Method, apparatus and device for monitoring state of data node and medium

Abstract

The application provides a state monitoring method, device and equipment of a data node and a medium. The method comprises the following steps: predicting a data volume change rate threshold in a first period to be executed according to historical data of the data node; obtaining first data processed by the data node in the first period, calculating an actual data volume change rate of the first data in the first period according to the first data; determining whether the data node is in an abnormal state based on the data volume change rate threshold and the actual data volume change rate, and performing an abnormal processing operation if the data node is in the abnormal state; adding the first data into the historical data, obtaining updated historical data, and using the updated historical data to predict a corresponding data volume change rate threshold in a second period; the second period is the next period adjacent to the first period. The scheme provided by the application can reduce false negatives and false positives in the data node state monitoring process.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data processing transmission, and particularly relates to a state monitoring method and device of a data node, equipment and a medium. BACKGROUND

[0002] In a system related to big data, cloud computing or the Internet of Things, a data node for data processing transmission needs to be monitored to ensure the stability and reliability of the data node and guarantee normal operation of the system.

[0003] In the related art, data flow passing through the data node is monitored, the relationship between the data flow and a flow threshold value is analyzed, and it is determined whether the data node is in an abnormal state.

[0004] However, the flow threshold value is generally a static value, and it is difficult to reflect the speed and trend of data change by analyzing the relationship between the data flow and the flow threshold value, which leads to false negatives or false positives of the related art solution. SUMMARY

[0005] The application aims to provide a state monitoring method, device, equipment and medium of a data node, which at least solves the problem of false negatives or false positives of the related art solution of monitoring the data node by using a static threshold value.

[0006] In a first aspect, an embodiment of the application discloses a state monitoring method of a data node, and the method comprises the following steps.

[0007] According to historical data of the data node, a data volume change rate threshold value in a first period to be executed is predicted, and the historical data comprises data processed by the data node in a historical period;

[0008] First data processed by the data node in the first period is obtained, and an actual measured data volume change rate of the first data in the first period is calculated according to the first data;

[0009] Based on the data volume change rate threshold value and the actual measured data volume change rate, it is determined whether the data node is in an abnormal state, and an abnormal processing operation is performed if the data node is in the abnormal state;

[0010] The first data is added into the historical data, updated historical data is obtained, and the updated historical data is used for predicting a corresponding data volume change rate threshold value in a second period; the second period is a next period adjacent to the first period.

[0011] In a second aspect, an embodiment of the application discloses a state monitoring device of a data node, and the device comprises the following steps.

[0012] The threshold adjusting module is configured to predict a data volume change rate threshold in a first period to be executed according to historical data of the data node, wherein the historical data includes data processed by the data node in a historical period.

[0013] The obtaining module is configured to obtain first data processed by the data node in the first period, and calculate a measured data volume change rate of the first data in the first period according to the first data.

[0014] The analyzing module is configured to determine whether the data node is in an abnormal state based on the data volume change rate threshold and the measured data volume change rate, and perform an abnormal processing operation in a case where the data node is in the abnormal state.

[0015] The historical data updating module is configured to add the first data into the historical data, and obtain updated historical data, wherein the updated historical data is used to predict a corresponding data volume change rate threshold in a second period, and the second period is a next period adjacent to the first period.

[0016] In a third aspect, an electronic device is disclosed, including a processor and a memory, the memory stores programs or instructions executable on the processor, and the programs or instructions are executed by the processor to implement the steps of the method according to the first aspect.

[0017] In a fourth aspect, a readable storage medium is disclosed, the readable storage medium stores programs or instructions, and the programs or instructions are executed by a processor to implement the steps of the method according to the first aspect.

[0018] In summary, in the embodiments of the present application, the data volume change rate threshold in the first period to be executed by the data node is predicted according to the historical data of the data node, in the first period, the first data processed by the data node is obtained and the measured data volume change rate is calculated, whether the data node is in an abnormal state is determined by comparing the measured data volume change rate and the data volume change rate threshold, in a case where the data node is in an abnormal state, an abnormal processing operation is performed, and the first data is recorded in the historical data to predict the data volume change rate threshold in the second period. Through the above technical solutions, the state of the data node is monitored by determining whether the data volume change rate in the period exceeds the threshold, and the dynamic adjustment mechanism of the data volume change rate threshold is realized by prediction based on the historical data. Compared with related technologies, the dynamic adjustment mechanism of the threshold in the present application is more suitable for the dynamic change of the data volume, the data volume change rate can reflect the change trend and speed of the data processed by the data node, so that the present application can more efficiently and accurately identify the abnormal state of the data node, and reduce the occurrence of false negatives or false positives. BRIEF DESCRIPTION OF DRAWINGS

[0019] In the drawings:

[0020] Figure 1 is a step flow chart of a data node state monitoring method provided by an embodiment of the present application;

[0021] Figure 2 is another step flow chart of a data node state monitoring method provided by an embodiment of the present application;

[0022] Figure 3 is still another step flow chart of a data node state monitoring method provided by an embodiment of the present application;

[0023] Figure 4 is a block diagram of a data node state monitoring apparatus provided by an embodiment of the present application;

[0024] Figure 5 is a block diagram of an electronic device of an embodiment provided by an embodiment of the present application;

[0025] Figure 6 is a block diagram of an electronic device of another embodiment provided by an embodiment of the present application. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0027] The terms "first", "second", and the like in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be exchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally a category and do not limit the number of objects, for example, the first object can be one or more. In addition, "and / or" in the specification and claims indicates at least one of the connected objects, and the character " / ", generally indicates that the front and rear associated objects are in an "or" relationship.

[0028] Reference Figure 1 , which shows a step flow chart of a data node state monitoring method provided by an embodiment of the present application, the method comprising:

[0029] Step 101, predicting a data volume change rate threshold in a first period to be executed according to historical data of the data node; the historical data includes data processed by the data node in a historical period.

[0030] The scheme of the present application first predicts the data volume change rate threshold of the first period to be executed according to the historical data of the data node; by periodically obtaining the data volume of the data processed by the data node, the periodic data volume change rate is calculated, and by comparing the data volume change rate and the data volume change rate threshold, the condition that the data node is in an abnormal state is further judged. In the scheme of the present application, the data volume change rate threshold can be a dynamic change value obtained according to the prediction of the historical data, and the data volume change rate threshold can dynamically change with the update of the historical data; the dynamic change of the control data volume change rate threshold realizes the dynamic adjustment mechanism of the data volume change rate threshold, and through the dynamic adjustment mechanism of the data volume change rate threshold, the dynamic change of the data volume can be adapted, so that the subsequent judgment according to the data volume change rate threshold is more efficient and accurate.

[0031] Illustratively, the historical data can be obtained from the running log file of the data node, or can be obtained by the way of burying points in the data node.

[0032] Illustratively, the first period can be one hour, half an hour.

[0033] Illustratively, the unit of data volume can be bit (bit, b), or byte (Byte, B).

[0034] Step 102, obtaining first data processed by the data node in the first period, and calculating a measured data volume change rate of the first data in the first period according to the first data.

[0035] In the monitoring method, by obtaining the first data processed by the data node in the first period, the total data volume processed by the data node in the first period, that is, the data volume of the first data, can be obtained, and the measured data volume change rate of the first data processed by the data node in the first period can be further calculated.

[0036] The measured data amount change rate can be calculated according to the total data amount. At the beginning time and the end time of the first period, two total data amounts can be obtained respectively. The total data amount at the beginning time can be consistent with the total data amount at the end time of the historical period adjacent to the first period. The total data amount at the end time is the total data amount accumulated from the beginning time to the end time. The difference between the total data amount at the end time of the first period and the total data amount at the beginning time of the first period is obtained. The ratio of the difference to the total data amount at the beginning time of the first period is obtained. The percentage of the ratio is the change rate of the total data amount processed by the data node in the first period, that is, the measured data amount change rate.

[0037] For example, the first data can be obtained through the running log file of the data node or the way of burying points monitoring.

[0038] For example, the measured data amount change rate can be 5%, 0, or -5%.

[0039] Step 103, based on the data amount change rate threshold and the measured data amount change rate, determining whether the data node is in an abnormal state, and performing an abnormal processing operation if the data node is in an abnormal state.

[0040] After calculating the measured data amount change rate, it is compared with the predicted data amount change rate threshold to determine whether the data node is in an abnormal state. It can be understood that when the measured data amount change rate is outside the range defined by the data amount change rate threshold, the data node is in an abnormal state.

[0041] For example, the predicted data amount change rate threshold is 5%, and the measured data amount change rate is 8%. It is determined that the data node is in an abnormal state. At this time, the abnormal processing operation of the data node is performed.

[0042] Step 104, adding the first data into the historical data to obtain updated historical data, the updated historical data being used for predicting a corresponding data amount change rate threshold in a second period; the second period being the next period adjacent to the first period.

[0043] The first data is added into the historical data to obtain updated historical data, and the updated historical data is used to predict a corresponding data volume change rate threshold in a second period to realize dynamic adjustment of the data volume change rate threshold, and the second period is a next period adjacent to the first period; through updating the historical data, the data volume change rate threshold is adjusted accordingly, and the adjusted data volume change rate threshold is compared with a measured data volume change rate of second data processed by the data node in the second period to determine whether the data node is in an abnormal state in the second period.

[0044] For example, the updated historical data can include a data volume change rate threshold of the first period predicted by a previous historical period of the first period.

[0045] In summary, in the embodiments of the present application, the data volume change rate threshold in the first period to be executed by the data node is predicted through the historical data of the data node; in the first period, the first data processed by the data node is obtained and a measured data volume change rate is calculated, whether the data node is in an abnormal state is determined by comparing the measured data volume change rate with the data volume change rate threshold, and an abnormal processing operation is performed in the case that the data node is in an abnormal state; the first data is recorded in the historical data to predict the data volume change rate threshold in the second period. Through the above technical solutions, the scheme of monitoring the state of the data node by judging whether the data volume change rate in the period exceeds the threshold is realized, and the dynamic adjustment mechanism of the data volume change rate threshold is realized through the prediction of the historical data. Compared with related technologies, the dynamic adjustment mechanism of the threshold of the present scheme is more suitable for the dynamic change of the data volume, and the data volume change rate can reflect the change trend and speed of the data processed by the data node, so that the present scheme can more efficiently and accurately identify the abnormal state of the data node and reduce the occurrence of false negatives or false positives.

[0046] Reference Figure 2 which actually shows another step flowchart of the state monitoring method of the data node provided by the embodiments of the present application, and the method comprises:

[0047] Step 201: obtaining a data volume change rate threshold in a third period; the third period is a historical period adjacent to the first period to be executed;

[0048] Steps 201 to 203 give an implementation of predicting the data volume change rate threshold in the first period. First, the data volume change rate threshold in the third period can be obtained from the historical data to make adjustment on the basis of the data volume change rate threshold in the third period in the subsequent steps to obtain the adjusted data volume change threshold in the third period.

[0049] The third period, the first period and the second period are three periods in sequence, for example, in the same day, the third period is 10:00-11:00, the first period is 11:00-12:00, and the second period is 12:00-13:00.

[0050] It can be understood that the third period and the first period are two adjacent periods, and the third period has occurred, and the third data processed by the data node in the third period is recorded in the historical data.

[0051] Step 202: statistics of the comparison result of the measured data amount change rate in each historical period and the data amount change rate threshold value, and the node state of the data node in each historical period;

[0052] The comparison result of the measured data amount change rate in each historical period and the data amount change rate threshold value, and the node state of the data node in each historical period are counted, so as to analyze the data amount change rate threshold value in the historical period, and determine how to adjust the data amount change rate threshold value in the third period in the subsequent step.

[0053] For example, the measured data amount change rate in the historical period can be less than / greater than / equal to the data amount change rate threshold value of the period.

[0054] Step 203: adjusting the data amount change rate threshold value in the third period according to the respective comparison result and the node state of all historical periods, and taking the adjusted data amount change threshold value in the third period as the data amount change rate threshold value in the first period to be executed;

[0055] The data amount change rate threshold value in the third period is adjusted by analyzing the respective comparison result and the node state of the historical period, and the adjusted data amount change rate in the third period will be taken as the predicted data amount change rate threshold value in the first period to be executed.

[0056] For example, the respective comparison result of all historical periods can be consistent, that is, the measured amount change rate in all historical periods is greater than the data amount change rate threshold value of the historical period, or the measured amount change rate in all historical periods is less than the data amount change rate threshold value of the historical period; the respective comparison result of all historical periods can be inconsistent, for example, for a part of the historical periods, the measured data amount change rate is less than the data amount change rate threshold value, and for another part of the historical periods, the measured data amount change rate is greater than the data amount change rate threshold value.

[0057] Optionally, step 203 can include:

[0058] Sub-step 2031: If there are multiple consecutive target historical periods with consistent comparison results and the node states of the target historical periods are all preset target states, adjust the data volume change rate threshold in the third period within a historical time period including multiple historical periods.

[0059] In the process of analyzing the comparison results in the historical periods, if there are multiple consecutive target historical periods and the node states of the target historical periods are all preset target states within a historical time period including multiple historical periods, it can be considered that the data volume change rate of the third period needs to be adjusted, wherein the target historical period represents consistent comparison results in the period.

[0060] For example, in a historical time period including 10 historical periods, the measured data volume change rates of 6 consecutive historical periods are all greater than the data volume change rate threshold, and the 6 historical periods are target periods. If the node states of the 6 historical periods are all preset target states, it can be considered that the data volume change rate of the third period needs to be adjusted. The 6 consecutive historical periods can include the third period, that is, the third period is the last historical period in the 6 consecutive historical periods.

[0061] Optionally, the ratio of the number of the multiple consecutive target historical periods with consistent comparison results to the total number of the historical periods in the historical time period is greater than or equal to 50%.

[0062] When the number of consecutive target historical periods is more than half of the number of historical periods in the selected historical time period, the node state of the data node can be further checked to determine whether the data volume change rate threshold in the third period needs to be adjusted.

[0063] It should be noted that the ratio greater than or equal to 50% is only one possible implementation scheme. The ratio of the number of consecutive target historical periods to the total number of historical periods in the historical time period also needs to be set according to the total number of historical periods of the data node.

[0064] Optionally, the target state includes an overrun state and a non-overrun state; the data node in the non-overrun state represents that the data volume input into the data node does not exceed the processing capacity limit value of the data node; and the data node in the overrun state represents that the data volume input into the data node exceeds the processing capacity limit value of the data node. Sub-step 2031 can include:

[0065] Sub-step 20311: If there are multiple consecutive target historical periods in the historical time period including multiple historical periods, the measured data volume change rate of each target historical period is greater than the data volume change rate threshold, and the node state of each target historical period is the non-overrun state, the data volume change rate threshold in the third period is increased.

[0066] Sub-step 20312: If there are multiple consecutive target historical periods in the historical time period including multiple historical periods, the measured data volume change rate of each target historical period is less than or equal to the data volume change rate threshold in the historical period, and the node state of each target historical period is the overrun state, the data volume change rate threshold in the third period is decreased.

[0067] Sub-step 20311 and sub-step 20312 describe the adjustment method of the data volume change rate threshold in the third period; the target state of the data node includes the overrun state and the non-overrun state, which is used to represent whether the input data node data volume exceeds the limit of the data node processing capability.

[0068] For example, the case that the data node is in the overrun state can be obtained by reading the running log file of the data node.

[0069] In the case that the measured data volume change rate of the consecutive target historical periods is greater than the data volume change rate threshold, and the node state of the target historical period is the non-overrun state, it can be considered that in the above consecutive target historical periods, the data volume change rate threshold has been maintained at a small level, which cannot well reflect the maximum value of the data volume change. In this case, it is easy to misreport that the data node is in an abnormal state, that is, it is easy to judge that the data node actually in the normal state is in the abnormal state. Therefore, the data volume change rate threshold in the third period needs to be increased to make the predicted data volume change rate threshold of the first period to be executed more accurate and reduce the occurrence of misreporting in the subsequent steps.

[0070] In the case that the measured data volume change rate of the consecutive target historical periods is less than or equal to the data volume change rate threshold, and the node state of the target historical period is the overrun state, it can be considered that in the above consecutive target historical periods, the data volume change rate threshold has been maintained at a high level, which cannot well reflect the minimum value of the data volume change. In this case, it is easy to miss report that the data node is in an abnormal state, that is, it is easy to judge that the data node actually in the abnormal state is in the normal state. Therefore, the data volume change rate threshold in the third period needs to be decreased to make the predicted data volume change rate threshold of the first period to be executed more accurate and reduce the occurrence of missing report in the subsequent steps.

[0071] Step 204: continuously acquiring data volume information from a first time point of the first period to a second time point of the first period; the first time point includes a start time point of the first period, and the second time point includes an end time point of the first period; the first data includes data volume information of data processed by the data node at each time point in the first period.

[0072] Step 205: accumulating a total data volume of the first data according to the data volume information acquired from the first time point to the second time point.

[0073] Step 206: calculating a measured data volume change rate of the first data in the first period in combination with a total data volume of third data processed by the data node in a third period and the total data volume of the first data.

[0074] Steps 204 to 206 provide a method for calculating a data volume change rate; data volume information is continuously acquired from a first time point of a first period, and the data volume information is accumulated to obtain a total data volume of first data in the first period. The data volume information herein represents a data volume attribute of data, i.e., a size of data stored, transmitted, and processed in a computer system.

[0075] For example, the data volume information at each acquisition time point can be acquired through a running log file of the data node or through a method of setting a buried point monitoring in the data node.

[0076] In the same way, a total data volume of a third period can be acquired; the third period is a historical period adjacent to the first period; a measured data volume change rate of the first period is calculated in combination with the total data volume of the first period and the total data volume of the third period. Specifically, the total data volume of the first period and the total data volume of the third period are first subtracted, and then the result of the subtraction is divided by the total data volume of the third period; a percentage of a result of the division is the measured data volume change rate in the first period.

[0077] For example, the total data volume of the first period is 105 TB, and the total data volume of the third period is 100 TB; thus, the measured data volume change rate is (105-100) / 100x100%=5%.

[0078] Step 207: confirming that the data node is in an abnormal state when the measured data volume change rate is greater than the data volume change rate threshold.

[0079] Step 208: confirming that the data node is in a normal state when the measured data volume change rate is less than or equal to the data volume change rate threshold.

[0080] Step 207 to step 208 provide an embodiment of determining whether the data node is in an abnormal state by comparing the measured quantity change rate and the data quantity change rate threshold; when the measured quantity change rate is greater than the data quantity change rate threshold, it is considered that the data node is in an abnormal state, otherwise, when the measured quantity change rate is less than or equal to the data quantity change rate threshold, it is considered that the data node is in a normal state.

[0081] For example, the measured data quantity change rate in the first period is 8%, and the data quantity change rate threshold is 5%, so it is considered that the data node is in an abnormal state in the first period.

[0082] It should be noted that steps 207 and 208 are only for the case that the measured data quantity change rate and the data quantity change rate threshold are both positive values, because the data quantity processed by the node is likely to continue to decrease, so that the total data quantity in the current period is lower than the total data quantity in the last period, resulting in that the measured data quantity change rate is negative; when the measured data quantity change rate and the data quantity change rate threshold are both negative values, the absolute values of the two are compared.

[0083] For example, the measured data quantity change rate in the first period is -8%, and the data quantity change rate threshold is -5%, then |-8%| = 8% and |-5%| = 5% are compared, since 8% is greater than 5%, it is considered that the data node is in an abnormal state in the first period.

[0084] Step 209: in the case that it is determined that the data node is in an abnormal state, determining an abnormal type of the data node in the abnormal state according to the first data.

[0085] After confirming that the data node is in an abnormal state, the abnormal type can be further analyzed according to the obtained first data, so as to execute the abnormal processing operation.

[0086] For example, the abnormal type can include data loss, data congestion, node failure, etc., and the data loss can be further divided into data loss caused by network and data loss caused by equipment, and the node failure can be further divided into hardware failure and software failure.

[0087] Step 210: executing an abnormal processing operation matched with the abnormal type on the data node.

[0088] After analyzing the abnormal type, the related system device is controlled to execute the abnormal processing operation matched with the abnormal type, so as to solve the abnormality.

[0089] For example, the abnormal processing operation can include: starting a data recovery process for data loss, adjusting data routing for data congestion, and sending an alarm notification for hardware failure, etc.

[0090] Optionally, the first data further comprises data volume related information of data processed by the data node at each time point in the first period, and before step 209, the method further comprises:

[0091] Step 211: continuously acquiring the data volume related information from a first time point of the first period to a second time point of the first period; the first time point comprises a start time point of the first period, and the second time point comprises an end time point of the first period.

[0092] After step 209, the method further comprises:

[0093] Step 212: correcting the case that the data node is in an abnormal state according to the data volume related information.

[0094] Steps 211 and 212 provide another embodiment of comparing the data volume change rate with the data volume change rate threshold, in the acquired first data, not only the data volume information directly reflecting the data processing volume of the data node, but also the data volume related information, such as the number of data processed by the data node, the data size, the data volume related information can be converted into data volume information through a preset manner, and then the equivalent data volume change rate is calculated, thereby providing another comparison dimension for whether the data node is in an abnormal state, and thus the data volume related information can be collected and acquired, and the result of confirming the case that the data node is in an abnormal state in step 209 is corrected through the data volume related information.

[0095] In step 211, the data volume information and the data volume related information of the data node can be acquired simultaneously with step 204, that is, from the first time point of the first period to the second time point of the first period, the data volume information and the data volume related information of the data node are continuously acquired.

[0096] Optionally, step 212 can comprise:

[0097] Sub-step 2121: converting the data volume related information into equivalent data volume information of data processed by the data node through a preset conversion manner.

[0098] Sub-step 2122: accumulating the equivalent total data volume of the first data according to the equivalent data volume information acquired from the first time point to the second time point.

[0099] Sub-step 2123: combining the total data volume of the third data processed by the data node in the third period and the equivalent total data volume in the first period to calculate the equivalent data volume change rate of the first data in the first period.

[0100] Sub-step 2124: in combination with the equivalent data amount change rate, the measured data amount change rate and the data amount change rate threshold, the case that the data node is in abnormal state is corrected.

[0101] Sub-steps 2121 to 2124 represent the preparation steps before the case that the data node is in abnormal state is corrected by the data amount related information; including, the continuously collected data amount related information is converted into equivalent data amount information by a preset conversion method, and then the equivalent data amount information is converted.

[0102] For example, the data amount related information includes data number information, the total number of data obtained by step 211 is 1000, each data content is the same, and the equivalent data amount information of each data is converted into 8MB, so the equivalent total data amount is 1000x8=8000MB.

[0103] At this time, the equivalent data amount change rate in the first period can be calculated based on the equivalent total data amount of the first period and the measured total data amount of the third period or the equivalent total data amount of the third period, so as to correct the case that the data node is in abnormal state.

[0104] Optionally, sub-step 2124 can include:

[0105] Sub-step 21241: in the case that any one of the equivalent data amount change rate and the measured data amount change rate is greater than the data amount change rate threshold, it is determined that the data node is in abnormal state.

[0106] Sub-step 21242: in the case that the equivalent data amount change rate and the measured data amount change rate are both less than or equal to the data amount change rate threshold, it is determined that the data node is in normal state.

[0107] Sub-steps 21241 and 21242 represent two cases of correcting the case that the data node is in abnormal state by the equivalent data amount change rate; generally, the measured data amount change rate and the equivalent data amount change rate should be consistent or close, when the data node is in abnormal state due to node failure, there will be a large deviation between the two, therefore, the relationship between the two and the data amount change rate threshold can be analyzed, in the case that any one of the two is greater than the data amount change rate threshold, it can be considered that the data node fails, and the corresponding data node is in abnormal state. In the case that both are less than the data amount change rate threshold, it can be considered that the data node does not fail, and since the measured data change amount threshold is also less than the data amount change threshold, it can be further considered that the data node is in normal state.

[0108] Reference Figure 3Fig. 7 shows another flow chart of the method for monitoring the state of the data node according to an embodiment of the application, which comprises the following steps:

[0109] Step S1: determining a unit time period;

[0110] This step is equivalent to setting a first period for the data node. First, a unit time period is determined, which can be one hour, half an hour, etc., and can be adjusted according to the frequency requirement of monitoring the data node.

[0111] Step S2: collecting and recording data multiple times;

[0112] Step S3: forming a data volume comparison set;

[0113] Step S2 and Step S3 are equivalent to obtaining the total amount of data of the third data in the third period in Step S206 above, that is, the data volume information of the data node is continuously collected and recorded at the starting moment of the unit time period, and the data volume comparison set is formed by accumulation, thereby increasing the accuracy of the subsequent steps.

[0114] Step S4: collecting data in the next time period;

[0115] This step is equivalent to Step 204 above, and it can be understood that the unit time period and the next time period here are equivalent to the third period and the first period above, respectively.

[0116] Step S5: calculating the data volume change rate;

[0117] The data volume of the current time period is compared with the data volume of the previous time period, and the data volume change rate is calculated.

[0118] Step S6: determining whether the change rate exceeds a threshold value;

[0119] The calculated data volume change rate is compared with the change rate threshold value, so as to determine whether the data volume change is normal. If the change rate is within the change rate threshold value range, it is considered to be normal; otherwise, if the change rate exceeds the threshold value range, it is considered to be abnormal. It can be understood that the data anomaly here is equivalent to the abnormal state of the data node in the above.

[0120] Step S7: confirming data anomaly in the case where the change rate exceeds the threshold value;

[0121] Step S8: triggering a response mechanism;

[0122] The data volume anomaly enters the abnormality judgment and response phase, further analyzes the possible reasons, such as data loss, data congestion, node failure or other reasons, and triggers the corresponding response mechanism according to the analyzed abnormal type, such as sending an alarm notification, starting a data recovery process, adjusting data routing, etc. Abnormal handling measures.

[0123] Step S7 and step S8 correspond to step 209 and step 210 above.

[0124] Step S9: continuously monitor in the case of the change rate not exceeding the threshold value;

[0125] Step S10: dynamically adjust the threshold value in the process of continuous monitoring;

[0126] The process of dynamically adjusting the threshold value corresponds to steps 101 and 104 above, that is, dynamically adjusting the change rate threshold value according to the data amount change in the historical data, and it needs to be noted that the step of dynamically adjusting the threshold value can be independent of steps S1 to S9.

[0127] In summary, in the embodiments of the present application, the data amount change rate threshold value in the first period to be executed by the data node is predicted through the historical data of the data node; in the first period, the first data processed by the data node is obtained and the measured data amount change rate is calculated, the data node is judged to be in an abnormal state by comparing the measured data amount change rate with the data amount change rate threshold value; in the case of the data node being in an abnormal state, an abnormal processing operation is performed; the first data is recorded in the historical data to predict the data amount change rate threshold value of the second period. Through the above technical solution, the scheme of monitoring the state of the data node by judging whether the data amount change rate in the period exceeds the threshold value is realized, and the dynamic adjustment mechanism of the data amount change rate threshold value is realized through the historical data prediction. Compared with related technologies, the dynamic adjustment mechanism of the threshold value of the present scheme is more suitable for the dynamic change of the data amount, and the data amount change rate can reflect the change trend and change speed of the data processed by the data node, so that the present scheme can more efficiently and accurately identify the abnormal state of the data node, and reduce the occurrence of false negatives or false positives.

[0128] Reference Figure 4 which shows a data node state monitoring device 30 provided by an embodiment of the present application, the device 30 comprises:

[0129] The threshold adjustment module 301 is configured to predict the data amount change rate threshold value in the first period to be executed according to the historical data of the data node; the historical data includes data processed by the data node in the historical period;

[0130] The acquisition module 302 is configured to acquire the first data processed by the data node in the first period, and calculate the measured data amount change rate of the first data in the first period according to the first data;

[0131] The analysis and execution module 303 is configured to determine whether the data node is in an abnormal state based on the data amount change rate threshold value and the measured data amount change rate, and perform an abnormal processing operation in the case of the data node being in an abnormal state.

[0132] The updating module 304 is configured to add the first data into the historical data to obtain updated historical data, and the updated historical data is used to predict a corresponding data volume change rate threshold in a second period; the second period is a next period adjacent to the first period.

[0133] Optionally, the threshold adjusting module 301 can comprise:

[0134] The threshold obtaining sub-module is configured to obtain a data volume change rate threshold in a third period; the third period is a historical period adjacent to the first period.

[0135] The statistical sub-module is configured to statistically obtain a comparison result of a measured data volume change rate and a data volume change rate threshold in each historical period, and a node state of the data node in each historical period.

[0136] The adjustment executing sub-module is configured to adjust the data volume change rate threshold in the third period according to the comparison result and the node state of each historical period, and take the adjusted data volume change threshold in the third period as the data volume change rate threshold in the first period to be executed.

[0137] Optionally, the adjustment executing sub-module can comprise:

[0138] The adjustment executing unit is configured to, in a historical time period comprising a plurality of historical periods, if there are a plurality of target historical periods with consistent comparison results and the node states of the target historical periods are all preset target states, adjust the data volume change rate threshold in the third period.

[0139] Optionally, the target state comprises an overrun state and a non-overrun state; the data node in the non-overrun state represents that a data volume of data processed by the data node does not exceed a processing capacity limit value of the data node; the data node in the overrun state represents that the data volume of data processed by the data node exceeds the processing capacity limit value of the data node; and the adjustment executing unit can comprise:

[0140] The first adjustment executing sub-unit is configured to, in a historical time period comprising a plurality of historical periods, if there are a plurality of target historical periods with measured data volume change rates greater than the data volume change rate threshold and the node states of the target historical periods are all non-overrun states, increase the data volume change rate threshold in the third period.

[0141] The second adjustment execution subunit is configured to, in a historical time period including a plurality of historical periods, if there are a plurality of continuous target historical periods, and the measured data volume change rate of each of the target historical periods is less than or equal to the data volume change rate threshold in the historical period, and the node state of each of the target historical periods is an overrun state, reduce the data volume change rate threshold in the third period.

[0142] Optionally, the adjustment execution unit can include:

[0143] The first confirmation subunit is configured to confirm that the ratio of the number of the plurality of continuous target historical periods to the sum of the number of the historical periods in the historical time period is greater than or equal to 50%.

[0144] Optionally, the analysis execution module 303 can include:

[0145] The first analysis submodule is configured to, in the case that the measured data volume change rate is greater than the data volume change rate threshold, confirm that the data node is in an abnormal state.

[0146] The second analysis submodule is configured to, in the case that the measured data volume change rate is less than or equal to the data volume change rate threshold, confirm that the data node is in a normal state.

[0147] Optionally, the analysis execution module 303 can include:

[0148] The abnormality determination submodule is configured to, in the case that it is determined that the data node is in an abnormal state, determine an abnormal type of the data node in the abnormal state according to the first data.

[0149] The abnormality processing submodule is configured to perform an abnormality processing operation matched with the abnormal type on the data node.

[0150] Optionally, the first data includes data volume information of data processed by the data node at each time in the first period, and the acquisition module 302 can include:

[0151] The data volume acquisition submodule is configured to continuously acquire the data volume information from a first time of the first period to a second time of the first period; the first time includes a start time of the first period, and the second time includes an end time of the first period.

[0152] The first accumulation submodule is configured to accumulate the total data volume of the first data according to the data volume information acquired from the first time to the second time.

[0153] The first calculating sub-module is configured to calculate a measured data amount change rate of the first data in the first period by combining a total data amount of third data processed by the data node in a third period and a total data amount of the first data, wherein the third period is a historical period adjacent to the first period.

[0154] Optionally, the apparatus 30 further comprises:

[0155] The data amount related information obtaining module is configured to continuously obtain the data amount related information from a first time point of the first period to a second time point of the first period, wherein the first time point comprises a start time point of the first period, and the second time point comprises an end time point of the first period.

[0156] The analysis result correcting module is configured to correct the situation that the data node is in an abnormal state according to the data amount related information.

[0157] Optionally, the analysis result correcting module can comprise:

[0158] The conversion sub-module is configured to convert the data amount related information into equivalent data amount information of data processed by the data node by a preset conversion manner.

[0159] The second accumulating sub-module is configured to accumulate the equivalent total data amount of the first data by the equivalent data amount information obtained from the first time point to the second time point.

[0160] The second calculating sub-module is configured to calculate an equivalent data amount change rate of the first data in the first period by combining a total data amount of third data processed by the data node in a third period and the equivalent total data amount in the first period.

[0161] The correction executing sub-module is configured to correct the situation that the data node is in an abnormal state by combining the equivalent data amount change rate, the measured data amount change rate and the data amount change rate threshold.

[0162] Optionally, the correction executing sub-module can comprise:

[0163] The first executing unit is configured to confirm that the data node is in an abnormal state when any one of the equivalent data amount change rate and the measured data amount change rate is greater than the data amount change rate threshold.

[0164] The second executing unit is configured to confirm that the data node is in a normal state when both the equivalent data amount change rate and the measured data amount change rate are less than or equal to the data amount change rate threshold.

[0165] In summary, in the embodiment of the present application, the state monitoring device of the data node is used to predict the data volume change rate threshold in the first period to be executed by the data node through the historical data of the data node; in the first period, the first data processed by the data node is acquired and the measured data volume change rate is calculated, and whether the data node is in an abnormal state is judged by comparing the measured data volume change rate with the data volume change rate threshold; in the case that the data node is in an abnormal state, an abnormal processing operation is executed; the first data is recorded in the historical data to predict the data volume change rate threshold in the second period. Through the above technical solution, the scheme of monitoring the state of the data node by judging whether the data volume change rate in the period exceeds the threshold is realized, and the dynamic adjustment mechanism of the data volume change rate threshold is realized through the historical data prediction. Compared with the related art, the dynamic adjustment mechanism of the threshold of the present scheme is more suitable for the dynamic change of the data volume, and the data volume change rate can reflect the change trend and change speed of the data processed by the data node, so that the present scheme can more efficiently and accurately identify the abnormal state of the data node, and reduce the occurrence of false or false reporting. Through the present scheme, the related problems caused by the abnormal state of the data node can be found and solved in time, which helps to improve the stability of the system where the data node is located.

[0166] Referring to Figure 5 The electronic device 500 can include one or more of the following components: a processing component 502, a memory 504, a power supply component 506, a multimedia component 508, an audio component 510, an input / output (I / O) interface 512, a sensor component 514, and a communication component 516.

[0167] The processing component 502 usually controls the overall operation of the electronic device 500, such as operations associated with displaying, making phone calls, data communications, camera operations, and recording operations. The processing component 502 can include one or more processors 520 to execute instructions to complete all or part of the steps of the methods described above. In addition, the processing component 502 can include one or more modules to facilitate interaction between the processing component 502 and other components. For example, the processing component 502 can include a multimedia module to facilitate the interaction between the multimedia component 508 and the processing component 502.

[0168] The memory 504 is used for storing various types of data to support the operation of the electronic device 500. Examples of these data include instructions for any application programs or methods operating on the electronic device 500, contact data, phonebook data, messages, pictures, multimedia, etc. The memory 504 can be implemented by any type of volatile or nonvolatile memory devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0169] The power supply component 506 supplies power for various components of the electronic device 500. The power supply component 506 can include a power management system, one or more power supplies, and other components associated with generating, managing and distributing power for the electronic device 500.

[0170] The multimedia component 508 includes an interface for outputting various types of media content between the electronic device 500 and a user. In some embodiments, the interface can include a liquid crystal display (LCD) and a touch panel (TP). If the interface includes the touch panel, the interface can be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors for sensing touch, swiping and gestures on the touch panel. The touch sensor can not only sense the boundary of a touch or swiping action, but also detect duration and pressure associated with the touch or swiping action. In some embodiments, the multimedia component 508 includes a front camera and / or a rear camera. The front and / or rear camera can receive external multimedia data when the electronic device 500 is in an operation mode, such as a shooting mode or a multimedia mode. Each of the front and rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0171] The audio component 510 is for outputting and / or inputting audio signals. For example, the audio component 510 includes a microphone (MIC) for receiving external audio signals when the electronic device 500 is in an operation mode, such as a call mode, a recording mode and a voice recognition mode. The received audio signals can be further stored in the memory 504 or transmitted via the communication component 516. In some embodiments, the audio component 510 also includes a speaker for outputting audio signals.

[0172] The input / output (I / O) interface 512 provides an interface between the processing component 502 and peripheral interface modules, which can be a keyboard, a click wheel, a button, etc. These buttons can include, but are not limited to, a home button, a volume button, a start button and a lock button.

[0173] The sensor component 514 includes one or more sensors to provide the electronic device 500 with state assessments of various aspects. For example, the sensor component 514 can detect the open / closed position of the electronic device 500, the relative positioning of components, such as the display and keypad of the electronic device 500, the sensor component 514 can also detect changes in position of the electronic device 500 or a component of the electronic device 500, the presence or absence of user contact with the electronic device 500, the orientation or acceleration / deceleration of the electronic device 500, and the temperature changes of the electronic device 500. The sensor component 514 can include an optical sensor for detecting ambient light, a proximity sensor configured to detect the presence of nearby objects without any physical touch, or a CMOS or CCD image sensor for use in imaging applications. In some embodiments, the sensor component 514 can further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0174] The communication component 516 is configured to facilitate wired or wireless communication between the electronic device 500 and other devices. The electronic device 500 can access a wireless network based on a communication standard, such as WiFi, a cellular network standard (such as 2G, 3G, 4G, or 5G), or a combination thereof. In an example embodiment, the communication component 516 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In an example embodiment, the communication component 516 further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technology.

[0175] In an example embodiment, the electronic device 500 can be implemented with one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, or other electronic elements to implement a method for monitoring the state of a data node.

[0176] In an example embodiment, a non-transitory computer-readable storage medium including instructions, such as the memory 504 including instructions, is also provided, which can be executed by the processor 520 of the electronic device 500 to implement the above-described method. For example, the non-transitory storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disc, and an optical data storage device, etc.

[0177] Figure 6FIG. 6 is a block diagram of an electronic device 600 according to another embodiment of the present application. For example, the electronic device 600 can be provided as a server. Referring to FIG. 6, the electronic device 600 includes a processing component 622, which further includes one or more processors, and a memory resource represented by a memory 632 for storing instructions, such as an application program, executable by the processing component 622. The application program stored in the memory 632 can include one or more modules each corresponding to a set of instructions. In addition, the processing component 622 is configured to execute the instructions to perform a method for monitoring a state of a data node according to an embodiment of the present application. Figure 6 The electronic device 600 can further include a power supply component 626 configured to perform power management of the electronic device 600, a wired or wireless network interface 650 configured to connect the electronic device 600 to a network, and an input / output (I / O) interface 658. The electronic device 600 can operate based on an operating system stored in the memory 632, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or the like.

[0178] The electronic device 600 can further include a power supply component 626 configured to perform power management of the electronic device 600, a wired or wireless network interface 650 configured to connect the electronic device 600 to a network, and an input / output (I / O) interface 658. The electronic device 600 can operate based on an operating system stored in the memory 632, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or the like.

[0179] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the application being indicated by the following claims.

[0180] It is to be understood that the application is not limited to the precise construction described in the specification and shown in the drawings, and various modifications and changes can be effected therein by those skilled in the art without departing from the scope of the application. The scope of the application is to be limited only by the appended claims.

Claims

1. A method of state monitoring of a data node, characterized by, The method comprises: obtaining a data volume change rate threshold in a third period; the third period is a historical period adjacent to a first period; continuously counting comparison results of a measured data volume change rate and the data volume change rate threshold in each historical period, and a node state of the data node in each historical period; in a historical time period comprising a plurality of historical periods, if there are a plurality of continuous target historical periods, and the measured data volume change rates of the target historical periods are all greater than the data volume change rate threshold, and the node states of the target historical periods are all un-over-limit states, increasing the data volume change rate threshold in the third period; the data node in the un-over-limit state indicates that the data volume input into the data node does not exceed the processing capacity limit value of the data node; the data node in the over-limit state indicates that the data volume input into the data node exceeds the processing capacity limit value of the data node; in the historical time period comprising a plurality of historical periods, if there are a plurality of continuous target historical periods, and the measured data volume change rates of the target historical periods are all less than or equal to the data volume change rate threshold in the historical period, and the node states of the target historical periods are all over-limit states, decreasing the data volume change rate threshold in the third period; taking the adjusted data volume change threshold in the third period as a data volume change rate threshold to be executed in the first period; obtaining first data processed by the data node in the first period, and calculating a measured data volume change rate of the first data in the first period according to the first data; determining whether the data node is in an abnormal state based on the data volume change rate threshold and the measured data volume change rate, and performing an abnormal processing operation in the case that the data node is in the abnormal state; adding the first data into historical data, obtaining updated historical data, and the updated historical data is used for predicting a corresponding data volume change rate threshold in a second period; the second period is a next period adjacent to the first period; the historical data comprises data processed by the data node in the historical period.

2. The method of claim 1, wherein, The ratio of the number of the plurality of continuous target historical periods to the total number of the historical periods in the historical time period is greater than or equal to 50%.

3. The method of claim 1, wherein, The determination of whether the data node is in the abnormal state based on the data volume change rate threshold and the measured data volume change rate comprises: in the case that the measured data volume change rate is greater than the data volume change rate threshold, confirming that the data node is in the abnormal state; in the case that the measured data volume change rate is less than or equal to the data volume change rate threshold, confirming that the data node is in a normal state.

4. The method of claim 1, wherein, The performance of the abnormal processing operation in the case that the data node is in the abnormal state comprises: in the case that it is determined that the data node is in the abnormal state, determining an abnormal type of the data node in the abnormal state according to the first data; performing an abnormal processing operation matched with the abnormal type on the data node.

5. The method of claim 1, wherein, The first data includes data volume information of data processed by the data node at each time point in the first period; the first data processed by the data node in the first period is obtained, and based on the first data, a measured data volume change rate of the first data in the first period is calculated, including: The data volume information is continuously obtained from a first time point of the first period to a second time point of the first period; the first time point includes a start time point of the first period, and the second time point includes an end time point of the first period; The total data volume of the first data is accumulated based on the data volume information obtained from the first time point to the second time point; The measured data volume change rate of the first data in the first period is calculated in combination with a total data volume of third data processed by the data node in a third period and the total data volume of the first data; the third period is a historical period adjacent to the first period.

6. The method of claim 1, wherein, The first data further includes data volume related information of data processed by the data node at each time point in the first period; before the step of determining whether the data node is in an abnormal state based on the data volume change rate threshold and the measured data volume change rate, the method further includes: The data volume related information is continuously obtained from a first time point of the first period to a second time point of the first period; the first time point includes a start time point of the first period, and the second time point includes an end time point of the first period; After the step of determining whether the data node is in an abnormal state based on the data volume change rate threshold and the measured data volume change rate, the method further includes: The case that the data node is in an abnormal state is corrected based on the data volume related information.

7. The method of claim 6, wherein, The case that the data node is in an abnormal state is corrected based on the data volume related information, including: The data volume related information is converted into equivalent data volume information of data processed by the data node by a preset conversion method; The equivalent total data volume of the first data is accumulated based on the equivalent data volume information obtained from the first time point to the second time point; An equivalent data volume change rate of the first data in the first period is calculated in combination with a total data volume of third data processed by the data node in a third period and the equivalent total data volume in the first period; The case that the data node is in an abnormal state is corrected in combination with the equivalent data volume change rate, the measured data volume change rate, and the data volume change rate threshold.

8. The method of claim 7, wherein, The case that the data node is in an abnormal state is corrected in combination with the equivalent data volume change rate, the measured data volume change rate, and the data volume change rate threshold, including: In a case that any one of the equivalent data volume change rate and the measured data volume change rate is greater than the data volume change rate threshold, it is determined that the data node is in an abnormal state; In a case that both the equivalent data volume change rate and the measured data volume change rate are less than or equal to the data volume change rate threshold, it is determined that the data node is in a normal state.

9. An apparatus for monitoring the status of a data node, the apparatus comprising: The device includes: The third period acquisition module is configured to acquire a data volume change rate threshold in a third period; the third period is a historical period adjacent to the first period; The statistical module is configured to continuously count a comparison result of a measured data volume change rate and the data volume change rate threshold in each historical period, and a node state of the data node in each historical period; The first adjustment module is configured to, in a historical time period including a plurality of historical periods, if there are a plurality of continuous target historical periods whose measured data volume change rates are all greater than the data volume change rate threshold, and the node states of the target historical periods are all in an un-over-limit state, increase the data volume change rate threshold in the third period; the data node in the un-over-limit state indicates that a data volume input into the data node does not exceed a processing capacity limit value of the data node; the data node in the over-limit state indicates that the data volume input into the data node exceeds the processing capacity limit value of the data node; The second adjustment module is configured to, in a historical time period including a plurality of historical periods, if there are a plurality of continuous target historical periods whose measured data volume change rates are all less than or equal to the data volume change rate threshold in the historical period, and the node states of the target historical periods are all in an over-limit state, decrease the data volume change rate threshold in the third period; The threshold adjustment module is configured to take the adjusted data volume change threshold in the third period as a data volume change rate threshold to be executed in the first period; the historical data includes data processed by the data node in the historical period; The acquisition module is configured to acquire first data processed by the data node in the first period, and calculate a measured data volume change rate of the first data in the first period according to the first data; The analysis and execution module is configured to determine whether the data node is in an abnormal state based on the data volume change rate threshold and the measured data volume change rate, and perform an abnormal processing operation if the data node is in the abnormal state; The historical data update module is configured to add the first data into historical data, acquire updated historical data, and use the updated historical data to predict a corresponding data volume change rate threshold in a second period; the second period is a next period adjacent to the first period; the historical data includes data processed by the data node in the historical period.

10. An electronic device, comprising: A processor and a memory are included, the memory stores programs or instructions executable on the processor, and the programs or instructions are executed by the processor to implement steps of the method in any one of claims 1 to 8.

11. A readable storage medium, characterized by, The readable storage medium stores programs or instructions, and the programs or instructions are executed by the processor to implement steps of the method in any one of claims 1 to 8.

Images