Data node state monitoring method and device, equipment and medium

Abstract

The invention provides a data node state monitoring method and device, equipment and a medium, and the method comprises the steps: predicting a to-be-executed data volume change rate threshold value in a first period according to the historical data of a data node; acquiring first data processed by the data node in the first period, and calculating an actually measured data volume change rate of the first data in the first period according to the first data; determining whether the data node is in an abnormal state or not based on the data volume change rate threshold and the actually measured data volume change rate, and executing an exception handling operation when the data node is in the abnormal state; the first data is added into historical data, updated historical data is obtained, and the updated historical data is used for a data volume change rate threshold value corresponding to second period prediction; the second period is a next period adjacent to the first period. By adopting the scheme provided by the invention, missing report and false report in a data node state monitoring process can be reduced.

Description

Technical Field

[0001] The present application belongs to the field of data processing and transmission technology, and specifically relates to a data node status monitoring method, device, equipment and medium. Background Art

[0002] In big data, cloud computing, or IoT related systems, it is necessary to monitor the data nodes that perform data processing and transmission to ensure the stability and reliability of the data nodes and the normal operation of the system.

[0003] In the related art, the data flow through the data node is monitored, the relationship between the data flow and the flow threshold is analyzed, and it is determined whether the data node has an abnormal condition.

[0004] However, the traffic threshold is generally a static value. It is difficult to reflect the speed and trend of data changes by analyzing the relationship between data traffic and the traffic threshold, which makes it easy for related technical solutions to miss or misreport abnormal conditions. Summary of the invention

[0005] The present application aims to provide a data node status monitoring method, device, equipment and medium, which at least solves the problem of underreporting or false reporting of abnormal conditions caused by using static thresholds to monitor data nodes in related technologies.

[0006] In a first aspect, an embodiment of the present application discloses a method for monitoring the status of a data node, the method comprising:

[0007] Predicting a data volume change rate threshold within a first cycle to be executed based on historical data of the data node; the historical data includes data processed by the data node within the historical cycle;

[0008] Acquire first data processed by the data node in a first period, and calculate, based on the first data, a measured data volume change rate of the first data in the first period;

[0009] Based on the data volume change rate threshold and the measured data volume change rate, determining whether the data node is in an abnormal state, and performing an abnormality handling operation if the data node is in an abnormal state;

[0010] The first data is added into the historical data to obtain updated historical data, and the updated historical data is used for predicting the corresponding data volume change rate threshold for the second cycle; the second cycle is the next cycle adjacent to the first cycle.

[0011] In a second aspect, an embodiment of the present application discloses a data node status monitoring device, the device comprising:

[0012] A threshold adjustment module: used to predict a data volume change rate threshold within a first cycle to be executed based on historical data of the data node; the historical data includes data processed by the data node within the historical cycle;

[0013] An acquisition module: configured to acquire first data processed by the data node in a first period, and calculate a measured data volume change rate of the first data in the first period according to the first data;

[0014] Analysis module: used to determine whether the data node is in an abnormal state based on the data volume change rate threshold and the measured data volume change rate, and perform an abnormality handling operation when the data node is in an abnormal state;

[0015] A historical data updating module is used to add the first data into the historical data to obtain updated historical data, wherein the updated historical data is used to predict the corresponding data volume change rate threshold for the second cycle; the second cycle is the next cycle adjacent to the first cycle.

[0016] In a third aspect, an embodiment of the present application further discloses an electronic device, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0017] In a fourth aspect, an embodiment of the present application further discloses a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0018] In summary, in an embodiment of the present application, the data volume change rate threshold within the first cycle to be executed by the data node is predicted through the historical data of the data node; in the first cycle, the first data processed by the data node is obtained and the measured data volume change rate is calculated, and by comparing the measured data volume change rate and the data volume change rate threshold, it is determined whether the data node is in an abnormal state; when the data node is in an abnormal state, an exception handling operation is performed; the first data is recorded in the historical data for predicting the data volume change rate threshold of the second cycle. Through the above technical scheme, a scheme for monitoring the state of the data node by judging whether the data volume change rate within the cycle exceeds the threshold is implemented, and a dynamic adjustment mechanism of the data volume change rate threshold is implemented through historical data prediction. Compared with the related technology, the dynamic adjustment mechanism of the threshold of the present scheme is more adaptable to the dynamic changes of the data volume, and the data volume change rate can reflect the changing trend and change speed of the data processed by the data node, so that the present scheme can more efficiently and accurately identify the abnormal state of the data node and reduce the occurrence of underreporting or false alarms. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In the attached picture:

[0020] Figure 1 It is a step flow chart of a method for monitoring the status of a data node provided in an embodiment of the present application;

[0021] Figure 2 It is another step flow chart of the data node status monitoring method provided in an embodiment of the present application;

[0022] Figure 3 It is another step flow chart of the data node status monitoring method provided in the embodiment of the present application;

[0023] Figure 4 It is a block diagram of a data node status monitoring device provided in an embodiment of the present application;

[0024] Figure 5 is a block diagram of an electronic device according to an embodiment of the present application;

[0025] Figure 6 It is a block diagram of an electronic device of another embodiment provided by the embodiments of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0028] refer to Figure 1 , which shows a step flow chart of a data node status monitoring method provided by an embodiment of the present application, the method comprising:

[0029] Step 101: predicting a data volume change rate threshold within a first cycle to be executed based on historical data of the data node; the historical data includes data processed by the data node within the historical cycle.

[0030] The present application scheme first predicts the data volume change rate threshold of the first cycle to be executed based on the historical data of the data node; by periodically obtaining the data volume of the data node processing data, calculating the periodic data volume change rate, and comparing the data volume change rate and the data volume change rate threshold, further determine whether the data node is in an abnormal state. In the present application scheme, the data volume change rate threshold can be a dynamic change value obtained based on the historical data prediction, and the data volume change rate threshold can change dynamically with the update of historical data; controlling the dynamic change of the data volume change rate threshold realizes the dynamic adjustment mechanism of the data volume change rate threshold. Through the dynamic adjustment mechanism of the data volume change rate threshold, it can adapt to the dynamic change of the data volume, so that the subsequent judgment based on the data volume change rate threshold is more efficient and accurate.

[0031] Exemplarily, historical data can be obtained from the operation log file of the data node, or by monitoring the data node at the embedded point.

[0032] Exemplarily, the first period may be one hour or half an hour.

[0033] Exemplarily, the unit of the data amount may be bit (bit, b) or byte (Byte, B).

[0034] Step 102: Acquire first data processed by the data node in a first period, and calculate a measured data volume change rate of the first data in the first period based on the first data.

[0035] In the monitoring method, by obtaining the first data processed by the data node in the first cycle, the total amount of data processed by the data node in the first cycle, that is, the amount of the first data, can be obtained, and further the measured data amount change rate of the first data processed by the data node in the first cycle can be calculated;

[0036] The measured data volume change rate can be calculated based on the total data volume. At the start and end times of the first cycle, two total data volumes can be obtained respectively, the total data volume at the start time and the total data volume at the end time. It can be understood that the total data volume at the start time can be consistent with the total data volume at the end time of the historical cycle adjacent to the first cycle. The total data volume at the end time is the total data volume continuously accumulated from the start time to the end time. By subtracting the total data volume at the end time of the first cycle from the total data volume at the start time of the first cycle, and comparing the difference result with the total data volume at the start time of the first cycle, the percentage of the ratio obtained is the change rate of the total data volume processed by the data node in the first cycle, that is, the measured data volume change rate.

[0037] Exemplarily, the first data can be obtained through an operation log file of a data node or through point-of-care monitoring.

[0038] Exemplarily, the measured data volume change rate may be 5%, 0, or -5%.

[0039] Step 103: Based on the data volume change rate threshold and the measured data volume change rate, determine whether the data node is in an abnormal state, and perform an abnormality handling operation if the data node is in an abnormal state.

[0040] After calculating the measured data volume change rate, it is compared with the predicted data volume change rate threshold to determine whether the data node is in an abnormal state. It can be understood that when the measured data volume change rate is outside the range specified by the data volume change rate threshold, the data node is in an abnormal state.

[0041] Exemplarily, if the predicted data volume change rate threshold is 5%, and the measured data volume change rate is 8%, it is determined that the data node is in an abnormal state, and at this time, an abnormal processing operation of the data node will be executed.

[0042] Step 104: Add the first data into the historical data to obtain updated historical data, where the updated historical data is used to predict the corresponding data volume change rate threshold for the second cycle; the second cycle is the next cycle adjacent to the first cycle.

[0043] The first data is added into the historical data to obtain the updated historical data, and the updated historical data is used for predicting the corresponding data volume change rate threshold for the second cycle to realize dynamic adjustment of the data volume change rate threshold, and the second cycle is the next cycle adjacent to the first cycle; by updating the historical data, the data volume change rate threshold is adjusted accordingly, and the adjusted data volume change rate threshold will be compared with the actual data volume change rate of the second data processed by the data node in the second cycle in the second cycle to determine whether the data node is in an abnormal state in the second cycle.

[0044] Exemplarily, the updated historical data may include a data volume change rate threshold of the first period predicted by a previous historical period of the first period.

[0045] In summary, in an embodiment of the present application, the data volume change rate threshold within the first cycle to be executed by the data node is predicted through the historical data of the data node; in the first cycle, the first data processed by the data node is obtained and the measured data volume change rate is calculated, and by comparing the measured data volume change rate and the data volume change rate threshold, it is determined whether the data node is in an abnormal state; when the data node is in an abnormal state, an exception handling operation is performed; the first data is recorded in the historical data for predicting the data volume change rate threshold of the second cycle. Through the above technical scheme, a scheme for monitoring the state of the data node by judging whether the data volume change rate within the cycle exceeds the threshold is implemented, and a dynamic adjustment mechanism of the data volume change rate threshold is implemented through historical data prediction. Compared with the related technology, the dynamic adjustment mechanism of the threshold of the present scheme is more adaptable to the dynamic changes of the data volume, and the data volume change rate can reflect the changing trend and change speed of the data processed by the data node, so that the present scheme can more efficiently and accurately identify the abnormal state of the data node and reduce the occurrence of underreporting or false alarms.

[0046] refer to Figure 2 , in fact, shows another step flow chart of the data node status monitoring method provided by an embodiment of the present application, the method comprising:

[0047] Step 201: obtaining a data volume change rate threshold in a third cycle; the third cycle is a historical cycle adjacent to the first cycle to be executed;

[0048] Steps 201 to 203 provide an implementation method for predicting the data volume change rate threshold of the first period. First, the data volume change rate threshold of the third period can be obtained from the historical data, so that the subsequent steps can make adjustments based on the data volume change rate threshold of the third period to obtain the adjusted data volume change threshold in the third period.

[0049] The third cycle, the first cycle, and the second cycle are three consecutive cycles. For example, in the same day, the third cycle is 10:00-11:00, the first cycle is 11:00-12:00, and the second cycle is 12:00-13:00.

[0050] It can be understood that the third cycle and the first cycle are two adjacent cycles, and the third cycle has occurred, and the third data processed by the data node in the third cycle is recorded in the historical data.

[0051] Step 202: Counting the comparison results of the actual data volume change rate in each historical period and the data volume change rate threshold, as well as the node status of the data node in each historical period;

[0052] The comparison results of the measured data volume change rate and the data volume change rate threshold in the statistical historical period, as well as the node status of the data node in each historical period, are used to analyze the data volume change rate threshold in the historical period, so as to determine how to adjust the data volume change rate threshold for the third period in the subsequent steps.

[0053] Exemplarily, the measured data volume change rate in a historical period may be less than / greater than / equal to a data volume change rate threshold of the period.

[0054] Step 203: adjusting the data volume change rate threshold in the third cycle according to the comparison results and node states of all historical cycles, and using the adjusted data volume change rate threshold in the third cycle as the data volume change rate threshold in the first cycle to be executed;

[0055] By analyzing the comparison results and node states of each historical cycle, the data volume change rate threshold in the third cycle is adjusted, and the adjusted data volume change rate in the third cycle will be used as the predicted data volume change rate threshold in the first cycle to be executed.

[0056] Exemplarily, the comparison results of all historical periods may be consistent, that is, the measured quantity change rate in all historical periods is greater than the data volume change rate threshold of the historical period, or the measured quantity change rate in all historical periods is less than the data volume change rate threshold of the historical period; the comparison results of all historical periods may be inconsistent, for example, for some historical periods, the measured data volume change rate is less than the data volume change rate threshold, and for another part of the historical periods, the measured data volume change rate is greater than the data volume change rate threshold.

[0057] Optionally, step 203 may include:

[0058] Sub-step 2031: In a historical time period including multiple historical periods, if there are multiple consecutive target historical periods with consistent comparison results, and the node states of the target historical periods are all preset target states, then adjust the data volume change rate threshold within the third period.

[0059] In the process of analyzing the comparison results within the historical cycle, in a historical time period including multiple historical cycles, if there are multiple consecutive target historical cycles, and the node states of the target historical cycles are all preset target states, then it can be considered that the data volume change rate of the third cycle needs to be adjusted, where the comparison results represented by the target historical cycle are consistent in the cycle.

[0060] For example, in a historical time period including 10 historical cycles, if the measured data volume change rate of 6 consecutive historical cycles is greater than the data volume change rate threshold, then these 6 historical cycles are target cycles. If the node states of these 6 historical cycles are all preset target states, it can be considered that the quantity change rate of the third cycle needs to be adjusted; these 6 consecutive historical cycles can include the third cycle, that is, the third cycle is the last historical cycle of these 6 consecutive historical cycles.

[0061] Optionally, the ratio of the number of target historical cycles with consistent multiple consecutive comparison results to the total number of historical cycles in the historical time period is greater than or equal to 50%.

[0062] When the number of consecutive target historical cycles accounts for half or more of the number of historical cycles in the selected historical time period, the node status of the data node can be further checked to determine whether the data volume change rate threshold in the third cycle needs to be adjusted;

[0063] It should be noted that a ratio greater than or equal to 50% is only a feasible implementation plan. The ratio of the number of consecutive target historical cycles to the total number of historical cycles in the historical time period also needs to be set according to the total number of historical cycles of the data nodes.

[0064] Optionally, the target state includes an over-limit state and a non-over-limit state; the data node being in the non-over-limit state indicates that the amount of data input to the data node does not exceed the processing capacity limit of the data node; the data node being in the over-limit state indicates that the amount of data input to the data node exceeds the processing capacity limit of the data node; sub-step 2031 may include:

[0065] Sub-step 20311: In a historical time period including multiple historical periods, if there are multiple consecutive target historical periods whose measured data volume change rates are greater than the data volume change rate threshold, and the node status of the target historical periods is not exceeded, the data volume change rate threshold in the third period is increased.

[0066] Sub-step 20312: In a historical time period including multiple historical cycles, if there are multiple consecutive target historical cycles whose measured data volume change rates are less than or equal to the data volume change rate threshold in the historical cycle, and the node status of the target historical cycle is all in an over-limit status, lower the data volume change rate threshold in the third cycle.

[0067] Sub-steps 20311 and 20312 describe the specific method of adjusting the data volume change rate threshold within the third period; the target state of the data node includes an over-limit state and a non-over-limit state, which are used to characterize whether the input data volume of the data node exceeds the limit of the data node processing capacity.

[0068] Exemplarily, the situation in which a data node is in an over-limit state can be obtained by reading a running log file of the data node.

[0069] When the measured data volume change rate of consecutive target historical cycles is greater than the data volume change rate threshold, and the node status of the target historical cycle is not exceeded, it can be considered that in the above consecutive target historical cycles, the data volume change rate threshold has been maintained at a relatively small level and cannot well reflect the maximum value of the data volume change. In this case, it is easy to falsely report that the data node is in an abnormal state, that is, it is easy to judge that the data node is in an abnormal state when it is actually in a normal state. In this regard, it is necessary to increase the data volume change rate threshold in the third cycle to make the predicted data volume change rate threshold of the first cycle to be executed more accurate and reduce the occurrence of false alarms in subsequent steps.

[0070] When the measured data volume change rate of consecutive target historical cycles is less than or equal to the data volume change rate threshold, and the node status of the target historical cycle is all in the over-limit state, it can be considered that in the above-mentioned consecutive target historical cycles, the data volume change rate threshold has been maintained at a high level and cannot well reflect the minimum value of the data volume change. In this case, it is easy to miss the data node in an abnormal state, that is, it is easy to judge the data node that is actually in an abnormal state as being in a normal state. In this regard, it is necessary to lower the data volume change rate threshold in the third cycle to make the predicted data volume change rate threshold of the first cycle to be executed more accurate, so as to reduce the occurrence of missed reports in subsequent steps.

[0071] Step 204: Continuously acquire data volume information from the first moment of the first cycle to the second moment of the first cycle; the first moment includes the start moment of the first cycle, and the second moment includes the end moment of the first cycle; the first data includes the data volume information of the data processed by the data node at each moment in the first cycle.

[0072] Step 205: According to the data volume information obtained from the first moment to the second moment, the total data volume of the first data is accumulated and obtained.

[0073] Step 206: Calculate the actual data volume change rate of the first data in the first period by combining the total data volume of the third data processed by the data node in the third period and the total data volume of the first data.

[0074] Steps 204 to 206 provide a method for calculating the rate of change of data volume; starting from the first moment of the first cycle, data volume information is continuously acquired and accumulated to obtain the total data volume of the first data in the first cycle. The data volume information here characterizes the data volume attribute of the data, that is, the size of the data stored, transmitted and processed in the computer system.

[0075] Exemplarily, the data volume information at each collection moment can be obtained through the operation log file of the data node, and can also be obtained by setting up point-of-care monitoring on the data node.

[0076] In the same way, the total data volume of the third cycle can be obtained, and the third cycle is a historical cycle adjacent to the first cycle; the total data volume of the first cycle and the total data volume of the third cycle are combined to calculate the measured data volume change rate of the first cycle. Specifically, the total data volume of the first cycle and the total data volume of the third cycle are first subtracted, and then the difference result is compared with the total data volume of the third cycle. The percentage of the ratio result is the measured data volume change rate in the first cycle.

[0077] Exemplarily, the total data volume of the first cycle is 105 TB, and the total data volume of the third cycle is 100 TB, then the measured data volume change rate is (105-100) / 100×100%=5%.

[0078] Step 207: When the measured data volume change rate is greater than the data volume change rate threshold, confirm that the data node is in an abnormal state.

[0079] Step 208: When the measured data volume change rate is less than or equal to the data volume change rate threshold, confirm that the data node is in a normal state.

[0080] Steps 207 to 208 provide an implementation method for determining whether a data node is in an abnormal state by comparing the measured quantity change rate and the data volume change rate threshold; when the measured quantity change rate is greater than the data volume change rate threshold, the data node is considered to be in an abnormal state; conversely, when the measured quantity change rate is less than or equal to the data volume change rate threshold, the data node is considered to be in a normal state.

[0081] Exemplarily, if the measured data volume change rate in the first cycle is 8% and the data volume change rate threshold is 5%, it is considered that the data node is in an abnormal state in the first cycle.

[0082] It should be noted that step 207 and step 208 are only for the case where both the measured data volume change rate and the data volume change rate threshold are positive values, because the amount of data processed by the node is likely to continue to decline, making the total amount of data in the current cycle lower than the total amount of data in the previous cycle, resulting in the measured data volume change rate being a negative value; when the measured data volume change rate and the data volume change rate threshold are both negative values, it is necessary to take the absolute values ​​of the two for comparison.

[0083] Exemplarily, the measured data volume change rate in the first cycle is -8%, and the data volume change rate threshold is -5%. Then, by comparing |-8%|=8% and |-5%|=5%, since 8% is greater than 5%, it is considered that the data node in the first cycle is in an abnormal state.

[0084] Step 209: When it is determined that the data node is in an abnormal state, determine the abnormality type of the data node in the abnormal state according to the first data.

[0085] After confirming that the data is in an abnormal state at some point, the abnormality type can be further analyzed based on the acquired first data to facilitate the execution of the abnormality handling operation.

[0086] Exemplarily, the abnormality types may include data loss, data congestion, node failure, etc. Data loss can be further divided into data loss caused by network reasons and data loss caused by equipment reasons. Node failure can be further divided into hardware failure and software failure.

[0087] Step 210: Execute an exception handling operation matching the exception type on the data node.

[0088] After analyzing the exception type, control the relevant system equipment to perform exception handling operations that match the exception type to resolve the exception.

[0089] Exemplarily, the exception handling operation may include: starting a data recovery process for data loss, adjusting data routing for data congestion, sending an alarm notification for hardware failure, etc.

[0090] Optionally, the first data further includes information related to the amount of data processed by the data node at each moment in the first cycle. Before step 209, the following may also be included:

[0091] Step 211: starting from a first moment of the first cycle to a second moment of the first cycle, continuously acquiring information related to the data volume; the first moment includes a start moment of the first cycle, and the second moment includes an end moment of the first cycle.

[0092] After step 209, the following steps may also be included:

[0093] Step 212: Correct the abnormal state of the data node according to the data volume related information.

[0094] Step 211 and step 212 provide another implementation method for comparing the data volume conversion rate and the data volume change rate threshold. The first data obtained includes not only the data volume information that directly reflects the data processing volume of the data node, but also the data volume related information that can indirectly reflect the data processing volume of the data node, such as the number of data items processed by the data node and the data size. The data volume related information can be converted into data volume information in a preset manner, and then the equivalent data volume change rate is calculated, thereby providing another comparison dimension for whether the data node is in an abnormal state. Therefore, the data volume related information can be collected and obtained, and the result of confirming that the data node is in an abnormal state in step 209 can be corrected through the data volume related information.

[0095] Among them, step 211 can be executed simultaneously with the above step 204, that is, from the first moment of the first cycle to the end of the second cycle of the first cycle, the data volume information and data volume related information of the data node are continuously obtained.

[0096] Optionally, step 212 may include:

[0097] Sub-step 2121: convert the data volume related information into equivalent data volume information of the data processed by the data node through a preset conversion method.

[0098] Sub-step 2122: According to the equivalent data amount information obtained from the first moment to the second moment, the equivalent total data amount of the first data is accumulated and obtained.

[0099] Sub-step 2123: Calculate the change rate of the equivalent data volume of the first data in the first cycle by combining the total data volume of the third data processed by the data node in the third cycle and the equivalent total data volume in the first cycle.

[0100] Sub-step 2124: combining the equivalent data volume change rate, the measured data volume change rate and the data volume change rate threshold, correcting the situation where the data node is in an abnormal state.

[0101] Sub-steps 2121 to 2124 describe the preparatory steps before correcting the abnormal state of the data node through data volume related information; including converting the continuously collected data volume related information into equivalent data volume information through a preset conversion method, and then converting the equivalent data volume information.

[0102] Exemplarily, the data volume related information includes the number of data items. The total number of data items acquired through step 211 is 1000, each data item has the same content, and the equivalent data volume information of each data item is converted to 8MB, so the equivalent total data volume is 1000×8=8000MB.

[0103] At this time, the change rate of the equivalent data volume in the first cycle can be calculated based on the total amount of equivalent data in the first cycle and the total amount of measured data in the third cycle or the total amount of equivalent data in the third cycle to correct the situation where the data node is in an abnormal state.

[0104] Optionally, sub-step 2124 may include:

[0105] Sub-step 21241: when any one of the equivalent data volume change rate and the measured data volume change rate is greater than the data volume change rate threshold, confirm that the data node is in an abnormal state.

[0106] Sub-step 21242: When the equivalent data volume change rate and the measured data volume change rate are both less than or equal to the data volume change rate threshold, confirm that the data node is in a normal state.

[0107] Sub-steps 21241 and 21242 describe two situations in which the data node is in an abnormal state by correcting the equivalent data volume change rate; under normal circumstances, the measured data volume change rate and the equivalent data volume change rate should be consistent or close. When the data node is in an abnormal state due to a node failure, there will be a large deviation between the two. For this reason, the relationship between the two and the data volume change rate threshold can be analyzed. When either of the two is greater than the data volume change rate threshold, it can be considered that the data node is faulty and the corresponding data node is in an abnormal state. When both are less than the data volume change rate threshold, it can be considered that the data node is not faulty. Since the measured data change threshold is also less than the data volume change threshold, it can be further considered that the data node is in a normal state.

[0108] refer to Figure 3, which shows another step flow chart of the data node status monitoring method provided by an embodiment of the present application, the method comprising:

[0109] Step S1: Determine a unit time period;

[0110] This step is equivalent to setting the first cycle for the data node. First, determine the unit time period, which can be one hour, half an hour, etc., and can be adjusted according to the frequency requirements of monitoring data nodes.

[0111] Step S2: Collect and record data multiple times;

[0112] Step S3: forming a data volume comparison set;

[0113] Step S2 and step S3 are equivalent to obtaining the total amount of data of the third data of the third cycle in the above step S206, that is, at the beginning of the unit time period, continuously collect and record the data amount information of the data nodes, and form a data amount comparison set by accumulation to increase the accuracy of subsequent steps.

[0114] Step S4: collect data in the next time period;

[0115] This step is equivalent to the above step 204. It can be understood that the unit time period and the next time period here are respectively equivalent to the third cycle and the first cycle in the above text.

[0116] Step S5: Calculate the data volume change rate;

[0117] Compare the data volume of the current time period with the data volume of the previous time period and calculate the data volume change rate.

[0118] Step S6: Determine whether the change rate exceeds a threshold;

[0119] The calculated data volume change rate is compared with the change rate threshold to determine whether the data volume change is normal. If the change rate is within the change rate threshold, the data is considered normal; conversely, if the change rate exceeds the threshold, it is considered abnormal. It can be understood that the data abnormality here is equivalent to the data node being in an abnormal state in the above text.

[0120] Step S7: when the rate of change exceeds the threshold, confirm that the data is abnormal;

[0121] Step S8: triggering a response mechanism;

[0122] If the data volume is abnormal, the system will enter the abnormal judgment and response stage to further analyze the possible causes, such as data loss, data congestion, node failure or other reasons. Then, the corresponding response mechanism will be triggered according to the analyzed abnormal type, such as sending alarm notifications, starting data recovery process, adjusting data routing and other abnormal handling measures.

[0123] Step S7 and step S8 are equivalent to the above step 209 and step 210.

[0124] Step S9: When the rate of change does not exceed the threshold, continue monitoring;

[0125] Step S10: Dynamically adjust the threshold during continuous monitoring;

[0126] The process of dynamically adjusting the threshold is equivalent to step 101 and step 104 above, that is, dynamically adjusting the change rate threshold according to the change in the amount of data in the historical data. It should be noted that the step of dynamically adjusting the threshold can be independent of steps S1 to S9.

[0127] In summary, in an embodiment of the present application, the data volume change rate threshold within the first cycle to be executed by the data node is predicted through the historical data of the data node; in the first cycle, the first data processed by the data node is obtained and the measured data volume change rate is calculated, and by comparing the measured data volume change rate and the data volume change rate threshold, it is determined whether the data node is in an abnormal state; when the data node is in an abnormal state, an exception handling operation is performed; the first data is recorded in the historical data for predicting the data volume change rate threshold of the second cycle. Through the above technical scheme, a scheme for monitoring the state of the data node by judging whether the data volume change rate within the cycle exceeds the threshold is implemented, and a dynamic adjustment mechanism of the data volume change rate threshold is implemented through historical data prediction. Compared with the related technology, the dynamic adjustment mechanism of the threshold of the present scheme is more adaptable to the dynamic changes of the data volume, and the data volume change rate can reflect the changing trend and change speed of the data processed by the data node, so that the present scheme can more efficiently and accurately identify the abnormal state of the data node and reduce the occurrence of underreporting or false alarms.

[0128] refer to Figure 4 , which shows a data node status monitoring device 30 provided in an embodiment of the present application, the device 30 includes:

[0129] Threshold adjustment module 301: used to predict the data volume change rate threshold in the first cycle to be executed according to the historical data of the data node; the historical data includes the data processed by the data node in the historical cycle;

[0130] Acquisition module 302: used to acquire first data processed by the data node in a first period, and calculate the measured data volume change rate of the first data in the first period according to the first data;

[0131] Analysis execution module 303: used to determine whether the data node is in an abnormal state based on the data volume change rate threshold and the measured data volume change rate, and perform an abnormality handling operation when the data node is in an abnormal state;

[0132] Update module 304: used to add the first data into the historical data to obtain updated historical data, wherein the updated historical data is used to predict the corresponding data volume change rate threshold for the second cycle; the second cycle is the next cycle adjacent to the first cycle.

[0133] Optionally, the threshold adjustment module 301 may include:

[0134] A threshold acquisition submodule, used to acquire a data volume change rate threshold in a third period; the third period is a historical period adjacent to the first period;

[0135] The statistical submodule is used to count the comparison results of the actual data volume change rate in each historical period and the data volume change rate threshold, as well as the node status of the data node in each historical period.

[0136] The adjustment execution submodule is used to adjust the data volume change rate threshold within the third cycle according to the comparison results and node status of all historical cycles, and use the adjusted data volume change threshold within the third cycle as the data volume change rate threshold within the first cycle to be executed.

[0137] Optionally, the adjustment execution submodule may include:

[0138] The adjustment execution unit is used to adjust the data volume change rate threshold within the third period if there are multiple consecutive target historical periods with consistent comparison results within a historical time period including multiple historical periods, and the node states of the target historical periods are all preset target states.

[0139] Optionally, the target state includes an over-limit state and a non-over-limit state; the data node being in the non-over-limit state indicates that the amount of data processed by the data node does not exceed the processing capacity limit of the data node; the data node being in the over-limit state indicates that the amount of data processed by the data node exceeds the processing capacity limit of the data node; the adjustment execution unit may include:

[0140] The first adjustment execution subunit is used to increase the data volume change rate threshold in the third period if there are multiple consecutive target historical periods with measured data volume change rates greater than the data volume change rate threshold within the historical time period including multiple historical periods, and the node states of the target historical periods are all in a non-exceeding state;

[0141] The second adjustment execution sub-unit is used to reduce the data volume change rate threshold in the third period if, within a historical time period including multiple historical periods, there are multiple consecutive target historical periods in which the measured data volume change rates are less than or equal to the data volume change rate threshold in the historical period, and the node states of the target historical periods are all in an over-limit state.

[0142] Optionally, adjusting the execution unit may include:

[0143] The first confirmation subunit is used to confirm that the ratio of the number of cycles of the multiple consecutive target historical cycles to the total number of historical cycles in the historical time period is greater than or equal to 50%.

[0144] Optionally, the analysis execution module 303 may include:

[0145] A first analysis submodule, configured to confirm that the data node is in an abnormal state when the measured data volume change rate is greater than the data volume change rate threshold;

[0146] The second analysis submodule is used to confirm that the data node is in a normal state when the measured data volume change rate is less than or equal to the data volume change rate threshold.

[0147] Optionally, the analysis execution module 303 may include:

[0148] Abnormal determination submodule: used to determine the abnormal type of the abnormal state of the data node according to the first data when determining that the data node is in an abnormal state

[0149] Exception handling submodule: used to perform an exception handling operation matching the exception type on the data node.

[0150] Optionally, the first data includes data volume information of data processed by the data node at each time in the first period, and the acquisition module 302 may include:

[0151] A data volume acquisition submodule: used to continuously acquire the data volume information from a first moment of the first cycle to a second moment of the first cycle; the first moment includes a start moment of the first cycle, and the second moment includes an end moment of the first cycle;

[0152] A first accumulation submodule: accumulating and acquiring a total data volume of the first data according to the data volume information acquired from the first moment to the second moment;

[0153] The first calculation submodule is used to calculate the measured data volume change rate of the first data in the first cycle by combining the total data volume of the third data processed by the data node in the third cycle and the total data volume of the first data; the third cycle is a historical cycle adjacent to the first cycle.

[0154] Optionally, the device 30 further includes:

[0155] a data volume related information acquisition module, configured to continuously acquire the data volume related information from a first moment of the first cycle to a second moment of the first cycle; the first moment includes a start moment of the first cycle, and the second moment includes an end moment of the first cycle;

[0156] The analysis result correction module is used to correct the situation where the data node is in an abnormal state according to the information related to the data volume.

[0157] Optionally, the analysis result correction module may include:

[0158] A conversion submodule, used to convert the data volume related information into equivalent data volume information of the data processed by the data node through a preset conversion method;

[0159] A second accumulation submodule, configured to accumulate and obtain an equivalent total data volume of the first data according to the equivalent data volume information obtained from the first moment to the second moment;

[0160] A second calculation submodule, configured to calculate a change rate of an equivalent data volume of the first data in the first cycle by combining a total data volume of the third data processed by the data node in the third cycle and an equivalent total data volume in the first cycle;

[0161] The correction execution submodule is used to correct the situation where the data node is in an abnormal state by combining the equivalent data volume change rate, the measured data volume change rate and the data volume change rate threshold.

[0162] Optionally, the modification execution submodule may include:

[0163] A first execution unit, configured to confirm that the data node is in an abnormal state when any one of the equivalent data volume change rate and the measured data volume change rate is greater than the data volume change rate threshold;

[0164] The second execution unit is used to confirm that the data node is in a normal state when the equivalent data volume change rate and the measured data volume change rate are both less than or equal to the data volume change rate threshold.

[0165] In summary, in an embodiment of the present application, a state monitoring device for a data node is used to predict a data volume change rate threshold within a first cycle to be executed by the data node through historical data of the data node; in the first cycle, the first data processed by the data node is obtained and the measured data volume change rate is calculated, and by comparing the measured data volume change rate with the data volume change rate threshold, it is determined whether the data node is in an abnormal state; when the data node is in an abnormal state, an abnormal processing operation is performed; the first data is recorded in the historical data for predicting the data volume change rate threshold of the second cycle. Through the above technical scheme, a scheme for monitoring the state of the data node by judging whether the data volume change rate within the cycle exceeds the threshold is implemented, and a dynamic adjustment mechanism of the data volume change rate threshold is implemented through historical data prediction. Compared with the related technology, the dynamic adjustment mechanism of the threshold of the present scheme is more adaptable to the dynamic change of the data volume, and the data volume change rate can reflect the change trend and change speed of the data processed by the data node, so that the present scheme can more efficiently and accurately identify the abnormal state of the data node, and reduce the occurrence of underreporting or false alarms; through the present scheme, relevant problems caused by the abnormal state of the data node can be immediately discovered and solved, which helps to improve the stability of the system where the data node is located.

[0166] Reference Figure 5 , the electronic device 500 may include one or more of the following components: a processing component 502 , a memory 504 , a power component 506 , a multimedia component 508 , an audio component 510 , an input / output (I / O) interface 512 , a sensor component 514 , and a communication component 516 .

[0167] The processing component 502 generally controls the overall operation of the electronic device 500, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 502 may include one or more processors 520 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 502 may include one or more modules to facilitate the interaction between the processing component 502 and other components. For example, the processing component 502 may include a multimedia module to facilitate the interaction between the multimedia component 508 and the processing component 502.

[0168] The memory 504 is used to store various types of data to support the operation of the electronic device 500. Examples of such data include instructions for any application or method operating on the electronic device 500, contact data, phone book data, messages, pictures, multimedia, etc. The memory 504 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0169] The power supply component 506 provides power to the various components of the electronic device 500. The power supply component 506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 500.

[0170] The multimedia component 508 includes an interface that provides an output interface between the electronic device 500 and the user. In some embodiments, the interface may include a liquid crystal display (LCD) and a touch panel (TP). If the interface includes a touch panel, the interface may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 508 includes a front camera and / or a rear camera. When the electronic device 500 is in an operating mode, such as a shooting mode or a multimedia mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0171] The audio component 510 is used to output and / or input audio signals. For example, the audio component 510 includes a microphone (MIC), and when the electronic device 500 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is used to receive an external audio signal. The received audio signal can be further stored in the memory 504 or sent via the communication component 516. In some embodiments, the audio component 510 also includes a speaker for outputting audio signals.

[0172] The input / output I / O interface 512 provides an interface between the processing component 502 and the peripheral interface module, which may be a keyboard, a click wheel, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0173] The sensor assembly 514 includes one or more sensors for providing various aspects of status assessment for the electronic device 500. For example, the sensor assembly 514 can detect the open / closed state of the electronic device 500, the relative positioning of components, such as the display and keypad of the electronic device 500, and the sensor assembly 514 can also detect the position change of the electronic device 500 or a component of the electronic device 500, the presence or absence of user contact with the electronic device 500, the orientation or acceleration / deceleration of the electronic device 500, and the temperature change of the electronic device 500. The sensor assembly 514 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 514 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 514 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0174] The communication component 516 is used to facilitate wired or wireless communication between the electronic device 500 and other devices. The electronic device 500 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 516 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 516 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0175] In an exemplary embodiment, the electronic device 500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to implement a data node status monitoring method provided in an embodiment of the present application.

[0176] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 504 including instructions, and the instructions can be executed by a processor 520 of an electronic device 500 to perform the above method. For example, the non-transitory storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0177] Figure 6600 is a block diagram of an electronic device 600 according to another embodiment of the present invention. For example, the electronic device 600 may be provided as a server. Figure 6 The electronic device 600 includes a processing component 622, which further includes one or more processors, and a memory resource represented by a memory 632 for storing instructions that can be executed by the processing component 622, such as an application. The application stored in the memory 632 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 622 is configured to execute instructions to execute a data node status monitoring method provided in an embodiment of the present application.

[0178] The electronic device 600 may also include a power supply component 626 configured to perform power management of the electronic device 600, a wired or wireless network interface 650 configured to connect the electronic device 600 to a network, and an input / output (I / O) interface 658. The electronic device 600 may operate based on an operating system stored in the memory 632, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0179] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the application disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0180] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A data node status monitoring method, characterized in that: The method comprises: Predicting a data volume change rate threshold within a first cycle to be executed based on historical data of the data node; the historical data includes data processed by the data node within the historical cycle; Acquire first data processed by the data node in a first period, and calculate, based on the first data, a measured data volume change rate of the first data in the first period; Based on the data volume change rate threshold and the measured data volume change rate, determining whether the data node is in an abnormal state, and performing an abnormality handling operation if the data node is in an abnormal state; The first data is added into the historical data to obtain updated historical data, and the updated historical data is used for predicting the corresponding data volume change rate threshold for the second cycle; the second cycle is the next cycle adjacent to the first cycle.

2. The method according to claim 1, characterized in that The step of predicting the data volume change rate threshold in the first cycle to be executed based on historical data includes: Acquire a data volume change rate threshold within a third period; the third period is a historical period adjacent to the first period; Continuously counting the comparison results of the measured data volume change rate and the data volume change rate threshold in each historical period, as well as the node status of the data node in each historical period; According to the comparison results and node states of all historical cycles, the data volume change rate threshold in the third cycle is adjusted, and the adjusted data volume change rate threshold in the third cycle is used as the data volume change rate threshold in the first cycle to be executed.

3. The method according to claim 2, characterized in that The step of adjusting the data volume change rate threshold in the third period according to the comparison results and node states of all the historical periods includes: In a historical time period including multiple historical periods, if there are multiple consecutive target historical periods with consistent comparison results, and the node states of the target historical periods are all preset target states, then the data volume change rate threshold in the third period is adjusted.

4. The method according to claim 3, characterized in that The target state includes an over-limit state and a non-over-limit state; the data node is in the non-over-limit state, which indicates that the amount of data input to the data node does not exceed the processing capacity limit of the data node; the data node is in the over-limit state, which indicates that the amount of data input to the data node exceeds the processing capacity limit of the data node; within the historical time period including multiple historical periods, if there are multiple consecutive target historical periods with consistent comparison results, and the node states of the target historical periods are all preset target states, then adjusting the data amount change rate threshold within the third period, including: In a historical time period including multiple historical periods, if there are multiple consecutive target historical periods where the measured data volume change rates are all greater than the data volume change rate threshold, and the node states of the target historical periods are all in a non-exceeding state, the data volume change rate threshold in the third period is increased; In a historical time period including multiple historical cycles, if there are multiple consecutive target historical cycles whose measured data volume change rates are less than or equal to the data volume change rate threshold in the historical cycle, and the node status of the target historical cycle is all in an over-limit state, the data volume change rate threshold in the third cycle is lowered.

5. The method according to claim 3, characterized in that: The ratio of the number of target historical cycles with consistent multiple consecutive comparison results to the total number of historical cycles in the historical time period is greater than or equal to 50%.

6. The method according to claim 1, characterized in that The determining whether the data node is in an abnormal state based on the data volume change rate threshold and the measured data volume change rate includes: When the measured data volume change rate is greater than the data volume change rate threshold, confirming that the data node is in an abnormal state; When the measured data volume change rate is less than or equal to the data volume change rate threshold, it is confirmed that the data node is in a normal state.

7. The method according to claim 1, characterized in that When the data node is in an abnormal state, performing an abnormal processing operation includes: In the case where it is determined that the data node is in an abnormal state, determining, according to the first data, an abnormality type of the data node being in an abnormal state; An exception handling operation matching the exception type is performed on the data node.

8. The method according to claim 1, characterized in that: The first data includes data volume information of data processed by the data node at each moment in the first cycle; the obtaining the first data processed by the data node in the first cycle and calculating the measured data volume change rate of the first data in the first cycle according to the first data includes: From a first moment of the first cycle to a second moment of the first cycle, the data volume information is continuously acquired; the first moment includes a start moment of the first cycle, and the second moment includes an end moment of the first cycle; Accumulate and obtain a total data volume of the first data according to the data volume information obtained from the first moment to the second moment; Combined with the total data volume of the third data processed by the data node in the third period and the total data volume of the first data, the measured data volume change rate of the first data in the first period is calculated; the third period is a historical period adjacent to the first period.

9. The method according to claim 1, characterized in that: The first data also includes information related to the amount of data processed by the data node at each moment in the first cycle; before the step of determining whether the data node is in an abnormal state based on the data amount change rate threshold and the measured data amount change rate, the step also includes: Starting from a first moment of the first cycle to an end of a second moment of the first cycle, continuously acquiring information related to the data volume; the first moment includes a start moment of the first cycle, and the second moment includes an end moment of the first cycle; After the step of determining whether the data node is in an abnormal state based on the data volume change rate threshold and the measured data volume change rate, the method further includes: According to the data volume related information, the situation that the data node is in an abnormal state is corrected.

10. The method according to claim 9, characterized in that The correcting the abnormal state of the data node according to the data volume related information includes: Convert the data volume related information into equivalent data volume information of the data processed by the data node through a preset conversion method; Accumulating and acquiring the equivalent total data volume of the first data according to the equivalent data volume information obtained from the first moment to the second moment; Calculate the change rate of the equivalent data volume of the first data in the first cycle by combining the total data volume of the third data processed by the data node in the third cycle and the equivalent total data volume in the first cycle; The situation where the data node is in an abnormal state is corrected by combining the equivalent data volume change rate, the measured data volume change rate and the data volume change rate threshold.

11. The method according to claim 10, characterized in that The correcting the situation where the data node is in an abnormal state by combining the equivalent data volume change rate, the measured data volume change rate and the data volume change rate threshold includes: When any one of the equivalent data volume change rate and the measured data volume change rate is greater than the data volume change rate threshold, confirming that the data node is in an abnormal state; When the equivalent data volume change rate and the measured data volume change rate are both less than or equal to the data volume change rate threshold, it is confirmed that the data node is in a normal state.

12. A data node status monitoring device, characterized in that: The device comprises: A threshold adjustment module: used to predict a data volume change rate threshold within a first cycle to be executed based on historical data of the data node; the historical data includes data processed by the data node within the historical cycle; An acquisition module: configured to acquire first data processed by the data node in a first period, and calculate a measured data volume change rate of the first data in the first period according to the first data; An analysis and execution module: used to determine whether the data node is in an abnormal state based on the data volume change rate threshold and the measured data volume change rate, and perform an abnormality handling operation when the data node is in an abnormal state; Historical data updating module: used to add the first data into the historical data to obtain updated historical data, wherein the updated historical data is used to predict the corresponding data volume change rate threshold for the second cycle; the second cycle is the next cycle adjacent to the first cycle.

13. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the method according to any one of claims 1 to 10 are implemented.

14. A readable storage medium, characterized in that: The readable storage medium stores a program or an instruction, and when the program or the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images