A dynamic anomaly detection method and system for data lake

Abstract

The present invention relates to the field of data security, and provides a dynamic anomaly detection method and system for a data lake, by selecting some storage nodes in the data lake, recorded as detection nodes, real-time monitoring and recording the performance data of all detection nodes, recorded as first data, and establishing a first dynamic detection model based on the first data, and determining the abnormal level of the detection node through the first dynamic detection model. The method can significantly improve the response speed of identifying abnormal storage nodes, ensure that potential storage performance bottlenecks or other abnormal faults can be discovered quickly and in real time, and by classifying the abnormal levels of the detection nodes, it can quickly locate high abnormal level nodes and handle them in time to prevent the expansion of faults, thereby improving the reliability and security of various big data in the process of entering and leaving the lake, and maintaining the long-term stability of the storage layer in the data lake.

Description

Technical Field

[0001] The present invention relates to the field of data security, and in particular to a dynamic anomaly detection method for a data lake. Background Art

[0002] As a centralized repository, the data lake stores massive amounts of structured data, unstructured data, and other raw data from multiple sources and modalities. With the rapid growth of data volume, the high-speed read and write performance requirements for data in and out of the lake, and the complexity of data application scenarios, data lakes often face the need to frequently expand storage capacity to cope with the continuous increase in data.

[0003] Invention patent CN113342502B discloses a performance diagnosis method for a data lake, which compares the current performance of the data lake with the historical performance through a first anomaly detection threshold and a second anomaly detection threshold, thereby improving the efficiency of determining the performance of the data lake. However, the current mainstream data lakes mostly adopt a distributed storage architecture based on multiple physical storage nodes. In the data interaction process that depends on the real-time performance of the data lake, the storage nodes often face requests for high-speed writing of data in a short period of time, but it is usually more passive to expand the node capacity when receiving the request, which is easy to cause data writing failure. Therefore, it is necessary to monitor the real-time performance indicators of the storage nodes in advance, and to expand the capacity or other processing in advance when the node has abnormal problems such as sudden performance drop or node capacity exhaustion. Therefore, a dynamic detection method that combines real-time monitoring and predictive analysis is the key to effectively identify abnormal loads and performance bottlenecks in the data lake and improve its active safety performance when facing instantaneous storage pressure. Summary of the invention

[0004] The purpose of the present invention is to propose a dynamic anomaly detection method for a data lake to solve one or more technical problems existing in the prior art and at least provide a beneficial option or create conditions.

[0005] The present invention provides a dynamic anomaly detection method and system for a data lake, by selecting some storage nodes in the data lake, recorded as detection nodes, real-time monitoring and recording the performance data of all detection nodes, recorded as first data, establishing a first dynamic detection model based on the first data, and determining the abnormal level of the detection node through the first dynamic detection model. The method can significantly improve the response speed of identifying abnormal storage nodes, ensure that potential storage performance bottlenecks or other abnormal faults can be discovered quickly and in real time, and by classifying the abnormal levels of the detection nodes, it can quickly locate high abnormal level nodes and handle them in time to prevent the expansion of faults, thereby improving the reliability and security of various big data in the process of entering and leaving the lake, and maintaining the long-term stability of the storage layer in the data lake.

[0006] In order to achieve the above objective, according to one aspect of the present invention, a dynamic anomaly detection method for a data lake is provided, the method comprising the following steps:

[0007] Select some storage nodes in the data lake and record them as detection nodes;

[0008] Monitor and record the performance data of all detection nodes in real time, and record it as the first data;

[0009] Establishing a first dynamic detection model according to the first data;

[0010] The abnormality level of the detection node is determined through the first dynamic detection model.

[0011] Furthermore, the storage nodes in the data lake refer to the nodes responsible for storing and managing data. All storage nodes are organized in a distributed manner and together constitute the storage layer of the data lake. Each storage node stores part of the data. Sharding technology is used in the data lake to achieve horizontal expansion of data and storage load balancing (on storage nodes).

[0012] Furthermore, a method of selecting some storage nodes in the data lake and recording them as detection nodes is specifically as follows: among all the storage nodes in the data lake, select several storage nodes that have been frequently in a high-load state in history as detection nodes; or, select several storage nodes as detection nodes according to a random sampling method.

[0013] Preferably, the number of selected detection nodes does not exceed half of the number of all storage nodes in the data lake.

[0014] Furthermore, the method of real-time monitoring and recording the performance data of all detection nodes, which is recorded as the first data, is specifically as follows:

[0015] Note that there are M detection nodes in total, and PNode(k) represents the kth detection node among the M detection nodes;

[0016] The real-time IOPS of the detection node is used as the performance data. Starting from an arbitrary time T0, the IOPS value of the detection node is recorded every second until the time T1 is recorded.

[0017] All IOPS values ​​of PNode(k) recorded from time T0 to time T1 are stored in the array Ipo(k);

[0018] The M arrays Ipo(1), Ipo(2), ..., Ipo(M) corresponding to the M detection nodes PNode(1), PNode(2), ..., PNode(M) are used as the first data.

[0019] Furthermore, the interval between time T0 and time T1 is set to R seconds, R∈[2000,5000].

[0020] Furthermore, according to the first data, a method for establishing a first dynamic detection model is specifically as follows:

[0021] By using the first algorithm, the first segment value and the second segment value of each array in the first data are screened out, and the first segment value and the second segment value of the array Ipo(k) are stored in the array Prv(k) and the array Pov(k) respectively;

[0022] Let the length of array Prv(k) be Npr(k), and let Prv(x,k) represent the x-th value in array Prv(k), where x=1,2,…,Npr(k); subtract ln(Prv(x,k)) from Prv(x,k) as the first factor, and traverse from x=1 to x=Npr(k), then we get Npr(k) first factors, and accumulate these Npr(k) first factors and record the accumulated value as the second factor;

[0023] Set the model variable t, and use the Npr(k)th power of the expression obtained by subtracting the second factor from t as the third factor. The M arrays Prv(1), Prv(2), …, Prv(M) correspond to M third factors. Multiply these M third factors and record the expression obtained after the multiplication as the first dynamic detection model.

[0024] The beneficial effect of this step is: since the data in the data lake needs to be requested and interacted at any time, the storage nodes need to maintain a high real-time performance as much as possible. When some storage nodes are close to full load, these nodes need to be added to the background queue for capacity expansion or load balancing processing, and are no longer assigned to participate in the current storage task. However, when the scale of some storage tasks is particularly large, when a large amount of data needs to be continuously transmitted to a certain storage node, the node is prone to overload problems, thereby affecting the progress of the storage task. The method of this step establishes a first dynamic detection model by using the performance data of the detection node, and uses the front and back values ​​of the performance data as the basis for model establishment and the input of the model respectively, comprehensively determines the abnormal level of the detection node, and takes measures in advance to expand the capacity or remove it from the current task, thereby effectively reducing the problem of response delay in storage tasks and ensuring the overall stability and efficiency of the data lake during the storage process.

[0025] The principle of this step is: in the expression of the dynamic detection model, by the multiplication of the third factor, it can be seen that the performance data of each storage node is coupled to the model. Therefore, the calculation of the detection sub-value of a detection node actually takes into account the impact of the performance data of other detection nodes on the detection sub-value result of the node. The subsequent segment value is input into the model as t. It can be seen from the expression that the calculation of the detection sub-value is essentially still completely dependent on the performance data of all detection nodes (just using the scaled and sorted form). Therefore, it is ensured that when the model calculates the detection sub-value of any node, it is not limited to the individual performance of the node, but fully integrates the dynamic performance data of the entire detection node.

[0026] Furthermore, the method of filtering out the front segment value and the back segment value of each array in the first data by the first algorithm is specifically as follows:

[0027] For the M arrays Ipo(1), Ipo(2), ..., Ipo(M) in the first data, Ipo(k) represents the kth array in the M arrays; let the array Ipo(k) contain a total of N(k) values, and Ipo(i,k) represents the i-th value in the array Ipo(k), i is the sequence number, i=1,2,...,N(k);

[0028] Calculate Nh(i,k) through the first equation, traverse i from i=1 to i=N(k) in the first equation, and then obtain N(k) values ​​Nh(1,k), Nh(2,k), …, Nh(N(k),k), create a blank array Nh(k), and fill these N(k) values ​​into the array Nh(k) in ascending order according to the size of the value (that is, the values ​​in the array Nh(k) are sorted in ascending order);

[0029] Filter out the first and last values ​​of the array Ipo(k) from the array Nh(k);

[0030] The calculation method of the first equation is:

[0031] The value obtained by subtracting the minimum value in the array Ipo(k) from Ipo(i,k) is taken as the first value;

[0032] The value obtained by subtracting 1 from the maximum value in the array Ipo(k) is used as the second value;

[0033] The third value is the maximum value in the array Ipo(k) minus the minimum value in the array Ipo(k);

[0034] The fourth value is obtained by dividing the second value by the third value;

[0035] A value obtained by multiplying the first value by the fourth value and adding 1 is taken as Nh(i,k).

[0036] The beneficial effect of this step is: the array Ipo(k) is scaled by the first equation so that the range of the array can match the ln function in the first dynamic detection model, and at the same time, Ipo(k) is reorganized into Nh(k) in ascending order, and the front segment value and the back segment value are divided based on the array Nh(k), which can accurately locate the segmentation position of the front and back segment data, reduce the impact of the disordered form of the original data and the unscaling on the model performance, and provide an accurate data foundation for the model.

[0037] The principle of this step is: since the amount of data of the storage node performance data is large and its range is wide, using the original data to establish the detection model will cause the model to accumulate more errors and be difficult to converge. Therefore, the values ​​of the performance data of all storage nodes are scaled to the interval [1, MIpo(k)] through the first equation. This is because the lnx function in the model will not be negative in the interval [1, ∞), and the function curve is flatter as it approaches the latter part of the interval (that is, the difference between the function values ​​is smaller), and the curve is steeper when it is close to the point value 1. Therefore, the array Ipo(k) is scaled and reorganized to form an array Nh(k) in ascending order, which not only corresponds to the characteristics of the x-axis from small to large in function, but also can be easily divided to obtain the front and rear values;

[0038] The front segment value is the interval of the ln function with a steeper value, and the back segment value is the interval of the ln function with a flatter value. It can be seen that the ln function shows an increasingly flat trend when extending along x=1. After a certain point value, its curve is close to a constant curve. The process of selecting the front and back segment values ​​based on Nh(k) (see S1 to S4) is to use the variable E0 as the point value and then divide it to obtain the front and back segment values, so that the steeper front segment value of the ln function (better discrimination) is used as the basis for establishing the detection model, which can more accurately reflect the storage pressure of each storage node.

[0039] Because a slight change in performance data will cause a large difference in function values ​​(the growth rate of the steep section of the lnx function is fast), a more accurate node abnormality level division weight is provided, and the back segment value is used as the model input to reduce the difference in the model output (that is, the detection sub-value of each detection node), because the performance pressure difference of the detection node is relatively small (much lower than the pressure difference between an unloaded node and a fully loaded node, and the detection node generally has a tendency to be close to full load). On the contrary, if the front segment value is used as the model input, the differences between the multiple detection sub-values ​​output by the model will be large, and the effectiveness of using it as a method to determine the abnormality level of the detection node will be lost. Therefore, by dividing the performance data into the front and back segments, the first dynamic detection model can have a higher accuracy in detecting the current real-time storage pressure of each detection node.

[0040] Furthermore, the method of selecting the front-end value and the back-end value of the array Ipo(k) from the array Nh(k) is as follows:

[0041] S1, set variable E0 = 1 (the reason for setting the initial value to 1 is to correspond to the scaling made by the first equation above), Nh(j,k) represents the jth value in the array Nh(k), j is the sequence number, j = 1, 2, ..., N(k), set variable c, the initial value of variable c is set to 1, and go to S2;

[0042] S2, record the value of Nh(c+1,k)-Nh(c,k) as DNh, increase the value of variable E0 by DNh (i.e. E0= E0+DNh), and go to S3;

[0043] S3, if the value of variable E0 is greater than or equal to the maximum value in array Nh(k), or the value of variable c is equal to N(k), then go to S4; if the value of variable E0 is less than the maximum value in array Nh(k), then go to S2;

[0044] S4, let the value of variable c be p, record Nh(1,k), Nh(2,k), …, Nh(p,k) in array Nh(k) as the first part of the values ​​of array Ipo(k) and store them in array Prv(k), record Nh(p+1,k), Nh(p+2,k), …, Nh(N(k),k) in array Nh(k) as the second part of the values ​​of array Ipo(k) and store them in array Pov(k).

[0045] Furthermore, the method for determining the abnormality level of the detection node through the first dynamic detection model is specifically as follows:

[0046] Each value in the array Pov(k) is input as a model variable t into the first dynamic detection model, and then multiple model output values ​​are obtained (the number of model output values ​​is consistent with the length of the array Pov(k)). The cumulative value of these multiple model output values ​​is recorded as PL(k). From k=1 to k=M, PL(1), PL(2), ..., PL(M) are obtained.

[0047] Take PL(1), PL(2), …, PL(M) as the detection sub-values ​​of the detection nodes PNode(1), PNode(2), …, PNode(M), and mark the M0 detection nodes with the highest detection sub-values ​​as high abnormality levels.

[0048] Furthermore, the value of M0 is set to M0=M×A1, where A1 is a decimal in the interval [0.25, 0.5].

[0049] The beneficial effect of this step is: the detection sub-value of each detection node is calculated through the first dynamic detection model, and the detection sub-value is used to further determine the abnormality level of the detection node, so that the detection system can timely and accurately locate the detection nodes with high abnormality levels and process them, effectively preventing potential storage performance abnormalities, and ensuring that the data lake can quickly adjust the capacity allocation of storage nodes and optimize storage performance when facing high concurrency and large data volume requests.

[0050] Preferably, after determining the abnormality level of the detection node, it also includes dynamically expanding the capacity of the detection node with a high abnormality level, or migrating the data in the detection node with a high abnormality level to a storage node in the data lake that is not selected as a detection node.

[0051] The present invention also provides a dynamic anomaly detection system for a data lake, the dynamic anomaly detection system for a data lake comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, the processor implementing the steps in a dynamic anomaly detection method for a data lake when executing the computer program, the dynamic anomaly detection system for a data lake can be run in computing devices such as desktop computers, notebook computers, mobile phones, cell phones, tablet computers, PDAs, and cloud data centers, the executable system may include, but not limited to, a processor, a memory, and a server cluster, the processor executing the computer program runs in the following system units:

[0052] The node selection unit is used to select some storage nodes in the data lake, which are recorded as detection nodes;

[0053] A data monitoring unit, used to monitor and record the performance data of all detection nodes in real time, recorded as first data;

[0054] A model building unit, used to build a first dynamic detection model according to the first data;

[0055] The node detection unit is used to determine the abnormality level of the detection node through the first dynamic detection model.

[0056] The beneficial effects of the present invention are as follows: the method can significantly improve the response speed of identifying abnormal storage nodes, ensure that potential storage performance bottlenecks or other abnormal faults can be discovered quickly and in real time, and by classifying the detection nodes into abnormal levels, it is possible to quickly locate high-abnormality level nodes and handle them in a timely manner to prevent the expansion of faults, thereby improving the reliability and security of various big data in the process of entering and leaving the lake, and maintaining the long-term stability of the storage layer in the data lake. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The above and other features of the present invention will become more obvious by describing in detail the embodiments shown in the accompanying drawings. The same reference numerals in the accompanying drawings of the present invention represent the same or similar elements. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other accompanying drawings can be obtained based on these accompanying drawings without creative work. In the accompanying drawings:

[0058] Figure 1 Shown is a flow chart of a dynamic anomaly detection method for a data lake;

[0059] Figure 2 Shown is a system architecture diagram of a dynamic anomaly detection system for a data lake. DETAILED DESCRIPTION

[0060] The following will be combined with the embodiments and drawings to clearly and completely describe the concept, specific structure and technical effects of the present invention, so as to fully understand the purpose, scheme and effect of the present invention. It should be noted that the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0061] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0062] like Figure 1 The flowchart of a dynamic anomaly detection method for a data lake according to the present invention is shown below. Figure 1 A dynamic anomaly detection method for a data lake according to an embodiment of the present invention is described below.

[0063] The present invention proposes a dynamic anomaly detection method for a data lake, the method comprising the following steps:

[0064] Select some storage nodes in the data lake and record them as detection nodes;

[0065] Monitor and record the performance data of all detection nodes in real time, and record it as the first data;

[0066] Establishing a first dynamic detection model according to the first data;

[0067] The abnormality level of the detection node is determined through the first dynamic detection model.

[0068] Furthermore, the storage nodes in the data lake refer to the nodes responsible for storing and managing data. All storage nodes are organized in a distributed manner and together constitute the storage layer of the data lake. Each storage node stores part of the data. Sharding technology is used in the data lake to achieve horizontal expansion of data and storage load balancing (on storage nodes).

[0069] Furthermore, a method of selecting some storage nodes in the data lake and recording them as detection nodes is specifically as follows: among all the storage nodes in the data lake, select several storage nodes that have been frequently in a high-load state in history as detection nodes; or, select several storage nodes as detection nodes according to a random sampling method.

[0070] Specifically, the number of selected detection nodes does not exceed half of the number of all storage nodes in the data lake.

[0071] Furthermore, the method of real-time monitoring and recording the performance data of all detection nodes, which is recorded as the first data, is specifically as follows:

[0072] Note that there are M detection nodes in total, and PNode(k) represents the kth detection node among the M detection nodes, where k is the sequence number, k=1,2,…,M;

[0073] The real-time IOPS of the detection node is used as the performance data. Starting from an arbitrary time T0, the IOPS value of the detection node is recorded every second until the time T1 is recorded. Then, each detection node is recorded with multiple IOPS values ​​from time T0 to time T1.

[0074] All IOPS values ​​of PNode(k) recorded from time T0 to time T1 are stored in the array Ipo(k);

[0075] The M arrays Ipo(1), Ipo(2), ..., Ipo(M) corresponding to the M detection nodes PNode(1), PNode(2), ..., PNode(M) are used as the first data.

[0076] Furthermore, the interval between time T0 and time T1 is set to R seconds, R∈[2000,5000].

[0077] Furthermore, according to the first data, a method for establishing a first dynamic detection model is specifically as follows:

[0078] By using the first algorithm, the first segment value and the second segment value of each array in the first data are screened out, and the first segment value and the second segment value of the array Ipo(k) are stored in the array Prv(k) and the array Pov(k) respectively;

[0079] Let the length of array Prv(k) be Npr(k), and let Prv(x,k) represent the x-th value in array Prv(k), where x=1,2,…,Npr(k); subtract ln(Prv(x,k)) from Prv(x,k) as the first factor, and traverse from x=1 to x=Npr(k), then we get Npr(k) first factors, and accumulate these Npr(k) first factors and record the accumulated value as the second factor;

[0080] Set the model variable t, and use the Npr(k)th power of the expression obtained by subtracting the second factor from t as the third factor, so that the array Prv(k) uniquely corresponds to a third factor, and the M arrays Prv(1), Prv(2), ..., Prv(M) correspond to M third factors. Multiply these M third factors, and record the expression obtained after the multiplication as the first dynamic detection model;

[0081] The specific mathematical expression is:

[0082]

[0083] Where t is the model variable, and ln(Prv(x,k)) represents the natural logarithm of Prv(x,k).

[0084] Furthermore, the method of filtering out the front segment value and the back segment value of each array in the first data by the first algorithm is specifically as follows:

[0085] For the M arrays Ipo(1), Ipo(2), ..., Ipo(M) in the first data, Ipo(k) represents the kth array in the M arrays; let the array Ipo(k) contain a total of N(k) values, and Ipo(i,k) represents the i-th value in the array Ipo(k), i is the sequence number, i=1,2,...,N(k);

[0086] Calculate Nh(i,k) through the first equation, traverse i from i=1 to i=N(k) in the first equation, and then obtain N(k) values ​​Nh(1,k), Nh(2,k), …, Nh(N(k),k), create a blank array Nh(k), and fill these N(k) values ​​into the array Nh(k) in ascending order according to the size of the value (that is, the values ​​in the array Nh(k) are sorted in ascending order);

[0087] Filter out the first and last values ​​of the array Ipo(k) from the array Nh(k);

[0088] The calculation method of the first equation is:

[0089] The value obtained by subtracting the minimum value in the array Ipo(k) from Ipo(i,k) is taken as the first value;

[0090] The value obtained by subtracting 1 from the maximum value in the array Ipo(k) is used as the second value;

[0091] The third value is the maximum value in the array Ipo(k) minus the minimum value in the array Ipo(k);

[0092] The fourth value is obtained by dividing the second value by the third value;

[0093] The value obtained by multiplying the first value by the fourth value and adding 1 is taken as Nh(i,k);

[0094] The specific mathematical expression is:

[0095]

[0096] Where DIpo(k) represents the minimum value in the array Ipo(k), and MIpo(k) represents the maximum value in the array Ipo(k).

[0097] Furthermore, the method of selecting the front-end value and the back-end value of the array Ipo(k) from the array Nh(k) is as follows:

[0098] S1, set variable E0 = 1 (the reason for setting the initial value to 1 is to correspond to the scaling made by the first equation above), use Nh(j,k) to represent the j-th value in the array Nh(k), j is the sequence number, j=1,2,…,N(k), set variable c, the initial value of variable c is set to 1, use Nh(c,k) to represent the c-th value corresponding to the value of variable c in the array Nh(k), and go to S2;

[0099] S2, record the value of Nh(c+1,k)-Nh(c,k) as DNh, increase the value of variable E0 by DNh (i.e. E0= E0+DNh), and go to S3;

[0100] S3, if the value of variable E0 is greater than or equal to the maximum value in array Nh(k), or the value of variable c is equal to N(k), then go to S4; if the value of variable E0 is less than the maximum value in array Nh(k), then go to S2;

[0101] S4, let the value of variable c be p, record Nh(1,k), Nh(2,k), …, Nh(p,k) in array Nh(k) as the first part of the values ​​of array Ipo(k) and store them in array Prv(k), record Nh(p+1,k), Nh(p+2,k), …, Nh(N(k),k) in array Nh(k) as the second part of the values ​​of array Ipo(k) and store them in array Pov(k).

[0102] If c in S4 is equal to N(k), it means that there are no outliers in the array Nh(k) or there are only a few outliers that account for a very small proportion of the total value (outliers are values ​​outside the sample area composed of most values). At this time, the array Prv(k) is equal to the array Nh(k) itself, and the array Pov(k) is empty. It is worth noting that this situation is relatively rare, because no matter when the number of outliers is slightly more or there are no outliers, the c value will stop before the loop increases to N(k).

[0103] Furthermore, the method for determining the abnormality level of the detection node through the first dynamic detection model is specifically as follows:

[0104] Each value in the array Pov(k) is input as a model variable t into the first dynamic detection model, and then multiple model output values ​​are obtained (the number of model output values ​​is consistent with the length of the array Pov(k)). The cumulative value of these multiple model output values ​​is recorded as PL(k). From k=1 to k=M, PL(1), PL(2), ..., PL(M) are obtained.

[0105] Take PL(1), PL(2), …, PL(M) as the detection sub-values ​​of the detection nodes PNode(1), PNode(2), …, PNode(M), and mark the M0 detection nodes with the highest detection sub-values ​​as high abnormality levels.

[0106] Further, the value of M0 is set to M0=M×A1, and A1 is set to 0.25.

[0107] The beneficial effect of this step is: the detection sub-value of each detection node is calculated through the first dynamic detection model, and the detection sub-value is used to further determine the abnormality level of the detection node, so that the detection system can timely and accurately locate the detection nodes with high abnormality levels and process them, effectively preventing potential storage performance abnormalities, and ensuring that the data lake can quickly adjust the capacity allocation of storage nodes and optimize storage performance when facing high concurrency and large data volume requests.

[0108] Specifically, after determining the abnormality level of the detection node, it also includes dynamically expanding the detection node with a high abnormality level, or migrating the data in the detection node with a high abnormality level to a storage node in the data lake that is not selected as a detection node.

[0109] The dynamic anomaly detection system for a data lake includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the above-mentioned embodiment of the dynamic anomaly detection method for a data lake are implemented. The dynamic anomaly detection system for a data lake can be run on computing devices such as desktop computers, laptops, mobile phones, cell phones, tablet computers, PDAs, and cloud data centers. The executable system may include, but is not limited to, a processor, a memory, and a server cluster.

[0110] An embodiment of the present invention provides a dynamic anomaly detection system for a data lake, such as Figure 2 As shown, a dynamic anomaly detection system for a data lake in this embodiment includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the above-mentioned embodiment of a dynamic anomaly detection method for a data lake are implemented. The processor executes the computer program to run in the following system units:

[0111] The node selection unit is used to select some storage nodes in the data lake, which are recorded as detection nodes;

[0112] A data monitoring unit, used to monitor and record the performance data of all detection nodes in real time, recorded as first data;

[0113] A model building unit, used to build a first dynamic detection model according to the first data;

[0114] The node detection unit is used to determine the abnormality level of the detection node through the first dynamic detection model.

[0115] The dynamic anomaly detection system for a data lake can be run on computing devices such as desktop computers, laptops, PDAs, and cloud data centers. The dynamic anomaly detection system for a data lake includes, but is not limited to, a processor and a memory. Those skilled in the art can understand that the example is only an example of a dynamic anomaly detection method and system for a data lake, and does not constitute a limitation on a dynamic anomaly detection method and system for a data lake. It may include more or fewer components than the example, or a combination of certain components, or different components. For example, the dynamic anomaly detection system for a data lake may also include input and output devices, network access devices, buses, etc.

[0116] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete component gate circuits or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the dynamic anomaly detection system for a data lake, and uses various interfaces and lines to connect the various sub-areas of the entire dynamic anomaly detection system for a data lake.

[0117] The memory can be used to store the computer program and / or module, and the processor implements the various functions of the dynamic anomaly detection method and system for a data lake by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0118] The present invention provides a dynamic anomaly detection method and system for a data lake, by selecting some storage nodes in the data lake, recorded as detection nodes, real-time monitoring and recording the performance data of all detection nodes, recorded as first data, according to the first data, a first dynamic detection model is established, and the abnormal level of the detection node is determined by the first dynamic detection model. The method can significantly improve the response speed of identifying abnormal storage nodes, ensure that potential storage performance bottlenecks or other abnormal faults can be discovered quickly and in real time, and by classifying the abnormal levels of the detection nodes, high abnormal level nodes can be quickly located and processed in time to prevent the expansion of faults, thereby improving the reliability and security of various big data in the process of entering and leaving the lake, and maintaining the long-term stability of the storage layer in the data lake. Although the description of the present invention has been quite detailed and several embodiments are described in particular, it is not intended to be limited to any of these details or embodiments or any special embodiments, so as to effectively cover the predetermined scope of the present invention. In addition, the above describes the present invention with the foreseeable embodiments of the inventor, the purpose of which is to provide a useful description, and those non-substantial changes to the present invention that have not yet been foreseen can still represent equivalent changes to the present invention.

Claims

1. A dynamic anomaly detection method for a data lake, characterized in that: The method comprises the following steps: Select some storage nodes in the data lake and record them as detection nodes; Monitor and record the performance data of all detection nodes in real time, and record it as the first data; Establishing a first dynamic detection model according to the first data; Determining the abnormality level of the detection node through the first dynamic detection model; The method of real-time monitoring and recording the performance data of all detection nodes as the first data is as follows: there are M detection nodes in total, PNode(k) represents the kth detection node among the M detection nodes, k is a sequence number, k=1,2,…,M; Taking the real-time IOPS of the detection node as the performance data, starting from an arbitrary time T0, the IOPS value of the detection node is recorded every second until the time T1 is recorded, and each detection node is recorded with multiple IOPS values ​​from time T0 to time T1; all IOPS values ​​of PNode(k) recorded from time T0 to time T1 are stored in the array Ipo(k); Taking M arrays Ipo(1), Ipo(2), ..., Ipo(M) corresponding to M detection nodes PNode(1), PNode(2), ..., PNode(M) as first data; The storage nodes in the data lake refer to the nodes responsible for storing and managing data. All storage nodes are organized in a distributed manner and together form the storage layer of the data lake. Each storage node stores part of the data. Sharding technology is used in the data lake to achieve horizontal expansion of data and storage load balancing. According to the first data, the method for establishing the first dynamic detection model is specifically as follows: By using the first algorithm, the first segment value and the second segment value of each array in the first data are screened out, and the first segment value and the second segment value of the array Ipo(k) are stored in the array Prv(k) and the array Pov(k) respectively; Let the length of array Prv(k) be Npr(k), and let Prv(x,k) represent the x-th value in array Prv(k), where x=1,2,…,Npr(k); subtract ln(Prv(x,k)) from Prv(x,k) as the first factor, and traverse from x=1 to x=Npr(k), then we get Npr(k) first factors, and accumulate these Npr(k) first factors and record the accumulated value as the second factor; Set the model variable t, and use the Npr(k)th power of the expression obtained by subtracting the second factor from t as the third factor, so that the array Prv(k) uniquely corresponds to a third factor, and the M arrays Prv(1), Prv(2), ..., Prv(M) correspond to M third factors. Multiply these M third factors, and record the expression obtained after the multiplication as the first dynamic detection model; The method of screening out the front segment value and the back segment value of each array in the first data by the first algorithm is specifically as follows: For the M arrays Ipo(1), Ipo(2), ..., Ipo(M) in the first data, Ipo(k) represents the kth array in the M arrays; let the array Ipo(k) contain a total of N(k) values, and Ipo(i,k) represents the i-th value in the array Ipo(k), i is the sequence number, i=1,2,...,N(k); Calculate Nh(i,k) through the first equation, traverse i from i=1 to i=N(k) in the first equation, and then obtain N(k) values ​​Nh(1,k), Nh(2,k), …, Nh(N(k),k), create a blank array Nh(k), and fill these N(k) values ​​into the array Nh(k) in ascending order according to the value size; filter out the first and last values ​​of the array Ipo(k) from the array Nh(k); The calculation method of the first equation is: the value obtained by subtracting the minimum value in the array Ipo(k) from Ipo(i,k) is taken as the first value; the value obtained by subtracting 1 from the maximum value in the array Ipo(k) is taken as the second value; the value obtained by subtracting the minimum value in the array Ipo(k) from the maximum value in the array Ipo(k) is taken as the third value; the second value is divided by the third value as the fourth value; the value obtained by multiplying the first value by the fourth value plus 1 is taken as Nh(i,k).

2. The dynamic anomaly detection method for a data lake according to claim 1, characterized in that: The method of selecting some storage nodes in the data lake and recording them as detection nodes is specifically as follows: among all the storage nodes in the data lake, select several storage nodes that have been frequently in a high-load state in history as detection nodes; or select several storage nodes as detection nodes according to a random sampling method.

3. The dynamic anomaly detection method for a data lake according to claim 1, characterized in that: The specific method of selecting the front and back values ​​of the array Ipo(k) from the array Nh(k) is: S1, set variable E0=1, use Nh(j,k) to represent the jth value in array Nh(k), j is the sequence number, j=1,2,…,N(k), set variable c, the initial value of variable c is set to 1, use Nh(c,k) to represent the cth value corresponding to the value of variable c in array Nh(k), and go to S2; S2, record the value of Nh(c+1,k)-Nh(c,k) as DNh, increase the value of variable E0 by DNh, and go to S3; S3, if the value of variable E0 is greater than or equal to the maximum value in array Nh(k), or the value of variable c is equal to N(k), then go to S4; if the value of variable E0 is less than the maximum value in array Nh(k), then go to S2; S4, let the value of variable c be p, record Nh(1,k), Nh(2,k), …, Nh(p,k) in array Nh(k) as the first part of the values ​​of array Ipo(k) and store them in array Prv(k), record Nh(p+1,k), Nh(p+2,k), …, Nh(N(k),k) in array Nh(k) as the second part of the values ​​of array Ipo(k) and store them in array Pov(k).

4. The dynamic anomaly detection method for a data lake according to claim 1, characterized in that: The method for determining the abnormal level of the detection node through the first dynamic detection model is specifically as follows: Each value in the array Pov(k) is input as a model variable t into the first dynamic detection model, and then multiple model output values ​​are obtained. The cumulative value of these multiple model output values ​​is recorded as PL(k). From k=1 to k=M, PL(1), PL(2), ..., PL(M) are obtained. Take PL(1), PL(2), …, PL(M) as the detection sub-values ​​of detection nodes PNode(1), PNode(2), …, PNode(M), and mark the M0 detection nodes with the highest detection sub-values ​​as high abnormality levels, where the value of M0 is set to M0=M×A1, and A1 is a decimal in the interval [0.25, 0.5].

5. A dynamic anomaly detection system for a data lake, characterized in that: The dynamic anomaly detection system for a data lake comprises: a processor, a memory, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, the steps of the dynamic anomaly detection method for a data lake described in any one of claims 1 to 4 are implemented. The dynamic anomaly detection system for a data lake runs on a desktop computer, a laptop computer, a PDA, or a computing device in a cloud data center.

Images