Data monitoring method and device

Abstract

The invention provides a data monitoring method and device. The method comprises the steps that the data type of to-be-monitored data is determined; the data type comprises a numeric type or a non-numeric type; based on the data type of the to-be-monitored data, inputting the to-be-monitored data into a corresponding sub-tree of a data monitoring model, and obtaining the path length of the to-be-monitored data passing through the sub-tree; determining whether the to-be-monitored data is abnormal or not based on the path length; wherein the data monitoring model comprises a plurality of isolated trees, each isolated tree is composed of a numeric type sub-tree and a non-numeric type sub-tree, numeric type data to be monitored are input into the numeric type sub-tree, and non-numeric type data to be monitored are input into the non-numeric type sub-tree.

Description

Technical Field

[0001] The present disclosure relates to the technical field of data analysis and monitoring, and in particular to a data monitoring method and device. Background Art

[0002] As data size and complexity continue to grow, if abnormal data is not discovered in a timely manner, it may have an immeasurable impact on the entire system or business. Traditional data anomaly monitoring methods have certain limitations in terms of the timeliness and efficiency of data anomaly monitoring, and cannot effectively cope with the challenges of the current growing data size and complexity. Summary of the invention

[0003] The present disclosure provides a data monitoring method and device.

[0004] According to a first aspect of the present disclosure, there is provided a data monitoring method, comprising:

[0005] Determine the data type of the data to be monitored; the data type includes a numerical type or a non-numerical type;

[0006] Based on the data type of the data to be monitored, input the data to be monitored into a corresponding subtree of a data monitoring model, and obtain the path length of the data to be monitored passing through the subtree;

[0007] Based on the path length, determining whether the data to be monitored is abnormal;

[0008] The data monitoring model includes multiple isolated trees, each of which is composed of a numerical subtree and a non-numerical subtree. The numerical data to be monitored is input into the numerical subtree, and the non-numerical data to be monitored is input into the non-numerical subtree.

[0009] In the above scheme, based on the data type of the data to be monitored, the data to be monitored is input into a subtree corresponding to the data monitoring model, and the path length of the data to be monitored passing through the subtree is obtained, including:

[0010] In response to the data type of the data to be monitored being numerical, the data to be monitored is input into the numerical subtree of each isolated tree included in the data monitoring model, and the path length of the data to be monitored passing through the numerical subtrees of all isolated trees is determined.

[0011] In the above scheme, based on the data type of the data to be monitored, the data to be monitored is input into a subtree corresponding to the data monitoring model, and the path length of the data to be monitored passing through the subtree is obtained, including:

[0012] In response to the data type of the data to be monitored being non-numeric, the data to be monitored is input into the non-numeric subtree of each isolated tree included in the data monitoring model, and the path length of the data to be monitored passing through the non-numeric subtrees of all isolated trees is determined.

[0013] In the above solution, the method further includes training the data monitoring model, including:

[0014] Constructing a first original data set based on the original data; the first original data set includes numerical features and non-numerical features;

[0015] The features are split according to the information gain of the numerical features in the first original data set from high to low, and the numerical subtrees of the isolated tree are constructed; the non-numerical features in the first original data set are randomly split to construct the non-numerical subtrees of the isolated tree;

[0016] Constructing an initial number of isolated trees based on numerical subtrees and non-numerical subtrees; the initial number of isolated trees constitute an original data monitoring model;

[0017] Determine a first number of isolated trees in the data monitoring model and a first ratio of the sub-data set to the first original data set based on a particle swarm optimization algorithm;

[0018] Based on the first number of isolated trees and a first ratio of the sub-data sets to the first original data set, adjusting the original data monitoring model to obtain a data monitoring model;

[0019] The data monitoring model is trained based on the training set.

[0020] In the above solution, constructing the first original data set based on the original data includes:

[0021] Constructing a second original data set based on the original data; the second original data set includes a table, each row of the table represents a data subject, and each column represents a feature corresponding to the data subject; the feature includes a numerical feature and a non-numerical feature;

[0022] The non-numeric features are subjected to dimensionality reduction processing, and the non-numeric features after the dimensionality reduction processing are concatenated with the numerical features to obtain a first original data set.

[0023] In the above solution, the dimensionality reduction processing of the non-numerical features includes:

[0024] Converting the non-numeric feature into a third quantitative data;

[0025] Calculating the mean of the third quantity data to determine a mean vector;

[0026] Determine a centralization matrix according to a difference between the third quantity data and the mean vector;

[0027] Determining a covariance matrix according to the centering matrix and the transpose of the centering matrix;

[0028] Performing eigendecomposition on the covariance matrix to determine eigenvalues ​​and eigenvectors;

[0029] The eigenvalue and the eigenvector are taken as the basis of the subspace, and the dimension of the centered matrix is ​​reduced to obtain non-numerical features after the dimension reduction.

[0030] In the above solution, the method of determining the first number of isolated trees in the data monitoring model and the first proportion of the sub-data set to the first original data set based on the particle swarm optimization algorithm includes:

[0031] Set the initial proportion of the sub-dataset and the initial number of isolated trees, and build the original data monitoring model based on the initial proportion and initial number;

[0032] Based on the proportion of the sub-dataset and the range of the number of isolated trees, a particle swarm is constructed; wherein the number of isolated trees is the current position of the particles in the particle swarm, and the proportion of the sub-dataset is the current speed of the particles in the particle swarm;

[0033] Calculating an objective function value corresponding to each particle in the particle swarm according to the current position and current speed of each particle in the particle swarm, wherein the objective function value is used to represent the monitoring accuracy of the data monitoring model;

[0034] Update the position and velocity of each particle, and calculate the updated objective function value corresponding to each particle in the particle swarm based on the current position and current velocity;

[0035] Update the current position and current speed of the target particle in the particle swarm by using the ideal position of the target particle, the current position of the target particle and the ideal position of the particle swarm, wherein the target particle is any particle in the particle swarm, and the ideal position represents the particle position that maximizes the objective function value;

[0036] Repeat the above two steps until, in response to satisfying the iteration end condition, the first number of isolated trees in the data monitoring model and the first proportion of the sub-data set to the first original data set are determined based on the ideal position of the target particle.

[0037] In the above scheme, the step of segmenting the features in the first original data set in descending order of information gain of the numerical features to construct the numerical subtree of the isolated tree includes:

[0038] Calculate the information gain of the numerical features in the first original data set;

[0039] The numerical feature with the largest information gain is used as the root node of the isolation tree, and the remaining numerical features are used as the target segmentation points of the isolation tree in the order of gain information from high to low to construct the numerical subtree of the isolation tree.

[0040] In the above solution, the training of the data monitoring model based on the training set includes:

[0041] Inputting the monitoring sample data in the training set into the data monitoring model to obtain the predicted type of the monitoring sample data;

[0042] Determining a loss function of the data monitoring model according to the annotation type of the sample data and the prediction type;

[0043] The target segmentation points of each isolated tree in the data monitoring model are adjusted based on the loss function.

[0044] According to a second aspect of the present disclosure, a data monitoring device is provided, the device comprising:

[0045] A data type determination unit, used to determine the data type of the data to be monitored; the data type includes a numerical type or a non-numerical type;

[0046] A monitoring unit, configured to input the data to be monitored into a subtree corresponding to a data monitoring model based on the data type of the data to be monitored, and obtain a path length of the data to be monitored passing through the subtree;

[0047] a determination unit, configured to determine whether the data to be monitored is abnormal based on the path length;

[0048] The data monitoring model includes multiple isolated trees, each of which is composed of a numerical subtree and a non-numerical subtree. The numerical data to be monitored is input into the numerical subtree, and the non-numerical data to be monitored is input into the non-numerical subtree.

[0049] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which:

[0051] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts.

[0052] Figure 1A first optional flow chart of the data monitoring method provided by the embodiment of the present disclosure is shown;

[0053] Figure 2 A second optional flow chart of the data monitoring method provided by the embodiment of the present disclosure is shown;

[0054] Figure 3 A third optional flow chart of the data monitoring method provided by the embodiment of the present disclosure is shown;

[0055] Figure 4 A fourth optional flow chart of the data detection method provided by the embodiment of the present disclosure is shown;

[0056] Figure 5 An optional schematic diagram of a second original data set provided by an embodiment of the present disclosure is shown;

[0057] Figure 6 An optional schematic diagram of a first original data set provided by an embodiment of the present disclosure is shown;

[0058] Figure 7 A fifth optional flow chart of the data monitoring method provided in the embodiment of the present disclosure is shown;

[0059] Figure 8 A fifth optional flow chart of the data monitoring method provided in the embodiment of the present disclosure is shown;

[0060] Fig. 9 A schematic diagram of training a data monitoring model provided by an embodiment of the present disclosure is shown;

[0061] Fig.10 An optional structural schematic diagram of a data monitoring device provided by an embodiment of the present disclosure is shown;

[0062] Fig.11 A schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0063] In order to make the purpose, features, and advantages of the present disclosure more obvious and easy to understand, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0064] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0065] In the following description, the terms "first\second" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein.

[0066] Unless otherwise defined, all technical and scientific terms used in this disclosure have the same meaning as those commonly understood by those skilled in the art to which this disclosure belongs. The terms used in this disclosure are only for the purpose of describing the embodiments of this disclosure and are not intended to limit this disclosure.

[0067] It should be understood that in the various embodiments of the present disclosure, the size of the serial number of each implementation process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure.

[0068] As the era of data-driven smart business has arrived, the scale and complexity of data are growing. In such a huge ocean of data, abnormal data is hidden in it. If it is not discovered in time, it may have an immeasurable impact on the entire system or business.

[0069] At present, there are mainly three traditional methods for data anomaly monitoring:

[0070] 1) The operation and maintenance personnel set thresholds based on their experience to detect abnormal states of the current data. However, this method is highly dependent on the experience of the operation and maintenance personnel. Different operation and maintenance personnel may set different thresholds, resulting in greater uncertainty in the results. The method of manually setting thresholds is subjective, and abnormal data may be missed and cannot be fully checked. With the dynamic changes of data, fixed thresholds are often difficult to adapt to various complex situations, which is a huge challenge for operation and maintenance personnel. In addition, the manual method also has the problems of high labor costs and low efficiency.

[0071] 2) Monitor through regularization and statistical analysis, and monitor abnormal conditions through statistical results of deviations between data and the mean. However, this method is often ineffective when facing non-normally distributed data and cannot handle large-scale, high-dimensional data. At the same time, it may not be sensitive enough to small changes in data, making it difficult to accurately detect potential abnormalities, and cannot be applied to data quality monitoring and early warning scenarios with high real-time requirements.

[0072] 3) Monitor through some data management tools (such as Data Covernance). Although these tools can realize data monitoring to a certain extent, they are usually single-function, lack flexibility and pertinence, rely on rules and templates, use predefined rules for early warning, and are relatively complex to manage. When used commercially, they need to be customized according to needs, which increases additional development investment. In addition, data management tools are highly complex and difficult for non-technical personnel. Moreover, when processing large-scale data, their timeliness and efficiency are often difficult to meet actual needs.

[0073] In summary, traditional data anomaly monitoring methods have certain limitations in terms of timeliness and efficiency, and cannot effectively cope with the challenges of the growing size and complexity of current data. Therefore, there is an urgent need for a new data monitoring method that can quickly and accurately detect abnormal data and provide strong guarantees for the stable operation of systems and businesses. Based on this, the embodiments of the present disclosure provide a data monitoring method to at least solve some or all of the above technical problems.

[0074] In some embodiments, the data monitoring method provided by the embodiments of the present disclosure is applicable to the financial field, such as fraud monitoring and risk assessment, and can detect abnormal trading behavior or analyze market fluctuations in a timely manner; it is applicable to the field of network security, such as intrusion detection by monitoring network traffic data, identifying abnormal network behavior, or analyzing the user's online behavior pattern for user behavior analysis; it is applicable to the medical field, monitoring medical data, timely discovering abnormal disease outbreak trends or changes in patient symptoms, and can also monitor the operating data of medical equipment, detect equipment failures and abnormal conditions; it is applicable to the logistics field, it can monitor the logistics data in the supply chain, timely discover abnormal logistics delays, and can also monitor the operating data of transport vehicles to detect abnormal conditions; it is applicable to the energy field, it can monitor the operating data of the power system, timely discover abnormal power loads, voltage fluctuations, etc., and can also monitor the data in the petrochemical production process to detect abnormal conditions, such as equipment failures, process parameter abnormalities, etc.; it is applicable to the field of production and manufacturing, it can monitor the quality of the production process for the quality-related data of the product in the production process, and can also detect the raw materials. For the convenience of description, the data monitoring in the field of production and manufacturing is used as an example for explanation in the embodiments of the present disclosure.

[0075] Figure 1 A first optional flow chart of the data monitoring method provided in the embodiment of the present disclosure is shown, and will be explained according to each step.

[0076] Step S101, determining the data type of the data to be monitored.

[0077] In some embodiments, the data types of the data to be monitored include numerical and non-numerical types; the numerical data to be monitored include data composed of numbers, and the non-numerical data to be monitored include data composed of non-numbers (such as text such as weight units and length units).

[0078] In some embodiments, the carrier implementing the data monitoring method (hereinafter referred to as the carrier) determines the data type of the data to be monitored based on the presentation form of the data to be monitored, whether data operation tests can be performed, etc. The carrier can be a computer program, electronic circuit, database, mobile application, electronic device, cloud computing platform, distributed system, artificial intelligence framework, mathematical model, automation tool and microcontroller, etc., which can implement software or hardware of the algorithm and method flow.

[0079] Step S102: based on the data type of the data to be monitored, input the data to be monitored into a subtree corresponding to a data monitoring model, and obtain the path length of the data to be monitored passing through the subtree.

[0080] In some embodiments, the data monitoring model includes multiple isolated trees, each of which includes a numerical subtree and a non-numerical subtree; the binary tree branching criteria of the numerical subtree and the non-numerical subtree are different.

[0081] In some embodiments, if the data type of the data to be monitored is a numeric type, it is input into the numeric subtree included in each isolated tree to obtain the path length of the data to be monitored through all the numeric subtrees. Alternatively, if the data type of the data to be monitored is a non-numeric type, it is input into the non-numeric subtree included in each isolated tree to obtain the path length of the data to be monitored through all the non-numeric subtrees.

[0082] Step S103: determining whether the data to be monitored is abnormal based on the path length.

[0083] In some embodiments, the carrier determines an abnormality score of the data to be monitored based on the path length, and determines whether the data to be monitored is abnormal based on the abnormality score and a preset threshold.

[0084] In this way, the data monitoring method provided by the embodiment of the present disclosure fully takes into account the properties of different types of data, and uses different subtrees to process numerical data to be monitored and non-numerical data to be monitored, so that the data monitoring model is more flexible in processing data of different data types and can better adapt to different data scenarios.

[0085] Figure 2 A second optional flow chart of the data monitoring method provided in the embodiment of the present disclosure is shown and will be explained according to each step.

[0086] Step S201, determining the data type of the data to be monitored.

[0087] The specific step flow of step S201 is the same as that of step S101 and will not be repeated here.

[0088] Step S202: input the data to be monitored into the numerical subtree of each isolated tree included in the data monitoring model, and determine the path length of the data to be monitored through the numerical subtrees of all isolated trees.

[0089] In some embodiments, the numerical subtree of each isolated tree is determined based on the information gain comparison result. Specifically, when constructing the data monitoring model, the carrier calculates the information entropy of a certain numerical feature (the form or format is the same as the data to be monitored) in the training set; based on the information entropy, calculates the information gain obtained when the training set is divided by the certain numerical feature; sorts the information gains of all numerical features from high to low, and divides the node decision points of the numerical subtree, that is, divides the features in order from high to low according to the information gain; the numerical feature with the largest information gain is the root node, and so on, to construct the numerical subtree of the isolated tree.

[0090] In some embodiments, the carrier inputs the data to be monitored into the numerical subtree, and determines the path length of the data to be monitored through the numerical subtrees of all isolated trees.

[0091] Step S203: input the data to be monitored into the non-numeric subtree of each isolated tree included in the data monitoring model, and determine the path length of the data to be monitored through the non-numeric subtrees of all isolated trees.

[0092] In some embodiments, the non-numeric subtree of each isolated tree is determined based on a random splitting method. Specifically, a non-numeric feature is randomly selected from a training set, and a split point is randomly selected between the minimum value and the maximum value of the non-numeric feature to construct the non-numeric subtree.

[0093] In some optional embodiments, the carrier may perform digitization and dimension reduction on the non-numerical features and the data to be monitored, and process the non-numerical features and the data to be monitored after dimension reduction.

[0094] In some embodiments, the carrier inputs the data to be monitored after dimension reduction into a non-numeric subtree, and determines the path length of the data to be monitored through the non-numeric subtrees of all isolated trees.

[0095] Step S204: determining whether the data to be monitored is abnormal based on the path length.

[0096] In some embodiments, the data to be monitored is any attribute that describes a subject data; if the subject is a product, the data to be monitored includes product size (at least one of length, width and height), price, gross weight, net weight, unit of weight, material density, strength, production cycle and one of inventory.

[0097] In some embodiments, the carrier determines an abnormality score corresponding to the data to be monitored based on the number of data attributes included in the subject corresponding to the data to be monitored and the path length corresponding to the data to be monitored; and determines whether the data to be monitored is abnormal based on the abnormality score and a preset threshold corresponding to the data to be monitored.

[0098] Specifically, the number of data attributes included in the subject includes the number of all attributes of the data included in the subject. For example, if the subject is a product, which includes price, gross weight, net weight, unit of weight, material density, strength, production cycle and inventory, then the number of data attributes included in the subject is 8.

[0099] In some embodiments, the data set to be monitored is represented by a matrix Y M×K The data monitoring process is to let all the monitored data included in each row of data in the monitored data set traverse each isolated tree of the data monitoring model; then calculate the monitored data y in the traversal process ij The length of the path through each isolated tree h(y ij ), the walking method is the same as the isolation forest model, each step is counted as 1, and finally y is calculated based on all path lengths ij The abnormal score s(y ij ,ψ),

[0100] In some optional embodiments, the carrier may calculate the abnormality score of the data to be monitored based on the following formula:

[0101]

[0102] Wherein, it is assumed that the data to be monitored y ij Represents the data in the i-th row and j-th column of the data set to be monitored; the i-th row of the data set to be monitored is the data included in the subject, and the j-th column is the data corresponding to a certain attribute (or a certain column), such as the j-th row is the price, gross weight, etc. ψ is the number of data attributes included in the subject; h(y ij ) is the path length of the data to be monitored through the data monitoring model; c(ψ) is the average length of the data passing through the data monitoring model, and the calculation method includes:

[0103]

[0104] In some embodiments, if the subject includes too few data attributes, indicating that the subject is abnormal, the corresponding path length is set to a fixed value. For example, if the subject includes less than 2 data attributes, the average length of the data passing through the data monitoring model is 0 or 1. After being brought into formula (1), the obtained abnormal score will be significantly higher. In formula (2), H(ψ) is calculated as follows:

[0105] H(ψ)=ln (ψ)+γ (3)

[0106] where γ is Euler's constant; E(h(y ij )) is y ij The average path length of all isolated trees in the data monitoring model.

[0107] In some embodiments, the carrier can set different preset thresholds based on the data monitoring business scenario, subject and data attributes, and determine whether the abnormality score is greater than or equal to the preset threshold. If the abnormality score is greater than or equal to the preset threshold, the data to be monitored is abnormal; if the abnormality score is less than the preset threshold, the data to be monitored is normal (i.e., normal data).

[0108] In some optional embodiments, taking the monitored data set including data corresponding to 1611 subjects (i.e., 1611 data items), and the data attributes corresponding to each subject are 25, i.e., 25 data attributes as an example, the preset threshold is set to 0.9, and the monitoring results show that there are 49 subjects with abnormal data and 1562 subjects with normal positive data.

[0109] Using the data monitoring method provided by the embodiment of the present disclosure, the TP, TN, FP, and FN results are as follows:

[0110] Positive Negative True 1486 44 False 76 5

[0111] The TP, TN, FP, and FN results using traditional isolation forests are as follows:

[0112] Positive Negative True 1238 82 False 253 38

[0113] The recall rate, F1 score and calculation time of the data monitoring method (based on PIPSO-IForest) provided by the embodiment of the present disclosure are as follows:

[0114] Precision Recall F1 Time PIPSO-IForest 0.95 0.99 0.97 0.4

[0115] The recall rate, F1 score, and calculation time using the traditional isolation forest are as follows:

[0116]

[0117] By comparison, the data monitoring method provided by the embodiment of the present disclosure has significant improvements in accuracy, recall rate, F1 score and time compared with the traditional isolation forest data monitoring method.

[0118] In some optional embodiments, after the carrier confirms that the data to be monitored is abnormal data, it can also automatically trigger an early warning reminder and present the detected abnormal data to be monitored to the operation and maintenance personnel in a visual manner, including charts, dashboards, etc., to facilitate users to quickly understand the data abnormalities.

[0119] In this way, through the data monitoring method provided by the embodiment of the present invention, different subtrees are used to process numerical data and non-numerical data, which will have more advantages in terms of targeting and reliability, thereby achieving good monitoring and early warning effects, solving the abnormal monitoring of the product's own master data in production and manufacturing, improving the timeliness of data quality monitoring, and providing basic data guarantees for subsequent production or sales and other aspects.

[0120] Figure 3 A third optional flow chart of the data monitoring method provided in the embodiment of the present disclosure is shown and will be explained according to each step.

[0121] Step S301: construct a first original data set based on original data.

[0122] In some embodiments, the carrier constructs a second original data set based on the original data, and the second original data set is a table, each row represents a data subject, and each column represents the attributes (or data) corresponding to the data subject. For example, if the subject is a product, the attributes include price, gross weight, net weight, unit of weight, material density, strength, production cycle and inventory; the features corresponding to the attributes include numerical features and non-numerical features.

[0123] In some embodiments, the carrier extracts key features from the original data to construct a second original data set; the extraction rules can be set manually, such as extracting data related to product quality, cost, and inventory.

[0124] In some embodiments, the carrier performs numerical processing on the non-numerical features and reduces their dimensionality, and determines that a data set obtained by concatenating the numerical features and the non-numerical features after dimensionality reduction is the first original data set.

[0125] Step S302: construct an isolation tree based on the first original data set.

[0126] In some embodiments, the carrier sets an initial ratio and an initial number; based on the initial ratio and the initial number, determines a plurality of sub-data sets used in constructing an isolated tree; and determines the number of isolated trees to be constructed based on the initial number. The initial ratio includes the ratio of the sub-data sets to the first original data set.

[0127] In some embodiments, the carrier calculates the information gain of the numerical features in any sub-dataset; the numerical feature with the largest information gain is used as the root node of the isolation tree, and the remaining numerical features are used as the target segmentation points of the isolation tree in order of gain information from high to low to construct the numerical subtree of the isolation tree. The non-numerical subtree of the isolation tree is constructed by randomly segmenting the non-numerical data; the numerical subtree and the non-numerical subtree constitute the isolation tree. In this way, an initial number of isolation trees are constructed based on all sub-datasets, and all isolation trees constitute the initial data monitoring model.

[0128] Step S303: determining a first number of isolated trees in the data monitoring model and a first proportion of the sub-data set to the first original data set based on a particle swarm optimization algorithm.

[0129] In some embodiments, in order to achieve the optimal effect of the data monitoring model, hyperparameter optimization is required, that is, determining the optimal number of isolated trees and the proportion of the sub-datasets to the first original data set; the carrier constructs a particle swarm based on the initial number and the initial proportion; wherein the number of isolated trees is the current position of the particles in the particle swarm, and the proportion of the sub-datasets is the current speed of the particles in the particle swarm; according to the current position and current speed of each particle in the particle swarm, the objective function value corresponding to each particle in the particle swarm is calculated, and the objective function value is used to represent the monitoring accuracy of the data monitoring model; the position and speed of each particle are updated, and based on the current position and current speed, the updated objective function value corresponding to each particle in the particle swarm is calculated; the current position and current speed of the target particle in the particle swarm are updated through the ideal position of the target particle, the current position of the target particle and the ideal position of the particle swarm, wherein the target particle is any particle in the particle swarm, and the ideal position represents the particle position that maximizes the objective function value; repeating the above two steps until, in response to meeting the iteration end condition, the first number of isolated trees and the first proportion of the sub-datasets to the first original data set in the data monitoring model are determined based on the ideal position of the target particle.

[0130] Step S304: training the data monitoring model based on the training set.

[0131] In some embodiments, the carrier inputs the monitoring sample data in the training set into the data monitoring model to obtain the predicted type of the monitoring sample data; determines the loss function of the data monitoring model according to the annotation type of the sample data and the predicted type; and adjusts each target segmentation point of each isolated tree in the data monitoring model based on the loss function.

[0132] The method involved in steps S301 to S304 is the training method of the data monitoring model used in steps S101 to S103, or steps S201 to S204.

[0133] In this way, the data monitoring method provided by the embodiment of the present disclosure can effectively capture abnormal data in multiple dimensions by reducing the dimensionality of high-dimensional data and extracting key information from the original data; combine information gain and random segmentation to determine the characteristics of isolated trees, improve the flexibility of the isolated forest, and enable it to better adapt to the abnormal monitoring needs of data with different attributes; and improve the efficiency of the data monitoring model based on the particle swarm optimization algorithm.

[0134] Figure 4 A fourth optional flow chart of the data detection method provided in the embodiment of the present disclosure is shown and will be explained according to each step.

[0135] Figure 4 FIG. 4 is a further description of step S301 .

[0136] In some embodiments, the traditional isolation forest is segmented based on randomly selected features. Its randomness improves computational efficiency, but it is also a double-edged sword and has shortcomings in data anomaly monitoring, including:

[0137] 1) The interpretability is poor, especially for data with high dimensions and correlation and redundancy between them, which makes the data points sparse in space, and the random selection also makes the overall model interpretability poor.

[0138] 2) For high-dimensional data processing, random feature selection leads to poor segmentation results.

[0139] 3) In terms of parameter selection, such as the number of trees and subsampling rate, different parameters have an impact on the abnormal monitoring results. How to select parameters needs to be determined by combining experience and experiments.

[0140] Therefore, when monitoring data for anomalies, considering the above-mentioned defects of the traditional isolation forest, the dimension of the product master data is high-dimensional and complex. Based on this, the embodiment of the present disclosure makes improvements and finally constructs an integrated isolation forest PIPSO-IForest model (i.e., data monitoring model), which may include:

[0141] Step S401, extract key features from the original data and construct a second original data set.

[0142] In some embodiments, the carrier collects original historical data from the business system, cleans, de-duplicates and formats the original historical data to obtain the original data.

[0143] In some embodiments, the original data set includes multi-dimensional descriptive features of the subject, such as price, gross weight, net weight, weight unit, material density, strength, production cycle, inventory, origin, production date and color, etc. When monitoring the data, some data may not need to be monitored. Therefore, the carrier performs principal component analysis (PCA) on the original data to extract the key features that need to be monitored in the original data, removes redundant features, and then presents the extracted key features in a tabular form to construct a second original data set.

[0144] Figure 5 An optional schematic diagram of a second original data set provided by an embodiment of the present disclosure is shown.

[0145] like Figure 5 As shown, each row represents a product, and each column represents the corresponding attributes of the product, such as gross weight (GROSS_WEIGHT), net weight (NET_WEIGHT), length (LENGTH), width (WIDTH), height (HEIGHT), weight unit (WEIGHT_UNIT), price (PRICE) and discount rate (DISCOUNT_RATE).

[0146] In this way, by extracting key features and removing redundant features through principal component analysis, the classification features during binary tree partitioning can be improved when the isolation forest is subsequently constructed.

[0147] Step S402: performing dimensionality reduction processing on the non-numerical features in the second original data set.

[0148] In some embodiments, the carrier digitizes the non-numerical features in the second original data set, converts them into third quantitative data, and then performs dimensionality reduction on the digitized non-numerical features, which may include:

[0149] Compute the mean vector of non-numeric features:

[0150]

[0151] Among them, μ is the mean vector of non-numerical features; n is the number of all non-numerical features, that is, the third number; x i is the i-th non-numeric feature.

[0152] The non-numerical features are decentralized, and the centralized matrix obtained includes:

[0153] Xb=Xa-μ (5)

[0154] Among them, Xb is the centered matrix, and Xa is the original non-numerical matrix, that is, the matrix corresponding to the non-numerical data after digitization in the second data set.

[0155] Construct the covariance matrix V of the centralized matrix Xb. The calculation formula is as follows:

[0156]

[0157] Where: Xb T is the transpose of Xb.

[0158] Perform eigendecomposition on the covariance matrix V to obtain the eigenvalue λ of each dimension j and the corresponding eigenvector w j , and sort the eigenvalues ​​λ in descending order j , where j is the count amount, representing the jth dimension.

[0159] Select the first k eigenvalues ​​Λ=[λ 1 ,λ 2 ,…,λ k ] and the corresponding eigenvector W = [w 1 ,w 2 ,…,w k ] as the basis of the subspace, the k-dimensional data after dimensionality reduction is:

[0160]

[0161] Wherein, X2 is the non-data-type feature matrix after dimension reduction. The non-numerical features involved in step S402 are the features after data processing.

[0162] Step S403: construct a first original data set based on the numerical features in the second original data set and the non-numerical features after dimensionality reduction.

[0163] In some embodiments, the matrix corresponding to the numerical features in the second original data set is a numerical matrix X1, which is combined with the non-numerical feature matrix X2 after dimensionality reduction to obtain the first original data set [X1, X2], where X1 and X2 have the same number of rows.

[0164] Figure 6 An optional schematic diagram of a first original data set provided by an embodiment of the present disclosure is shown. Figure 6 is based on Figure 5 The conversion and dimensionality reduction are obtained. Due to space reasons, only Figure 5 The first 10 lines of content; Figure 6 As shown, the weight unit has been changed from Figure 5 Convert the non-numeric values ​​in the data and reduce their dimension into numerical data.

[0165] Thus, by the data monitoring method described in the embodiment of the present disclosure, the key features of the original data are extracted by principal component analysis, which makes up for the defect that the isolation forest cannot effectively process or the complexity increases for high-dimensional data with high correlation and redundancy between dimensions. It will be an effective way to improve the isolation forest for data quality monitoring and early warning in combination with the principal component analysis algorithm. Specifically, the original data set is reduced in dimension by principal component analysis, and then the principal component after dimension reduction is selected as the feature input to the isolation forest model for quality monitoring. In this way, irrelevant or redundant features can be removed, the feature dimension can be reduced, and key feature information can be extracted, which helps to reduce computational complexity and improve the training efficiency of the model. For the second original data set, the data type is segmented, and the numerical features keep the original features, such as price, size, discount rate, etc. For the non-numerical feature data, the dimension is reduced to extract the key feature information, such as weight unit, product type, etc. Finally, the original numerical data set X1 and the original non-numerical data set X2 after dimension reduction are spliced ​​by column to form a new data set as the first original data set.

[0166] Figure 7 A fifth optional flow chart of the data monitoring method provided in the embodiment of the present disclosure is shown and will be explained according to each step.

[0167] In some embodiments, the traditional isolation forest selects branch features randomly. Although the randomness of feature selection can simplify the model and improve computational efficiency, it will also ignore some important features, causing information loss and lack of interpretability, resulting in low model stability. Therefore, choosing appropriate branch features is the first important aspect of optimizing the isolation forest.

[0168] The disclosed embodiment uses a combination of information gain comparison and random feature selection. By calculating the information gain of each feature attribute, a good decision can be made on the tree partition attribute. The random selection method can simplify the model and improve efficiency. The advantages of both are combined to balance the contradiction between feature selection and randomness, and improve the efficiency and interpretability of the model. The combination of random selection and information gain achieves a certain balance in performance, while retaining the randomness and high performance of the isolation forest.

[0169] Step S501, calculating the information gain of each numerical feature in the first original data set.

[0170] In some embodiments, assuming that the information entropy is H(X1), X1 represents a numerical feature set, according to the calculation result of the information entropy, select the attribute with larger information entropy as the tree node splitting feature, as the optimization branch criterion, and construct an isolated forest binary tree based on this. This can make the generation of isolated trees more targeted. Specifically, the carrier can calculate the information entropy by the following formula:

[0171]

[0172] Among them, P p Represents the probability that X1 is p.

[0173] Assume that a numerical feature in the numerical feature set X1 is F i , then the numerical F i The information gain obtained by partitioning the numerical feature set X1 is:

[0174]

[0175] Where v is each attribute with v values, X1 v It is feature F i Data set at each branch node.

[0176] Step S502: construct a numerical subtree in descending order of information gain.

[0177] In some embodiments, a combination of information gain comparison and random feature selection is used. If the original data is numerical, the information gain method is used to divide the first part. If the original data is non-numerical, the dimensionality is reduced and then divided randomly.

[0178] Specifically, the information gain is calculated for numerical features, such as product price, size, discount rate, etc., to determine which numerical features have higher information gain. It is used for the first half of the isolation tree (i.e., the numerical subtree), and the decision points of each node of the tree are divided from high to low according to the information gain, i.e., the split features. The feature with the largest information gain is used as the root node, and so on. The features of each node are selected by comparing the information gain of each feature.

[0179] Taking a specific example as an example, in the first original data set, the information gain results of each attribute include: price 0.179277, discount rate 0.265170, net weight 0.159690, length 0.147297, width 0.071146. The discount rate is the most important division feature, and the other branches are divided as branch attributes in descending order by information gain. Branch training is performed on each sub-data set in the optimization training set according to the information gain score criterion, and the best branch attributes of each node of each sub-data set are selected, and then evolved into a data monitoring model.

[0180] Step S503: construct a non-numeric subtree in a random partitioning manner.

[0181] In some embodiments, the carrier uses non-numeric features as features of the second half of the isolation tree (i.e., non-numeric subtree), such as weight unit, product model, etc. For these features, the original strategy of the isolation forest is adopted, that is, a feature is randomly selected and a split point is randomly selected between the minimum and maximum values ​​of the feature.

[0182] In this way, the data monitoring method provided by the embodiment of the present disclosure fully considers the properties of numerical and non-numerical features, making the data monitoring model more flexible in processing different types of features to better adapt to different data scenarios. The advantages of both are combined to balance the contradiction between feature selection and randomness, and improve the efficiency and interpretability of the model.

[0183] Figure 8 A fifth optional flow chart of the data monitoring method provided in the embodiment of the present disclosure is shown and will be explained according to each step.

[0184] In some embodiments, after determining the numerical subtrees and non-numerical subtrees of each isolated tree, in order to achieve the optimal effect of the data monitoring model, hyperparameter optimization is required, that is, determining the most appropriate number of isolated trees and the ratio of the number of sub-data sets to the first original data set. A smaller number of sub-data sets and a suitable number of isolated trees can reduce training time and improve the training efficiency of the data monitoring model.

[0185] Step S601, constructing a particle swarm based on the proportion of the sub-datasets and the range of the number of isolated trees.

[0186] In some embodiments, the carrier first constructs an initial data monitoring model based on the initial ratio and initial number, and the aforementioned isolated tree construction method (steps S501 to S503). Based on the ratio of the sub-datasets and the range of the number of isolated trees, a particle swarm is constructed. Each particle in the particle swarm represents a hyperparameter combination, the current position of the particle represents the number of isolated trees in the data monitoring model, and the current speed of the particle represents the ratio of the sub-dataset.

[0187] For example, the number of isolated trees is set between 50 and 200, and the proportion of the sub-dataset is set between 0.1 and 0.5.

[0188] In the disclosed embodiment, the particle swarm optimization algorithm is used to determine the hyperparameters of the data monitoring model. The particle swarm optimization is a random search algorithm based on group collaboration. By simulating the foraging behavior of a flock of birds, a group of random particles (random solutions) are initialized, and then the position and speed of the particles are continuously updated in the iterative process to find the optimal solution. It finds the optimal solution through collaboration and information sharing between individuals in the group.

[0189] Step S602, calculating the objective function value corresponding to each particle in the particle swarm according to the current position and current speed of each particle in the particle swarm.

[0190] In some embodiments, the objective function is used to characterize the accuracy of the data monitoring model, that is, the accuracy of correctly identifying abnormal data. The carrier confirms the objective function corresponding to each particle and calculates the objective function value corresponding to each particle in the particle group based on the current position and current speed.

[0191] Step S603, updating the position and velocity of each particle, and calculating the updated objective function value corresponding to each particle in the particle swarm based on the current position and current velocity.

[0192] In some embodiments, the position of particle a in D-dimensional space is represented by a vector x a =(x 1 ,x 2 ,…,x D ,), the flight speed is represented by vector v a =(v 1 ,v 2 ,…,v D ,).

[0193] Each particle has a fitness value determined by the objective function and knows its best position pbest found so far and its current position x a In addition, each particle also knows the best position gbest found by all particles in the entire group so far (gbest is the best value in pbest). Then the optimal solution is found through iteration.

[0194] Step S604: updating the current position and current speed of the target particle in the particle swarm according to the ideal position of the target particle, the current position of the target particle and the ideal position of the particle swarm.

[0195] In some embodiments, the particle updates itself by tracking pbest and gbest. After finding these two optimal values, the particle updates its speed and position by the following formula:

[0196] v a =ω×v a +c 1 ×rand()×(pbest a -x a )+c 2 ×rand()×(gbest-x a )

[0197] Among them, ω is the inertia factor, c1, c2 are learning factors, and rand() is a random number between (0,1).

[0198] x i =x i +v i

[0199] Step S605 , in response to satisfying the iteration end condition, determining a first number of isolated trees in the data monitoring model and a first ratio of the sub-data set to the first original data set based on the ideal position of the target particle.

[0200] In some embodiments, the carrier repeats step S603 and step S604, and redetermines the objective function value of the particle each time the particle changes position until an iteration end condition is met; the iteration end condition includes reaching a maximum number of iterations or the objective function value converges to a target threshold.

[0201] In some embodiments, after the iteration is completed, the hyperparameter combination with the best performance is selected as the final optimized integrated isolation forest configuration, that is, the ideal position and the ideal speed of the target particle.

[0202] Taking a specific embodiment as an example, after calculation, the number of optimal isolated trees is 57.58, and the first ratio of the optimal sub-dataset to the first original data set is 0.478. Therefore, the first number of isolated trees of the data monitoring model is determined to be 58, and the first ratio is 0.478. The sub-dataset is determined based on the first ratio, the data monitoring model is constructed based on the first number, and the data monitoring model is trained based on the sub-dataset.

[0203] Fig. 9 A training diagram of a data monitoring model provided by an embodiment of the present disclosure is shown.

[0204] like Fig. 9 As shown, 58 isolated trees are constructed based on the method described in step S501 to step S503 and the first quantity calculated, 58 sub-data sets are determined based on the first ratio, and the data monitoring model is trained.

[0205] In this way, through the data monitoring method provided by the embodiment of the present invention, the particle swarm optimization algorithm is used to find the optimal hyperparameter configuration of the data monitoring model, thereby improving the reliability of abnormal data monitoring, and finally using the improved isolation forest model as an integrated device as the final data monitoring model (PIPSO-IForest model); it can better adapt to specific data characteristics and improve monitoring effects.

[0206] Fig.10 An optional structural schematic diagram of a data monitoring device provided in an embodiment of the present disclosure is shown, and will be described according to each part.

[0207] In some embodiments, the data monitoring device 900 includes a data type determination unit 901 , a monitoring unit 902 , and a determination unit 903 .

[0208] The data type determination unit 901 is used to determine the data type of the data to be monitored; the data type includes a numerical type or a non-numerical type;

[0209] The monitoring unit 902 is configured to input the data to be monitored into a subtree corresponding to a data monitoring model based on the data type of the data to be monitored, and obtain a path length of the data to be monitored passing through the subtree;

[0210] The determining unit 903 is used to determine whether the data to be monitored is abnormal based on the path length;

[0211] The data monitoring model includes multiple isolated trees, each of which consists of a numerical subtree and a non-numerical subtree. The numerical data to be monitored is input into the numerical subtree, and the non-numerical data to be monitored is input into the non-numerical subtree.

[0212] The monitoring unit 902 can be used to input the data to be monitored into the numerical subtree of each isolated tree included in the data monitoring model in response to the data type of the data to be monitored being numerical, and determine the path length of the data to be monitored passing through the numerical subtrees of all isolated trees.

[0213] The monitoring unit 902 can be used to input the data to be monitored into the non-numeric subtree of each isolated tree included in the data monitoring model in response to the data type of the data to be monitored being non-numeric, and determine the path length of the data to be monitored passing through the non-numeric subtrees of all isolated trees.

[0214] In some embodiments, the data monitoring device 900 further includes a training unit 904 .

[0215] The training unit 904 is used to train the data monitoring model and can be used to:

[0216] Constructing a first original data set based on the original data; the first original data set includes numerical features and non-numerical features;

[0217] The features are split according to the information gain of the numerical features in the first original data set from high to low, and the numerical subtrees of the isolated tree are constructed; the non-numerical features in the first original data set are randomly split to construct the non-numerical subtrees of the isolated tree;

[0218] Constructing an initial number of isolated trees based on numerical subtrees and non-numerical subtrees; the initial number of isolated trees constitute an original data monitoring model;

[0219] Determine a first number of isolated trees in the data monitoring model and a first ratio of the sub-data set to the first original data set based on a particle swarm optimization algorithm;

[0220] Based on the first number of isolated trees and a first ratio of the sub-data sets to the first original data set, adjusting the original data monitoring model to obtain a data monitoring model;

[0221] The data monitoring model is trained based on the training set.

[0222] The training unit 904 may be used to construct a second original data set based on the original data; the second original data set includes a table, each row of the table represents a data subject, and each column represents a feature corresponding to the data subject; the feature includes a numerical feature and a non-numerical feature;

[0223] The non-numeric features are subjected to dimensionality reduction processing, and the non-numeric features after the dimensionality reduction processing are concatenated with the numerical features to obtain a first original data set.

[0224] The training unit 904 may be used to convert the non-numeric feature into a third quantity data;

[0225] Calculating the mean of the third quantity data to determine a mean vector;

[0226] Determine a centralization matrix according to a difference between the third quantity data and the mean vector;

[0227] Determining a covariance matrix according to the centering matrix and the transpose of the centering matrix;

[0228] Performing eigendecomposition on the covariance matrix to determine eigenvalues ​​and eigenvectors;

[0229] The eigenvalue and the eigenvector are taken as the basis of the subspace, and the dimension of the centered matrix is ​​reduced to obtain non-numerical features after the dimension reduction.

[0230] The training unit 904 may be used to set an initial ratio of the sub-datasets and an initial number of isolated trees, and to construct a data monitoring model based on the initial ratio and initial number;

[0231] Based on the proportion of the sub-dataset and the range of the number of isolated trees, a particle swarm is constructed; wherein the number of isolated trees is the current position of the particles in the particle swarm, and the proportion of the sub-dataset is the current speed of the particles in the particle swarm;

[0232] Calculating an objective function value corresponding to each particle in the particle swarm according to the current position and current speed of each particle in the particle swarm, wherein the objective function value is used to represent the monitoring accuracy of the data monitoring model;

[0233] Update the position and velocity of each particle, and calculate the updated objective function value corresponding to each particle in the particle swarm based on the current position and current velocity;

[0234] Update the current position and current speed of the target particle in the particle swarm by using the ideal position of the target particle, the current position of the target particle and the ideal position of the particle swarm, wherein the target particle is any particle in the particle swarm, and the ideal position represents the particle position that maximizes the objective function value;

[0235] Repeat the above two steps until, in response to satisfying the iteration end condition, the first number of isolated trees in the data monitoring model and the first proportion of the sub-data set to the first original data set are determined based on the ideal position of the target particle.

[0236] The training unit 904 may be used to calculate the information gain of the numerical features in the first original data set;

[0237] The numerical feature with the largest information gain is used as the root node of the isolation tree, and the remaining numerical features are used as the target segmentation points of the isolation tree in the order of gain information from high to low to construct the numerical subtree of the isolation tree.

[0238] The training unit 904 may be used to input the monitoring sample data in the training set into the data monitoring model to obtain the predicted type of the monitoring sample data;

[0239] Determining a loss function of the data monitoring model according to the annotation type of the sample data and the prediction type;

[0240] The target segmentation points of each isolated tree in the data monitoring model are adjusted based on the loss function.

[0241] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0242] Fig.11 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0243] like Fig.11 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0244] Multiple components in the electronic device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0245] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as a data monitoring method. For example, in some embodiments, the data monitoring method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the data monitoring method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the data monitoring method in any other appropriate manner (e.g., by means of firmware).

[0246] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0247] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0248] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0249] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0250] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0251] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0252] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0253] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0254] The above is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A data monitoring method, the method comprising: Determine the type of data to be monitored; The data type includes a numeric type or a non-numeric type; Based on the data type of the data to be monitored, input the data to be monitored into a corresponding subtree of a data monitoring model, and obtain the path length of the data to be monitored passing through the subtree; Based on the path length, determining whether the data to be monitored is abnormal; The data monitoring model includes multiple isolated trees, each of which is composed of a numerical subtree and a non-numerical subtree. The numerical data to be monitored is input into the numerical subtree, and the non-numerical data to be monitored is input into the non-numerical subtree.

2. The method according to claim 1, wherein based on the data type of the data to be monitored, the data to be monitored is input into a subtree corresponding to a data monitoring model, and the path length of the data to be monitored passing through the subtree is obtained, comprising: In response to the data type of the data to be monitored being numerical, the data to be monitored is input into the numerical subtree of each isolated tree included in the data monitoring model, and the path length of the data to be monitored passing through the numerical subtrees of all isolated trees is determined.

3. The method according to claim 1, wherein based on the data type of the data to be monitored, the data to be monitored is input into a subtree corresponding to a data monitoring model, and the path length of the data to be monitored passing through the subtree is obtained, comprising: In response to the data type of the data to be monitored being non-numeric, the data to be monitored is input into the non-numeric subtree of each isolated tree included in the data monitoring model, and the path length of the data to be monitored passing through the non-numeric subtrees of all isolated trees is determined.

4. The method according to claim 1, further comprising training the data monitoring model, comprising: constructing a first original data set based on the original data; The first original data set includes numerical features and non-numerical features; The features are split according to the information gain of the numerical features in the first original data set from high to low, and the numerical subtrees of the isolated tree are constructed; Randomly split the non-numeric features in the first original data set to construct non-numeric subtrees of the isolated tree; Construct an initial number of isolated trees based on numerical subtrees and non-numerical subtrees; The initial number of isolated trees constitute the original data monitoring model; Determine a first number of isolated trees in the data monitoring model and a first ratio of the sub-data set to the first original data set based on a particle swarm optimization algorithm; Based on the first number of isolated trees and a first ratio of the sub-datasets to the first original data set, adjusting the original data monitoring model to obtain a data monitoring model; The data monitoring model is trained based on the training set.

5. The method according to claim 4, wherein constructing the first original data set based on the original data comprises: Constructing a second original data set based on the original data; the second original data set includes a table, each row of the table represents a data subject, and each column represents a feature corresponding to the data subject; the feature includes a numerical feature and a non-numerical feature; The non-numeric features are subjected to dimensionality reduction processing, and the non-numeric features after the dimensionality reduction processing are concatenated with the numerical features to obtain a first original data set.

6. According to the method of claim 5, the dimensionality reduction processing of the non-numerical features comprises: Converting the non-numeric feature into a third quantitative data; Calculating the mean of the third quantity data to determine a mean vector; Determine a centralization matrix according to a difference between the third quantity data and the mean vector; Determining a covariance matrix according to the centering matrix and the transpose of the centering matrix; Performing eigendecomposition on the covariance matrix to determine eigenvalues ​​and eigenvectors; The eigenvalue and the eigenvector are taken as the basis of the subspace, and the dimension of the centered matrix is ​​reduced to obtain non-numerical features after the dimension reduction.

7. The method according to claim 4, wherein determining the first number of isolated trees in the data monitoring model and the first proportion of the sub-dataset to the first original data set based on the particle swarm optimization algorithm comprises: Set the initial proportion of the sub-dataset and the initial number of isolated trees, and build the original data monitoring model based on the initial proportion and initial number; Based on the proportion of the sub-dataset and the range of the number of isolated trees, a particle swarm is constructed; wherein the number of isolated trees is the current position of the particles in the particle swarm, and the proportion of the sub-dataset is the current speed of the particles in the particle swarm; Calculating an objective function value corresponding to each particle in the particle swarm according to the current position and current speed of each particle in the particle swarm, wherein the objective function value is used to represent the monitoring accuracy of the data monitoring model; Update the position and velocity of each particle, and calculate the updated objective function value corresponding to each particle in the particle swarm based on the current position and current velocity; Update the current position and current speed of the target particle in the particle swarm by using the ideal position of the target particle, the current position of the target particle and the ideal position of the particle swarm, wherein the target particle is any particle in the particle swarm, and the ideal position represents the particle position that maximizes the objective function value; Repeat the above two steps until, in response to satisfying the iteration end condition, the first number of isolated trees in the data monitoring model and the first proportion of the sub-data set to the first original data set are determined based on the ideal position of the target particle.

8. The method according to claim 4, wherein the step of segmenting the features in the first original data set in descending order of information gain of the numerical features to construct the numerical subtree of the isolated tree comprises: Calculate the information gain of the numerical features in the first original data set; The numerical feature with the largest information gain is used as the root node of the isolation tree, and the remaining numerical features are used as the target segmentation points of the isolation tree in the order of gain information from high to low to construct the numerical subtree of the isolation tree.

9. The method according to claim 4, wherein the training of the data monitoring model based on the training set comprises: Inputting the monitoring sample data in the training set into the data monitoring model to obtain the predicted type of the monitoring sample data; Determining a loss function of the data monitoring model according to the annotation type of the sample data and the prediction type; The target segmentation points of each isolated tree in the data monitoring model are adjusted based on the loss function.

10. A data monitoring device, comprising: A data type determination unit, used to determine the data type of the data to be monitored; The data type includes a numeric type or a non-numeric type; A monitoring unit, configured to input the data to be monitored into a subtree corresponding to a data monitoring model based on the data type of the data to be monitored, and obtain a path length of the data to be monitored passing through the subtree; a determination unit, configured to determine whether the data to be monitored is abnormal based on the path length; The data monitoring model includes multiple isolated trees, each of which is composed of a numerical subtree and a non-numerical subtree. The numerical data to be monitored is input into the numerical subtree, and the non-numerical data to be monitored is input into the non-numerical subtree.

Images