Enterprise data management platform

Abstract

The invention provides an enterprise data management platform, and belongs to the technical field of data processing. The enterprise data management platform comprises a first anomaly judgment module used for judging target data based on an isolated forest algorithm model to determine first abnormal data; the abnormal value determination module is used for determining a first target abnormal value based on related data of the first abnormal data in response to the first abnormal data meeting a first related condition; the processor is also used for responding to the condition that the first abnormal data does not meet the first related condition and determining a second target abnormal value based on a target clustering algorithm; and the second abnormality judgment module is used for judging whether the first abnormal data is real data or not based on the first target abnormal value, or judging whether the first abnormal data is real data or not based on the second target abnormal value, and sending a judgment result to the target equipment. The accuracy and reliability of abnormal data detection can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to an enterprise data management platform. Background Art

[0002] With the rapid development of information technology, the amount of data in various industries has exploded. At the enterprise operation level, data covers multiple key links such as production, sales, finance, customer relations, etc., and has become an important basis for enterprise decision-making and development.

[0003] However, among the numerous data in an enterprise, the existence of abnormal data cannot be ignored. It may be caused by failure of data collection equipment, such as sensor failure and recording wrong values; it may also be human factors, such as negligence during data entry. Abnormal data will seriously interfere with the results of data analysis and lead to wrong decisions. Existing technologies can perform preliminary data processing, but there are still many deficiencies in the monitoring of abnormal data. The judgment accuracy of abnormal data is low and it is easy to misjudge.

[0004] Therefore, there is an urgent need for a stable and reliable enterprise data management platform. Summary of the invention

[0005] The disclosed embodiment provides an enterprise data management platform to solve the problem that abnormal data in enterprise data is difficult to judge.

[0006] The disclosed embodiment provides an enterprise data management platform, including: a first abnormality judgment module, used to judge target data based on an isolation forest algorithm model to determine first abnormal data; an outlier determination module, configured to determine a first target outlier based on the relevant data of the first outlier in response to the first outlier data satisfying the first relevant condition; and to determine a second target outlier based on the target clustering algorithm in response to the first outlier data not satisfying the first relevant condition; The second abnormality judgment module is used to judge whether the first abnormal data is real data based on the first target abnormal value, or judge whether the first abnormal data is real data based on the second target abnormal value, and send the judgment result to the target device.

[0007] In an exemplary embodiment of the present disclosure, the enterprise data management platform further includes: a related data determination module; A related data determination module, used to determine a plurality of related data of the first abnormal data, and weights corresponding to the plurality of related data; The outlier determination module is further configured to determine first prediction data based on a plurality of related data and weights corresponding to the plurality of related data; A first target abnormal value is determined based on a difference value between the first predicted data and the first abnormal data.

[0008] In an exemplary embodiment of the present disclosure, the relevant data determination module is further configured to: Introducing the data in the first related data set as independent variables into the first linear regression model one by one, and determining the second linear regression model based on the first standard; deriving the independent variables in the second linear regression model one by one, and determining the third linear regression model based on the second standard; until the first condition is met, the target linear regression model is obtained; The first linear regression model is a preset linear regression model; the dependent variable of the first linear regression model is the first abnormal data; the independent variable in the target linear regression model is the related data of the first abnormal data; The parameters of the independent variables in the target linear regression model are normalized to obtain weights corresponding to the relevant data; wherein the first relevant data set is a data set consisting of predetermined data related to the first abnormal data.

[0009] In an exemplary embodiment of the present disclosure, the relevant data determination module is further configured to: Determine the first significance probability value of the introduced variable based on the target test rule; In response to the first significance probability value being less than the first probability value, adding the introduced variable to the first linear regression model to obtain a second linear regression model; The relevant data determination module is also specifically used for: Determine the second significance probability value of the derived variable based on the target test rule; In response to the second significance probability value being greater than the second probability value, the derived variable is removed from the second linear regression model to obtain a third linear regression model.

[0010] In an exemplary embodiment of the present disclosure, the enterprise data management platform further includes: a standard adjustment module; a criteria adjustment module, configured to reduce the first probability value based on a first adjustment step size in response to the amount of data in the first correlation data set being greater than the first correlation amount; In response to the number of independent variables in the second linear regression model being greater than the second correlation number, the second probability value is increased based on a second adjustment step size.

[0011] In an exemplary embodiment of the present disclosure, the enterprise data management platform further includes: a relevant condition determination module; The correlation condition determination module is used to respond that the first abnormal data satisfies the first correlation condition in response to the first abnormal data having a correlation data quantity greater than a first correlation quantity or an importance level greater than a preset level.

[0012] In an exemplary embodiment of the present disclosure, the outlier determination module is further configured to: Performing clustering processing on a data set consisting of the first abnormal data and the historical first abnormal data based on a target clustering algorithm to obtain a target clustering result; In response to the first abnormal data being in the target cluster and the distance between the first abnormal data and the cluster center of the target cluster being less than the first distance, determining a second target abnormal value based on the first method; In response to the first abnormal data not being in the target cluster and / or the distance between the first abnormal data and the cluster center of the target cluster being greater than or equal to the first distance, determining a second target abnormal value based on a second manner; The calculation standards of the first method and the second method are different.

[0013] In an exemplary embodiment of the present disclosure, the enterprise data management platform further includes: a distance adjustment module; a distance adjustment module, configured to reduce the first distance based on the first distance adjustment step in response to the density of the target cluster being greater than the first density; In response to the density of the target cluster being less than the second density, the first distance is increased based on the second distance adjustment step size.

[0014] In an exemplary embodiment of the present disclosure, the enterprise data management platform further includes: a first data processing module; The first data processing module is configured to send abnormal information to the first device in response to the first abnormal data not being real data.

[0015] In an exemplary embodiment of the present disclosure, the enterprise data management platform further includes: a second data processing module; The second data processing module is used to control the second device to store the first abnormal data and related data of the first abnormal data in response to the first abnormal data not being real data.

[0016] The beneficial effects of the enterprise data management platform provided by the embodiments of the present disclosure are: The present disclosure uses the first abnormal judgment module and the isolation forest algorithm model to preliminarily screen the target data, and can quickly and effectively identify potential first abnormal data. The present disclosure takes into account the diversity of abnormal data. When the first abnormal data meets a specific first related condition, the first target abnormal value is determined based on the relevant information of these data, which helps to accurately capture abnormalities related to logic or data patterns. When the first abnormal data does not meet these conditions, the target clustering algorithm is used to determine the second target abnormal value, which improves the accuracy and reliability of abnormal data detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0018] Figure 1 is a structural diagram of a first enterprise data management platform provided by an embodiment of the present disclosure; Figure 2 It is a structural diagram of the second enterprise data management platform provided by the embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] In order to enable people in the technical field to better understand the present solution, the technical solution in the embodiment of the present solution will be clearly described below in conjunction with the drawings in the embodiment of the present solution. Obviously, the described embodiment is an embodiment of a part of the present solution, not all of the embodiments. Based on the embodiments in the present solution, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present solution.

[0020] The term "including" and any other variations in the specification and claims of this solution and the above drawings mean "including but not limited to", and is intended to cover non-exclusive inclusions and is not limited to the examples listed in the text. In addition, the terms "first" and "second" are used to distinguish different objects, not to describe a specific order.

[0021] The following is a detailed description of the implementation of the present disclosure in conjunction with the specific drawings: Figure 1 This is a schematic diagram of the structure of the first enterprise data management platform provided by the embodiment of the present disclosure. Figure 1 , the enterprise data management platform includes: The first abnormality judgment module 10 is used to judge the target data based on the isolation forest algorithm model to determine the first abnormal data.

[0022] The outlier determination module 11 is used to determine a first target outlier value based on the related data of the first outlier data in response to the first outlier data satisfying the first related condition; and is also used to determine a second target outlier value based on the target clustering algorithm in response to the first outlier data not satisfying the first related condition.

[0023] The second abnormality judgment module 12 is used to judge whether the first abnormal data is real data based on the first target abnormal value, or judge whether the first abnormal data is real data based on the second target abnormal value, and send the judgment result to the target device.

[0024] In this embodiment, the isolation forest algorithm model can process and judge the data in the target data and its historical data. The training process of the isolation forest algorithm is relatively simple and efficient. It mainly builds a tree structure through random sampling and recursive partitioning, and does not require a large number of parameter iteration updates like some deep learning models. Therefore, when faced with large-scale data sets, the isolation forest algorithm can complete training in a shorter time and quickly give anomaly detection results. Therefore, the isolation forest algorithm is selected in this article to process a large amount of enterprise data and make preliminary judgments on anomalies. The parameters of the isolation forest algorithm during the training process can be set according to the inherent parameters of the isolation forest model or the parameters when solving similar problems.

[0025] Target data refers to a piece of data received by the enterprise's central data platform, or it can be data from a predetermined data list. Because the amount of data collected by the enterprise is relatively large, it will take more resources and time to judge the outliers of each piece of data, so only some of the data can be judged and processed.

[0026] The target data and historical data are processed and judged by the isolation forest algorithm model, and the data obtained is the first abnormal data.

[0027] Considering that some data itself has strong volatility or has large fluctuations due to some objective reasons, rather than due to manual filling errors or the submitter's subjective desire to hide, the first abnormal data is only preliminarily determined to be data that may be abnormal, not necessarily abnormal data. The purpose of the abnormal value determination module 11 is to determine the abnormal value of the first abnormal data. The larger the abnormal value, the greater the possibility that the first abnormal data is abnormal false data.

[0028] Different methods of determining outliers should be used for different data. If the first abnormal data meets the first relevant condition, the first target outlier value can be determined based on the relevant data of the first abnormal data. The first target outlier value is the outlier value corresponding to the first abnormal data. Conversely, if the first abnormal data does not meet the first relevant condition, the second target outlier value can be determined based on the target clustering algorithm. The second target outlier value is also the outlier value corresponding to the first abnormal data. The difference between the first target outlier value and the second target outlier value is only the different determination methods.

[0029] The first relevant condition can be judged in the following manner: In one embodiment of the present disclosure, the enterprise data management platform further includes: a relevant condition determination module 15.

[0030] The correlation condition determination module 15 is configured to respond that the first abnormal data satisfies the first correlation condition in response to the number of correlation data of the first abnormal data being greater than the first correlation number or the importance level of the first abnormal data being greater than a preset level.

[0031] In this embodiment, the setting of the first related condition is based on the characteristics of two different means, taking into account that a large number of related data means more information dimensions to describe and understand the first abnormal data. These rich data can reflect the characteristics, relationships and trends of the first abnormal data from different angles, and thus the analysis and prediction based on these data are more comprehensive and accurate. For example, in the sales data of an enterprise, when the amount of customer information, product information, market environment information and other data related to a certain abnormal sales data is large, the cause of the abnormal sales data can be analyzed more accurately to determine whether it is really abnormal and the degree of abnormality.

[0032] Secondly, if the first abnormal data itself has a high importance level, it means that it plays a key role in the business decision-making and analysis of the enterprise, and its abnormality may have a significant impact. For example, the core financial indicator data of the enterprise, the key performance data of major projects, etc. For these important data, more sophisticated and targeted analysis methods are needed to determine the abnormal values ​​based on their related data, so as to more fully explore the information behind the data, ensure that the judgment of its abnormal situation is accurate, and avoid major decision-making errors caused by wrong judgments.

[0033] Therefore, when the number of relevant data of the first abnormal data is greater than the first relevant number, or the importance level of the first abnormal data is greater than the preset level, the first abnormal data meets the first relevant condition. The first relevant number and the preset level can be determined in advance based on experience or actual scenarios. The importance level of the first abnormal data can be preset by relevant personnel during the operation of the enterprise. For example, data related to finance, key performance or sales can be appropriately preset to a higher importance level, and outdated or invalid data can be preset to a lower importance level.

[0034] In this embodiment, the purpose of the second abnormality judgment module 12 is to determine whether the first abnormal data is real data. Real data should be understood as correct data that is not caused by human error. The opposite data is false data.

[0035] Considering that the outlier value of the first outlier data may be the first target outlier value or the second target outlier value, and the first target outlier value and the second target outlier value have different sources, that is, they are obtained in different ways, different thresholds should be set to determine whether the first outlier data is real data.

[0036] For example, in response to the first target abnormality value being greater than or equal to the first abnormality threshold, the first abnormal data is not true data.

[0037] In response to the first target abnormality value being less than the first abnormality threshold, the first abnormal data is true data.

[0038] In response to the second target abnormality value being greater than or equal to the second abnormality threshold, the first abnormal data is not true data.

[0039] In response to the second target abnormality value being less than the second abnormality threshold, the first abnormal data is true data.

[0040] The first abnormal threshold and the second abnormal threshold can be adjusted according to experience or experimental process.

[0041] Through the judgment of the second abnormality judgment module 12, it can be obtained whether the first abnormal data is real data, and the judgment result can also be sent to a related device, which can be a computer or mobile phone dedicated to the enterprise manager.

[0042] It can be concluded from the above that the present disclosure uses the first abnormality judgment module 10 to perform preliminary screening of the target data using the isolation forest algorithm model, and can quickly and effectively identify potential first abnormal data. The present disclosure takes into account the diversity of abnormal data. When the first abnormal data meets a specific first related condition, the first target abnormal value is determined based on the relevant information of these data, which helps to accurately capture abnormalities related to logic or data patterns. When the first abnormal data does not meet these conditions, the target clustering algorithm is used to determine the second target abnormal value, which improves the accuracy and reliability of abnormal data detection.

[0043] From the above description, it can be seen that if the number of related data of the first abnormal data is greater than the first related number or the importance level of the first abnormal data is greater than the preset level, the judgment of the first abnormal data is determined based on the data related thereto, and the related data of the first abnormal data and the first target abnormal value can be determined as follows: Figure 2 Schematic diagram of the structure of the second enterprise data management platform provided by the embodiment of the present disclosure. Figure 2 In one embodiment of the present disclosure, the enterprise data management platform further includes: a related data determination module 13.

[0044] The relevant data determination module 13 is specifically used to determine a plurality of relevant data of the first abnormal data, and weights corresponding to the plurality of relevant data.

[0045] The outlier determination module 11 is further specifically configured to determine first prediction data based on a plurality of related data and weights corresponding to the plurality of related data.

[0046] A first target abnormal value is determined based on a difference value between the first predicted data and the first abnormal data.

[0047] In this embodiment, we can first determine the related data of the first abnormal data and their corresponding weights, where the weight is the degree of influence of the related data on the first abnormal data. Then, we can determine a predicted value based on the related data and its corresponding weight, and then calculate the difference between the predicted value and the first abnormal data, and then determine the first target abnormal value based on this difference.

[0048] The relevant data of the first outlier and its corresponding weight can be determined as follows: In one embodiment of the present disclosure, the relevant data determination module 13 is further configured to: The data in the first related data set are introduced as independent variables into the first linear regression model one by one, and the second linear regression model is determined based on the first standard; the independent variables in the second linear regression model are introduced one by one, and the third linear regression model is determined based on the second standard; until the first condition is met, the target linear regression model is obtained.

[0049] The first linear regression model is a preset linear regression model; the dependent variable of the first linear regression model is the first abnormal data; and the independent variable in the target linear regression model is the related data of the first abnormal data.

[0050] The parameters of the independent variables in the target linear regression model are normalized to obtain weights corresponding to the relevant data; wherein the first relevant data set is a data set consisting of predetermined data related to the first abnormal data.

[0051] In this embodiment, it should be noted that the execution time of the relevant data determination module 13 is performed before all the steps, that is, the relevant data and the corresponding weights of each data should be determined in advance by the relevant data determination module 13, otherwise the judgment of the first relevant condition in the outlier determination module cannot be executed.

[0052] The first relevant data set is a data set consisting of data predetermined to be related to the first abnormal data, and should be understood as a data set consisting of data that is predetermined by humans through common sense or experience and may be related to the first abnormal data. It is a data set that may contain irrelevant data. It should be noted that when determining the first relevant data set, if there is data that cannot be determined to be related, it should also be included in the first relevant data set to avoid inaccurate subsequent judgments.

[0053] The first linear regression model is a preset linear regression model. The dependent variable of the first linear regression model is the first abnormal data. There is no independent variable. The first linear regression model can be ,in is the first abnormal data, is the intercept, is the error term.

[0054] Introduction process: Select independent variables one by one from the first relevant data set , which is introduced into the model to obtain , using the least squares method to estimate the model parameters , based on the first criterion, determining whether the introduced variable should remain in the first linear regression model, until the last data in the first related data set is introduced, the obtained linear regression model is the second linear regression model.

[0055] Induction process: Based on the second linear regression model that has introduced multiple independent variables, each independent variable is derived one by one. , based on the second criterion, determining whether the derived variable should be eliminated from the second linear regression model, until the last independent variable of the second linear regression model is derived, thereby obtaining a third linear regression model; It is determined whether the first condition is met. If the first condition is not met, the above-mentioned introduction process and extraction process are repeated until the first condition is met to obtain the target linear regression model.

[0056] In this process, the first criterion refers to determining whether to add the introduced variable to the variable of the first linear regression model through the target test rule, and the second criterion refers to determining whether to add the derived variable to the variable of the first linear regression model through the target test rule, specifically: In one embodiment of the present disclosure, the relevant data determination module 13 is further configured to determine a first significance probability value of the introduced variable based on a target verification rule.

[0057] In response to the first significance probability value being less than the first probability value, an introduction variable is added to the first linear regression model to obtain a second linear regression model.

[0058] The relevant data determination module 13 is specifically further used to determine the second significance probability value of the derived variable based on the target test rule.

[0059] In response to the second significance probability value being greater than the second probability value, the derived variable is removed from the second linear regression model to obtain a third linear regression model.

[0060] In this embodiment, the target test rule can be a t test or an F test. In this embodiment, considering that the t test is mainly used to test whether a single independent variable has a significant effect on the dependent variable, the F test is mainly used to test the significance of the entire regression model, that is, to determine whether all independent variables as a whole have a significant effect on the dependent variable. Therefore, the t test is selected as the target test rule to obtain a significant probability value, that is, the first significant probability value of the introduced variable or the second significant probability value of the derived variable.

[0061] The first significance probability value and the second significance probability value are essentially the same, and both are significance probability values ​​of the corresponding data.

[0062] In the introduction process, we start from a model without any independent variables and consider introducing independent variables into the model one by one. For each variable to be selected, its corresponding value is calculated. If the first significant probability value is less than the preset first probability value for introduction, the introduced variable is introduced into the model. Then continue to perform the same test on the remaining independent variables until no independent variable meets the introduction conditions.

[0063] In the process of eliciting, the independent variables in the obtained second linear regression model are derived one by one. The significance probability value corresponding to each independent variable is calculated. If the second significance probability value is greater than the preset second probability value for elimination, the variable is eliminated from the model. Then the remaining independent variables are tested until no independent variable meets the elimination condition.

[0064] The smaller the significance probability value, the more significant the influence of the variable on the independent variable, and it can be introduced into the model. On the contrary, the larger the significance probability value, the less significant the influence of the variable on the independent variable, and it can be removed from the model.

[0065] The parameters of each independent variable in the final target regression model can be normalized and used as their corresponding weights. The positive and negative signs of the parameter estimates of the independent variables can intuitively reflect the direction of the relationship between the independent variable and the dependent variable. If the parameter estimate is positive, it means that the independent variable and the dependent variable are positively correlated, that is, when the independent variable increases, the dependent variable also tends to increase; if the parameter estimate is negative, it indicates a negative correlation. The absolute value of the parameter reflects the relative size of the influence of the independent variable on the dependent variable to a certain extent. The larger the absolute value, the more significant the influence of the independent variable on the dependent variable. Therefore, it can be normalized and used as its corresponding weight.

[0066] After the above derivation and description, the target regression model of each data can be obtained. When the first target outlier value needs to be calculated, the target regression model corresponding to the first outlier data can be obtained according to the pre-stored target regression model corresponding to each data, and then the relevant data of the current first outlier data is input into the target regression model, and the value of the dependent variable is obtained, which is the first predicted data. The first target outlier value can be determined according to the difference between the first predicted data and the first outlier data. The difference between the first predicted data and the first outlier data can be the absolute value of the data difference after the first predicted data and the first outlier data are normalized. The purpose of normalization is to eliminate the problem of inconsistent evaluation standards caused by inconsistent units or data bases of different data. For example, the first target outlier value can be obtained through Table 1:

[0067] It can be concluded from the above that the present disclosure can further process the first abnormal data by introducing the relevant data determination module 13 and the abnormal value determination module 11, and further considers other data related to the abnormal data and their weights, so as to more accurately determine the abnormal value. This embodiment uses the linear regression model (including the introduction and derivation process) through the relevant data determination module 13 to determine the data related to the first abnormal data and its weight. This embodiment uses the significance probability value to determine which data has a significant impact on the abnormal data, thereby assigning reasonable weights to them, thereby improving the accuracy and reliability of abnormal data detection.

[0068] In one embodiment of the present disclosure, the enterprise data management platform further includes: a standard adjustment module 14 .

[0069] The standard adjustment module 14 is configured to reduce the first probability value based on a first adjustment step in response to the data quantity in the first correlation data set being greater than the first correlation quantity.

[0070] In response to the number of independent variables in the second linear regression model being greater than the second correlation number, the second probability value is increased based on a second adjustment step size.

[0071] In this embodiment, considering that when the number of data in the first related data set is greater than the first related number, it means that there are more independent variables to choose from. In order to avoid introducing too many insignificant variables and causing model overfitting, the first probability value can be appropriately lowered. The adjusted value is the first adjustment step, and the first adjustment step can be determined by the first formula.

[0072] The first formula can be: ,in, represents the first adjustment step, is the first adjustment coefficient, the value range can be adjusted according to the actual situation, and is used to control the overall size of the adjustment step. represents the number of data in the first related dataset, is the first relevant quantity, represents the variance of the first related data set, Indicates the number of independent variables in the current regression model. The first related number can be preset based on experience. is a preset maximum number of independent variables, which is used to normalize the impact of model complexity. The value of should be related to the amount of data in the first related data set. The larger the amount of data in the first related data set, The larger the value of is, the smaller the amount of data in the first related data set is. The smaller the value, for example, it can be a certain proportion of the data in the first relevant data set, and rounded up, for example, it can be three-fifths of the amount of data in the first relevant data set.

[0073] The logic of the first formula is: the degree to which the number of data in the data set exceeds , when the number of data in the first related data set Exceeding the first relevant quantity When the number is larger, it means there are more independent variables to choose from. In stepwise regression, if it is not strictly controlled, it is easy to introduce too many insignificant variables, resulting in overfitting of the model. An overfitted model performs well on the training data, but has poor generalization ability on new data. Therefore, the greater the degree of excess, the larger the adjustment step size is needed to reduce the first probability value, making the standard for introducing variables more stringent, so as to screen out the variables that really have a significant impact on the dependent variable.

[0074] The variance of the dataset It reflects the degree of data dispersion. The larger the variance, the greater the fluctuation of the data, and the more complex and unstable the relationship between variables. In this case, more caution is needed when introducing variables, because a seemingly significant variable may only show correlation with the dependent variable due to random fluctuations in the data, not a true causal relationship. Therefore, the larger the variance, the larger the adjustment step should be to reduce the risk of introducing false significant variables.

[0075] Model complexity , using the number of independent variables already in the current model to measure the complexity of the model. The more complex the model (i.e. The larger the independent variable, the more likely it is to have overfitting problems. As the number of independent variables increases, the model will overfit the noise and outliers in the training data and ignore the true laws of the data. Therefore, when the model complexity is high, a larger adjustment step size is required to reduce the first probability value and strictly control the introduction of new variables to prevent the model from becoming too complex.

[0076] Similarly, the second adjustment step is used to increase the second probability value when the number of independent variables in the second linear regression model is greater than the second correlation number. The second adjustment step can be determined according to the second formula.

[0077] The second formula can be: ,in, represents the second adjustment step, is the second adjustment coefficient, the value range can be adjusted according to the actual situation, and is used to control the overall size of the adjustment step. represents the number of independent variables in the second linear regression model, is the second relevant quantity, Represents the covariance matrix of the independent variables in the second linear regression model, and its determinant is , the coefficient of determination of the current regression model after each induction is (which can be calculated based on, for example, the Akaike Information Criterion or the Bayesian Information Criterion), is a very small positive number used to avoid the situation where the covariance matrix determinant is zero, for example, it can be 0.001. The second correlation number can be preset based on experience.

[0078] The logic of the second formula is: the degree to which the number of data in the data set exceeds , when the number of independent variables in the second linear regression model Exceeding the second relevant quantity If the value is too high, it means that the current model contains some unimportant independent variables. In the reverse elimination process of stepwise regression, if the standard is too strict, some variables that contribute to the model may be mistakenly eliminated, causing the model to lose important information and the fitting effect to deteriorate. Therefore, the greater the degree of excess, the larger the adjustment step is needed to increase the second probability value, making the standard for eliminating variables more relaxed and retaining more potentially useful variables.

[0079] The determinant of the covariance matrix of the data set , the determinant of the covariance matrix reflects the degree of correlation between the independent variables. The smaller the determinant value, the stronger the correlation between the independent variables, and the more likely it is to have multicollinearity problems. Multicollinearity can lead to unstable model parameter estimation and make the significance test results of some variables unreliable. In this case, it is necessary to be more cautious when eliminating variables, because a variable may seem insignificant, which may be due to the influence of multicollinearity, but in fact it has an impact on the dependent variable. Therefore, the smaller the determinant value, the larger the adjustment step should be to increase the second probability value and reduce the possibility of incorrectly eliminating variables.

[0080] Goodness of fit of the model , after adjustment (i.e., each time the lead is drawn) To measure the goodness of fit of the model. The higher the goodness of fit, the stronger the model's ability to explain the data. When the model already has a good fit, removing variables may destroy this good fit and cause the model performance to deteriorate. Therefore, the higher the goodness of fit, the larger the adjustment step should be to increase the second probability value, remove variables more carefully, and ensure the stability and accuracy of the model.

[0081] It can be concluded from the above that in this embodiment, when the number of data in the first related data set exceeds the preset first related number, the standard adjustment module 14 will appropriately reduce the first probability value according to the first adjustment step, taking into account the size, variance and complexity of the data set of the current model, ensuring that only variables that truly have a significant impact on the dependent variable will be introduced into the model, effectively avoiding the overfitting problem caused by introducing too many insignificant variables, and improving the generalization ability of the model. In this embodiment, when the number of independent variables in the second linear regression model exceeds the preset second related number, the standard adjustment module 14 will increase the second probability value according to the second adjustment step, taking into account the degree of correlation of the independent variables of the data set and the goodness of fit of the model, ensuring that important variables will not be mistakenly eliminated, helping to retain useful information in the model, preventing the degradation of model performance due to the elimination of too many variables, and improving the accuracy and reliability of abnormal data detection.

[0082] In one embodiment of the present disclosure, the outlier determination module 11 is further configured to: Performing clustering processing on a data set consisting of the first abnormal data and the historical first abnormal data based on a target clustering algorithm to obtain a target clustering result; In response to the first abnormal data being in the target cluster and the distance between the first abnormal data and the cluster center of the target cluster being less than the first distance, determining a second target abnormal value based on the first method; In response to the first abnormal data not being in the target cluster and / or the distance between the first abnormal data and the cluster center of the target cluster being greater than or equal to the first distance, determining a second target abnormal value based on a second manner; The calculation standards of the first method and the second method are different.

[0083] In this embodiment, the target clustering algorithm may be a density-based clustering algorithm, which performs a clustering operation on a data set consisting of the first abnormal data and the historical first abnormal data. The purpose of clustering is to divide data points into different clusters so that data points within the same cluster have high similarity, while data points in different clusters have large differences.

[0084] Considering that the isolation forest mainly judges anomalies based on the degree of isolation of data points, when the data distribution is relatively complex, especially when there are uniform density areas, data points at the edge of these areas or relatively sparse positions will be misjudged as abnormal points. Because the isolation forest is not very good at capturing the local density characteristics of the data, it may give wrong judgments for those data points that seem to be isolated globally but are normal in the local uniform density area. The density-based clustering algorithm (Density-Based Spatial Clustering of Applications with Noise, DBSCAN) is a density-based spatial clustering algorithm that identifies clusters and noise points by defining the density of data points. In areas with uniform density, DBSCAN can accurately divide the data points in these areas into a cluster, and can well distinguish whether the points at the boundary of density change belong to a cluster or anomaly. It can find clusters of any shape, and is very sensitive to local density changes in data, and can effectively identify uniform density areas in the data and abnormal points therein. Therefore, in this embodiment, the DBSCAN algorithm is considered to be used to obtain the target clustering result through clustering processing, that is, the data set is divided into several clusters. Among them, the target cluster is the cluster with the smallest data standard deviation in the cluster. The smaller the standard deviation, the more uniform the density. Therefore, the clustering results of DBSCAN and the anomaly detection results of Isolation Forest are comprehensively judged. If the point is located inside the uniform density cluster identified by DBSCAN and the distance from the cluster center is within a certain range, then it can be considered that the point may not be a true anomaly point, but a misjudgment caused by the limitations of the Isolation Forest algorithm, and it is removed from the anomaly point set. On the contrary, if the noise points identified by DBSCAN also have a high anomaly score in the Isolation Forest algorithm, then these points can be further confirmed as anomalies, improving the accuracy of anomaly detection.

[0085] Therefore, when the first abnormal data is in the target cluster and the distance between the first abnormal data and the cluster center of the target cluster is less than the first distance, the second target abnormal value is determined based on the first method. When the first abnormal data is not in the target cluster and / or the distance between the first abnormal data and the cluster center of the target cluster is greater than or equal to the first distance, the second target abnormal value is determined based on the second method. The first distance can be determined empirically.

[0086] Both the first method and the second method can determine the second outlier value based on the distance between the first outlier data and the cluster center of the target cluster, but the difference lies in the different calculation standards of the first method and the second method, which can be specifically reflected in the different slopes of the formula or the different corresponding relationships in the mapping table.

[0087] For example, the first method is the third formula, which can be , the second method can be the fourth formula, which can be ,in, represents the second outlier, Represent the first slope and the second slope respectively, which can be determined experimentally. represents the initial outlier value, which can be determined experimentally.

[0088] In one embodiment of the present disclosure, the enterprise data management platform further includes: a distance adjustment module 16; a distance adjustment module 16, configured to reduce the first distance based on the first distance adjustment step in response to the density of the target cluster being greater than the first density; In response to the density of the target cluster being less than the second density, the first distance is increased based on the second distance adjustment step size.

[0089] In this embodiment, considering that for clusters with higher density, the distance between data points is relatively small, the distance threshold should be set smaller to set the first distance more strictly; while for clusters with lower density, the distance threshold should be increased accordingly to set the first distance more loosely.

[0090] Therefore, when the density of the target cluster is greater than the first density, the first distance is reduced based on the first distance adjustment step; when the density of the target cluster is less than the second density, the first distance is increased based on the second distance adjustment step. The first distance adjustment step and the second distance adjustment step can be determined based on experiments.

[0091] It can be concluded from the above that the present disclosure uses the DBSCAN algorithm to cluster the first abnormal data and the historical first abnormal data through the abnormal value determination module 11, and combines the abnormal detection results of the isolation forest algorithm to make up for the misjudgment that may be caused by the isolation forest under complex data distribution, especially for data points in the uniform density area, DBSCAN can accurately identify its clustering attributes, thereby avoiding misjudging these normal points as abnormal points. In this embodiment, the distance adjustment module 16 can dynamically adjust the first distance according to the density of the target cluster. For clusters with higher density, the module will reduce the first distance to more strictly screen outliers; and for clusters with lower density, the module will increase the first distance to set the outlier standard more loosely, so that the anomaly detection is more in line with the actual situation of the data, and the accuracy and reliability of abnormal data detection are improved.

[0092] In one embodiment of the present disclosure, the enterprise data management platform further includes: a first data processing module 17; The first data processing module 17 is configured to send abnormal information to the first device in response to the first abnormal data not being real data.

[0093] The enterprise data management platform further includes: a second data processing module 18; The second data processing module 18 is configured to control the second device to store the first abnormal data and related data of the first abnormal data in response to the first abnormal data not being real data.

[0094] In this embodiment, the first data processing module 17 can send relevant abnormal information to the first device after obtaining that the first abnormal data is not real data. The first device can be the mobile phone of the relevant personnel or the relevant device of the data uploader, so as to remind them to make corrections or review again.

[0095] The second data processing module 18 can store the first abnormal data and the related data of the first abnormal data as evidence. In the present application, the related data of the first abnormal data is assumed to be real data.

[0096] It can be concluded from the above that in the present disclosure, when the first data processing module 17 determines that the first abnormal data is not real data, it can send abnormal information to the first device, which is very important for timely discovery and correction of data errors, because it can ensure that relevant personnel (such as data uploaders, data reviewers or data managers) obtain error information in time and take necessary corrective measures. The second data processing module 18 can retain the first abnormal data and its related data as evidence to facilitate enterprise management.

[0097] The above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. An enterprise data management platform, characterized in that: include: A first abnormality judgment module, used to judge the target data based on the isolation forest algorithm model to determine the first abnormal data; an abnormal value determination module, configured to determine a first target abnormal value based on the correlation data of the first abnormal data in response to the first abnormal data satisfying a first correlation condition; further configured to determine, in response to the first abnormal data not satisfying the first correlation condition, a second target abnormal value based on a target clustering algorithm; The second abnormality judgment module is used to judge whether the first abnormal data is real data based on the first target abnormal value, or judge whether the first abnormal data is real data based on the second target abnormal value, and send the judgment result to the target device.

2. The enterprise data management platform according to claim 1, characterized in that: Also includes: Related data determination module; The related data determination module is used to determine a plurality of related data of the first abnormal data, and weights corresponding to the plurality of related data; The outlier determination module is further configured to determine first prediction data based on the plurality of related data and weights corresponding to the plurality of related data; A first target abnormal value is determined based on a difference value between the first predicted data and the first abnormal data.

3. The enterprise data management platform according to claim 2, characterized in that: The relevant data determination module is specifically used for: Introducing the data in the first related data set as independent variables into the first linear regression model one by one, and determining the second linear regression model based on the first standard; deriving the independent variables in the second linear regression model one by one, and determining the third linear regression model based on the second standard; until the first condition is met, the target linear regression model is obtained; The first linear regression model is a preset linear regression model; the dependent variable of the first linear regression model is the first abnormal data; The independent variable in the target linear regression model is the related data of the first abnormal data; The parameters of the independent variables in the target linear regression model are normalized to obtain weights corresponding to the relevant data; wherein the first relevant data set is a data set consisting of predetermined data related to the first abnormal data.

4. The enterprise data management platform according to claim 3, characterized in that: The relevant data determination module is also specifically used for: Determine the first significance probability value of the introduced variable based on the target test rule; In response to the first significance probability value being less than a first probability value, adding the introduced variable to the first linear regression model to obtain the second linear regression model; The relevant data determination module is also specifically used for: Determining a second significance probability value of the induced variable based on the target test rule; In response to the second significance probability value being greater than the second probability value, the derived variable is removed from the second linear regression model to obtain the third linear regression model.

5. The enterprise data management platform according to claim 4, characterized in that: Also includes: Standard adjustment module; The standard adjustment module is configured to reduce the first probability value based on a first adjustment step size in response to the amount of data in the first correlation data set being greater than a first correlation amount; In response to the number of independent variables in the second linear regression model being greater than a second correlation number, the second probability value is increased based on a second adjustment step size.

6. The enterprise data management platform according to claim 1, characterized in that: Also includes: Related condition determination module; The correlation condition determination module is configured to respond that, in response to the number of correlation data of the first abnormal data being greater than a first correlation number, or the importance level of the first abnormal data being greater than a preset level, the first abnormal data satisfies a first correlation condition.

7. The enterprise data management platform according to claim 1, characterized in that: The outlier determination module is further specifically used for: Performing clustering processing on a data set consisting of the first abnormal data and the historical first abnormal data based on the target clustering algorithm to obtain a target clustering result; In response to the first abnormal data being in a target cluster and the distance between the first abnormal data and the cluster center of the target cluster being less than a first distance, determining the second target abnormal value based on a first manner; In response to the first abnormal data not being in the target cluster and / or the distance between the first abnormal data and the cluster center of the target cluster being greater than or equal to the first distance, determining the second target abnormal value based on a second manner; The calculation standards of the first method and the second method are different.

8. The enterprise data management platform according to claim 7, characterized in that: Also includes: Distance adjustment module; The distance adjustment module is configured to reduce the first distance based on a first distance adjustment step in response to the density of the target cluster being greater than a first density; In response to the density of the target cluster being less than a second density, the first distance is increased based on a second distance adjustment step size.

9. The enterprise data management platform according to claim 1, characterized in that: Also includes: A first data processing module; The first data processing module is configured to send abnormal information to the first device in response to the first abnormal data not being real data.

10. The enterprise data management platform according to claim 1, characterized in that: Also includes: A second data processing module; The second data processing module is used for controlling the second device to store the first abnormal data and related data of the first abnormal data in response to the first abnormal data not being real data.

Images