A data cleaning method based on a Bayesian formula, a terminal and a storage medium

Abstract

The application discloses a data cleaning method based on a Bayesian formula, a terminal and a storage medium, and the method comprises the following steps: acquiring original data and pre-defined prior knowledge; constructing a Bayesian network and an association relationship according to the prior knowledge, performing causal reasoning on the original data according to the Bayesian network, and obtaining a probability score of the Bayesian network; determining an association relationship score of the original data according to the association relationship, and cleaning the original data according to the sum of the probability score and the association relationship score, to obtain a cleaned data version. The application combines the user knowledge prior information which is easy to master, the modeling capability of the Bayesian network on dirty data and the association relationship of mutual information existing in the data, performs scanning and cleaning on the original data, reduces the difficulty of data cleaning, and improves the accuracy and recall rate of data cleaning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to a data cleaning method based on Bayes formula, a terminal and a storage medium. BACKGROUND

[0002] Data in the real world has errors such as missing, misspelling and incompleteness, as an effective method to solve data errors, data cleaning has been a research hotspot; data cleaning is a process of re-examining and checking data, aiming to delete duplicate information, correct existing errors, and provide data consistency. Data cleaning is the last procedure to find and correct identifiable errors in data files, including checking data consistency, processing invalid values and missing values, etc. Unlike questionnaire review, data cleaning after input is generally completed by computer rather than manually.

[0003] The existing data cleaning methods mainly have the following forms:

[0004] 1. BayesWipe: This technology directly models the original data by using the Bayes network, and directly runs on the data to clean the data, which can realize easy-to-use causal reasoning cleaning, but the concept of Bayes cleaning is not perfect: Pr(g|t) = Pr(t)Pr(t|g) / Pr(g) is obtained by conversion of the Bayes formula, where g is a clean value, t is an observed value, and only the prior judgment of whether the data is discrete or continuous is required to be provided by the user, and the internal data type is preprocessed according to the provided corresponding pre-processing, lacking the description of expert knowledge prior, resulting in low cleaning accuracy.

[0005] 2. PClean: This technology probabilistically programs the original data, encodes the distribution of each column of data and the connection relationship between columns of data, and can achieve very good precision, recall and running time effect. But this technology requires the user to have a very good understanding of the data and the knowledge of probability programming as support, and the use cost is very high, and it is not easy to use.

[0006] 3. Holoclean: This technology applies probabilistic reasoning to feature engineering, converts expert knowledge into factor graph reasoning for vectorization of data and labeling (clean / dirty), and then trains a classifier for the entire data cleaning. This technology does not use causal inference but is based on correlation reasoning, which simplifies the modeling difficulty and fully utilizes the mutual information of nodes, and can achieve relatively good precision, recall and running time. But it also requires a high cost of expert, the user needs to develop relevant data rules, needs the knowledge support of reasoning system deepdive, and cannot explain the results.

[0007] Therefore, the prior art remains to be improved. SUMMARY

[0008] The technical problem solved by the present application is that, in view of the defects of the prior art, the present application provides a data cleaning method based on Bayes formula, a terminal and a storage medium, to solve the technical problems of high difficulty and low cleaning accuracy in data cleaning methods.

[0009] The technical solution adopted by the present application to solve the technical problem is as follows:

[0010] In a first aspect, the present application provides a data cleaning method based on Bayes formula, comprising:

[0011] obtaining original data and pre-defined prior knowledge;

[0012] constructing a Bayes network and a correlation relationship according to the prior knowledge, performing causal reasoning on the original data according to the Bayes network, and obtaining a probability score of the Bayes network;

[0013] determining a correlation relationship score of the original data according to the correlation relationship, and cleaning the original data according to the sum of the probability score and the correlation relationship score, to obtain a cleaned data version.

[0014] In an implementation mode, the obtaining of the original data and the pre-defined prior knowledge comprises:

[0015] defining the prior knowledge in advance according to input constraints.

[0016] In an implementation mode, the constructing of the Bayes network and the correlation relationship according to the prior knowledge, and the performing of the causal reasoning on the original data according to the Bayes network to obtain the probability score of the Bayes network comprises:

[0017] constructing a global directed graph generation algorithm, and performing structure learning according to the directed graph generation algorithm and the prior knowledge to obtain a learned Bayes network;

[0018] scanning the original data according to the learned Bayes network, and reasoning to obtain the probability of each data frame taking each data.

[0019] In an implementation mode, the constructing of the global directed graph generation algorithm, and the performing of the structure learning according to the directed graph generation algorithm and the prior knowledge to obtain the learned Bayes network comprises:

[0020] obtaining an inverse covariance matrix through decomposition of nodes, and generating an upper triangular matrix through the inverse covariance matrix;

[0021] The first directed graph edges are obtained according to the upper triangular matrix, and the second directed graph edges are added or deleted according to the prior knowledge and operation instructions, to obtain a directed acyclic graph;

[0022] Parameter learning is performed according to the directed acyclic graph, to obtain a conditional probability table of the Bayesian network;

[0023] The learned Bayesian network is obtained according to the conditional probability table.

[0024] In an implementation manner, the original data is scanned according to the learned Bayesian network, and a probability of each data frame taking each data is inferred, including:

[0025] The original data is scanned in a record unit;

[0026] In each record, each attribute is set as missing, and a missing value is derived through other attributes associated with the attribute;

[0027] The missing value is taken as the probability of each data frame taking each data.

[0028] In an implementation manner, the association relationship of the original data is determined according to the association relationship, and the original data is cleaned according to a sum of the probability score and the association relationship score, to obtain a cleaned data version, including:

[0029] The distance between different values in the same column in the Bayesian network is determined;

[0030] The relationship score between each value in the value range of each two columns in the Bayesian network is determined;

[0031] The sum of the distance and the relationship score of each value in the Bayesian network is calculated, to obtain a final probability score;

[0032] The original data is cleaned according to the final probability score and an observation value, to obtain a cleaned data version.

[0033] In an implementation manner, the relationship score between each value in the value range of each two columns in the Bayesian network is determined, including:

[0034] In the value range of each two columns, the score of each value commonly appearing in data is calculated;

[0035] A co-occurrence dictionary is obtained according to the common occurrence score, and the prior knowledge is intervened in the co-occurrence dictionary;

[0036] The relationship between each value in the value range of each two columns is scored according to the prior knowledge, to obtain the relationship score.

[0037] In an implementation manner, the data cleaning method based on the Bayesian formula further comprises:

[0038] When reasoning the current value, all parent nodes and child nodes directly connected with the current value attribute node are excluded.

[0039] The correlation relationship score of the current value is calculated according to the correlation relationship of the remaining nodes, and the current value is normalized to 0 to 1.

[0040] In a second aspect, the present application further provides a terminal comprising a processor and a memory, wherein the memory stores a data cleaning program based on the Bayesian formula, and the data cleaning program based on the Bayesian formula is used to implement the operations of the data cleaning method based on the Bayesian formula in the first aspect when executed by the processor.

[0041] In a third aspect, the present application further provides a storage medium, which is a computer readable storage medium, and the storage medium stores a data cleaning program based on the Bayesian formula, and the data cleaning program based on the Bayesian formula is used to implement the operations of the data cleaning method based on the Bayesian formula in the first aspect when executed by a processor.

[0042] The technical scheme of the present application has the following effects:

[0043] The present application obtains original data and prior knowledge defined in advance, constructs a Bayesian network and a correlation relationship according to the prior knowledge, performs causal reasoning on the original data according to the Bayesian network to obtain a probability score of the Bayesian network, determines a correlation relationship score of the original data according to the correlation relationship, and cleanses the original data according to the sum of the probability score and the correlation relationship score to obtain a cleaned data version. The present application combines user knowledge prior information easy to master, the modeling capability of the Bayesian network on dirty data, and the correlation relationship of mutual information existing in the data, scans and cleanses the original data, reduces the difficulty of data cleansing, and improves the accuracy and recall rate of data cleansing. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical scheme in the embodiments of the present application or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor based on the drawings shown.

[0045] Figure 1 is a flowchart of a data cleaning method based on the Bayesian formula in an implementation manner of the present application.

[0046] Figure 2 is a schematic diagram of a data cleaning framework based on a Bayesian formula in an implementation manner of the present application.

[0047] Figure 3 is a functional schematic diagram of a terminal in an implementation manner of the present application.

[0048] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0049] In order to make the object, technical solutions and advantages of the present application more clear and explicit, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0050] Exemplary method

[0051] In the conventional data cleaning manner, one is to obtain the data to be cleaned through the conversion of the Bayesian formula, and requires the user to provide the prior judgment of whether the data is discrete or continuous, and the internal data type is preprocessed according to the provided corresponding pre-processing, which lacks the prior description of expert knowledge, resulting in low cleaning accuracy. Another is to program the original data by probability, encode the distribution of each column of data and the connection relationship between the columns of data, which requires the user to have a very good understanding of the data and the knowledge of probability programming as support, and the use cost is very high and not easy to use.

[0052] In view of the above technical problems, in the present embodiment, the original data is scanned and cleaned by combining the user knowledge prior information easy to use, the modeling ability of the Bayesian network on dirty data and the correlation relationship of the mutual information existing in the data, thereby reducing the difficulty of data cleaning and improving the accuracy and recall rate of data cleaning.

[0053] As shown in Figure 1 The present embodiment provides a data cleaning method based on a Bayesian formula, which comprises the following steps:

[0054] Step S100, obtaining original data and pre-defined prior knowledge.

[0055] In the present embodiment, the data cleaning method based on the Bayesian formula is applied to a terminal, which includes but is not limited to a computer, a mobile terminal and other devices.

[0056] In the present embodiment, by combining the user knowledge prior information easy to use, the modeling ability of the Bayesian network on dirty data and the correlation relationship of the mutual information existing in the data, a new Bayesian formula is designed to scan and clean the original data.

[0057] In the embodiment, a bclean system for data cleaning is arranged in the terminal, and data is cleaned based on the bclean system; before the data cleaning method in the embodiment is implemented, corresponding constraint conditions are input into the bclean system, and the prior knowledge is defined according to the input constraint conditions; the constraint conditions are user-defined sql (a way of defining the prior knowledge by the user, similar to defining a data table); by inputting the user-defined sql into the bclean system, the bclean system can generate the prior knowledge required for data cleaning.

[0058] Specifically, in an implementation manner of the embodiment, the step S100 includes the following steps:

[0059] In step S001, the prior knowledge is defined according to the input constraint conditions.

[0060] In an implementation manner of the embodiment, the prior knowledge is used to describe statistical indexes of the original data, including a maximum length (value) of the original data, a minimum length (value) of the original data, a data type, a pattern form that meets the original data, and the like.

[0061] In an implementation manner of the embodiment, when the corresponding constraint conditions are input into the bclean system, the original data that needs to be cleaned also needs to be input into the bclean system; the original data can be data in a certain database, or data in multiple databases; in the original data, there are strong correlation and weak correlation between the data, the strong correlation is a cause-effect relationship, and in the embodiment, the strong correlation is a node relationship of a Bayesian network, similar to that when A=1 and B=2, according to the relationship of (A, B)→C, C=3 can be obtained; the weak correlation is an association relationship, and in the embodiment, the weak correlation is similar to a statistical co-occurrence frequency, that is, (A=1, B=2) appears 3 times at the same time, and (A=1, B=3) appears 1 time at the same time, so the association relationship of (A, B) is that the initial distribution of (A=1, B=2) is 0.75, and the initial distribution of (A=1, B=3) is 0.25.

[0062] In the embodiment, by inputting the original data and the user-defined simple sql constraint into the bclean system, the Bayesian network can be generated in the bclean system, and the association relationship between the data is constructed, so that the probability score of the Bayesian network and the probability score of the association relationship are calculated for the required data, and the data is cleaned by using the calculated probability score.

[0063] As Figure 1As shown, in an implementation form of the embodiment of the present application, the data cleaning method based on the Bayesian formula further comprises the following steps:

[0064] In step S200, a Bayesian network and a correlation relationship are constructed according to the prior knowledge, and a probability score of the Bayesian network is obtained by performing causal reasoning on the original data according to the Bayesian network.

[0065] In this embodiment, the algorithm model for data cleaning is a new Bayesian formula converted based on the Bayesian formula, that is, the new Bayesian formula is converted from Pr(G|Obs)=Pr(Obs|G)Pr(G) / Pr(T) to lnPr(G)=(lnPr(G|Obs))+(lnPr(Obs)-lnPr(Obs|G)), and the new Bayesian formula is divided into two parts to solve, and the probability score of the Bayesian network and the probability score of the correlation relationship are calculated respectively; it can be understood that the data cleaning is performed by solving the clean data G that can maximize lnPr(G), and the original data Obs needs to be preprocessed by the user-defined prior knowledge, which can perform rough error detection on the data.

[0066] Specifically, in an implementation form of the embodiment, step S200 comprises the following steps:

[0067] In step S201, a global directed graph generation algorithm is constructed, and structure learning is performed according to the directed graph generation algorithm and the prior knowledge to obtain a learned Bayesian network.

[0068] In step S202, the original data is scanned according to the learned Bayesian network, and the probability of each data frame taking each data is inferred.

[0069] In this embodiment, for the first part (lnPr(G|Obs)) of the new Bayesian formula, it is defined as the result of causal reasoning of G by the observed value Obs in this embodiment, that is, it is defined as the Bayesian network; the process of the first part is to infer the clean value T[A] of the current attribute A under the data T, and there is a causal relationship (B, C)→A, so that the value of A is regarded as a missing value, and then the final value of A is determined according to the values of B and C, such as when (B=1, C=2) A=3, and when (B=3, C=4) A=7. That is, B and C are observation values, and the cleaning value is inferred based on the observation values.

[0070] For the construction method of the first part, there are traditional structure learning algorithms, but most of them are based on enumeration algorithm and greedy algorithm, which makes it consume a lot of time when learning the network structure of multiple nodes and multiple degrees, and a local optimal structure is learned, that is, it is not the best from the global data.

[0071] In this embodiment, a global directed graph generation algorithm based on FDX (FDX is a structure learning algorithm) is used for structure learning.

[0072] Specifically, in one implementation of this embodiment, step S201 includes the following steps:

[0073] Step S201a, obtaining an inverse covariance matrix by decomposing a node, and generating an upper triangular matrix by the inverse covariance matrix;

[0074] Step S201b, obtaining first directed graph edges according to the upper triangular matrix, and adding or deleting second directed graph edges according to the prior knowledge and operation instructions to obtain a directed acyclic graph;

[0075] Step S201c, performing parameter learning according to the directed acyclic graph to obtain a conditional probability table of the Bayesian network;

[0076] Step S201d, obtaining the learned Bayesian network according to the conditional probability table.

[0077] Specifically, in this embodiment, the inverse covariance matrix obtained by decomposing a node is used to generate an upper triangular matrix; that is, by the formula O = (I-B) V (I-B)T, where O is the inverse covariance matrix, I is the unit matrix, V is the diagonal matrix, and B is the upper triangular matrix to be obtained.

[0078] Further, the edges of the directed graph are obtained according to the upper triangular matrix, and the appropriate edges are intervened in the obtained directed graph edges according to the operation instructions input by the user in the system; that is, the directed graph obtained by FDX can be modified by the user to become another directed acyclic graph. Then, the obtained directed graph is subjected to parameter learning to obtain a conditional probability table of the entire network; wherein the conditional probability table is used to give the correct probability of each candidate clean value in the cleaning stage. It can be understood that the probability score of the data to be cleaned in the Bayesian network can be calculated according to the conditional probability table.

[0079] In this embodiment, the learned Bayesian network is obtained according to the conditional probability table, that is, the conditional probability table is the learned parameter of the first part (lnPr(G|Obs)); each data in the original data can be scanned by the conditional probability table to obtain the probability of each data frame taking each data under each attribute, that is, the probability score of the data to be cleaned in the Bayesian network.

[0080] Specifically, in one implementation of this embodiment, step S202 includes the following steps:

[0081] Step S202a, scanning the original data in a record unit;

[0082] Step S202b, setting each attribute as missing in each record, and deriving the missing value through other associated attributes;

[0083] Step S202c, taking each data in each data frame as the probability of the missing value.

[0084] Specifically, in the embodiment, each data in the original data can be scanned in a record unit; that is, one record is scanned each time, and in each record, each attribute is first treated as missing, and then the missing value is inferred through other associated attributes, and the scanning is completed, that is, the first cleaning is completed; it can be understood that the first cleaning is a coarse cleaning process.

[0085] In the embodiment, the first part (lnPr(G|Obs)) realizes the cleaning process of the causal relationship in the original data, and the part mainly affects the cleaning effect, that is, the cleaning process of the first part is mainly performed through the inference of the causal relationship, when the network structure of the first part is good, the first part can have high cleaning accuracy and recall rate, and vice versa.

[0086] As shown in the embodiment of the present application, the data cleaning method based on the Bayesian formula further includes the following steps: Figure 1 Step S300, determining the association relationship score of the original data according to the association relationship, and cleaning the original data according to the sum of the probability score and the association relationship score to obtain a cleaned data version.

[0087] In the embodiment, for the second part (lnPr(Obs)-lnPr(Obs|G)) of the new Bayesian formula, the embodiment regards the association relationship score as the score of the observation value minus the score of the candidate clean value; wherein the association relationship is the association relationship of other nodes (mutual information nodes) except the causal relationship, and the candidate value score is obtained by solving the converted Bayesian formula; the second part only considers the association relationship score of the current inference node and all other nodes (mutual information nodes) not directly connected to it in the Bayesian network.

[0088] Specifically, in one implementation manner of the embodiment, step S300 includes the following steps:

[0089] Step S301, determining the distance between different values in the same column in the Bayesian network;

[0090]

[0091] ​Step S302, determining the relationship score between each value in the value range of each two columns in the Bayesian network;

[0092] Step S303, calculating the sum of the distance and the relationship score of each value in the Bayesian network to obtain a final probability score;

[0093] Step S304, cleaning the original data according to the final probability score and the observation value to obtain a cleaned data version.

[0094] In the embodiment, the second part (lnPr(Obs)-lnPr(Obs|G)) needs to determine the distance score between different values in the same column in the Bayesian network, and determine the relationship score between each value in the value range of each two columns in the Bayesian network, and by adding the distance score and the relationship score of each data, the probability score of the causal relationship and the association relationship of each data in the original data is obtained, that is, the final probability score is obtained; the original data is cleaned according to the final probability score and the observation value to obtain a cleaned data version.

[0095] Specifically, in an implementation manner of the embodiment, step S303 includes the following steps:

[0096] Step S302a, calculating the score of the common occurrence of each value in the data in the value range of each two columns;

[0097] Step S302b, obtaining a co-occurrence dictionary according to the common occurrence score, and intervening the prior knowledge in the co-occurrence dictionary;

[0098] Step S302c, scoring the relationship between each value in the value range of each two columns according to the prior knowledge to obtain the relationship score.

[0099] In the embodiment, in the calculation of different values between different two columns, the score of the common occurrence of the values in the two columns of data is calculated and recorded as a co-occurrence dictionary; wherein, the co-occurrence dictionary is measured by a score rather than an occurrence frequency, that is, (A=1, B=2) occurs 3 times, (A=3, B=2) occurs 1 time, but since there is a restriction condition A<2, (A=1, B=2) scores 3+1+1=5, (A=3, B=2) scores 1-1+1=1), wherein the user's prior information is intervened (as described above, the user's prior information is used for scoring, that is, a penalty is applied or a normal score is applied), when the data of a certain row slightly violates the user's requirement, a penalty (score deduction) is applied to the score, and the minimum is 0.

[0100] Further, the co-occurrence dictionary records mutual information between columns (nodes), i.e. the association relationship. For clean value inference of the current value, all parent nodes and child nodes directly connected to the current attribute node are excluded, and then the association relationship score of the current value is calculated according to the obtained association relationship, and the value is normalized to 0 to 1.

[0101] In the embodiment, the second part (lnPr(Obs)-lnPr(Obs|G)) is secondary to the influence of the cleaning effect, and mainly performs credibility inference. When the current value is a correct value, a positive score is added when the current value is inferred to be correct by the Bayesian network, and more scores are added when the current value is inferred to be incorrect by the Bayesian network. When the current value is an incorrect value, a penalty is added when the current value is inferred to be incorrect by the Bayesian network, and more penalties are added when the current value is inferred to be correct by the Bayesian network.

[0102] In an implementation manner of the embodiment of the application, the data cleaning method based on the Bayesian formula further includes the following steps:

[0103] In step S400, when inferring the current value, all parent nodes and child nodes directly connected to the current attribute node are excluded.

[0104] In step S500, the association relationship score of the current value is calculated according to the association relationship of the remaining nodes, and the current value is normalized to 0 to 1.

[0105] In the embodiment, the traversal inference increases unnecessary workload, and when the Bayesian network is too large (the structure and the conditional probability table are too large), the cleaning time is greatly increased. Therefore, the association relationship can be used to preliminarily filter the inference space in the bclean system. Moreover, since the association relationship is a mutual information dictionary, the access speed is fast, the association relationship score of each observation value is calculated by first using the traversal, and when the score is higher than the threshold value set by the user, the observation value is regarded as a correct value and is not inferred, so that the subsequent process can be avoided. In a specific application scenario, the threshold value set by the user can be set to 0.1 to 0.6, because most values fall in the interval of 0.5 to 0.6 after normalization.

[0106] As shown in FIG. 4, the association relationship score of each observation value is calculated according to the traversal of the mutual information dictionary. Figure 2 Figure 2 ​The structure and flow in the method are shown in the figure, in an actual application scenario of the embodiment, the cleaning process is: inputting the original data and the user-defined simple sql (a way of defining the prior knowledge of the user, similar to defining a data table) constraint into the bclean system, then the bclean generates the Bayesian network and constructs the association relationship, then calculates the probability score of the Bayesian network and the probability score of the association relationship for the required data respectively, finally adds the final probability score, and selects the value with the maximum score, if the same as the observed value, it is considered that the current value is correct and does not need to be cleaned; if different from the observed value, it is considered that the observed value is wrong, and the value with the maximum score is changed. After scanning the entire data, a cleaner version is returned to the user.

[0107] The embodiment achieves the following technical effects through the above technical scheme:

[0108] The embodiment combines the easy-to-use user knowledge prior information, the modeling capability of the Bayesian network on dirty data, and the association relationship of mutual information existing in the data to scan and clean the original data, thereby reducing the difficulty of data cleaning and improving the accuracy and recall rate of data cleaning.

[0109] Exemplary device

[0110] Based on the above embodiment, the application further provides a terminal, comprising: a processor, a memory, an interface, a display screen and a communication module connected through a system bus; wherein the processor is used to provide computing and control capability; the memory comprises a storage medium and an internal memory; the storage medium stores an operating system and a computer program; the internal memory provides an environment for the operating system and the computer program in the storage medium; the interface is used to connect external devices, such as mobile terminals and computer devices; the display screen is used to display corresponding information; and the communication module is used to communicate with a cloud server or a mobile terminal.

[0111] The computer program is executed by the processor to implement the operation of the data cleaning method based on the Bayesian formula.

[0112] Those skilled in the art can understand that, Figure 3 The principle block diagram shown in the figure is only a block diagram of part of the structure related to the application scheme, and does not constitute a limitation on the terminal to which the application scheme is applied, and the specific terminal can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0113] In one embodiment, a terminal is provided, comprising: a processor and a memory, the memory storing a data cleaning program based on Bayesian formula, the data cleaning program based on Bayesian formula being used for implementing operations of the data cleaning method based on Bayesian formula as described above when executed by the processor.

[0114] In one embodiment, a storage medium is provided, the storage medium storing a data cleaning program based on Bayesian formula, the data cleaning program based on Bayesian formula being used for implementing operations of the data cleaning method based on Bayesian formula as described above when executed by the processor.

[0115] It can be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware, and the computer program can be stored in a non-volatile storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments of the present application can include non-volatile and / or volatile memory.

[0116] In summary, the present application provides a data cleaning method based on Bayesian formula, a terminal and a storage medium. The method comprises: obtaining original data and pre-defined prior knowledge; constructing a Bayesian network and an association relationship according to the prior knowledge, performing causal reasoning on the original data according to the Bayesian network to obtain a probability score of the Bayesian network; determining an association relationship score of the original data according to the association relationship, and cleaning the original data according to the sum of the probability score and the association relationship score to obtain a cleaned data version. The present application combines easy-to-use user knowledge prior information, the modeling ability of Bayesian network on dirty data and the association relationship of mutual information existing in data to scan and clean the original data, thereby reducing the difficulty of data cleaning and improving the accuracy and recall rate of data cleaning.

[0117] It should be understood that the application of the present application is not limited to the above examples, and those skilled in the art can make improvements or changes according to the above description, and all these improvements and changes should belong to the protection scope of the appended claims of the present application.

Claims

1. A data cleaning method based on Bayes formula, characterized in that, The data cleaning method based on the Bayesian formula comprises: obtaining original data and pre-defined prior knowledge; constructing a Bayesian network and an association relationship according to the prior knowledge, performing causal reasoning on the original data according to the Bayesian network, and obtaining a probability score of the Bayesian network; determining an association relationship score of the original data according to the association relationship, and cleaning the original data according to a sum of the probability score and the association relationship score to obtain a cleaned data version; the constructing a Bayesian network and an association relationship according to the prior knowledge, performing causal reasoning on the original data according to the Bayesian network, and obtaining a probability score of the Bayesian network, comprises: constructing a global directed graph generation algorithm, and performing structure learning according to the directed graph generation algorithm and the prior knowledge to obtain a learned Bayesian network; scanning the original data according to the learned Bayesian network, and reasoning to obtain a probability of each data frame taking each data; the constructing a global directed graph generation algorithm, and performing structure learning according to the directed graph generation algorithm and the prior knowledge to obtain a learned Bayesian network, comprises: obtaining an inverse covariance matrix through decomposition of a node, and generating an upper triangular matrix through the inverse covariance matrix; obtaining a first directed graph edge according to the upper triangular matrix, and adding or deleting a second directed graph edge according to the prior knowledge and an operation instruction to obtain a directed acyclic graph; performing parameter learning according to the directed acyclic graph to obtain a conditional probability table of the Bayesian network; obtaining the learned Bayesian network according to the conditional probability table; the determining an association relationship score of the original data according to the association relationship, and cleaning the original data according to a sum of the probability score and the association relationship score to obtain a cleaned data version, comprises: determining a distance between different values in a same column in the Bayesian network; determining a relationship score between each value within a value range of each two columns in the Bayesian network; calculating a sum of the distance and the relationship score of each value in the Bayesian network to obtain a final probability score; cleaning the original data according to the final probability score and an observation value to obtain a cleaned data version. 2.The data cleaning method based on Bayesian formula according to claim 1, characterized in that, the obtaining original data and pre-defined prior knowledge, previously comprises: pre-defining the prior knowledge according to an input constraint condition. 3.The data cleaning method based on Bayesian formula according to claim 1, characterized in that, the scanning the original data according to the learned Bayesian network, and reasoning to obtain a probability of each data frame taking each data, comprises: scanning the original data in a record unit; setting each attribute as missing in each record, and deriving a missing value through other associated attributes; taking the missing value as the probability of each data frame taking each data. 4.The data cleaning method based on Bayesian formula according to claim 1, characterized in that, the determining a relationship score between each value within a value range of each two columns in the Bayesian network, comprises: calculating a score of each value commonly appearing in data within a value range of each two columns; obtaining a co-occurrence dictionary according to the commonly appearing score, and intervening the prior knowledge in the co-occurrence dictionary; According to the prior knowledge, a relationship between each value in the value range of each two columns is scored to obtain the relationship score. 5.The data cleaning method based on Bayesian formula according to claim 1, characterized in that, The data cleaning method based on the Bayesian formula further comprises: When reasoning the current value, all parent nodes and child nodes directly connected with the current value attribute node are excluded. The association relationship score of the current value is calculated according to the association relationship of the remaining nodes, and the current value is normalized to 0 to 1.

6. A terminal, characterized by comprising: Comprise: A processor and a memory, the memory stores a data cleaning program based on the Bayesian formula, and the data cleaning program based on the Bayesian formula is used to implement the operation of the data cleaning method based on the Bayesian formula in any one of claims 1-5 when executed by the processor.

7. A storage medium, characterized by The storage medium is a computer readable storage medium, and the storage medium stores a data cleaning program based on the Bayesian formula, and the data cleaning program based on the Bayesian formula is used to implement the operation of the data cleaning method based on the Bayesian formula in any one of claims 1-5 when executed by the processor.

Images