Intelligent data entry method based on Bayesian algorithm

Abstract

The present invention discloses an intelligent data entry method based on a Bayesian algorithm, comprising the following specific steps: extracting multi-link data, pre-processing the multi-link data, and mining and cleaning multi-link similar data based on a Bayesian algorithm. On the basis of extracting and pre-processing the multi-link data, the present invention uses the posterior probability of the Bayesian algorithm to calculate the node probability of the data in the multi-link, uses the dimension value of the similar data as a mining parameter, and mines the similar data in the link. Then, the optimal Bayesian network is set by setting the structural parameters, and the parent node of each node in all similar data is input therein to determine the maximum likelihood value of the data. The cleaning process is completed by eliminating the data with the maximum likelihood value, and the mining accuracy and cleaning accuracy are relatively high, so the effectiveness is relatively strong.

Description

Technical Field

[0001] The invention relates to the field of networks, and in particular to an intelligent data entry method based on a Bayesian algorithm. Background Art

[0002] The multi-link in the network is the main channel for data transmission. By setting up multiple links, it can ensure that the data to be transmitted is transmitted in the most appropriate and convenient path. Due to the large amount of data in the multi-link, it is easy to cause the link channel to be blocked, reducing the efficiency of data transmission. This is because there is a lot of similar data in the multi-link data, which seriously affects the efficiency of the multi-link transmission channel. Therefore, in order to reduce the amount of similar data in the multi-link and improve the network transmission efficiency, there are currently three multi-link similar data mining detection and cleaning methods on the market:

[0003] The first method is to divide the data attributes based on the optimization of the SNM algorithm, and use the three-interval sorting algorithm to sort the data to reduce the comparison range. Finally, the data similarity is judged by summing the attribute weights and similar data is eliminated. This method can improve the matching degree of data similarity, but due to the large amount of data in the matching process, there are problems such as long time consumption;

[0004] The second method is based on the K-Modes clustering method. It uses the information entropy theory to determine the weight of data attributes and simplify the attribute dimensions. Then, it compares the data of each group according to the importance of the data attributes, determines their similarity based on the threshold, and eliminates similar data after completing the detection of each data set.

[0005] The third method is to design a preprocessing module, a cloud clustering module, a similar data identification module and an entity division module, and complete the cleaning of similar data through data space clustering, similarity calculation, etc. The above methods have a high control over the packet loss of link similar data, but there is a problem of poor accuracy in similar data extraction. Summary of the invention

[0006] The purpose of the present invention is to provide an intelligent data entry method based on Bayesian algorithm to solve the problems raised in the above background technology.

[0007] To achieve the above object, the present invention provides the following technical solution: an intelligent data entry method based on a Bayesian algorithm, comprising the following specific steps:

[0008] Step 1: Determine the existence status of data in the multi-link through the adjacency matrix, calculate the node degree of the multi-link data, and extract the multi-link data information according to the data out-degree and in-degree;

[0009] Step 2: Determine the limited range of pre-processed data by calculating the multi-link radius. On the basis of secondary noise reduction, calculate the dimension of multi-link data based on this range, and then complete the pre-processing of data through dimensionality reduction operation;

[0010] Step 3: Use the posterior probability of the Bayesian algorithm to calculate the node probability of the data in the multi-link, use the dimension value of the similar data as the mining parameter, and mine the similar data in the link;

[0011] Step 4: Set the optimal Bayesian network by setting the structural parameters, input the parent node of each node in all similar data into it, determine the maximum likelihood value of the data, and complete the cleaning process by eliminating the data with the maximum likelihood value.

[0012] Preferably, in the step 1, the multi-link data extraction should first determine whether there is data on the multi-link through the adjacency matrix, and set the data set on the multi-link to be represented as G. Then the adjacency matrix at this time can be represented as:

[0013] L=(I ij ) n×n

[0014] In the formula, I ij represents the link data on the jth row on the i-th row;

[0015] Since the distance between point data on the link during the collection process is a key factor affecting data extraction, the maximum distance between two data in the multi-link is determined, and data is collected on this basis. At this time, the maximum straight-line distance between two data on the multi-link is expressed as:

[0016]

[0017] In the formula, Z ij Represents the diameter of the multi-link, Z max Represents the maximum straight-line distance between two data;

[0018] The degree of data on a link is determined based on the state of data on multiple links and the maximum straight-line distance. The degree of data represents its importance on the link, which is:

[0019]

[0020] On this basis, the average value of the data degree on multiple links is determined as follows:

[0021]

[0022] In the formula, n represents the total amount of data on multiple links, represents the average degree value;

[0023] Divide the data degree on the link into out-degree And in-degree, different data degrees at this time represent all the data of the data degree ambition on the link, thus obtaining the data on the multi-link in these two states, namely:

[0024]

[0025]

[0026] In the formula, Represents the out-degree data status, Represents the in-degree data state, and n represents the number of data.

[0027] Preferably, if the node is at the edge of the link, the adjacency matrix at this time is 1, and if the node is not at the edge of the link, the adjacency matrix of the test is 0.

[0028] Preferably, the data degree on the link is divided into out-degree And in-degree, at this time, different data degrees represent all the data of the data degree ambition on the link, thus obtaining the data on the multi-link in these two states, namely:

[0029]

[0030]

[0031] In the formula, Represents the out-degree data status, Represents the in-degree data state, and n represents the number of data.

[0032] Preferably, the multi-link data preprocessing in step 2 includes setting two data within the multi-link data sensing range to be adjacent to each other. At this time, the data on the link is represented as:

[0033]

[0034] In the formula, e represents the Euclidean distance between two data, 1 represents that the two data are adjacent, and 0 represents that the two data are not adjacent;

[0035] In the denoising process, the initial value of the node data is set to i, and the adjacent data is set to j. Then the data set is expressed as:

[0036] D={d 1 ,d 2 ,d 3 ...d m}

[0037] Where, d m Represents the multi-link data composition factor. After noise reduction, the data in the formula is obtained:

[0038]

[0039] In the formula, v represents the data noise reduction threshold, and c represents the average value of the data in the set.

[0040] Preferably, in the multi-link data denoising process, due to the different noise levels, secondary denoising is required to ensure the validity of the data. The process is as follows:

[0041]

[0042] In the formula, s represents the location of the data center. After denoising the above multi-link data, we need to further reduce the data with higher dimensions. First, we establish a multi-link data set as follows:

[0043] B = {b 1 ,b 2 ,...b n}

[0044] Solve the partial derivative of this type of data to get the dimension of the data. If the dimension is 0 or less than 0, reduce the dimension of this part of the data. If it is greater than 0, it is determined that the data does not need to be reduced in dimension, that is:

[0045]

[0046] In the formula, E represents the degree of membership of the data, y ij Represents the dimension of the data.

[0047] Preferably, in step 3, the independent relationship of the object of study is decomposed using a Bayesian network, and its basic form is a product form, which can be expressed as:

[0048]

[0049] In the formula, pa(p) represents the parent node of node i in the Bayesian network, and the research object is analyzed through the independent conditions existing in the algorithm;

[0050] First, determine the similarity between them, namely:

[0051]

[0052] In the formula, same represents the initial similarity of the data, y t Represents the historical similarity of data, x t represents the error range of similarity calculation, G represents the total number of multi-link similar data;

[0053] According to the determined multi-link similarity data, the posterior probability calculation of the Bayesian algorithm is used to determine the node probability of the multi-link similarity data, which can be expressed as:

[0054] P(Q=q|U=u)

[0055] In the formula, Q represents the set of similar nodes, and U represents the set of query data;

[0056] According to the determined probability of similar data nodes of multi-link data, a mining model of multi-link similar data is constructed, and the mining of similar data is completed according to the constructed model, namely:

[0057] wi=aP+bp+s(a,b)ΣP

[0058] In the formula, w i represents the target similar link data, a represents the measurement value of the mined similarity data, b represents the similarity between links, and s represents the weight value of different data. The Bayesian network is used to train the above similar data to complete the cleaning of the mined data;

[0059] On this basis, the optimal structure of the Bayesian network is determined and set as B s ,get:

[0060] P(B s )=D∑P

[0061] According to the determined optimal network structure, the similarity data obtained above are input into the network in the form of nodes, where each similarity data represents a feature of a dimension, and its set can be expressed as:

[0062] F={f 1 ,f 2 ,f 3 ,...f m}

[0063] In the formula, represents M records, and then the parent node of each node in all similar data is represented as e(V i ), by initializing the node value to a null value, the process is as follows:

[0064]

[0065] According to the determined empty set, the scoring function is set by the Bayesian algorithm, namely:

[0066]

[0067] In the formula, Represents the scoring result value;

[0068] After completing the evaluation of similar data, the maximum likelihood value of the Bayesian algorithm is obtained and this value is used as the optimal result of cleaning to obtain:

[0069]

[0070] In the formula, K(D|θ) represents the set of Bayesian network structure variables, L represents the combination of similar data parent nodes, P represents the probability of the parent node, and e(V i ) represents the maximum likelihood value of the cleaned data.

[0071] Technical effects and advantages of the present invention:

[0072] (1) Based on the extraction and preprocessing of multi-link data, the present invention uses the posterior probability of the Bayesian algorithm to calculate the node probability of the data in the multi-link, takes the dimension value of the similar data as the mining parameter, and mines the similar data in the link. Then, the optimal Bayesian network is set by setting the structural parameters, and the parent node of each node in all similar data is input into it, and the maximum likelihood value of the data is determined. The cleaning process is completed by eliminating the data with the maximum likelihood value, which has high mining accuracy and cleaning accuracy, so it is more effective. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 It is a schematic diagram of a multi-link directed graph of the present invention.

[0074] Figure 2 This is a schematic diagram of the multi-link data perception range of the present invention.

[0075] Figure 3 This is a comparison chart of the accuracy of multi-link similarity data mining using different methods of the present invention.

[0076] Figure 4 Comparison diagram of cleaning errors of multi-link similar data by different methods of the present invention DETAILED DESCRIPTION

[0077] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0078] The present invention provides Figure 1-4 The intelligent data entry method based on the Bayesian algorithm shown includes the following specific steps:

[0079] Step 1: Determine the existence status of data in the multi-link through the adjacency matrix, calculate the node degree of the multi-link data, and extract the multi-link data information according to the data out-degree and in-degree;

[0080] Multi-link data extraction should first determine whether there is data on the multi-link through the adjacency matrix. Set the data set on the multi-link to be represented as G, then the adjacency matrix can be expressed as:

[0081] L=(I ij ) n×n

[0082] In the formula, I ij Represents the link data on the jth link on the i-th row. If it is on the edge of the link, the adjacency matrix at this time is 1. If it is not on the edge of the link, the adjacency matrix of the test is 0.

[0083] Since the distance between point data on the link during the collection process is a key factor affecting data extraction, the maximum distance between two data in the multi-link is determined, and data is collected on this basis. At this time, the maximum straight-line distance between two data on the multi-link is expressed as:

[0084]

[0085] In the formula, Z ij Represents the diameter of the multi-link, Z max Represents the maximum straight-line distance between two data;

[0086] The degree of data on a link is determined based on the state of data on multiple links and the maximum straight-line distance. The degree of data represents its importance on the link, which is:

[0087]

[0088] On this basis, the average value of the data degree on multiple links is determined as follows:

[0089]

[0090] In the formula, n represents the total amount of data on multiple links, represents the average degree value;

[0091] Divide the data degree on the link into out-degree And in-degree, at this time, different data degrees represent all the data of the data degree ambition on the link, thus obtaining the data on the multi-link in these two states, namely:

[0092]

[0093]

[0094] In the formula, Represents the out-degree data status, Represents the in-degree data state, and n represents the number of data.

[0095] Divide the data degree on the link into out-degree And in-degree, at this time, different data degrees represent all the data of the data degree ambition on the link, thus obtaining the data on the multi-link in these two states, namely:

[0096]

[0097]

[0098] In the formula, Represents the out-degree data status, Represents the in-degree data state, and n represents the number of data;

[0099] Step 2: Determine the limited range of pre-processed data by calculating the multi-link radius. On the basis of secondary noise reduction, calculate the dimension of multi-link data based on this range, and then complete the pre-processing of data through dimensionality reduction operation, laying a good data foundation for the subsequent data cleaning process;

[0100] Multi-link data preprocessing includes setting two data within the multi-link data sensing range to be adjacent. At this time, the data on the link is represented as:

[0101]

[0102] In the formula, e represents the Euclidean distance between two data, 1 represents that the two data are adjacent, and 0 represents that the two data are not adjacent;

[0103] In the denoising process, the initial value of the node data is set to i, and the adjacent data is set to j. Then the data set is expressed as:

[0104] D={d 1 ,d 2 ,d 3 ...d m}

[0105] Where, d m Represents the multi-link data composition factor. After noise reduction, the data in the formula is obtained:

[0106]

[0107] In the formula, v represents the data noise reduction threshold, and c represents the average value of the data in the set;

[0108] In the multi-link data denoising process, due to the different noise levels, secondary denoising is required to ensure the validity of the data. The process is as follows:

[0109]

[0110] In the formula, s represents the location of the data center. After denoising the above multi-link data, we need to further reduce the data with higher dimensions. First, we establish a multi-link data set as follows:

[0111] B = {b 1 ,b 2 ,...b n}

[0112] Solve the partial derivative of this type of data to get the dimension of the data. If the dimension is 0 or less than 0, reduce the dimension of this part of the data. If it is greater than 0, it is determined that the data does not need to be reduced in dimension, that is:

[0113]

[0114] In the formula, E represents the degree of membership of the data, y ij Represents the dimension of the data;

[0115] Step 3: Use the posterior probability of the Bayesian algorithm to calculate the node probability of the data in the multi-link, use the dimension value of the similar data as the mining parameter, and mine the similar data in the link;

[0116] The Bayesian network is used to decompose the independent relationship of the research object. Its basic form is a product form, which can be expressed as:

[0117]

[0118] In the formula, pa(p) represents the parent node of node i in the Bayesian network, and the research object is analyzed through the independent conditions existing in the algorithm;

[0119] First, determine the similarity between them, namely:

[0120]

[0121] In the formula, same represents the initial similarity of the data, y t Represents the historical similarity of data, x t represents the error range of similarity calculation, G represents the total number of multi-link similar data;

[0122] According to the determined multi-link similarity data, the posterior probability calculation of the Bayesian algorithm is used to determine the node probability of the multi-link similarity data, which can be expressed as:

[0123] P(Q=q|U=u)

[0124] In the formula, Q represents the set of similar nodes, and U represents the set of query data;

[0125] According to the determined probability of similar data nodes of multi-link data, a mining model of multi-link similar data is constructed, and the mining of similar data is completed according to the constructed model, namely:

[0126] wi=aP+bp+s(a,b)∑P

[0127] In the formula, w i represents the target similar link data, a represents the measurement value of the mined similarity data, b represents the similarity between links, and s represents the weight value of different data. The Bayesian network is used to train the above similar data to complete the cleaning of the mined data;

[0128] On this basis, the optimal structure of the Bayesian network is determined and set as B s ,get:

[0129] P(B s )=D∑P

[0130] According to the determined optimal network structure, the similarity data obtained above are input into the network in the form of nodes, where each similarity data represents a feature of a dimension, and its set can be expressed as:

[0131] F={f 1 ,f 2 ,f 3 ,...f m}

[0132] In the formula, represents M records, and then the parent node of each node in all similar data is represented as e(V i ), by initializing the node value to a null value, the process is as follows:

[0133]

[0134] According to the determined empty set, the scoring function is set by the Bayesian algorithm, namely:

[0135]

[0136] In the formula, Represents the scoring result value;

[0137] After completing the evaluation of similar data, the maximum likelihood value of the Bayesian algorithm is obtained and this value is used as the optimal result of cleaning to obtain:

[0138]

[0139] In the formula, K(D|θ) represents the set of Bayesian network structure variables, L represents the combination of similar data parent nodes, P represents the probability of the parent node, and e(V i) represents the maximum likelihood value of the cleaned data;

[0140] Step 4: Set the optimal Bayesian network by setting the structural parameters, input the parent node of each node in all similar data into it, determine the maximum likelihood value of the data, and complete the cleaning process by eliminating the data with the maximum likelihood value.

[0141] Embodiment 1:

[0142] The multi-link data selected comes from the Solvakia data set. In this combination, a total of 2000 data are selected, of which 800 are similar data and the rest are normal data. The relevant parameters of the experimental sample data are shown in Table 1:

[0143]

[0144] Table 1 Experimental sample data related parameters

[0145] According to the experimental scheme set above, by comparison, the mining accuracy and cleaning error of multi-link similar data are used as indicators to verify the application performance of the first method and the second method;

[0146] Taking the mining accuracy of multi-link similarity data as the key indicator, the performance of the first monkey method and the second method is compared. Figure 3 As shown;

[0147] according to Figure 3 It can be seen from the experimental data in that there is a certain difference in the mining accuracy of sample data between the first method and the second method. Among them, the mining accuracy of the present invention is the highest among the three methods, and its mining accuracy can reach up to 98.51%. Therefore, it can be considered that the method in this paper can more accurately mine similar data in multiple links, and can lay a good data foundation for the subsequent data cleaning link;

[0148] On the basis of ensuring the accuracy of multi-link similar data mining, the cleaning errors of the first and second methods for similar data are further verified. The cleaning error results of different methods are shown in Figure 4 As shown;

[0149] according to Figure 4It can be seen from the experimental data in that there are differences in the errors of cleaning similar data in multiple links using the first method and the second method. Among them, the lowest cleaning error of the first method is about 3.85%, the lowest cleaning error of the second method is about 2.01%, and the lowest cleaning error of the present invention is about 1.22%. In contrast, the cleaning error of the method in this paper is lower. This is because the method in this paper uses the Bayesian algorithm to build the optimal network and determine the maximum likelihood value of the data, thereby improving the effect of data cleaning.

[0150] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An intelligent data entry method based on Bayesian algorithm, characterized in that: The specific steps include: Step 1: Determine the existence status of data in the multi-link through the adjacency matrix, calculate the node degree of the multi-link data, and extract the multi-link data information according to the data out-degree and in-degree; Step 2: Determine the limited range of pre-processed data by calculating the multi-link radius. On the basis of secondary noise reduction, calculate the dimension of multi-link data based on this range, and then complete the pre-processing of data through dimensionality reduction operation; Step 3: Use the posterior probability of the Bayesian algorithm to calculate the node probability of the data in the multi-link, use the dimension value of the similar data as the mining parameter, and mine the similar data in the link; Step 4: Set the optimal Bayesian network by setting the structural parameters, input the parent node of each node in all similar data into it, determine the maximum likelihood value of the data, and complete the cleaning process by eliminating the data with the maximum likelihood value; In step three, the degree of similarity is first determined, namely: In the formula, same represents the initial similarity of the data, y t Represents the historical similarity of data, x t represents the error range of similarity calculation, G represents the total number of multi-link similar data; According to the determined multi-link similarity data, the posterior probability calculation of the Bayesian algorithm is used to determine the node probability of the multi-link similarity data, which can be expressed as: P(Q=qU=u) In the formula, Q represents the set of similar nodes, and U represents the set of query data; According to the determined probability of similar data nodes of multi-link data, a mining model of multi-link similar data is constructed, and the mining of similar data is completed according to the constructed model, namely: wi=aP+bp+s(a,b)∑P In the formula, w i represents the target similar link data, a represents the measurement value of the mined similarity data, b represents the similarity between links, and s represents the weight value of different data; the Bayesian network is used to train the above similar data to complete the cleaning of the mined data; On this basis, the optimal structure of the Bayesian network is determined and set as B s ,get: P(B s )=D∑P According to the determined optimal network structure, the similarity data obtained above are input into the network in the form of nodes, where each similarity data represents a feature of a dimension, and its set can be expressed as: <h2 style=";text-align:left;direction:ltr">F = {f1,f2,f3,...f<h2 style=";text-align:left;direction:ltr"> m <h2 style=";text-align:left;direction:ltr">} In the formula, represents M records, and then the parent node of each node in all similar data is represented as e(V i ), by initializing the node value to a null value, the process is as follows: According to the determined empty set, the scoring function is set by the Bayesian algorithm, namely: In the formula, Represents the scoring result value; After completing the evaluation of similar data, the maximum likelihood value of the Bayesian algorithm is obtained and this value is used as the optimal result of cleaning to obtain: In the formula, K(Dθ) represents the set of Bayesian network structure variables, L represents the combination of similar data parent nodes, P represents the probability of the parent node, and e(V i ) represents the maximum likelihood value of the cleaned data.

2. The intelligent data entry method based on Bayesian algorithm according to claim 1, characterized in that: In the step 1, the multi-link data extraction should first determine whether there is data on the multi-link through the adjacency matrix, and set the data set on the multi-link to be represented as G. Then the adjacency matrix at this time can be represented as: L=(I ij ) n×n In the formula, I ij represents the link data on the jth row on the i-th row; Since the distance between point data on the link during the collection process is a key factor affecting data extraction, the maximum distance between two data in the multi-link is determined, and data is collected on this basis. At this time, the maximum straight-line distance between two data on the multi-link is expressed as: In the formula, Z ij Represents the diameter of the multi-link, Z max Represents the maximum straight-line distance between two data; The degree of data on a link is determined based on the state of data on multiple links and the maximum straight-line distance. The degree of data represents its importance on the link, which is: On this basis, the average value of the data degree on multiple links is determined as follows: In the formula, n represents the total amount of data on multiple links, and r represents the average degree value; Divide the data degree on the link into out-degree And in-degree, at this time, different data degrees represent all the data of the data degree ambition on the link, thus obtaining the data on the multi-link in these two states, namely: In the formula, Represents the out-degree data status, Represents the in-degree data state, and n represents the number of data.

3. The intelligent data entry method based on Bayesian algorithm according to claim 2 is characterized in that: If it is at the edge of the link, the adjacency matrix at this time is 1, and if it is not at the edge of the link, the adjacency matrix of the test is 0.

4. The intelligent data entry method based on Bayesian algorithm according to claim 2 is characterized in that: Divide the data degree on the link into out-degree And in-degree, at this time, different data degrees represent all the data of the data degree ambition on the link, thus obtaining the data on the multi-link in these two states, namely: In the formula, Represents the out-degree data status, Represents the in-degree data state, and n represents the number of data.

5. The intelligent data entry method based on Bayesian algorithm according to claim 1, characterized in that: The multi-link data preprocessing in step 2 includes setting two data within the multi-link data sensing range to be adjacent. At this time, the data on the link is represented as: In the formula, e represents the Euclidean distance between two data, 1 represents that the two data are adjacent, and 0 represents that the two data are not adjacent; In the denoising process, the initial value of the node data is set to i, and the adjacent data is set to j. Then the data set is expressed as: <h2 style=";text-align:left;direction:ltr">D = {d1,d2,d3...d<h2 style=";text-align:left;direction:ltr"> m <h2 style=";text-align:left;direction:ltr">} Where, d m Represents the multi-link data composition factor. After noise reduction, the data in the formula is obtained: In the formula, v represents the data noise reduction threshold, and c represents the average value of the data in the set.

6. The intelligent data entry method based on Bayesian algorithm according to claim 5, characterized in that: In the multi-link data denoising process, due to the different noise levels, secondary denoising is required to ensure the validity of the data. The process is as follows: In the formula, s represents the location of the data center. After denoising the above multi-link data, we need to further reduce the data with higher dimensions. First, we establish a multi-link data set as follows: B={b1,b2,...b n } Solve the partial derivative of this type of data to get the dimension of the data; if the dimension is 0 or less than 0, reduce the dimension of this part of the data; if it is greater than 0, it is determined that the data does not need to be reduced in dimension, that is: In the formula, E represents the degree of membership of the data, y ij Represents the dimension of the data.

7. The intelligent data entry method based on Bayesian algorithm according to claim 1, characterized in that: In step 3, the independent relationship of the object of study is decomposed using the Bayesian network, and its basic form is a product form, which can be expressed as: In the formula, pa(p) represents the parent node of node i in the Bayesian network, and the research object is analyzed through the independent conditions existing in the algorithm.

Images