A data cleaning method based on improved SNM algorithm in trial auxiliary affairs

Abstract

The invention discloses a data cleaning method for trial auxiliary things based on an improved SNM algorithm, and the data cleaning method comprises the following steps: S1, determining the initial value of the window according to the influencing factors of the window; S2, estimating the probability density of the window by a non-parametric regression method, and determining the threshold of the window; S3, matching the data set in the window, and determining whether to adjust the window size according to the matching result; S4, determining the adjustment step length of the window according to the step length calculation formula; S5, performing word segmentation, approximate word extraction and conjunction on the window, assigning weight values ​​of M1, M2 and M3 in turn, and confirming whether the data is similar data. The accuracy and efficiency of the graphics of the invention are significantly improved, and the larger the data volume, the more obvious the effect. The improved algorithm can save more than 40% of time and improve the accuracy by 60%.

Description

Technical Field

[0001] The present invention relates to the technical field of data cleaning of trial auxiliary matters, and in particular to a data cleaning method for trial auxiliary matters based on an improved SNM algorithm. Background Art

[0002] With the rapid development of the information age, more and more information data are carried by various industries. At present, the momentum of information development in the court industry is getting stronger and stronger. Under the general trend of digital China, the court has the idea of ​​integrating the existing relatively messy information systems and information data, hoping to form a complete guidance system by sorting out the existing historical data, and try to guide the trial and trial-related auxiliary affairs. However, most of the current information systems are in an independent state, and there will be errors and inconsistent semantic expressions when entering data, resulting in more time spent on the larger the amount of data, resulting in reduced work efficiency. In addition, the basic idea of ​​the traditional SNM algorithm is to sort the data set by keywords first, and then set a fixed-size window to check the repeatability of the data in the window. After the check, the window moves down, the first data in the window moves out, and the first data under the window moves in. According to the analysis, it can be seen that the algorithm has the following shortcomings:

[0003] 1. A window that is too small affects the data accuracy, while a window that is too large affects the comparison efficiency, resulting in the inability to determine the window size;

[0004] 2. Keywords have a great impact on the results of repetitive data processing. If the actual similar data are sorted by keywords and are far apart and larger than the window size, the duplicate records cannot be removed.

[0005] Currently, no effective solution has been proposed for the problems in the related technologies. Summary of the invention

[0006] In view of the problems in the related technology, the present invention proposes a data cleaning method for trial auxiliary matters based on an improved SNM algorithm to overcome the above-mentioned technical problems existing in the existing related technology.

[0007] To this end, the specific technical solution adopted by the present invention is as follows:

[0008] A data cleaning method for trial auxiliary matters based on an improved SNM algorithm, including the following steps:

[0009] S1. Determine the initial value of the window according to the influencing factors of the window;

[0010] S2, using non-parametric regression method to estimate the probability density of the window and determine the threshold of the window;

[0011] S3, matching the data set in the window, and determining whether to adjust the window size according to the matching result;

[0012] S4. Determine the window adjustment step size according to the step size calculation formula;

[0013] S5. Segment the window, extract similar words, and perform conjunctions, assign weight values ​​M1, M2, and M3 in sequence, and confirm whether the data is similar data.

[0014] Furthermore, the step of determining the initial value of the window according to the influencing factors of the window further includes the following steps:

[0015] S11, determine the window influencing factors;

[0016] S12. Let the size of the window data set be W i , the number of window data similarity comparisons is W i -1;

[0017] S13. Adjust the window to a reasonable range according to the comparison result, and determine the initial value of the window.

[0018] Furthermore, the influencing factors include data set size and data quality.

[0019] Furthermore, the method of estimating the probability density of the window by using a non-parametric regression method and determining the threshold of the window further includes the following steps:

[0020] S21, estimating the probability density of the window using a nonparametric regression method;

[0021] S22, calculate the probability density and determine the rejection region;

[0022] S23. Calculate the threshold value according to the distribution of the rejection region.

[0023] Furthermore, matching the data set in the window and determining whether to adjust the window size according to the matching result further includes the following steps:

[0024] S31, let the data set in the window be {D m , D m+1 , ..., D m+n}, compare Dm, and calculate Sim(D m , D m+1 ) size;

[0025] S32, when compared to Dm + n, if Sim(D m , D m+n )>T min , then adjust the window according to the comparison result;

[0026] S33、If Sim(D m , D m+n )<T max , then adjust the window according to the comparison result;

[0027] Among them, D m , D m+1 , ..., D m+n All are data sets, Sim(D m , D m+1 ) is the size of the window value, T min is the minimum similarity threshold of the window value, T max is the maximum similarity threshold of the window value.

[0028] Furthermore, the step of adjusting the window according to the comparison result further includes the following steps:

[0029] If Sim(D m , D m+n )>T min , then expand the window;

[0030] If Sim(D m , D m+n )<T max , the window will be reduced.

[0031] Furthermore, the step of determining the window adjustment step size according to the step size calculation formula further includes the following steps:

[0032] S41, determine the size of the window, and calculate Sim(D m , D m+n ) The size of the window values ​​for the mth and m+nth windows;

[0033] S42: Determine whether to increase the minimum similarity threshold of the window value.

[0034] Furthermore, the calculation formula of the enlarged step length is as follows:

[0035]

[0036] Where Tmin is the minimum similarity threshold of the window value, Sim(D m , D m+1 ) is the size of the window value, and W is the step size.

[0037] Furthermore, the calculation formula for reducing the step length is as follows:

[0038]

[0039] Where Tmax is the maximum similarity threshold of the window value, Sim(D m , D m+1 ) is the size of the window value, and W is the step size.

[0040] Furthermore, the formula for confirming whether the data is similar data is as follows:

[0041]

[0042] Where Sim(S1, S2) is the similarity, S1 and S2 are data, M1, M2 and M3 are weight values.

[0043] The beneficial effects of the present invention are as follows: from the running efficiency graph and the accuracy graph of the improved algorithm of the present invention, it can be seen that the graphics accuracy and efficiency in the present invention are significantly improved. The larger the amount of data, the more obvious the effect. It is mainly used for data cleaning. When the early data set reaches the order of tens of millions, compared with the nearest neighbor sorting algorithm, the improved algorithm can save more than 40% of the time and improve the accuracy by 60%. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 It is a flow chart of a data cleaning method for trial auxiliary matters based on an improved SNM algorithm according to an embodiment of the present invention;

[0046] Figure 2 is an operating efficiency diagram of a data cleaning method for trial auxiliary matters based on an improved SNM algorithm according to an embodiment of the present invention;

[0047] Figure 3 It is an accuracy graph of a data cleaning method for trial auxiliary matters based on an improved SNM algorithm according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] To further illustrate each embodiment, the present invention provides drawings, which are part of the disclosure of the present invention and are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, ordinary technicians in the field should be able to understand other possible implementations and advantages of the present invention. The components in the figures are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0049] According to an embodiment of the present invention, a data cleaning method for trial auxiliary matters based on an improved SNM algorithm is provided.

[0050] The present invention is further described with reference to the accompanying drawings and specific embodiments. Figure 1-Figure 3 As shown, according to the data cleaning method for trial auxiliary matters based on the improved SNM algorithm according to an embodiment of the present invention, the data cleaning method includes the following steps:

[0051] S1. Determine the initial value of the window according to the influencing factors of the window;

[0052] In one embodiment, the step of determining the initial value of the window according to the influencing factors of the window further comprises the following steps:

[0053] S11, determine the window influencing factors;

[0054] S12. Let the size of the window data set be W i , the number of window data similarity comparisons is W i -1;

[0055] S13, adjusting the window to a reasonable range according to the comparison result, and determining the initial value of the window;

[0056] In one embodiment, the influencing factors include data set size and data quality.

[0057] S2, using non-parametric regression method to estimate the probability density of the window and determine the threshold of the window;

[0058] In one embodiment, the method of estimating the probability density of the window by non-parametric regression and determining the threshold of the window further comprises the following steps:

[0059] S21, estimating the probability density of the window using a nonparametric regression method;

[0060] S22, calculate the probability density and determine the rejection region;

[0061] S23. Calculate the threshold value according to the distribution of the rejection region.

[0062] S3, matching the data set in the window, and determining whether to adjust the window size according to the matching result;

[0063] In one embodiment, matching the data set in the window and determining whether to adjust the window size according to the matching result further comprises the following steps:

[0064] S31, let the data set in the window be {D m , D m+1 , ..., D m+n}, compare Dm, and calculate Sim(D m , D m+1 ) size;

[0065] S32, when compared to Dm + n, if Sim(D m , D m+n )>T min , then adjust the window according to the comparison result;

[0066] S33、If Sim(D m , D m+n )<T max , then adjust the window according to the comparison result;

[0067] Among them, D m , D m+1 , ..., D m+n All are data sets, Sim(D m , D m+1 ) is the size of the window value, T min is the minimum similarity threshold of the window value, T max is the maximum similarity threshold of the window value;

[0068] In one embodiment, adjusting the window according to the comparison result further comprises the following steps:

[0069] If Sim(D m , D m+n )>T min , then expand the window;

[0070] If Sim(D m , D m+n )<T max , the window will be reduced.

[0071] S4. Determine the window adjustment step size according to the step size calculation formula;

[0072] In one embodiment, determining the window adjustment step size according to the step size calculation formula further comprises the following steps:

[0073] S41, determine the size of the window, and calculate Sim(D m , D m+n ) The size of the window values ​​for the mth and m+nth windows;

[0074] S42: Determine whether to increase the minimum similarity threshold of the window value.

[0075] In one embodiment, the calculation formula of the enlarged step length is as follows:

[0076]

[0077] Where Tmin is the minimum similarity threshold of the window value, Sim(D m , D m+1 ) is the size of the window value, and W is the step size.

[0078] In one embodiment, the calculation formula for the reduction step size is as follows:

[0079]

[0080] Where Tmax is the maximum similarity threshold of the window value, Sim(D m , D m+1 ) is the size of the window value, and W is the step size.

[0081] S5. Segment the window, extract similar words, and perform conjunctions, assign weight values ​​M1, M2, and M3 in sequence, and confirm whether the data is similar data.

[0082] In one embodiment, the formula for confirming whether the data is similar data is as follows:

[0083]

[0084] Where Sim(S1, S2) is the similarity, S1 and S2 are data, M1, M2 and M3 are weight values.

[0085] In specific applications, participle:

[0086] For example, the address "No. 115, Dazhou Road, Yuhuatai District, Nanjing City, Jiangsu Province" is segmented to get "Jiangsu Province", "Nanjing City", "Yuhuatai District", "Dazhou Road", "No. 115"

[0087] Semantic Extraction:

[0088] Perform approximate word matching on the segmented words and extract semantics: "Jiangsu Province" = "Jiangsu", "Nanjing City" = "Nanjing", "Yuhuatai District" = "Yuhuatai"

[0089] conjunction:

[0090] Connect sentences by using fixed hyphens, "Jiangsu / Nanjing / Yuhuatai / Dazhou Road / No. 116";

[0091] An example of extracting three keywords and sorting them separately is as follows:

[0092] For example, in personnel information, extract the three fields of "ID card", "name" and "address", sort them respectively, assign weight values, and finally confirm whether the data is similar data.

[0093] In summary, with the help of the above technical solution of the present invention, it can be seen from the running efficiency graph and accuracy graph of the improved algorithm of the present invention that the accuracy and efficiency of the graph in the present invention are significantly improved. The larger the amount of data, the more obvious the effect. It is mainly used for data cleaning. When the early data set reaches the order of tens of millions, compared with the nearest neighbor sorting algorithm, the improved algorithm can save more than 40% of time and improve the accuracy by 60%.

[0094] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data cleaning method based on improved SNM algorithm in trial auxiliary matters, It is characterized in that The data cleaning method includes the following steps: S1. Determine the initial value of the window according to the influencing factors of the window; S2, using non-parametric regression method to estimate the probability density of the window and determine the threshold of the window; S3, matching the data set in the window, and determining whether to adjust the window size according to the matching result; S4. Determine the window adjustment step size according to the step size calculation formula; S5: Segment the window, extract similar words, and perform conjunctions, and assign weight values ​​M in turn. 1 、M 2 and M 3 , and confirm whether the data is similar data; The matching of the data set in the window and determining whether to adjust the window size according to the matching result also includes the following steps: S31, let the data set in the window be {D m , D m+1 , ..., D m+n }, compare Dm, and calculate Sim(D m , D m+1 ) size; S32, when compared to Dm + n, if Sim(D m , D m+n )>T min , then adjust the window according to the comparison result; S33、If Sim(D m , D m+n )<T max , then adjust the window according to the comparison result; Among them, D m , D m+1 , ..., D m+n All are data sets, Sim(D m , D m+1 ) is the size of the window value, T min is the minimum similarity threshold of the window value, T max is the maximum similarity threshold of the window value; The step of adjusting the window according to the comparison result further comprises the following steps: If Sim(D m , D m+n )>T min , then expand the window; If Sim(D m , D m+n )<T max , the window will be reduced.

2. According to claim 1, a data cleaning method based on the improved SNM algorithm in trial auxiliary matters, It is characterized in that The step of determining the initial value of the window according to the influencing factors of the window further comprises the following steps: S11, determine the window influencing factors; S12. Let the size of the window data set be W i , the number of window data similarity comparisons is W i -1; S13. Adjust the window to a reasonable range according to the comparison result, and determine the initial value of the window.

3. According to claim 1, a data cleaning method based on the improved SNM algorithm in trial auxiliary matters, It is characterized in that The influencing factors include data set size and data quality.

4. According to claim 1, a data cleaning method based on the improved SNM algorithm in trial auxiliary matters, It is characterized in that The method of estimating the probability density of the window by non-parametric regression and determining the threshold of the window also includes the following steps: S21, estimating the probability density of the window using a nonparametric regression method; S22, calculate the probability density and determine the rejection region; S23. Calculate the threshold value according to the distribution of the rejection region.

5. According to claim 4, a data cleaning method based on the improved SNM algorithm in trial auxiliary matters, It is characterized in that Determining the window adjustment step size according to the step size calculation formula further comprises the following steps: S41, determine the size of the window, and calculate Sim(D m , D m+n ) The size of the window values ​​for the mth and m+nth windows; S42: Determine whether to increase the minimum similarity threshold of the window value.

6. According to claim 5, a data cleaning method based on the improved SNM algorithm in trial auxiliary matters, It is characterized in that The calculation formula of the enlarged step length is as follows: Where Tmin is the minimum similarity threshold of the window value, Sim(D m , D m+1 ) is the size of the window value, and W is the step size.

7. According to claim 6, a data cleaning method based on the improved SNM algorithm in trial auxiliary matters, It is characterized in that The calculation formula for the reduction step length is as follows: Where Tmax is the maximum similarity threshold of the window value, Sim(D m , D m+1 ) is the size of the window value, and W is the step size.

8. According to claim 7, a data cleaning method based on the improved SNM algorithm in trial auxiliary matters, It is characterized in that The formula for confirming whether the data is similar data is as follows: In the formula, Sim(S1, S2) is the similarity, S1 and S2 are data, M 1 、M 2 and M 3 All are weight values.

Images