Big data cleaning method and system based on artificial intelligence

Abstract

The invention discloses a big data cleaning method and system based on artificial intelligence, and particularly relates to the field of big data: efficient and accurate data de-duplication is realized through a systematized big data cleaning process in combination with application of byte-by-byte matching analysis, similarity calculation, clustering analysis and a machine learning model; through similarity evaluation and classification of unmatched target fields, in combination with a clustering algorithm, features are generated, and a machine learning model is trained, the accuracy and reliability of a duplicate removal process are optimized, the risk of mistakenly deleting key customer records is remarkably reduced, the integrity of enterprise customer data is guaranteed, and the enterprise customer data processing efficiency is improved. And the operation efficiency and decision-making quality of customer relationship management are improved.

Description

Technical Field

[0001] The present invention relates to the field of big data, and in particular to a big data cleaning method and system based on artificial intelligence. Background Art

[0002] When conducting customer relationship management, companies usually collect customer data from multiple channels (such as websites, social media, and telephone). This data may contain duplicate or erroneous information. Through data cleaning, the accuracy of customer information can be ensured, thereby improving the effectiveness of marketing activities and customer satisfaction. Among them, big data cleaning refers to the process of preprocessing the original big data, aiming to remove or correct errors, inconsistencies, duplicate values, and irrelevant information in the data to ensure the accuracy and completeness of the data.

[0003] In customer relationship management, big data cleaning technologies include data deduplication, data consistency verification, and missing data processing. Duplicate customer records are identified and deleted through data deduplication algorithms to ensure that each customer appears only once. However, during the data deduplication process, the data deduplication algorithm may mistakenly identify two customer records that are actually different as duplicate records and delete one of them. This mistaken deletion usually occurs when the data fields do not match exactly, but there is enough similarity that the algorithm misjudges, for example, two customers may have the same name or similar contact information. And the consequences of mistaken deletion can be extremely serious, especially in customer relationship management. This may lead to the loss of customer information, making it impossible for the company to identify or contact important customers, affecting customer experience and business operations. Especially in high-value customer groups, mistakenly deleting a key customer's record may directly lead to revenue loss and even affect customer loyalty and brand reputation. Summary of the invention

[0004] The purpose of the present invention is to provide a big data cleaning method and system based on artificial intelligence to solve the deficiencies in the background technology.

[0005] In order to achieve the above object, the present invention provides the following technical solution: a big data cleaning method based on artificial intelligence, comprising the following steps:

[0006] S1: Integrate and pre-process duplicate customer record data from several data sources of the enterprise, determine the target fields for deduplication, perform byte-by-byte matching analysis on the target fields, and divide them into matching target fields and non-matching target fields based on the analysis results;

[0007] S2: Select the corresponding similarity algorithm according to the different data types of the unmatched target field, and calculate the comprehensive similarity score of the unmatched target field by combining the similarities of all data types of the unmatched target field through the weighted average method;

[0008] S3: after comparing the obtained comprehensive similarity score of the unmatched target field with the gradient standard threshold, the similarity of the unmatched target field is divided into different levels, namely, a high similarity level, a general similarity level and a low similarity level;

[0009] S4: extract the non-matching target fields with high similarity levels and establish a data set. Treat each non-matching target field as an independent data entry, cluster the data set through a clustering algorithm, and generate a cluster to which each non-matching target field belongs.

[0010] S5: Perform association analysis on the cluster to which each unmatched target field belongs to determine the association relationship of the data items within the cluster, and generate new intra-cluster features based on the association relationship;

[0011] S6: Add the generated new features to the dataset, use the machine learning model for training and testing, and incorporate the trained machine learning model into the data cleaning process to predict the accuracy of deduplication and make corresponding processing decisions based on the prediction results.

[0012] Preferably, in S1, a byte-by-byte matching analysis is performed on the target field, and it is divided into a matching target field and a non-matching target field according to the analysis result. Specifically, each pair of strings to be compared is compared one by one using a selected function. If each character of the two strings is consistent, it is considered a complete match; if any character is different, it is considered a non-match.

[0013] Preferably, in S2, the comprehensive similarity score of the unmatched target field is calculated, specifically:

[0014] According to the importance of each field, a weight is assigned to the similarity score of each field, and the similarity score of each field is multiplied by the corresponding weight to obtain a weighted similarity score. The weighted similarity scores of all fields are summed up to obtain the final comprehensive similarity score.

[0015] Preferably, in S3, after comparing the obtained comprehensive similarity score of the unmatched target field with the gradient standard threshold, the similarity of the unmatched target field is divided into different levels, specifically: comparing the obtained comprehensive similarity score of the unmatched target field with the gradient standard threshold, the gradient standard threshold includes a first standard threshold and a second standard threshold, and the first standard threshold is less than the second standard threshold, and comparing the comprehensive similarity score of the unmatched target field with the first standard threshold and the second standard threshold respectively;

[0016] If the comprehensive similarity score of the unmatched target field is greater than the second standard threshold, it is classified as a high similarity level; if the comprehensive similarity score of the unmatched target field is greater than or equal to the first standard threshold and less than or equal to the second standard threshold, it is classified as a general similarity level; if the comprehensive similarity score of the unmatched target field is less than the first standard threshold, it is classified as a low similarity level.

[0017] Preferably, the data set is input into a hierarchical clustering algorithm for cluster analysis to generate a cluster to which each unmatched target field belongs, specifically:

[0018] Calculate the distance between each pair of data points and generate an N×N distance matrix, where N is the number of data points. In agglomerative hierarchical clustering, treat each data point as an independent cluster. Start the clustering process. According to the distance matrix, find the two closest clusters. Merge the two closest clusters. Update the distance matrix. Repeat the merging of the closest cluster pairs until the stopping condition is met. Visualize the clustering process through a dendrogram to show the order and distance of cluster merging. Determine the number of clusters based on the dendrogram. Assign a unique cluster identifier to each generated cluster. Assign each unmatched target field to the cluster to which it belongs, and record the cluster ID of the field.

[0019] Preferably, in S5, after performing association analysis on the cluster to which each unmatched target field belongs, the association relationship of the data items in the cluster is determined, and a new intra-cluster feature is generated based on the association relationship, specifically:

[0020] The intra-cluster correlation feature is generated according to the correlation between different data items in the cluster. The intra-cluster correlation feature is the intra-cluster correlation index. The intra-cluster correlation index is obtained by:

[0021] Perform statistics on the data within the cluster, select two data items X and Y to calculate the association index, count the number of times X=x appears in the data within the cluster, divide it by the total number of samples, and get the marginal probability P(x), which is expressed as: In the formula, count(X=x) is the number of times X=x appears, N is the total number of samples, and the number of times Y=y appears in the data within the cluster is counted and divided by the total number of samples to obtain the marginal probability P(y), which is expressed as: In the formula, count(Y=y) is the number of times Y=y appears; count the number of times X=x and Y=y appear simultaneously in the data within the cluster, and divide it by the total number of samples to get P(x,y), which is expressed as: Calculate the intra-cluster association index, the expression is: Where PM represents the intra-cluster correlation index.

[0022] Preferably, a data clustering quality feature is generated according to the association relationship of the data items in the cluster, and the data clustering quality feature is a clustering quality anomaly index, wherein the clustering quality anomaly index is obtained by executing a clustering algorithm, dividing the data set into k clusters, each cluster containing a number of data points, and the centroid of each cluster is the average vector of all the data points in the cluster, and calculating the centroid of each cluster, the expression is: In the formula, C i is the centroid of the ith cluster, |C i | is the number of data points in the i-th cluster, x j is the vector representation of data point j in the i-th cluster, and the Euclidean distance between the centroids of each pair of clusters is calculated, which represents the separation degree M between clusters. i,j , the expression is: M i,j =∥C i -C j ∥; For each cluster, calculate the average distance between all data points in the cluster and the cluster centroid, indicating the compactness of the cluster, which is recorded as S i , the expression is: In the formula, ||x j -C i || represents the Euclidean distance between data point j and the centroid of the i-th cluster. For each cluster i, find the cluster j closest to it and calculate the clustering quality anomaly index, which is expressed as: Where RG is the clustering quality anomaly index.

[0023] Preferably, in S6, the generated new features are added to the data set, and the machine learning model is used for training and testing, specifically:

[0024] The intra-cluster association index and clustering quality anomaly index are converted into feature vectors, and the feature vectors are used as inputs of the machine learning model. The machine learning model uses the accuracy value label of the data deduplication process predicted by each group of feature vectors as the prediction target, and minimizes the sum of the prediction errors of the accuracy value labels of all data deduplication processes as the training target. The machine learning model is trained until the sum of the prediction errors converges, and the model training is stopped. The accuracy value of the data deduplication process is determined according to the model output results, wherein the machine learning model is a polynomial regression model.

[0025] Preferably, the acquired accuracy value of the data deduplication process is compared with a preset reference threshold of the accuracy value of the data deduplication process. If the accuracy value of the data deduplication process is greater than or equal to the reference threshold of the accuracy value, it indicates that the accuracy of the data deduplication process is high, and the deduplication result is marked as valid and applied to the actual data processing flow; if the accuracy value of the data deduplication process is less than the reference threshold of the accuracy value, it indicates that the accuracy of the data deduplication process is low, and the result is further reviewed or the parameters are adjusted before deduplication is performed again.

[0026] The present invention also provides a big data cleaning system based on artificial intelligence, including a target field matching module, a similarity calculation module, a similarity level division module, a cluster analysis module, an association analysis module and an integration module;

[0027] Target field matching module: Integrate and pre-process duplicate customer record data from several data sources of the enterprise, determine the target fields for deduplication, perform byte-by-byte matching analysis on the target fields, and divide them into matching target fields and non-matching target fields based on the analysis results;

[0028] Similarity calculation module: selects the corresponding similarity algorithm according to the different data types of the unmatched target field, and calculates the comprehensive similarity score of the unmatched target field by combining the similarities of all data types of the unmatched target field through the weighted average method;

[0029] Similarity level division module: after comparing the obtained comprehensive similarity score of the unmatched target field with the gradient standard threshold, the similarity of the unmatched target field is divided into different levels, namely, high similarity level, general similarity level and low similarity level;

[0030] Cluster analysis module: extracts the non-matching target fields with high similarity levels and creates a data set. Each non-matching target field is treated as an independent data entry. The data set is clustered using a clustering algorithm to generate a cluster to which each non-matching target field belongs.

[0031] Association analysis module: After performing association analysis on the cluster to which each unmatched target field belongs, determine the association relationship of the data items within the cluster, and generate new intra-cluster features based on the association relationship;

[0032] Integration module: Add the generated new features to the dataset, use the machine learning model for training and testing, and incorporate the trained machine learning model into the data cleaning process to predict the accuracy of deduplication and make corresponding processing decisions based on the prediction results.

[0033] In the above technical solution, the technical effects and advantages provided by the present invention are:

[0034] 1. The present invention significantly improves the accuracy and automation of the data deduplication process by introducing a polynomial regression model and a systematic big data cleaning process. By combining byte-by-byte matching analysis, similarity calculation, cluster analysis, and the comprehensive application of machine learning models, the risk of accidentally deleting important customer records is effectively reduced, ensuring the integrity and consistency of customer data. This process not only improves the quality of data management, but also improves the efficiency of customer relationship management, reducing the risk of customer churn and revenue loss due to data errors.

[0035] 2. The present invention realizes intelligent deduplication quality control by converting the intra-cluster association index and the clustering quality anomaly index into feature vectors and using a machine learning model to predict the accuracy of the data deduplication process. The system can automatically identify potential deduplication problems and make adjustments based on actual conditions, thereby optimizing the overall process of data cleaning while ensuring the quality of deduplication, and improving the reliability of enterprise data operations and the accuracy of decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0037] Figure 1 The present invention is a flow chart of the method.

[0038] Figure 2 It is a system module diagram of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] Example 1, please refer to Figure 1 As shown, the big data cleaning method based on artificial intelligence described in this embodiment includes the following steps:

[0041] S1: Integrate and pre-process duplicate customer record data from several data sources of the enterprise, determine the target fields for deduplication, perform byte-by-byte matching analysis on the target fields, and divide them into matching target fields and non-matching target fields based on the analysis results;

[0042] S2: Select the corresponding similarity algorithm according to the different data types of the unmatched target field, and calculate the comprehensive similarity score of the unmatched target field by combining the similarities of all data types of the unmatched target field through the weighted average method;

[0043] S3: after comparing the obtained comprehensive similarity score of the unmatched target field with the gradient standard threshold, the similarity of the unmatched target field is divided into different levels, namely, a high similarity level, a general similarity level and a low similarity level;

[0044] S4: extract the non-matching target fields with high similarity levels and establish a data set. Treat each non-matching target field as an independent data entry, cluster the data set through a clustering algorithm, and generate a cluster to which each non-matching target field belongs.

[0045] S5: Perform association analysis on the cluster to which each unmatched target field belongs to determine the association relationship of the data items within the cluster, and generate new intra-cluster features based on the association relationship;

[0046] S6: Add the generated new features to the dataset, use the machine learning model for training and testing, and incorporate the trained machine learning model into the data cleaning process to improve the accuracy of the data cleaning and deduplication process.

[0047] Among them, in S1, the duplicate customer record data of several data sources of the enterprise are integrated and preprocessed, the target fields for deduplication are determined, and the target fields are subjected to byte-by-byte matching analysis. According to the analysis results, they are divided into matching target fields and non-matching target fields, specifically:

[0048] Collect data: Gather all customer records from multiple data sources (e.g., CRM system, email lists, customer order systems, social media data).

[0049] Data format unification: Unify data from different sources into the same format, such as unifying dates into the YYYY-MM-DD format and formatting phone numbers into international formats.

[0050] Data cleaning: Remove blank characters and special characters: Remove extra spaces and special characters (such as ",",".","-", etc.) from each field to ensure data consistency.

[0051] Format strings: Ensure that the strings being compared use the same format. This includes: Case unification: Convert all strings to all lowercase or all uppercase to avoid mismatches caused by case differences. Example: "John Smith" is converted to "john smith".

[0052] Remove leading and trailing spaces: Remove leading and trailing spaces from a string to avoid mismatches caused by extra spaces. Example: "john smith" is converted to "john smith".

[0053] Remove special characters: Remove or unify special characters (such as "-", ".", ", etc.). Example: "John-Smith" is converted to "johnsmith".

[0054] Choose a programming language or tool, such as Python, SQL, JavaScript, etc., which have built-in string comparison functions. For example, Python: use the lower() method to compare strings. Example: john smith.lower() == "John Smith".lower().

[0055] Use the selected function to compare each pair of strings that need to be compared one by one. Complete match: If every character of the two strings is the same, it is considered a complete match. Mismatch: If any character is different, it is considered a mismatch.

[0056] Record the comparison results: Matching records: Mark the fully matched strings as "matched". Unmatching records: Mark the unmatched strings as "unmatched". Output matching results: Store the matching and unmatching results in the corresponding data structure or file for subsequent processing.

[0057] Handling mismatches: Manual review: For important data records, manual review can be performed to check whether the mismatch is caused by preprocessing or other factors. Based on the comparison results, the preprocessing rules or matching criteria are adjusted to ensure the accuracy of the algorithm.

[0058] S2: Select the corresponding similarity algorithm according to the different data types of the unmatched target field, and calculate the comprehensive similarity score of the unmatched target field by combining the similarities of all data types of the unmatched target field through the weighted average method.

[0059] Determine the data type of the unmatched target field: Text fields: such as name, address, company name, etc. Numeric fields: such as phone number, zip code, age, etc. Date fields: such as date of birth, registration date, etc. Categorical fields: such as gender, country, city, and other categorical information.

[0060] Text field similarity algorithms include:

[0061] Levenshtein distance: Calculates the edit distance (the minimum number of operations required for insertion, deletion, or substitution) between two strings.

[0062] Jaro-Winkler distance: specially used for similarity calculation of short strings, taking into account the situation of character exchange, and is suitable for names, place names, etc.

[0063] Cosine similarity: Converts a string into a vector and calculates the cosine angle between two vectors for similarity analysis of long text fields.

[0064] Numeric field similarity algorithms include:

[0065] Euclidean distance: Calculates the distance between two values, suitable for calculating direct numerical differences.

[0066] Absolute difference: Takes the absolute difference between two values, suitable for scenarios that do not require complexity.

[0067] Normalized Difference: Normalize the numeric field and then calculate the difference. This is suitable for situations where the numeric range varies greatly.

[0068] Date field similarity algorithms include:

[0069] Date Difference: Calculates the difference in days between two dates, usually setting the similarity based on the difference.

[0070] Month or year difference: Sometimes only the month or year is compared, ignoring the specific date, which is suitable for some scenarios.

[0071] Categorical field similarity algorithms include:

[0072] Jaccard similarity: Calculates the ratio of the intersection and union between two classification sets, which is used for the similarity of classification labels.

[0073] Hamming distance: Calculates the number of different bits between two strings or bit sequences of the same length, for binary or multi-class data.

[0074] Apply the selected algorithm: For each unmatched target field, apply the corresponding similarity algorithm to calculate the similarity score for each field. Text field: For example, for the two names "John Smith" and "Jon Smith", the similarity score is calculated using the Levenshtein distance.

[0075] Numeric fields: For example, for the two phone numbers "1234567890" and "1234567891", the similarity score is calculated using the Euclidean distance.

[0076] Date field: For example, for two dates "2023-08-15" and "2023-08-20", the date difference is calculated and converted into a similarity score.

[0077] Categorical fields: For example, for two country codes "US" and "CA", the similarity score is calculated using Jaccard similarity.

[0078] Assign weights to the similarity score for each field based on its importance. The weights should be assigned to reflect the relative importance of the field to the overall similarity. For example, a name might have a weight of 0.5, a phone number might have a weight of 0.3, and an address might have a weight of 0.2.

[0079] Normalize weights: Make sure the sum of all field weights is 1.

[0080] Weighted similarity score: Multiply the similarity score of each field by the corresponding weight to get the weighted similarity score. The formula is: weighted similarity = field similarity score × field weight; for example, if the similarity score of the name is 0.9 and the weight is 0.5, the weighted similarity is 0.9 × 0.5 = 0.45. Sum the weighted similarity scores of all fields to get the final comprehensive similarity score. The formula is: comprehensive similarity score = Σ (weighted similarity score of each field); for example, if the weighted similarity scores of the name, phone number, and address are 0.45, 0.24, and 0.18 respectively, the comprehensive similarity score is 0.45 + 0.24 + 0.18 = 0.87.

[0081] S3: After comparing the obtained comprehensive similarity score of the unmatched target field with the gradient standard threshold, the similarity of the unmatched target field is divided into different levels, namely, a high similarity level, a general similarity level and a low similarity level.

[0082] The obtained comprehensive similarity score of the unmatched target field is compared with the gradient standard threshold, the gradient standard threshold includes a first standard threshold and a second standard threshold, and the first standard threshold is less than the second standard threshold, and the comprehensive similarity score of the unmatched target field is compared with the first standard threshold and the second standard threshold respectively;

[0083] If the comprehensive similarity score of the unmatched target field is greater than the second standard threshold, it means that the similarity of the unmatched target field is high, and it is classified as a high similarity level;

[0084] If the comprehensive similarity score of the unmatched target field is greater than or equal to the first standard threshold and less than or equal to the second standard threshold, it means that the similarity of the unmatched target field is in the middle, and it is classified as a general similarity level;

[0085] If the comprehensive similarity score of the unmatched target field is less than the first standard threshold, it means that the similarity of the unmatched target field is low, and it is classified as a low similarity level.

[0086] High similarity level processing: records can be automatically merged or marked as duplicates.

[0087] General similarity level processing: records marked for manual review will be further analyzed to decide whether to merge or delete.

[0088] Low similarity level processing: Usually kept as independent records, without merging or deleting.

[0089] S4: After extracting the unmatched target fields with high similarity levels, a data set is established. Each unmatched target field is treated as an independent data entry. The data set is clustered by a clustering algorithm to generate a cluster to which each unmatched target field belongs.

[0090] Filter records: Extract all unmatched target fields that are classified as "high similarity level" from the deduplication analysis. The combined similarity score of these fields is higher than the second criterion threshold.

[0091] Create data structure: Treat each highly similar unmatched target field as an independent data entry to construct a new data set.

[0092] Record field information: retain relevant field information (such as field name, source data source, similarity score, etc.) for subsequent processing.

[0093] Vectorization: For text data, you can use vectorization techniques (such as TF-IDF, word embedding, etc.) to convert text into numerical vectors. For example, use TF-IDF to process text fields and represent each field as a vector.

[0094] Data cleaning: Remove missing values ​​or outliers to ensure the consistency and completeness of the data set.

[0095] Select a clustering algorithm. Clustering algorithms include:

[0096] K-means: It is suitable for situations where the amount of data is large and it is desired to divide the data into a fixed number of clusters. Choice of K: Determine the K value (the number of clusters) by the elbow rule or silhouette coefficient.

[0097] DBSCAN: Suitable for finding clusters of arbitrary shapes and can handle noisy data. Parameter selection: Determine the epsilon (neighborhood radius) and min_samples (minimum number of points) parameters.

[0098] Hierarchical clustering: Suitable for understanding hierarchical structures in data, especially when there are obvious hierarchical relationships in the dataset.

[0099] The data set is input into the hierarchical clustering algorithm for cluster analysis to generate the cluster to which each unmatched target field belongs, specifically:

[0100] Choose a distance metric: Common distance metrics include Euclidean distance, Manhattan distance, etc. Choose an appropriate distance metric based on the characteristics of the data.

[0101] Compute distance matrix: Compute the distance between each pair of data points using the selected distance metric, generating an N×N distance matrix, where N is the number of data points.

[0102] Agglomerative hierarchical clustering: Start with each data point as a cluster and gradually merge the nearest clusters until the desired number of clusters is formed.

[0103] Divisive hierarchical clustering: Starting with the entire dataset as one cluster, it is gradually split into multiple clusters until the desired number of clusters is formed.

[0104] The clustering process includes:

[0105] Initialization: In agglomerative hierarchical clustering, each data point is considered as an independent cluster and the clustering process begins.

[0106] Calculate the nearest cluster pair: Find the two closest clusters based on the distance matrix.

[0107] Merge clusters: merge the two closest clusters and update the distance matrix. The merging method can be: Single link method (min): the distance between the closest point of the two clusters. Full link method (max): the distance between the farthest point of the two clusters. Average link method (average): the average distance of all points between the two clusters. Centroid method (centroid): the distance between the centroid of the merged cluster and other clusters.

[0108] Repeated merging: Repeatedly merge the closest cluster pairs until a stopping condition is met, such as reaching a preset number of clusters or reaching a maximum distance.

[0109] Draw a dendrogram: Visualize the clustering process through a dendrogram to show the order and distance of cluster merging. Interpret the dendrogram: The horizontal axis of the dendrogram represents the samples, and the vertical axis represents the distance of merging. The higher the branches, the farther the distance between clusters.

[0110] Select the number of clusters: Determine the appropriate number of clusters based on the dendrogram. You can select a "cut" point on the dendrogram to truncate the branches of the dendrogram and determine the number of clusters.

[0111] Cluster identifier generation: Each generated cluster is assigned a unique cluster identifier (such as Cluster_1, Cluster_2, etc.).

[0112] Data point assignment: Assign each unmatched target field (data point) to the cluster to which it belongs and record the cluster ID of the field.

[0113] Record cluster information: record the characteristic information of the cluster (such as the center point within the cluster, the number of samples in the cluster, etc.) to facilitate subsequent analysis.

[0114] S5: After performing association analysis on the cluster to which each unmatched target field belongs, determine the association relationship of the data items within the cluster, and generate new intra-cluster features based on the association relationship.

[0115] Collect all data items in each cluster and organize them into a data set. Make sure the data set contains the original field information of each data item and the relevant cluster identifier (cluster ID). Select key features (such as text fields, numerical fields, date fields, etc.) that may have association relationships from the data items in the cluster. These features will be used for association analysis.

[0116] The intra-cluster correlation feature is generated according to the correlation between different data items in the cluster. The intra-cluster correlation feature is the intra-cluster correlation index. The intra-cluster correlation index is obtained by:

[0117] Perform statistics on the data within the cluster, select two data items X and Y to calculate the association index, count the number of times X=x appears in the data within the cluster, divide it by the total number of samples, and get the marginal probability P(x), which is expressed as: In the formula, count(X=x) is the number of times X=x appears, N is the total number of samples, and the number of times Y=y appears in the data within the cluster is counted and divided by the total number of samples to obtain the marginal probability P(y), which is expressed as: In the formula, count(Y=y) is the number of times Y=y appears; count the number of times X=x and Y=y appear simultaneously in the data within the cluster, and divide it by the total number of samples to get P(x,y), which is expressed as: Calculate the intra-cluster association index, the expression is: In the formula, PM represents the intra-cluster association index, that is, the strength of the association between specific values ​​x and y that appear together in the cluster. The higher the PM value, the stronger the association between the two values.

[0118] The larger the intra-cluster correlation index is, the stronger the correlation between the data items in the cluster is, indicating that in the data cleaning and deduplication process, the similar records identified are more likely to be actual duplicate records. This means that the accuracy of the cleaning and deduplication process is higher, reducing the risk of accidental deletion and incorrect merging, thereby ensuring the integrity and reliability of the data.

[0119] A data clustering quality feature is generated according to the association relationship of the data items in the cluster, wherein the data clustering quality feature is a clustering quality anomaly index, wherein the clustering quality anomaly index is obtained by:

[0120] Execute the clustering algorithm to divide the data set into k clusters. Each cluster contains a number of data points. The centroid of each cluster is the average vector of all data points in the cluster. The centroid of each cluster is calculated as follows: In the formula, C i is the centroid of the ith cluster, |C i | is the number of data points in the i-th cluster, x j is the vector representation of data point j in the i-th cluster, and the Euclidean distance between the centroids of each pair of clusters is calculated, which represents the separation degree M between clusters. i,j , the expression is: M i,j =∥C i -C j ∥; For each cluster, calculate the average distance between all data points in the cluster and the centroid of the cluster, indicating the compactness of the cluster, which is recorded as S i , the expression is: In the formula, ||x j -C i || represents the Euclidean distance between data point j and the centroid of the i-th cluster. For each cluster i, find the cluster j closest to it (that is, the cluster pair with the largest DB value) and calculate the clustering quality anomaly index, which is expressed as: Where RG is the clustering quality anomaly index.

[0121] The larger the clustering quality anomaly index is, the lower the accuracy of the data cleaning and deduplication process is. This means that there are more abnormal or bad clusters in the clustering results, the compactness of data points within the cluster is poor or the difference between clusters is not obvious, resulting in a higher risk of misjudgment or missed judgment, thereby reducing the effect and reliability of data cleaning and deduplication.

[0122] S6: Add the generated new features to the dataset, use the machine learning model for training and testing, and incorporate the trained machine learning model into the data cleaning process to predict the accuracy of deduplication and make corresponding processing decisions based on the prediction results.

[0123] The intra-cluster association index and clustering quality anomaly index are converted into feature vectors, and the feature vectors are used as inputs of the machine learning model. The machine learning model uses the accuracy value label of the data deduplication process predicted by each group of feature vectors as the prediction target, and minimizes the sum of the prediction errors of the accuracy value labels of all data deduplication processes as the training target. The machine learning model is trained until the sum of the prediction errors converges, and the model training is stopped. The accuracy value of the data deduplication process is determined according to the model output results, wherein the machine learning model is a polynomial regression model.

[0124] The method for obtaining the accuracy value of the data deduplication process is: from the feature vector training data of the trained machine learning model, obtain the corresponding function expression: CR=f1(PM, RG); where f1 is the output function of the model, PM is the intra-cluster association index, RG is the clustering quality anomaly index, and CR is the accuracy value of the data deduplication process.

[0125] The obtained accuracy value of the data deduplication process is compared with a preset reference threshold value of the accuracy value of the data deduplication process. If the accuracy value of the data deduplication process is greater than or equal to the reference threshold value of the accuracy value, it means that the accuracy of the data deduplication process is high and the deduplication result is acceptable. The deduplication result is marked as valid and applied to the actual data processing flow.

[0126] If the accuracy value of the data deduplication process is less than the reference threshold of the accuracy value, it means that the accuracy of the data deduplication process is low and there are problems with the deduplication result. You can choose to further review the result or adjust the parameters and perform deduplication again.

[0127] The trained polynomial regression model is integrated into the automated process of data cleaning and deduplication. During each deduplication process, the intra-cluster association index and cluster quality anomaly index are calculated and input into the model to predict the accuracy of deduplication and make corresponding processing decisions based on the prediction results.

[0128] When manual review or automatic system judgment indicates that it is necessary to adjust the deduplication algorithm or parameters. According to the specific characteristics of the data and the review feedback, adjust the relevant parameters in the deduplication process, such as the weighting coefficient of the similarity algorithm, the similarity threshold, the distance metric of the clustering algorithm, etc. After adjusting the parameters, rerun the deduplication algorithm, generate new deduplication results, and evaluate their accuracy again. After multiple adjustments and tests, find the best parameter combination to ensure the highest accuracy and stability of the deduplication process.

[0129] The system runs continuously in the production environment, and regularly checks the actual effect and accuracy of the deduplication results. The system regularly monitors the deduplication process of new data, and automatically triggers parameter optimization or manual review processes when accuracy drops.

[0130] In this application, by converting the intra-cluster association index and the cluster quality anomaly index into the first eigenvector and using a polynomial regression model to predict the accuracy value of the data deduplication process, enterprises can effectively judge the quality of the deduplication operation. If the predicted accuracy value is higher than the pre-set threshold, the deduplication result can be considered reliable; if it is lower than the threshold, further inspection or adjustment is required. This process will greatly improve the overall accuracy and efficiency of data cleaning and deduplication.

[0131] In this embodiment, the customer duplicate record data of several data sources of the enterprise are integrated and preprocessed to determine the target field for deduplication, and the target field is subjected to byte-by-byte matching analysis, and is divided into matching target fields and non-matching target fields according to the analysis results; the corresponding similarity algorithm is selected according to the different data types of the non-matching target field, and the comprehensive similarity score of the non-matching target field is calculated by combining the similarities of all data types of the non-matching target field through the weighted average method; after comparing the obtained comprehensive similarity score of the non-matching target field with the gradient standard threshold, the similarity of the non-matching target field is divided into different levels, and the similarity of the non-matching target field is divided into different levels. Divided into high similarity level, general similarity level and low similarity level; extract the unmatched target fields of high similarity level and establish a data set, take each unmatched target field as an independent data entry, cluster the data set through clustering algorithm, and generate the cluster to which each unmatched target field belongs; perform association analysis on the cluster to which each unmatched target field belongs, determine the association relationship of the data items in the cluster, and generate new intra-cluster features based on the association relationship; add the generated new features to the data set, train and test them using machine learning models, and incorporate the trained machine learning models into the data cleaning process to improve the accuracy of data cleaning and deduplication processes. It can effectively identify and process complex duplicate records, and the generated intra-cluster features further improve the decision quality of deduplication. Finally, the use of machine learning models to automate the entire process not only reduces manual intervention, but also greatly improves the efficiency and reliability of deduplication operations, thereby ensuring the integrity and consistency of enterprise data.

[0132] Example 2, please refer to Figure 2 As shown, the big data cleaning system based on artificial intelligence described in this embodiment includes a target field matching module, a similarity calculation module, a similarity level division module, a clustering analysis module, an association analysis module and an integration module;

[0133] Target field matching module: Integrate and pre-process duplicate customer record data from several data sources of the enterprise, determine the target fields for deduplication, perform byte-by-byte matching analysis on the target fields, and divide them into matching target fields and non-matching target fields based on the analysis results;

[0134] Similarity calculation module: selects the corresponding similarity algorithm according to the different data types of the unmatched target field, and calculates the comprehensive similarity score of the unmatched target field by combining the similarities of all data types of the unmatched target field through the weighted average method;

[0135] Similarity level division module: after comparing the obtained comprehensive similarity score of the unmatched target field with the gradient standard threshold, the similarity of the unmatched target field is divided into different levels, namely, high similarity level, general similarity level and low similarity level;

[0136] Cluster analysis module: extracts the non-matching target fields with high similarity levels and creates a data set. Each non-matching target field is treated as an independent data entry. The data set is clustered using a clustering algorithm to generate a cluster to which each non-matching target field belongs.

[0137] Association analysis module: After performing association analysis on the cluster to which each unmatched target field belongs, determine the association relationship of the data items within the cluster, and generate new intra-cluster features based on the association relationship;

[0138] Integration module: Add the generated new features to the dataset, use the machine learning model for training and testing, and incorporate the trained machine learning model into the data cleaning process to predict the accuracy of deduplication and make corresponding processing decisions based on the prediction results.

[0139] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0140] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0141] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0142] In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0143] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

Claims

1. A big data cleaning method based on artificial intelligence, characterized by: The following steps are involved: S1: Integrate and pre-process duplicate customer record data from several data sources of the enterprise, determine the target fields for deduplication, perform byte-by-byte matching analysis on the target fields, and divide them into matching target fields and non-matching target fields based on the analysis results; S2: Select the corresponding similarity algorithm according to the different data types of the unmatched target field, and calculate the comprehensive similarity score of the unmatched target field by combining the similarities of all data types of the unmatched target field through the weighted average method; S3: after comparing the obtained comprehensive similarity score of the unmatched target field with the gradient standard threshold, the similarity of the unmatched target field is divided into different levels, namely, a high similarity level, a general similarity level and a low similarity level; S4: extract the non-matching target fields with high similarity levels and establish a data set. Treat each non-matching target field as an independent data entry, cluster the data set through a clustering algorithm, and generate a cluster to which each non-matching target field belongs. S5: Perform association analysis on the cluster to which each unmatched target field belongs to determine the association relationship of the data items within the cluster, and generate new intra-cluster features based on the association relationship; S6: Add the generated new features to the dataset, use the machine learning model for training and testing, and incorporate the trained machine learning model into the data cleaning process to predict the accuracy of deduplication and make corresponding processing decisions based on the prediction results.

2. The big data cleaning method based on artificial intelligence according to claim 1, characterized in that: In S1, a byte-by-byte matching analysis is performed on the target field, and it is divided into a matching target field and a non-matching target field according to the analysis result. Specifically, each pair of strings to be compared is compared one by one using the selected function. If each character of the two strings is consistent, it is considered a complete match; if any character is different, it is considered a non-match.

3. The big data cleaning method based on artificial intelligence according to claim 1, characterized in that: In S2, the comprehensive similarity score of the unmatched target fields is calculated as follows: According to the importance of each field, a weight is assigned to the similarity score of each field, and the similarity score of each field is multiplied by the corresponding weight to obtain a weighted similarity score. The weighted similarity scores of all fields are summed up to obtain the final comprehensive similarity score.

4. The big data cleaning method based on artificial intelligence according to claim 1, characterized in that: In S3, after comparing the obtained comprehensive similarity score of the unmatched target field with the gradient standard threshold, the similarity of the unmatched target field is divided into different levels, specifically: comparing the obtained comprehensive similarity score of the unmatched target field with the gradient standard threshold, the gradient standard threshold includes a first standard threshold and a second standard threshold, and the first standard threshold is less than the second standard threshold, and comparing the comprehensive similarity score of the unmatched target field with the first standard threshold and the second standard threshold respectively; If the comprehensive similarity score of the unmatched target field is greater than the second standard threshold, it is classified as a high similarity level; If the comprehensive similarity score of the unmatched target field is greater than or equal to the first standard threshold and less than or equal to the second standard threshold, it is classified as a general similarity level; If the comprehensive similarity score of the unmatched target field is less than the first standard threshold, it is classified as a low similarity level.

5. The big data cleaning method based on artificial intelligence according to claim 1, characterized in that: The data set is input into the hierarchical clustering algorithm for cluster analysis to generate the cluster to which each unmatched target field belongs, specifically: Calculate the distance between each pair of data points and generate an N×N distance matrix, where N is the number of data points. In agglomerative hierarchical clustering, treat each data point as an independent cluster. Start the clustering process. According to the distance matrix, find the two closest clusters. Merge the two closest clusters. Update the distance matrix. Repeat the merging of the closest cluster pairs until the stopping condition is met. Visualize the clustering process through a dendrogram to show the order and distance of cluster merging. Determine the number of clusters based on the dendrogram. Assign a unique cluster identifier to each generated cluster. Assign each unmatched target field to the cluster to which it belongs, and record the cluster ID of the field.

6. The big data cleaning method based on artificial intelligence according to claim 5, characterized in that: In S5, after performing association analysis on the cluster to which each unmatched target field belongs, the association relationship of the data items in the cluster is determined, and new intra-cluster features are generated based on the association relationship, specifically: The intra-cluster correlation feature is generated according to the correlation between different data items in the cluster. The intra-cluster correlation feature is the intra-cluster correlation index. The intra-cluster correlation index is obtained by: Perform statistics on the data within the cluster, select two data items X and Y to calculate the association index, count the number of times X=x appears in the data within the cluster, divide it by the total number of samples, and get the marginal probability P(x), which is expressed as: In the formula, count(X=x) is the number of times X=x appears, N is the total number of samples, and the number of times Y=y appears in the data within the cluster is counted and divided by the total number of samples to obtain the marginal probability P(y), which is expressed as: In the formula, count(Y=y) is the number of times Y=y appears; count the number of times X=x and Y=y appear simultaneously in the data within the cluster, and divide it by the total number of samples to get P(x,y), which is expressed as: Calculate the intra-cluster association index, the expression is: Where PM represents the intra-cluster correlation index.

7. The big data cleaning method based on artificial intelligence according to claim 6, characterized in that: The data clustering quality feature is generated according to the association relationship of the data items in the cluster. The data clustering quality feature is the clustering quality anomaly index. The clustering quality anomaly index is obtained by executing a clustering algorithm to divide the data set into k clusters. Each cluster contains a number of data points. The centroid of each cluster is the average vector of all data points in the cluster. The centroid of each cluster is calculated. The expression is: In the formula, C i is the centroid of the ith cluster, |C i | is the number of data points in the i-th cluster, x j is the vector representation of data point j in the i-th cluster, and the Euclidean distance between the centroids of each pair of clusters is calculated, which represents the separation degree M between clusters. i,j , the expression is: M i,j =∥C i -C j ∥; For each cluster, calculate the average distance between all data points in the cluster and the cluster centroid, indicating the compactness of the cluster, which is recorded as S i , the expression is: In the formula, ||x j -C i || represents the Euclidean distance between data point j and the centroid of the i-th cluster. For each cluster i, find the cluster j closest to it and calculate the clustering quality anomaly index, which is expressed as: Where RG is the clustering quality anomaly index.

8. The method for cleaning big data based on artificial intelligence according to claim 7, characterized in that: In S6, the generated new features are added to the dataset and trained and tested using the machine learning model, specifically: The intra-cluster association index and clustering quality anomaly index are converted into feature vectors, and the feature vectors are used as inputs of the machine learning model. The machine learning model uses the accuracy value label of the data deduplication process predicted by each group of feature vectors as the prediction target, and minimizes the sum of the prediction errors of the accuracy value labels of all data deduplication processes as the training target. The machine learning model is trained until the sum of the prediction errors converges, and the model training is stopped. The accuracy value of the data deduplication process is determined according to the model output results, wherein the machine learning model is a polynomial regression model.

9. The big data cleaning method based on artificial intelligence according to claim 8, characterized in that: The obtained accuracy value of the data deduplication process is compared with a preset reference threshold of the accuracy value of the data deduplication process. If the accuracy value of the data deduplication process is greater than or equal to the reference threshold of the accuracy value, it means that the accuracy of the data deduplication process is high, and the deduplication result is marked as valid and applied to the actual data processing flow; If the accuracy value of the data deduplication process is less than the reference threshold of the accuracy value, it means that the accuracy of the data deduplication process is low, and the result should be further reviewed or the parameters should be adjusted before re-deduplicate.

10. An artificial intelligence-based big data cleaning system, used to implement an artificial intelligence-based big data cleaning method according to any one of claims 1 to 9, characterized in that: It includes target field matching module, similarity calculation module, similarity level classification module, cluster analysis module, association analysis module and integration module; Target field matching module: Integrate and pre-process duplicate customer record data from several data sources of the enterprise, determine the target fields for deduplication, perform byte-by-byte matching analysis on the target fields, and divide them into matching target fields and non-matching target fields based on the analysis results; Similarity calculation module: selects the corresponding similarity algorithm according to the different data types of the unmatched target field, and calculates the comprehensive similarity score of the unmatched target field by combining the similarities of all data types of the unmatched target field through the weighted average method; Similarity level division module: after comparing the obtained comprehensive similarity score of the unmatched target field with the gradient standard threshold, the similarity of the unmatched target field is divided into different levels, namely, high similarity level, general similarity level and low similarity level; Cluster analysis module: extracts the non-matching target fields with high similarity levels and creates a data set. Each non-matching target field is treated as an independent data entry. The data set is clustered using a clustering algorithm to generate a cluster to which each non-matching target field belongs. Association analysis module: After performing association analysis on the cluster to which each unmatched target field belongs, determine the association relationship of the data items within the cluster, and generate new intra-cluster features based on the association relationship; Integration module: Add the generated new features to the dataset, use the machine learning model for training and testing, and incorporate the trained machine learning model into the data cleaning process to predict the accuracy of deduplication and make corresponding processing decisions based on the prediction results.

Images