Method for cleaning ten millions of data based on LLM + Spark

Abstract

The invention provides a ten-million-level data cleaning method based on LLM + Spark, and belongs to the technical field of big data. Candidate rules are generated through a big language model, a rule warehouse is formed through semantic check and unit testing, a rule set and a data set are subjected to cross matching through a rule adaptation evaluation model, and an optimal rule combination is obtained; in a Spark cluster, first-stage cleaning based on format verification and a regular expression and second-stage cleaning based on vector retrieval and semantic rewriting are executed, meanwhile, a processing process is optimized by applying an inference rhythm shaping algorithm, a partition rebalance algorithm and an abnormal propagation suppression algorithm, an overall processing analysis report is generated, and iterative optimization is triggered. The technical problems that in the ten-million-level large-scale data cleaning process, rule generation depends on manual configuration, and multi-source heterogeneous data features cannot be dynamically adapted, so that the cleaning efficiency is low, and the accuracy is insufficient are solved.

Description

Technical Field

[0001] This invention belongs to the field of big data technology, specifically, it relates to a method for cleaning tens of millions of data points based on LLM+Spark. Background Technology

[0002] In the field of big data processing, cleaning tens of millions of data points typically employs a Spark-based distributed processing framework combined with a predefined rule base for data filtering and correction. Traditional techniques rely on data engineers manually writing cleaning rules based on business experience, using regular expressions, numerical range constraints, and format validation to batch process structured and unstructured data, and leveraging the MapReduce computing model to achieve parallel cleaning operations. However, the rule generation and adaptation process in traditional techniques is entirely dependent on manual configuration. When faced with multi-source heterogeneous data from different sources, time periods, and field types, a fixed rule set struggles to cover complex data characteristics, resulting in low rule coverage, high false positive rates, long manual configuration cycles, and difficulties in iterative optimization, failing to address dynamic changes in data characteristics. Existing technologies lack a quantitative assessment mechanism for data complexity and an intelligent matching mechanism between rules and data, preventing cleaning rules from automatically adjusting and optimizing according to data characteristics. This leads to poor rule adaptability and insufficient cleaning accuracy, especially when processing massive datasets of tens of millions of data points, resulting in low efficiency. In other words, existing technologies suffer from low cleaning efficiency and insufficient accuracy due to the reliance on manual configuration for rule generation during the cleaning of massive amounts of data (tens of millions of records) and the inability to dynamically adapt to the characteristics of multi-source heterogeneous data. Summary of the Invention

[0003] In view of this, the present invention provides a method for cleaning tens of millions of data points based on LLM+Spark, which can solve the technical problems in the existing technology where rule generation in the process of cleaning tens of millions of large-scale data points depends on manual configuration and cannot dynamically adapt to the characteristics of multi-source heterogeneous data, resulting in low cleaning efficiency and insufficient accuracy.

[0004] This invention is implemented as follows: This invention provides a method for cleaning tens of millions of data points based on LLM+Spark, including the following steps:

[0005] Structured and unstructured data sources are connected to the system through data synchronization tools, and the data is partitioned and stored in a distributed file system according to the data source and timestamp, forming partitioned storage data;

[0006] Statistical analysis is performed on the partitioned storage data to extract missing rate, number of unique values, field dependency and semantic relevance as data feature set. Data complexity index is calculated based on the data feature set, and 500 to 2000 records are extracted from each partition as representative sample set according to the data complexity index.

[0007] The representative sample set is assembled with field names, data sources, timestamps, and historical cleaning logs into input text. A private large language model is called to generate a candidate rule list. Each rule in the candidate rule list is subjected to semantic checks and unit tests and applied to the representative sample set to obtain rule coverage. Rules with rule coverage below the coverage threshold are marked as low-confidence rules and submitted for manual review. Rules that pass the review are written into the rule repository and labeled with version number and scope of application.

[0008] The rules in the rule warehouse are divided into N rule sets according to the applicable field types, and the partitioned storage data is divided into N datasets according to the data feature set. The N rule sets and the N datasets are then compared. The cross-matching process generates a rule data matching report, which is then input into the rule adaptation evaluation model to obtain the optimal rule combination for each dataset.

[0009] The optimal rule combination is broadcast to each execution node via Spark cluster. Each execution node performs the first stage of cleaning on the allocated data partition. The first stage of cleaning uses format validation rules, regular expression rules, and numerical range rules to filter and correct the data. Records that cannot pass the first stage of cleaning are marked as suspicious record subsets. The second stage of cleaning is performed on the suspicious record subsets. The second stage of cleaning converts the suspicious record subsets into record vectors through an embedding model and retrieves the standard metadata text corresponding to the highest similarity standard vector in the vector library. The suspicious record subsets are semantically rewritten and corrected according to the standard metadata text to form the cleaned data.

[0010] An overall processing analysis report is generated for the cleaned data. The overall processing analysis report includes a comparison of data quality indicators before and after cleaning, rule hit rate statistics, and distribution of abnormal records. The overall processing analysis report is compared with the expected rule behavior and submitted for manual review. After the review is approved, the corresponding optimal rule combination is marked as the formal application status and the next round of iteration optimization is triggered.

[0011] The data complexity index is calculated using a normalized weighted summation. The missing rate, the ratio of the number of unique values ​​to the total number of records, the field dependency strength, and the semantic relevance score in the data feature set are normalized and then weighted and summed with weight coefficients of 0.3, 0.2, 0.25, and 0.25.

[0012] The input layer of the rule adaptation evaluation model receives numerical features from four dimensions in the rule data matching report: rule coverage, data pass rate, field matching degree, and semantic consistency score.

[0013] The rule adaptation evaluation model consists of three fully connected layers with 128, 64, and 32 neurons respectively. The activation function is a modified linear unit function. The backbone network output is processed through an attention mechanism layer to evaluate the N-class rule sets against the N-class dataset. Weighted scoring is applied to the various combinations.

[0014] The establishment of the training dataset for the rule adaptation evaluation model includes collecting 1,000 to 5,000 sets of rule-data matching cases from historical data cleaning projects, labeling matching cases with a matching quality score higher than 80 as positive samples and matching cases with a score lower than 60 as negative samples, and dividing the training set and validation set in a 7:3 ratio.

[0015] The rule-fit evaluation model training uses the cross-entropy loss function, and the adaptive moment estimation optimization algorithm is used to adjust the model parameters. The learning rate is set to 0.001, the batch size is set to 32, and the training rounds are set to 100.

[0016] The first stage of cleaning uses format validation rules, regular expression rules, and numerical range rules to filter and correct the data, and marks records that cannot pass the first stage of cleaning as a subset of suspicious records.

[0017] In the second stage of cleaning, the suspicious record subset is converted into a 768-dimensional record vector through an embedding model. The standard vector with the highest similarity is retrieved from the vector library to obtain the corresponding standard metadata text. The suspicious record subset is then semantically rewritten and corrected based on the standard metadata text.

[0018] The overall processing and analysis report includes a comparison of data quality indicators before and after cleaning, statistics on rule hit rates, and distribution of abnormal records. The comparison of data quality indicators includes four dimensions: completeness indicators, accuracy indicators, consistency indicators, and timeliness indicators.

[0019] The vector library stores standard vectors of 100,000 to 1 million standard records and their corresponding standard metadata text. During retrieval, the cosine similarity between the record vectors of the suspicious record subset and all standard vectors in the vector library is calculated, and the standard vector with the highest similarity and a similarity value greater than 0.85 is selected as a reference.

[0020] This invention constructs a representative sample extraction mechanism based on data complexity metrics, combines a large language model to generate candidate rules, and forms a rule repository through semantic checking and unit testing. Then, a rule adaptation evaluation model is used to cross-match and evaluate the rule set with the dataset to obtain the optimal rule combination, achieving automated rule generation and dynamic adaptation. This scheme constructs data complexity metrics by quantifying missing rate, number of unique values, field dependency, and semantic relevance, ensuring that representative samples accurately reflect the characteristics of partitioned data. The candidate rules generated by the large language model based on sample features and historical cleaning logs are targeted. The rule adaptation evaluation model, through a comprehensive evaluation of rule coverage, data pass rate, field matching degree, and semantic consistency score, can automatically match the optimal rule combination for datasets with different characteristics, avoiding the subjectivity and limitations of manual configuration. In summary, this invention, through the collaborative mechanism of data complexity quantification, large language model rule generation, and rule adaptation evaluation model, solves the technical problems of low cleaning efficiency and insufficient accuracy in large-scale data cleaning processes involving tens of millions of data points, where rule generation relies on manual configuration and cannot dynamically adapt to the characteristics of multi-source heterogeneous data. Attached Figure Description

[0021] Figure 1 This is a flowchart of the method of the present invention.

[0022] Figure 2 This is a heat map showing the cross-matching popularity of rule-based data.

[0023] Figure 3 This is a distribution diagram of the vector similarity search results for suspicious records.

[0024] Figure 4 This is a graph showing the iterative changes in the node flow pressure drop index. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below.

[0026] like Figure 1 The diagram shown is a flowchart of a method for cleaning tens of millions of data points based on LLM+Spark, provided by this invention. This method includes the following steps:

[0027] S01. Connect structured and unstructured data sources to the system using data synchronization tools, and partition and store the data in a distributed file system according to the data source and timestamp to form partitioned storage data;

[0028] S02. Perform statistical analysis on the partitioned storage data, extract missing rate, number of unique values, field dependency and semantic relevance as data feature set, calculate data complexity index based on the data feature set, and extract 500 to 2000 records from each partition as representative sample set according to the data complexity index.

[0029] S03. Assemble the representative sample set with field names, data sources, timestamps and historical cleaning logs into input text, call the private large language model to generate a candidate rule list, perform semantic checks and unit tests on each rule in the candidate rule list and apply it to the representative sample set to obtain rule coverage, mark rules with rule coverage below 60% as low confidence rules and submit them for manual review, write the approved rules into the rule repository and mark the version number and applicable scope label;

[0030] S04. Divide the rules in the rule warehouse into N rule sets according to the applicable field types, and divide the partitioned storage data into N datasets according to the data feature set. Then, perform a cross-sectional analysis between the N rule sets and the N datasets. The cross-matching process generates a rule data matching report, which is then input into the rule adaptation evaluation model to obtain the optimal rule combination for each dataset.

[0031] S05. The optimal rule combination is broadcast to each execution node via the Spark cluster. Each execution node performs the first stage of cleaning on the allocated data partition. The first stage of cleaning uses format validation rules, regular expression rules, and numerical range rules to filter and correct the data. Records that cannot pass the first stage of cleaning are marked as suspicious record subsets. The second stage of cleaning is performed on the suspicious record subsets. The second stage of cleaning converts the suspicious record subsets into record vectors through an embedding model and retrieves the standard vector with the highest similarity in the vector library to obtain the corresponding standard metadata text. The suspicious record subsets are semantically rewritten and corrected according to the standard metadata text to form the cleaned data.

[0032] S06. Generate an overall processing analysis report for the cleaned data. The overall processing analysis report includes a comparison of data quality indicators before and after cleaning, rule hit rate statistics, and distribution of abnormal records. Compare the overall processing analysis report with the expected rule behavior and submit it for manual review. After the review is approved, mark the corresponding optimal rule combination as the formal application status and trigger the next round of iteration optimization.

[0033] It also includes: applying an inference rhythm shaping algorithm to monitor GPU power fluctuations and voltage drops during the first and second stage cleaning processes; evenly distributing large language model inference requests according to time slices and performing queue shaping on peak requests; applying a partition rebalancing algorithm to monitor the input / output traffic and data exchange traffic of each partition; iteratively migrating hot key values ​​and large partitions to low-load nodes; and applying an anomaly propagation suppression algorithm to identify high-connectivity fields on the field dependency graph and preferentially applying strengthening rule corrections to the high-connectivity fields to suppress the propagation of dirty data to downstream fields.

[0034] The data complexity index is calculated as follows: the missing rate, the ratio of the number of unique values ​​to the total number of records, the field dependency strength, and the semantic relevance score of the data feature set are normalized and then weighted and summed. The weight coefficients are 0.3, 0.2, 0.25, and 0.25, respectively. The normalization process is to divide each index value by the maximum value of the index in all partitions.

[0035] The inference rhythm shaping algorithm monitors the power ripple amplitude and transient voltage drop of the GPU power supply module. When the voltage drop exceeds 5% of the rated voltage, it triggers a request shaping mechanism to rearrange the queue of large language model inference requests to be processed according to time windows, so that the number of requests in each time window remains balanced. The length of the time window is 100 to 200 milliseconds. For requests that exceed the average request rate by more than 30% during peak periods, delay jitter processing is implemented, with a delay time range of 50 to 150 milliseconds.

[0036] The partition rebalancing algorithm treats the input / output flow and data exchange flow of each data partition as the flow process of fluid through a pipe, and regards nodes with excessive load as narrow pipe sections with high fluid resistance. It identifies blockage points by calculating the flow pressure drop index of each node. The flow pressure drop index is the ratio of the current processing time of a node to the average processing time of all nodes. When the flow pressure drop index exceeds 1.5, the top 10% of hot key values ​​with the highest key value frequency and the top 5% of large partitions with the largest data volume on the node are migrated to low-load nodes with a flow pressure drop index of less than 0.8. The migration process is iterated until the flow pressure drop index of all nodes is within the range of 0.8 to 1.2.

[0037] The anomaly propagation suppression algorithm constructs a field dependency graph, where fields in the data table are nodes, and the reference relationships, computational dependencies, and semantic associations between fields are directed edges. The connectivity of each field node is calculated, where connectivity is the sum of the number of incoming edges pointing to the field node and the number of outgoing edges emanating from the field node. The top 20% of field nodes by connectivity are marked as high-connectivity fields. Strong rule correction with a rule strength coefficient of 1.0 is preferentially applied to these high-connectivity fields, while regular rule correction with a rule strength coefficient of 0.6 to 0.8 is applied to the remaining fields. The rule strength coefficient represents the rule correction confidence threshold. Strong rule correction only corrects records with a confidence score higher than 90%, while regular rule correction corrects records with a confidence score higher than 70%. This hierarchical correction strategy prevents outliers in a single field from propagating to multiple downstream fields through field dependencies.

[0038] The structure of the rule adaptation evaluation model is as follows: the input layer receives numerical features from the rule data matching report, including rule coverage, data pass rate, field matching degree, and semantic consistency score (four dimensions in total). After passing through a normalization layer, these features are input into a backbone network containing three fully connected layers. The number of neurons in the three fully connected layers are 128, 64, and 32, respectively, and the activation function is a modified linear unit function. The output of the backbone network is processed through an attention mechanism layer to evaluate the N-class rule sets against the N-class datasets. The rule combinations are weighted and scored, and finally the rule selection probability distribution corresponding to each class of dataset is generated through the output layer. The rule combination with the highest probability is selected as the optimal rule combination for that class of dataset.

[0039] The steps for establishing the training dataset for the rule adaptation evaluation model include: collecting 1,000 to 5,000 sets of rule-data matching cases from historical data cleaning projects. Each set of matching cases includes a rule feature vector, a data feature vector, a matching processing result, and a manually labeled matching quality score. The rule feature vector consists of rule type, rule complexity, applicable field type, and historical success rate. The data feature vector consists of the data complexity index, field type distribution, missing value ratio, and outlier ratio. Matching cases with a matching quality score higher than 80 are labeled as positive samples, and matching cases with a score lower than 60 are labeled as negative samples. The training set and validation set are divided in a 7:3 ratio.

[0040] The training steps of the rule adaptation evaluation model include: using the cross-entropy loss function to measure the difference between the rule selection probability distribution predicted by the rule adaptation evaluation model and the actual rule applicability labels; using the adaptive moment estimation optimization algorithm to adjust the parameters of the rule adaptation evaluation model; setting the learning rate to 0.001; setting the batch size to 32; setting the training epochs to 100; evaluating the accuracy of the rule adaptation evaluation model on the validation set every 10 epochs; stopping training early when the accuracy of the validation set no longer improves for 5 consecutive epochs; and saving the parameters of the rule adaptation evaluation model when the accuracy of the validation set is the highest as the final model.

[0041] In the second stage of cleaning, the embedding model converts text records into 768-dimensional record vector representations. The vector library stores standard vectors of 100,000 to 1,000,000 standard records and their corresponding standard metadata text. During retrieval, the cosine similarity between the record vectors of the suspicious record subset and all standard vectors in the vector library is calculated. The standard vector with the highest similarity and a similarity value greater than 0.85 is selected as a reference. If the similarity of all standard vectors is less than 0.85, the records in the suspicious record subset are marked as items for manual review.

[0042] The data quality indicator comparison in the overall processing and analysis report includes four dimensions: completeness indicator comparison, accuracy indicator comparison, consistency indicator comparison, and timeliness indicator comparison. The completeness indicator comparison is the ratio of the number of non-empty fields to the total number of fields after cleaning to the ratio before cleaning. The accuracy indicator comparison is the ratio of the number of records that pass rule verification to the total number of records to the ratio before cleaning. The consistency indicator comparison is the ratio of the number of records whose field dependencies meet the constraints to the total number of records to the ratio before cleaning. The timeliness indicator comparison is the ratio of the number of records whose timestamp field is within a reasonable range from the current time to the total number of records to the ratio before cleaning.

[0043] The rule hit rate is the ratio of the number of records in the cleaned data where each rule in the rule warehouse is actually applied to the total number of records in the cleaned data. The abnormal record distribution is the distribution of the number of records marked as manual review items in the cleaned data in each data partition.

[0044] The expected rule behavior refers to the data partition range and expected number of correction records corresponding to the preset applicable scope label of each rule in the rule repository.

[0045] Furthermore, the present invention also provides a computer-based system for cleaning tens of millions of data points using LLM and Spark. The computer is equipped with a storage medium that stores program instructions, which execute the above-described method when the computer is run.

[0046] The specific implementation methods of the above steps are described in detail below.

[0047] The specific implementation of step S01 is as follows: First, a connection channel is established with various data sources by configuring a data synchronization tool. The data synchronization tool supports reading structured data from relational databases and unstructured text data from object storage systems. For structured data sources, an incremental synchronization method is used to capture newly added or changed data records at fixed time intervals. For unstructured data sources, a full scan method is used to periodically obtain a file list and compare file modification timestamps to filter files to be synchronized. After the data is accessed, a partition key value is constructed based on the data source identifier and the data generation timestamp. The partition key value format is the data source name with an underscore followed by the timestamp (year, month, day). According to the partition key value, the data is written to different directory paths of the distributed file system to form physically isolated partitioned storage data. The purpose of partitioned storage is to facilitate independent access to the data partitions of each computing node during subsequent parallel processing without generating resource competition. At the same time, the partition structure also provides a natural organizational form for data traceability and version management.

[0048] The specific implementation of step S02 is as follows: First, a statistical analysis task is started for each partition in the partitioned storage data. The statistical analysis task traverses all records in the partition and calculates the proportion of missing values ​​to the total number of records for each field as the missing rate. The number of unique values ​​in each field is counted as the number of unique values. The linear correlation between the numerical changes of the fields is calculated using the Pearson correlation coefficient method as the field dependency strength. The similarity between the vector representations of the text content of the fields is calculated using the cosine similarity method as the semantic relevance score. The missing rate, the number of unique values, the field dependency strength, and the semantic relevance score are combined into a data feature set. Based on the data feature set... The data complexity index is used to quantify the difficulty of data cleaning. A higher data complexity index indicates more serious data quality problems and requires more refined cleaning rules. The number of samples to be extracted is determined based on the data complexity index: 500 records are extracted when the data complexity index is below 0.3, 1000 to 1500 records are extracted when the data complexity index is between 0.3 and 0.7, and 2000 records are extracted when the data complexity index is above 0.7. The sample extraction adopts a stratified random sampling method to ensure that the distribution of samples on various data features is consistent with the overall data. The representative sample set after extraction is used to reduce the amount of computation and maintain the representativeness of the rules during subsequent rule generation.

[0049] The specific implementation of step S03 is as follows: First, the data records in the representative sample set are organized in tabular form, and the corresponding field names, data source identifiers, timestamps, and historical cleaning log texts are collected. The data records and metadata are concatenated into structured input text according to a predetermined template. The input text contains clear field descriptions and data quality problem descriptions to guide the large language model to understand the cleaning requirements. The input text is submitted to the privately deployed large language model interface. The large language model analyzes the data patterns and quality problems in the input text based on its pre-trained language understanding and context learning capabilities and generates targeted cleaning rule suggestions. The candidate rule list is returned in structured text form, containing rule names, rules, and other information. The rules are described in terms of type, applicable fields, and rule logic. Semantic checks are performed on each rule in the candidate rule list to verify the completeness and consistency of the rule logic. Unit test cases are constructed to verify the correctness of the rules. The rule coverage rate is the percentage of records where the rule correction is applied to the representative sample set, based on the number of records corrected. If the rule coverage rate is below 60%, the rule is considered to have a narrow scope of application or inaccurate logic and is marked as a low-confidence rule requiring human expert review and correction. When a rule passes the review and is written to the rule repository, a version number is automatically generated, recording the rule's creation time and modification history. Simultaneously, based on the rule's performance on the sample set, an applicable scope label is added to indicate the applicable data source type and field type.

[0050] The specific implementation of step S04 is as follows: First, all approved rules are read from the rule repository. The rules are then grouped according to the field type information in the rule's scope tag. The field types include four basic types: numeric, text, date, and enumeration, as well as combinations of each type. The rules are divided into N rule sets according to their field types. Simultaneously, the partitioned storage data is clustered into N data sets based on the field type distribution, missing rate range, and semantic relevance score range in the data feature set. Cartesian product matching is then performed between the N rule sets and the N data sets to generate... A rule is paired with data, and a small-batch preprocessing task is initiated for each pair. The preprocessing task applies the rule to the first 1000 records of the corresponding dataset and statistically analyzes the processing results. These results include rule coverage (the proportion of records successfully matched by the rule out of 1000 records), data pass rate (the proportion of records whose data quality indicators improved after processing out of the total number of matched records), field matching degree (the proportion of fields involved in the rule present in the data), and semantic consistency score (the degree to which the corrected field values ​​conform to the semantics of the fields). The processing results of each pairing combination are summarized into a rule data matching report. The rule data matching report is input into a pre-trained rule adaptation evaluation model. The rule adaptation evaluation model learns the adaptation pattern between rules and data based on a deep neural network structure. The model outputs the rule selection probability distribution corresponding to each data set and selects the rule set with the highest probability as the optimal rule combination for the data set. The process of determining the optimal rule combination utilizes the model's ability to learn from historical cleaning cases to avoid the computational complexity of exhaustive search.

[0051] The specific implementation of step S05 is as follows: First, the optimal rule combination is serialized into binary data on the master node of the Spark cluster and distributed to the memory of all execution nodes through a broadcast mechanism. The broadcast mechanism avoids the duplicate overhead of network transmission and ensures that each execution node receives the rule data only once. After receiving the rules, each execution node starts the first-stage cleaning task. The first-stage cleaning task reads the allocated data partition and applies the format validation rules in the optimal rule combination to each record to verify whether the field value conforms to the predetermined format, applies regular expression rules to match and replace strings that do not conform to the pattern, and applies numerical range rules to filter values ​​that exceed the reasonable range. Records that cannot be corrected by the format validation rules, regular expression rules, and numerical range rules are marked as a suspicious record subset. The second-stage cleaning task is started for the suspicious record subset. The second-stage cleaning task calls the embedding model to convert the text fields of the suspicious records into 768-dimensional numerical vectors. In the pre-built vector library, the vector similarity retrieval algorithm is used to find the standard vector most similar to the numerical vector. When the highest similarity exceeds 0.85, it is considered that a reference standard record has been found, and the standard metadata corresponding to the standard vector is read. Based on the standard metadata text, rewritten prompt words are constructed and a large language model is invoked to perform semantic-level correction on suspicious records. During the parallel execution of the first and second stage cleaning processes, an inference rhythm shaping algorithm is initiated. This algorithm collects voltage waveform data of the GPU power supply module in real time through a hardware monitoring interface. When a voltage drop exceeding 5% of the rated voltage is detected, a request reordering mechanism is triggered. The backlogged inference request queue is divided according to time windows and the requests are evenly distributed within each time window to avoid instantaneous power surges. The time window length is set to 100 to 200 milliseconds in response. A balance is struck between latency and power smoothing. Simultaneously, a partition rebalancing algorithm is initiated. This algorithm, drawing inspiration from the pressure balance principle in fluid mechanics, treats the data processing load as fluid flow. It calculates the flow drop index by continuously monitoring the task waiting time of each execution node. When the flow drop index exceeds 1.5, it is identified as a flow bottleneck. Hotspot keys and large partitions at the bottleneck are transferred to low-load nodes with a flow drop index below 0.8. This transfer operation is implemented through Spark's dynamic partition reallocation mechanism. The transfer operation is iteratively executed until the flow drop index of all nodes converges to between 0.8 and 1.In the equilibrium interval of 2, an anomaly propagation suppression algorithm is launched in parallel. This algorithm first scans the metadata of the data table to construct a field dependency graph. The nodes in the field dependency graph are field names, and directed edges represent the dependencies between fields. The sum of the in-degree and out-degree of each node is calculated as the connectivity index. The top 20% of nodes by connectivity are marked as high-connectivity fields. Strong rule corrections with a confidence threshold of 90% are preferentially applied to these high-connectivity fields, while conventional rule corrections with a confidence threshold of 70% are applied to the remaining fields. This tiered correction strategy references the critical node blocking principle in infectious disease control, cutting off the propagation path of outliers by prioritizing high-connectivity fields to prevent single-point anomalies from spreading to multiple downstream fields and forming cascading errors.

[0052] The specific implementation of step S06 is as follows: First, collect statistical information of the cleaned data, calculate the comparison of completeness indicators, accuracy indicators, consistency indicators, and timeliness indicators before and after cleaning. The completeness indicator comparison quantifies the missing value filling effect by statistically analyzing the change in the proportion of non-empty fields. The accuracy indicator comparison quantifies the error correction effect by statistically analyzing the change in the proportion of records that pass rule verification. The consistency indicator comparison quantifies the logical consistency improvement effect by statistically analyzing the change in the proportion of records that satisfy field dependency constraints. The timeliness indicator comparison quantifies the timeliness improvement effect by statistically analyzing the change in the proportion of records with timestamps within a reasonable range. At the same time, calculate the rule hit rate by statistically analyzing the actual application frequency of each rule in the rule warehouse in the cleaned data, and statistically analyze the distribution of the number of records marked as manual review items in each partition to form an abnormal record distribution map. The data quality indicators, rule hit rate statistics, and abnormal record distribution are summarized into an overall processing analysis report. This report is then compared with the expected rule behaviors recorded in the rule repository. The expected rule behaviors include the pre-defined applicable data partition range and the expected range of corrected records. When the deviation between the actual application and the expected rule behaviors exceeds 20%, an anomaly alarm is triggered, along with a detailed deviation analysis. The overall processing analysis report is then submitted to a manual review process. Review experts assess the cleaning quality based on the report content and decide whether to approve the optimal rule combination for formal application. After approval, the system updates the version number of the optimal rule combination and marks it as available for the production environment. Simultaneously, the next iteration optimization process is triggered. This next iteration extracts samples from the newly added data partitions and repeats steps S02 to S06 to continuously optimize the rule repository.

[0053] It should be noted that one of the key technical ideas of this invention is an automatic semantic-level rule generation mechanism based on a large language model. Traditional data cleaning relies on manually written rules, which can only cover limited data patterns and cannot understand the semantic relationships between fields. This invention utilizes the context learning and language understanding capabilities of a large language model to automatically summarize high-level rules containing semantic constraints from sample data. These semantic-level rules can identify the rationality of field values, logical consistency between fields, and cross-field dependencies. Compared with traditional rules, they have stronger generalization and adaptability, reducing rule maintenance costs and improving cleaning quality. The second key technical idea is a distributed task dynamic balancing algorithm based on a flow drop model. Traditional distributed computing systems using static partitioning allocation strategies are prone to data skew, leading to some nodes being overloaded while others are idle. This invention draws on fluid mechanics principles, treating task load as fluid flow and node processing capacity as pipe cross-sections. By continuously monitoring flow drop indicators and dynamically migrating hot data, it achieves adaptive load balancing. This dynamic balancing mechanism effectively eliminates long-tail tasks and improves overall processing throughput. The third key technical approach is an anomaly propagation suppression algorithm based on graph propagation theory. Traditional data cleaning processes data field by field independently, neglecting the dependencies between fields. This allows outliers in a single field to propagate through dependency chains, contaminating multiple downstream fields. This invention constructs a field dependency graph and identifies key field nodes with high connectivity. By prioritizing the application of strong rules at these key nodes, it cuts off the anomaly propagation path. This hierarchical correction strategy references the blocking mechanism in infectious disease control, preventing the cascading spread of anomalies with minimal correction cost and ensuring overall data consistency. The synergistic effect of these three technical approaches forms a closed-loop optimization system from rule generation and task scheduling to anomaly control. Semantic rules generated by a large language model provide intelligent cleaning capabilities, the dynamic balancing algorithm ensures efficient rule execution on large-scale data, and the anomaly propagation suppression algorithm ensures global consistency of the cleaning results. These three elements support each other to achieve high-quality automated cleaning of tens of millions of data points, significantly improving cleaning accuracy, processing speed, and automation compared to traditional methods.

[0054] It should be noted that this invention also solves the following technical problems: In the process of cleaning tens of millions of data points, existing technologies suffer from uneven load distribution across computing nodes and the propagation of abnormal data along field dependencies. To address the load imbalance problem, this invention uses a partition rebalancing algorithm to treat the input / output flow and data exchange flow of data partitions as a fluid flow process. It calculates the flow pressure drop index of each node to identify bottlenecks, migrates hot key values ​​and large partitions to low-load nodes, and iteratively executes the migration process until the flow pressure drop index of all nodes is balanced, achieving dynamic optimization of computing resources. To address the anomaly propagation problem, this invention constructs a field dependency graph and calculates field connectivity using an anomaly propagation suppression algorithm. For fields with high connectivity, strong rule correction is applied preferentially, correcting only high-confidence records. For other fields, conventional rule correction is applied. This hierarchical correction strategy prevents outliers from propagating to multiple downstream fields through field dependencies, avoiding cascading pollution caused by single-point anomalies.

[0055] Furthermore, this invention addresses the technical problem of system instability caused by GPU power fluctuations during large language model inference. In existing technologies, when large-scale concurrent calls to large language models are made for inference, the GPU power supply module experiences power ripple and transient voltage drops, affecting inference stability and accuracy. This invention monitors the voltage drop amplitude of the GPU power supply module using an inference rhythm shaping algorithm. When the voltage drop exceeds 5% of the rated voltage, a request shaping mechanism is triggered, rearranging the large language model inference request queue according to time windows to maintain a balanced number of requests within each time window. Requests exceeding the average request rate by more than 30% during peak periods are subject to delay jittering. By evenly distributing time slices and shaping the queue, the power fluctuation curve is smoothed, ensuring GPU power supply stability and the reliability of the inference process.

[0056] Specifically, the principle of this invention is as follows: The invention solves this technical problem by establishing a complete technical chain from data feature quantification to intelligent rule generation and dynamic rule adaptation. First, by calculating the data complexity index, the missing rate, number of unique values, field dependencies, and semantic relevance of the partitioned data are normalized and weighted to obtain a comparable quantified value of data complexity. Representative samples extracted based on this value accurately reflect the overall characteristics of the partition. Second, the semantic understanding capabilities of a large language model and the empirical knowledge from historical cleaning logs are used to generate targeted candidate rules. Semantic checks and unit tests ensure rule quality, and a rule coverage evaluation mechanism ensures rule effectiveness. Finally, a rule adaptation evaluation model constructs a multi-dimensional matching relationship between the rule set and the dataset. A neural network is used to comprehensively evaluate rule coverage, data pass rate, field matching degree, and semantic consistency scores. An attention mechanism is used to weight and score various combinations, automatically selecting the rule combination with the highest probability as the optimal solution. This achieves dynamic adaptation between rules and data features, avoiding the subjectivity of manual configuration and the limitations of fixed rule sets, thereby improving cleaning efficiency and accuracy.

[0057] The following provides a specific embodiment 1 of the present invention. The specific implementation of step S01 in this embodiment 1 is the same as that of the data access and partition storage process, and will not be described in detail here. The specific implementation of other steps is described in detail below.

[0058] The specific implementation of step S02 involves performing statistical analysis on the partitioned storage data and extracting representative samples. The formula for calculating the data complexity index is expressed as follows:

[0059] ;

[0060] In the formula, A dimensionless metric for data complexity; This represents the missing rate of the current partition, with a value ranging from 0 to 1. This represents the maximum missing rate across all partitions. The number of unique values ​​in the current partition; This represents the total number of records in the current partition. This represents the maximum ratio of the number of unique values ​​to the total number of records across all partitions. This represents the field dependency strength, with a value ranging from 0 to 1. This represents the maximum value of field dependency strength across all partitions. The semantic relevance score ranges from 0 to 1. This represents the maximum semantic relevance score across all partitions. The missing rate is also included. The number of unique values ​​is obtained by comparing the ratio of the number of null values ​​in all fields within a partition to the total number of fields. The field dependency strength is obtained by summing the number of unique values ​​for each field within the partition; The semantic relevance score is calculated by analyzing foreign key relationships and functional dependencies between fields, with an empirical value typically ranging from 0.3 to 0.7; This is obtained by calculating the word vector similarity of field names and the topic consistency of field content, with a default value of 0.5 to 0.8. The data complexity index is based on this calculation. The number of samples drawn from each partition The calculation is as follows:

[0061] ;

[0062] In the formula, The number of sample records extracted, in units of records; The base sample size is 500. The sample size range is adjusted to 1500 samples. This is a dimensionless indicator of data complexity. A stratified random sampling method is used to ensure that the sample covers all field types and numerical distribution ranges.

[0063] The specific implementation of step S03 involves generating candidate rules and reviewing them. A representative sample set, along with field names, data sources, timestamps, and historical cleansing logs, is assembled and input into a private large language model. The coverage of each rule in the candidate rule list generated by the model is then calculated. The calculation formula is as follows:

[0064] ;

[0065] In the formula, The rule coverage is dimensionless. The number of records in the representative sample set that conform to this rule, expressed in records; This represents the total number of records in the representative sample set, expressed in records. If the rule is deemed unreliable, it will be marked as a low-confidence rule and submitted for manual review. Approved rules will be written to the rule repository and labeled with their version number and scope tag.

[0066] The specific implementation of step S04 involves cross-matching rules and data. Rules in the rule repository are categorized according to the applicable field types. Class rule sets, and also divide partitioned storage data into data feature sets. Class dataset, for Class rule sets and Class datasets The second cross-matching process generates a rule data matching report, in which... The number of categories in the rule set and dataset is determined based on field types and data characteristics, typically ranging from 5 to 15. The input layer of the rule adaptation evaluation model receives numerical features from four dimensions: rule coverage, data pass rate, field matching degree, and semantic consistency score. After normalization, these features are fed into a backbone network containing three fully connected layers. The backbone network output is then processed through an attention mechanism layer... The rule combination is weighted and scored, and the output layer generates the rule selection probability distribution for each dataset class. The rule combination with the highest probability is selected as the optimal rule combination for that dataset class.

[0067] Step S05 is specifically implemented by performing data cleaning through a Spark cluster. Each execution node performs the first stage of cleaning on its assigned data partition, filtering and correcting the data using format validation rules, regular expression rules, and numerical range rules. For the subset of suspicious records that fail the first stage of cleaning, a second stage of cleaning is performed. The suspicious records are converted into 768-dimensional record vectors using an embedding model, and the standard vector with the highest similarity is retrieved from the vector library. Cosine similarity is then used. The calculation formula is as follows:

[0068] ;

[0069] In the formula, The cosine similarity score ranges from 0 to 1. The first of the suspicious record vectors Dimensional components; The first standard vector Dimensional components; This is the vector dimension index, with values ​​ranging from 1 to 768. The index with the highest similarity is selected. Using the standard vector as a reference, suspicious records are semantically rewritten and corrected based on the corresponding standard metadata text. During the cleaning process, an inference rhythm shaping algorithm is applied. When the GPU voltage drops by more than 5% of the rated voltage, a request shaping mechanism is triggered, rearranging the large language model inference request queue according to a time window of 100 to 200 milliseconds. Requests exceeding the average request rate by more than 30% during peak periods are subject to latency jitter processing, with a latency range of 50 to 150 milliseconds. A partition rebalancing algorithm is applied, with traffic drop metrics... The calculation formula is as follows:

[0070] ;

[0071] In the formula, This is a flow rate pressure drop indicator, dimensionless; The current processing time of the node, in seconds; This represents the average processing time across all nodes, in seconds. When... The most frequent 10% of key-value pairs and the largest 5% of partitions with the largest data volume on the node will be migrated to [location missing]. For low-load nodes, iteratively execute the migration process until all nodes are migrated. All values ​​are within the range of 0.8 to 1.2. An anomaly propagation suppression algorithm is applied to construct a field dependency graph, and the connectivity of each field node is calculated. :

[0072] ;

[0073] In the formula, The connectivity of the field nodes, expressed in rows; This represents the number of incoming edges pointing to the node in this field, in units of edges. This represents the number of outgoing edges emanating from this field node, expressed in terms of edges. The top 20% of field nodes by connectivity are marked as high-connectivity fields. Strong rule corrections with a rule strength coefficient of 1.0 are applied to high-connectivity fields, while regular rule corrections with rule strength coefficients ranging from 0.6 to 0.8 are applied to the remaining fields. Strong rule corrections only apply to records with a confidence level higher than 90%, while regular rule corrections apply to records with a confidence level higher than 70%.

[0074] The specific implementation of step S06 is to generate an overall processing analysis report. Integrity indicators. The calculation formula is as follows:

[0075] ;

[0076] In the formula, It is a completeness indicator and is dimensionless. This represents the number of non-empty fields, expressed in units of [number]. Total number of fields, in units of one. Accuracy metrics. The calculation formula is as follows:

[0077] ;

[0078] In the formula, As an accuracy indicator, it is dimensionless; The number of records that passed the rule verification, expressed in records; Total number of records, in rows. Consistency index. The calculation formula is as follows:

[0079] ;

[0080] In the formula, It is a consistency indicator and is dimensionless. The number of records in which the field dependencies satisfy the constraints; The total number of records, in units of entries, is the same as the accuracy metric. They have the same meaning. Timeliness indicator. The calculation formula is as follows:

[0081] ;

[0082] In the formula, It is a time-sensitive indicator and is dimensionless. The number of records whose timestamp field and the current time difference are within a reasonable range, in units of records; The total number of records, in units of entries, is the same as the accuracy metric. The meaning is the same. Rule hit rate The calculation formula is as follows:

[0083] ;

[0084] In the formula, The rule hit rate is dimensionless. The number of records in which the rule is actually applied, expressed in records; The total number of records, in units of entries, is the same as the accuracy metric. The meaning is the same. The overall processing analysis report is compared with the expected rule behavior and submitted for manual review. After the review is approved, the corresponding optimal rule combination is marked as the formal application status and the next round of iteration optimization is triggered.

[0085] It should be noted that the variables involved in this invention are explained in detail in Table 1.

[0086] Table 1. Variable Explanation Table

[0087]

[0088] To better understand and implement this invention, the following is a specific application scenario example 2: A production line is equipped with 156 sensors and detection devices, generating approximately 780,000 structured quality inspection data entries and 160,000 unstructured production log data entries daily, covering multiple dimensions such as dimensional measurement, surface quality inspection, assembly parameter recording, environmental monitoring, and equipment operating status. Due to the heterogeneous data sources, varying collection frequencies, and the presence of numerous missing and outlier values, traditional rule-based cleaning methods struggle to address the complex data quality issues, severely impacting the accuracy of subsequent quality analysis and prediction models. The technical team decided to adopt a data cleaning method based on LLM and Spark to clean 30 days of historical data. The records are systematically cleaned.

[0089] The technical team first used the Apache NiFi data synchronization tool to integrate structured data from the MES system and unstructured log data from the PLC controller into the Hadoop Distributed File System. The structured data included multiple fields such as product number, inner diameter, outer diameter, surface roughness, ambient temperature, equipment vibration amplitude, and timestamp; the unstructured data consisted of JSON-formatted equipment status logs and operation records. The system categorized data into four sources: MES data source, PLC data source, visual inspection data source, and environmental monitoring data source, and partitioned the data with timestamps every two hours. This resulted in 360 data partitions on HDFS, with each partition containing an average of 78,300 records and a storage capacity of 142GB.

[0090] After statistical analysis of the partitioned storage data, the technical team extracted key data features. Significant differences were found in the missing rates of different fields; the missing rate for the ambient temperature field was 0.034, while the missing rate for the equipment vibration amplitude field reached 0.287. The ratio of unique values ​​to the total number of records reflects the dispersion of a field; for the product number field, this ratio was 0.962, indicating that almost every record corresponds to a unique product. Field dependencies were quantified by analyzing the strength of association rules between fields; for example, the dependency strength between inner diameter and outer diameter was 0.845. Semantic relevance was obtained by calculating the cosine similarity between the field description text using a word embedding model; the semantic relevance score between the detection result field and the defect type field was 0.912.

[0091] When calculating data complexity metrics based on the data feature set, the missing rate, unique value ratio, dependency strength, and semantic relevance were normalized separately. The normalization method involved dividing each metric value by its maximum value across all partitions, followed by a weighted sum using weighting coefficients of 0.3, 0.2, 0.25, and 0.25. The results showed that partition 126 had a data complexity metric of 0.782, classifying it as a high-complexity partition; partition 243 had a data complexity metric of 0.341, classifying it as a low-complexity partition. Based on these metrics, the technical team extracted 1850 records from the high-complexity partition, 1200 records from the medium-complexity partition, and 650 records from the low-complexity partition, forming a representative sample set containing a total of 423,000 records.

[0092] The technical team assembled a representative sample set with field names, data source identifiers, timestamp information, and historical cleaning logs into input text, and then invoked a private large language model deployed on a local server cluster. This model, based on an open-source architecture with 7 billion parameters, was fine-tuned and trained using 3,000 industrial data cleaning cases. After receiving the input, the model generated a list of 427 candidate rules, covering four main categories: format validation rules, numerical range rules, regular expression rules, and semantic consistency rules. For each rule, the system validated its executability using a syntax parser and executed it on a representative sample set using a unit testing framework. Rule coverage statistics showed that 185 rules had a coverage rate exceeding 60% and were directly added to the rule repository; 242 rules had a coverage rate between 35% and 60% and were marked as low-confidence rules and submitted for manual review. The technical team manually evaluated these 242 low-confidence rules, and ultimately 163 rules passed the review and were added to the rule repository, bringing the total number of rules in the repository to 348. Each rule is labeled with a version number from v1.0 to v1.8 and an applicable scope label, such as "applies to size fields" or "applies to temperature sensor data".

[0093] like Figure 2 As shown, the technical team categorized the 348 rules in the rule repository into eight rule sets based on the applicable field types: size, temperature, vibration, surface quality, timestamp, identifier, status code, and mixed. Simultaneously, the 360 ​​data partitions were divided into eight datasets based on the principal component analysis results of the data feature sets. The system then performed [the necessary steps] on these eight rule sets and eight datasets. This involves 64 cross-matching processes, with each match calculating four metrics: rule coverage, data pass rate, field matching degree, and semantic consistency score. For example, when the size-based rule set is applied to a size-based dataset, the rule coverage is 0.892, the data pass rate is 0.834, the field matching degree is 0.956, and the semantic consistency score is 0.887; however, when the size-based rule set is applied to a temperature-based dataset, the rule coverage is only 0.147, and the data pass rate is 0.089. These 64 matching results constitute a rule data matching report, which is input to the rule adaptation evaluation model. The model's input layer receives 4-dimensional numerical features, which, after passing through a normalization layer, enter a 3-layer fully connected network containing 128, 64, and 32 neurons, respectively, with the ReLU function as the activation function. The backbone network output uses an attention mechanism layer to weight and score the 64 combinations, and the output layer generates the rule selection probability distribution for each dataset class. The model prediction results show that the optimal rule combination for the size-type dataset is the joint application of the size-type rule set and the mixed-type rule set, with a selection probability of 0.912; the optimal rule combination for the temperature-type dataset is the temperature-type rule set, with a selection probability of 0.867.

[0094] The technical team used the Spark cluster's broadcast mechanism to distribute the optimal rule combinations for each of the eight dataset categories to 16 execution nodes. Each execution node is configured with two 16-core processors, 128GB of memory, and one V100 GPU. After receiving the allocated data partitions, the nodes first perform the first stage of cleaning. The first stage of cleaning uses format validation rules to verify field formats, such as checking whether product numbers conform to a 12-digit alphanumeric format; using regular expression rules to match timestamp formats; and using numerical range rules to verify whether dimensional parameters are within the drawing tolerances. After the first stage of cleaning, 26.34 million records passed the validation, while 1.86 million records were marked as a suspicious subset due to format errors or out-of-bounds values.

[0095] During the second-stage cleaning of the suspicious record subset, the system converts each text record into a 768-dimensional record vector using an embedding model based on the Transformer architecture. The vector library pre-stores vector representations of 520,000 standard records and their corresponding standard metadata text; these standard records come from the detection data of historical qualified products. The system calculates the cosine similarity between the suspicious record vector and all standard vectors in the vector library, selecting the standard vector with the highest similarity as a reference. For example... Figure 3 As shown, when the similarity value is greater than 0.85, the system semantically rewrites and corrects suspicious records based on the standard metadata text. For example, if a record's surface roughness field value is "Ra3.2um", and the system finds a standard record with a similarity of 0.923, the correct format of the standard metadata text should be "3.2". The unit is uniformly specified in the field definition, and the system automatically corrects the field to "3.2". For records with a similarity value lower than 0.85, the system marks them as items requiring manual review; there are a total of 138,000 such records. After the second stage of cleaning, 1.722 million suspicious records were corrected and merged with the 26.34 million records that passed the first stage, forming a cleaned dataset containing 28.062 million records.

[0096] During the two-stage cleaning process, the technical team applied three optimization algorithms to ensure system stability and efficiency. The inference rhythm shaping algorithm monitors the power ripple and transient voltage drops of the GPU power supply module, triggering a shaping mechanism when the voltage drop exceeds 5% of the rated voltage. System records show that during peak processing periods, the GPU voltage dropped from the standard 1.2V to 1.14V, a drop of 5%, triggering request queue reordering. The system reduced the number of inference requests within a 150-millisecond time window from a peak of 187 to an average of 142. For the 45 requests exceeding the average request rate by 30%, latency jitter was implemented, with latency randomly distributed between 50 and 150 milliseconds, effectively preventing system instability caused by GPU power spikes.

[0097] The partition rebalancing algorithm treats data processing as a fluid flow process in a pipe. For example... Figure 4 As shown, the algorithm calculates the traffic drop index for each node, which is the ratio of the current processing time of a node to the average processing time of all nodes. Initially, node 7 took 1568 seconds to process a single partition, while the global average processing time was 982 seconds, resulting in a traffic drop index of 1.60, exceeding the threshold of 1.5. The system identified the key value "Process Code: OP230" as the most frequent value on this node, accounting for 12.3%, making it a hot key value. Simultaneously, the partition with partition ID P0187 had a data volume of 9.2GB, classifying it as a large partition. The algorithm migrated these hot key values ​​and large partitions to node 14, where the traffic drop index was 0.74, lower than 0.8. After three rounds of iterative migration, the traffic drop index for all nodes stabilized within the range of 0.83 to 1.17, and the overall processing time decreased from 2340 seconds to 1580 seconds. Figure 4 In the diagram, the gray broken line represents the changes in indicators at other nodes.

[0098] The anomaly propagation suppression algorithm constructed a field dependency graph containing 67 field nodes and 214 directed edges. The algorithm calculated the connectivity of each field. The inner diameter field had 8 incoming edges and 11 outgoing edges, resulting in a connectivity of 19, ranking first. The outer diameter field had a connectivity of 17, ranking second. The assembly clearance field had a connectivity of 15, ranking third. The top 20% of fields by connectivity (14 fields) were marked as high-connectivity fields. Strong rule correction with a rule strength coefficient of 1.0 was applied to these fields, correcting only records with a confidence level higher than 90%. Regular rule correction with a rule strength coefficient of 0.7 was applied to the remaining 53 fields, correcting records with a confidence level higher than 70%. This hierarchical correction strategy prevents outliers in the inner diameter field from propagating uncontrollably to 11 downstream fields such as assembly clearance, bearing clearance, and load-bearing capacity through dependencies, effectively controlling the scope of data pollution.

[0099] The technical team generated an overall processing and analysis report for the cleaned data. The report showed that the integrity index improved from 0.783 to 0.947, the accuracy index from 0.692 to 0.923, the consistency index from 0.728 to 0.906, and the timeliness index from 0.856 to 0.981. Rule hit rate statistics indicated that the average hit rate for format validation rules was 0.874, for numerical range rules it was 0.692, for regular expression rules it was 0.745, and for semantic consistency rules it was 0.623. Anomaly record distribution showed that the 138,000 records marked for manual review were mainly concentrated in partitions 126 to 138, corresponding to anomaly data generated during equipment maintenance. The technical team compared the overall processing and analysis report with the expected rule behaviors preset in the rule repository, finding that the deviation between the actual number of corrected records and the expected range was within 5%, and the matching degree between the rule application scope and the actual application partitions reached 92%. Once approved, the system marks the corresponding 8 optimal rule combinations as officially applied and automatically triggers the next round of iterative optimization process to prepare for real-time cleaning of the newly added daily production data.

[0100] The advancements of this invention compared to traditional methods are mainly reflected in three aspects. Traditional rule-based cleaning methods rely on manually formulated fixed rules, which are difficult to handle dynamic changes in data characteristics. This invention, however, automatically generates rules through a large language model and combines it with a rule adaptation evaluation model to achieve intelligent matching between rules and data, dynamically adjusting the cleaning strategy based on data complexity and field characteristics. Traditional methods apply a uniform cleaning intensity to all fields, leading to insufficient correction of key fields or excessive correction of minor fields. This invention identifies high-connectivity fields through an anomaly propagation suppression algorithm and implements a tiered correction strategy, blocking the spread of dirty data from a topological perspective. Traditional Spark jobs are prone to performance degradation due to uneven node load and GPU power fluctuations when processing large-scale data. This invention introduces an inference rhythm shaping algorithm and a partition rebalancing algorithm, providing dual protection for system stability from both hardware and scheduling levels, enabling stable and reliable completion of cleaning data of tens of millions of records in engineering practice.

[0101] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for cleaning tens of millions of data points based on a large language model and Spark, characterized in that, Includes the following steps: Structured and unstructured data sources are connected to the system through data synchronization tools, and the data is partitioned and stored in a distributed file system according to the data source and timestamp, forming partitioned storage data; Statistical analysis is performed on the partitioned storage data to extract missing rate, number of unique values, field dependency and semantic relevance as data feature set. Data complexity index is calculated based on the data feature set, and records are extracted from each partition as representative sample set according to the data complexity index. The representative sample set is assembled with field names, data sources, timestamps, and historical cleaning logs into input text. A private large language model is called to generate a candidate rule list. Each rule in the candidate rule list is subjected to semantic checks and unit tests and applied to the representative sample set to obtain rule coverage. Rules with rule coverage below the coverage threshold are marked as low-confidence rules and submitted for manual review. Rules that pass the review are written into the rule repository and labeled with version number and scope of application. The rules in the rule warehouse are divided into N rule sets according to the applicable field types, and the partitioned storage data is divided into N datasets according to the data feature set. The N rule sets and the N datasets are then compared. The cross-matching process generates a rule data matching report, which is then input into the rule adaptation evaluation model to obtain the optimal rule combination for each dataset. The optimal rule combination is broadcast to each execution node via Spark cluster. Each execution node performs the first stage of cleaning on the allocated data partition. The first stage of cleaning uses format validation rules, regular expression rules, and numerical range rules to filter and correct the data. Records that cannot pass the first stage of cleaning are marked as suspicious record subsets. The second stage of cleaning is performed on the suspicious record subsets. The second stage of cleaning converts the suspicious record subsets into record vectors through an embedding model and retrieves the standard metadata text corresponding to the highest similarity standard vector in the vector library. The suspicious record subsets are semantically rewritten and corrected according to the standard metadata text to form the cleaned data. An overall processing analysis report is generated for the cleaned data. The overall processing analysis report includes a comparison of data quality indicators before and after cleaning, rule hit rate statistics, and distribution of abnormal records. The overall processing analysis report is compared with the expected rule behavior and submitted for manual review. After the review is approved, the corresponding optimal rule combination is marked as the formal application status and the next round of iteration optimization is triggered.

2. The method according to claim 1, characterized in that, The data complexity index is calculated using normalized weighted summation. The missing rate, the ratio of the number of unique values ​​to the total number of records, the field dependency strength, and the semantic relevance score in the data feature set are normalized and then weighted and summed with weight coefficients of 0.3, 0.2, 0.25, and 0.

25.

3. The method according to claim 2, characterized in that, The representative sample set consists of 500 to 2000 records per partition.

4. The method according to claim 3, characterized in that, The coverage threshold is 60%.

5. The method according to claim 4, characterized in that, The input layer of the rule adaptation evaluation model receives numerical features from four dimensions in the rule data matching report: rule coverage, data pass rate, field matching degree, and semantic consistency score.

6. The method according to claim 5, characterized in that, The rule-fit evaluation model consists of three fully connected layers with 128, 64, and 32 neurons respectively. The activation function is a modified linear unit function. The backbone network output is processed by an attention mechanism layer to evaluate the N-class rule set against the N-class dataset. Weighted scoring is applied to the various combinations.

7. The method according to claim 6, characterized in that, The establishment of the training dataset for the rule-fit evaluation model includes collecting 1,000 to 5,000 sets of rule-data matching cases from historical data cleaning projects. Matching cases with a matching quality score higher than 80 are labeled as positive samples, and matching cases with a score lower than 60 are labeled as negative samples. The training set and validation set are divided in a 7:3 ratio.

8. The method according to claim 7, characterized in that, The rule-fit evaluation model training uses the cross-entropy loss function and the adaptive moment estimation optimization algorithm is used to adjust the model parameters. The learning rate is set to 0.001, the batch size is set to 32, and the training epochs are set to 100.

9. The method according to claim 8, characterized in that, The first stage of cleaning uses format validation rules, regular expression rules, and numerical range rules to filter and correct the data, marking records that cannot pass the first stage of cleaning as a subset of suspicious records.

10. The method according to claim 9, characterized in that, The second stage of cleaning involves converting the suspicious record subset into 768-dimensional record vectors using an embedding model. The standard vector with the highest similarity is retrieved from the vector library to obtain the corresponding standard metadata text. The suspicious record subset is then semantically rewritten and corrected based on the standard metadata text.

Images