An information processing method based on big data

Abstract

The application provides an information processing method based on big data, comprising: obtaining data blood relationship information including metadata of data sources, processing, conversion and other links according to a data quality traceability chain; determining the input and output data sets of each data processing link through metadata analysis to form a data flow direction diagram; backtracking the data flow direction diagram according to quality problem data to locate the upstream link that introduces the quality problem; obtaining the detailed processing logic and parameter configuration of the quality problem link to analyze the specific reasons for the quality problem; formulating a scheme of data repair and processing flow optimization for the quality problem reasons to repair the problem data; updating the optimized processing logic and parameter configuration to the corresponding link and re-executing the data processing flow; and performing quality inspection on the repaired data to ensure that the problem has been solved and the data quality meets the requirements.

Description

Technical Field

[0001] This invention relates to the field of information technology, and in particular to an information processing method based on big data. Background Technology

[0002] In big data information processing platforms, the aggregation and processing of massive amounts of heterogeneous data is intricate, involving numerous stages such as data collection, cleaning, storage, computation, and visualization. Due to the wide range of data sources, diverse data formats, and complex processing logic, data quality issues are highly likely to occur during the data flow process, such as missing data, data duplication, and data inconsistency. When this defective data is applied to business analysis or decision-making, it can be misleading and lead to business risks.

[0003] Therefore, big data platforms urgently need to build a data quality traceability system, but they face the following technical challenges: First, end-to-end data lineage tracing involves massive data processing nodes, each with different data transformation logic, leading to performance bottlenecks in the extraction and association of lineage information across the entire chain. Second, after complex serial and parallel processing, multi-source heterogeneous data undergoes significant changes in its data model, making it difficult to accurately map the correspondence between data items and affecting the accuracy of lineage analysis. Furthermore, the frequent job scheduling on big data platforms, along with ad-hoc queries and other non-standard data processing behaviors, significantly increases the cost of collecting and storing data lineage information. These factors combined have resulted in technical bottlenecks in data quality traceability in the big data environment, necessitating breakthroughs and innovations to achieve visible, manageable, and controllable data quality throughout its entire lifecycle. Summary of the Invention

[0004] This invention provides an information processing method based on big data, mainly including:

[0005] Based on the data quality traceability chain, obtain data lineage information, including metadata of data source, processing, transformation and other links;

[0006] Through metadata analysis, the input and output datasets of each data processing stage are determined, forming a data flow diagram;

[0007] Data quality monitoring rules are used to check the quality of input and output data at each stage and identify data with quality problems.

[0008] Based on the quality problem data, trace the data flow diagram back to locate the upstream link that introduced the quality problem;

[0009] Obtain detailed processing logic and parameter configurations for quality issues, and analyze the specific causes of quality problems;

[0010] Based on the causes of the quality problems, a plan was developed to optimize the data repair and processing procedures, and the problematic data was repaired.

[0011] Update the optimized processing logic and parameter configuration to the corresponding stage, and re-execute the data processing flow;

[0012] Perform a quality check on the repaired data to ensure that the problem has been resolved and the data quality meets the requirements.

[0013] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects:

[0014] This invention discloses a method for obtaining data lineage information and ensuring the quality and traceability of data processing at each stage through metadata analysis and quality checks. This method first collects metadata about the data's source, processing, and transformation stages, and then uses this metadata to construct a detailed data flow diagram. The data flow diagram not only shows the flow of data between each processing stage but also indicates the input and output datasets for each stage.

[0015] In this invention, particular attention is paid to the quality of the input and output datasets in the data processing stage. By employing data quality monitoring rules, this invention performs quality checks on the input and output data at each stage, enabling timely identification and location of data with quality issues. Once problematic data is identified, this invention uses a data flow diagram to trace and locate the upstream stage that introduced the quality problem, and obtains the detailed processing logic and parameter configuration of that stage.

[0016] By analyzing these processing logics and parameter configurations, this invention can identify the specific causes of data quality problems and formulate corresponding data repair and processing flow optimization solutions. This includes repairing the problematic data and updating the optimized processing logic and parameter configurations to the corresponding data processing stages. Subsequently, this invention re-executes the data processing flow and performs a second quality check on the repaired data to ensure that the problem has been resolved and the data quality meets the requirements.

[0017] Overall, this invention provides a comprehensive data quality control and management solution by establishing and utilizing data lineage information. This not only improves the transparency and traceability of data processing but also enhances data accuracy and reliability, ultimately ensuring the success of data processing projects and the output of high-quality data. This approach is particularly suitable for data-intensive industries such as finance, healthcare, and e-commerce, where data quality directly impacts the efficiency and effectiveness of business decisions and operations. Attached Figure Description

[0018] Figure 1 This is a flowchart of an information processing method based on big data according to the present invention.

[0019] Figure 2This is a schematic diagram of an information processing method based on big data according to the present invention.

[0020] Figure 3 This is another schematic diagram of an information processing method based on big data according to the present invention. Detailed Implementation

[0021] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0022] like Figure 1-3 This embodiment of an information processing method based on big data may specifically include:

[0023] Step S101: Based on the data quality traceability chain, obtain data lineage information, including metadata of data source, processing, transformation and other links.

[0024] Based on the data quality traceability chain, data lineage information is obtained, including metadata from data sources, processing, and transformation stages. A data lineage analysis algorithm is employed, using metadata extraction technology to extract metadata from data sources such as databases, logs, and ETL tools. This metadata includes information on data source, transformation, processing, and storage. Based on the extracted metadata, a data lineage graph is constructed, displaying the dependencies between data at different stages. Nodes in the data lineage graph represent data, and edges represent transformation relationships between data. Graph algorithms are used to analyze the data lineage graph to identify data quality issues. A network embedding algorithm is used to map the nodes in the data lineage graph into a low-dimensional vector space, obtaining vector representations of the data nodes. Based on the vector representations of the data nodes, the similarity between data nodes is calculated to determine their association. If the similarity between data nodes is below a threshold, it indicates a weak association and potential data quality issues. Based on the associations between data nodes, a community detection algorithm is used to divide the data lineage graph into multiple communities, each representing a group of closely related data nodes. By analyzing the attributes of data nodes within a community, the quality of data within that community is assessed. If the attributes of data nodes within a community are inconsistent, it indicates a problem with the quality of the data within the community. Based on the attributes of the data nodes within the community, anomaly detection algorithms are used to detect anomalous data nodes. Using the attributes of the data nodes as features, an anomaly detection model, such as the Isolation Forest algorithm, is constructed. Through the anomaly detection model, the anomalousness of each data node is calculated; data nodes with high anomalousness indicate potential quality problems. Based on the anomalous data nodes, root cause analysis algorithms are used to analyze the causes of the data quality problems. Starting from the anomalous data nodes, the data lineage graph is traversed to find the upstream nodes of the anomalous data nodes. The attributes and transformation logic of the upstream nodes are analyzed to determine whether the quality problem of the anomalous data nodes is introduced by the upstream nodes or by the data transformation logic. Based on the causes of the data quality problems, data quality improvement algorithms are used to generate data quality improvement suggestions. If the data quality problem is introduced by the upstream nodes, data quality improvement suggestions such as data cleaning and data repair are generated. If the data quality problem is introduced by the data transformation logic, improvement suggestions for optimizing the data transformation logic are generated.

[0025] For example, extracting metadata from database logs and finding that a certain data table is adding 1000 records per day, accounting for 5% of the total data volume, indicates that the data volume of this table is growing rapidly. For example, mapping 1000 nodes in a data lineage graph to a 100-dimensional vector space using a network embedding algorithm, and then calculating the similarity between nodes, reveals that 5 pairs of data nodes have a similarity below the threshold of 5, indicating weak associations and potential data quality issues. For example, using a community detection algorithm to divide 10,000 data nodes into 20 communities, it is found that within one community of 500 nodes, 30% of the data nodes have null values ​​for a certain field, while this proportion is below 5% in other communities, indicating poor data quality in this community. For example, using the Isolation Forest algorithm to perform anomaly detection on 5000 data nodes, it is found that 50 nodes have an anomaly score exceeding 6, requiring further investigation. For example, root cause analysis might reveal that a data quality issue in an anomalous data node was introduced by an upstream data processing node with 500 data transformation logic steps. A flaw in one of these steps caused the downstream node to malfunction. Data quality improvement suggestions could include cleaning 10% of noisy data from an upstream table and fixing a bug in a data transformation logic to eliminate downstream outliers. These improvements could reduce the overall data flow error rate from 5% to below 1%.

[0026] Step S102: Through metadata analysis, determine the input and output datasets for each data processing stage and form a data flow diagram.

[0027] To meet the needs of metadata analysis, metadata extraction techniques are employed to obtain metadata information from the data source, including data source type, data tables, fields, and data formats. Metadata analysis techniques are then used to analyze the relationships between metadata, constructing a metadata relationship graph. Based on this graph, data source tracing analysis is employed to analyze each stage of the data processing flow from bottom to top, identifying the input and output datasets for each stage. A data flow diagram is then drawn to clearly illustrate the flow of datasets within the data processing flow. For each data processing stage in the flow diagram, data feature analysis techniques are used to analyze the data characteristics of the input dataset, including data volume, distribution, and quality, thereby assessing the processing capacity and efficiency of that stage. Simultaneously, data feature comparison analysis is used to compare the differences in data characteristics between the input and output datasets of that stage, evaluating its processing effectiveness. Building upon the flow diagram, data dependency analysis is used to analyze the dependencies between different data processing stages. If the input dataset of a stage depends on the output of an upstream stage, corresponding dependency lines are added to the flow diagram. Through data dependency analysis, key and bottleneck stages in the data processing flow can be identified. Based on the results of data dependency analysis, critical path analysis algorithms, such as Huffman coding and Dijkstra's algorithm, are used to calculate the critical path of the data processing flow and identify the links that have the greatest impact on the overall efficiency of the process. For the links on the critical path, the data processing algorithms and implementation logic can be optimized to improve the overall efficiency of the process. Finally, the optimized data processing logic is applied to the data flow graph to update the original data flow relationships. Simultaneously, data monitoring and statistical analysis techniques are used to monitor and statistically analyze the optimized data processing flow, obtaining optimized data processing efficiency indicators to evaluate the optimization effect. Based on the data monitoring and statistical analysis results, the data processing flow can be further optimized, and the optimization results can be reflected in the data flow graph, forming a closed-loop continuous optimization process.

[0028] For example, metadata analysis techniques can be used to analyze the relationships between metadata and construct a metadata relationship graph. Suppose data source A contains 100 tables and data source B contains 80 tables. After relationship analysis, it is found that 60 of these tables are related, thus a metadata relationship graph with 60 nodes can be constructed. For example, data feature analysis techniques can be used to analyze the data characteristics of the input dataset in a data processing stage. Suppose a certain stage's input dataset contains 10 million records, the data distribution is normally distributed, and the data quality is 93%, then it can be judged that the data processing capability of this stage is strong and the data quality is high. For example, data feature comparison analysis techniques can be used to compare the differences in data features between the input and output datasets of a data processing stage. Suppose the input dataset contains 10 million records with a data quality of 95%; the output dataset contains 8 million records with a data quality of 98%, then it can be evaluated that the data processing effect of this stage is good and the data quality has improved. For example, critical path analysis algorithms can be used to calculate the critical path of the data processing flow. Assuming a data processing workflow comprises five stages, with processing times of 30 minutes, 20 minutes, 45 minutes, 15 minutes, and 25 minutes for each stage, the critical paths can be identified as stages 1, 3, and 5, which have the greatest impact on overall workflow efficiency. For example, data monitoring and statistical analysis techniques can be used to monitor and analyze the optimized data processing workflow. If the average processing time before optimization was 5 hours, and the average time after optimization is 2 hours, then the optimization effect can be assessed as good, and data processing efficiency has improved.

[0029] Step S103: Using data quality monitoring rules, perform quality checks on the input and output data of each stage to identify data with quality problems.

[0030] According to predefined data quality monitoring rules, the input and output data of each stage are checked for quality. Data profiling techniques are used to perform statistical analysis on the input data to obtain the basic characteristics and distribution of the data. Through data feature analysis, it is determined whether there are quality problems such as missing values, outliers, and duplicate values. If quality problems are found, the data is marked as problematic data, and the problem type and severity are recorded. Appropriate data cleaning methods are used to process the data based on the type and severity of the quality problems. For missing values, methods such as mean imputation, median imputation, and highest frequency value imputation can be used. For outliers, methods such as truncation and box plot methods can be used for identification and processing. For duplicate values, deduplication algorithms can be used for deduplication. Through data cleaning, a corrected version of the problematic data is obtained. The corrected data is compared with the original data, the data correction rate is calculated, the effectiveness of data cleaning is evaluated, and a preset threshold is set. If the data correction rate reaches the preset threshold, the corrected data is used as input data for subsequent stages. If the data correction rate does not reach the preset threshold, the causes of the data quality problems need to be further analyzed, and the data cleaning methods need to be adjusted. The corrected data undergoes a quality re-check using data validation techniques to verify its compliance with predefined data quality standards. Data quality assessment calculates quality indicators such as data completeness, accuracy, and consistency to determine if the data quality meets the standards. If the data meets the standards, it is marked as qualified and passed on to subsequent stages. If the data does not meet the standards, further analysis of the causes of the data quality problems is required, and corresponding measures are taken to address them. Through data quality monitoring and data cleaning, a high-quality dataset is obtained. This high-quality dataset serves as input for subsequent data analysis and data mining stages, ensuring the accuracy and reliability of data processing and analysis. Simultaneously, the data quality monitoring results and data cleaning process are recorded as a basis for data quality management, providing a reference for future data quality improvements.

[0031] For example, in the label: <result>For example, a quality check on a dataset containing 1000 records revealed 80 records with missing values, representing 8% of the dataset, indicating a moderate severity. For example, for a batch of numerical data, box plots were used for outlier detection, marking data points less than 5 times the interquartile range (IQR) of the lower quartile Q1 and greater than 5 times the IQR of the upper quartile Q3 as outliers. After processing, 20 outliers were removed from the dataset. For example, after deduplication of a dataset containing 5000 records, 4800 unique records were obtained, achieving a data correction rate of 96%, reaching the preset threshold of 95%. For example, through data quality assessment, a dataset was calculated to have 98% completeness, 95% accuracy, 99% consistency, and an overall quality index of 95%, meeting the predefined quality standard of 93%. For example, a statistical analysis of a dataset containing 10,000 records yields a mean of 85, a standard deviation of 11, a median of 82, and a mode of 85, indicating that the data follows a normal distribution.

[0032] Step S104: Based on the quality problem data, trace back the data flow diagram to locate the upstream link that introduced the quality problem.

[0033] Based on quality issue data, quality inspection records from each stage of the product manufacturing process are obtained. Data mining techniques are used to mine association rules between the quality issue data and the production stages to obtain association rules. These association rules are then used to determine which production stages primarily originate the quality issues. A production process data flow diagram is then created based on these association rules to identify the upstream stages from which the quality issue data primarily originates. If the quality issue data primarily originates from a particular upstream stage, that stage is highly likely to be the key stage introducing the quality issue. Production process parameters for the key stages are obtained, and a classification model between these parameters and quality issues is established using a support vector machine algorithm. This model is used to identify abnormal combinations of process parameters in the key stages. If the production process parameters exceed the normal process range, the batch of products is highly likely to have quality defects. Raw material quality inspection data is also obtained, and principal component analysis is used to reduce the dimensionality of the raw material quality data, obtaining the principal component scores. These scores are then used to determine whether the raw material quality is abnormal. If the raw material quality is abnormal, the quality issue in the batch of products is highly likely to be caused by the raw material quality. By comprehensively analyzing production process parameters and raw material quality inspection results, a quality problem tracing model is established using a decision tree algorithm. The tracing model determines whether the quality problem is introduced by a key process step or by abnormal raw material quality, thereby achieving quality problem tracing and locating the upstream link that introduced the quality problem.

[0034] For example, by performing association rule mining on 5000 data points on quality issues and production processes, the following association rules were obtained: Assembly → Quality Issues (Support = 30%, Confidence = 75%); Welding → Quality Issues (Support = 25%, Confidence = 80%); Painting → Quality Issues (Support = 20%, Confidence = 70%). This indicates that quality issues mainly originate from the assembly, welding, and painting processes. Analysis of the production process data flow diagram shows that the upstream links of the assembly process include parts processing and outsourced parts procurement; the upstream links of the welding process include metal parts processing; and the upstream link of the painting process is chemical raw material procurement. By analyzing 500 production process parameter data points from the assembly process, a classification model was established using a support vector machine algorithm. The model accuracy was 85%. The classification model determined that when the assembly gap parameter exceeds 5mm, the probability of component assembly failure is as high as 90%. Therefore, the assembly gap parameter is a key process parameter that introduces quality issues into the assembly process. 1000 weld joint mechanical property data from the welding process were analyzed, and principal component analysis (PCA) was used for dimensionality reduction. The first principal component contributed 60% of the variance, and the second principal component contributed 25%. When the first principal component score of a weld joint was below -5, the probability of welding quality failure was 95%. Therefore, the mechanical properties of welding materials are the main cause of quality problems introduced during the welding process. 1000 batches of chemical raw material quality inspection data were also analyzed, and PCA was used for dimensionality reduction. The first principal component contributed 70% of the variance. When the first principal component score of a chemical raw material was below -2, 90% of the batches of products had quality problems. Therefore, the quality of chemical raw materials is the main cause of quality problems introduced during the spraying process. A quality problem tracing model was established using a decision tree algorithm based on a comprehensive analysis of production process parameters and raw material quality inspection data. The model accuracy was 80%. The tracing model determined that among products with quality problems, 60% were caused by assembly gaps in the assembly process, 30% by welding material quality problems, and 10% by chemical raw material quality problems.

[0035] Step S105: Obtain the detailed processing logic and parameter configuration of the quality problem process, and analyze the specific causes of the quality problem.

[0036] Based on the quality data collected during the production process, data mining techniques are used to analyze the quality problem links. Association rule mining is performed on the quality data of each stage of the production process to obtain the correlations between quality problem links. Based on the results of association rule mining, the key links leading to quality problems are identified. Statistical analysis is performed on the process parameters of the key quality problem links, and principal component analysis is used to reduce the dimensionality of the process parameters, revealing the critical process parameters affecting quality problems. Cluster analysis is performed on the threshold ranges of the critical process parameters to determine the threshold intervals of each process parameter. If a process parameter exceeds the threshold interval, it is determined to be a cause of the quality problem. Based on the results of the quality problem cause analysis, a decision tree algorithm is used to classify the quality problems, obtaining the specific combination of process parameters leading to the quality problems. Analysis of the decision tree model reveals the detailed processing logic for the quality problems. Based on the branching conditions of each node in the decision tree model, the configuration of processing parameters for the quality problems is determined. The configuration of processing parameters for the quality problems is optimized using a genetic algorithm to find the optimal threshold parameters in the decision tree model, obtaining the optimal configuration of processing parameters for the quality problems. Based on the optimal configuration of processing parameters, the production process is adjusted and controlled to improve product quality and reduce the occurrence of quality problems.

[0037] For example, association rule mining: By performing association rule mining on 1000 pieces of production process quality data, it was found that when quality problems occur in the cutting stage, there is also an 85% probability of quality problems occurring in the grinding stage. This identifies cutting and grinding as key stages leading to quality problems. Principal component analysis: For example, by performing principal component analysis on 10 process parameters in the cutting stage, the dimensionality was reduced to 3 principal components, with a cumulative contribution rate of 95%. This identified cutting speed, cutting time, and the number of cuts as key process parameters affecting cutting quality. Cluster analysis: For example, by performing cluster analysis on cutting speed, the optimal threshold range for cutting speed was found to be 50–80 meters per minute. If the cutting speed is lower than 50 meters per minute or higher than 80 meters per minute, it is determined to be a cause of cutting quality problems. Decision Tree Classification: For example, by classifying 2000 quality problem samples using a decision tree, the following rules were obtained: when the cutting speed is below 50 meters / minute and the cutting time is greater than 10 seconds, uneven cuts will result; when the cutting speed is above 80 meters / minute and the number of cuts is more than 3, burrs will result. Genetic Algorithm Optimization: For example, by using a genetic algorithm to optimize the three parameters of cutting speed, cutting time, and number of cuts in the decision tree model, the optimal parameter configuration was obtained: cutting speed 65 meters / minute, cutting time 8 seconds, and number of cuts 2. Applying this configuration to the production process increased the product qualification rate from 90% to 95%.

[0038] Step S106: Based on the cause of the quality problem, formulate a data repair and processing flow optimization plan, and repair the problematic data.

[0039] Based on the analysis of the causes of the quality issues, the following data repair and processing optimization plan was formulated to repair the problematic data: A data quality issue list was obtained through data quality assessment. Based on the data quality issue list, data exploration analysis methods were used to determine the specific causes of the data quality issues. Corresponding data repair rules were designed for different causes of data quality issues. If missing values ​​exist, the expected value algorithm is used for data completion; if outliers exist, the local anomaly factor algorithm is used for anomaly detection and repair; if inconsistencies exist, data association analysis is used to identify and unify the inconsistent data. According to the data repair rules, data cleaning and transformation algorithms are used to repair the problematic data, resulting in a repaired dataset. Data quality verification is conducted to determine if the repaired dataset meets the quality requirements. If it meets the requirements, the repaired dataset is updated to the production environment; if it does not meet the requirements, data repair is repeated until the quality requirements are met. During the data repair process, data version management is used to record the process and results of each data repair, ensuring that the data repair process is traceable and reproducible. Based on lessons learned from the data repair process, optimize data processing workflows, establish data quality management systems, and standardize operational procedures for data collection, storage, and processing to improve data quality from the source. Regularly conduct data quality assessments and data repairs to normalize data quality management and continuously improve data quality.

[0040] For example, a data quality assessment is used to obtain a list of data quality issues. For instance, the assessment might find that 5% of the records in the dataset have missing values, 2% have outliers, and 1% have inconsistencies. Data exploration analysis is then used to determine the specific causes of these data quality issues. For example, the analysis might find that missing values ​​are mainly concentrated in the age field, accounting for 8% of the total records in that field; outliers are mainly found in the income field, with 20 records showing negative income values; and inconsistencies are mainly found in the address field, with 15 records showing address information that does not match other information. Corresponding data repair rules are designed for different causes of data quality issues: 1. If missing values ​​exist, the Expectation-Maximization (EM) algorithm is used for data completion. For example, for missing values ​​in the age field, the EEM algorithm estimates the average age of these missing values ​​to be 35 years old, and then fills all missing values ​​in the age field with 35 years old. 2. If outliers exist, the Local Anomaly Factor (LOF) algorithm is used for anomaly detection and repair. For example, the LEF algorithm detects 20 outlier records in the income field, and then corrects these outliers to the average value of 8000 yuan for that field. 3. If data inconsistencies exist, data correlation analysis is used to identify and unify the inconsistent data. For example, correlation analysis revealed 15 inconsistent records in the address field; after verification, the address information of these records was updated to the correct values. According to data repair rules, data cleaning and transformation algorithms are used to repair the problematic data, resulting in a repaired dataset. Data quality verification is conducted to determine if the repaired dataset meets quality requirements. For example, after quality verification, it was found that the proportion of missing values ​​in the repaired dataset was reduced to below 1%, the proportion of outliers was reduced to below 5%, and the inconsistency rate was reduced to below 1%, meeting data quality requirements. The repaired dataset is then updated to the production environment. During the data repair process, data version management is employed to record the process and results of each data repair, ensuring that the data repair process is traceable and reproducible. Based on lessons learned from the data repair process, data processing workflows are optimized, data quality management systems are established, and operational procedures for data collection, storage, and processing are standardized to improve data quality from the source. For example, a dual-verification rule for data entry is established, requiring that critical data be independently verified by two people; if the verification results of the two people are inconsistent, a second verification is required. Regular data quality assessments and data repairs are conducted to normalize data quality management and continuously improve data quality. For example, a monthly data quality assessment mechanism has been established to comprehensively evaluate data quality every month and promptly repair any quality issues found. The results of three consecutive months of data quality assessments show that data quality has continued to improve, with the data missing rate decreasing from 5% to 1%, the outlier rate decreasing from 5% to 2%, and the inconsistency rate decreasing from 1% to 0.5%.

[0041] Step S107: Update the optimized processing logic and parameter configuration to the corresponding stage, and re-execute the data processing flow.

[0042] Based on data quality analysis, issues with noisy and outlier data are identified. Data cleaning and repair techniques are employed to process the noisy and outlier data, resulting in data with improved accuracy and completeness. Rotation, translation, and scaling transformations are applied to this improved data to obtain a larger number of training samples. Feature importance analysis is used to identify key features affecting prediction results from the training samples. Based on these key features, the feature extraction process is optimized to obtain optimized features. New feature construction methods, including cross-features and statistical features, are introduced to obtain richer feature representations. Finally, the model is improved using attention mechanisms and residual connection techniques. Feature extraction and fusion capabilities; the model is trained using Adam and AdaGrad optimization algorithms to obtain a training model with improved convergence speed and training efficiency; cross-validation is used to evaluate the performance of the training model; the prediction results of the training model are analyzed using confusion matrix and ROC curve tools; the uncertainty of the prediction results is quantified using confidence assessment techniques to obtain reliable decision-making basis; the optimized end-to-end data processing flow is modularly designed and containerized for deployment to improve system maintainability and scalability; a model version management and update mechanism is established to obtain continuously optimized and iterative data processing and prediction models.

[0043] For example, data quality analysis revealed that 5% of the 10,000 data points contained noise, such as missing or outliers. Data cleaning and repair techniques, including filling in missing values ​​and smoothing outliers, improved the accuracy from 90% to 98% and the completeness from 95% to 99%. Data augmentation was then applied to the cleaned data, such as rotating images by 30 degrees, shifting them by 20 pixels, and scaling them by 8 times, increasing the number of training samples from 10,000 to 50,000. Feature importance analysis identified 20 key features with significant impact on prediction results from 200 features. Based on these key features, the feature extraction process was optimized, for example, by increasing feature richness, improving prediction accuracy from 85% to 90%. Ten cross features and five statistical features were introduced, increasing the number of features from 20 to 35. Attention mechanisms and residual connection techniques were used to improve the model, enhancing feature extraction and fusion capabilities by 20%. The Adam optimization algorithm was used to train the model, achieving a 2x faster convergence speed and a 50% higher training efficiency compared to the SGD optimization algorithm. The model performance was evaluated using 5-fold cross-validation, with an average accuracy of 95% and an average recall of 92%.1 Confusion matrix analysis showed a prediction accuracy of 98% on positive samples and 96% on negative samples; ROC curve analysis showed an AUC of 99.1 Confidence assessment showed an average prediction confidence of 95%, with 80% of samples having a confidence level greater than 90%.1 The data processing workflow was divided into five modules: data acquisition, data cleaning, feature engineering, model training, and model evaluation, and deployed using Docker containers, improving system maintainability and scalability by 30%.1 A model version management mechanism was established, iterating with a new version weekly. After three months of continuous optimization, the overall model performance improved.

[0044] Step S108: Perform a quality check on the repaired data to ensure that the problem has been resolved and the data quality meets the requirements.

[0045] The repaired data undergoes a quality check to ensure the issues have been resolved and the data quality meets requirements. First, Pearson correlation coefficient analysis is used to calculate the correlation between various fields in the repaired data. The obtained correlation coefficient matrix is ​​used to determine if any abnormal correlations exist between data fields. If abnormally correlated fields are found, the data in these fields is re-checked and repaired. Based on the characteristics of the abnormal fields, a local anomaly detection algorithm is used to identify outliers in these fields, and the identified abnormal data is removed or corrected. Next, a data quality assessment index system is constructed, and a weighted average comprehensive evaluation model is used to calculate the quality score of the repaired data. The quality score is compared with a threshold. If the quality score is below the threshold, it indicates that the data still has quality problems, requiring further analysis of the causes and remedial measures; if the quality score is above the threshold, it indicates that the data quality meets requirements. Finally, Monte Carlo simulation is used to conduct robustness tests on the repaired data. Through multiple simulations, robustness index values ​​are obtained, and the stability and reliability of the data quality are judged based on these robustness index values. If the robustness index value fluctuates significantly, it indicates that the data quality is unstable and further measures need to be taken to improve the robustness of the data quality; if the robustness index value fluctuates slightly, it indicates that the data quality is stable and reliable, and the repaired data can be used for subsequent analysis and application.

[0046] For example, a quality check is performed on the repaired data to ensure that the problem has been resolved and the data quality meets the requirements: Pearson correlation coefficient analysis is used to calculate the correlation between various fields of the repaired data. For example, in the calculated correlation coefficient matrix, the correlation coefficient between field A and field B is 95, indicating that the two fields are highly correlated and there may be data anomalies, requiring focused investigation. Based on the data characteristics of the abnormal fields, a local anomaly detection algorithm is used to identify outliers in these fields. For example, in 10,000 data entries in field C, this algorithm identifies 150 abnormal data entries, accounting for 5%, which are removed, and the correlation coefficient matrix is ​​recalculated. A data quality assessment index system is constructed, and a weighted average comprehensive evaluation model is used to calculate the quality score of the repaired data. For example, the index system contains 5 indicators, with weights of 3, 2, 2, 15, and 15 respectively. The scores of each indicator in the repaired data are 95, 90, 92, 96, and 97, and the weighted average comprehensive quality score is 985, which is higher than the preset threshold of 90, indicating that the data quality meets the requirements. Monte Carlo simulation is used to conduct robustness tests on the repaired data. For example, through 1000 simulations, the mean value of the data quality robustness index Cpk was 33, the standard deviation was 0.5, and the Cpk values ​​were all greater than 25 with small fluctuations, indicating that the data quality robustness after repair was good and could be used for subsequent analysis and applications.

[0047] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.< / result>

Claims

1. A big data-based informatization processing method, characterized by, The method comprises: According to the data quality traceability chain, the data blood relationship information is obtained, including the metadata of data source, processing and conversion link; Through metadata analysis, the input and output data sets of each data processing link are determined to form a data flow diagram; Using data quality monitoring rules, the quality of the input and output data of each link is checked to identify the data with quality problems; According to the quality problem data, the data flow diagram is traced back to locate the upstream link that introduces the quality problem; According to the quality problem data, the quality detection records of each link in the production process are obtained, and the data mining technology is used to mine the association rules of the quality problem data to obtain the association rules between the quality problem and the production link; Determine the production link where the quality problem mainly comes from through the association rules; According to the association rules between the quality problem and the production link, the production process data flow diagram is obtained, and the data flow diagram is used to determine the upstream link where the quality problem mainly comes from; If the quality problem data mainly comes from a certain upstream link, then the link is the key link that introduces the quality problem; The production process parameters of the key link are obtained, and a classification model between the production process parameters and the quality problem is established by using the support vector machine algorithm, and the abnormal process parameter combination of the key link is determined by the classification model; If the production process parameters exceed the normal process range, then the batch product has quality defects; The raw material quality detection data is obtained, the principal component analysis method is used to reduce the dimension of the raw material quality data, the principal component score of the raw material quality is obtained, and whether the raw material quality is abnormal is determined by the principal component score; If the raw material quality is abnormal, then the quality problem of the batch product is introduced by the raw material quality; Comprehensive analysis of production process parameters and raw material quality test results, using decision tree algorithm to establish quality problem traceability model, through traceability model to determine whether the quality problem is introduced by key process link or raw material quality abnormality, so as to realize quality problem traceability, locate the upstream link that introduces the quality problem; Obtain the detailed processing logic and parameter configuration of the quality problem link, and analyze the specific reasons for the quality problem; According to the quality data collected in the production process, the quality problem link is analyzed by using data mining technology; Through the association rule mining of the quality data of each link in the production process, the association relationship between the quality problem links is obtained; According to the results of association rule mining, the key link that causes the quality problem is determined; Statistical analysis is performed on the process parameters of the key quality problem link, and the principal component analysis method is used to reduce the dimension of the process parameters to obtain the key process parameters affecting the quality problem; The threshold range of the key process parameters is clustered and analyzed to determine the threshold interval of each process parameter; If the process parameter exceeds the threshold interval, it is determined as the cause of the quality problem; ​ ​ Based on the results of the quality problem cause analysis, the decision tree algorithm is used to classify the quality problems and obtain the specific process parameter combinations that lead to the quality problems; By analyzing the decision tree model, we can obtain the detailed processing logic for quality issues. Based on the branching conditions of each node in the decision tree model, determine the configuration of processing parameters for quality issues; The parameter configuration for handling quality problems is optimized by using a genetic algorithm to find the optimal threshold parameter configuration in the decision tree model. By adjusting and controlling the production process according to the optimal processing parameters, product quality can be improved and the occurrence of quality problems can be reduced. Based on the causes of the quality problems, a plan was developed to optimize the data repair and processing procedures, and the problematic data was repaired. Update the optimized processing logic and parameter configuration to the corresponding stage, and re-execute the data processing flow; Perform a quality check on the repaired data to ensure that the problem has been resolved and the data quality meets the requirements.

2. The method of claim 1, wherein, The process of obtaining data lineage information based on the data quality traceability chain includes metadata about the data source, processing, and transformation stages, including: Based on the data quality traceability chain, obtain data lineage information, including metadata about the data source, processing and transformation stages; The data lineage analysis algorithm is used to extract metadata from databases, logs and ETL tool data sources through metadata extraction technology; Metadata includes information about the source, transformation, processing, and storage of data; Based on the extracted metadata, a data lineage map is constructed to show the dependencies between data at different stages; In a data lineage graph, nodes represent data, and edges represent the transformation relationships between data. By using graph algorithms to analyze the pedigree data, data quality issues were identified. A network embedding algorithm is used to map nodes in the data lineage graph to a low-dimensional vector space to obtain vector representations of data nodes; Based on the vector representation of the data nodes, calculate the similarity between the data nodes and determine the association between them; If the similarity between data nodes is below the threshold, it indicates that the relationship between the data nodes is weak and there may be problems with the data quality. Based on the relationships between data nodes, a community discovery algorithm is used to divide the data lineage graph into multiple communities, with each community representing a group of closely related data nodes. By analyzing the attributes of data nodes within the community, we can determine the quality of the data within the community. If the attributes of data nodes within a community are inconsistent, it indicates that there is a problem with the quality of the data within the community. Based on the attributes of data nodes within the community, an anomaly detection algorithm is used to detect abnormal data nodes within the community. Using the attributes of data nodes within the community as features, an anomaly detection model is constructed using the isolated forest algorithm; The anomaly detection model is used to calculate the anomaly score of each data node. Data nodes with high anomaly scores indicate that there are quality problems. Based on the abnormal data nodes, the root cause analysis algorithm is used to analyze the causes of data quality problems; Starting from the abnormal data node, traverse the data lineage graph to find the upstream node of the abnormal data node, analyze the attributes and transformation logic of the upstream node, and determine whether the quality problem of the abnormal data node is introduced by the upstream node or by the data transformation logic. Based on the causes of data quality problems, data quality improvement algorithms are used to generate data quality improvement suggestions. If the data quality problem is introduced by the upstream node, then data quality improvement suggestions such as data cleaning or data repair will be generated. If the data quality problem is caused by the data transformation logic, then suggestions for improving the data transformation logic will be generated.

3. The method of claim 1, wherein, The process involves analyzing metadata to determine the input and output datasets for each data processing stage, forming a data flow diagram, including: Based on the requirements of metadata analysis, metadata extraction technology is used to obtain metadata information of the data source. The metadata information of the data source includes metadata of data source type, data table, fields and data format. Metadata analysis techniques are used to analyze the relationships between metadata and construct a metadata relationship graph. Based on the metadata association map, data source tracing analysis technology is used to analyze each link in the data processing flow from bottom to top to determine the input and output datasets of each link; Based on this, a data flow diagram is drawn to clearly show the process of data transfer in the data processing flow; For each data processing stage in the data flow diagram, data feature analysis technology is used to analyze the data characteristics of the input dataset of that stage, including data volume, data distribution and data quality, so as to determine the data processing capability and efficiency of that stage. At the same time, by using data feature comparison and analysis technology, the differences in data features between the input dataset and the output dataset of this step are compared to evaluate the data processing effect of this step; Based on the data flow diagram, data dependency analysis technology is used to analyze the dependencies between various data processing links; If the input dataset of a certain stage depends on the output of the upstream stage, then add the corresponding dependency connection in the data flow graph; Data dependency analysis identifies key and bottleneck links in the data processing flow. Based on the results of data dependency analysis, a critical path analysis algorithm is used, such as Huffman coding or Dijkstra algorithm, to calculate the critical path of the data processing flow and determine the link that has the greatest impact on the efficiency of the entire process. For each link on the critical path, the data processing algorithm and implementation logic are optimized to improve the efficiency of the entire process. Finally, the optimized data processing logic is applied to the data flow diagram to update the original data flow relationships; At the same time, data monitoring and statistical analysis technologies are used to monitor and statistically analyze the optimized data processing flow to obtain optimized data processing efficiency indicators for evaluating the optimization effect. Based on the data monitoring and statistical analysis results, the data processing flow is further optimized, and the optimization results are reflected in the data flow diagram, forming a closed-loop continuous optimization process.

4. The method of claim 1, wherein, The aforementioned data quality monitoring rules are used to perform quality checks on the input and output data at each stage, identifying data with quality problems, including: According to predefined data quality monitoring rules, the input and output data of each step are checked for quality. Data profiling techniques are used to perform statistical analysis on the input data to obtain the basic characteristics and distribution of the data; Data feature analysis is used to determine whether there are quality problems in the data, including missing values, outliers, and duplicate values. If the data has quality issues, mark it as problematic data and record the type and severity of the problem; Based on the type and severity of the data quality problem, appropriate data cleaning methods are used to process it; For missing values, imputation is performed, using one of the following methods: mean imputation, median imputation, or highest frequency value imputation. Outliers are identified and processed using truncation and box plot methods. For duplicate values, a deduplication algorithm is used to remove duplicates. Through data cleaning, a corrected version of the data with quality issues was obtained; The corrected data is compared with the original data, the data correction rate is calculated, the effectiveness of data cleaning is evaluated, and a preset threshold is set. If the data correction rate reaches the preset threshold, the corrected data will be used as the input data for subsequent steps. If the data correction rate does not reach the preset threshold, further analyze the reasons for the data quality problems and adjust the data cleaning methods. The corrected data is re-checked for quality, and data validation techniques are used to verify whether the data meets the predefined data quality standards. Data quality assessment involves calculating data quality indicators to determine whether the data quality meets the standards; these quality indicators include data completeness, accuracy, and consistency. If the data quality meets the standards, it is marked as qualified data and passed on to the next step; If the data quality does not meet the standards, further analyze the reasons for the data quality problems and take corresponding measures to deal with them; High-quality datasets are obtained through data quality monitoring and data cleaning; Using high-quality datasets as input for subsequent data analysis and data mining ensures the accuracy and reliability of data processing and analysis. At the same time, the data quality monitoring results and data cleaning process will be recorded as a basis for data quality management and to provide a reference for subsequent data quality improvement.

5. The method of claim 1, wherein, The aforementioned plan, based on the causes of the quality issues, includes developing a data repair and processing workflow optimization scheme to repair the problematic data, including: Based on the analysis of the causes of the quality problems, the following data repair and processing optimization plan was formulated to repair the problematic data: A list of data quality problems was obtained through data quality assessment; Based on the list of data quality issues, data exploration analysis methods are used to determine the specific causes of the data quality problems. Design corresponding data repair rules for different causes of data quality problems; If the data contains missing values, the expected value algorithm is used to complete the data. If outliers are found in the data, the local anomaly factor algorithm is used to detect and repair the outliers. If inconsistencies exist in the data, data correlation analysis will be used to identify the inconsistent data and then unify the data. Based on the data repair rules, data cleaning and transformation algorithms are used to repair the problematic data, resulting in a repaired dataset. Data quality verification is used to determine whether the repaired dataset meets the quality requirements. If the requirements are met, the repaired dataset will be updated to the production environment; If the requirements are not met, the data will be repaired again until the quality requirements are met. During the data repair process, data version management is adopted to record the process and results of each data repair, ensuring that the data repair process is traceable and reproducible; Based on experience gained during the data repair process, optimize the data processing workflow, formulate a data quality management system, and standardize the operating procedures for data collection, storage, and processing to improve data quality from the source. Regularly conduct data quality assessments and data repairs to normalize data quality management and continuously improve data quality.

6. The method of claim 1, wherein, The step of updating the optimized processing logic and parameter configuration to the corresponding stage and re-executing the data processing flow includes: Based on data quality analysis, identify quality issues related to noisy and outlier data. Data cleaning and repair techniques are used to process the noisy data and outliers, resulting in data with improved accuracy and completeness. By performing rotation, translation, and scaling transformations on the data that improves accuracy and completeness, a larger number of training sample data can be obtained. By analyzing feature importance, key features that influence the prediction results are identified from the training sample data; Based on the key features, the feature extraction process is optimized to obtain optimized features; Introduction Novel feature construction methods using cross-features and statistical features yield rich feature representations; Improve the model's feature extraction and fusion capabilities based on attention mechanisms and residual connection techniques; The model was trained using the Adam and AdaGrad optimization algorithms, resulting in a training model with improved convergence speed and training efficiency. The performance of the trained model was evaluated using cross-validation. The prediction results of the trained model are analyzed using confusion matrix and ROC curve tools. Based on confidence assessment techniques, the uncertainty of the prediction results is quantified to obtain a reliable basis for decision-making; The optimized end-to-end data processing flow is modularly designed and containerized for deployment, improving the maintainability and scalability of the system. Establish a model version management and update mechanism to obtain continuously optimized and iterative data processing and prediction models.

7. The method of claim 1, wherein, The process of performing a quality check on the repaired data to ensure that the problem has been resolved and the data quality meets the requirements includes: Perform a quality check on the repaired data to ensure that the problem has been resolved and the data quality meets the requirements; First, the correlation between various fields of the repaired data is calculated using the Pearson correlation coefficient analysis method. By obtaining the correlation coefficient matrix, it is determined whether there are abnormal correlations between the data fields. If any fields are found to be abnormal, the data in these fields will be re-examined and repaired. Based on the data characteristics of the abnormal fields, a local anomaly detection algorithm is used to identify outliers in these fields, and the identified abnormal data is removed or corrected. Then, by constructing a data quality assessment index system, a weighted average comprehensive evaluation model is used to calculate the quality score of the repaired data. The quality score is compared with a threshold. If the quality score is lower than the threshold, it indicates that the data still has quality problems and it is necessary to further analyze the causes of the data quality problems and take measures to repair them. If the quality score is higher than the threshold, it indicates that the data quality meets the requirements; Finally, the Monte Carlo simulation method was used to conduct robustness tests on the repaired data. Through multiple simulations, robustness index values ​​of the data were obtained, and the stability and reliability of the data quality were judged based on the robustness index values. If the robustness index value fluctuates significantly, it indicates that the data quality is unstable and further measures need to be taken to improve the robustness of the data quality. If the robustness index value fluctuates little, it indicates that the data quality is stable and reliable, and the repaired data can be used for subsequent analysis and application.

Images