Informatization processing method based on big data
Through data quality traceability chain and metadata analysis, a data flow chart is built, and the quality problems in the big data platform are identified and repaired, and the performance bottleneck of data blood relationship tracking and traceability is solved, the quality management of the entire life cycle of data is realized, and the quality of the data is improved. Accuracy and reliability of data processing.
Patent Information
- Application Number
- CN202410900703.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-05
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-07-05
AI Technical Summary
Data quality problems in big data platforms lead to misleading data processing and business risks, and there are performance bottlenecks in data blood relationship tracking and quality traceability, making it difficult to achieve visual, manageable and controllable quality throughout the data life cycle.
The blood relationship information is obtained through the data quality traceability chain, metadata analysis is used to build a data flow chart, use quality monitoring rules to identify problems, trace the upstream links, analyze and repair quality problems, optimize processing logic and parameter configuration, and ensure data quality.
Improves the transparency and traceability of data processing, enhances the accuracy and reliability of data, and ensures the success and high-quality output of data processing projects.
Smart Images

Figure CN119003495A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of information technology, and in particular to an information processing method based on big data. Background Art
[0002] In the big data information processing platform, the aggregation and processing of massive heterogeneous data is complicated, involving many links such as data collection, cleaning, storage, calculation, and visualization. Due to the wide range of data sources, diverse data formats, and complex processing logic, data is prone to quality problems during the circulation process, such as data missing, data duplication, and data inconsistency. When these defective data are used for business analysis or decision-making, they will be misleading and bring business risks.
[0003] Therefore, the big data platform urgently needs to build a data quality traceability system, but it faces the following technical difficulties: First, end-to-end data lineage relationship tracking involves massive data processing nodes, and the data conversion logic of each node is different. The extraction and association of full-link lineage information faces performance bottlenecks. Secondly, after the complex serial and parallel processing of multi-source heterogeneous data, the data model has changed significantly, and the corresponding relationship of data items is difficult to accurately map, affecting the accuracy of lineage analysis. In addition, the job scheduling of the big data platform is frequent, and there are non-standard data processing behaviors such as ad hoc queries, which leads to a sharp increase in the collection and storage costs of data lineage information. These factors have led to the technical bottleneck of data quality traceability in the big data environment, and it is urgent to break through innovation to achieve the quality of data throughout the life cycle. Visualization, management, and control. Summary of the invention
[0004] The present invention provides an information processing method based on big data, which mainly includes:
[0005] According to the data quality traceability chain, obtain data lineage information, including metadata of data sources, processing, conversion and other links;
[0006] Through metadata analysis, the input and output data sets of each data processing link are determined to form a data flow diagram;
[0007] Adopt data quality monitoring rules to check the quality of input and output data at each stage and identify data with quality problems;
[0008] Based on the quality problem data, trace back the data flow diagram to locate the upstream link that introduces the quality problem;
[0009] Obtain the detailed processing logic and parameter configuration of the quality problem link, and analyze the specific causes of the quality problem;
[0010] Develop data repair and processing flow optimization plans based on the causes of quality problems and repair problematic data;
[0011] Update the optimized processing logic and parameter configuration to the corresponding links, and re-execute the data processing flow;
[0012] Perform quality checks on the repaired data to ensure that the problem has been resolved and the data quality meets the requirements.
[0013] The technical solution provided by the embodiment of the present invention may have the following beneficial effects:
[0014] The present invention discloses a method for obtaining data kinship information and ensuring the quality and traceability of data processing in each link through metadata analysis and quality inspection. This method first collects metadata of data sources, processing and conversion links, and uses these metadata to construct a detailed data flow diagram. The data flow diagram not only shows the flow of data between various processing links, but also indicates the input and output data sets of each link.
[0015] In the present invention, special attention is paid to the quality of the input and output data sets of the data processing link. By adopting data quality monitoring rules, the present invention performs quality checks on the input and output data of each link, and can timely identify and locate data with quality problems. Once the problematic data is identified, the present invention will use the data flow diagram to trace and locate the upstream link that introduces the quality problem, and obtain the detailed processing logic and parameter configuration of the link.
[0016] By analyzing these processing logics and parameter configurations, the present invention can determine the specific causes of data quality problems, and formulate corresponding data repair and processing flow optimization solutions for these causes. This includes repairing the problematic data and updating the optimized processing logic and parameter configuration to the corresponding data processing links. Subsequently, the present invention re-executes the data processing flow and performs another quality check on the repaired data to ensure that the problem has been resolved and the data quality meets the requirements.
[0017] In general, the present invention provides a comprehensive data quality control and management solution by establishing and utilizing data lineage information. This not only improves the transparency and traceability of data processing, but also enhances the accuracy and reliability of data, ultimately ensuring the success of data processing projects and high-quality data output. This method is particularly suitable for data-intensive industries such as finance, healthcare, and e-commerce, where data quality directly affects the efficiency and effectiveness of business decisions and operations. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 The present invention is a flowchart of an information processing method based on big data.
[0019] Figure 2It is a schematic diagram of an information processing method based on big data of the present invention.
[0020] Figure 3 This is another schematic diagram of an information processing method based on big data of the present invention. DETAILED DESCRIPTION
[0021] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.
[0022] like Figure 1-3 In this embodiment, a method for information processing based on big data may specifically include:
[0023] Step S101, according to the data quality traceability chain, obtain data lineage information, including metadata of data source, processing, conversion and other links.
[0024] According to the data quality traceability chain, the data lineage relationship information is obtained, including metadata of data sources, processing, conversion and other links. The data lineage analysis algorithm is used to extract metadata from data sources such as databases, logs, and ETL tools through metadata extraction technology. Metadata includes information such as the source, conversion, processing, and storage of data. According to the extracted metadata, a data lineage map is constructed to show the dependency relationship between data in different links. The nodes in the data lineage map represent data, and the edges represent the conversion relationship between data. Through the graph algorithm, the data lineage map is analyzed to find data quality problems. The network embedding algorithm is used to map the nodes in the data lineage map to a low-dimensional vector space to obtain the vector representation of the data nodes. According to the vector representation of the data nodes, the similarity between the data nodes is calculated to determine the association relationship between the data nodes. If the similarity between the data nodes is lower than the threshold, it means that the association relationship between the data nodes is weak and there may be problems with the data quality. According to the association relationship between the data nodes, the community discovery algorithm is used to divide the data lineage map into multiple communities, each of which represents a group of closely related data nodes. By analyzing the attributes of the data nodes within the community, the quality problems of the data within the community can be determined. If the attributes of the data nodes within the community are inconsistent, it means that there is a problem with the quality of the data within the community. According to the attributes of the data nodes within the community, an anomaly detection algorithm is used to detect abnormal data nodes within the community. The attributes of the data nodes within the community are used as features to build an anomaly detection model, such as the isolation forest algorithm. Through the anomaly detection model, the degree of abnormality of each data node is calculated. Data nodes with high abnormality indicate that there may be quality problems. According to the abnormal data nodes, the root cause analysis algorithm is used to analyze the cause of the data quality problem. Starting from the abnormal data node, traverse the data lineage map, find the upstream node of the abnormal data node, analyze the attributes and conversion logic of the upstream node, and determine whether the quality problem of the abnormal data node is introduced by the upstream node or by the data conversion logic. According to the cause of the data quality problem, the data quality improvement algorithm is used to generate data quality improvement suggestions. If the data quality problem is introduced by the upstream node, data quality improvement suggestions such as data cleaning and data repair are generated. If the data quality problem is introduced by the data conversion logic, improvement suggestions for optimizing the data conversion logic are generated.
[0025] For example, by extracting metadata from database logs, it is found that a certain data table adds 1,000 records every day, accounting for 5% of the total data volume. It can be judged that the data volume of this data table is growing rapidly. For example, by using a network embedding algorithm to map 1,000 nodes in the data lineage map to a 100-dimensional vector space, and then calculating the similarity between nodes, it is found that the similarity of 5 pairs of data nodes is lower than the threshold of 5, indicating that their association is weak and the data quality may be problematic. For example, by using a community discovery algorithm to divide 10,000 data nodes into 20 communities, it is found that within a community of 500 nodes, 30% of the data nodes have a null value for a field, while the proportion of other communities is below 5%, indicating that the data quality of this community is poor. For example, by using an isolation forest algorithm to detect anomalies on 5,000 data nodes, it is found that 50 of the nodes have anomalies exceeding 6, which need to be investigated in detail. For example, after root cause analysis, it was found that the data quality problem of an abnormal data node was introduced by an upstream data processing node. The node had 500 data conversion logics, and a defect in one of the logics caused the abnormality of the downstream node. For example, data quality improvement suggestions include: cleaning 10% of the noise data in an upstream data table, fixing a bug in a data conversion logic to eliminate downstream outliers. These improvements can reduce the error rate of the entire data flow from 5% to less than 1%.
[0026] Step S102, through metadata analysis, determine the input and output data sets of each data processing link to form a data flow diagram.
[0027] According to the needs of metadata analysis, metadata extraction technology is used to obtain metadata information of data sources, including metadata such as data source type, data table, field, data format, etc. Metadata analysis technology is used to analyze the association relationship between metadata and construct a metadata association map. According to the metadata association map, data traceability analysis technology is used to analyze each link in the data processing process from bottom to top, and the input data set and output data set of each link are determined. On this basis, a data flow diagram is drawn to clearly show the flow process of the data set in the data processing process. For each data processing link in the data flow diagram, data feature analysis technology is used to analyze the data features of the input data set of this link, including data volume, data distribution, data quality, etc., so as to judge the data processing capability and efficiency of this link. At the same time, through data feature comparison analysis technology, the data feature differences between the input data set and the output data set of this link are compared to evaluate the data processing effect of this link. On the basis of the data flow diagram, data dependency analysis technology is used to analyze the dependency relationship between each data processing link. If the input data set of a link depends on the output of the upstream link, the corresponding dependency connection line is added in the data flow diagram. Through data dependency analysis, the key links and bottleneck links in the data processing process can be determined. According to the results of data dependency analysis, critical path analysis algorithms such as Huffman coding and Dijkstra algorithm are used to calculate the critical path of the data processing flow and determine the link that has the greatest impact on the efficiency of the entire process. For the links on the critical path, their data processing algorithms and implementation logic can be optimized to achieve the purpose of improving the efficiency of the entire process. Finally, the optimized data processing logic is applied to the data flow diagram to update the original data flow relationship. At the same time, data monitoring and data statistical analysis techniques are used to monitor and statistically analyze the optimized data processing process to obtain the optimized data processing efficiency index for evaluating the optimization effect. According to the results of data monitoring and statistical analysis, the data processing process can be further optimized, and the optimization results can be reflected in the data flow diagram to form a closed-loop continuous optimization process.
[0028] For example, by using metadata analysis technology, the association relationship between metadata is analyzed and a metadata association map is constructed. Assuming that data source A contains 100 data tables and data source B contains 80 data tables, after association analysis, it is found that there is an association relationship between 60 data tables, then a metadata association map containing 60 nodes can be constructed. For example, data feature analysis technology is used to analyze the data features of the input data set in the data processing link. Assuming that the input data set of a certain link contains 10 million records, the data distribution is normally distributed, and the data quality is 93%, it can be judged that the data processing capability of this link is strong and the data quality is high. For example, by using data feature comparison analysis technology, the data feature differences between the input data set and the output data set of the data processing link are compared. Assuming that the input data set contains 10 million records and the data quality is 95%; the output data set contains 8 million records and the data quality is 98%, it can be evaluated that the data processing effect of this link is good and the data quality is improved. For example, the critical path analysis algorithm is used to calculate the critical path of the data processing flow. Assuming that the data processing process includes 5 links, and the data processing time of each link is 30 minutes, 20 minutes, 45 minutes, 15 minutes and 25 minutes respectively, it can be calculated that the critical path is the first link, the third link and the fifth link, which have the greatest impact on the efficiency of the entire process. For example, data monitoring and data statistical analysis technology are used to monitor and statistically analyze the optimized data processing process. Assuming that the average time taken for the data processing process before optimization is 5 hours, and the average time taken for the data processing process after optimization is 2 hours, it can be evaluated that the optimization effect is good and the data processing efficiency is improved.
[0029] Step S103, using data quality monitoring rules, conduct quality checks on the input and output data of each link to identify data with quality problems.
[0030] According to the pre-defined data quality monitoring rules, the input and output data of each link are checked for quality. Data profiling technology is used to perform statistical analysis on the input data to obtain the basic characteristics and distribution of the data. Through data feature analysis, it is determined whether the data has quality problems such as missing values, outliers, and duplicate values. If the data has quality problems, it is marked as quality problem data, and the type and severity of the problem are recorded. According to the type and severity of the data quality problem, the corresponding data cleaning method is used to handle it. For missing values, mean filling, median filling, and highest frequency value filling can be used to fill them. For outliers, truncation method, box plot method and other methods can be used to identify and process. For duplicate values, deduplication algorithm can be used for deduplication. Through data cleaning, a corrected version of the quality problem data is obtained. The corrected data is compared with the original data, the data correction rate is calculated, the effect of data cleaning is evaluated, and a preset threshold is set. If the data correction rate reaches the preset threshold, the corrected data is used as the input data for the subsequent link. If the data correction rate does not reach the preset threshold, it is necessary to further analyze the cause of the data quality problem and adjust the data cleaning method. Recheck the quality of the corrected data and use data verification technology to verify whether the data meets the predefined data quality standards. Through data quality assessment, calculate the quality indicators such as data integrity, accuracy, and consistency to determine whether the data quality meets the standards. If the data quality meets the standards, it will be marked as qualified data and passed to the subsequent links. If the data quality does not meet the standards, it is necessary to further analyze the causes of the data quality problems and take appropriate measures to deal with them. Through data quality monitoring and data cleaning, a high-quality data set is obtained. The high-quality data set is used as the input for subsequent data analysis, data mining and other links to ensure the accuracy and reliability of data processing and analysis. At the same time, the data quality monitoring results and data cleaning process are recorded as the basis for data quality management and provide a reference for subsequent data quality improvement.
[0031] For example, in the tag: <result>For example, a data set containing 1,000 records is quality checked. Through data analysis, it is found that 80 records have missing values, accounting for 8%, with a medium severity. For example, for a batch of numerical data, the box plot method is used for outlier detection, and data less than 5 times the interquartile range (IQR) of the lower quartile Q1 and greater than 5 times the IQR of the upper quartile Q3 are marked as outliers. After processing, 20 outliers in the data set are removed. For example, after deduplication of a data set containing 5,000 records, 4,800 unique records are obtained, and the data correction rate is 96%, reaching the preset 95% threshold. For example, through data quality assessment, the completeness of a data set is calculated to be 98%, the accuracy is 95%, the consistency is 99%, and the comprehensive quality index is 95%, reaching the predefined 93% quality standard. For example, a statistical analysis of a data set containing 10,000 records shows that the mean of the data is 85, the standard deviation is 11, the median is 82, and the mode is 85, and the data presents a normal distribution.
[0032] Step S104, based on the quality problem data, trace back the data flow diagram to locate the upstream link that introduced the quality problem.
[0033] According to the quality problem data, the quality inspection records of each link in the product production process are obtained, and the data mining technology is used to mine the association rules of the quality problem data to obtain the association rules between the quality problem and the production link. The association rules are used to determine which production links the quality problem mainly comes from. According to the association rules between the quality problem and the production link, the production process data flow diagram is obtained, and the data flow diagram is used to determine which upstream links the quality problem data mainly comes from. If the quality problem data mainly comes from a certain upstream link, then this link is very likely to be the key link that introduces the quality problem. The production process parameters of the key link are obtained, and the support vector machine algorithm is used to establish a classification model between the production process parameters and the quality problem. The abnormal process parameter combination of the key link is determined by the classification model. If the production process parameters exceed the normal process range, then the batch of products is very likely to have quality defects. The raw material quality inspection data is obtained, and the principal component analysis method is used to reduce the dimension of the raw material quality data to obtain the principal component score of the raw material quality. The principal component score is used to determine whether the raw material quality is abnormal. If the raw material quality is abnormal, then the quality problem of this batch of products is very likely to be introduced by the raw material quality. Comprehensively analyze the production process parameters and raw material quality test results, and use the decision tree algorithm to establish a quality problem traceability model. The traceability model can be used to determine whether the quality problem is introduced by key process links or by abnormal raw material quality, thereby tracing the quality problem and locating the upstream link that introduces the quality problem.
[0034] For example, by mining association rules for 5,000 pieces of quality problem data and production link data, the following association rules are obtained: Assembly link → Quality problem (support = 30%, confidence = 75%) Welding link → Quality problem (support = 25%, confidence = 80%) Spraying link → Quality problem (support = 20%, confidence = 70%) It can be judged that quality problems mainly come from assembly link, welding link and spraying link. According to the production process data flow diagram analysis, the upstream links of the assembly link include parts processing link and outsourcing parts procurement link, the upstream links of the welding link include metal parts processing link, and the upstream link of the spraying link is the chemical raw material procurement link. By analyzing 500 pieces of production process parameter data of the assembly link, the support vector machine algorithm is used to establish a classification model. The model accuracy is 85%. According to the classification model, when the assembly gap parameter exceeds 5mm, the probability of unqualified component assembly is as high as 90%. Therefore, the assembly gap parameter is the key process parameter that introduces quality problems in the assembly link. The mechanical properties data of 1000 welded joints in the welding process were analyzed, and the principal component analysis method was used for dimensionality reduction. The variance contribution rate of the first principal component was 60%, and the variance contribution rate of the second principal component was 25%. When the first principal component score of the welded joint was lower than -5, the probability of unqualified welding quality was 95%. Therefore, the mechanical properties of the welding material are the main reason for the introduction of quality problems in the welding process. The quality inspection data of 1000 batches of chemical raw materials were analyzed, and the principal component analysis method was used for dimensionality reduction. The variance contribution rate of the first principal component was 70%. When the first principal component score of the chemical raw materials was lower than -2, 90% of the batches of products had quality problems. Therefore, the quality of chemical raw materials is the main reason for the introduction of quality problems in the spraying process. The production process parameters and raw material quality inspection data were comprehensively analyzed, and the decision tree algorithm was used to establish a quality problem traceability model. The model accuracy rate was 80%. According to the traceability model, among the products with quality problems, 60% were introduced by the assembly gap in the assembly process, 30% were introduced by welding material quality problems, and 10% were introduced by chemical raw material quality problems.
[0035] Step S105, obtaining the detailed processing logic and parameter configuration of the quality problem link, and analyzing the specific causes of the quality problem.
[0036] According to the quality data collected during the production process, data mining technology is used to analyze the quality problem links. By mining the association rules of the quality data of each link in the production process, the association relationship between the quality problem links is obtained. According to the results of the association rule mining, the key links that cause quality problems are determined. The process parameters of the key quality problem links are statistically analyzed, and the process parameters are reduced by the principal component analysis method to obtain the key process parameters that affect the quality problem. The threshold range of each process parameter is determined by clustering analysis of the threshold range of the key process parameters. If the process parameter exceeds the threshold range, it is determined to be the cause of the quality problem. According to the results of the quality problem cause analysis, the decision tree algorithm is used to classify the quality problems and obtain the specific process parameter combination that causes the quality problem. The detailed processing logic of the quality problem is obtained by analyzing the decision tree model. According to the branch conditions of each node in the decision tree model, the processing parameter configuration of the quality problem is determined. The processing parameter configuration of the quality problem is optimized, and the threshold parameters in the decision tree model are optimized by genetic algorithm to obtain the optimal quality problem processing parameter configuration. According to the optimal processing parameter configuration, the production process is adjusted and controlled to improve product quality and reduce the occurrence of quality problems.
[0037] For example, association rule mining: for example, by mining association rules on 1,000 production process quality data, it is found that when quality problems occur in the cutting link, there is an 85% probability that quality problems will occur in the grinding link, thereby determining that cutting and grinding are the key links that lead to quality problems. Principal component analysis: for example, by performing principal component analysis on 10 process parameters in the cutting link, the dimension is reduced to 3 principal components, and the cumulative contribution rate reaches 95%, thereby determining that cutting speed, cutting time, and number of cutting times are the key process parameters that affect cutting quality. Cluster analysis: for example, by performing cluster analysis on cutting speed, the optimal threshold interval of cutting speed is obtained to be 50 to 80 meters / minute. If the cutting speed is lower than 50 meters / minute or higher than 80 meters / minute, it is determined to be the cause of the cutting quality problem. Decision tree classification: For example, by using decision tree classification on 2,000 quality problem sample data, the following rules are obtained: when the cutting speed is lower than 50 m / min and the cutting time is greater than 10 seconds, the cut will be uneven; when the cutting speed is higher than 80 m / min and the number of cuts is more than 3 times, the cut will be burred. Genetic algorithm optimization: For example, by using genetic algorithm to optimize the three parameters of cutting speed, cutting time, and cutting times in the decision tree model, the optimal parameter configuration is obtained: cutting speed 65 m / min, cutting time 8 seconds, cutting times 2 times. When this configuration is applied to the production process, the product qualification rate is increased from the original 90% to 95%.
[0038] Step S106, formulate a plan for data repair and processing flow optimization based on the cause of the quality problem, and repair the problematic data.
[0039] According to the cause analysis of quality problems, the following data repair and processing flow optimization plan is formulated to repair the problem data: Through data quality assessment, a list of data quality problems is obtained. According to the list of data quality problems, the data exploration and analysis method is used to obtain the specific causes of data quality problems. According to different causes of data quality problems, corresponding data repair rules are designed. If there are missing values in the data, the maximum expectation algorithm is used for data completion; if there are outliers in the data, the local anomaly factor algorithm is used to detect and repair abnormal data; if there are inconsistencies in the data, inconsistent data is found through data association analysis, and data unification is performed. According to the data repair rules, the data cleaning and conversion algorithm is used to repair the problem data and obtain the repaired data set. Through data quality verification, it is determined whether the repaired data set meets the quality requirements. If it meets the requirements, the repaired data set is updated to the production environment; if it does not meet the requirements, the data is repaired again until it meets the quality requirements. In the data repair process, data version management is used to record the process and results of each data repair to ensure that the data repair process is traceable and reproducible. Based on the lessons learned from the data repair process, we will optimize the data processing process, formulate a data quality management system, standardize the operating procedures for data collection, storage, processing and other links, and improve data quality from the source. We will regularly conduct data quality assessment and data repair, normalize data quality management, and continuously improve data quality.
[0040] For example, through data quality assessment, a list of data quality problems is obtained. For example, the assessment found that 5% of the records in the data set have missing values, 2% of the records have outliers, and 1% of the records have inconsistencies. The data exploration and analysis method is used to obtain the specific causes of data quality problems. For example, the analysis found that the missing values are mainly concentrated in the age field, accounting for 8% of the total number of records in this field; outliers mainly appear in the income field, of which 20 records have negative income values; inconsistencies mainly appear in the address field, of which 15 records have address information that does not match other information. According to different causes of data quality problems, corresponding data repair rules are designed: 1. If there are missing values in the data, the maximum expectation algorithm is used to complete the data. For example, for the missing values in the age field, the average age of this part of the missing values is estimated to be 35 years old through the maximum expectation algorithm, and then 35 years old is used to fill the missing values of all age fields. 2. If there are outliers in the data, the local anomaly factor algorithm is used to detect and repair abnormal data. For example, 20 abnormal records in the income field are detected through the local anomaly factor algorithm, and then these abnormal values are corrected to the average value of the field, 8,000 yuan. 3 If there are inconsistencies in the data, the inconsistent data is found through data association analysis and the data is unified. For example, through association analysis, 15 inconsistent records in the address field are found. After verification, the address information of these records is updated to the correct value. According to the data repair rules, the data cleaning and conversion algorithm is used to repair the problem data to obtain the repaired data set. Through data quality verification, it is determined whether the repaired data set meets the quality requirements. For example, after quality verification, it is found that the missing value ratio of the repaired data set is reduced to less than 1%, the abnormal value ratio is reduced to less than 5%, and the inconsistency rate is reduced to less than 1%, which meets the data quality requirements. The repaired data set is updated to the production environment. In the process of data repair, data version management is used to record the process and results of each data repair to ensure that the data repair process is traceable and reproducible. Based on the lessons learned in the data repair process, the data processing process is optimized, the data quality management system is formulated, and the operating procedures of data collection, storage, processing and other links are standardized to improve data quality from the source. For example, a double verification rule for data entry is formulated, requiring that the key data entered must be independently verified by two people. If the verification results of the two people are inconsistent, a review is required. Regularly carry out data quality assessment and data repair, normalize data quality management, and continuously improve data quality. For example, a monthly data quality assessment mechanism has been established to conduct a comprehensive assessment of data quality every month and promptly repair any quality problems found. The results of data quality assessment for three consecutive months show that data quality continues to improve, with the data missing rate reduced from 5% to 1%, the outlier rate reduced from 5% to 2%, and the inconsistency rate reduced from 1% to 0.5%.
[0041] Step S107, update the optimized processing logic and parameter configuration to the corresponding links, and re-execute the data processing flow.
[0042] According to data quality analysis, the quality problems of noise data and outlier data are obtained; data cleaning and repair technology is used to process the noise data and outliers to obtain data with improved accuracy and integrity; more training sample data are obtained by rotating, translating and scaling the data with improved accuracy and integrity; through feature importance analysis, the key features that affect the prediction results are determined from the training sample data; according to the key features, the feature extraction process is optimized to obtain optimized features; new feature construction methods such as cross features and statistical features are introduced to obtain rich feature representations; according to the attention mechanism and residual connection technology, the model is improved. Feature extraction and fusion capabilities; use Adam and AdaGrad optimization algorithms to train the model to obtain a training model with improved convergence speed and training efficiency; use cross-validation method to evaluate the performance of the training model; use confusion matrix and ROC curve tools to analyze the prediction results of the training model; based on confidence assessment technology, quantify the uncertainty of the prediction results to obtain a reliable basis for decision-making; modularize and containerize the optimized end-to-end data processing process to improve the maintainability and scalability of the system; establish a model version management and update mechanism to obtain a continuously optimized and iterative data processing and prediction model.
[0043] For example, data quality analysis found that among 10,000 data, 5% of the data had noise, such as missing values or outliers. Data cleaning and repair technology was used to improve the accuracy of the data from 90% to 98%, and the completeness from 95% to 99% by filling missing values and smoothing outliers. Data enhancement was performed on the cleaned data, such as rotating the image data by 30 degrees, translating by 20 pixels, and scaling by 8 times, so that the number of training samples increased from 10,000 to 50,000. Through feature importance analysis, 20 key features with a greater impact on the prediction results were selected from 200 features. Based on the selected key features, the feature extraction process was optimized, such as improving the richness of the features, so that the prediction accuracy was increased from 85% to 90%. 10 cross features and 5 statistical features were introduced to increase the number of features from the original 20 to 35. The attention mechanism and residual connection technology were used to improve the model, so that the feature extraction and fusion capabilities were improved by 20%. The Adam optimization algorithm is used to train the model. Compared with the SGD optimization algorithm, the convergence speed is increased by 2 times and the training efficiency is increased by 50%. The 5-fold cross-validation method is used to evaluate the model performance, with an average accuracy of 95% and an average recall of 92%. 1 Through confusion matrix analysis, the prediction accuracy of the model on positive samples is 98%, and the prediction accuracy on negative samples is 96%; through ROC curve analysis, the AUC value reaches 99. 1 According to the confidence evaluation, the average confidence of the prediction results is 95%, and the proportion of samples with a confidence greater than 90% is 80%. 1 The data processing process is divided into five modules: data collection, data cleaning, feature engineering, model training, and model evaluation, and Docker container deployment is used to improve the maintainability and scalability of the system by 30%. 1 Establish a model version management mechanism, iterate a new version every week, and after continuous optimization for 3 months, the overall performance of the model is improved.
[0044] Step S108, perform a quality check on the repaired data to ensure that the problem has been resolved and the data quality meets the requirements.
[0045] The repaired data is quality checked to ensure that the problem has been solved and the data quality meets the requirements. First, the Pearson correlation coefficient analysis method is used to calculate the correlation between the various fields of the repaired data. The obtained correlation coefficient matrix is used to determine whether there is abnormal correlation between the data fields. If there are abnormally correlated fields, the data of these fields are rechecked and repaired. According to the data characteristics of the abnormal fields, the local abnormal factor detection algorithm is used to identify the abnormal values in these fields, and the identified abnormal data are eliminated or corrected. Then, by constructing a data quality evaluation index system, the weighted average comprehensive evaluation model is used to calculate the quality score of the repaired data, and the quality score is compared with the threshold. If the quality score is lower than the threshold, it indicates that the data still has quality problems, and it is necessary to further analyze the causes of the data quality problems and take measures to repair them; if the quality score is higher than the threshold, it indicates that the data quality meets the requirements. Finally, the Monte Carlo simulation method is used to test the robustness of the repaired data. Through multiple simulations, the robustness index value of the data is obtained, and the stability and reliability of the data quality are judged according to the robustness index value. If the robustness index value fluctuates greatly, it indicates that the data quality is unstable and further measures need to be taken to improve the robustness of the data quality; if the robustness index value fluctuates slightly, it indicates that the data quality is stable and reliable, and the repaired data can be used for subsequent analysis and application.
[0046] Exemplarily, the quality of the repaired data is checked to ensure that the problem has been solved and the data quality meets the requirements: the Pearson correlation coefficient analysis method is used to calculate the correlation between the various fields of the repaired data. For example, in the calculated correlation coefficient matrix, the correlation coefficient between field A and field B is 95, indicating that the two fields are highly correlated, and there may be data anomalies, which need to be checked in detail. According to the data characteristics of the abnormal fields, the local anomaly factor detection algorithm is used to identify the abnormal values in these fields. For example, among the 10,000 data in field C, 150 abnormal data are identified by the algorithm, accounting for 5%. After removing them, the correlation coefficient matrix is calculated again. By constructing a data quality assessment index system, a weighted average comprehensive evaluation model is used to calculate the quality score of the repaired data. For example, the index system contains 5 indicators, and the weights of each indicator are 3, 2, 2, 15, and 15 respectively. The scores of each indicator of the repaired data are 95, 90, 92, 96, and 97. The comprehensive quality score after weighted average is 985, which is higher than the preset threshold of 90, indicating that the data quality meets the requirements. The Monte Carlo simulation method is used to perform robustness testing on the repaired data. For example, after 1000 simulations, the mean of the data quality robustness indicator Cpk is 33, the standard deviation is 05, the Cpk values are all greater than 25, and the fluctuation is small, indicating that the data quality is robust after repair and can be used for subsequent analysis applications.
[0047] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.< / result>
Claims
1. An information processing method based on big data, characterized in that: The method comprises: According to the data quality traceability chain, obtain data lineage information, including metadata of data source, processing, and conversion links; Through metadata analysis, the input and output data sets of each data processing link are determined to form a data flow diagram; Adopt data quality monitoring rules to check the quality of input and output data at each stage and identify data with quality problems; Based on the quality problem data, trace back the data flow diagram to locate the upstream link that introduces the quality problem; Obtain the detailed processing logic and parameter configuration of the quality problem link, and analyze the specific causes of the quality problem; Develop data repair and processing flow optimization plans based on the causes of quality problems and repair problematic data; Update the optimized processing logic and parameter configuration to the corresponding links, and re-execute the data processing flow; Perform quality checks on the repaired data to ensure that the problem has been resolved and the data quality meets the requirements.
2. The method according to claim 1, wherein: According to the data quality traceability chain, the data lineage information is obtained, including metadata of data sources, processing, and conversion links, including: According to the data quality traceability chain, obtain data lineage information, including metadata of data sources, processing and conversion links; Using data lineage analysis algorithms and metadata extraction technology, metadata is extracted from databases, logs, and ETL tool data sources; Metadata includes information about the source, conversion, processing, and storage of data; Based on the extracted metadata, a data lineage map is constructed to show the dependency relationship between data in different links; The nodes in the data lineage graph represent data, and the edges represent the conversion relationship between data; Through graph algorithms, we analyze the data lineage graph and find data quality issues; Using the network embedding algorithm, the nodes in the data lineage graph are mapped into a low-dimensional vector space to obtain the vector representation of the data nodes; According to the vector representation of data nodes, the similarity between data nodes is calculated to determine the association relationship between data nodes; If the similarity between data nodes is lower than the threshold, it means that the association between data nodes is weak and there may be problems with data quality; According to the association relationship between data nodes, a community discovery algorithm is used to divide the data lineage graph into multiple communities, each of which represents a group of closely related data nodes; By analyzing the attributes of data nodes within the community, we can determine the quality issues of data within the community. If the attributes of the data nodes within the community are inconsistent, it means that there are problems with the quality of the data within the community; According to the attributes of the data nodes within the community, an anomaly detection algorithm is used to detect abnormal data nodes within the community; The attributes of the data nodes within the community are used as features to build an anomaly detection model using the isolation forest algorithm; The anomaly detection model is used to calculate the anomaly degree of each data node. Data nodes with high anomaly degrees indicate quality problems. Based on the abnormal data nodes, the root cause analysis algorithm is used to analyze the causes of data quality problems; Starting from the abnormal data node, traverse the data lineage graph, find the upstream node of the abnormal data node, analyze the attributes and conversion logic of the upstream node, and determine whether the quality problem of the abnormal data node is introduced by the upstream node or by the data conversion logic; According to the causes of data quality problems, data quality improvement algorithms are used to generate data quality improvement suggestions; If the data quality problem is introduced by the upstream node, generate data quality improvement suggestions for data cleaning or data repair; If the data quality problem is introduced by the data conversion logic, improvement suggestions for optimizing the data conversion logic are generated.
3. The method according to claim 1, wherein: The metadata analysis is used to determine the input and output data sets of each data processing link, and form a data flow diagram, including: According to the needs of metadata analysis, metadata extraction technology is used to obtain metadata information of the data source, where the metadata information of the data source includes data source type, data table, field and data format metadata; Through metadata analysis technology, the relationship between metadata is analyzed and a metadata association map is constructed; According to the metadata association map, data traceability analysis technology is used to analyze each link in the data processing process from bottom to top, and the input and output data sets of each link are determined; On this basis, draw a data flow diagram to clearly show the flow of data sets in the data processing process; For each data processing link in the data flow diagram, data feature analysis technology is used to analyze the data features of the input data set of this link, including data volume, data distribution and data quality, so as to judge the data processing capacity and efficiency of this link; At the same time, through data feature comparison and analysis technology, the data feature differences between the input data set and the output data set of this link are compared to evaluate the data processing effect of this link; Based on the data flow diagram, data dependency analysis technology is used to analyze the dependency relationship between various data processing links; If the input data set of a link depends on the output of the upstream link, add the corresponding dependency line in the data flow diagram; Through data dependency analysis, we can determine the key links and bottleneck links in the data processing process; According to the results of data dependency analysis, a critical path analysis algorithm is used, such as Huffman coding or Dijkstra algorithm, to calculate the critical path of the data processing flow and determine the link that has the greatest impact on the efficiency of the entire process; Focus on optimizing the data processing algorithms and implementation logic for the key links, so as to improve the efficiency of the entire process; Finally, the optimized data processing logic is applied to the data flow diagram to update the original data flow relationship; At the same time, data monitoring and data statistical analysis technology are used to monitor and statistically analyze the optimized data processing process to obtain the optimized data processing efficiency index for evaluating the optimization effect; Based on the results of data monitoring and statistical analysis, the data processing process is further optimized, and the optimization results are reflected in the data flow diagram, forming a closed-loop continuous optimization process.
4. The method according to claim 1, wherein: The data quality monitoring rules are used to check the quality of input and output data in each link and identify data with quality problems, including: According to the pre-defined data quality monitoring rules, the input and output data of each link are quality checked; Use data profiling techniques to perform statistical analysis on input data to obtain the basic characteristics and distribution of the data; By analyzing data characteristics, determine whether the data has quality problems, including missing values, outliers, and duplicate values; If the data has quality problems, it will be marked as quality problem data, and the type and severity of the problem will be recorded; According to the type and severity of data quality issues, use corresponding data cleaning methods to deal with them; For missing values, fill them in. The filling method can be one of mean filling, median filling and highest frequency value filling; For outliers, the truncation method and box plot method were used for identification and processing; For duplicate values, a deduplication algorithm is used to remove duplicate values; Through data cleaning, a revised version of the quality problem data is obtained; Compare the corrected data with the original data, calculate the data correction rate, evaluate the effect of data cleaning, and set the preset threshold; If the data correction rate reaches the preset threshold, the corrected data will be used as input data for subsequent links; If the data correction rate does not reach the preset threshold, further analyze the cause of the data quality problem and adjust the data cleaning method; Recheck the quality of the corrected data and use data verification technology to verify whether the data meets the predefined data quality standards; Through data quality assessment, the quality indicators of the data are calculated to determine whether the data quality meets the standards; the quality indicators include the completeness, accuracy and consistency of the data; If the data quality meets the standard, it will be marked as qualified data and passed to the subsequent links; If the data quality does not meet the standards, further analyze the causes of the data quality issues and take appropriate measures to deal with them; Obtain high-quality data sets through data quality monitoring and data cleaning; Use high-quality data sets as input for subsequent data analysis and data mining to ensure the accuracy and reliability of data processing and analysis; At the same time, the data quality monitoring results and data cleaning process are recorded as the basis for data quality management and provide a reference for subsequent data quality improvements.
5. The method according to claim 1, wherein: The method of tracing back the data flow diagram based on the quality problem data to locate the upstream link that introduces the quality problem includes: Based on the quality problem data, we obtain the quality inspection records of each link in the product production process, and use data mining technology to mine the association rules of the quality problem data to obtain the association rules between quality problems and production links; Use association rules to determine which production links the quality problems mainly come from; According to the association rules between quality problems and production links, obtain the data flow diagram of the production process, and use the data flow diagram to determine which upstream links the quality problem data mainly comes from; If the quality problem data mainly comes from a certain upstream link, then this link is the key link that introduces the quality problem; Obtain the production process parameters of key links, use the support vector machine algorithm to establish a classification model between production process parameters and quality issues, and use the classification model to determine the abnormal process parameter combination of key links; If the production process parameters are outside the normal process range, the batch of products will have quality defects; Obtain raw material quality inspection data, use principal component analysis to reduce the dimension of raw material quality data, obtain the principal component score of raw material quality, and judge whether the raw material quality is abnormal based on the principal component score; If the quality of raw materials is abnormal, the quality problem of this batch of products is introduced by the quality of raw materials; Comprehensively analyze the production process parameters and raw material quality test results, and use the decision tree algorithm to establish a quality problem traceability model. The traceability model can be used to determine whether the quality problem is introduced by key process links or by abnormal raw material quality, thereby tracing the quality problem and locating the upstream link that introduces the quality problem.
6. The method according to claim 1, wherein: The detailed processing logic and parameter configuration of the quality problem acquisition link and the analysis of the specific causes of the quality problems include: Based on the quality data collected during the production process, data mining technology is used to analyze the quality problem links; By mining association rules on the quality data of each link in the production process, the association relationship between the links of quality problems can be obtained; According to the results of association rule mining, determine the key links that lead to quality problems; Conduct statistical analysis on process parameters of key quality issues, use principal component analysis to reduce the dimension of process parameters, and obtain key process parameters that affect quality issues; By clustering the threshold ranges of key process parameters, the threshold intervals of each process parameter can be determined; If the process parameters exceed the threshold range, it is determined to be the cause of the quality problem; According to the results of the quality problem cause analysis, the decision tree algorithm is used to classify the quality problems and obtain the specific process parameter combination that causes the quality problems; Obtain detailed processing logic of quality issues through analysis of decision tree models; Determine the processing parameter configuration of quality issues based on the branch conditions of each node in the decision tree model; Optimize the configuration of quality problem processing parameters, use genetic algorithm to optimize the threshold parameters in the decision tree model, and obtain the optimal quality problem processing parameter configuration; According to the optimal processing parameter configuration, the production process is adjusted and controlled to improve product quality and reduce the occurrence of quality problems.
7. The method according to claim 1, wherein: According to the causes of quality problems, a plan for data repair and process optimization is formulated to repair the problematic data, including: According to the cause analysis of quality problems, formulate the following data repair and processing flow optimization plan to repair the problematic data: obtain a list of data quality problems through data quality assessment; According to the list of data quality problems, use data exploration and analysis methods to obtain the specific causes of data quality problems; Design corresponding data repair rules according to different causes of data quality problems; If there are missing values in the data, the maximum expectation algorithm is used to complete the data; If there are outliers in the data, the local outlier factor algorithm is used to detect and repair the outlier data; If there is inconsistency in the data, find out the inconsistent data through data association analysis and unify the data; According to the data repair rules, the data cleaning and conversion algorithms are used to repair the problematic data and obtain the repaired data set; Through data quality verification, determine whether the repaired data set meets the quality requirements; If it meets the requirements, the repaired dataset will be updated to the production environment; If it does not meet the requirements, the data will be repaired again until it meets the quality requirements; During the data repair process, data version management is used to record the process and results of each data repair to ensure that the data repair process is traceable and reproducible; Based on the experience gained in the data repair process, we optimized the data processing process, formulated a data quality management system, standardized the operating procedures for data collection, storage and processing, and improved data quality from the source; Carry out data quality assessment and data repair regularly, normalize data quality management, and continuously improve data quality.
8. The method according to claim 1, wherein: The optimized processing logic and parameter configuration are updated to the corresponding links, and the data processing flow is re-executed, including: According to data quality analysis, obtain noise data and outlier data quality issues; Using data cleaning and repair technology to process the noise data and outliers to obtain data with improved accuracy and completeness; Obtaining a larger amount of training sample data by rotating, translating and scaling the data with improved accuracy and completeness; Determining key features that affect prediction results from the training sample data through feature importance analysis; According to the key features, the feature extraction process is optimized to obtain optimized features; Introduction New feature construction methods of cross-features and statistical features to obtain rich feature representations; Improve the model's feature extraction and fusion capabilities based on the attention mechanism and residual connection technology; The model is trained using Adam and AdaGrad optimization algorithms to obtain a training model with improved convergence speed and training efficiency; Using a cross-validation method to evaluate the performance of the training model; Analyze the prediction results of the training model through confusion matrix and ROC curve tools; According to the confidence assessment technology, the uncertainty of the prediction results is quantified to obtain a reliable basis for decision-making; Modularize and containerize the optimized end-to-end data processing process to improve the maintainability and scalability of the system; Establish a model version management and update mechanism to obtain continuously optimized and iterative data processing and prediction models.
9. The method according to claim 1, wherein: The repaired data is quality checked to ensure that the problem has been resolved and the data quality meets the requirements, including: Perform quality checks on the repaired data to ensure that the problem has been resolved and the data quality meets the requirements; First, the Pearson correlation coefficient analysis method is used to calculate the correlation between the various fields of the repaired data. Through the obtained correlation coefficient matrix, it is determined whether there is an abnormal correlation between the data fields; If there are abnormal related fields, recheck and repair the data of these fields; According to the data characteristics of abnormal fields, a local anomaly factor detection algorithm is used to identify abnormal values in these fields, and the identified abnormal data is removed or corrected; Then, by constructing a data quality assessment index system, a weighted average comprehensive evaluation model is used to calculate the quality score of the repaired data, and the quality score is compared with the threshold. If the quality score is lower than the threshold, it indicates that there are still quality problems in the data, and it is necessary to further analyze the causes of the data quality problems and take measures to repair them. If the quality score is higher than the threshold, it means that the data quality meets the requirements; Finally, the Monte Carlo simulation method is used to test the robustness of the repaired data. Through multiple simulations, the robustness index value of the data is obtained, and the stability and reliability of the data quality are judged according to the robustness index value; If the robustness index value fluctuates greatly, it indicates that the data quality is unstable and further measures need to be taken to improve the robustness of the data quality; If the robustness index value fluctuates slightly, it indicates that the data quality is stable and reliable, and the repaired data can be used for subsequent analysis and application.
Citation Information
Patent Citations
Method and device for tracing data quality and hardware processor
CN105844390A
Data information processing method and apparatus
CN105868521A
Data blood relationship sorting and monitoring method
CN116955503A
Data quality inspection method based on artificial intelligence
CN117951128A
Data quality analysis method and system
CN118113689A
Cited By
Human umbilical cord blood stem cell detection and screening system
CN119432581A
IT work order risk management and control method based on combination of large and small models
CN119904109A
Data management method and system for digital government affairs
CN120508554A
A data governance method and system for digital government affairs
CN120508554B
Multi-source heterogeneous data analysis method and system based on machine learning and electronic equipment
CN120804597A
