A Machine Learning-Based Processing Data Quality Monitoring System and Method

Abstract

This invention discloses a machine learning-based data quality monitoring system and method, belonging to the field of data analysis technology. The method includes the following steps: acquiring multi-source processed data and sorting it by time; performing null value and reasonableness checks on each data value and marking them accordingly; analyzing the data vector using the Isolation Forest algorithm, outputting a global anomaly score and decomposing it into local scores, marking them as machine learning anomalies; selecting a random forest regression model or an LSTM model to predict replacement values ​​based on the anomaly markings; performing range verification on the replacement values, deciding whether to replace them or submit them for manual review based on the anomaly type; calculating the residual between the processed data value and the replacement value, fitting Gaussian components using a Gaussian Model (GMM) and assigning semantics, defining the processing method based on cluster semantics and membership degrees. This method can effectively improve the situation in existing technologies where anomaly data may be misjudged and corrected data may still be anomaly.

Description

Technical Field

[0001] This invention relates to the field of data analysis technology, specifically a processing data quality monitoring system and method based on machine learning. Background Technology

[0002] In the field of intelligent manufacturing, processing data quality inspection is a crucial link in ensuring product quality and production efficiency. Traditional inspection methods mainly rely on single rule verification or statistical analysis, such as setting fixed threshold ranges to determine whether parameters are abnormal. With the development of the Industrial Internet of Things (IIoT), multi-source sensor data (such as equipment operating parameters and environmental indicators) and time-series characteristics have placed higher demands on quality inspection, prompting the introduction of machine learning technology to identify complex anomalies. However, existing solutions still have limitations in terms of the accuracy of anomaly localization, multi-parameter correlation analysis, and the adaptability of repair strategies, making it difficult to meet the needs of high-precision quality control.

[0003] Existing technologies typically employ static strategies in the data correction stage, such as mean imputation, linear interpolation, or single-model prediction of replacement values, without fully considering the dynamic correlations and temporal characteristics of parameters. This leads to the possibility that the corrected data may still contain potential anomalies: on the one hand, when multiple related parameters in the original data are abnormal, a simple collaborative correction model may introduce new biases; on the other hand, machine learning models trained on historical data struggle to cope with sudden changes in production processes, causing predicted replacement values ​​to deviate from the true values. Furthermore, the post-correction verification stage often lacks or relies solely on basic rule validation, failing to identify outliers that fall within the acceptable range but do not conform to production logic. This allows defective data to flow into subsequent analysis processes, impacting the accuracy of decision-making. Summary of the Invention

[0004] The purpose of this invention is to provide a processing data quality monitoring system and method based on machine learning to solve the problems raised in the prior art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a method for monitoring the quality of processed data based on machine learning, the method comprising the following steps:

[0006] Step 1: Obtain multi-source processing data and sort it by time. Perform null value and reasonableness checks on each data value and mark it accordingly.

[0007] Step 2: Use the Isolation Forest algorithm to analyze the data vector, output the global anomaly score and decompose it into local scores, and mark them as machine learning anomalies;

[0008] Step 3: Select either the random forest regression model or the LSTM model to predict replacement values ​​based on the anomaly labeling.

[0009] Step 4: Perform range validation on the replacement values, and decide whether to replace them or submit them for manual review based on the type of anomaly.

[0010] Step 5: Calculate the residual between the processed data value and the replacement value, fit the Gaussian component with GMM and assign semantics, and define the processing method according to the cluster semantics and membership degree.

[0011] In step 1, historical processing data is obtained through equipment operation sensors, environmental monitoring sensors, or from the production management system, and is represented as [P1, P2, ..., P]. n Sort in chronological order;

[0012] Where P1, P2, ..., P n This represents the processing data for the 1st, 2nd, ..., nth type;

[0013] At time t i Data vector X(t) i ) is represented as: X(t) i )=[P1(t i ),P2(t i ),…,P n (t i )];

[0014] Among them, t i P1(t) represents the i-th time point; i ),P2(t i ),…,P n (t i ) represents t i Processing data P1, P2, ..., P corresponding to time points n The value;

[0015] Perform null value detection on each processed data value, and mark it as a null value if it is detected.

[0016] The reasonableness of each processing data value is checked based on a preset range. When the processing data value is not within the preset range, it is marked as abnormal.

[0017] In step 2, when the processed data value is within a preset range, the isolated forest algorithm is used to process the data vector X(t). i Perform anomaly analysis and output a global anomaly score s. i ;

[0018] Through feature importance analysis, the global anomaly score s i Decomposed into each parameter P j (t i ), thus obtaining the local anomaly score s j (t iBased on feature importance weights, the global anomaly score is proportionally allocated to each parameter;

[0019] Where j∈{1,2,…,n}; P j (t i ) represents t i Processing data P corresponding to time moment j Value, s j (t i ) represents t i Real-time processing data P j The corresponding local anomaly score;

[0020] Set threshold T s When s j (t i )>T s At that time, machine learning anomaly labeling is performed.

[0021] In step 3, when the processing data value P j (t i ) is marked as null, anomaly, or machine learning anomaly and the data vector X(t) i Except for P j (t i When all other values ​​are not marked as null, anomalous, or machine learning anomalies, calculate P. j Correlation with other processing data, setting thresholds and selecting a subset of relevant parameters {P} a}; where a≠j, a∈{1,2,…,n};

[0022] Construct a random forest regression model, with P k (t i Given the input, predict the replacement value B. j (t i );

[0023] When the processing data value P j (t i ) is marked as null, anomaly, or machine learning anomaly and the data vector X(t) i Except for P j (t i When any of the remaining values ​​are marked as null, anomaly, or machine learning anomaly, or when P j When the correlation with all other processing data does not meet the set threshold requirement, the P-based LSTM time series model is used. j Sequence modeling alone, predicting replacement value B j (t i ).

[0024] In step 4, the generated replacement value B is... j (ti Perform secondary rule validation:

[0025] When B j (t i When not within the preset range:

[0026] When the processing data value P j (t i If a value is marked as null or abnormal, it will be submitted for manual review.

[0027] When the processing data value P j (t i When a P-value is marked as a machine learning anomaly, retain P. j (t i )value;

[0028] In step 5, when B j (t i When the value is within the preset range: Calculate the replacement value B. j (t i ) and the processing data value P j (t i The residual r j (t i ):r j (t i )=|B j (t i )-P j (t i When the processed data value is empty, fill it with the most recent value that is within the preset range, and then calculate the residual.

[0029] Collect anomaly scores for corresponding processed data values ​​from historical data. j (t i and the corresponding residual r j (t i Manual labeling: normal, abnormal, or suspicious;

[0030] Normal indicates that, after manual verification, the replacement value is more reliable than the processing data value and can be directly replaced; Abnormal indicates that, after manual verification, the replacement value is unreliable and the processing data value should be retained, i.e., the processing data value that does not need correction; Suspicious indicates that further verification is required.

[0031] Form a two-dimensional training set {(s(m),r(m))} m=1 M Where M represents the number of corresponding processed data values, m∈{1,2,…,M};

[0032] In two-dimensional space, with {(s(m),r(m))} as input, a Gaussian mixture model (GMM) is used to fit K Gaussian components; for each component k, the semantics of normal cluster, suspicious cluster, or anomalous cluster are assigned based on the label with the most internal number.

[0033] After generating the new replacement value, obtain its corresponding (s,r) and calculate the posterior membership vector. ;

[0034] Where, π k N([s,r]|μ) represents the weight of the k-th Gaussian component; k ,Σ k ) represents the probability density function of a two-dimensional Gaussian distribution; μ k Σ represents the mean vector of the k-th Gaussian component; k Let represent the covariance matrix of the k-th Gaussian component; l represents the index variable in the summation symbol Σ, which iterates through all K Gaussian components;

[0035] Find the maximum posterior cluster k* = arg max k γ k and its membership degree γ k* ;

[0036] Based on the cluster semantics and membership strength corresponding to k*: when the cluster semantics corresponding to k* is a normal cluster and γ k* When the value exceeds the preset normal threshold, a replacement value is used; when the cluster semantics corresponding to k* are abnormal clusters and γ k* If the value exceeds the preset abnormal threshold, the processed data value is retained; otherwise, it is submitted for manual review.

[0037] A machine learning-based data quality monitoring system includes a data preprocessing module, an anomaly detection module, a replacement prediction module, and a verification and auditing module.

[0038] The data preprocessing module acquires multi-source processed data and sorts it by time. It performs null and reasonableness checks on each data value and marks it accordingly. The anomaly detection module analyzes the data vector using the Isolation Forest algorithm, outputs a global anomaly score, decomposes it into local scores, and marks them as machine learning anomalies. The replacement prediction module selects either a random forest regression model or an LSTM model to predict replacement values ​​based on the anomaly markings. The verification and review module performs range verification on the replacement values ​​and decides whether to replace them or submit them for manual review based on the anomaly type. It calculates the residual between the processed data value and the replacement value, fits Gaussian components using a Gaussian Model (GMM) and assigns semantics, defining the processing method based on cluster semantics and membership degree.

[0039] The output of the data preprocessing module is connected to the input of the anomaly detection module; the output of the anomaly detection module is connected to the input of the replacement prediction module; and the output of the replacement prediction module is connected to the input of the verification and auditing module.

[0040] The data preprocessing module includes a data acquisition unit and a sorting and detection unit;

[0041] The data acquisition unit is used to acquire multi-source data from equipment sensors or production systems; the sorting and detection unit is used to generate data vectors by sorting them by time, perform null value detection and rationality detection, and mark outliers.

[0042] The output of the data acquisition unit is connected to the input of the sorting and detection unit; the output of the sorting and detection unit is connected to the input of the anomaly detection module.

[0043] The anomaly detection module includes a global analysis unit and a local decomposition unit;

[0044] The global analysis unit is used to calculate the global anomaly score using the isolated forest algorithm; the local decomposition unit is used to decompose the global anomaly score into specific data based on feature importance, and perform machine learning anomaly labeling based on thresholds.

[0045] The output of the global analysis unit is connected to the input of the local decomposition unit; the output of the local decomposition unit is connected to the input of the replacement prediction module.

[0046] The replacement prediction module includes a model selection unit and a prediction execution unit;

[0047] The model selection unit is used to select a random forest or LSTM model based on the anomaly labeling; the prediction execution unit is used to execute the selected model to generate replacement values.

[0048] The output of the model selection unit is connected to the input of the prediction execution unit; the output of the prediction execution unit is connected to the input of the verification and auditing module.

[0049] The verification and auditing module includes a rule verification unit, a residual clustering unit, and a decision execution unit;

[0050] The rule verification unit is used to perform a second verification on the replacement value within a preset range; the residual clustering unit is used to calculate the residual between the processed data value and the replacement value, construct the GMM model and assign semantic labels; the decision execution unit is used to decide whether to replace, retain or manually review based on the cluster semantics and membership degree.

[0051] The output of the rule verification unit is connected to the input of the residual clustering unit; the output of the residual clustering unit is connected to the input of the decision execution unit.

[0052] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention combines rule detection (preset range) and machine learning (isolated forest) to achieve multi-level anomaly recognition from single parameters to global vectors, thereby improving detection accuracy; This invention uses Gaussian mixture model (GMM) to perform posterior verification of the repair results, and combines anomaly scores and residuals to construct semantic clusters, thereby reducing erroneous repairs and lowering the cost of manual review; This invention provides differentiated processing strategies for different anomaly scenarios (single-parameter anomalies, multi-parameter anomalies), especially adopting collaborative repair for highly correlated parameters and using time-series models for data with strong time-series characteristics. Attached Figure Description

[0053] Figure 1 This is a schematic diagram illustrating the steps of a machine learning-based method for monitoring the quality of processed data according to the present invention.

[0054] Figure 2 This is a flowchart illustrating a machine learning-based processing data quality monitoring system according to the present invention. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] Example: Figures 1-2 As shown, the present invention provides a technical solution, a method for monitoring the quality of processed data based on machine learning, the method comprising the following steps:

[0057] Step 1: Obtain multi-source processing data and sort it by time. Perform null value and reasonableness checks on each data value and mark it accordingly.

[0058] Step 2: Use the Isolation Forest algorithm to analyze the data vector, output the global anomaly score and decompose it into local scores, and mark them as machine learning anomalies;

[0059] Step 3: Select either the random forest regression model or the LSTM model to predict replacement values ​​based on the anomaly labeling.

[0060] Step 4: Perform range validation on the replacement values, and decide whether to replace them or submit them for manual review based on the type of anomaly.

[0061] Step 5: Calculate the residual between the processed data value and the replacement value, fit the Gaussian component with GMM and assign semantics, and define the processing method according to the cluster semantics and membership degree.

[0062] In step 1, historical processing data is obtained through equipment operation sensors, environmental monitoring sensors, or from the production management system, and is represented as [P1, P2, ..., P]. n Sort in chronological order;

[0063] Where P1, P2, ..., P n This represents the processing data for the 1st, 2nd, ..., nth type;

[0064] At time t i Data vector X(t) i ) is represented as: X(t) i )=[P1(t i ),P2(t i ),…,P n (t i )];

[0065] Among them, t i P1(t) represents the i-th time point; i ),P2(t i ),…,P n (t i ) represents t i Processing data P1, P2, ..., P corresponding to time points n The value;

[0066] Perform null value detection on each processed data value, and mark it as a null value if it is detected.

[0067] The reasonableness of each processing data value is checked based on a preset range. When the processing data value is not within the preset range, it is marked as abnormal.

[0068] In step 2, when the processed data value is within a preset range, the isolated forest algorithm is used to process the data vector X(t). i Perform anomaly analysis and output a global anomaly score s. i ;

[0069] Through feature importance analysis, the global anomaly score s i Decomposed into each parameter P j (t i ), thus obtaining the local anomaly score s j (t i Based on feature importance weights, the global anomaly score is proportionally allocated to each parameter;

[0070] Where j∈{1,2,…,n}; Pj (t i ) represents t i Processing data P corresponding to time moment j Value, s j (t i ) represents t i Real-time processing data P j The corresponding local anomaly score;

[0071] Set threshold T s When s j (t i )>T s At that time, machine learning anomaly labeling is performed.

[0072] In step 3, when the processing data value P j (t i ) is marked as null, anomaly, or machine learning anomaly and the data vector X(t) i Except for P j (t i When all other values ​​are not marked as null, anomalous, or machine learning anomalies, calculate P. j Correlation with other processing data, setting thresholds and selecting a subset of relevant parameters {P} a}; where a≠j, a∈{1,2,…,n};

[0073] Construct a random forest regression model, with P k (t i Given the input, predict the replacement value B. j (t i );

[0074] When the processing data value P j (t i ) is marked as null, anomaly, or machine learning anomaly and the data vector X(t) i Except for P j (t i When any of the remaining values ​​are marked as null, anomaly, or machine learning anomaly, or when P j When the correlation with all other processing data does not meet the set threshold requirement, the P-based LSTM time series model is used. j Sequence modeling alone, predicting replacement value B j (t i ).

[0075] In step 4, the generated replacement value B is... j (t i Perform secondary rule validation:

[0076] When B j (t iWhen not within the preset range:

[0077] When the processing data value P j (t i If a value is marked as null or abnormal, it will be submitted for manual review.

[0078] When the processing data value P j (t i When a P-value is marked as a machine learning anomaly, retain P. j (t i )value;

[0079] In step 5, when B j (t i When the value is within the preset range: Calculate the replacement value B. j (t i ) and the processing data value P j (t i The residual r j (t i ):r j (t i )=|B j (t i )-P j (t i )|;

[0080] Collect anomaly scores for corresponding processed data values ​​from historical data. j (t i and the corresponding residual r j (t i Manual labeling: normal, abnormal, or suspicious;

[0081] Normal indicates that, after manual verification, the replacement value is more reliable than the processing data value and can be directly replaced; Abnormal indicates that, after manual verification, the replacement value is unreliable and the processing data value should be retained, i.e., the processing data value that does not need correction; Suspicious indicates that further verification is required.

[0082] Form a two-dimensional training set {(s(m),r(m))} m=1 M Where M represents the number of corresponding processed data values, m∈{1,2,…,M};

[0083] In two-dimensional space, with {(s(m),r(m))} as input, a Gaussian mixture model (GMM) is used to fit K Gaussian components; for each component k, the semantics of normal cluster, suspicious cluster, or anomalous cluster are assigned based on the label with the most internal number.

[0084] After generating the new replacement value, obtain its corresponding (s,r) and calculate the posterior membership vector. ;

[0085] Where, π k N([s,r]|μ) represents the weight of the k-th Gaussian component; k ,Σ k ) represents the probability density function of a two-dimensional Gaussian distribution; μ k Σ represents the mean vector of the k-th Gaussian component; k Let represent the covariance matrix of the k-th Gaussian component; l represents the index variable in the summation symbol Σ, which iterates through all K Gaussian components;

[0086] Find the maximum posterior cluster k* = arg max k γ k and its membership degree γ k* ;

[0087] Based on the cluster semantics and membership strength corresponding to k*: when the cluster semantics corresponding to k* is a normal cluster and γ k* When the value exceeds the preset normal threshold, a replacement value is used; when the cluster semantics corresponding to k* are abnormal clusters and γ k* If the value exceeds the preset abnormal threshold, the processed data value is retained; otherwise, it is submitted for manual review.

[0088] A machine learning-based data quality monitoring system includes a data preprocessing module, an anomaly detection module, a replacement prediction module, and a verification and auditing module.

[0089] The data preprocessing module acquires multi-source processed data and sorts it by time. It performs null and reasonableness checks on each data value and marks it accordingly. The anomaly detection module analyzes the data vector using the Isolation Forest algorithm, outputs a global anomaly score, decomposes it into local scores, and marks them as machine learning anomalies. The replacement prediction module selects either a random forest regression model or an LSTM model to predict replacement values ​​based on the anomaly markings. The verification and review module performs range verification on the replacement values ​​and decides whether to replace them or submit them for manual review based on the anomaly type. It calculates the residual between the processed data value and the replacement value, fits Gaussian components using a Gaussian Model (GMM) and assigns semantics, defining the processing method based on cluster semantics and membership degree.

[0090] The output of the data preprocessing module is connected to the input of the anomaly detection module; the output of the anomaly detection module is connected to the input of the replacement prediction module; and the output of the replacement prediction module is connected to the input of the verification and auditing module.

[0091] The data preprocessing module includes a data acquisition unit and a sorting and detection unit;

[0092] The data acquisition unit is used to acquire multi-source data from equipment sensors or production systems; the sorting and detection unit is used to generate data vectors by sorting them by time, perform null value detection and rationality detection, and mark outliers.

[0093] The output of the data acquisition unit is connected to the input of the sorting and detection unit; the output of the sorting and detection unit is connected to the input of the anomaly detection module.

[0094] The anomaly detection module includes a global analysis unit and a local decomposition unit;

[0095] The global analysis unit is used to calculate the global anomaly score using the isolated forest algorithm; the local decomposition unit is used to decompose the global anomaly score into specific data based on feature importance, and perform machine learning anomaly labeling based on thresholds.

[0096] The output of the global analysis unit is connected to the input of the local decomposition unit; the output of the local decomposition unit is connected to the input of the replacement prediction module.

[0097] The replacement prediction module includes a model selection unit and a prediction execution unit;

[0098] The model selection unit is used to select a random forest or LSTM model based on the anomaly labeling; the prediction execution unit is used to execute the selected model to generate replacement values.

[0099] The output of the model selection unit is connected to the input of the prediction execution unit; the output of the prediction execution unit is connected to the input of the verification and auditing module.

[0100] The verification and auditing module includes a rule verification unit, a residual clustering unit, and a decision execution unit;

[0101] The rule verification unit is used to perform a second verification on the replacement value within a preset range; the residual clustering unit is used to calculate the residual between the processed data value and the replacement value, construct the GMM model and assign semantic labels; the decision execution unit is used to decide whether to replace, retain or manually review based on the cluster semantics and membership degree.

[0102] The output of the rule verification unit is connected to the input of the residual clustering unit; the output of the residual clustering unit is connected to the input of the decision execution unit.

[0103] In this embodiment, an automotive parts processing plant manufactures engine blocks. During production, several key parameters need to be monitored in real time, including machining temperature, cutting force, spindle speed, and feed rate, to ensure product quality and production efficiency. The following is a detailed implementation process of the processing plant strictly adhering to a machine learning-based machining data quality monitoring method for full-process quality monitoring of production data:

[0104] Acquire and preprocess multi-source machining data: Historical machining data from the past week is obtained through equipment operation sensors, environmental monitoring sensors, and the production management system. The sensors collect data every 10 minutes. The collected machining data includes machining temperature (P1), cutting force (P2), spindle speed (P3), and feed rate (P4), which are represented as a data sequence [P1,P2,P3,P4].

[0105] All acquired data are sorted chronologically according to their acquisition time to ensure the continuity of the data over time. At the 10th recording time t10, a data vector X(t) is formed. 10 )=[P1(t 10 ),P2(t 10 ),P3(t 10 ),P4(t 10 )], where P1(t 10 ) represents t 10 The processing temperature value at time t, P2(t) 10 P3(t) 10 P4(t) 10 ) represent the values ​​of the corresponding parameters at that moment.

[0106] Null value checks are performed on each processed data value individually, using null detection functions in data processing software or programming languages, such as the `isnull()` function in Python. The checks revealed null values ​​for t. 10 Cutting force P2(t) at time t 10 If a value is null, immediately mark the data as null and add a special marker field to the data record.

[0107] Based on the pre-set normal range of parameters, the rationality of each machining data value is checked. The specific pre-set ranges are: machining temperature 80-120℃, cutting force 50-80N, spindle speed 1000-2000 rpm, and feed rate 10-30 mm / min. When t is detected... 15 Processing temperature P1(t) at time 15 The temperature was 75℃, which was outside the preset processing temperature range, and an anomaly was immediately marked.

[0108] Analyzing data vectors using the Isolation Forest algorithm: For data within a preset range, t20 The data vector X(t) at time t 20 Taking a data point as an example, we apply the Isolation Forest algorithm for anomaly analysis. The Isolation Forest algorithm constructs multiple isolated trees, using the length of the isolated path of a data point within the trees as a measure of anomaly severity, and ultimately outputs a global anomaly score s. 20 .

[0109] Through feature importance analysis, the global anomaly score s 20 This is decomposed into each parameter. Specifically, a tree-based feature importance calculation method is used, such as calculating the average gain of each feature's contribution to sample partitioning across all isolated trees, thus obtaining t. 20 The local anomaly score s1(t) corresponding to the processing temperature P1 at time step 20 ).

[0110] Set threshold T s The threshold is set to 0.6, and the local anomaly score for each parameter is compared to this threshold. After calculation, s1(t) 20 )=0.7, since s1(t 20 )>T s The abnormal condition is met, therefore for P1(t) 20 To perform machine learning anomaly labeling.

[0111] Choose an appropriate model to predict replacement values: For data that is marked as an anomaly, such as P2(t) 10 (Negative values) To calculate the correlation between P2 and other processing data (P1, P3, P4) using statistical methods, the Pearson correlation coefficient method can be used. Calculations showed a high correlation between P2 and P3 (spindle speed), with a correlation coefficient of 0.6. A correlation threshold of 0.5 was set, and a subset of relevant parameters {P3} was selected. Subsequently, a random forest regression model was constructed, and P3(t) was used to calculate the correlation coefficient between P2 and P3 (spindle speed). 10 Using t as model input, the model is used to predict P2(t). 10 The replacement value B2(t) 10 For example, for P1(t) 15 (outliers), and the data vector X(t) 15 Except for P1(t) 15 All other values ​​were not marked as null, outlier, or machine learning anomaly, and the same random forest regression model was used to predict replacement values.

[0112] If multiple data values ​​are marked as anomalous, for example, P1(t) 30 (outliers), P4(t) 30 (Machine learning outliers), and the data vector X(t) 30If any values ​​other than these two are marked as anomalous, then the P1 and P4 sequences are modeled separately using an LSTM time series model. The LSTM model is then used to predict the replacement value B1(t). 30 ) and B4(t 30 ).

[0113] Perform range validation on replacement values: Perform secondary rule validation on all generated replacement values, comparing the replacement values ​​with the preset normal range of parameter values. If B1(t) 15 ) = 125℃, which is not within the preset range of 80-120℃, because P1(t 15 If B2(t) is marked as abnormal, it will be submitted for manual review according to the rules, and relevant technical personnel will be notified via email or a dedicated production data review platform for processing. 10 If the value is 70N, which is within the preset range of 50-80N, then proceed to the next step of processing.

[0114] Based on the residual and semantic definition processing method: when the replacement value is within the preset range, use B2(t) 10 Taking t as an example, calculate the replacement value B2(t) 10 ) and processing data value P2(t 10 The residual r2(t) 10 The absolute value difference is used for calculation, i.e., r2(t) 10 )=|B2(t 10 )-P2(t 10 )|.

[0115] Data labeling and training set construction: Collecting outlier scores s2(t) of cutting force data values ​​from historical data. i ) and the corresponding residual r2(t i The organization has organized professional production data analysts to manually annotate these data, and the annotation labels are divided into three categories: "normal", "abnormal" and "suspicious".

[0116] Gaussian Mixture Model Fitting and Semantic Assignment: A Gaussian Mixture Model (GMM) is used to fit the two-dimensional training set, with the number of fitted Gaussian components K set to 3. For each fitted Gaussian component k, the most frequent label type within it is counted, and the component is assigned the semantic meaning of "normal cluster", "suspicious cluster" or "abnormal cluster" based on the label type.

[0117] Posterior membership calculation and decision processing: After generating a new replacement value, obtain its corresponding outlier score s and residual r. Using the probability density function and weight parameters of a Gaussian mixture model, calculate the posterior membership vector γ. k ;

[0118] Take the index of the component with the largest value in the posterior membership vector, k*=arg maxk γ k and its membership degree γ k * The processing decision is made based on the cluster semantics and membership strength corresponding to k: when the cluster semantics corresponding to k is "normal cluster" and γ k When the value exceeds the preset normal threshold (set to 0.8), a replacement value is used; when the cluster semantics corresponding to k are "abnormal clusters" and γ k *When the value exceeds the preset abnormal threshold (set to 0.7), the processed data value will be retained; otherwise, it will be submitted for manual review.

[0119] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A method for monitoring the quality of processed data based on machine learning, characterized in that: The method includes the following steps: Step 1: Obtain multi-source processing data and sort it by time. Perform null value and reasonableness checks on each data value and mark it accordingly. Step 2: Use the Isolation Forest algorithm to analyze the data vector, output the global anomaly score and decompose it into local scores, and mark them as machine learning anomalies; Step 3: Select either the random forest regression model or the LSTM model to predict replacement values ​​based on the anomaly labeling. Step 4: Perform range validation on the replacement values, and decide whether to replace them or submit them for manual review based on the type of anomaly. Step 5: Calculate the residual between the processed data value and the replacement value, fit the Gaussian component with GMM and assign semantics, and define the processing method according to the cluster semantics and membership degree; In step 1, historical processing data is obtained through equipment operation sensors, environmental monitoring sensors, or from the production management system, and is represented as [P1, P2, ..., P]. n Sort in chronological order; Where P1, P2, ..., P n This represents the processing data for the 1st, 2nd, ..., nth type; At time t i Data vector X(t) i ) is represented as: X(t) i )=[P1(t i ),P2(t i ),…,P n (t i )]; Among them, t i P1(t) represents the i-th time point; i ),P2(t i ),…,P n (t i ) represents t i Processing data P1, P2, ..., P corresponding to time points n The value; Perform null value detection on each processed data value, and mark it as a null value if it is detected. The reasonableness of each processing data value is checked based on a preset range. When the processing data value is not within the preset range, it is marked as abnormal. In step 2, when the processed data value is within a preset range, the isolated forest algorithm is used to process the data vector X(t). i Perform anomaly analysis and output a global anomaly score s. i ; Through feature importance analysis, the global anomaly score s i Decomposed into each parameter P j (t i ), thus obtaining the local anomaly score s j (t i ); Where j∈{1,2,…,n}; P j (t i ) represents t i Processing data P corresponding to time moment j Value, s j (t i ) represents t i Real-time processing data P j The corresponding local anomaly score; Set threshold T s When s j (t i )>T s At that time, machine learning anomaly labeling is performed.

2. The method for monitoring the quality of processed data based on machine learning according to claim 1, characterized in that: In step 3, when the processing data value P j (t i ) is marked as null, anomaly, or machine learning anomaly and the data vector X(t) i Except for P j (t i When all other values ​​are not marked as null, anomalous, or machine learning anomalies, calculate P. j Correlation with other processing data, setting thresholds and selecting a subset of relevant parameters {P} a }; where a≠j, a∈{1,2,…,n}; Construct a random forest regression model, with P k (t i Given the input, predict the replacement value B. j (t i ); When the processing data value P j (t i ) is marked as null, anomaly, or machine learning anomaly and the data vector X(t) i Except for P j (t i When any of the remaining values ​​are marked as null, anomaly, or machine learning anomaly, or when P j When the correlation with all other processing data does not meet the set threshold requirement, the P-based LSTM time series model is used. j Sequence modeling alone, predicting replacement value B j (t i ).

3. The method for monitoring the quality of processed data based on machine learning according to claim 2, characterized in that: In step 4, the generated replacement value B is... j (t i Perform secondary rule validation: When B j (t i When not within the preset range: When the processing data value P j (t i If a value is marked as null or abnormal, it will be submitted for manual review. When the processing data value P j (t i When a P-value is marked as a machine learning anomaly, retain P. j (t i )value; In step 5, when B j (t i When the value is within the preset range: Calculate the replacement value B. j (t i ) and the processing data value P j (t i The residual r j (t i ):r j (t i )=|B j (t i )-P j (t i )|; Collect anomaly scores for corresponding processed data values ​​from historical data. j (t i and the corresponding residual r j (t i Manual labeling: normal, abnormal, or suspicious; Normal indicates that the processing data value will be directly replaced with the replacement value after manual confirmation; Abnormal indicates that the processing data value will be retained after manual confirmation; Suspicious indicates that further verification is required. Form a two-dimensional training set {(s(m),r(m))} m=1 M Where M represents the number of corresponding processed data values, m∈{1,2,…,M}; In two-dimensional space, with {(s(m),r(m))} as input, a Gaussian mixture model (GMM) is used to fit K Gaussian components; for each component k, the semantics of normal cluster, suspicious cluster, or anomalous cluster are assigned based on the label with the most internal number. After generating the new replacement value, obtain its corresponding (s,r) and calculate the posterior membership vector. ; Where, π k N([s,r]|μ) represents the weight of the k-th Gaussian component; k ,Σ k ) represents the probability density function of a two-dimensional Gaussian distribution; μ k Σ represents the mean vector of the k-th Gaussian component; k Let represent the covariance matrix of the k-th Gaussian component; l represents the index variable in the summation symbol Σ, which iterates through all K Gaussian components; Find the maximum posterior cluster k* = arg max k γ k and its membership degree γ k* ; Based on the cluster semantics and membership strength corresponding to k*: when the cluster semantics corresponding to k* is a normal cluster and γ k* When the value exceeds the preset normal threshold, a replacement value is used; when the cluster semantics corresponding to k* are abnormal clusters and γ k* If the value exceeds the preset abnormal threshold, the processed data value is retained; otherwise, it is submitted for manual review.

4. A machine learning-based processing data quality monitoring system, applied to the machine learning-based processing data quality monitoring method described in any one of claims 1-3, characterized in that: The system includes a data preprocessing module, an anomaly detection module, a replacement prediction module, and a verification and auditing module; The data preprocessing module is used to acquire multi-source processed data and sort it according to time, and to perform null value and reasonableness checks on each data value and mark it accordingly; The anomaly detection module is used to analyze data vectors using the isolated forest algorithm, output a global anomaly score and decompose it into local scores, and mark them as machine learning anomalies. The replacement prediction module is used to select either a random forest regression model or an LSTM model to predict replacement values ​​based on the anomaly labeling situation. The verification and review module is used to perform range verification on the replacement value, and decide whether to replace it or submit it for manual review based on the anomaly type; calculate the residual between the processed data value and the replacement value, fit the Gaussian component with GMM and assign semantics, and define the processing method according to the cluster semantics and membership degree. The output of the data preprocessing module is connected to the input of the anomaly detection module; The output of the anomaly detection module is connected to the input of the replacement prediction module; The output of the replacement prediction module is connected to the input of the verification and audit module.

5. The processing data quality monitoring system based on machine learning according to claim 4, characterized in that: The data preprocessing module includes a data acquisition unit and a sorting and detection unit; The data acquisition unit is used to acquire multi-source data from equipment sensors or production systems; The sorting and detection unit is used to generate a data vector by sorting it by time, perform null value detection and rationality detection, and mark outliers. The output of the data acquisition unit is connected to the input of the sorting and detection unit; The output of the sorting detection unit is connected to the input of the anomaly detection module.

6. The processing data quality monitoring system based on machine learning according to claim 5, characterized in that: The anomaly detection module includes a global analysis unit and a local decomposition unit; The global analysis unit is used to calculate the global anomaly score using the isolated forest algorithm; The local decomposition unit is used to decompose the global anomaly score into specific data based on feature importance, and perform machine learning anomaly labeling based on thresholds; The output of the global analysis unit is connected to the input of the local decomposition unit; The output of the local decomposition unit is connected to the input of the replacement prediction module.

7. The processing data quality monitoring system based on machine learning according to claim 6, characterized in that: The replacement prediction module includes a model selection unit and a prediction execution unit; The model selection unit is used to select a random forest or LSTM model based on the anomaly labeling situation; The prediction execution unit is used to execute the selected model to generate replacement values; The output of the model selection unit is connected to the input of the prediction execution unit; The output of the prediction execution unit is connected to the input of the verification and auditing module.

8. The machine learning-based processing data quality monitoring system according to claim 7, characterized in that: The verification and auditing module includes a rule verification unit, a residual clustering unit, and a decision execution unit; The rule verification unit is used to perform a second verification on the replacement value within a preset range; the residual clustering unit is used to calculate the residual between the processed data value and the replacement value, construct the GMM model and assign semantic labels; the decision execution unit is used to decide whether to replace, retain or manually review based on the cluster semantics and membership degree. The output of the rule verification unit is connected to the input of the residual clustering unit; The output of the residual clustering unit is connected to the input of the decision execution unit.

Images