A data mining and analysis method and system for industrial data

Abstract

The present invention discloses a data mining and analysis method and system for industrial data, comprising: collecting raw data from a data source and performing cleaning and conversion; defining a problem, creating a data mining library for online analysis and processing; constructing a mathematical mining model, and evaluating and optimizing the model; and using the optimized model to perform data mining analysis and obtain the optimal decision. The present invention pre-sets a large number of data processing and operator nodes, and implements streaming modeling by dragging and dropping, making it convenient for users to quickly build data mining models. The mining results are analyzed and presented through a variety of rich visualization methods. The organic combination of online data analysis and machine learning and process optimization automates data management and analysis, thereby more effectively performing data analysis.

Description

Technical Field

[0001] The present invention relates to the field of data engineering technology, and in particular to a data mining and analysis method and system for industrial data. Background Art

[0002] With the widespread and deepening digitalization of enterprises, business processes are becoming increasingly automated, generating vast amounts of data. This data and the resulting information are valuable assets for businesses, faithfully documenting their fundamental operational status. However, faced with such a vast amount of data, traditional data analysis methods, such as data retrieval and statistical analysis, can only capture surface information, failing to capture its underlying, deeper insights. This leaves managers facing a dilemma of abundant data but insufficient knowledge. Extracting valuable knowledge from this data for business decision-making is crucial, and data mining emerged to address this need.

[0003] However, due to the diverse sources of data and the numerous uncertainties inherent in semi-structured and unstructured data, existing technologies are difficult to scientifically calculate and evaluate due to the lack of historical data and corresponding statistical information. Therefore, alternative technologies and methods are needed to enhance the value of enterprise data mining. While numerous data mining algorithms and case studies exist, they are primarily used for network data. These algorithms suffer from significant overfitting in the industrial sector and cannot be directly used, requiring optimization and adjustment. Summary of the Invention

[0004] In view of the above-mentioned problems, the present invention is proposed.

[0005] Therefore, the technical problem solved by the present invention is: how to process massive multi-source heterogeneous data and obtain its inherent deep information.

[0006] To solve the above technical problems, the present invention provides the following technical solutions: a data mining and analysis method for industrial data, comprising:

[0007] Collect raw data from data sources and clean and transform it;

[0008] Perform problem definition and create a data mining library for online analytical processing;

[0009] Build mathematical mining models, and evaluate and optimize the models;

[0010] Use the optimized model to conduct data mining analysis to obtain the optimal decision.

[0011] As a preferred solution of the data mining and analysis method for industrial data described in the present invention, the collection of raw data includes: the data collection method is divided into offline collection and real-time collection, and the collected data is uniformly aggregated into the data warehouse; offline collection is based on Dolphin Scheduling and integrates Datax services, supports data sources in the form of library tables, interfaces, FTP files, time series data definitions, CSV file import, and time series database data import, supports conversion and mapping between structured and semi-structured data, and real-time collection is based on the fink+kafka message and cdc log modes; different types of data collected in different modes are finally connected to a unified data warehouse.

[0012] As a preferred solution of the data mining and analysis method for industrial data described in the present invention, wherein: the cleaning and conversion include: an autoencoder composed of an encoder and a decoder, learning the representation of the data by minimizing the reconstruction error;

[0013] Input data D raw , after passing through encoder E and decoder D, the output is the repaired data D a The formula is:

[0014] D a =D(E(D raw ))

[0015] The loss function L is expressed as:

[0016] L=||D raw -D(E(D raw ))

[0017] Real-time data is processed and filtered through Apache Flink; the CEP engine is used to detect complex event patterns in real-time data streams. Let the CEP rule be φ CEP , the processed data set D CEP Expressed as:

[0018] D CEP ={s i ∈D stream |φ CEP (s i )}

[0019] The event pattern for detecting simultaneous abnormalities in temperature and pressure is expressed as:

[0020] φ CEP (s i )=(s i [temp]>threshold1)∧(s i [pres]>threshold2)

[0021] For detected event data, use autoencoders to perform real-time repair:

[0022] s′ i =D(E(s i ))

[0023] Automatically detect outliers and missing values in the data and repair them; for missing values, the autoencoder will fill in the missing values during the reconstruction process; for outliers, the autoencoder will reconstruct them into normal values.

[0024] As a preferred solution of the data mining and analysis method for industrial data described in the present invention, the creation of a data mining library includes recording detailed data through fine-grained data analysis using a detailed model; the data recording formula is expressed as:

[0025] Detail_Model={d1,d2,...,d n}

[0026] Among them, d i Represents the i-th detailed data record; the storage formula is expressed as:

[0027]

[0028] Among them, Size(d i ) represents the storage size of the i-th data record;

[0029] Use the aggregation model to summarize data analysis and aggregate data by dimension. The data aggregation formula is expressed as:

[0030] Aggregate_Model={α1,a2,...,a m}

[0031] Among them, a i Represents the i-th aggregate data record; the storage formula is expressed as:

[0032]

[0033] Among them, Size(a i ) represents the storage size of the i-th aggregate data record;

[0034] Use a unique model to manage unique data. The formula is:

[0035] Unique_Model={u1,u2,...,u p}

[0036] Among them, u i Represents the i-th unique data record; the uniqueness constraint formula is expressed as:

[0037]

[0038] This formula ensures that each unique data record is not repeated in the collection; storage formula:

[0039]

[0040] Among them, Size(u i ) represents the storage size of the i-th unique data record; an appropriate data model is selected according to the data characteristics.

[0041] As a preferred embodiment of the data mining and analysis method for industrial data of the present invention, the online analytical processing includes establishing a series of hypotheses and obtaining data analysis conclusions by confirming or refuting the hypotheses through online analytical processing (OLAP);

[0042] Loading data from the data warehouse to the OLAP system, the data loading formula is expressed as:

[0043] OLAP_Data(t)=Load(Warehouse_Data(t))

[0044] Where OLAP_Data(t) represents the data loaded into the OLAP system at time t; Warehouse_Data(t) represents the data stored in the data warehouse at time t; Load represents the operation of loading data from the data warehouse; Build a star or snowflake data model and define fact tables and dimension tables;

[0045] Perform advanced aggregation operations on the data. The advanced data aggregation formula is expressed as:

[0046]

[0047] w represents the weight vector, which indicates the weight of each data point; x represents the data value vector, which indicates the value of each data point; w i represents the weight of the i-th data point; x i represents the value of the i-th data point;

[0048]

[0049] Among them, μ w represents the weighted average;

[0050] The multidimensional array aggregation formula is expressed as:

[0051]

[0052] Among them, A i,j,kRepresents the (i, j, k)th element in the aggregated multidimensional array; D l,m,n Represents the (l,m,n)th element in the original data array; f i,j,k (l,m,n) represents the aggregation function, which defines how to aggregate the original data into a new multidimensional array; L,M,N represent the dimensions of the original data array;

[0053] Slice and dice data in different dimensions to support complex queries; the slicing formula is expressed as:

[0054] Slice(D,d,v)={r|r∈D∧r[d]=v}

[0055] Where D represents the dataset; d represents the dimension to be sliced; v represents the specific value of the slice dimension; r represents the data record that meets the conditions;

[0056] The cutting formula is expressed as:

[0057]

[0058] Among them, {d1,d2,...,d k} represents multiple dimensions to be sliced; {v1,v2,...,v k} represents a specific value of the slicing dimension;

[0059] Supports data drilling and roll-up operations to perform data analysis at different levels; the drilling formula is expressed as:

[0060] Drill_Down(D,h,l)={r|r∈D∧r[h]∈l}

[0061] Among them, h represents the high-level dimension; l represents the low-level dimension set.

[0062] As a preferred solution of the data mining and analysis method for industrial data described in the present invention, the construction of the mathematical mining model includes extracting key features using advanced feature engineering technology. The feature engineering formula is expressed as:

[0063] Feature i =f(T,S,X)

[0064] Where T represents time series features, S represents spatial features, X represents other features, and f represents feature extraction function; the deep learning model formula is expressed as:

[0065] h t =o(W h ·h t-1 +W x ·x t +b h )

[0066] c t =f(W c c t-1 +W ic ·x t +b c )

[0067] y t =W y ·h t +b y

[0068] Among them, h t represents the hidden state of the current time step; c t Indicates cell status; x t Indicates input; W h 、W x 、W c 、W ic 、W y represents the weight matrix, b h 、b c 、b y represents bias, σ represents activation function;

[0069] The loss function formula is expressed as:

[0070]

[0071] Among them, yi represents the true value, represents the predicted value, and θ represents the model parameters.

[0072] As a preferred solution of the data mining and analysis method for industrial data described in the present invention, wherein: the data mining and analysis includes calculating relative deviation and weighted deviation;

[0073] Δ i Indicates the relative deviation of the i-th data point;

[0074]

[0075] Among them, X i represents the actual observation value at the i-th time point; represents the predicted value at the i-th time point; let W i represents the weighted deviation of the i-th data point, which is weighted by combining the sigmoid function and the absolute error;

[0076]

[0077] Calculate the anomaly score, the formula is expressed as:

[0078]

[0079] M represents the total number of data points; H represents the set value of high deviation; if the deviation exceeds H, the data point is considered to be abnormal; (W i >H) represents the indicator function; when the weighted deviation W i When the set value H is exceeded, the function value is 1, otherwise it is 0;

[0080] Threshold represents the set abnormal ratio threshold. When the deviation is greater than H, calculation and judgment are performed; when AS>threshold, it is judged as data abnormality; when AS≤threshold, the prediction model parameters and deviation setting values are adjusted, and the abnormality judgment is re-performed.

[0081] A data mining and analysis system for industrial data using any of the methods described in the present invention, comprising:

[0082] The data collection and preprocessing module uses Kafka and Flink for data collection and real-time processing, as well as data cleaning and standardization.

[0083] Data mining library and online analysis, creating a data mining library, using Doris for data storage and OLAP analysis;

[0084] Mathematical mining module, building LSTM model, and using Bayesian optimization to tune model parameters;

[0085] The data mining and decision-making module uses optimized models for data mining analysis; Tableau is used for data visualization and decision support.

[0086] A computer device comprises: a memory and a processor; the memory stores a computer program, wherein the processor implements the steps of any one of the methods of the present invention when executing the computer program.

[0087] A computer-readable storage medium stores a computer program thereon, wherein the computer program, when executed by a processor, implements the steps of any one of the methods of the present invention.

[0088] The beneficial effects of the present invention are as follows: 1. It provides users with a visual modeling interface, pre-sets a large number of data processing and operator nodes, and implements streaming modeling by dragging and dropping, which facilitates users to quickly build data mining models. Modeling is mainly based on dragging and dropping, which greatly reduces the threshold for participation in modeling work. 2. Data mining results are often not easy to understand, and it is seamlessly integrated with other visualization functions to analyze and display the mining results through a variety of rich visualization means. 3. It is compatible with small factory mode and cloud edge mode, multiple development modes, and reduces the skill level requirements for developers. It has high flexibility and usability in different scenarios. The computing engine is suitable for online visual analysis of massive data. 4. The organic combination of online data analysis and machine learning (ML), process optimization, automates data management and analysis, and thus performs data analysis more effectively. It reduces the efficiency problems and caliber deviations brought about by the current reliance on IT processing, allowing users to gain deeper insights. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort. Among them:

[0090] Figure 1 This is an overall flow chart of a data mining and analysis method for industrial data provided by the first embodiment of the present invention. DETAILED DESCRIPTION

[0091] To make the above-mentioned objects, features, and advantages of the present invention more clearly understood, the following detailed description of the specific embodiments of the present invention is given in conjunction with the accompanying drawings. It is obvious that the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in this field without creative work should fall within the scope of protection of the present invention.

[0092] Example 1

[0093] Reference Figure 1 , as an embodiment of the present invention, provides a data mining and analysis method for industrial data, comprising:

[0094] S1: Collect raw data from data sources and perform cleaning and transformation.

[0095] Furthermore, the collection of raw data includes: data collection methods are divided into offline collection and real-time collection, and the collected data are uniformly aggregated into a data warehouse.

[0096] Furthermore, offline collection is based on the integration of Datax services with Dolphin Scheduling, supporting data sources in the form of library tables, interfaces, FTP files, time series data definitions, CSV file import, and time series database data import. It supports conversion and mapping between structured and semi-structured data, and real-time collection is implemented based on the Fink+Kafka message and CDC log modes. Different types of data collected in different modes are ultimately connected to a unified data warehouse.

[0097] Furthermore, the cleaning and conversion includes that the autoencoder is composed of an encoder and a decoder, and learns the representation of the data by minimizing the reconstruction error.

[0098] Furthermore, input data D raw , after passing through encoder E and decoder D, the output is the repaired data D a The formula is:

[0099] D a =D(E(D raw ))

[0100] Furthermore, the loss function L is expressed as:

[0101] L=||D raw -D(E(D raw ))

[0102] Furthermore, real-time data is processed and filtered through Apache Flink; the CEP engine is used to detect complex event patterns in real-time data streams. Let the CEP rule be φ CEP , the processed data set D CEP Expressed as:

[0103] D CEP ={s i ∈D stream |φ CEP (s i )}

[0104] Furthermore, the event pattern for detecting simultaneous abnormalities in temperature and pressure is expressed as:

[0105] φ CEP (s i )=(s i [temp]>threshold1)∧(s i [pres]>threshold2)

[0106] Furthermore, the detected event data is repaired in real time using an autoencoder:

[0107] s′ i =D(E(si ))

[0108] Furthermore, it automatically detects outliers and missing values in the data and repairs them; for missing values, the autoencoder fills in the missing values during the reconstruction process; for outliers, the autoencoder reconstructs them into normal values.

[0109] It should be noted that the excavation objectives and problems must be clearly defined. A clear problem definition can help excavators better select appropriate analysis methods and techniques. Hydraulic pump troubleshooting is often performed post-fault. In contrast, predictive maintenance schedules maintenance activities by predicting faults based on the hydraulic pump's previous condition, offering significant advantages such as automation and high efficiency. Hydraulic pump fault prediction utilizes sensors to collect excavator data and employs appropriate algorithms to assess the pump's health, predicting failures before they occur.

[0110] S2: Define the problem and create a data mining library for online analysis and processing.

[0111] Furthermore, the creation of the data mining library includes recording each piece of data in detail for fine-grained data analysis; the data recording formula is:

[0112] Detail_Model={d1,d2,...,d n}

[0113] Among them, d i Represents the i-th detailed data record; storage formula:

[0114]

[0115] Among them, Size(d i ) represents the storage size of the i-th data record.

[0116] Furthermore, for summary data analysis, data is aggregated by dimension. The data aggregation formula is expressed as:

[0117] Aggregate_Model={a1,a2,...,a m}

[0118] Among them, a i Represents the i-th aggregate data record; aggregate function formula:

[0119] a i =Aggregate Function(d i1 ,d i2 ,...,d ik )

[0120] Among them, Aggregate Function represents the aggregation function; d ij Indicates the jth detailed data record participating in the aggregation; storage formula:

[0121]

[0122] Among them, Size(a i ) represents the storage size of the i-th aggregate data record.

[0123] Furthermore, the data uniqueness formula is used for unique data management to ensure the uniqueness of data records. The formula is expressed as:

[0124] Unique_Model={u1,u2,…,u p}

[0125] Among them, u i Represents the i-th unique data record; the uniqueness constraint formula is expressed as:

[0126]

[0127] Furthermore, this formula ensures that each unique data record is not repeated in the collection; storage formula:

[0128]

[0129] Among them, Size(u i ) represents the storage size of the i-th unique data record; an appropriate data model is selected according to the data characteristics.

[0130] Furthermore, the online analytical processing includes establishing a series of hypotheses and obtaining data analysis conclusions by confirming or refuting the hypotheses through online analytical processing (OLAP).

[0131] Furthermore, to load data from the data warehouse into the OLAP system, the data loading formula is expressed as:

[0132] OLAP_Data(t)=Load(Warehouse_Data(t))

[0133] Among them, OLAP_Data(t) represents the data loaded into the OLAP system at time t; Warehouse_Data(t) represents the data stored in the data warehouse at time t; Load represents the operation of loading data from the data warehouse; build a star or snowflake data model and define fact tables and dimension tables.

[0134] Furthermore, we can perform advanced aggregation operations on the data. The advanced data aggregation formula is expressed as:

[0135]

[0136] Among them, w represents the weight vector, which represents the weight of each data point; x represents the data value vector, which represents the value of each data point; w i represents the weight of the i-th data point; x i Represents the value of the i-th data point.

[0137]

[0138] Among them, μ w Represents a weighted average.

[0139] Furthermore, the multidimensional array aggregation formula is expressed as:

[0140]

[0141] Among them, A i,j,k Represents the (i, j, k)th element in the aggregated multidimensional array; D l,m,n Represents the (l,m,n)th element in the original data array; f i,j,k (l,m,n) represents the aggregation function, which defines how to aggregate the original data into a new multidimensional array; L,M,N represent the dimensional size of the original data array.

[0142] Furthermore, data slicing and dicing operations can be performed on different dimensions to support complex queries; the slicing formula is expressed as:

[0143] Slice(D,d,v)={r|r∈D∧r[d]=v}

[0144] Where D represents the dataset, d represents the dimension to be sliced, v represents the specific value of the slice dimension, and r represents the data record that meets the conditions.

[0145] Furthermore, the slicing formula is expressed as:

[0146]

[0147] Among them, {d1,d2,...,d k} represents multiple dimensions to be sliced; {v1,v2,...,v k} represents a specific value for the slicing dimension.

[0148] Furthermore, it supports data drill-down and roll-up operations to perform data analysis at different levels; the drill-down formula is expressed as:

[0149] Drill_Down(D,h,l)={r|r∈D∧r[h]∈l}

[0150] Among them, h represents the high-level dimension; l represents the low-level dimension set.

[0151] It should be noted that this process utilizes the platform's large-scale data processing capabilities to standardize heterogeneous data. During the visual modeling process, all data processing logic is encapsulated into operators, such as deduplication, complementation, dictionary conversion, data desensitization, ID encoding, value range cleaning, time domain cleaning, null value cleaning, duplicate cleaning, outlier cleaning, word frequency cleaning, and slope cleaning. Users orchestrate these operators from a natural language perspective and then publish offline scheduling jobs or real-time monitoring tasks for data processing. Ultimately, model data that meets the requirements of model training is generated and archived in the data mining library.

[0152] S3: Build mathematical mining models and evaluate and optimize the models.

[0153] Furthermore, the construction of the mathematical mining model includes extracting key features using advanced feature engineering technology. The feature engineering formula is expressed as:

[0154] Feature i =f(T, S, X)

[0155] Where T represents time series features, S represents spatial features, X represents other features, and f represents feature extraction function; the deep learning model formula is expressed as:

[0156] h t =σ(W h ·h t-1 +W x ·x t +b h )

[0157] c t =f(W c c t-1 +W ic ·x t +b c )

[0158] y t =W y ·h t +b y

[0159] Among them, h t represents the hidden state of the current time step; c t Indicates cell status; x t Indicates input; W h 、W x 、W c 、W ic 、W yrepresents the weight matrix, b h 、b c 、b y represents the bias, and σ represents the activation function.

[0160] Furthermore, the loss function formula is expressed as:

[0161]

[0162] Among them, y i represents the true value, represents the predicted value, and θ represents the model parameters.

[0163] It should be noted that the collected raw data is verified and processed to become clean and complete data suitable for modeling. First, the data type of each field is verified to ensure it is logically consistent. Next, its value range is verified to ensure it is realistic. After a series of processing, the raw data is converted to the data we need. Data outside the actual value range of each field is considered abnormal and deleted. For columns with >80% null values, missing values are deleted. Both continuous and discrete variables are filled with missing values using the up-fill method. Continuous variables are reduced using the Z-score standardization method, while discrete variables are one-hot encoded.

[0164] S4: Use the optimized model to conduct data mining analysis to obtain the optimal decision.

[0165] Furthermore, the data mining analysis includes calculating relative deviation and weighted deviation.

[0166] Furthermore, Δ i Indicates the relative deviation of the i-th data point.

[0167]

[0168] Among them, X i represents the actual observation value at the i-th time point; represents the predicted value at the i-th time point. Let W i Represents the weighted deviation of the i-th data point, which is weighted by combining the sigmoid function and the absolute error.

[0169]

[0170] Furthermore, the anomaly score is calculated as follows:

[0171]

[0172] Where M represents the total number of data points; H represents the set value of high deviation; if the deviation exceeds H, the data point is considered to be abnormal; (Wi >H) represents the indicator function; when the weighted deviation W i When the set value H is exceeded, the function value is 1, otherwise it is 0.

[0173] Furthermore, threshold represents the set abnormal ratio threshold. When the deviation is greater than H, calculation and judgment are performed; when AS>threshold, it is judged as data abnormality; when AS≤threshold, the prediction model parameters and the deviation setting value are adjusted, and the abnormality judgment is re-performed.

[0174] It should be noted that data mining and OLAP are complementary. Before taking action based on data mining findings, it's important to examine the impact of such actions on the business. OLAP tools can answer these questions. Furthermore, in the early stages of knowledge discovery, OLAP tools have other uses. They can help you explore data, identify variables that are important to a problem, and uncover anomalies and interdependent variables. These operations can lead to a better understanding of the data and accelerate the knowledge discovery process.

[0175] On the other hand, this embodiment also provides a data mining and analysis system for industrial data, which includes:

[0176] The data collection and preprocessing module uses Kafka and Flink for data collection and real-time processing, as well as data cleaning and standardization.

[0177] Data mining library and online analysis, create a data mining library, use Doris for data storage and OLAP analysis.

[0178] Mathematical mining module, builds LSTM model, and uses Bayesian optimization to tune model parameters.

[0179] The data mining and decision-making module uses optimized models for data mining analysis; Tableau is used for data visualization and decision support.

[0180] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0181] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0182] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0183] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0184] Example 2

[0185] The following is an embodiment of the present invention, which provides a data mining and analysis method for industrial data. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0186] Define the objectives and problems to be explored. Hydraulic pump failures are related to wear or oil contamination, and since the power comes from the engine, select engine and oil parameters. The collected raw data is verified and processed to transform it into clean, complete data suitable for modeling. First, verify that the data type of each field is logically consistent. Next, check that its value range is realistic. After a series of processing, convert the raw data into the data we need. Data outside the actual value range of each field is considered anomaly and deleted. For missing values, columns with a null value ratio greater than 80% are deleted. For both continuous and discrete variables, fill in missing values using the up-fill method. Continuous variables are reduced using the Z-score standardization method, while discrete variables are one-hot encoded.

[0187] The relevant operating statistical parameters of the hydraulic pump after preprocessing are shown in the following table.

[0188] Table 1 Hydraulic pump related operating statistical parameters

[0189]

[0190] If a hydraulic pump fails, based on the three manifestations of the failure and combined with the current sensor data, the pump pressure can be predicted for analysis: If the hydraulic pump fails, the generated pump pressure P will deviate from the normal pump pressure P' by △P. If the deviation △P / P exceeds a certain value w and the proportion exceeds the threshold, it is determined that the hydraulic pump has failed. Let the daily pump pressure be Y dayi , the corresponding predicted value is Y i , the fault judgment formula is expressed as:

[0191]

[0192] When the calculated result is greater than the set threshold, it is predicted to be a fault; otherwise, it is predicted to be normal.

[0193] Model Verification. Based on the aforementioned model, we obtained pump pressure predictions. We calculated the deviation (ΔP / P) for each data point between the actual and predicted values. We also calculated the daily percentage of different deviations (±5%, +10%, etc.). By analyzing the trend chart of failure time and deviation percentage, we derived a set of model settings: w = 20%, threshold = 20%. Using these model settings, we verified the prediction accuracy of other hydraulic pumps and adjusted the model parameters to ensure generalizability.

[0194] The ultimate analysis goal is to predict whether the hydraulic pump is faulty, which is a binary classification problem. The confusion matrix is a method for evaluating the accuracy of classification problems. For binary classification problems, the combination of the true class and the predicted class is divided into four categories: true positive, false positive, true negative, and false negative. Let TP, FP, TN, and FN represent the corresponding number of samples. The resulting confusion matrix is shown in the following table:

[0195] Table 2 Evaluation sample table

[0196] The real situation Positive prediction results Negative prediction results Positive example 15 4 Negative example 21 69

[0197] Precision P = TP / (TP+FP) = 15 / (15+21) = 0.42

[0198] Recall R = TP / (TP+FN) = 15 / (15+4) = 0.79

[0199] It can be concluded that all hydraulic pump system failures are predicted, but false alarms also occur from time to time. Therefore, it is necessary to optimize the judgment method to reduce false alarms.

[0200] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A data mining and analysis method for industrial data, characterized in that: include: Collect raw data from data sources and clean and transform it; Perform problem definition and create a data mining library for online analytical processing; Build mathematical mining models, and evaluate and optimize the models; Use the optimized model to conduct data mining analysis to obtain the best decision; Problem definition includes clarifying the mining objectives and problems. This helps select appropriate analysis methods and techniques. Hydraulic pump fault prediction uses sensors to collect excavator data and uses appropriate algorithms to assess the health of the hydraulic pump and predict faults before they occur. The data mining analysis includes calculating relative deviation and weighted deviation; Δ i Indicates the relative deviation of the i-th data point; Among them, X i represents the actual observation value at the i-th time point; represents the predicted value at the i-th time point; let W i represents the weighted deviation of the i-th data point, which is weighted by combining the sigmoid function and the absolute error; Calculate the anomaly score, the formula is expressed as: M represents the total number of data points; H represents the set value of high deviation; if the deviation exceeds H, the data point is considered to be abnormal; (W i >H) represents the indicator function; when the weighted deviation W i When the set value H is exceeded, the function value is 1, otherwise it is 0; Threshold represents the set abnormal ratio threshold. When the deviation is greater than H, calculation and judgment are performed; when AS>threshold, it is judged as data abnormality; when AS≤threshold, the prediction model parameters and deviation setting values are adjusted, and the abnormality judgment is re-performed.

2. The data mining and analysis method for industrial data according to claim 1, characterized in that: The collection of raw data includes: data collection methods are divided into offline collection and real-time collection, and the collected data are uniformly aggregated into the data warehouse; offline collection is based on Dolphin Scheduling and integrates Datax services, supports data sources in the form of library tables, interfaces, FTP files, time series data definitions, CSV file import, and time series database data import, supports conversion and mapping between structured and semi-structured data, and real-time collection is based on the fink+kafka message and cdc log modes; different types of data collected in different modes are ultimately connected to a unified data warehouse.

3. The data mining and analysis method for industrial data according to claim 2, characterized in that: The cleaning and conversion include that the autoencoder is composed of an encoder and a decoder, and learns the representation of the data by minimizing the reconstruction error; Input data D raw , after passing through encoder E and decoder D, the output is the repaired data D a The formula is: D a =D(E(D raw )) The loss function L is expressed as: L=||D raw -D(E(D raw ))|| 2 Process and filter real-time data through Apache Flink; Use the CEP engine to detect complex event patterns in real-time data streams. Let the CEP rule be φ CEP , the processed data set D CEP Expressed as: D CEP ={s i ∈D stream ∣φ CEP (s i )} The event pattern for detecting simultaneous abnormalities in temperature and pressure is expressed as: φ CEP (s i )=(s i [temp]>threshold1)∧(s i [pres]>threshold2) For detected event data, use autoencoders to perform real-time repair: s′ i =D(E(s i )) Automatically detect outliers and missing values in the data and repair them; for missing values, the autoencoder will fill in the missing values during the reconstruction process; for outliers, the autoencoder will reconstruct them into normal values.

4. The data mining and analysis method for industrial data according to claim 3, characterized in that: The creation of the data mining library includes recording detailed data through fine-grained data analysis using a detailed model; the data recording formula is expressed as: Detail_Model={d1,d2,…,d n } Among them, d i Represents the i-th detailed data record; the storage formula is expressed as: Among them, Size(d i ) represents the storage size of the i-th data record; Use the aggregation model to summarize data analysis and aggregate data by dimension. The data aggregation formula is expressed as: Aggregate_Model={a1,a2,…,a m } Among them, a i Represents the i-th aggregate data record; the storage formula is expressed as: Among them, Size(a i ) represents the storage size of the i-th aggregate data record; Use a unique model to manage unique data. The formula is: Unique_Model={u1,u2,…,u p } Among them, u i Represents the i-th unique data record; the uniqueness constraint formula is expressed as: This formula ensures that each unique data record is not repeated in the collection; storage formula: Among them, Size(u i ) represents the storage size of the i-th unique data record; an appropriate data model is selected according to the data characteristics.

5. The data mining and analysis method for industrial data according to claim 4, characterized in that: The online analytical processing includes establishing a series of hypotheses and obtaining data analysis conclusions by confirming or refuting the hypotheses through online analytical processing (OLAP); Loading data from the data warehouse to the OLAP system, the data loading formula is expressed as: OLAP_Data(t)=Load(Warehouse_Data(t)) Where OLAP_Data(t) represents the data loaded into the OLAP system at time t; WareHouse_Data(t) represents the data stored in the data warehouse at time t; Load represents the operation of loading data from the data warehouse; Build a star or snowflake data model and define fact tables and dimension tables; Perform advanced aggregation operations on the data. The advanced data aggregation formula is expressed as: w represents the weight vector, which indicates the weight of each data point; x represents the data value vector, which indicates the value of each data point; w i represents the weight of the i-th data point; x i represents the value of the i-th data point; Among them, μ w represents the weighted average; The multidimensional array aggregation formula is expressed as: Among them, A i,j,k Represents the (i, j, k)th element in the aggregated multidimensional array; D l,m,n Represents the (l, m, n)th element in the original data array; f i,j,k (l,m,n) represents the aggregation function, which defines how to aggregate the original data into a new multidimensional array; L,M,N represent the dimensions of the original data array; Slice and dice data in different dimensions to support complex queries; the slicing formula is expressed as: Slice(D,d,v)={r|r∈D∧r[d]=v} Where D represents the dataset; d represents the dimension to be sliced; v represents the specific value of the slice dimension; r represents the data record that meets the conditions; The cutting formula is expressed as: Among them, {d1, d2, ..., d k } represents multiple dimensions to be sliced; {v1,v2,…,v k } represents a specific value of the slicing dimension; Supports data drilling and roll-up operations for data analysis at different levels; the drilling formula is expressed as: Drill_Down(D,h,l)={r|r∈D∧r[h]∈l} Among them, h represents the high-level dimension; l represents the low-level dimension set.

6. The data mining and analysis method for industrial data according to claim 5, characterized in that: The construction of the mathematical mining model includes extracting key features using advanced feature engineering technology. The feature engineering formula is expressed as: Feature i =f(T,S,X) Where T represents time series features, S represents spatial features, X represents other features, and f represents feature extraction function; the deep learning model formula is expressed as: h t =σ(W h ·h t-1 +W x ·x t +b h ) c t =f(W c ·c t-1 +W ic ·x t +b c ) y t =W y ·h t +b y Among them, h t represents the hidden state of the current time step; c t Indicates cell status; x t Indicates input; W h 、W x 、W c 、W ic 、W y represents the weight matrix, b h 、b c 、b y represents the bias, and σ represents the activation function.

7. A data mining and analysis system for industrial data using the method according to any one of claims 1 to 6, characterized in that: The data collection and preprocessing module uses Kafka and Flink for data collection and real-time processing, as well as data cleaning and standardization. Data mining library and online analysis, creating a data mining library, using Doris for data storage and OLAP analysis; Mathematical mining module, building LSTM model, and using Bayesian optimization to tune model parameters; The data mining and decision-making module uses optimized models for data mining analysis; Tableau is used for data visualization and decision support.

8. A computer device comprising: memory and processor; The memory stores a computer program, wherein the processor implements the steps of the data mining and analysis method for industrial data according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data mining and analysis method for industrial data according to any one of claims 1 to 6 are implemented.

Images