Data mining analysis method and system for industrial data

Through the construction of data collection, cleaning and mathematical mining models, the deep information acquisition and overfitting problems of industrial data are solved, efficient analysis and decision-making support of industrial data are realized, and flexible and automated data management are provided.

CN119066117AActive Publication Date: 2024-12-03LUCULENT SMART TECHNOLOGIES CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202410979787.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-22
Publication Date
2024-12-03
Estimated Expiration
2044-07-22

AI Technical Summary

Technical Problem

It is difficult for the existing technology to effectively process massive multi-source heterogeneous industrial data and obtain its deep-level information. The existing data mining algorithms have overfitting in the industrial field, making it difficult to directly apply.

Method used

A data mining analysis method is proposed, including data collection and cleaning, creating data mining libraries, building mathematical mining models, and performing data mining analysis through optimization models, using automatic encoder to process data, and using Apache Flink and CEP engine to detect complex event patterns , build star or snowflake data models, perform advanced aggregation and multi-dimensional array processing, combine deep learning models to extract key features and perform exception scoring.

Benefits of technology

Realize deep information mining of industrial data, provide a visual modeling interface, reduce modeling threshold, is compatible with small factory mode and cloud edge mode, improves the automation efficiency of data management and analysis, and reduces efficiency problems and caliber deviations. , providing deeper insight.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119066117A_ABST
    Figure CN119066117A_ABST
Patent Text Reader

Abstract

The invention discloses a data mining analysis method and system for industrial data. The method comprises the steps that original data are collected from a data source and cleaned and converted; performing problem definition, and creating a data mining library for online analysis processing; constructing a mathematical mining model, and evaluating and optimizing the model; and performing data mining analysis by using the optimized model to obtain an optimal decision. According to the method, a large number of data processing and operator nodes are preset, streaming modeling is achieved in a dragging mode, and a user can conveniently and rapidly construct a data mining model. And the mining result is analyzed and displayed through various and rich visual means. On-line data analysis and machine learning are organically combined, and the process is optimized, so that data management and analysis are automated, and data analysis is more effectively carried out.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data engineering technology, and in particular to a data mining and analysis method and system for industrial data. Background Art

[0002] With the popularization and deepening of enterprise digitalization, the business operation process of enterprises is becoming increasingly automated, and a large amount of data is generated in the process of enterprise operation. These data and the information generated are valuable assets of enterprises, and they truthfully record the essential status of enterprise operation. However, in the face of such a large amount of data, traditional data analysis methods, such as data retrieval and statistical analysis, can only obtain surface information of the data, but cannot obtain its internal and deep information. Managers are faced with the dilemma of rich data but poor knowledge. How to mine useful knowledge for enterprise operation decision-making from these data is very important, and data mining is born to meet this need.

[0003] However, due to the wide variety of data sources, and the large number of semi-structured and unstructured problems with uncertain factors, many factors are difficult to calculate and evaluate scientifically with existing technologies due to the lack of historical data and corresponding statistical information. Therefore, other technologies and methods need to be applied to improve the value mining of enterprise data. Although there are many data mining algorithms and cases, they are still mainly used for network data. These algorithms have obvious overfitting phenomena in the industrial field and cannot be used directly, and some optimization adjustments need to be made. Summary of the invention

[0004] In view of the above-mentioned problems, the present invention is proposed.

[0005] Therefore, the technical problem solved by the present invention is: how to process massive multi-source heterogeneous data and obtain its inherent deep information.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: a data mining and analysis method for industrial data, comprising:

[0007] Collect raw data from data sources and clean and transform it;

[0008] Perform problem definition and create a data mining library for online analytical processing;

[0009] Build mathematical mining models, and evaluate and optimize them;

[0010] Use the optimized model to conduct data mining analysis and obtain the best decision.

[0011] As a preferred solution of the data mining and analysis method for industrial data described in the present invention, wherein: the collection of raw data includes, the data collection method is divided into offline collection and real-time collection, and the collected data is uniformly aggregated into the data warehouse; offline collection is based on Dolphin Scheduling Integration Datax service, supports data sources in the form of library tables, interfaces, FTP files, time series data definitions, CSV file import, and time series database data import, supports conversion and mapping between structured and semi-structured data, and real-time collection is based on fink+kafka messages and cdc log modes; different types of data collected in different modes are ultimately connected to a unified data warehouse.

[0012] As a preferred solution of the data mining and analysis method for industrial data described in the present invention, wherein: the cleaning and conversion include, the autoencoder is composed of an encoder and a decoder, and the representation of the data is learned by minimizing the reconstruction error;

[0013] Input data D raw , after passing through encoder E and decoder D, the repaired data D is output a The formula is:

[0014] D a =D(E(D raw ))

[0015] The loss function L is expressed as:

[0016] L=||D raw -D(E(D raw ))

[0017] Process and filter real-time data through Apache Flink; use the CEP engine to detect complex event patterns in real-time data streams, and set the CEP rule to φ CEP , the processed data set D CEP It is expressed as:

[0018] D CEP ={s i ∈D stream |φ CEP (s i )}

[0019] The event pattern for detecting both abnormal temperature and pressure is expressed as:

[0020] φ CEP (s i )=(s i [temp]>threshold1)∧(s i [pres]>threshold2)

[0021] For detected event data, use an autoencoder to perform real-time repair:

[0022] s′ i =D(E(s i ))

[0023] Automatically detect outliers and missing values ​​in the data and repair them; for missing values, the autoencoder will fill in the missing values ​​during the reconstruction process; for outliers, the autoencoder will reconstruct them into normal values.

[0024] As a preferred solution of the data mining and analysis method for industrial data described in the present invention, the creation of a data mining library includes recording detailed data through fine-grained data analysis using a detailed model; the data recording formula is expressed as:

[0025] Detail_Model = {d1, d2, ..., d n}

[0026] Among them, d i Represents the i-th detailed data record; the storage formula is expressed as:

[0027]

[0028] Among them, Size(d i ) represents the storage size of the i-th data record;

[0029] Use the aggregation model to summarize data analysis and aggregate the data by dimension. The data aggregation formula is expressed as:

[0030] Aggregate_Model={α1,a2,...,a m}

[0031] Among them, a i Represents the i-th aggregate data record; the storage formula is expressed as:

[0032]

[0033] Among them, Size(a i ) represents the storage size of the i-th aggregate data record;

[0034] Use a unique model to manage unique data. The formula is:

[0035] Unique_Model={u1,u2,...,u p}

[0036] Among them, u i Represents the i-th unique data record; the uniqueness constraint formula is expressed as:

[0037]

[0038] This formula ensures that each unique data record is not repeated in the collection; storage formula:

[0039]

[0040] Among them, Size(u i ) represents the storage size of the i-th unique data record; an appropriate data model is selected according to the data characteristics.

[0041] As a preferred solution of the data mining and analysis method for industrial data described in the present invention, the online analytical processing includes establishing a series of hypotheses, and obtaining data analysis conclusions by confirming or refuting the hypotheses through online analytical processing OLAP;

[0042] Load data from the data warehouse to the OLAP system. The data loading formula is expressed as:

[0043] OLAP_Data(t)=Load(Warehouse_Data(t))

[0044] Among them, OLAP_Data(t) represents the data loaded into the OLAP system at time t; Warehouse_Data(t) represents the data stored in the data warehouse at time t; Load represents the operation of loading data from the data warehouse; Build a star or snowflake data model and define fact tables and dimension tables;

[0045] Perform advanced aggregation operations on the data. The advanced data aggregation formula is expressed as:

[0046]

[0047] w represents the weight vector, which indicates the weight of each data point; x represents the data value vector, which indicates the value of each data point; w i represents the weight of the i-th data point; x i represents the value of the i-th data point;

[0048]

[0049] Among them, μ w represents the weighted average;

[0050] The multidimensional array aggregation formula is expressed as:

[0051]

[0052] Among them, A i,j,kRepresents the (i, j, k)th element in the aggregated multidimensional array; D l,m,n Represents the (l,m,n)th element in the original data array; f i,j,k (l,m,n) represents the aggregation function, which defines how to aggregate the original data into a new multidimensional array; L,M,N represents the dimension size of the original data array;

[0053] Data slicing and dicing operations are performed in different dimensions to support complex queries; the slicing formula is expressed as:

[0054] Slice(D,d,v)={r|r∈D∧r[d]=v}

[0055] Where D represents the data set; d represents the dimension to be sliced; v represents the specific value of the slice dimension; r represents the data record that meets the conditions;

[0056] The block cutting formula is expressed as:

[0057]

[0058] where {d1,d2,...,d k} represents multiple dimensions to be sliced; {v1,v2,...,v k} represents a specific value of the slicing dimension;

[0059] Supports data drilling and roll-up operations to perform data analysis at different levels; the drilling formula is expressed as:

[0060] Drill_Down(D,h,l)={r|r∈D∧r[h]∈l}

[0061] Among them, h represents the high-level dimension; l represents the set of low-level dimensions.

[0062] As a preferred solution of the data mining and analysis method for industrial data described in the present invention, the construction of the mathematical mining model includes extracting key features using advanced feature engineering technology, and the feature engineering formula is expressed as:

[0063] Feature i =f(T,S,X)

[0064] Among them, T represents time series features, S represents spatial features, X represents other features, and f represents feature extraction function; the deep learning model formula is expressed as:

[0065] h t =o(W h ·h t-1 +W x ·x t +b h )

[0066] c t =f(W c ·c t-1 +W ic ·x t +b c )

[0067] y t =W y ·h t +b y

[0068] Among them, h t represents the hidden state of the current time step; c t Indicates the cell state; x t Indicates input; W h , W x , W c , W ic , W y represents the weight matrix, b h , b c , b y represents bias, σ represents activation function;

[0069] The loss function formula is expressed as:

[0070]

[0071] Among them, yi represents the true value, represents the predicted value, and θ represents the model parameters.

[0072] As a preferred solution of the data mining and analysis method for industrial data described in the present invention, wherein: the data mining and analysis includes calculating relative deviation and weighted deviation;

[0073] Δ i Represents the relative deviation of the i-th data point;

[0074]

[0075] Among them, X i represents the actual observation value at the i-th time point; represents the predicted value at the i-th time point; let W i represents the weighted deviation of the i-th data point, which is weighted by combining the sigmoid function and the absolute error;

[0076]

[0077] Calculate the anomaly score, the formula is expressed as:

[0078]

[0079] M represents the total number of data points; H represents the set value of high deviation; if the deviation exceeds H, the data point is considered to be abnormal; (W i >H) represents the indicator function; when the weighted deviation W i When the set value H is exceeded, the function value is 1, otherwise it is 0;

[0080] Threshold represents the set abnormal proportion threshold. When the deviation is greater than H, calculation and judgment are performed; when AS>threshold, it is judged as data abnormality; when AS≤threshold, the prediction model parameters and the deviation setting value are adjusted, and the abnormality judgment is re-performed.

[0081] A data mining and analysis system for industrial data using any of the methods described in the present invention, comprising:

[0082] The data collection and preprocessing module uses Kafka and Flink for data collection and real-time processing, as well as data cleaning and standardization;

[0083] Data mining library and online analysis, create a data mining library, use Doris for data storage and OLAP analysis;

[0084] Mathematical mining module, building LSTM model, and using Bayesian optimization to tune model parameters;

[0085] The data mining and decision-making module uses the optimized model for data mining analysis; Tableau is used for data visualization and decision support.

[0086] A computer device comprises: a memory and a processor; the memory stores a computer program, wherein the processor implements the steps of any one of the methods of the present invention when executing the computer program.

[0087] A computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of any one of the methods of the present invention.

[0088] The beneficial effects of the present invention are as follows: 1. It provides users with a visual modeling interface, pre-sets a large number of data processing and operator nodes, and implements streaming modeling by dragging and dropping, which is convenient for users to quickly build data mining models. Modeling is mainly based on dragging and dropping, which greatly reduces the threshold for participation in modeling work. 2. Data mining results are often not easy to understand, and they are seamlessly integrated with other visualization functions to analyze and display the mining results through a variety of rich visualization methods. 3. It is compatible with both the small factory mode and the cloud edge mode, and multiple development modes, which reduces the requirements for the skill level of developers and has high flexibility and availability in different scenarios. The computing engine is suitable for online visual analysis of massive data. 4. Online data analysis and machine learning (ML) are organically combined and processes are optimized to automate data management and analysis, thereby performing data analysis more effectively. It reduces the efficiency problems and caliber deviations caused by the current reliance on IT processing, allowing users to gain deeper insights. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0090] Figure 1 An overall flow chart of a data mining and analysis method for industrial data provided by the first embodiment of the present invention. DETAILED DESCRIPTION

[0091] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0092] Example 1

[0093] Reference Figure 1 , is an embodiment of the present invention, and provides a data mining and analysis method for industrial data, comprising:

[0094] S1: Collect raw data from data sources and clean and transform them.

[0095] Furthermore, the collection of raw data includes that the data collection method is divided into offline collection and real-time collection, and the collected data is uniformly aggregated into a data warehouse.

[0096] Furthermore, offline collection is based on the integration of Datax services with Dolphin Scheduling, supports data sources in the form of library tables, interfaces, FTP files, time series data definitions, CSV file import, and time series database data import, supports conversion and mapping between structured and semi-structured data, and real-time collection is implemented based on the fink+kafka message and cdc log modes; different types of data collected in different modes are ultimately connected to a unified data warehouse.

[0097] Furthermore, the cleaning and converting includes that the autoencoder is composed of an encoder and a decoder, and learns the representation of the data by minimizing the reconstruction error.

[0098] Furthermore, input data D raw , after passing through encoder E and decoder D, the repaired data D is output a The formula is:

[0099] D a =D(E(D raw ))

[0100] Furthermore, the loss function L is expressed as:

[0101] L=||D raw -D(E(D raw ))

[0102] Furthermore, real-time data is processed and filtered through Apache Flink; the CEP engine is used to detect complex event patterns in real-time data streams. Let the CEP rule be φ CEP , the processed data set D CEP It is expressed as:

[0103] D CEP ={s i ∈D stream |φ CEP (s i )}

[0104] Furthermore, the event pattern for detecting both abnormal temperature and pressure is expressed as:

[0105] φ CEP (s i )=(s i [temp]>threshold1)∧(s i [pres]>threshold2)

[0106] Furthermore, the detected event data is repaired in real time using an autoencoder:

[0107] s′ i =D(E(si ))

[0108] Furthermore, outliers and missing values ​​in the data are automatically detected and repaired; for missing values, the autoencoder will fill in the missing values ​​during the reconstruction process; for outliers, the autoencoder will reconstruct them into normal values.

[0109] It should be noted that the mining objectives and problems need to be clearly defined. Clear problem definition can help mining personnel better choose appropriate analysis methods and techniques. For hydraulic pump failure maintenance, post-maintenance is mostly adopted. In contrast, predictive maintenance can arrange maintenance activities by predicting the previous state of the hydraulic pump, which has significant advantages such as automation and high efficiency. Hydraulic pump failure prediction uses sensors to collect data information of the excavator, and uses appropriate algorithms to evaluate the health status of the hydraulic pump and predict it before the failure occurs.

[0110] S2: Define the problem and create a data mining library for online analysis and processing.

[0111] Furthermore, the creation of the data mining library includes being used for fine-grained data analysis and recording each piece of data in detail; the data recording formula is:

[0112] Detail_Model = {d1, d2, ..., d n}

[0113] Among them, d i Represents the i-th detailed data record; storage formula:

[0114]

[0115] Among them, Size(d i ) represents the storage size of the i-th data record.

[0116] Furthermore, it is used to summarize data analysis and aggregate data by dimension. The data aggregation formula is expressed as:

[0117] Aggregate_Model={a1,a2,...,a m}

[0118] Among them, a i Represents the i-th aggregate data record; the aggregate function formula is:

[0119] a i =Aggregate Function(d i1 ,d i2 ,...,d ik )

[0120] Among them, Aggregate Function represents the aggregation function; ij Indicates the jth detailed data record participating in the aggregation; storage formula:

[0121]

[0122] Among them, Size(a i ) represents the storage size of the i-th aggregate data record.

[0123] Furthermore, the data uniqueness formula is used for unique data management to ensure the uniqueness of data records. The formula is expressed as:

[0124] Unique_Model={u1,u2,…,u p}

[0125] Among them, u i Represents the i-th unique data record; the uniqueness constraint formula is expressed as:

[0126]

[0127] Furthermore, the formula ensures that each unique data record is not repeated in the collection; storage formula:

[0128]

[0129] Among them, Size(u i ) represents the storage size of the i-th unique data record; an appropriate data model is selected according to the data characteristics.

[0130] Furthermore, the online analytical processing includes establishing a series of hypotheses and obtaining data analysis conclusions by confirming or refuting the hypotheses through online analytical processing (OLAP).

[0131] Furthermore, to load data from the data warehouse to the OLAP system, the data loading formula is expressed as:

[0132] OLAP_Data(t)=Load(Warehouse_Data(t))

[0133] Among them, OLAP_Data(t) represents the data loaded into the OLAP system at time t; Warehouse_Data(t) represents the data stored in the data warehouse at time t; Load represents the operation of loading data from the data warehouse; build a star or snowflake data model and define fact tables and dimension tables.

[0134] Furthermore, advanced data aggregation operations are performed on the data. The advanced data aggregation formula is expressed as:

[0135]

[0136] Among them, w represents the weight vector, which represents the weight of each data point; x represents the data value vector, which represents the value of each data point; w i represents the weight of the i-th data point; x i Represents the value of the i-th data point.

[0137]

[0138] Among them, μ w Represents a weighted average.

[0139] Furthermore, the multidimensional array aggregation formula is expressed as:

[0140]

[0141] Among them, A i,j,k Represents the (i, j, k)th element in the aggregated multidimensional array; D l,m,n Represents the (l,m,n)th element in the original data array; f i,j,k (l,m,n) represents the aggregation function, which defines how to aggregate the original data into a new multidimensional array; L,M,N represents the dimension size of the original data array.

[0142] Furthermore, data slicing and dicing operations are performed on different dimensions to support complex queries; the slicing formula is expressed as:

[0143] Slice(D,d,v)={r|r∈D∧r[d]=v}

[0144] Where D represents the data set, d represents the dimension to be sliced, v represents the specific value of the slice dimension, and r represents the data record that meets the conditions.

[0145] Furthermore, the slicing formula is expressed as:

[0146]

[0147] where {d1,d2,...,d k} represents multiple dimensions to be sliced; {v1,v2,...,v k} indicates a specific value for the slicing dimension.

[0148] Furthermore, it supports data drilling and roll-up operations to perform data analysis at different levels; the drilling formula is expressed as:

[0149] Drill_Down(D,h,l)={r|r∈D∧r[h]∈l}

[0150] Among them, h represents the high-level dimension; l represents the set of low-level dimensions.

[0151] It should be noted that this process uses the platform's large-scale data processing capabilities to standardize various heterogeneous data. In the process of visual modeling, all data processing logic is encapsulated into operation operators such as deduplication, complement, dictionary conversion, data desensitization, ID encoding, value domain cleaning, time domain cleaning, null value cleaning, duplicate cleaning, outlier cleaning, word frequency cleaning, slope cleaning, etc. Users perform task scheduling on these operation operators from the perspective of natural language, and then publish offline scheduling jobs or real-time monitoring tasks for data processing, and finally form model data that meets the needs of model training and archive them in the data mining library.

[0152] S3: Build mathematical mining models and evaluate and optimize the models.

[0153] Furthermore, the construction of the mathematical mining model includes extracting key features using advanced feature engineering technology, and the feature engineering formula is expressed as:

[0154] Feature i =f(T, S, X)

[0155] Among them, T represents time series features, S represents spatial features, X represents other features, and f represents feature extraction function; the deep learning model formula is expressed as:

[0156] h t =σ(W h ·h t-1 +W x ·x t +b h )

[0157] c t =f(W c ·c t-1 +W ic ·x t +b c )

[0158] y t =W y ·h t +b y

[0159] Among them, h t represents the hidden state of the current time step; c t Indicates the cell state; x t Indicates input; W h , W x , W c , W ic , W yrepresents the weight matrix, b h , b c , b y represents the bias, and σ represents the activation function.

[0160] Furthermore, the loss function formula is expressed as:

[0161]

[0162] Among them, y i represents the true value, represents the predicted value, and θ represents the model parameters.

[0163] It should be noted that the collected raw data is verified and processed to be clean and complete data that can be used for modeling. First, it is necessary to verify whether the data type of each field is logical. Then see whether its value range is in line with the actual situation, and then convert the raw data into the data we need through a series of processing. Data that exceeds the actual value range of each field is considered as abnormal deletion. For columns with missing values ​​​​deleting null values ​​​​> 80%, both continuous variables and discrete variables use the upward filling method to fill missing values. The reduction process uses the Zscore standardization method for continuous variables, and the discrete variables use one-hot encoding.

[0164] S4: Use the optimized model to conduct data mining analysis and obtain the optimal decision.

[0165] Furthermore, the data mining analysis includes calculating relative deviation and weighted deviation.

[0166] Furthermore, Δ i Represents the relative deviation of the ith data point.

[0167]

[0168] Among them, X i represents the actual observation value at the i-th time point; represents the predicted value at the i-th time point. Let W i Represents the weighted deviation of the i-th data point, which is weighted by combining the sigmoid function and the absolute error.

[0169]

[0170] Furthermore, the anomaly score is calculated as follows:

[0171]

[0172] Where M represents the total number of data points; H represents the set value of high deviation; if the deviation exceeds H, the data point is considered to be abnormal; (Wi >H) represents the indicator function; when the weighted deviation W i When the set value H is exceeded, the function value is 1, otherwise it is 0.

[0173] Furthermore, threshold represents the set abnormal proportion threshold. When the deviation is greater than H, calculation and judgment are performed; when AS>threshold, it is judged as data abnormality; when AS≤threshold, the prediction model parameters and the deviation setting value are adjusted, and the abnormality judgment is re-performed.

[0174] It should be noted that data mining and OLAP are complementary to each other. Before taking action based on the conclusions from data mining, you need to verify what impact such action will have on the enterprise. OLAP tools can answer these questions. In addition, in the early stages of knowledge discovery, OLAP tools have other uses. They can help you explore data, find out which variables are more important to a problem, and discover abnormal data and variables that affect each other. These operations can better understand the data and speed up the process of knowledge discovery.

[0175] On the other hand, this embodiment also provides a data mining and analysis system for industrial data, which includes:

[0176] The data collection and preprocessing module uses Kafka and Flink for data collection and real-time processing, as well as data cleaning and standardization.

[0177] Data mining library and online analysis, create a data mining library, use Doris for data storage and OLAP analysis.

[0178] Mathematical mining module, builds LSTM model, and uses Bayesian optimization to tune model parameters.

[0179] The data mining and decision-making module uses the optimized model for data mining analysis; Tableau is used for data visualization and decision support.

[0180] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0181] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0182] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0183] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0184] Example 2

[0185] The following is an embodiment of the present invention, which provides a data mining and analysis method for industrial data. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0186] Define the goals and problems that need to be mined; hydraulic pump failure is related to hydraulic pump wear or oil contamination, and the power comes from the engine, so select engine parameters and oil parameters, and transform the collected raw data into clean and complete data that can be used for modeling after verification and processing. First, it is necessary to verify whether the data type of each field is logical. Then see whether its value range is in line with the actual situation, and then convert the raw data into the data we need after a series of processing. Data that exceeds the actual value range of each field is considered abnormal and deleted. For missing values, delete columns with a null value ratio of >80%, and use the upward filling method for continuous variables and discrete variables to fill missing values. The Zscore standardization method is used for reduction processing of continuous variables, and the unique hot encoding is used for discrete variables.

[0187] The relevant operating statistical parameters of the hydraulic pump after preprocessing are shown in the following table.

[0188] Table 1 Hydraulic pump related operation statistical parameters

[0189]

[0190] If the hydraulic pump fails, according to the three manifestations of the failure, combined with the current sensor data, the pump pressure can be predicted for analysis: if the hydraulic pump fails, the generated pump pressure P will deviate from the normal pump pressure P' by △P. If the daily deviation △P / P exceeds a certain value w and the proportion exceeds the threshold, it is determined to be a hydraulic pump failure. Let the daily pump pressure be Y dayi , the corresponding predicted value is Y i , the fault judgment formula is expressed as:

[0191]

[0192] When the calculated result is greater than the set threshold, it is predicted to be a fault; otherwise, it is predicted to be normal.

[0193] Model verification. The pump pressure prediction value is obtained according to the above model. The deviation △P / P of each data is calculated according to the actual value and the predicted value. The daily proportion of different deviations (±5%, +10%, etc.) is calculated. Combining the trend chart analysis of failure time and deviation proportion, a set of model setting values ​​can be obtained: w = 20%, threshold = 20%. According to the model setting values, the prediction accuracy of other hydraulic pumps is verified, and the model parameters are adjusted to make it have a certain generalization ability.

[0194] The final analysis goal is to predict whether the hydraulic pump is faulty, which is a binary classification problem. The confusion matrix is ​​a method to evaluate the accuracy of classification problems. For binary classification problems, the combination of the true category and the predicted category is divided into four cases: true positive examples, false positive examples, true negative examples, and false negative examples. Let TP, FP, TN, and FN represent the corresponding number of samples. The confusion matrix we get is shown in the following table:

[0195] Table 2 Evaluation sample table

[0196] Reality Positive prediction results Negative prediction results Positive example 15 4 Negative example 21 69

[0197] Precision P = TP / (TP+FP) = 15 / (15+21) = 0.42

[0198] Recall R = TP / (TP+FN) = 15 / (15+4) = 0.79

[0199] It can be concluded that all hydraulic pump system failures are predicted, but false alarms occur from time to time, so it is necessary to optimize the judgment method to reduce false alarms.

[0200] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A data mining and analysis method for industrial data, characterized in that: include: Collect raw data from data sources and clean and transform it; Perform problem definition and create a data mining library for online analytical processing; Build mathematical mining models, and evaluate and optimize them; Use the optimized model to conduct data mining analysis and obtain the best decision.

2. The data mining and analysis method for industrial data according to claim 1, characterized in that: The collection of raw data includes: data collection methods are divided into offline collection and real-time collection, and the collected data are uniformly aggregated into a data warehouse; offline collection is based on Dolphin Scheduling's integrated Datax service, supports data sources in the form of library tables, interfaces, FTP files, time series data definitions, CSV file imports, and time series database data imports, supports conversion and mapping between structured and semi-structured data, and real-time collection is based on fink+kafka messages and cdc log modes; different types of data collected in different modes are ultimately connected to a unified data warehouse.

3. The data mining and analysis method for industrial data according to claim 2, characterized in that: The cleaning and conversion include that the autoencoder is composed of an encoder and a decoder, and learns the representation of the data by minimizing the reconstruction error; Input data D raw , after passing through encoder E and decoder D, the repaired data D is output a The formula is: D a =D(E(D raw )) The loss function L is expressed as: L=||D raw -D(E(D raw )) Process and filter real-time data through Apache Flink; Use the CEP engine to detect complex event patterns in real-time data streams. Let the CEP rule be φ CEP , the processed data set D CEP It is expressed as: D CEP ={s i ∈D stream |f CEP (s i )} The event pattern for detecting both abnormal temperature and pressure is expressed as: φ CEP (s i )=(s i [temp]>threshold1)Λ(s i [pres]>threshold2) For detected event data, use an autoencoder to perform real-time repair: s i =D(E(s i )) Automatically detect outliers and missing values ​​in the data and repair them; for missing values, the autoencoder will fill in the missing values ​​during the reconstruction process; for outliers, the autoencoder will reconstruct them into normal values.

4. The data mining and analysis method for industrial data according to claim 3, characterized in that: The creation of the data mining library includes recording detailed data by fine-grained data analysis using a detailed model; the data recording formula is expressed as: Detail_Model={d1,d2,…,d n } Among them, d i Represents the i-th detailed data record; the storage formula is expressed as: Among them, Size(d i ) represents the storage size of the i-th data record; Use the aggregation model to summarize data analysis and aggregate the data by dimension. The data aggregation formula is expressed as: Aggregate_Model={a1,a2,…,a m } Among them, a i Represents the i-th aggregate data record; the storage formula is expressed as: Among them, Size(a i ) represents the storage size of the i-th aggregate data record; Use a unique model to manage unique data. The formula is: Unique_Model={u1,u2,…,u p } Among them, u i Represents the i-th unique data record; the uniqueness constraint formula is expressed as: This formula ensures that each unique data record is not repeated in the collection; storage formula: Among them, Size(u i ) represents the storage size of the i-th unique data record; an appropriate data model is selected according to the data characteristics.

5. The data mining and analysis method for industrial data according to claim 4, characterized in that: The online analytical processing includes establishing a series of hypotheses and obtaining data analysis conclusions by confirming or refuting the hypotheses through online analytical processing (OLAP); Load data from the data warehouse to the OLAP system. The data loading formula is expressed as: OLAP_Data(t)=Load(Warehouse_Data(t)) Among them, OLAP_Data(t) represents the data loaded into the OLAP system at time t; Warehouse_Data(t) represents the data stored in the data warehouse at time t; Load represents the operation of loading data from the data warehouse; Build a star or snowflake data model and define fact tables and dimension tables; Perform advanced aggregation operations on the data. The advanced data aggregation formula is expressed as: w represents the weight vector, which indicates the weight of each data point; x represents the data value vector, which indicates the value of each data point; w i represents the weight of the i-th data point; x i represents the value of the i-th data point; Among them, μ w represents the weighted average; The multidimensional array aggregation formula is expressed as: Among them, A i,j,k Represents the (i, j, k)th element in the aggregated multidimensional array; D l,m,n Represents the (l,m,n)th element in the original data array; f i,j,k (l,m,n) represents the aggregation function, which defines how to aggregate the original data into a new multidimensional array; L,M,N represents the dimension size of the original data array; Data slicing and dicing operations are performed in different dimensions to support complex queries; the slicing formula is expressed as: Slice(D,d,v)={r|r∈D∧r[d]=v} Where D represents the data set; d represents the dimension to be sliced; v represents the specific value of the slice dimension; r represents the data record that meets the conditions; The cutting formula is expressed as: Among them, {d1,d2,…,d k } represents multiple dimensions to be sliced; {v1,v2,…,v k } represents a specific value of the slicing dimension; Supports data drilling and roll-up operations to perform data analysis at different levels; the drilling formula is expressed as: Drill_Down(D,h,l)={r|r∈D∧r[h]∈l} Among them, h represents the high-level dimension; l represents the set of low-level dimensions.

6. The data mining and analysis method for industrial data according to claim 5, characterized in that: The construction of the mathematical mining model includes extracting key features using advanced feature engineering technology. The feature engineering formula is expressed as: Feature i =f(T,S,X) Among them, T represents time series features, S represents spatial features, X represents other features, and f represents feature extraction function; the deep learning model formula is expressed as: h t =σ(W h ·h t-1 +W x ·x t +b h ) c t =f(W c ·c t-1 +W ic ·x t +b c ) y t =W y ·h t +b y Among them, h t represents the hidden state of the current time step; c t Indicates the cell state; x t Indicates input; W h , W x , W c , W ic , W y represents the weight matrix, b h 、b c 、b y represents bias, σ represents activation function; The loss function formula is expressed as: Among them, yi represents the true value, represents the predicted value, and θ represents the model parameters.

7. The data mining and analysis method for industrial data according to claim 6, characterized in that: The data mining analysis includes calculating relative deviation and weighted deviation; Δ i Represents the relative deviation of the i-th data point; Among them, X i represents the actual observation value at the i-th time point; represents the predicted value at the i-th time point; let W i represents the weighted deviation of the i-th data point, which is weighted by combining the sigmoid function and the absolute error; Calculate the anomaly score, the formula is expressed as: M represents the total number of data points; H represents the set value of high deviation; if the deviation exceeds H, the data point is considered to be abnormal; (W i >H) represents the indicator function; when the weighted deviation W i When the set value H is exceeded, the function value is 1, otherwise it is 0; Threshold represents the set abnormal proportion threshold. When the deviation is greater than H, calculation and judgment are performed; when AS>threshold, it is judged as data abnormality; when AS≤threshold, the prediction model parameters and the deviation setting value are adjusted, and the abnormality judgment is re-performed.

8. A data mining and analysis system for industrial data using the method according to any one of claims 1 to 7, characterized in that: The data collection and preprocessing module uses Kafka and Flink for data collection and real-time processing, as well as data cleaning and standardization; Data mining library and online analysis, create a data mining library, use Doris for data storage and OLAP analysis; Mathematical mining module, building LSTM model, and using Bayesian optimization to tune model parameters; The data mining and decision-making module uses the optimized model for data mining analysis; Tableau is used for data visualization and decision support.

9. A computer device comprising: Memory and processor; The memory stores a computer program, and is characterized in that: when the processor executes the computer program, the steps of the data mining and analysis method for industrial data are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a data mining analysis method for industrial data are implemented.

Citation Information

Patent Citations

  • Data mining analysis method

    CN117453764A

  • Carbon dot catalyst co-doped with nitrogen and sulfur derived from feathers, and its manufacturing method and application

    KR1020240113715A

  • Data-centric automatic data mining

    US20060136462A1

  • Data mining application programming interface

    US6865573B1