Industrial Internet of Things heterogeneous data prediction method and device based on attention mechanism

Abstract

The invention discloses an attention mechanism-based industrial Internet of Things heterogeneous data prediction method. The method comprises the following steps of: obtaining a historical data set and constructing random filling data according to a random number; performing data filling operation on the initial monitoring data based on randomly generated filling data to obtain reconstructed monitoring data under the same dimension; forming a data set by the initial monitoring data, the corresponding reconstructed monitoring data and the labels; training a neural network prediction model by using the data set to obtain a monitoring data prediction model; and inputting the current initial monitoring data into the monitoring data prediction model to obtain reconstructed monitoring data of the next time node. The invention further provides an industrial Internet of Things heterogeneous data prediction device. According to the method provided by the invention, the equipment operation state and the process parameters in the industrial scene can be effectively coupled, so that the real-time performance of the health state evaluation of the processing equipment is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent manufacturing, and particularly relates to an industrial Internet of Things heterogeneous data prediction method and device based on an attention mechanism. Background Technique

[0002] In the field of industrial Internet of Things, real-time monitoring and predictive analysis, as key technologies to improve production efficiency and ensure equipment reliability, are significantly restricted by the problem of heterogeneous data processing. Due to the significant differences in sensor configurations of different models of equipment in the industrial production environment, the collected time-series data often exhibits multi-dimensional heterogeneous characteristics, specifically manifested as inconsistent data feature dimensions. Traditional data-driven methods usually require the input data to have a unified structured feature dimension. This rigid constraint forces system developers to face a dilemma: either build a dedicated prediction model for a specific device group or use data filling techniques to forcibly unify the data dimensions. The former will generate exponentially increasing computational costs during large-scale deployment, while the latter relies on traditional methods such as linear interpolation and mean filling. When dealing with industrial time-series data with non-linear dynamic correlations, it often leads to the loss of key feature information or the aggravation of noise interference. More notably, the coupling relationship between the equipment operating state and process parameters in the industrial scenario has time-varying characteristics, and static data alignment strategies are difficult to effectively capture this dynamic evolution law. The resulting data distortion phenomenon directly affects the generalization ability of the prediction model. This underlying data processing defect not only reduces the real-time performance of equipment health status assessment but may also lead to misjudgments of key process parameters, ultimately resulting in increased maintenance costs and decreased decision-making reliability.

[0003] Patent document CN119415904A discloses a device fault warning method, device, medium, and system based on the industrial Internet of Things, including: collecting various operating data of the device; determining the initial attenuation factor of each operating data according to the volatility of each operating data; for any operating data, adjusting the initial attenuation factor based on the noise probability of each discrete data in the operating data to obtain the adaptive attenuation factor of the operating data; using the exponentially weighted average algorithm for prediction based on the adaptive attenuation factors of each operating data to obtain several predicted values of each operating data; and performing device fault warning according to the several predicted values of each operating data.

[0004] Patent document CN119363624A discloses an industrial Internet of Things system anomaly detection method based on multivariate time series data, including: S1, dividing the original multivariate time series data into an original training set and an original test set; S2, performing normalization processing on the original training set and the original test set, and then performing time window segmentation to obtain a new training set and a new test set; S3, constructing a GTVformer model; S4, the GTVformer model trains the data in the new training set in two stages, and trains two Transformer models in an adversarial manner to reconstruct the normal input window data; S5, using the trained GTVformer model to predict the data in the new test set to obtain a marking result. Summary of the Invention

[0005] The purpose of the present invention is to provide an industrial Internet of Things heterogeneous data prediction method and device based on an attention mechanism. This method can effectively couple the operating states of devices and process parameters in industrial scenarios, thereby realizing the real-time evaluation of the health status of processing equipment.

[0006] In order to achieve the first object of the present invention, the following technical solutions are provided: An industrial Internet of Things heterogeneous data prediction method based on an attention mechanism, including the following steps:

[0007] Obtain a historical data set and construct random filling data according to a random number. The historical data set includes initial monitoring data collected by different types of sensors;

[0008] Perform a data filling operation on the initial monitoring data based on the randomly generated filling data to obtain reconstructed monitoring data in the same dimension;

[0009] Label the initial monitoring data and the corresponding reconstructed monitoring data with time nodes, and form a data set with the initial monitoring data, the corresponding reconstructed monitoring data, and the labels;

[0010] Use the data set to train a pre-constructed neural network prediction model to obtain a monitoring data prediction model for predicting the monitoring data at the next time node;

[0011] Input the current initial monitoring data into the monitoring data prediction model to obtain the reconstructed monitoring data at the next time node.

[0012] Specifically, the neural network prediction model includes a preprocessing module, a feature extraction module, and a prediction module;

[0013] The preprocessing module is used to perform a data filling operation on the input initial monitoring data to generate corresponding reconstructed monitoring data and input it into the feature extraction module;

[0014] The feature extraction module is used to extract the data features of the input reconstructed monitoring data and input the extracted data features into the prediction module;

[0015] The prediction module makes a prediction based on the input data features to output a prediction result.

[0016] Specifically, the specific process of the data filling operation is as follows:

[0017] Using the initial monitoring data as the query volume of the attention mechanism, taking the constructed random filling data as the key and value of the attention mechanism, calculating the similarity between the initial monitoring data and the random filling data to obtain the corresponding similarity score, and calculating the attention weight by combining the similarity score with the masked activation function to obtain the corresponding attention weight;

[0018] Multiplying the attention weight by the random filling data to output the corresponding filling data;

[0019] Concatenating the initial monitoring data with the corresponding filling data to construct the corresponding reconstructed monitoring data.

[0020] Specifically, the attention mechanism includes additive attention, scaled dot-product attention, or general attention.

[0021] Specifically, during the calculation of the attention weight, the filling data is masked by the masked activation function.

[0022] Specifically, the expression of the masked activation function is as follows:

[0023]

[0024] where x[:l val means selecting the first l val terms of x.

[0025] Specifically, the concatenation operation is implemented using the Concatenate function.

[0026] Specifically, during the training process, the neural network prediction model is trained using the mean absolute error or / and the mean squared error.

[0027] To achieve the second object of the present invention, the following technical solution is provided: An industrial Internet of Things heterogeneous data prediction device for performing the steps of the above-mentioned industrial Internet of Things heterogeneous data prediction method based on the attention mechanism.

[0028] Compared with the prior art, the beneficial effects of the present invention:

[0029] It can avoid the limitations of traditional filling strategies, generate new features through the attention mechanism, and attach these generated features to the original data as filling data, thereby achieving the real-time evaluation of the device health status. Description of the Drawings

[0030] Figure 1 It is a flowchart of the industrial Internet of Things heterogeneous data prediction method provided in this embodiment;

[0031] Figure 2 It is a schematic diagram of the prediction results of different injection molding devices provided in this embodiment. Detailed Implementation Manner

[0032] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Usually, the components of the embodiments of the present invention described and illustrated herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0033] As Figure 1 shown, a kind of industrial Internet of Things heterogeneous data prediction method provided in this embodiment is as follows:

[0034] The given input data is the original data X and the maximum value l of the number of features in the original data max , and the dimension of the original data is: batch size n batch , historical data length T his , number of data features l val .

[0035] First, initialize the random filling data E, and the data dimension is 1, historical data length T his , maximum number of data features l max .

[0036] Take the input original data as the query volume of the attention mechanism, that is, Q←X; take the filling data as the key and value of the attention mechanism, that is, K←E, V←E.

[0037] Calculate the similarity score S between the query and the key, and calculate the attention weight. The calculation method here can be selected from different forms such as additive attention, scaled dot-product attention, or general attention according to the selected attention mechanism.

[0038] For additive attention, the similarity function is calculated by Equation (1):

[0039] S add (Q, K) = tanh(W q Q + W k K) (1)

[0040] where S add is the similarity score corresponding to additive attention, and W q , W k are learnable parameters.

[0041] The calculation method of the attention weight of additive attention is as shown in Equation (2):

[0042]

[0043] where α add is the additive attention weight, W v is a learnable parameter, softmax is a commonly used activation function in neural networks, and the softmax mask function is the masked softmax function, where the padded data is masked, and its calculation method is as shown in Equation (3),

[0044]

[0045] where x[:l val represents selecting the first l val terms of x.

[0046] The calculation methods of the similarity function and attention weight of scaled dot - product attention are as shown in Equations (4) and (5):

[0047]

[0048] α dot = softmax mask (S dot (Q, K)) (5)

[0049] where S dot (Q, K) is the similarity function of scaled dot - product attention, d k is the dimension of the key sequence, and α dot is the attention weight of scaled dot - product attention.

[0050] The calculation methods of the similarity function and attention weight of general attention are as shown in Equations (6) and (7):

[0051] S gen (Q, K) = QW rK(6)

[0052] α gen = softmax mask (S gen (Q, K))(7)

[0053] where S gen (Q, K) is the similarity function of general attention, W r is a trainable parameter, and α gen is the attention weight of general attention.

[0054] Multiply the attention weight by the value to obtain the final padding data, as shown in Equation (8):

[0055] P = αV(8)

[0056] where P is the padding data and α is the attention weight calculated by the selected attention mechanism.

[0057] Concatenate the padding data with the original data to obtain the finally padded data, as shown in Equation (9):

[0058] X pad = Concatenate(X[:, :, :l val , E[:, :, l val :l max , -1)(9)

[0059] where X pad is the finally padded data, the Concatenate(a, b, -1) function represents concatenating a and b along the last dimension, and X[:, :, :l val represents selecting the first l val values for the last dimension of X.

[0060] After completing the data padding operation, input the padded data X pad into the prediction model. This method uses an encoder-decoder long short-term memory neural network (LSTM) based on temporal attention. The encoder-decoder architecture includes an encoder and a decoder. The encoder encodes the input data into a context vector of a fixed dimension, and the decoder generates the final prediction value based on the context vector. The encoder in this method is an LSTM neural network, and the decoder is an LSTM neural network based on temporal attention. In the encoder, the LSTM neural network performs calculations for each time step, as shown in Equations (10), (11), and (12):

[0061] i t = σ(W i [h t-1 , Xt +b i ) (10)

[0062] f t = σ(W f [h t-1 , X t +b f ) (11)

[0063] o t = σ(W o [h t-1 , X t +b o ) (12)

[0064] where i t , f t , o t are the input gate, forget gate, and output gate respectively, t is the historical time step, W i , W f , W o are trainable network weights, b i , b f , b o are bias vectors, h t-1 is the hidden layer state, X t is the input data at the current historical time step, σ represents the sigmoid activation function, and its expression is

[0065] The update methods of the neuron state and hidden layer state of the LSTM neural network are shown in Equations (13) and (14):

[0066] c t = f t ⊙ c t-1 + i t ⊙ tanh(W c [h t-1 , X t +b c ) (13)

[0067] h t = o t ⊙ tanh(c t ) (14)

[0068] where c t is the neuron state, h t is the hidden layer state, W c and b c are the learnable network weight and bias vector respectively, and ⊙ represents element-wise multiplication. Dropout is used at the end of the encoder to prevent overfitting during network training. The final neuron state and the hidden layer state are passed to the decoder as context vectors.

[0069] In the decoder, first, the neuron state and the hidden layer state of the LSTM network are initialized according to the context vector, as shown in Equations (15) and (16):

[0070]

[0071] where are respectively the initial neuron state and the hidden layer state of the LSTM network in the decoder.

[0072] In the decoder, first, the time attention mechanism is used to extract the key features from the historical monitoring data. The final layer hidden state of the decoder at the previous time step is used as the query volume of the attention mechanism, i.e., Q dec ← h dec [-1]; the final layer hidden states of all time steps of the encoder are used as the key and value of the attention mechanism, i.e., K dec ← h enc , V dec ← h enc . Additive attention, scaled dot-product attention, general attention, etc. can be used in time attention, and the calculation methods are shown in Equations (1) to (7).

[0073] The historical data features extracted by the time attention mechanism are shown in Equation (17):

[0074] Ccotext = αV (17)

[0075] where Context is the historical data features extracted, and α is the attention weight calculated by the time attention mechanism.

[0076] The historical data features are concatenated with the input of the decoding layer, as shown in Equation (18):

[0077] X dec,cat = Concatenate(Context, X dec , -1) (18)

[0078] where X dec,cat is the concatenated decoder input, X dec is the original decoder input, and the Concatenate(a, b, -1) function represents concatenating a and b along the last dimension.

[0079] The calculation process of the LSTM network is shown in Equations (10)-(14), and finally the final prediction value is obtained through the fully connected layer and the softmax activation function, as shown in Equation (19):

[0080]

[0081] Among them, is the final prediction value, Y dec is the output value of the decoder, W y and b y are the learnable network weights and bias vectors of the fully connected layer respectively, and the calculation method of the softmax function is

[0082] The loss function is calculated according to the prediction value. The forms of the loss function include the mean absolute error or the mean squared error, etc. At the same time, a masked loss function is adopted. When calculating the prediction accuracy, only the data of the original length is considered for comparison, and its calculation method is shown in Equations (14) and (15):

[0083]

[0084] Among them, L mask,MAE and L mask,MSE represent the masked mean absolute error and the masked mean squared error loss functions respectively.

[0085] During the training process, the learnable parameters are continuously updated, etc. If the number of training times reaches the set value n epoch , or the number of consecutive training times with the loss function value not decreasing reaches the set value n patience , stop training. The finally obtained model can achieve accurate prediction of time series.

[0086] This embodiment also provides an industrial Internet of Things heterogeneous data prediction device for performing the steps of the industrial Internet of Things heterogeneous data prediction method provided in the above embodiment.

[0087] To better illustrate the technical effects of the method provided in this embodiment, it is verified in the actual injection molding production under different dimensions. In the barrel heating process of injection molding, due to the different numbers of barrel temperature segments of different molding machines and the different numbers of installed sensors, the collected data dimensions are different. To solve this problem, the method is applied for verification. Experiments collect data from three different injection molding devices, as Figure 2 shown, which are an injection molding machine with four temperature segments, a temperature control platform with five temperature segments, and an injection molding machine with six temperature segments.

[0088] In addition, the terms "upper", "lower", "inner", "outer", "front", and "rear" are for descriptive purposes only and should not be construed as indicating or implying relative importance. Unless specifically stated otherwise, the relative steps, numerical expressions, and values of the components and steps set forth in these embodiments do not limit the scope of the present invention.

[0089] Of course, the above are only specific embodiments of the present invention and are not intended to limit the scope of implementation of the present invention. Any equivalent changes or modifications made according to the structure, features, and principles described in the scope of the patent application of the present invention should be included in the scope of the patent application of the present invention.

[0090] Finally, it should be noted that the above embodiments are only specific implementation manners of the present invention, used to illustrate the technical solutions of the present invention, rather than limiting it. The protection scope of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that any person skilled in the art within the technical scope disclosed by the present invention can still modify the technical solutions described in the foregoing embodiments or can easily conceive of changes, or perform equivalent replacements on some of the technical features; and these modifications, changes, or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention and should all be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. An industrial Internet of Things heterogeneous data prediction method based on an attention mechanism, characterized in that It includes the following steps: Obtain a historical data set and construct random filling data according to random numbers. The historical data set includes initial monitoring data collected by different types of sensors; Perform data filling operations on the initial monitoring data based on the randomly generated filling data to obtain reconstructed monitoring data in the same dimension; Label the initial monitoring data and the corresponding reconstructed monitoring data with time nodes, and form a data set by combining the initial monitoring data, the corresponding reconstructed monitoring data and labels; Use the data set to train a pre-constructed neural network prediction model to obtain a monitoring data prediction model for predicting the monitoring data at the next time node; Input the current initial monitoring data into the monitoring data prediction model to obtain the reconstructed monitoring data at the next time node.

2. The industrial Internet of Things heterogeneous data prediction method based on the attention mechanism according to claim 1, wherein The neural network prediction model includes a preprocessing module, a feature extraction module and a prediction module; The preprocessing module is used to perform data filling operations on the input initial monitoring data to generate corresponding reconstructed monitoring data and input them into the feature extraction module; The feature extraction module is used to extract the data features of the input reconstructed monitoring data and input the extracted data features into the prediction module; The prediction module makes predictions according to the input data features and outputs prediction results.

3. The method for predicting heterogeneous data in industrial Internet of Things based on attention mechanism according to claim 1 or 2, characterized in that The specific process of the data filling operation is as follows: Use the initial monitoring data as the query volume of the attention mechanism, use the constructed random filling data as the key and value of the attention mechanism, calculate the similarity between the initial monitoring data and the random filling data to obtain the corresponding similarity score, and use the similarity score to combine the masked activation function to calculate the attention weight to obtain the corresponding attention weight; Multiply the attention weight by the random filling data to output the corresponding filling data; Perform a concatenation operation on the initial monitoring data and the corresponding filling data to construct the corresponding reconstructed monitoring data.

4. The method for predicting heterogeneous data of industrial Internet of Things based on attention mechanism according to claim 3, characterized in that, The attention mechanism includes additive attention, scaled dot-product attention or general attention.

5. The industrial Internet of Things heterogeneous data prediction method based on the attention mechanism according to claim 3, wherein During the calculation of the attention weight, the filling data is masked by the masked activation function.

6. The method for predicting heterogeneous data of industrial Internet of Things based on attention mechanism according to claim 3, characterized in that The expression of the masked activation function is as follows: Among them, x[:l val means selecting the first l val terms of x.

7. The method for predicting heterogeneous data in industrial Internet of Things based on attention mechanism according to claim 3, wherein The concatenation operation is implemented using the Concatenate function.

8. The method for predicting heterogeneous data of industrial Internet of Things based on attention mechanism according to claim 1, characterized in that During the training process, the mean absolute error or / and the mean squared error are used to train the neural network prediction model.

9. An industrial Internet of Things heterogeneous data prediction device, characterized in that, Steps for performing the attention mechanism-based industrial Internet of Things heterogeneous data prediction method according to any one of claims 1 to 8.

Images