Adaptive fitting and prediction method applied to multi-modal industrial big data prediction

Abstract

The invention discloses a self-adaptive fitting and prediction method applied to multi-modal industrial big data prediction, and the method is used for processing and predicting multi-modal data in a table form, converting the data into an embedded sequence matrix, and achieving efficient modeling through a self-attention mechanism. The processing flow of the embedded sequence naturally supports multi-modal data, and various information such as digits, texts, images and the like can be uniformly processed. And new modal data can be quickly accessed only by introducing a pre-trained embedded encoder, so that the adaptability of the model is remarkably improved. According to the method, the data fitting effect and the calculation cost can be flexibly balanced, and an efficient training and deployment solution is provided. The method facilitates the implementation of transfer learning and continuous learning, and remarkably improves the practical application value of the model in an industrial data scene. Especially under the complex multi-modal data processing requirement, efficient and unified modeling is achieved, the development and maintenance cost is reduced, and an effective technical path is provided for intelligent analysis of industrial data.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning and industrial data processing, and in particular to an adaptive fitting and prediction method applied to multimodal industrial big data prediction. Background Art

[0002] Tabular data prediction tasks are widespread. Tabular data prediction in power systems is a class of application problems with important practical significance, widely present in fields such as electricity, energy, and manufacturing. Tabular data is typically stored in a structured form and contains multiple types of feature columns, such as numerical features (current, voltage, power, etc.), categorical features (equipment type, operating status, etc.), and image features (equipment status images, etc.). It is multimodal, multiformat, has varying data sizes, and is complex and changeable. The multimodal properties and diverse patterns of power system data pose many challenges to fault detection and classification tasks, such as handling feature heterogeneity, modeling feature interactions, and data missingness. These challenges require models to effectively integrate and utilize multiple types of feature information.

[0003] Among existing solutions, boosting-based methods, such as Gradient Boosting Decision Trees (GBDT) and its optimized version, xgboost, have demonstrated effectiveness for purely numerical and categorical tabular data. xgboost improves the prediction accuracy and training efficiency of tabular data by introducing techniques such as a weighted tree-structured combination strategy, regularization terms, and distributed parallel computing. While these methods can effectively capture nonlinear relationships between numerical and categorical data and are commonly used for tasks such as feature selection and feature importance analysis, the hyperparameter tuning process for the gradient boosting tree method is complex and time-consuming, and its ability to scale to large data volumes is limited, making it difficult to adapt to continuous learning scenarios in power systems. While methods like xgboost outperform traditional neural networks in certain tasks, their limited ability to handle multimodal data imposes stringent requirements on data preprocessing and feature engineering, hindering their further development and application to more complex real-world problems. With the rapid development of deep learning technology, neural network-based methods have gradually been applied to tabular data prediction tasks, with networks such as TabNet being a representative example. TabNet utilizes an attention mechanism to distribute learning paths, dynamically selecting features. This improves model interpretability compared to traditional single-layer perceptron architectures and reduces the burden of feature engineering to a certain extent. Compared to xgboost, TabNet possesses inherent advantages in processing large-scale and multimodal data. In particular, its transfer learning and continuous learning capabilities are directly adaptable to deep learning frameworks, demonstrating potential in equipment condition monitoring and multi-source data fusion. However, deep learning networks like TabNet currently have poor adaptability to small and sparse datasets, making them prone to overfitting. Most existing deep learning models struggle to balance predictive performance and computational efficiency, and their deployment in industrial environments, in particular, still faces numerous technical bottlenecks. Summary of the Invention

[0004] The purpose of the present invention is to address the defects and shortcomings of the above-mentioned existing technologies and propose an adaptive fitting and prediction method for multimodal industrial big data prediction. This method is aimed at the task of power system equipment fault detection and classification. By efficiently modeling multimodal sensor data (numerical time series signals such as current, voltage, temperature, equipment model categories, thermal imaging images, etc.), it solves the problems of insufficient modeling of time series correlation, difficulty in fusing multimodal features, and poor robustness to missing sensor data in traditional methods.

[0005] The technical solution adopted by the present invention to solve the technical problem is: an adaptive fitting and prediction method applied to multimodal industrial big data prediction, the method comprising the following steps:

[0006] Step 1: Data preprocessing: convert the original data into numerical and embedded form through a multimodal encoder. For numerical features, use methods such as mean variance normalization or Box-Cox transformation to transform; for categorical features, embed them using the category embedding matrix Where C is the number of categories and d is the embedding dimension;

[0007] Step 2: Combine numerical features and categorical features into a unified embedding sequence matrix to form a unified embedding X = [x scaled , x encoded , t]. Where t is a learnable additional tag embedding used to distinguish different data parts;

[0008] Step 3: After the embedding matrix X is input, it is processed by multiple layers of self-attention modules. Each layer of modules relies on learnable weights. Calculate the query matrix, key matrix and value matrix Q = XW through the embedding matrix X Q , K=XW K , V=XW V , where d h is the head dimension, and then the scalar scaling dot product attention mechanism is used to calculate the output of this layer, as shown in the formula As shown. The self-attention process uses the multi-head attention technique to divide the embedding dimension into H attention heads and obtain the processing results separately. The implementation of multi-head attention concatenates the results and maps them back to the original space: MultiHead(Q, K, V) = Concat(head1, ..., head H )W O .in is the linear mapping matrix, and H is the number of heads.

[0009] Step 4: Pass the fully connected layer weights Map features to output target values, where b is the bias, and the final prediction result is obtained.

[0010] For model training, random attention masks at the embedding element level can be used for data augmentation to suppress overfitting and increase model performance. The specific approach is to obtain a unified embedding X∈R b×s×d Generate the corresponding random mask Applied to the autonomous process, some features do not participate in the reasoning process, where b is the size of a batch of data and s is the total length of the embedded sequence of the unified embedding.

[0011] The additional tag embedding t can be a learnable parameter of the network or a fixed absolute position encoding such as sine and cosine encoding. Choosing the appropriate tag embedding to improve model capabilities depends on whether the specific problem requires processing temporal features or spatially related features.

[0012] Furthermore, the unified feature representation in step 1 of the present invention includes a logarithmic feature space Category feature space Mapping with other modal feature spaces to obtain a feature representation of unified dimension Φ s (X s ),Φ c (X c ) etc. Among them, M s and M c Represent the vector dimensions of the numerical feature space and the category feature space, Φ s and Φ c They respectively represent the mapping functions that transform the corresponding feature space into feature representations of uniform dimension.

[0013] Furthermore, the construction expression of the unified embedding matrix in step 2 of the present invention is:

[0014]

[0015] Where M is the total length of all modal feature sequences, and d is the unified embedding dimension.

[0016] Furthermore, the random feature mask in step 3 of the present invention is generated by generating a mask matrix M∈{0,1} M Implementation, get the final input:

[0017] X masked =(X+P)⊙M

[0018] Where P is the position code and ⊙ represents element-by-element multiplication.

[0019] Furthermore, the loss function used in step 4 of the present invention is a weighted combination of mean square error, Huber loss and regularization term:

[0020]

[0021] Among them, α, β and λ are weight coefficients, is the mean square error, is the Huber loss, is the regularization term, and θ represents the trainable parameters of the model.

[0022] Beneficial effects:

[0023] 1. This paper uses a multi-layer self-attention mechanism to efficiently capture global feature relationships. Attention masks can be added at the embedding element level to block some feature data, making it robust to missing data and reducing preprocessing requirements.

[0024] 2. The present invention supports the unified processing of multimodal data features such as numerical and categorical features, and has an adaptive and flexible tag embedding scheme for time series data and spatially correlated data, which enhances the adaptability of multimodal data and reduces the difficulty of feature engineering.

[0025] 3. The network structure and scale of the present invention are flexible and variable, which can not only increase the model scale to adapt to extremely complex and high-precision fitting tasks, but also reduce the model scale to facilitate distributed operation and low-cost deployment. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1a This is a diagram showing how the training set fitting accuracy changes with the number of training steps on the Electrical Fault detection and classification dataset.

[0027] Figure 1b This is a diagram showing how the validation set fitting accuracy changes with the number of training steps on the Electrical Fault detection and classification dataset.

[0028] Figure 1c This is a diagram showing how the training set loss changes with the number of training steps on the Electrical Fault detection and classification dataset.

[0029] Figure 1d This is a diagram showing how the validation set loss changes with the number of training steps on the Electrical Fault detection and classification dataset.

[0030] Figure 2 This is part of the Electrical Fault detection and classification dataset, showing a schematic diagram of numerical features and categorical features.

[0031] Figure 3 This is an image of a dam associated with the Electrical Fault detection and classification dataset, showing a schematic diagram of image features.

[0032] Figure 4 It is the overall process framework diagram of the model processing multimodal data.

[0033] Figure 5 This is a diagram of the working principle of the model's self-attention and multi-layer perceptron modules. DETAILED DESCRIPTION

[0034] This paper presents a multimodal modeling method for large-scale tabular data in power systems. It uses a self-attention mechanism to efficiently model power system data while simultaneously handling multimodal data and missing data. By constructing a unified feature representation framework, it enables joint modeling of heterogeneous features such as numerical values, categories, and images. The method transforms multimodal tabular data into an embedding sequence matrix and uses a multi-layer self-attention mechanism for feature extraction and classification modeling.

[0035] In terms of formal description, consider a multimodal power system dataset

[0036]

[0037] Where N is the amount of data in the dataset, X i represents the data sample, y i Represents the label output corresponding to the sample. Assume that each sample X i Including numerical feature space Category feature space and image feature space Three completely different modal data, target variables The core task of this invention is to learn a mapping function from the multimodal feature space to the real number domain in Represents the tensor product of multimodal features.

[0038] To achieve the above goals, we first define a set of feature transformation strategies to map heterogeneous features into a unified d-dimensional embedding space. Numerical features (such as current, voltage, etc.) are standardized and expressed as Φ s (X s ); the category features are mapped to Φ through the learnable embedding matrix c (X c ); The image features are extracted using the pre-trained ResNet50 network to extract the feature sequence Φ v (X v ). Then, a unified embedding matrix is constructed by concatenating all modal features in the sequence dimension, as shown in Equation 2:

[0039]

[0040] In order to preserve the position information in the sequence, the model can distinguish the position information of different features in the sequence, and introduce a learnable position encoding. Get the enhanced embedded sequence x enhanced =X+t. Based on this representation, the modeling goal is transformed into estimating conditional expectation And optimize by minimizing the mean square error loss function:

[0041]

[0042] The core idea of this method is to capture the complex interactions between multimodal features by constructing a unified embedding space and introducing a multi-layer self-attention mechanism. At the same time, targeted preprocessing and optimization methods are designed based on the characteristics of power system data, demonstrating high precision and strong robustness in power fault detection and classification tasks.

[0043] The reasoning process of the present invention specifically includes the following:

[0044] The core of the inference process lies in extracting a unified embedding representation from the raw power system data and capturing the deep relationships between modalities through a multi-layer self-attention mechanism. The main steps include feature processing and unified embedding generation, as well as feature modeling based on the attention mechanism.

[0045] First, the original features are preprocessed to achieve consistent representation across modalities. Quantile Transformation is used to map it to a space that obeys the standard normal distribution, expressed as Φ Q (X s ), as shown in Formula 4:

[0046]

[0047] where Φ -1 is the inverse cumulative distribution function of the standard normal distribution, rank(x i ) is the ranked index of the eigenvalue. The quantile transformation calculates the ranked percentile of each eigenvalue and maps it using the inverse standard normal cumulative distribution function. This transformation effectively eliminates the influence of extreme values while preserving the relative relationships between eigenvalue distributions, enhancing the model's robustness to the input data distribution.

[0048] For category features (such as device type, operating status, etc.), by introducing a learnable embedding matrix Achieve feature transformation. The embedding matrix not only performs well in reducing the representation dimension, but also can capture the semantic similarity between categories through learning, thereby improving the model's expressive power and fault diagnosis performance. The generation of category embedding is achieved through matrix lookup, that is, mapping the category index to the embedding vector space. The transformation of category features can be expressed as in Equation 5. Represents the embedding vector of category j.

[0049]

[0050] For image features A pre-trained deep convolutional network (such as ResNet50) is used to extract its spatial features, as shown in Equation 6:

[0051]

[0052] N v is the number of tokens in the image after segmentation. This method preserves the spatial structure of the image while achieving a unified representation with features from other modalities. The advantage of a pretrained network is that it effectively captures both global and local information about the image. Embedding its output as input not only reduces reliance on feature engineering but also ensures robust representation. The high-dimensional features of the image are further segmented and processed to generate a serialized embedded representation that can be unified with other modalities.

[0053] The embedding representations of all modalities are concatenated in the sequence dimension to form a unified embedding matrix as shown in Equation 7:

[0054]

[0055] In order to further enhance the model's ability to perceive sequence position information, learnable position encoding is introduced Fusion of position information into X by addition allows the model to take into account their absolute position in the sequence while capturing contextual relationships between feature values.

[0056] In the multi-layer self-attention mechanism processing stage, the Grouped Query Attention (GQA) mechanism is introduced to model the complex interaction relationship between features in the enhanced feature sequence X. First, the query, key, and value matrices are calculated:

[0057] Q=XW Q , K=XW K , V=XW V #1) Equation 8

[0058] in is a learnable weight matrix. The attention score is calculated by scaling the dot product attention:

[0059]

[0060] The self-attention mechanism dynamically adjusts the weight of each feature by calculating the correlation between them, capturing the nonlinear interactions present in multimodal data. Experiments have shown that this mechanism performs well in modeling the association between device status and signal patterns in power system fault diagnosis tasks. The introduction of the multi-head attention mechanism further enhances the modeling capability by computing different subspace features in parallel:

[0061] MultiHead(Q,K,V)=Concat(head1,...,head H )W o #1) Formula 10

[0062] Note that head i represents the output of the i-th attention head, is the output weight matrix. To ensure modeling stability, residual connections and normalization layers (RMSNorm) are used, as shown in Equation 11. Where γ is a learnable scaling parameter.

[0063]

[0064] By stacking multiple layers of self-attention modules, high-order feature relationships in the sequence are gradually extracted. The nonlinear expression capability is enhanced by the residual feedforward network, and the final feature representation is obtained:

[0065]

[0066] The last token in the feature sequence is selected to summarize the global information as the main basis for prediction. Finally, the last token of the sequence is taken and the fault type classification task is completed through the fully connected layer:

[0067]

[0068] Considering that multimodal data may be partially missing in real scenarios, this paper designs a random masking mechanism to improve the robustness of the model in dealing with missing value scenarios. During the training process, some input features are randomly masked to simulate the impact of missing values, thereby improving the robustness of the model in the inference phase. This masking mechanism acts on the embedding sequence by generating a binary mask matrix, thereby ensuring that the masking operation has controllable and consistent processing of the input features. By generating a mask matrix Simulate feature loss. This is how we obtain the embedding sequence that is input to the model. This process can be expressed as Equation 14, where the ⊙ symbol represents an element-wise multiplication operation.

[0069] X masked =(X unifid +P)⊙M#1) Formula 14

[0070] The training process of the simulation experiment includes the following:

[0071] To ensure model stability during training and generalization during inference, this paper employs multiple optimization strategies in terms of loss function design, optimization algorithm selection, and regularization strategies. The core objective function is based on the mean squared error (MSE), which optimizes the model's primary task by minimizing the squared difference between the predicted value and the true value. Its mathematical expression is shown in Equation 15:

[0072]

[0073] This loss directly optimizes the model's prediction accuracy for the target variable. On this basis, in order to enhance the model's robustness in scenarios with outliers and class imbalance, a balanced loss function is adopted to adjust the weights of samples from different categories. At the same time, given that certain fault types in the power system may account for a relatively small proportion, Focal Loss is introduced as an auxiliary objective to improve the model's ability to detect rare fault types. Its definition is:

[0074]

[0075] Where δ is a threshold hyperparameter that controls the sensitivity of outlier processing. Huber loss can degenerate into MSE when the error is small, and adopt a linear growth strategy when the error is large, thereby effectively suppressing the impact of outliers.

[0076] In order to further improve the generalization ability of the model and suppress overfitting problems, L2 regularization constraints are imposed on all network weights during training to strike a balance between constraint strength and model flexibility. Its definition is shown in Equation 17, where ω l Represents the weight parameter of the lth layer. By limiting the parameter size, it helps reduce the dependence of overly complex models on training data. L2 regularization can effectively improve the generalization ability of the model.

[0077]

[0078] The final loss function is a weighted combination of the above three parts, defined as Equation 18. α, β, and λ are weight coefficients, and their specific values are determined through cross-validation.

[0079]

[0080] In the selection of optimization algorithm, the present invention adopts the adaptive gradient descent algorithm (Adam) for parameter update, and its iterative process can be expressed as:

[0081]

[0082] in and are the first-order and second-order moment estimates, η is the learning rate, and ∈ is the numerical stability correction term.

[0083] To avoid the gradient explosion problem during training, a gradient clipping strategy is introduced. The gradient range is limited to [-κ, κ] during each parameter update:

[0084]

[0085] Where κ is the clipping threshold.

[0086] To verify the performance of the method of the present invention, training is performed on the dataset Electrical Fault detection and classification and the training effect is checked.

[0087] Before training a model, the dataset needs to be divided into a training set and a validation set. Only the training set will participate in the actual training process, while the validation set will only be used to obtain model loss and accuracy data to check for overfitting. An overfitted model only achieves extremely high accuracy on the training set and is almost incapable of making accurate predictions on data not used for training. If the model loss on the validation set increases rather than decreases as training progresses, gradually becoming significantly larger than the loss on the training set, this indicates that the model has lost its generalization ability and is overfitting. If the model loss on both the training and validation sets steadily decreases as training progresses, this indicates that the model is not overfitting. Lower loss values indicate a better fit to the data. This value should decrease as training progresses, eventually stabilizing at a certain level. The accuracy value indicates the accuracy of the model's predictions, with a maximum value of 100%. Higher values indicate greater accuracy in the model's predictions for real-world problems. Its value generally increases as loss decreases.

[0088] like Figure 1c and Figure 1d As shown in , the model loss on both the training set and the test set is steadily decreasing, indicating that the model fits the data correctly. Figure 1a and Figure 1b As shown in the figure, acc steadily increases and stabilizes at a very high value, indicating that the trained model has high accuracy in practical problems.

[0089] The simulation experiments of the present invention are carried out based on the following hyperparameters, specifically including:

[0090] Table 1 Data processing hyperparameters

[0091] Configuration Name Configuration Values batch_size 256 test_size 0.2 drop_last TRUE

[0092] Table 2 Model definition hyperparameters

[0093] Configuration Name Configuration Values hidden_size 128 head_dim 32 num_heads 4 num_kv_heads 4 attn_dropout 0 mlp_dropout 0 attn_bias TRUE mlp_bias TRUE mlp_hidden_size_factor 4 use_rmsnorm FALSE num_hidden_layers 4 dropout_rate 0.1

[0094] Table 3 Learning rate hyperparameters

[0095]

[0096]

[0097] The above are only preferred embodiments of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An adaptive fitting and prediction method for multimodal industrial big data prediction, characterized in that: The following steps are involved: Step 1: Preprocess and uniformly encode the multimodal raw data, map the features of different modalities into a unified d-dimensional embedding space, and obtain a unified feature representation; Step 2: Concatenate the unified feature representations in step 1 above in the sequence dimension to construct a unified embedding matrix, and introduce learnable positional encoding to enhance sequence information; Step 3: Apply a random feature mask to the embedding matrix in step 2 above to simulate the feature missing scenario, improve the model's robustness to missing data, and obtain the final model input; Step 4: Model the input data through a multi-layer attention network and obtain the final prediction result through a fully connected layer.

2. The adaptive fitting and prediction method for multimodal industrial big data prediction according to claim 1 is characterized in that: The unified feature representation in step 1 includes a logarithmic feature space Category feature space Mapping with other modal feature spaces to obtain a feature representation of unified dimension Φ s (X s ),Φ c (X c ), where M s and M c Represent the vector dimensions of the numerical feature space and the category feature space, Φ s and Φ c They respectively represent the mapping functions that transform the corresponding feature space into feature representations of uniform dimension.

3. The adaptive fitting and prediction method for multimodal industrial big data prediction according to claim 1, characterized in that: The construction expression of the unified embedding matrix in step 2 is: Where M is the total length of all modal feature sequences, and d is the unified embedding dimension.

4. The adaptive fitting and prediction method for multimodal industrial big data prediction according to claim 1, characterized in that: The feature mask used in step 3 is generated by generating a mask matrix M∈{0, 1} M Implementation, get the final input: X masked =(X+P)☉M Where P is the position code and ⊙ represents element-by-element multiplication.

5. The adaptive fitting and prediction method for multimodal industrial big data prediction according to claim 1, characterized in that: The loss function used in step 4 is a weighted combination of mean square error, Huber loss and regularization term: Among them, α, β and λ are weight coefficients, is the mean square error, is the Huber loss, is the regularization term, and θ represents the trainable parameters of the model.

Images