Industrial production table data classification prediction method based on capsule network

Abstract

The invention relates to an industrial production table data classification prediction method based on a capsule network, and the method comprises the steps: obtaining to-be-detected industrial production table data, carrying out the feature extraction after preprocessing, inputting the extracted features into a trained capsule network, and obtaining a prediction result. The training process of the capsule network comprises the following steps: acquiring industrial production table data with a specific label, performing feature embedding, and performing feature extraction of different granularities by using a plurality of feature extractors; combining the embedded features with advanced features extracted by a plurality of feature extractors, and coding the combined features into a primary capsule; transmitting the output information of the primary capsule to a high-level capsule through a self-attention static routing mechanism, performing classification prediction, and calculating marginal loss; inputting the classification prediction result into a feature reconstruction network for reconstruction, and calculating reconstruction loss; and iteratively training the capsule network according to the marginal loss and the reconstruction loss. The accuracy of the prediction result can be obviously improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of big data analysis, and in particular relates to a capsule network-based industrial production table data classification prediction method. Background Art

[0002] In recent years, with the significant improvement of computing power and the rapid development of artificial intelligence technology, the computer field has achieved rapid development, especially in the core field of processing and parsing industrial production tabular structured data. Due to its structured characteristics and key role in the decision-making process, industrial production tabular data has been widely infiltrated into various sub-fields, such as process control in manufacturing, posture detection of robots, gas prediction in coal mines, etc.

[0003] The prediction function of industrial production tabular data is particularly favored by the market, and it has become the core of many industrial applications. Enterprises and researchers are looking for tabular data prediction solutions that can more effectively solve practical problems, because these solutions can provide decision makers with accurate and reliable data support to promote business growth and risk control.

[0004] In the research progress of tabular data prediction tasks, the methods can be roughly divided into three categories: traditional machine learning methods, hybrid learning methods, and Transformer-based methods. Among them, methods based on traditional machine learning, such as XGBoost and LightGBM, are widely praised for their excellent efficiency and prediction accuracy. These methods have been widely used in the industry for their efficient implementation and excellent interpretability.

[0005] However, with the development of deep learning technology, hybrid learning methods that combine decision trees and neural networks have begun to emerge. This type of method cleverly combines the non-differentiable model in traditional machine learning methods with deep neural networks, not only taking advantage of the advantages of decision trees in processing rules and logic, but also taking advantage of the powerful feature extraction capabilities of neural networks to capture more complex nonlinear relationships between data.

[0006] The method based on the Transformer architecture is an innovation in the field of tabular data processing. This architecture allows the model to capture long-distance dependencies in tabular data through a deep self-attention mechanism, providing a deep understanding of complex data patterns. In particular, Transformer has shown great potential in the processing of heterogeneous data.

[0007] Geoffrey Hinton and his team proposed a new neural network based on a structure called capsule, and applied it to the field of image processing, proving that it is effective in understanding spatial relationships in high-level data. The capsule network captures multiple features of an image through a series of primary capsules, which represent different entities or parts of objects in the image. Each capsule outputs a vector whose direction and length represent the various attributes and probability of existence of a specific entity. The traditional capsule network uses a dynamic routing mechanism to transfer information between capsules in different layers, and the dynamic routing mechanism requires iterative calculation of the weights of different capsules based on the data each time. This process is computationally intensive and very time-consuming.

[0008] Nevertheless, these advanced methods also have challenges, especially in reducing invalid feature interactions. When directly processing high-dimensional complex tabular data, comprehensive feature interactions may lead to incorrect feature representations, which not only increases the uncertainty of the model, but also greatly increases the computational cost. Therefore, researchers have been exploring how to balance feature interactions at an appropriate scale to improve the accuracy and efficiency of predictions. An effective feature interaction strategy should be able to distinguish which feature interactions are beneficial and which may be noise, thereby enhancing the generalization ability of the model while maintaining the computational efficiency of the model. Summary of the invention

[0009] In view of the pain point that the current neural network-based model cannot solve the feature interaction problem of industrial production tabular data well, the present invention proposes a classification and prediction method for industrial production tabular data based on capsule networks. The method includes: preprocessing the acquired tabular data, then performing corresponding feature extraction on it, inputting the processed data and features into the trained capsule network, generating the basic representation of the primary capsule through some trainable multivariate Gaussian kernels, forwarding the output of the primary capsule to the advanced capsule through an attention-based routing algorithm, achieving excellent classification results, and also designing a reconstruction network for reconstructing the industrial production tabular data to reduce the overfitting of the model.

[0010] The present invention proposes a classification prediction method for industrial production table data based on a capsule network, comprising: obtaining the industrial production table data to be detected, extracting features after preprocessing, inputting the extracted features into a trained capsule network, and obtaining a prediction result;

[0011] The training process of the capsule network includes:

[0012] S1: Obtain industrial production table data with specific labels and use it as training sample data after preprocessing;

[0013] S2: embedding the training sample data to obtain embedded features; using J parallel feature extractors to extract features of different granularities from the training sample data;

[0014] S3: Merge the embedded features and the high-level features extracted by the J parallel feature extractors, encode the merged features into primary capsules, and obtain primary capsule features;

[0015] S4: Pass the primary capsule features to the advanced capsule through the self-attention static routing mechanism for classification prediction, obtain the classification prediction results of different attribute category features in the industrial production table data, and calculate the marginal loss L;

[0016] S5: Input the classification prediction result into the feature reconstruction network, perform feature reconstruction, and calculate the reconstruction loss MSE between the reconstructed features and the embedded features;

[0017] S6: Iteratively train the capsule network according to the marginal loss L and the reconstruction loss MSE to obtain the trained capsule network.

[0018] Beneficial effects of the present invention:

[0019] 1. The present invention applies capsule networks to classification prediction of industrial tabular data. During the network training stage, multiple feature extractors are used to capture high-level features of industrial tabular data of different granularities. Subsequently, multiple trainable multivariate Gaussian kernels are used to capture different data patterns or different data forms of the tabular data. This effectively avoids the full-link interaction of features and improves the accuracy of prediction.

[0020] 2. In view of the shortcomings of the existing capsule network dynamic routing method, the present invention proposes a self-attention static routing mechanism, which can autonomously capture the relationship between primary capsules and advanced capsules, effectively reducing the calculation time and improving the prediction efficiency.

[0021] 3. The present invention also designs a reconstruction network for tabular data, which is used to reconstruct tabular features and calculate reconstruction loss. The capsule network is iteratively trained according to the reconstruction loss and marginal loss, and prediction is performed using the trained capsule network, which can significantly improve the accuracy of the prediction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a schematic diagram of the overall process of an embodiment of the present invention;

[0023] Figure 2 Schematic diagram of the network structure of the feature extractor in an embodiment of the present invention;

[0024] Figure 3 Schematic diagram of a self-attention static routing mechanism in an embodiment of the present invention;

[0025] Figure 4 It is a schematic diagram of the structure of the reconstructed network in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances. This is just a way of distinguishing objects with the same attributes when describing the embodiments of this application.

[0028] The embodiment of the present invention proposes a classification prediction method for industrial production table data based on capsule network, referring to Figure 1 As shown, the method includes:

[0029] Obtain the industrial production table data to be tested, extract features after preprocessing, input the extracted features into the trained capsule network to obtain the prediction results;

[0030] The training process of the capsule network includes:

[0031] S1: Obtain industrial production table data with specific labels and use them as training sample data after preprocessing.

[0032] S2: embed the training sample data to obtain embedded features; use J parallel feature extractors to extract features of different granularities from the training sample data.

[0033] S3: Merge the embedded features and the high-level features extracted by J parallel feature extractors, encode the merged features into primary capsules, and obtain primary capsule features.

[0034] S4: Pass the primary capsule features to the advanced capsule through the self-attention static routing mechanism for classification prediction, obtain the classification prediction results of different attribute category features in the industrial production table data, and calculate the marginal loss L.

[0035] S5: Input the classification prediction result into the feature reconstruction network, perform feature reconstruction, and calculate the reconstruction loss MSE between the reconstructed features and the embedded features.

[0036] S6: Iteratively train the capsule network according to the marginal loss L and the reconstruction loss MSE to obtain the trained capsule network.

[0037] Figure 1 Schematic diagram of the overall process of an embodiment of the present invention. Figure 1 In the embodiment of the present invention, the overall process includes two process stages: the training stage and the prediction stage. In the training stage, industrial production table data with specific labels are obtained, and the embedded features are obtained through feature embedding processing of the embedding layer Embedding, and then multiple parallel feature extractors are used to extract features of different granularities on the training sample data to obtain high-level features of different granularities; the embedded features and high-level features of different granularities are merged, and the merged features are encoded into the primary capsules in the capsule network; the output information in the primary capsule is passed to the high-level capsule through the self-attention static routing mechanism, and classification prediction is performed to obtain the classification prediction results of different attribute category features in the industrial production table data, and the marginal loss L is calculated; the classification prediction results are input into the feature reconstruction network, feature reconstruction is performed, and the reconstruction loss MSE between the reconstructed features and the embedded features is calculated; the capsule network is iteratively trained according to the marginal loss L and the reconstruction loss MSE to obtain the trained capsule network. In the prediction stage, the industrial production table data to be detected is obtained, and feature extraction is performed after preprocessing, and the extracted features are input into the trained capsule network to obtain the prediction results.

[0038] In step S1, open source industrial production table data with specific labels is obtained and used as training samples after preprocessing.

[0039] Specifically: The embodiment of the present invention selects a classification dataset in the industrial field based on the openml--CC18 standard. The benchmark dataset meets the conditions of sufficient sample size, few missing values, and balanced positive and negative samples. Taking the APS fault dataset in IDA industry as an example, the data in this dataset is about the equipment status of a large industrial machine. It is required to predict whether the value of each point is normal, which is a binary classification task. The dataset contains 170 digital features, some of which have known specific meanings, and some of which have unknown specific meanings. Please refer to Table 1 for details.

[0040] Table 1 APS fault data set

[0041]

[0042] Since the data set is mainly continuous values, a unified approach is adopted to standardize the data using z-score so that the processed data conforms to the standard normal distribution, that is, the mean is 0 and the standard deviation is 1.

[0043] The training sample data set is divided into a ratio of 7:3, with 70% as the training set and 30% as the test set. In deep learning, the neural network samples a batch of data each time, which is equivalent to a random sampling of the full data. These samples should have similar distributions in the feature space.

[0044] Specifically, in step S2, the training sample data is input into the embedding layer Embedding, feature embedding is performed, and the embedding features of the training sample data are obtained.

[0045] In a preferred embodiment, reference Figure 3 As shown, in step S3, the feature extractor includes a feature selection module and a feature extraction module; the mutual information I of the training sample data is measured by the feature selection module, and the preliminary features f' of the training sample data are obtained through sparse mask matrix processing according to the mutual information; high-level features are extracted from the preliminary features f' by the feature extraction module; the training sample data are processed respectively by the feature selection modules and feature extraction modules in J parallel feature extractors to obtain high-level features of different attribute categories in the training sample data at different granularities.

[0046] Specific:

[0047] Figure 2 Schematic diagram of the network structure of the feature extractor. Figure 2 In , the feature extractor includes mutual information measurement, generating sparse mask matrix and two linear layers, linner1 and linner2 represent linear layers, and ⊙ represents Hagem product.

[0048] Reference Figure 2 As shown, the feature extraction network aims to explore potential feature combinations and extract high-level features while minimizing the number of parameters.

[0049] For the feature selection module, assume that the input training sample data contains m scalar features, R m is a real vector, through a learnable sparse mask W∈R m To select a set of scalar features, the learnable mask W is obtained by a sparse mapping M mask formed.

[0050] In some embodiments, feature selection is performed on the training sample data to obtain preliminary features, and the specific process includes:

[0051] S301: Calculate the mutual information between each random variable X and the target value Y to measure the degree of dependence between the feature and the target value. The calculation formula is as follows:

[0052]

[0053] In the formula, I(X;Y) represents the mutual information between random variable X and target value Y. Random variable X comes from training sample data, and target value Y is the label of training sample data. Mutual information measures the amount of information shared between random variable X and target value Y. p(x,y) represents the joint probability distribution function of random variable X and target value Y, p(x) represents the marginal probability distribution of random variable x, and p(x,y) represents the marginal probability distribution of target value y.

[0054] S302: Sort the importance of features according to mutual information, select the top C features, set these C features to 1 in the sparse mask, and set other features to 0 in the sparse mask; generate a sparse mask matrix for the mutual information matrix generated above through the sparsemax activation function, and the sparsemax activation function is expressed as:

[0055]

[0056] Where sparsemax(t) means that different processing is performed on the input feature t in different ranges.

[0057] S303: Extract the feature vector f of the training sample data, and multiply it element-by-element with the learned sparse mask W to obtain the preliminary feature f'.

[0058] f′=W⊙f (3)

[0059] W∈sparsemax(M mask )

[0060] Where W represents the sparse mask matrix, ⊙ represents element-by-element multiplication, f represents the feature vector of the training sample data, and M mask Represents a sparse mapping matrix.

[0061] The learned sparse mask W is multiplied element by element with the input vector f to obtain the output vector f′. The final preliminary feature is It is shown that by calculating sparse masks and applying multiplication operations, effective features are preliminarily selected, which is simple and can select the same features for different instances.

[0062] For the feature extraction module, it is necessary to extract corresponding high-level features from the primary features, such as extracting the mechanical physical index from the current sensor, temperature sensor, and vibration sensor to reflect the overall health status of the sensor.

[0063] In a preferred embodiment, a feature extraction module is used to extract high-level features from the preliminary features f', and the specific process includes:

[0064] S311: Input the preliminary feature f' into the feature extraction module, multiply the preliminary feature f' by the parameter matrix W1 to obtain the attention vector q, and map the attention vector q to the interval (0,1) through a Logistic function. Specifically:

[0065] Reference Figure 3 As shown, the preliminary feature f' statistics pass through the first linear layer linner1, and then multiply the generated sparse mask matrix element by element. The obtained attention vector q will be multiplied again with the initial feature after linner2 to complete a high-level feature extraction. The specific details are as follows:

[0066] In the feature extraction module, by accepting the output of the feature selection module, that is, the preliminary feature f′∈R m , initialize a learnable weight matrix Represents a weight parameter matrix composed of d m-dimensional vectors. Multiply the preliminary feature f' by the weight matrix W1 to obtain an attention vector q. Then, a Logistic function is used to map the attention vector q to the interval (0,1), making the function smoother and easier to derive. The specific formula is as follows:

[0067] q=Logistic(W1f′) (4)

[0068] Where q represents the attention vector, W1 represents the first weight matrix, and f' represents the preliminary features after mutual information measurement.

[0069] S312: Multiply the attention vector q with the embedded vector W2f, and process it through the Relu activation function to obtain the high-level feature f * , which is specifically expressed as:

[0070] f * =ReLU(q⊙(W2f)) (5)

[0071] In the formula, f * represents the output feature vector after being processed by the Relu activation function, q represents the attention vector, ⊙ represents element-by-element multiplication, W2 represents the second weight matrix, and f represents the initial feature.

[0072] In this way, the module maps the preliminary features f′ to the final high-level features f * And the d scalar features in the output feature vector are regarded as independent features, completing a high-level feature extraction. Figure 2As shown, each feature extractor performs multiple feature extractions on high-level features by stacking multiple feature extraction modules to obtain high-level features of different granularities. The embodiment of the present invention uses J (J≥3) parallel feature extractors (the weight matrix W in each feature extractor is different) to perform feature extraction of different granularities on the training sample data. In this way, as many feature combinations as possible can be covered, so that the valid information between each feature extraction module may be further grouped by other feature extraction modules, and the invalid information is discarded; finally, the training sample data is extracted by J (J≥3) parallel feature extractors to obtain high-level features of different granularities under different attribute categories in the training sample data.

[0073] In the field of deep learning, capsule networks (CapsNets) have been widely studied, especially when processing image data. Capsule networks capture multiple features of an image through a series of primary capsules, which represent different entities or parts of an object in the image. Each capsule outputs a vector whose direction and length represent various attributes and the probability of existence of a specific entity. However, when dealing with structured data, especially tabular data, the situation is different. Tabular data (such as database records or rows in Excel tables) usually contains multiple features, which may represent different attributes of an entity. For example: in industrial data records, when describing the status of a machine, there may be multiple features such as current, speed, weight, etc. Therefore, primary capsules are used to represent different profiles of a record in tabular data, rather than different parts of an object.

[0074] Step S4 specifically includes:

[0075] S401: concatenate the embedded features obtained after processing in S2 with the high-level features of different granularities obtained after processing in S3 to obtain the combined feature x. A feature vector x of dimension m.

[0076] S402: Encode the merged feature x using multiple multivariate Gaussian kernels trained with the network to obtain primary capsule features. The primary capsule features are specifically expressed as:

[0077]

[0078] Among them, u i represents the primary capsule feature, k i (x,μ i ,Σ i ) represents the probability density function of the i-th Gaussian distribution, μ i represents the location parameter vector, and μ i Represents the center point position vector of the Gaussian kernel in multidimensional space; ∑i represents the scale parameter, and ∑ i represents the distribution width of the feature, x represents the feature after the embedded feature is merged with the high-level features extracted by J parallel feature extractors, i represents the i-th Gaussian distribution, (·) T represents transpose, represents the normalization constant in the normal distribution, and m represents the dimension of the eigenvector x. In the multivariate Gaussian distribution, these parameters define the shape and location of the probability density function.

[0079] By continuously learning the position parameter μ i and scale parameter ∑ i , a better primary capsule network is trained, whose output is the primary capsule feature, so as to find the most appropriate representation (i.e., fine-grained feature vector representation) for each input training sample data set.

[0080] In some preferred embodiments, the low-level capsule features are passed to the high-level capsules through a self-attention static routing mechanism.

[0081] The embodiment of the present invention uses the self-attention static routing mechanism to transfer the output of the primary capsule to the advanced capsule, thereby improving the routing effect from the primary capsule to the advanced capsule. Figure 3 shown.

[0082] like Figure 3 As shown, the total input of the nth high-level capsule in the l+1th layer of the capsule network is the weighted sum of the prediction vectors of all primary capsules from layer l. The prediction vector of the primary capsule is the output of formula (6), i.e., the primary capsule feature. The prediction vector of each primary capsule is This is done by The capsule is generated by matrix multiplication of the capsule and the weight matrix. represents the total input of the nth high-level capsule in the l+1th layer, which is obtained by the weighted sum of all the "prediction vectors" from the lower-layer capsules (i.e., all the primary capsules in the lth layer of the capsule network) represents the output result of the nth primary capsule in the lth layer of the capsule network, Represents the set of all primary capsule output result vectors of layer l in the capsule network with dimension d. All weight matrices of layer l in the capsule network is embedded into all affine transformations between primary capsules of two adjacent layers. Figure 3 middle, represents the logarithmic prior matrix containing all weights of layer l in the capsule network, which are learned simultaneously with other weights; represents the matrix containing all coupling coefficients generated by the self-attention algorithm.

[0083] Step S5 is specifically as follows:

[0084] S501: Calculate the output prediction tensor of all primary capsules in the lth layer of the capsule network, which is specifically expressed as:

[0085]

[0086] In the formula, Represents the output prediction tensor of all primary capsules in the lth layer of the capsule network, d represents the dimension of the capsule network, n l represents the nth primary capsule in the lth layer of the capsule network, n l+1 represents the nth primary capsule in the l+1th layer of the capsule network, represents the transposed feature of the nth primary capsule from the lth layer in the capsule network, (·) T represents transpose, Represents the weight matrix of the lth layer in the capsule network.

[0087] S502: Output prediction tensor of all primary capsules in layer l of the capsule network As the input of the high-level capsule in the l+1th layer, the total input of the nth high-level capsule in the l+1th layer in the capsule network is obtained through the self-attention static routing mechanism The specific expression is:

[0088]

[0089] In the formula, represents the total input of the nth high-level capsule in the l+1th layer of the capsule network, Represents the transpose of the output prediction tensor of all primary capsules in the lth layer of the capsule network. These vectors predict the nth l+1 The properties of each capsule; represents the logarithmic prior matrix containing all weights of the lth layer in the capsule network; represents the matrix containing all coupling coefficients generated by the self-attention algorithm in the lth layer of the capsule network.

[0090] In some preferred embodiments, the coupling coefficient matrix is ​​calculated based on the self-attention tensor.

[0091] Specifically, the coupling coefficient matrix is ​​calculated by the self-attention tensor Calculated, the self-attention tensor formula is as follows:

[0092]

[0093] In the formula, represents the attention weight matrix of all capsules in layer l to the position of the nth capsule in layer l+1 in the capsule network, Represents the relationship matrix between all capsules in the l+1th layer of the capsule network, Representing the relationship matrix The transpose of represents the balance coefficient, d represents the dimension of the feature vector of the lth layer in the capsule prediction network, l represents the number of layers, and n represents the nth capsule (including primary capsules or advanced capsules). It helps stabilize training and maintain a balance between coupling coefficients and logarithmic priors.

[0094] The calculation formula of coupling coefficient is:

[0095]

[0096] In the formula, Represents the coupling coefficient matrix from all capsules in the lth layer to the nth capsule in the l+1th layer in the capsule network, represents the self-attention matrix from the nth capsule in the lth layer to the nth capsule in the l+1th layer in the capsule prediction network, It represents the influence of all capsules in layer l on the nth capsule in layer l+1 in the capsule prediction network. is a three-dimensional attention tensor, is a two-dimensional attention tensor; represents the sum of the elements of the self-attention matrix from the nth capsule in the lth layer to the nth capsule in the l+1th layer in the capsule network, It represents the cumulative score of all capsules in the lth layer on the nth capsule in the l+1th layer in the capsule prediction network.

[0097] Therefore, the coupling coefficients between a capsule in layer l of the capsule network and all capsules in the previous layer, i.e., layer l+1, add up to 1. In the case of multiple capsule layers stacked together, the capsule network remains unchanged in order to create a deeper hierarchy.

[0098] For each class to which a sample belongs, multiple advanced capsules are used to make predictions.

[0099] For CapsNet in the image field, the vector length of the high-level capsule feature is used to represent the probability of existence of a semantic. For the tabular data of the present invention, the vector length of the high-level capsule is also used to represent the probability, which represents the probability of a certain target category in a binary or multi-classification data set.

[0100] The present invention combines the above ideas. In the final link of high-level capsule prediction of categories, inspired by the ensemble learning method, for each category, the same number of high-level capsules are used to predict the classification (that is, multiple high-level capsules are used to predict the same category). Assume that the number of high-level capsules used for prediction of a certain category is z, and the total number of categories is K. For the final prediction result, the specific formula is:

[0101]

[0102] v k =||y j ||2, 1≤k≤K (12)

[0103] in, Represents the classification prediction result of the capsule network, y j represents the feature vector output by the jth high-level capsule in the capsule network, v k represents the L2 norm of the output feature vector of the jth advanced capsule under the kth attribute category, ||·||2 represents the L2 norm, K represents the total number of different attribute categories, z represents the total number of advanced capsules, It represents the sum of the L2 norms of z high-level capsules under the k-th attribute category.

[0104] Specifically, for multiple advanced capsules under each category, the sum of the vector lengths of the advanced capsules is used to represent the current possible probability, and then the maximum value is taken as the final result. At the same time, in the specific implementation, 20% of the advanced capsules in each group will be dropped to prevent overfitting.

[0105] Calculate the marginal loss function L, which represents the sum of the losses from the first category to the last category. The marginal loss function L follows the setting of the original capsule network (that is, the marginal loss function L is Max-Margin Loss), and its calculation formula is as follows:

[0106]

[0107] Among them, L represents the marginal loss function of the capsule network, which is the sum of the losses of all categories; K represents the total number of different attribute categories, I k represents the average length of the output feature vector of the high-level capsule responsible for predicting the k-th category; T k Represents the label of the kth category. For a binary classification problem, if the kth category is the target category, then T k =1, otherwise T k =0; t + is a constant threshold, indicating the threshold of the positive class; when the target class exists, if I k <t+ , there will be losses; t - is a constant threshold, indicating the threshold of the negative class; when the target class does not exist, if I k >t - , will generate loss, λ represents a scaling factor, which is usually used to adjust the loss of non-target categories.

[0108] For each category k, the loss function consists of two parts. k = 1, that is, category k is the target category, and the loss is max(0,t + -I k ) 2 , which means that when an instance does belong to category k, we hope that I k (The average length of the high-level capsule feature vector of category k) must be at least greater than a threshold t + Usually this threshold is set to 0.9. k Less than t + , a positive loss value will be generated; if it is greater than or equal to t + , no loss will occur.

[0109] When T k = 0, that is, category k is not the target category, and the loss is λmax(0,I k -t - ) 2 , where λ is a regularization parameter used to weigh the loss of non-target categories. Similarly, we hope that I k is less than a small threshold t - , this value is usually set to 0.1. k Greater than t - , a loss will occur; if it is less than or equal to t - , no loss will occur.

[0110] This design enables the loss function to encourage the capsule output length of the target class to be greater than t. + At the same time, it also penalizes capsules of non-target categories whose output length is greater than t - The overall goal is to make the network better at identifying the correct category and suppressing other categories.

[0111] In some preferred embodiments, the classification prediction result of the capsule prediction network is selected The output vectors of the advanced capsules that predict the correct categories are merged, and the merged vectors are input into the reconstruction network. After feature selection and feature updating, the reconstructed industrial production table data is output.

[0112] In S5, the output information of the high-level capsule is input into the feature reconstruction network for feature reconstruction and the reconstruction loss is calculated. Specifically:

[0113] Select the classification prediction results of the capsule prediction network The output vectors of the advanced capsules that predict the correct category are merged to obtain the merged vector V, and the formula is as follows:

[0114] V=concat(h1,h2,h3......h p ) (13)

[0116] Where V is the merged vector, h p It represents the feature vector of a certain advanced capsule under a certain category, and p represents the number of advanced capsules under a certain category.

[0117] By concatenating the feature vectors of the high-level capsules output above, the full semantics of each high-level capsule is obtained, and the generated vector is used to reconstruct the original features of the table through the reconstruction network. This step can add regularization to the network, and can also use the reconstruction network to fill in missing values ​​when there are missing values ​​in the original data.

[0118] In the original capsule network for images, a reconstruction network is usually connected to the back of the network to reconstruct the image. As the network training process is continuously optimized, the original image can be restored, and the output result can also be used for the image content completion task. Inspired by this, the embodiment of the present invention is specially designed for industrial production tabular data. A reconstruction network is designed to enhance network regularization by minimizing the difference between the original tabular data and the reconstructed tabular data to reduce overfitting. At the same time, when the original data is missing, the missing value can be interpolated through the reconstructed data.

[0119] Figure 4 It is a schematic diagram of the structure of the reconstructed network in an embodiment of the present invention.

[0120] like Figure 4 As shown, the concatenated vector V l It will pass through a linear layer LN, and then the vector will pass through a fully connected layer and an attention module AE respectively. The vector matrix obtained by the two will be multiplied and added to the original vector matrix to obtain the output vector matrix of the feature selection module. Then this matrix will pass through a layer normalization Norm, and the resulting vector matrix will pass through the linear layer Linear1 and the linear layer Linner2 respectively. The vector matrix passing through the linear layer Linner1 will be activated by the tanh activation function, and the final results will be multiplied as the output result of a feature update.

[0121] Specifically, the present invention designs a special conversion layer to reconstruct the table data. The reconstruction network completes the reconstruction of the table data by stacking two layers of conversion layers. The conversion layer is a neural network layer with conversion function, such as a fully connected layer. The conversion layer includes two parts, namely, feature selection and feature update modules, which are as follows:

[0122]

[0123] Among them, Z l represents the final output matrix of the feature selection module, that is, the result after transformation, W Q , W K , W V and W v Represents different weight matrices, V l represents the input vector matrix, The value used to balance the dot product is generated by the Query, Key, and Value matrices. The dot product of the Query and Key is calculated and scaled to obtain the attention score matrix. The softmax function is applied to this matrix to normalize it into a weight matrix. Then, this weight matrix is ​​multiplied with the Value matrix to obtain a weighted feature matrix. Finally, this weighted feature matrix is ​​combined with another learnable weight matrix W. v Multiply the transformed input feature matrix to get the final output feature matrix Z l .

[0124] The feature update module is as follows:

[0125] v′=tanh(Z l W3+b1)⊙(Z l W4+b2) (15)

[0126] Wherein, v′ represents the final output matrix of the feature update module, W3 represents the third weight matrix in the reconstruction network, W4 represents the fourth weight matrix in the reconstruction network, b1 represents the first bias in the reconstruction network, b2 represents the second bias in the reconstruction network, and tanh(.) represents the tanh activation function.

[0127] First, the input matrix Z l Perform two linear transformations Z respectively l W1 and Z l W2 and add the corresponding biases b1 and b2. Then, the tanh activation function is applied to the result of the first linear transformation to obtain the intermediate result after nonlinear activation. Finally, the activated intermediate result is multiplied element by element with the result of the second linear transformation to obtain the final output vector.

[0128] Through the feature selection and feature update modules, effective feature subsets can be selected and the feature representation part can be dynamically adjusted to adapt to the learning needs of the model and improve the accuracy of the reconstructed network.

[0129] The reconstruction network is constructed based on the conversion layer, which is specifically expressed as:

[0130] H=FC(T2(G+T1(G)) (16)

[0131] Among them, H represents the reconstructed industrial production tabular data, G represents the vector concatenation of the output results of each high-level capsule under the same category, T1 represents the first conversion layer, T2 represents the second conversion layer; FC is the fully connected layer, the purpose of which is to map the input to the tabular data that needs to be restored in the end.

[0132] The input feature G passes through the first conversion layer, and the generated new feature representation is concatenated with the initial feature and then input into the second conversion layer, which helps to maintain the integrity of the input information. Finally, a fully connected layer is used to map the output to the final result.

[0133] The concatenated vector is further transformed through the reconstruction network designed above to obtain the reconstructed tabular data. The MSE loss between the training sample data (i.e., the original tabular data) and the reconstructed tabular data is calculated as the reconstruction loss and added to the subsequent overall loss function.

[0134] The reconstruction loss follows the classic MSE loss, a bunch of u observations in the data set, assuming is the observation value of the capsule network for the i-th observation, y i is the corresponding true value, and the specific formula is as follows:

[0135]

[0136] In the formula, MSE stands for mean square error, which represents the average of the squares of the errors between the predicted value and the actual value. u represents the number of samples, which represents the number of samples in the data set. represents the observation value of the capsule network for the i-th observation, y i is the corresponding true value.

[0137] Considering the impact of reconstruction loss on the network, appropriately reducing the weight of reconstruction loss helps the network fitting work, so that reconstruction loss can fine-tune the marginal loss to obtain better results.

[0138] Specifically, the capsule network uses the sum of marginal loss and reconstruction loss as its overall loss, that is, the overall loss during network training = L + MSE. By iteratively optimizing the capsule network through marginal loss and reconstruction loss, a better network training effect can be obtained, making the trained capsule network more accurate in predicting the industrial production table data.

[0139] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, which can include: ROM, RAM, disk or CD, etc.

[0140] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A classification prediction method for industrial production table data based on capsule network, characterized in that: include: Obtain the industrial production table data to be tested, extract features after preprocessing, input the extracted features into the trained capsule network to obtain the prediction results; The training process of the capsule network includes: Obtain industrial production table data with specific labels and use them as training sample data after preprocessing; The training sample data is embedded to obtain embedded features; J parallel feature extractors are used to extract features of different granularities from the training samples respectively; Merge the embedded features and the high-level features extracted by J parallel feature extractors, encode the merged features into primary capsules, and obtain primary capsule features; The primary capsule features are passed to the advanced capsules through the self-attention static routing mechanism for classification prediction. The classification prediction results of different attribute category features in the industrial production table data are obtained, and the marginal loss L is calculated. The classification prediction result is input into the feature reconstruction network to perform feature reconstruction, and the reconstruction loss MSE between the reconstructed features and the embedded features is calculated; The capsule network is iteratively trained according to the marginal loss L and the reconstruction loss MSE to obtain the trained capsule network.

2. The method for classifying and predicting industrial production table data based on capsule network according to claim 1, characterized in that: The feature extractor includes a feature selection module and a feature extraction module; The mutual information I of the training sample data is measured through the feature selection module, and the preliminary features f' of the training sample data are obtained through sparse mask matrix processing based on the mutual information; The feature extraction module extracts high-level features from the preliminary features f'; the feature selection module and the feature extraction module in J parallel feature extractors are used to process the training sample data respectively to obtain high-level features of different attribute categories in the training sample data at different granularities.

3. The method for classifying and predicting industrial production table data based on capsule network according to claim 2, characterized in that: The feature extraction module extracts high-level features from the preliminary features f'. The specific process includes: S311: Multiply the preliminary feature f' by the parameter matrix W1 to obtain the attention vector q, and map the attention vector q to the interval (0,1) through a Logistic function, that is: q=Logistic(W1f′) Where q represents the attention vector, W1 represents the first weight matrix, and f' represents the preliminary features of the training sample data; S312: Multiply the attention vector q with the embedded vector W2f, and process it through the Relu activation function to obtain the high-level feature f * , which is specifically expressed as: f * =ReLU(q⊙(W2f)) Among them, f * represents the high-level features of the training sample data, ⊙ represents element-by-element multiplication, W2 represents the second weight matrix, and f represents the initial features of the training sample data.

4. The method for classifying and predicting industrial production table data based on capsule network according to claim 1, characterized in that: The combined features are encoded by the optimized multivariate Gaussian kernel to obtain the primary capsule features, which are specifically expressed as: you i =k i (x,μ i ,S i ), Among them, u i represents the primary capsule feature, k i (x,μ i ,Σ i ) represents the probability density function of the i-th Gaussian distribution, μ i Represents the center point position vector of the Gaussian kernel in multidimensional space, ∑ i represents the distribution width of the feature, x represents the feature after the embedded feature is merged with the high-level features extracted by J parallel feature extractors, (·) T represents transpose, and m represents the dimension of the feature vector x.

5. The method for classifying and predicting industrial production table data based on capsule network according to claim 1, characterized in that: The primary capsule features are passed to the high-level capsule through the self-attention static routing mechanism. The specific process includes: In the formula, Represents the output prediction tensor of all primary capsules in the lth layer of the capsule network, d represents the dimension of the capsule network, n l represents the nth primary capsule in the lth layer of the capsule network, n l+1 represents the nth primary capsule in the l+1th layer of the capsule network, represents the transposed feature of the nth primary capsule from the lth layer in the capsule network, (·) T represents transpose, Represents the weight matrix of the lth layer in the capsule network; In the formula, represents the total input of the nth high-level capsule in the l+1th layer of the capsule network, represents the transpose of the output prediction tensor of all primary capsules in layer l of the capsule network, represents the logarithmic prior matrix containing all weights of the lth layer in the capsule network, Represents the coupling coefficient matrix generated by the self-attention algorithm in the lth layer of the capsule network.

6. The method for classifying and predicting industrial production table data based on capsule network according to claim 5, characterized in that: The coupling coefficient matrix is ​​calculated based on the self-attention tensor, which is specifically expressed as: in, Represents the coupling coefficient matrix from all capsules in the lth layer to the nth capsule in the l+1th layer in the capsule network, Represents the self-attention matrix from the nth capsule in the lth layer to the nth capsule in the l+1th layer in the capsule network; represents the sum of the elements of the self-attention matrix from the nth capsule in the lth layer to the nth capsule in the l+1th layer in the capsule network, Represents the cumulative score of all capsules in the lth layer of the capsule network on the nth capsule in the l+1th layer; In the formula, represents the attention weight matrix of all capsules in layer l to the nth capsule in layer l+1 in the capsule network, Represents the relationship matrix between all capsules in the l+1th layer of the capsule network, Representing the relationship matrix The transpose of represents the balance coefficient, and d represents the dimension of the feature vector of the l+1th layer in the capsule network.

7. The method for classifying and predicting industrial production table data based on capsule network according to claim 1, characterized in that: The classification prediction results of different attribute category features in the industrial production table data are specifically as follows: v k =||y j ||2,1≤k≤K in, Represents the classification prediction result of the capsule network, y j represents the feature vector output by the jth high-level capsule in the capsule network, v k represents the L2 norm of the output feature vector of the jth advanced capsule under the kth attribute category, ||·||2 represents the L2 norm, K represents the total number of different attribute categories, z represents the total number of advanced capsules, It represents the sum of the L2 norms of z high-level capsules under the k-th attribute category.

8. The method for classifying and predicting industrial production table data based on capsule network according to claim 1 or 7, characterized in that: The marginal loss function is specifically: L represents the marginal loss function of the capsule network, where L is the sum of the losses of all different attribute categories; K represents the total number of different attribute categories, I k T represents the average length of the feature vector of the advanced capsule responsible for predicting the kth category; k Represents the label of the kth category. For a binary classification problem, if the kth category is the target category, then T k =1, otherwise T k =0; t + represents the threshold of the positive class; t - Represents the threshold of the negative class.

9. The method for classifying and predicting industrial production table data based on capsule network according to claim 1, characterized in that: Select the classification prediction results of the capsule prediction network The output vectors of the advanced capsules that predict the correct categories are merged, and the merged vectors are input into the reconstruction network. After feature selection and feature updating, the reconstructed industrial production table data is output.

10. The method for classifying and predicting industrial production table data based on capsule network according to claim 1, characterized in that: The reconstruction loss is specifically: In the formula, MSE represents the reconstruction loss, u represents the number of training sample data, represents the predicted value of the capsule network for the i-th training sample data, y i Representation and The corresponding true value.

Images