Orthogonal matrix-auto-encoder-based disordered multi-classification variable coding method
Through orthogonal matrix encoding and Bayesian optimization autoencoder, the dimensionality reduction of disordered multi-classification variables is solved, which solves the dimension and order problems when inputting disordered multi-classification variables in neural networks, and improves the accuracy of target prediction.
Patent Information
- Application Number
- CN202411806284.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-10
- Publication Date
- 2025-04-04
AI Technical Summary
Neural network models such as BP and LSTM only support numerical input in target prediction. Existing coding methods may lead to dimensional disasters and misleading sequences, making it difficult to effectively deal with disordered multi-classified variables.
The disordered multi-classification variable is encoded using an orthogonal matrix, combined with Bayesian optimization autoencoder hyperparameters, orthogonal numerical vector dimensionality reduction, and finally input the low-dimensional numerical encoding into the neural network model for target prediction.
Through orthogonal matrix encoding and autoencoder dimensionality reduction, the dimensionality problem and sequence misleading of disordered multi-classification variables when inputting into neural networks is solved, and the accuracy of prediction results and the richness of data are improved.
Smart Images
Figure CN119760501A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to an unordered multi-classification variable encoding method based on an orthogonal matrix-autoencoder, and belongs to the technical field of data processing and management. Background Art
[0002] Target prediction refers to predicting the target associated with the input through a certain method based on the existing data. Target prediction is becoming increasingly important in the fields of finance, medical care, manufacturing, electricity, weather, transportation, etc. Excellent prediction results can identify potential risks, provide reliable prediction data, and enhance users' risk management capabilities; they can also formulate executable future plans and programs to provide decision support for enterprises.
[0003] Studies have shown that artificial neural networks have better performance and target prediction effects under the condition of multi-classification variable input. Neural networks are the mainstream model for solving target prediction problems at this stage. It should be noted that neural network models such as BP and LSTM only support numerical data input, but a large part of the data set contains non-numerical variables, that is, categorical variables. For example, the binary categorical variable "drug response" contains two categories of "positive and negative", and "assessment results" is also a binary categorical variable, containing two categories of "pass and fail"; for example, the multi-classification variable "occupation" contains categories such as "teachers, firefighters, construction workers, doctors...". Numerical variables are easier to interpret, but when categorical variables dominate in the data set, it becomes difficult to view data trends and make predictions. Therefore, before entering categorical variables into neural networks, a method must be adopted to map categorical variables to numerical values.
[0004] Categorical variable encoding methods include One-hot encoding, label encoding, target encoding, and Embedding encoding. Among them, One-hot encoding is applicable to unordered categorical variables. It uses 0 and 1 to generate an N-dimensional binary vector for N categories, and only one position in the vector is 1, and the rest are 0. Label encoding is applicable to ordered categorical variables. It maps each category to an integer starting from 0. Target encoding uses the target mean corresponding to the categorical variable to replace the categorical variable. Embedding randomly generates a weight vector according to the specified embedding dimension, and maps the text to a word vector according to the index of the categorical variable.
[0005] An unordered multi-categorical variable is a categorical variable that contains several (usually more than 3) categories with no order or distance difference between the categories.
[0006] Some existing encoding methods are not fully applicable to unordered multi-classification variables. For example, when there are a large number of categories, applying One-hot encoding or applying Embedding and setting a large embedding dimension will result in the risk of dimensionality explosion, and the data after One-hot encoding is too sparse. At the same time, Embedding and label encoding will introduce additional order and distance to the originally unordered categorical variables. Target encoding will affect the ability of the neural network model to extract information and there is a risk of overfitting.
[0007] In general, there is currently no encoding method for converting unordered multi-categorical variables into numerical codes with unordered variable characteristics and smaller dimensions. Summary of the invention
[0008] The present invention aims to solve the problem that neural network models such as BP and LSTM only support numerical input when performing target prediction. The existing encoding methods may produce dimensionality disasters and may also introduce additional order misleading and distance deviations in neural network calculations. Therefore, it is necessary to convert unordered multi-classification variables into numerical codes with unordered variable characteristics and smaller dimensions.
[0009] The technical solution adopted by the present invention to solve the above-mentioned problem is: the steps of the present invention include:
[0010] Step 1: Use an orthogonal matrix to encode unordered multi-classification variables;
[0011] Step 2: Use Bayesian optimization to optimize the autoencoder hyperparameters.
[0012] Step 3: Use the optimized autoencoder to perform dimensionality reduction of orthogonal numerical vectors;
[0013] Step 4: Use low-dimensional numerical encoding to input into the neural network model for target prediction.
[0014] Furthermore, the orthogonal matrix used in step 1 to encode unordered multi-classification variables includes:
[0015] Step 101: Construct an N-order square matrix based on the number of categories N of the multi-classification variable
[0016] Step 102: Use Gram-Schmidt orthogonal transformation to perform QR decomposition of a to obtain an orthogonal matrix
[0017] Step 103: Map the orthogonal matrix and the original unordered multi-classification variable one by one according to the position index.
[0018] Furthermore, in step 101, an N-order square matrix is constructed according to the number of categories N of the multi-classification variable.
[0019] Setting up unordered multi-class variables The sample size is The variable contains N categories c i ,i=1,2,…N,N≥3, There are multiple data of the same category, where category c i There are m pieces of data, c i1 , c i2 , …c im , m≥1.
[0020] c i =(c i1 ,c i2 …c im ),m≥1
[0021] Using encoding method f, class c i Mapped to a value n i , so that the unordered multi-classification variable Mapping to numeric variables
[0022]
[0023] The numerical data of categorical data of the same category under the same encoding method mapping are the same.
[0024] There are m identical categories c in i , under the f mapping, c i1 , c i2 , …c im Convert to numerical value n i1 , n i2 ,…n im , and n i1 =n i2 =…=n im .
[0025]
[0026] c i =(c i1 , c i2 , …c im ), n i =(n i1 , n i2 ,…n im ), m≥1
[0027] n i1 =n i2 =…=n im
[0028] According to the number of categories N of the unordered multi-classification variable, set the N-order square matrix A=[a1,a2,…,a i , …, a N ], where a i is the column vector of A,
[0029] Further, in step 102, the Gram-Schmidt orthogonal transformation is used to perform QR decomposition of A to obtain an orthogonal matrix
[0030] The Gram-Schmidt orthogonal transformation can transform a non-orthogonal basis into an orthogonal basis by first orthogonalizing the non-orthogonal basis and then normalizing it.
[0031]
[0032] Q=[q1,q2,…,q i ,…,q N ]
[0033] The orthogonal matrix Q consists of N column vectors, Q = [q1, q2, ..., q i ,…,q N ],in It represents the orthogonal basis obtained by orthogonal transformation of N-order square matrix A, which is also the column vector that constitutes the orthogonal matrix Q. Represents the column vector q i The value of each dimension in .
[0034] An orthogonal matrix has the property that both row vectors and column vectors are unit vectors that are orthogonal to each other.
[0035] QQ T =Q T Q=E
[0036]
[0037] ‖q i ‖2=1
[0038] Among them, Q T represents the transposed matrix of the orthogonal matrix, E represents the N-order identity matrix, q i T Indicates q i The transposed vector of ||q i ||2 means q i The module is q i Length.
[0039] Therefore, mapping the unordered multi-classification variables into orthogonal numerical codes can ensure that different categories have independent positions, equal distances, and the same sizes, while increasing data richness and avoiding underfitting in the subsequent prediction process due to poor data.
[0040] Furthermore, in step 103, the orthogonal matrix and the original unordered multi-classification variable are mapped one by one according to the position index:
[0041] Construct an N-order unit matrix E, and use matrix multiplication to map the orthogonal matrix to the original unordered multi-classification variable one by one according to the position index to obtain the orthogonal numerical encoding matrix X consists of row vectors.
[0042]
[0043] Among them, x i =q i T , x i is the row vector of X.
[0044] Furthermore, in step 2, Bayesian optimization is used to optimize the autoencoder hyperparameters:
[0045] The autoencoder consists of an input layer, a hidden layer, a latent space, and an output layer, each with a different number of nodes. The structure from the input layer to the latent space is called an encoder, and the structure from the latent space to the output layer is called a decoder. The number of nodes in the input layer is the original dimension of the data, the number of nodes in the latent space is the dimension that the data needs to be compressed, and the number of nodes in the hidden layer is between the original dimension and the compressed dimension.
[0046] The structure of the autoencoder model is not fixed. Since the dataset may contain multiple unordered multi-classification variables with different numbers of categories, the number of hidden layers and nodes in the autoencoder is adjusted according to the size of the dataset and the required level of data compression. Therefore, parameters such as the number of layers and nodes in the hidden layer of the autoencoder are key factors affecting the performance of the autoencoder and the effect of data dimensionality reduction.
[0047] Bayesian optimization is suitable for black-box optimization problems with complex objective functions, no analytical expressions, and high computational costs. The core idea is to use a probabilistic proxy model (such as Gaussian process or random forest) to approximate the objective function, and use this model to select the next evaluation point so that the objective function can converge to the optimal value as quickly as possible. It is an optimization method based on prior information. There are two core parts in this process: probability model and acquisition strategy.
[0048] Bayes theorem formula:
[0049]
[0050] Where x represents the hyperparameter sequence, which in this method represents the number of hidden layers and nodes in the autoencoder; y represents the target output. p(y|x) represents the probability of y occurring under the condition that x occurs, that is, the posterior probability of y. p(x) represents the marginal probability of x occurring, p(y) represents the prior probability of y occurring independently, and p(x|y) represents the probability of x occurring under the condition that y occurs.
[0051] The TPE (Tree-structured Parzen Estimator) algorithm under the optuna framework is used to implement Bayesian optimization technology. TPE is a tree-structured Bayesian optimization method. In TPE, two density functions are used to define p(x|y).
[0052]
[0053] TPE uses kernel density estimation (KDE) to model the probability distribution of the objective function value. Specifically, TPE divides the objective function value into two parts: one is the probability distribution of the better objective value l(x), and the other is the probability distribution of the worse objective value g(x). * Indicates the threshold value.
[0054] TPE uses expected improvement (EI) as the acquisition function.
[0055]
[0056] Among them, p M (y|x) represents the Bayesian formula, γ=p(y<y * ) represents a certain quantile of TPE.
[0057] It follows that:
[0058]
[0059] The value of EI is proportional to Depends on the ratio of the two probabilities, so we need to find the ratio The largest x, in each iteration, returns the maximum EI.
[0060] Furthermore, in step 3, the optimized autoencoder is used to perform dimensionality reduction of orthogonal numerical vectors:
[0061] The working mechanism of the autoencoder is as follows: the encoder compresses the input data into a pre-set low-dimensional code through a series of transformations f, and the decoder tries to reconstruct the low-dimensional code into the original input data through a series of transformations g. By minimizing the reconstruction error between input and output, the autoencoder is iteratively trained, and ultimately the autoencoder is expected to learn the abstract feature representation z of the sample.
[0062] The working principle of the autoencoder for data dimensionality reduction is shown in the formula.
[0063]
[0064] z=f(W1x+b1)
[0065]
[0066] in, Represents the data space, represents the latent space. z=f(W1x+b1) represents the encoder, f is the activation function of the encoder, W1 is the weight matrix, x is the input data, b1 is the bias term, and z is the data after the encoder dimensionality reduction. represents the decoder, g is the activation function of the decoder, W2 is the weight matrix, b2 is the bias term, It is the data after the decoder upgrades the dimension. f, g = arg min f，g ‖xg(f(x))‖2 represents the training objective, i.e. minimizing the reconstruction error.
[0067] Using ReLU as the activation function, ReLU is relatively simple to calculate and has relatively stable performance. It can avoid gradient vanishing during model training, increase the model's nonlinear fitting ability, and has good performance on multiple tasks and data sets. The ReLU formula is f(x) = max(0, x).
[0068] Taking the autoencoder model with 4 hidden layers and data compression dimension of 1 as an example, X N×N represents an orthogonal numerical encoding matrix, Represents the numerical matrix reconstructed by the autoencoder. Represents the process data compressed by the coding layer. Represents the process data reconstructed by the decoding layer, let Represents the encoded data after dimensionality reduction.
[0069] X, X (1) , X (2) , They all consist of row vectors.
[0070] The change of data dimension during the encoding and decoding process of the orthogonal numerical coding matrix through AE is shown in the formula.
[0071]
[0072] Among them, x i and is an N-dimensional row vector, x (1) i and is an a-dimensional row vector, x (2) i and is a b-dimensional row vector, z i is the encoded value after dimensionality reduction.
[0073] Finally, the orthogonal numerical coding matrix is obtained after dimension reduction.
[0074]
[0075] Since the values of the same category are the same after encoding, the numerical encoding after dimensionality reduction is expanded to the original sample size Get the encoded representation of all unordered multi-class variables.
[0076]
[0077] Among them, c i1 =c i2 =c i Represents an unordered multi-class variable The category, z i1 =z i2 =z i Indicates category c i The encoding after dimensionality reduction by orthogonal coding.
[0078] Furthermore, in step 4, low-dimensional numerical encoding is used to input into the neural network model for target prediction:
[0079] The unordered multi-classification variables are mapped to low-dimensional numerical codes, and the processed data sets are input into the BP neural network for target prediction. The target prediction results and evaluation index results prove the applicability and effectiveness of the unordered multi-classification variable encoding method based on orthogonal matrix-autoencoder.
[0080] The beneficial effects of the present invention are as follows: the present invention solves the problem that neural network models such as BP and LSTM only support numerical input when carrying out target prediction. Existing encoding methods may produce dimensionality disasters and may also introduce additional order misleading and distance deviations in neural network calculations. Therefore, it is necessary to convert unordered multi-classification variables into numerical codes with unordered variable characteristics and smaller dimensions. By proposing an outlier detection method based on two new anomaly factors, an orthogonal matrix is used to numerically encode unordered multi-classification variables to ensure that the codes are independent in position, equal in distance, and the same in size; secondly, an autoencoder is used to carry out representation learning on the numerical codes and then perform dimensionality reduction to ensure that the data input to the neural network learns the relevant characteristics of the orthogonal matrix while being low-dimensional. BRIEF DESCRIPTION OF THE DRAWINGS
[0081] Figure 1 is a flow chart of the present invention;
[0082] Figure 2 This is a structural example and working principle of the autoencoder used in the present invention;
[0083] Figure 3 It is the neural network target prediction result after the present invention performs disordered multi-classification variable encoding;
[0084] Figure 4 It is the target prediction result of the neural network after encoding the disordered multi-classification variables using the one-hot encoding method;
[0085] Figure 5 It is the target prediction result of the neural network after the label encoding method encodes the disordered multi-classification variables;
[0086] Figure 6 It is the neural network target prediction result after the target encoding method encodes the disordered multi-classification variables. DETAILED DESCRIPTION
[0087] Combination Figure 1 The present embodiment is described. The steps of the unordered multi-classification variable encoding method based on an orthogonal matrix-autoencoder described in the present embodiment include:
[0088] Step 1: Use an orthogonal matrix to encode unordered multi-classification variables;
[0089] Step 101: Construct an N-order square matrix based on the number of categories N of the multi-classification variable
[0090] Setting up unordered multi-class variables The sample size is The variable contains N categories c i , i=1,2,…N,N≥3, There are multiple data of the same category, where category c i There are m pieces of data, c i1 , c i2 , …c im , m≥1.
[0091] c i =(c i1 , c i2 , …c im ), m≥1 using encoding method f, class c i Mapped to a value n i , so that the unordered multi-classification variable Mapping to numeric variables
[0092]
[0093] The numerical data of categorical data of the same category under the same encoding method mapping are the same.
[0094] There are m identical categories c in i , under the f mapping, c i1 , c i2 , …c im Convert to numerical value n i1 , n i2 ,…n im , and n i1 =n i2 =…=n im .
[0095]
[0096] c i =(c i1 , c i2 , …c im ), n i =(n i1 , n i2 ,…n im ), m≥1
[0097] n i1 =n i2 =…=n im
[0098] According to the number of categories N of the unordered multi-classification variable, set the N-order square matrix A=[a1,a2,…,a i , …, a N ], where a i is the column vector of A,
[0099] Step 102: Use Gram-Schmidt orthogonal transformation to perform QR decomposition of A and obtain an orthogonal matrix
[0100] The Gram-Schmidt orthogonal transformation can transform a non-orthogonal basis into an orthogonal basis by first orthogonalizing the non-orthogonal basis and then normalizing it.
[0101]
[0102] Q=[q1,q2,…,q i ,…,q N ]
[0103] The orthogonal matrix Q consists of N column vectors, Q = {q1, q2, ..., q i ,…,q N ],in It represents the orthogonal basis obtained by orthogonal transformation of N-order square matrix A, which is also the column vector that constitutes the orthogonal matrix Q. Represents the column vector q i The value of each dimension in .
[0104] An orthogonal matrix has the property that both row vectors and column vectors are unit vectors that are orthogonal to each other.
[0105] QQ T =Q T Q=E
[0106]
[0107] ||q i ||2=1
[0108] Among them, Q T represents the transposed matrix of the orthogonal matrix, E represents the N-order identity matrix, q i T Indicates q i The transposed vector of ||q i ||2 means q i The module is q i Length.
[0109] Therefore, mapping the unordered multi-classification variables into orthogonal numerical codes can ensure that different categories have independent positions, equal distances, and the same sizes, while increasing data richness and avoiding underfitting in the subsequent prediction process due to poor data.
[0110] Step 103: Map the orthogonal matrix and the original unordered multi-classification variable one by one according to the position index.
[0111] Construct an N-order unit matrix E, and use matrix multiplication to map the orthogonal matrix to the original unordered multi-classification variable one by one according to the position index to obtain the orthogonal numerical encoding matrix X consists of row vectors.
[0112]
[0113] Among them, x i =q i T , x i is the row vector of X.
[0114] Step 2: Use Bayesian optimization to optimize the autoencoder hyperparameters.
[0115] The autoencoder consists of an input layer, a hidden layer, a latent space, and an output layer, each with a different number of nodes. The structure from the input layer to the latent space is called an encoder, and the structure from the latent space to the output layer is called a decoder. The number of nodes in the input layer is the original dimension of the data, the number of nodes in the latent space is the dimension that the data needs to be compressed, and the number of nodes in the hidden layer is between the original dimension and the compressed dimension. An example of an autoencoder structure is as follows: Figure 2 shown.
[0116] The structure of the autoencoder model is not fixed. Since the dataset may contain multiple unordered multi-classification variables with different numbers of categories, the number of hidden layers and nodes in the autoencoder is adjusted according to the size of the dataset and the required level of data compression. Therefore, parameters such as the number of layers and nodes in the hidden layer of the autoencoder are key factors affecting the performance of the autoencoder and the effect of data dimensionality reduction.
[0117] Bayesian optimization is suitable for black-box optimization problems with complex objective functions, no analytical expressions, and high computational costs. The core idea is to use a probabilistic proxy model (such as Gaussian process or random forest) to approximate the objective function, and use this model to select the next evaluation point so that the objective function can converge to the optimal value as quickly as possible. It is an optimization method based on prior information. There are two core parts in this process: probability model and acquisition strategy.
[0118] Bayes' theorem formula:
[0119]
[0120] Where x represents the hyperparameter sequence, which in this method represents the number of hidden layers and nodes in the autoencoder; y represents the target output. p(y|x) represents the probability of y occurring under the condition that x occurs, that is, the posterior probability of y. p(x) represents the marginal probability of x occurring, p(y) represents the prior probability of y occurring independently, and p(x|y) represents the probability of x occurring under the condition that y occurs.
[0121] The TPE (Tree-structured Parzen Estimator) algorithm under the optuna framework is used to implement Bayesian optimization technology. TPE is a tree-structured Bayesian optimization method. In TPE, two density functions are used to define p(y|x).
[0122]
[0123] TPE uses kernel density estimation (KDE) to model the probability distribution of the objective function value. Specifically, TPE divides the objective function value into two parts: one is the probability distribution of the better objective value l(x), and the other is the probability distribution of the worse objective value g(x). * Indicates the threshold value.
[0124] TPE uses expected improvement (EI) as the acquisition function.
[0125]
[0126] Among them, p M (y|x) represents the Bayesian formula, γ=p(y<y * ) represents a certain quantile of TPE.
[0127] It follows that:
[0128]
[0129] The value of EI is proportional to Depends on the ratio of the two probabilities, so we need to find the ratio The largest x, in each iteration, returns the maximum EI.
[0130] Step 3: Use the optimized autoencoder to perform dimensionality reduction of orthogonal numerical vectors;
[0131] The working mechanism of the autoencoder is as follows: the encoder compresses the input data into a pre-set low-dimensional code through a series of transformations f, and the decoder tries to reconstruct the low-dimensional code into the original input data through a series of transformations g. By minimizing the reconstruction error between input and output, the autoencoder is iteratively trained, and ultimately the autoencoder is expected to learn the abstract feature representation z of the sample.
[0132] The working principle of the autoencoder for data dimensionality reduction is shown in the formula.
[0133]
[0134] z=f(W1x+b1)
[0135]
[0136] in, Represents the data space, represents the latent space. z=f(W1x+b1) represents the encoder, f is the activation function of the encoder, W1 is the weight matrix, x is the input data, b1 is the bias term, and z is the data after the encoder dimensionality reduction. represents the decoder, g is the activation function of the decoder, W2 is the weight matrix, b2 is the bias term, It is the data after the decoder upgrades the dimension. f, g = arg min f，g ||xg(f(x))||2 represents the training objective, which is to minimize the reconstruction error.
[0137] Using ReLU as the activation function, ReLU is relatively simple to calculate and has relatively stable performance. It can avoid gradient vanishing during model training, increase the model's nonlinear fitting ability, and has good performance on multiple tasks and data sets. The ReLU formula is f(x) = max(0, x).
[0138] Taking the autoencoder model with 4 hidden layers and data compression dimension of 1 as an example, X N×N represents an orthogonal numerical encoding matrix, Represents the numerical matrix reconstructed by the autoencoder. Represents the process data compressed by the coding layer. Represents the process data reconstructed by the decoding layer, let Represents the encoded data after dimensionality reduction.
[0139] X, X (1) , X (2) , They all consist of row vectors.
[0140] The change of data dimension during the encoding and decoding process of the orthogonal numerical coding matrix through AE is shown in the formula.
[0141]
[0142] Among them, x i and is an N-dimensional row vector, x (1) i and is an a-dimensional row vector, x (2) i and is a b-dimensional row vector, z i is the encoded value after dimensionality reduction.
[0143] Finally, the orthogonal numerical coding matrix is obtained after dimension reduction.
[0144]
[0145] Since the values of the same category are the same after encoding, the numerical encoding after dimensionality reduction is expanded to the original sample size Get the encoded representation of all unordered multi-class variables.
[0146]
[0147] Among them, c i1 =c i2 =c i Represents an unordered multi-class variable The category, z i1 =z i2 =z i Indicates category c i The encoding after dimensionality reduction by orthogonal coding.
[0148] Step 4: Use low-dimensional numerical encoding to input into the neural network model for target prediction.
[0149] The unordered multi-classification variables are mapped to low-dimensional numerical codes, and the processed data sets are input into the BP neural network for target prediction. The target prediction results and evaluation index results prove the applicability and effectiveness of the unordered multi-classification variable encoding method based on orthogonal matrix-autoencoder.
[0150] In this implementation, after the design of the unordered multi-classification variable encoding method based on the orthogonal matrix-autoencoder is completed, the unordered multi-classification variable encoding method based on the orthogonal matrix-autoencoder is used to map the unordered multi-classification variable encoding into a low-dimensional numerical encoding, and input it into the neural network for target prediction. The specific steps are as follows:
[0151] Step 1: Use an orthogonal matrix to encode unordered multi-classification variables;
[0152] Step 2: Use Bayesian optimization to optimize the autoencoder hyperparameters;
[0153] Step 3: Use the optimized autoencoder to reduce the dimension of orthogonal numerical vectors;
[0154] Step 4: Use low-dimensional numerical encoding to input into the neural network model for target prediction;
[0155] After the above steps, the target prediction results of the data set can be obtained.
[0156] The present invention uses a public data set and a BP neural network to verify the target prediction results after encoding unordered multi-classification variables.
[0157] The sample size of the dataset, the number of variables, the number of categorical variables, the number of categories contained in the categorical variables, and the targets used for prediction are shown in Table 1.
[0158] Table 1 Dataset
[0159]
[0160] The categorical variables condition(3) and joint(3) in the Gait dataset contain a small number of categories, and leg(2) is a binary categorical variable, so the label data in the original data is used for the experiment. Only the multi-categorical variable Subject in the Gait dataset is encoded.
[0161] Three classic evaluation indicators are used to evaluate the performance of different encoding methods in the BP neural network target prediction task, including Mean Absolute Error (MAE), Root Mean Square Error (RMSE), coefficient of determination (R 2 ).
[0162]
[0163]
[0164]
[0165] Where n is the number of samples, y i is the true value, is the predicted value, is the average of the true values.
[0166] MAE is the average of the absolute errors, and RMSE is the square root of the average of the sum of squared errors. Both MAE and RMSE are indicators that measure the deviation between the predicted value and the true value. The smaller the value, the better the final prediction result. 2 It is used to evaluate the linear relationship between the true value and the predicted value. The closer the value is to 1, the better the model fit is.
[0167] When using autoencoders for encoding and dimensionality reduction, the unordered multi-classification variables are divided into training sets and validation sets according to a ratio of 7:3. The early stopping strategy is introduced, and the patience parameter of early stopping is set to 100. The number of hidden layers of AE and the number of nodes in each layer are obtained by Bayesian optimization. The number of hidden layers of the encoder is set to no more than 5, and the decoder and encoder structures are symmetrical. The relevant parameters of the autoencoder are shown in Table 2 below. Because the sample size of the experimental data set is small, the batch size is set to the sample size of the data set.
[0168] Table 2 Related parameters
[0169]
[0170] When using BP neural network for target prediction, the data set is divided into training set, validation set and test set in a ratio of 7:2:1. Since the sample size of the experimental data set is small, the batch size is set to the sample size of the data set. In order to quickly conduct experiments and obtain a better prediction result, the 5-fold cross-validation grid search hyperparameter optimization method is used. In order to improve model performance and avoid overfitting, L2 Regularization and early stopping strategies are introduced, and the patience parameter of early stopping is set to 100. The BP neural network parameters are shown in Table 3 below.
[0171] Table 3 BP neural network parameters
[0172]
[0173] One embodiment of the present invention is that after the present invention performs disordered multi-classification variable encoding with one-hot encoding, label encoding, and target encoding, the target prediction results of the processed data set in the BP neural network model are respectively as follows: Figures 3 to 6 shown.
[0174] One embodiment of the present invention is that after the present invention performs disordered multi-classification variable encoding with one-hot encoding, label encoding, and target encoding, the target prediction index results of the processed data set in the BP neural network model are shown in Table 4 below.
[0175] Table 4 Results of target prediction indicators
[0176]
[0177] The examples verify that the disordered multi-classification variables processed by the present invention can meet the data input requirements of the subsequent neural network model, promote the data learning ability of the neural network, and improve the target prediction results of the neural network.
[0178] The above is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this profession can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent replacement and improvement made to the above embodiments without departing from the content of the technical solution of the present invention, based on the technical essence of the present invention, within the spirit and principles of the present invention, still fall within the protection scope of the technical solution of the present invention.
Claims
1. A method for encoding unordered multi-classification variables based on orthogonal matrix-autoencoder, characterized in that: The following steps are involved: Step 1: Use an orthogonal matrix to encode unordered multi-classification variables; Step 2: Use Bayesian optimization to optimize the autoencoder hyperparameters. Step 3: Use the optimized autoencoder to perform dimensionality reduction of orthogonal numerical vectors; Step 4: Use low-dimensional numerical encoding to input into the neural network model for target prediction.
2. The method for encoding unordered multi-classification variables based on an orthogonal matrix-autoencoder according to claim 1, characterized in that: Step 1 includes the following sub-steps: Step 101: Construct an N-order square matrix based on the number of categories N of the multi-classification variable Step 102: Use Gram-Schmidt orthogonal transformation to perform QR decomposition of A and obtain an orthogonal matrix Step 103: Map the orthogonal matrix and the original unordered multi-classification variable one by one according to the position index.
3. The method for encoding unordered multi-classification variables based on an orthogonal matrix-autoencoder according to claim 2, characterized in that: The specific method of step 101 is: Setting up unordered multi-class variables The sample size is The variable contains N categories c i ,i=1,2,…N,N≥3, There are multiple data of the same category, where category c i There are m pieces of data, c i1 , c i2 , …c im , m≥1; c i =(c i1 ,c i2 ,…c im ),m≥1 Using encoding method f, class c i Mapped to a value n i , so that the unordered multi-classification variable Mapping to numeric variables The numerical data of the same category of categorical data mapped under the same coding method are the same; There are m identical categories c in i , under the f mapping, c i1 , c i2 , …c im Convert to numerical value n i1 , n i2 ,…n im , and n i1 =n i2 =…=n im ; c i =(c i1 ,c i2 ,…c im ),n i =(n i1 ,n i2 ,…n im ),m≥1 n i1 =n i2 =…=n im According to the number of categories N of the unordered multi-classification variable, set the N-order square matrix A={a1,a2,…,a i , …, a N ], where a i is the column vector of A, 4. The method for encoding unordered multi-classification variables based on an orthogonal matrix-autoencoder according to claim 2, characterized in that: The specific method of step 102 is: Gram-Schmidt orthogonal transformation can convert non-orthogonal basis into orthogonal basis, firstly orthogonalize the non-orthogonal basis, and then normalize it; Q=[q1,q2,…,q i ,…,q N ] The orthogonal matrix Q consists of N column vectors, Q = [q1, q2, ..., q i , …, q N ],in It represents the orthogonal basis obtained by orthogonal transformation of N-order square matrix A, which is also the column vector that constitutes the orthogonal matrix Q. Represents the column vector q i The value of each dimension in ; An orthogonal matrix has the property that both row vectors and column vectors are unit vectors that are orthogonal to each other. QQ T =Q T Q=E ||q i ||2＝1 Among them, Q T represents the transposed matrix of the orthogonal matrix, E represents the N-order identity matrix, q i T Indicates q i The transposed vector of ||q i ||2 means q i The module is q i Length; Mapping unordered multi-classification variables into orthogonal numerical codes can ensure that different categories have independent positions, equal distances, and the same sizes, while increasing data richness and avoiding underfitting in subsequent prediction processes due to poor data.
5. The method for encoding unordered multi-classification variables based on an orthogonal matrix-autoencoder according to claim 2, characterized in that: The specific method of step 103 is: Construct an N-order unit matrix E, and use matrix multiplication to map the orthogonal matrix to the original unordered multi-classification variable one by one according to the position index to obtain the orthogonal numerical encoding matrix X consists of row vectors; in, is the row vector of X.
6. The method for encoding unordered multi-classification variables based on an orthogonal matrix-autoencoder according to claim 1, characterized in that: Step 2: The autoencoder includes an input layer, a hidden layer, a latent space, and an output layer, each with a different number of nodes. The structure from the input layer to the latent space is called an encoder, and the structure from the latent space to the output layer is called a decoder. The number of nodes in the input layer is the original dimension of the data, the number of nodes in the latent space is the dimension that the data needs to be compressed, and the number of nodes in the hidden layer is between the original dimension and the compressed dimension. The structure of the autoencoder model is not fixed; the number of hidden layers and nodes in the autoencoder is adjusted based on the size of the dataset and the level of data compression required; The Bayesian optimization process consists of two core parts: probability model and acquisition strategy; Bayes theorem formula: Where x represents the hyperparameter sequence, which in this method represents the number of hidden layers and nodes in the autoencoder; y represents the target output; p(y|x) represents the probability of y occurring under the condition that x occurs, that is, the posterior probability of y; p(x) represents the marginal probability of x occurring, p(y) represents the prior probability of y occurring independently, and p(x|y) represents the probability of x occurring under the condition that y occurs; Use the TPE algorithm under the optuna framework to implement Bayesian optimization technology; in TPE, two density functions are used to define p(x|y); TPE uses kernel density estimation KDE to model the probability distribution of the objective function value; specifically, TPE divides the objective function value into two parts: one is the probability distribution of good target values l(x), and the other is the probability distribution of bad target values g(x); where y * Indicates the threshold value; TPE uses the expected improvement EI as the acquisition function; Among them, p M (y|x) represents the Bayesian formula, γ=p(y<y * ) represents a certain quantile of TPE; It turns out that: The value of eI is proportional to Depends on the ratio of two probabilities, find the ratio The largest x, in each iteration, returns the maximum EI.
7. The method for encoding unordered multi-classification variables based on an orthogonal matrix-autoencoder according to claim 1, characterized in that: Step 3: The working mechanism of the autoencoder is as follows: the encoder compresses the input data into a pre-set low-dimensional code through a series of transformations f, and the decoder attempts to reconstruct the low-dimensional code into the original input data through a series of transformations g. By minimizing the reconstruction error between input and output, the autoencoder is iteratively trained, and ultimately the autoencoder is expected to learn the abstract feature representation of the sample. The working principle of the autoencoder to carry out data dimensionality reduction is shown in the formula; f： g： in, Represents the data space, represents the latent space; represents the encoder, f is the activation function of the encoder, W1 is the weight matrix, x is the input data, b1 is the bias term, and z is the data after the encoder dimensionality reduction; represents the decoder, g is the activation function of the decoder, W2 is the weight matrix, b2 is the bias term, It is the data after the decoder upgrades the dimension; represents the training objective, i.e. minimizing the reconstruction error; Use ReLU as the activation function. The ReLU formula is: X N×N represents an orthogonal numerical encoding matrix, Represents the numerical matrix reconstructed by the autoencoder; let Represents the process data compressed by the coding layer; Represents the process data reconstructed by the decoding layer, let Represents the encoded data after dimensionality reduction; X,X (1) ,X (2) , All consist of row vectors; The data dimension change during the encoding and decoding process of the orthogonal numerical coding matrix through AE is shown in the formula: in, and is an N-dimensional row vector, and is an a-dimensional row vector, and is a b-dimensional row vector, is the encoded value after dimensionality reduction; Finally, the orthogonal numerical coding matrix is obtained after dimension reduction. Expand the reduced-dimensional numerical encoding to the original sample size Get the encoded representation of all unordered multi-class variables: Among them, c i1 =c i2 =c i Represents an unordered multi-class variable Category, Indicates category c i The encoding after dimensionality reduction by orthogonal coding.
8. The method for encoding unordered multi-classification variables based on an orthogonal matrix-autoencoder according to claim 1, characterized in that: Step 4: The unordered multi-classification variables are mapped into low-dimensional numerical codes, and the processed data sets are input into the BP neural network for target prediction. The target prediction results and evaluation index results are used to prove the applicability and effectiveness of the unordered multi-classification variable encoding method based on orthogonal matrix-autoencoder.