Novel width learning regression prediction method with deep and flat structures

Abstract

A width learning system (BLS) is a newest proposed machine learning algorithm, and has extremely high calculation efficiency and good approximation capability. Existing researches prove that in some cases, deep learning can be replaced by width learning, and even a better effect can be achieved. However, width learning only has a feature mapping layer and an enhancement layer, and a shallower structure enables the feature learning ability to be insufficient, resulting in poor performance. Therefore, the invention provides a novel width learning structure, and a plurality of feature mapping layer and enhancement layer structures are adopted, so that the feature learning capability and the nonlinear mapping capability of a width learning model are improved. Meanwhile, a sparse encoder is adopted to reconstruct original features, the original features are added into input of an output layer, robustness of the model is enhanced, and original information is prevented from being lost. In addition, each multi-layer BLS structure is regarded as a BLS block, and the fitting capability and robustness are further improved by stacking the BLS.

Description

Technical field:

[0001] The present invention relates to the field of machine learning, and more specifically, to a novel wide learning regression prediction method with a deep and flat structure. The method combines the advantages of deep learning and wide learning, aims to improve the accuracy and efficiency of regression prediction, and is applicable to multiple fields such as big data analysis, financial prediction, medical diagnosis, industrial control, etc.

[0002] Traditional machine learning methods are often limited by model complexity and computational cost when processing complex data. This paper proposes a new wide learning system that uses the characteristics of depth and flat structure to reduce computational complexity and training time while maintaining the model's prediction ability. By stacking and optimizing the outputs of multiple shallow models, the method of the present invention can more effectively capture the high-dimensional features and nonlinear relationships of the data, and is widely applicable to application scenarios that require high-precision regression prediction. Background technology:

[0003] In the era of big data, deep learning has become an important tool for solving complex problems and realizing artificial intelligence. As a branch of machine learning, deep learning imitates the working principle of the human brain as much as possible, and learns and understands data through a hierarchical network structure. Its powerful feature learning and abstract representation capabilities have enabled it to achieve major breakthroughs in image recognition, natural language processing, speech recognition and other fields. However, since deep learning models are usually composed of millions or even hundreds of millions of parameters, and most deep learning models require a large number of iterative optimization processes to train the model, which requires extremely high computing resources and unbearable time costs, this limits the application of deep learning models in resource-constrained environments.

[0004] Due to the above limitations, the industrial application community urgently needs a model with fast computing speed and excellent performance. Recently, a new neural network model has been proposed: breadth learning (BLS). Breadth learning does not adopt a structure that continuously stacks in depth like deep learning, but builds a model in a generalized way. BLS generates feature mapping nodes and enhancement nodes through input data, connects the feature mapping nodes and enhancement nodes to the output layer, and then uses the ridge regression algorithm to calculate the weight of the output layer. Therefore, compared with the deep learning model, BLS has the following advantages: the weights of the feature nodes and enhancement nodes of BLS are randomly generated and there is no iterative optimization process, so the training speed is fast; due to its good nonlinear mapping ability, BLS has excellent performance; BLS can expand the number of nodes or update parameters according to data, and can be easily applied to streaming data scenarios. The following is a brief introduction to BLS.

[0005] BLS is a network structure that only expands in width without expanding in depth. In BLS, the input data is linearly transformed and mapped into multiple groups of feature nodes, and then the multiple groups of feature nodes are further transformed, and the transformation results are nonlinearly activated to obtain enhanced nodes. Among them, both feature nodes and enhanced nodes are generated by randomly generating weight matrices and bias terms. After obtaining the feature nodes and enhanced nodes, the feature nodes and enhanced nodes are aggregated as the input A of the output layer, and the weight of the output layer is obtained through the ridge regression algorithm. In detail, the weight W of the output layer is obtained by calculating the pseudo-inverse of A and matrix multiplying the pseudo-inverse with the output of the training data.

[0006] Due to its excellent performance, BLS has attracted the attention of academia and industry since its inception, and has made significant progress both in theory and in application. Fu et al. applied BLS to fault diagnosis, converted the original vibration signal into feature nodes and enhanced nodes, and proposed an adaptive incremental learning strategy to quickly adjust the network structure. Chu et al. proposed a weighted BLS based on BLS to deal with noise and outliers in industrial processes. In short, it uses a weighted penalty factor to constrain the contribution of each sample to modeling, and assigns higher and lower weights to normal samples and abnormal samples respectively. Feng et al. combined width learning with fuzzy systems, replaced the feature nodes in BLS with a set of TS fuzzy subsystems, and determined the center of the Gaussian membership function and the number of fuzzy rules through the k-means algorithm. Liu et al. proposed a generalized learning system with a stacked structure: StackedBLS. StackedBLS regards each BLS as a separate BLS block, the output of the previous block as the input of the new block, the residual of the lower block as the expected output of the new block, and finally merges the output of all blocks and regards them as the true prediction result of the model. This design method is not only extended in a broad sense, but also needs to be stacked at a deep level. Guo et al. proposed a hierarchical generalized learning system HBLS with deep and sparse feature learning. HBLS sets multiple feature mapping layers to enhance the feature learning ability of BLS, and uses sparse autoencoders to reconstruct the features of each layer, eliminate redundant features and generate more sparse and compact features. Summary of the invention:

[0007] Although BLS has excellent performance, it still has some shortcomings: (1) Due to the random generation of feature nodes and enhancement nodes, some key information in the original data may be lost. (2) There is only one mapping layer and enhancement layer in BLS, which may not be enough to extract sufficient features in some cases.

[0008] To address these problems, the present invention proposes a new width learning structure. First, the original data is reconstructed using a sparse autoencoder to obtain a more sparse and compact feature representation. This process can be regarded as an unsupervised learning process. At the same time, due to the shallow structure and random feature mapping method of BLS, its feature learning ability may be insufficient in some cases. Therefore, inspired by HBLS, the present invention not only adopts a multi-layer feature mapping layer structure, but also sets multiple enhancement layers to further improve the performance of the model. In addition, by stacking multiple blocks, BLS is not only expanded in width, but also further expanded in depth. In order to verify the performance advantages of this model, experiments were conducted on the public datasets Cleveland, Strike, Quake, Pyrim and Basketball. The main contributions are as follows:

[0009] (1) In order to prevent the loss of key information of the original data during the random mapping process, the original data is merged into the input of the BLS output layer. At the same time, in order to prevent the excessive influence of the original data from causing overfitting of the model, the original data is reconstructed through a sparse autoencoder to generate a more sparse and compact feature representation of the original data.

[0010] (2) A multi-layer feature mapping layer and an enhanced layer structure DHBLS (Double-Hierarchical Broad Learning System) are used to enhance the model's feature extraction and nonlinear learning capabilities, extract more key features from the original data, and thus improve model performance. In order to further improve model performance, a DHBLS structure is regarded as a DHBLS block, and multiple DHBLS blocks are stacked to obtain StackedDHBLS (Stacked Double-Hierarchical Broad Learning System), and the final output of the model is obtained by residual connection.

[0011] (3) Experiments are conducted on five data sets, and the results show that the proposed method is superior to the basic model in performance. Description of the drawings:

[0012] Figure 1 It is the feature representation extraction module.

[0013] Figure 2 For internal breadth learning module - DHBLS.

[0014] Figure 3 A novel wide learning model with deep and flat structure - StackedDHBLS Specific implementation method:

[0015] In the present invention, we propose a new width learning structure. First, the original data is reconstructed using a sparse autoencoder (SAE) to obtain a more sparse and compact feature representation, and this sparse representation is used to make up for the information lost in the random process. This process can be regarded as an unsupervised learning process. At the same time, due to the shallow structure and random feature mapping method of BLS, its feature learning ability may be insufficient in some cases. Therefore, we not only adopt a multi-layer feature mapping layer structure, but also set up multiple enhancement layers to further improve the performance of the model. In addition, by stacking multiple BLS blocks, residual connections are established between multiple adjacent blocks, so BLS is not only expanded in width, but also further expanded in depth. Next, the four parts of extracting key data information, multi-layer BLS structure module-DHBLS, residual connection of multiple DHBLS blocks and prediction output will be explained respectively.

[0016] Step 1: Extract key data information

[0017] The feature mapping of BLS is a random process, so in some cases, key information in the data may be lost, so we need to find a way to make up for this information. Directly merging the training data with all hidden nodes is the simplest and most direct method, but this method may lead to overfitting of the model, so we propose a label-based sparse autoencoder to reconstruct the original data and then merge it with all hidden nodes to improve model performance. Figure 1 The process of extracting key information from raw data is given.

[0018] Sparse autoencoder is a network-based autoencoder model, whose goal is to reconstruct input data by learning sparse representation. Compared with traditional autoencoders, SAE introduces sparsity penalty terms to promote sparse representation, and more important features can often be obtained through SAE. The present invention uses alternating direction multiplier method (ADMM) as a sparse autoencoder. SAE not only serves each feature mapping layer, but also extracts feature representations that imply original key information from the original data to make up for the information lost in the random mapping process.

[0019] Assume that there is existing training data (X, Y), where Indicates input, Represents the output. During the model training process, the original data information is added to the input of the output layer. In order to prevent the model from overfitting, a sparse encoder is used to reconstruct the original data to extract a more sparse and compact feature representation. In ADMM, a feature representation β is obtained through the input x and output Y of the training data, so that A·β≈Y, where A·β represents the key information F of the data.

[0020] Step 2: Multi-layer BLS structure module - DHBLS

[0021] There is only one mapping layer and one enhancement layer in BLS, which may not be enough to extract sufficient features in some cases. Figure 2 As shown, in order to solve this problem, the present invention not only adopts a multi-layer and multi-group feature learning framework to extract high-level features, but also adopts multiple enhancement layers to enhance the nonlinear expression ability of the model, thereby further improving the model performance. For the first layer, the calculation formula is as follows:

[0022] Z n =φ(XW e +b e ) (1)

[0023] H m =ξ(Z n W m +b m ) (2)

[0024] The weight W and the bias term b are randomly generated weight matrices, φ and ξ represent activation functions, Z and H represent feature nodes and enhancement nodes respectively, and n groups of feature nodes and m groups of enhancement nodes can be obtained through feature mapping and enhancement mapping. The calculation method of the second layer and higher layers is slightly different. Assuming that the current module has K1 and K2 feature mapping layers and enhancement layers, each feature mapping layer and enhancement layer can be expressed as:

[0025]

[0026]

[0027] in Respectively represent all nodes of the i-th feature group and enhancement group in the k-th (k-1)-th layer. Since then, all mapping processes of the input data X are completed, and the input A of the output layer is obtained after multi-layer mapping, which is expressed as:

[0028]

[0029] Next, the sparsely processed feature F is merged with all nodes to obtain A x ([F[A]), and obtain the corresponding weights of the output layer through ridge regression. The objective function of DHBLS is basically the same as that of BLS, both of which solve the following Tikhonov regularization minimization problem:

[0030]

[0031] A x The pseudo-inverse of can be obtained by

[0032]

[0033] Therefore, the output layer weights can be calculated as

[0034]

[0035] So far, we have obtained a trained DHBLS block, which can be summarized as follows:

[0036]

[0037] in represents the prediction result of the DHBLS block, ψ(·) represents the hierarchical grouping feature mapping function, and ζ(·) represents the hierarchical grouping nonlinear enhancement function.

[0038] Step 3: Residual connection of multiple BLS blocks

[0039] The present invention further improves the fitting ability by connecting multiple DHBLS blocks through residuals. Figure 3 The overall framework is described. Assuming there are B DHBLS blocks in StackedDHBLS, the high-level DHBLS calculation process is slightly different, so it is explained here:

[0040]

[0041] in represents the prediction result of the bth DHBLS block, It represents the input of the bth DHBLS block, that is, the hidden node of the previous block is used as the input of the next DHBLS block. It is worth noting that the reconstructed feature F extracted from the input data is not added to the high-level DHBLS. This is because the bottom-level DHBLS block contributes the most to the model. Adding the key information F extracted from the original data to the DHBLS block is sufficient to make up for the original information lost due to randomness. If it is added to the high-level DHBLS block, it may lead to structural redundancy.

[0042] At the same time, in order to make full use of the output of each DHBLS block, residual connections are established between adjacent blocks, and the expected output of each block can be expressed as

[0043]

[0044] Therefore, the weight of each block is W out It can be obtained by solving the following proximal optimization problem:

[0045]

[0046] Step 4: Predict output

[0047] Through the above three parts, the prediction output of each module can be obtained. Since residual connections are made between each adjacent module, the final prediction output of the model is approximately the sum of the outputs of all DHBLS blocks:

[0048]

Claims

1. A novel width learning regression prediction method with deep and flat structure, which is divided into four parts: extracting key information of data, multi-layer width learning structure module, residual connection module and prediction output, characterized by: Extracting key data information includes the following three steps: (1) The user inputs the training data (X, Y) into the model, where Indicates input, Represents the output. After obtaining the input data, the alternating method multiplier method (ADMM) is used as the sparse autoencoder, X is used as the input of the sparse autoencoder, and Y is used as the target output of the sparse autoencoder (2) Solve the following optimization problems through ADMM: where β represents the sparse autoencoder solution. (3) A sparse representation β is found through a sparse autoencoder, such that A·β≈Y, where A·β represents the key information F of the data. The multi-layer width learning structure module consists of the following five steps: (4) Perform feature mapping on the input X of the training data, and map the original data through the randomly generated weight matrix W and the bias term b. The calculation formula can be expressed as: Z n =φ(XW ei +b ei ),i=1,2,...,n. Where W ei and the bias term b ei is a randomly generated weight matrix used to map the original features, Z n Represents the output of the feature mapping layer, Φ represents the activation function, and through feature mapping, n groups of feature nodes Zn=[Z1,Z2,...,Zn] can be obtained, and the sparse autoencoder can be used to encode W ei Adjustments are made to obtain more sparse and compact mapping features. (5) After the first layer mapping is completed, the mapping features are no longer handed over to the enhancement layer for nonlinear mapping, but the second layer and higher layer feature mapping are performed. Assuming that the current module has a total of K feature mapping layers, except for the first layer, each of the remaining feature mapping layers can be expressed as: Where Z i k (Z i k-1 ) represents all nodes of the i-th feature group in the k-th (k-1)-th layer. (6) After the multi-layer feature mapping is completed, the final feature node Z is obtained n , and the first-layer enhanced node H is obtained by nonlinear mapping by the enhanced module. The formula is as follows: H m =ξ(Z n W m +b m ) Where W m and b m is a randomly generated weight matrix, H m represents the output of the enhancement layer, ξ represents the activation function, and after the enhancement mapping, m groups of enhancement nodes Hn=[H1,H2,...,Hn] are obtained. (7) After obtaining the first layer of enhanced nodes, the feature nodes and enhanced nodes are no longer merged, but the second layer and higher layer enhanced mapping are performed. Assuming that the current module has a total of K enhanced layers, except for the first layer, each of the remaining enhanced layers can be expressed as: Among them, H i k (H i k-1 ) represent all nodes of the i-th enhancement group in the k-th (k-1)-th layer respectively. (8) The feature nodes and enhancement nodes are combined to obtain the input A of the output layer of the current module: [F|Z|H], and the corresponding weight of the output layer is obtained by ridge regression. Its objective function is to solve the Tikhonov regularization minimization problem: According to the formula Get the pseudo-inverse of A + Thus, the weight W of the output layer is obtained out =A + Y, so far we have a trained module, which can be summarized as: in represents the prediction result of the DHBLS block, ψ(·) represents the hierarchical grouping feature mapping function, and ζ(·) represents the hierarchical grouping nonlinear enhancement function. The residual connection module consists of the following two steps: (9) Assume that there are B modules in total, and establish residual connections between each adjacent module. Each module uses the input A of the output layer of the previous module. b-1 As the input of the bth module, the expected output is expressed as It is worth noting that, except for the bottom-level module, the A of the remaining modules does not include the extracted key information F, because the bottom-level module contributes the most to the model, and adding F to the module is sufficient to make up for the information lost due to random mapping. (10) Obtain the weight of each output layer by solving the proximal optimization problem: Prediction output: Through the above three parts, the prediction output of each module can be obtained. Since residual connections are made between each adjacent module, the final prediction output of the model is approximately the sum of the outputs of all DHBLS blocks:

Images