Neural network initialization method based on matrix decomposition

Abstract

The invention introduces a neural network initialization method based on matrix decomposition, and aims to optimize the training speed and accuracy of a neural network and solve the problems that the generalization of a traditional initialization method is limited and an activation function is limited. According to the method, sampling is carried out from Gaussian distribution to generate an initial weight matrix, eigenvalue decomposition is carried out on the matrix, the weight matrix is initialized by controlling the value taking mode of eigenvalues, and finally matrix reconstruction is carried out, so that the training performance of the network is improved, and reference is provided for understanding the influence of different initialization methods on the performance of the auto-encoder. Experimental results reveal the advantages of the initialization method in the auto-encoder model.

Description

Technical Field

[0001] The invention is used in the field of neural network initialization, in particular to a weight matrix initialization based on matrix decomposition. Background Art

[0002] As a key machine learning technology, deep learning has achieved remarkable success in many practical application fields such as computer vision, speech recognition, and games. However, as the complexity of neural networks increases, the problem of over-parameterization becomes increasingly prominent, that is, the number of network parameters far exceeds the number of training samples. In order to prevent overfitting and accelerate the training process, regularization techniques are widely used. Experiments have shown that regularization is crucial to improving model performance, among which parameter matrix initialization is a typical regularization method that has been successful in many machine learning applications.

[0003] The initialization strategy of neural networks is one of the research focuses. Correctly initializing network parameters (such as weights and biases) is crucial for effective learning during training. Improper initialization may cause the network to lose optimization ability or fall into local optimality. With the continuous development of neural network architecture and optimization technology, the initialization strategy also needs to be constantly updated. In 2010, Xavier initialization was proposed and became one of the earliest and widely used initialization techniques. This method initializes weights by a Gaussian distribution with zero mean and variance scaled based on the input number, ensuring that the gradient maintains a consistent proportion in each unit. Xavier initialization is effective for neurons with sigmoid and tanh activation functions, but its effectiveness has been challenged after the rise of ReLU activation function. He et al. found that Xavier initialization may cause gradient explosion or disappearance in ReLU networks, so they proposed an improved initialization method to stabilize the network by controlling the forward or reverse variance. Subsequently, methods such as Fixup initialization and LSUV initialization were proposed one after another to further optimize the initialization strategy of the weight matrix.

[0004] Although existing methods have made some progress in weight matrix initialization, they mainly focus on scaling the overall weight matrix and do not directly control the eigenvalues ​​or eigenvectors of the weight matrix. Miller et al. proposed a new regularization technique called "Eigeninit", which initializes network weights by generating random eigenvalues ​​from a known distribution, directly controlling the eigenvalues ​​to better guide the network learning process. Eigeninit significantly improves the convergence speed in Koopman autoencoders, but it was previously limited to Koopman linear operators and autoencoders.

[0005] The neural network initialization methods proposed above have been successful to a certain extent, but these methods also have problems such as limited generalization and limited activation functions. It is crucial to use reasonable initialization methods to optimize the neural network training speed, improve accuracy, and reduce losses. Summary of the invention

[0006] In order to overcome the problem that the neural network is prone to fall into the local optimal solution, solve the generalization problem of Eigeninit and speed up the training speed of the neural network, the technical solution of the present invention is:

[0007] A neural network initialization method based on matrix decomposition, including matrix initialization, matrix decomposition and matrix reconstruction.

[0008] The initialization matrix is ​​generated before the neural network training, starting from the element sampling of Gaussian distribution, and a matrix U0 is generated. The weight matrix of each layer randomly generates a matrix of corresponding dimension for subsequent initialization adjustment.

[0009] The matrix decomposition described above modifies the initialization of each layer of the autoencoder when initializing the weight matrix. Unlike the initialization of the weight matrix elements in other methods, we choose to initialize the eigenvalues ​​of the weight matrix. For the generated matrix U0, the matrix is ​​decomposed using the following formula:

[0010] U0=VΛV -1 (1)

[0011] where Λ is a diagonal matrix with eigenvalue λ j distributed along its main diagonal. V is a vector consisting of eigenvectors υ i The eigenvalues ​​of a matrix composed of are usually complex numbers, so it can also be written as In this regard, we apply the following method to modify the eigenvalues ​​of a diagonal matrix:

[0012] r j =|λ j |∈R (2)

[0013] That is, let the modulus of its eigenvalue be randomly generated from the specified distribution (Gaussian distribution, double Gaussian distribution, uniform distribution, small range uniform distribution, etc.), while ensuring that its phase θ j No change occurs, so we get a new diagonal array

[0014]

[0015] The matrix reconstruction is to obtain a new diagonal matrix Then, use the inverse operation of matrix decomposition Get the new weight matrix U.

[0016] Beneficial effects: The initialization method shows significant beneficial effects in the autoencoder network. By directly controlling the eigenvalues ​​and distribution of the weight matrix, it can effectively improve the convergence performance during training and better adjust the properties of the model. Its flexible parameter selection enables it to adapt to the initial eigenvalue size and phase of different network architectures and data sets, thereby promoting training convergence. This method provides an important reference for understanding the impact of different initialization methods on the performance of autoencoders, and preliminarily reveals its potential advantages in autoencoder models. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic diagram of the principle of the experimental automatic encoder of the present invention;

[0018] Figure 2 This is a schematic diagram of the experimental automatic encoder SAE of the present invention:

[0019] Figure 3 It is a schematic diagram of the experimental automatic encoder DAE of the present invention; DETAILED DESCRIPTION

[0020] The content of the present invention is further described below in conjunction with the accompanying drawings.

[0021] like Figure 1 Shown is an autoencoder, a neural network architecture used for dimensionality reduction, anomaly detection, and generative modeling. It consists of an encoder and a decoder. The encoder maps high-dimensional input data to a low-dimensional vector space, while the decoder maps the low-dimensional vector space back to a high-dimensional space to "reconstruct" the input data.

[0022] The reconstruction process includes obtaining the input, obtaining the input code through the encoding function, and then restoring the original input through the decoding function. Assume that the input is x∈R d :

[0023] h=f c (x) = S e (W e x+b e ) (4)

[0024] X r =f d (x) = S d (W d h+b d ) (5)

[0025] Encoding function f e and the decoding function f d The parameters W e and W d, the encoder and decoder weights are represented. The encoder and decoder biases are represented by b e and b d Representation. Usually a nonlinear function is used. The goal is to find a set of parameters during training to minimize the error.

[0026] like Figure 2 The figure shows that the Denoising Autoencoder (DAE) is a special type of autoencoder whose core concept is to train the autoencoder by adding noise to the input data, thereby learning more robust and meaningful representations. Specifically, DAE generates noisy data by adding noise distribution to the original data, and then feeds these noisy data into the encoder. The encoder compresses the noisy data into hidden variable representations, and then the decoder reconstructs the original data using these hidden variables to restore the clean data without noise. During the training process, the reconstruction loss between the original data and the reconstructed data is calculated using the mean square error (MSE), and the parameters of the encoder and decoder are updated through the back-propagation algorithm to minimize the reconstruction loss. In this way, the encoder is able to learn stable and useful features from the input data while filtering out the noise. Therefore, the hidden variable representation contains the main information of the input data instead of simply copying the input data. Throughout the training process, adding noise forces the encoder to develop a more robust representation capability, which ultimately enables the representation learned by the encoder to better decode and reconstruct the original data.

[0027] like Figure 3 The Stacked Autoencoder (SAE) shown in the figure adopts a hierarchical encoding learning method to extract multi-level representations of the input data by stacking multiple autoencoders. The process is mainly divided into three stages: first, the original input data is trained unsupervisedly using the first autoencoder to learn low-level features; second, the features learned by the previous layer of autoencoders are used as input to train subsequent autoencoders to extract higher-level feature representations; finally, after all autoencoders have learned to extract multi-level features, the entire model is fine-tuned end-to-end using a supervised learning method, the cost / loss function is minimized through the back-propagation algorithm, and labeled data is used to improve the model's parameter learning. Through these three stages, SAE is able to learn the hidden structure of the input data hierarchically, extract high-level representations with physical meaning, and optimize the relationship between features through fine-tuning, thereby improving classification and other tasks. The overall learning process of SAE is similar to that of the autoencoder

[0028] The specific implementation process of the present invention includes the following steps:

[0029] S1: Preliminary data preprocessing: First download the MNIST, Cifar100 and Cifar10 datasets, and divide the entire dataset into training set, validation set and test set in a ratio of 8:1:1.

[0030] S2: Implementation of initialization method: The present invention compares the initialization effects of Xavier, Kaiming, Fixup, LSUV and the initialization proposed in this paper on the autoencoder. First, the above five initialization methods are set on SAE and DAE, and the network parameters are adjusted to be consistent.

[0031] S3: Run the configured model in the same epoch to compare the effect of initialization, set the same threshold, and compare the time and loss of different methods to reach the threshold to evaluate the effectiveness of the initialization method.

[0032] Table 1 shows the initialization evaluation effect on DAE in the present invention:

[0033] Table 1 shows the DAE training results in the present invention.

[0034]

[0035] Table 2 shows the initialization evaluation effect on SAE in the present invention:

[0036] Table 2 shows the SAE training results in the present invention.

[0037]

[0038] In summary, the present invention provides a reference for understanding the impact of different initialization methods on the performance of the autoencoder. The experimental results reveal the advantages of the initialization method in the autoencoder model.

Claims

1. An initialization method based on matrix decomposition, characterized in that: First, generate a matrix from a Gaussian distribution, then perform matrix decomposition, regenerate the eigenvalues ​​of the diagonal matrix, and reconstruct the matrix through the inverse operation of matrix decomposition; The matrix decomposition uses a singular value decomposition method to decompose the original matrix into a matrix, a product of a diagonal matrix and a matrix transpose, and then randomly generates the modulus of its eigenvalue from a specific distribution, and ensures that its phase does not change; The reconstruction is to generate a new matrix of features and then use matrix multiplication to generate a new matrix as the weight matrix of each layer of the neural network.

2. According to the method described in claim 1, the specific implementation is to define a Gaussian distribution sampler with a mean of 0 and a standard deviation of std / n_units, where std is a preset standard deviation and n_units is the dimension of the matrix; use the sampler to generate a matrix Omega of size (n_units, n_units); perform eigenvalue decomposition on the matrix Omega to obtain eigenvalues ​​and eigenvectors; and return the matrix Omega.

3. The method according to claim 2, characterized in that The uniformly distributed eigenvalue initialization step comprises: Initialize the real and imaginary parts of the eigenvalues ​​to random numbers in the range of [-maxmin, maxmin], where maxmin is the preset maximum and minimum value.

4. The method according to claim 2, characterized in that: The step of initializing the eigenvalues ​​of the small-range uniform distribution includes: The real and imaginary parts of the eigenvalues ​​are initialized to random numbers in the range of [-0.05, 0.05] respectively.

5. The method according to claim 2, characterized in that: The Gaussian distribution eigenvalue initialization step comprises: The real and imaginary parts of the eigenvalues ​​are initialized to Gaussian distributed random numbers with mean 0 and standard deviation std, respectively.

6. The method according to claim 2, characterized in that The step of initializing the eigenvalues ​​of the double Gaussian distribution includes: The real and imaginary parts of the eigenvalues ​​are initialized to the sum of two Gaussian-distributed random numbers, one of which has a mean of 1 and the other has a mean of -1, and their standard deviations are both std.

7. The method according to claim 2, characterized in that The step of reconstructing the matrix comprises: Reconstruct a new matrix using matrix multiplication based on the reinitialized eigenvalues ​​and the original eigenvectors.

8. The method according to claim 2, characterized in that: The steps of generating the matrix Omega include: Generate a matrix of size (n_units, n_units) using the sampler and convert it to a real matrix.

9. The method according to claim 2, characterized in that: The eigenvalue decomposition step comprises: Use the np.linalg.eig function in the NumPy library to perform eigenvalue decomposition on the matrix Omega and obtain the eigenvalues ​​and eigenvectors.

10. The method according to claim 2, characterized in that The returned matrix is ​​a matrix that has been initialized by eigenvalues ​​or Gaussian initialization, and the matrix is ​​of floating point type.

Images