A multi-label learning method of recurrent neural network based on GRU
Through the multi-label learning method of recursive neural network based on GRU, the problem of difficulty in learning sample features and gradient disappearance in the prior art is solved, and higher multi-label classification accuracy and long-term memory ability are achieved.
Patent Information
- Application Number
- CN201810112481.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2018-02-05
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2038-02-05
AI Technical Summary
Existing neural network multi-label classification is difficult to effectively learn the basic characteristics of samples, and gradient vanishing problems are prone to occur during backpropagation, resulting in the inability to effectively process long-term associated information.
The multi-label learning method of recursive neural network based on GRU is adopted to effectively learn the characteristics of sequence data through the GRU structure, and obtain the multi-label vector of the sample, simplifying the model structure and training process, and reducing the risk of gradient disappearance.
It improves the accuracy of multi-label classification, can effectively perform long-term memory, and avoids gradient vanishing problems during backpropagation, and obtains more accurate feature representations.
Smart Images

Figure CN108319980B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of recursive neural network multi-label learning, and in particular to a GRU-based recursive neural network multi-label learning method. Background Art
[0002] In the field of machine learning, multi-label classification plays an important role in classification problems. The traditional label classification problem is to learn a single label of a sample from a label set. Such a problem is called a binary classification problem (or text and network data filtering problem). In the multi-label classification problem, each sample has a set of related labels. In recent years, many methods have emerged to solve the multi-label classification problem. The methods are basically divided into two types. One is to convert the problem into a traditional single-label classification. This method learns each label in the label set separately through multiple binary classifiers. The second is to adjust the existing algorithm to adapt to multi-label classification.
[0003] For modeling of sequence data, models based on neural networks have achieved excellent results in broadcast speech recognition, language models, and video classification. Most of these models belong to two types of neural networks, feedforward neural networks and recurrent neural networks. Neural networks with recursive structures can represent moment information step by step. The recursive nature of the recursive neural network (RNN) is that the state of its hidden layer at a certain moment depends on the output state at the previous moment. This cycle enables the model to store complex information of any length of time. The hidden layer state can be regarded as the memory unit of the model. Its structure and expansion are as follows: Figure 1 shown.
[0004] The traditional RNN recursive hidden layer network module is relatively simple, such as Figure 2 As shown. In theory, RNN can learn sequence information of infinite length. However, practice has shown that the larger the time interval, the weaker the RNN's learning ability will be. And this loop structure is difficult to train because it is prone to gradient vanishing and gradient explosion. Mikolov prevents gradient explosion by gradient clipping, which is simple and feasible. This allows these models to be trained using gradient descent and BPTT (Back-propagation Through Time) algorithms on large data sets. Since their gradients may decay exponentially when backpropagating along time, recurrent neural networks still have the problem of gradient vanishing. This allows RNN to only process short-distance information and ignore long-term associated information.
[0005] In order to solve the gradient vanishing problem, many structures have been proposed. Among them is the long short-term memory unit (LSTM) recurrent neural network. It performs well in handwriting recognition and phoneme classification problems. LSTM has a carefully designed structure that contains many gates. These gates control the amount of information flowing to the hidden state. This allows the neural network to remember information for a long time. However, the LSTM network structure is more complex, which will take more training time and is prone to gradient vanishing during backpropagation. To solve this problem, a GRU structure with a simpler structure was proposed. This structure is easier to implement and simpler to train than LSTM. Its structure is as follows Figure 3 shown.
[0006] The existing neural network multi-label classification cannot effectively learn the basic features of the sample, the implementation structure is complex, and the gradient vanishing problem is prone to occur during back propagation. Based on this, the present invention provides a GRU-based recursive neural network multi-label learning method, which can fully utilize RNN to obtain effective feature representation of the sample and improve the accuracy of multi-label classification. The GRU structure can effectively perform long-term memory, it is simpler than the LSTM structure, and the gradient vanishing problem is not prone to occur during back propagation. Summary of the invention
[0007] In view of the defects of the prior art, the present invention provides a multi-label learning method of a recursive neural network based on GRU, which effectively learns the features of sequence data through a recursive neural network, obtains the representation of sample features, and learns the multi-label vector of the sample based on the obtained representation. The GRU structure of RNN is easier to implement and simpler to train than LSTM, and can remember the features of the previous moment for a long time, so that the obtained feature representation is more accurate.
[0008] To achieve the above object, the present invention provides a GRU-based recursive neural network multi-label learning method, the method comprising the following steps:
[0009] Step S1: Initialize system parameters θ = (W, U, B), where W represents the weight matrix between neural network layers; U represents the weight matrix when data is input into the neural network; B represents the bias matrix between neural network layers;
[0010] Step S2: Input sample Calculate the hidden state h of the RNN output at each moment t , where the sample y i is a sample x i The multi-label vector of
[0011] Step S3: Calculate the context vector h TAnd the output layer output z i ;
[0012] Step S4: Calculate the predicted output Calculate the loss L i , determine the objective function J;
[0013] Step S5: solving the gradient of θ=(W, U, B) according to the gradient descent method and the BPTT algorithm;
[0014] Step S6: Determine the learning rate η and update the weight gradient W = W-η·δ W , where δ W Represents the partial derivative of the weight matrix obtained by the gradient descent method during the back propagation of the neural network;
[0015] Step S7: Determine whether the neural network has reached stability. If so, execute step S8; if not, return to step S2 and iteratively update the model parameters;
[0016] Step S8: Output the optimized model.
[0017] Wherein, the step S2 further comprises: i Do the normalization so that its value is in [0, 1], and then In As the input value of RNN at time t, it is input into the hidden layer to obtain the hidden state h t .
[0018] Wherein, the step S3 further comprises: Each component in is input into the hidden layer as the T input value of RNN at T time points, and a context vector h is obtained. T ,in,
[0019] In the above formula, the symbol Represents the dot product operation between matrices.
[0020] Wherein, the step S4 further comprises: using the context vector h T Calculate sample x i Multi-label vector Among them, the joint probability of the i-th sample label pair is:
[0021]
[0022] L i for: The objective function J is determined as:
[0023]
[0024] in express Assume that the output of the jth neuron in the output layer is j=1,...,C, the output of the softmax layer is The actual output is k=1,...,C, The value of is 0 or 1, and λ represents the scale coefficient.
[0025] Wherein, the step S6 further includes determining the learning rate by the following method: η = Ae -λn , where n is the number of iterations in the network training process, 1≤A≤50, 0.0001≤λ≤0.001; or, η(k)=e -λ (k-1), where 0.0001≤λ≤0.001 and k is the number of iterations.
[0026] Wherein, the step S8 further includes: when the determined objective function J is minimized, outputting the optimal model.
[0027] The present invention provides a GRU-based recursive neural network multi-label learning method, which can fully utilize RNN to obtain effective feature representation of samples and improve the accuracy of multi-label classification. The GRU structure can effectively perform long-term memory. It is simpler than the LSTM structure and is not prone to the gradient disappearance problem during back propagation.
[0028] The features and advantages of the present invention will become clear through reference to the following drawings and detailed description of specific embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 is a schematic diagram of a recursive neural network deployed in the prior art;
[0030] Figure 2 It is a schematic diagram of a recursive module in a standard RNN in the prior art;
[0031] Figure 3 It is a schematic diagram of the structure of the gate recursive unit GRU in the prior art;
[0032] Figure 4 Schematic diagram of the GRU-based RNN multi-label classifier in the present invention;
[0033] Figure 5 It is a schematic diagram of the model parameter updating process of the GRU-based recursive neural network multi-label learning method of the present invention. DETAILED DESCRIPTION
[0034] In order to make the technical solution of the present invention clearer and more understandable, it will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0035] Figure 4 is a schematic diagram of a GRU-based RNN multi-label classifier in the present invention, through which the context vector h can be obtained T , and then output through the softmax layer Then use the multi-vector label y i Construct the loss function L for the i-th sample label pair i The specific implementation process is as follows:
[0036] Assume that the sample label pair Contains N training examples, where y i is a sample x i The multi-label vector of i=1,…,N. We take the sample x i Do the normalization so that its value is in [0, 1]. First, i Zero mean, and then use variance standardization, as follows:
[0037]
[0038] where μ and σ represent X={x1,…,x N}, and ε is the scale coefficient.
[0039] we will As the input of RNN at T moments, it finally outputs a context vector h T , which contains all the information before time T. The hidden layer adopts the GRU structure, and the propagation formula of each gate is as follows:
[0040]
[0041] Among them, the symbol Represents the dot product operation between matrices.
[0042] Finally, the context vector h T Prediction example x i Multi-label vector The joint probability of the i-th sample label pair is
[0043]
[0044] in express Assume that the output of the jth neuron in the output layer is j=1,…,C, the output of the softmax layer is The actual output is k=1,…,C. Then, the loss function of the i-th sample label pair is defined as the form of log likelihood cost function, which is expressed as follows
[0045]
[0046] in The value of is 0 or 1. The objective function of the model is defined as
[0047]
[0048] The first term on the right side of the above equation is the sample label pair The total loss of is , and the second term is the regularization term, where θ is the model parameter, including the weight matrix w, u, v and the bias vector B, and λ represents the scale coefficient.
[0049] Figure 5 The figure is a schematic diagram of the model parameter update process of the GRU-based recursive neural network multi-label learning method of the present invention. In this method, the model parameters are solved according to the objective function through the gradient descent method and the BPTT algorithm. The specific steps are as follows:
[0050] Step S1: Initialize system parameters θ = (W, U, B), where W represents the weight matrix between neural network layers; U represents the weight matrix when data is input into the neural network; B represents the bias matrix between neural network layers;
[0051] In this step, θ=(W,U,B) means that the parameters in the weight matrix θ include W, U, B, and the initialization system parameters θ=(W,U,B) refers to the initial values of input W, U, B. The initial values of W, U, B can be set in advance according to actual conditions.
[0052] Step S2: Input sample Calculate the hidden state h of the RNN output at each moment t , where the sample y i is a sample x i The multi-label vector of
[0053] In this step, the sample x i Do the normalization so that its value is in [0, 1], and then In As the input value of RNN at time t, it is input into the hidden layer to obtain the hidden state h t .
[0054] Step S3: Calculate the context vector h T And the output layer output z i ;
[0055] In this step, Each component in is input into the hidden layer as the T input value of RNN at T time points, and a context vector h is obtained. T ,in, In the above formula, the symbol Represents the dot product operation between matrices.
[0056] Step S4: Calculate the predicted output Calculate the loss L i , determine the objective function J;
[0057] In this step, the context vector h is used T Calculate sample x i Multi-label vector Among them, the joint probability of the i-th sample label pair is:
[0058]
[0059] L i for: The objective function J is determined as:
[0060]
[0061] in express Assume that the output of the jth neuron in the output layer is j=1,…,C, the output of the softmax layer is The actual output is k=1,…,C, The value of is 0 or 1, and λ represents the scale coefficient.
[0062] Step S5: solving the gradient of θ=(W, U, B) according to the gradient descent method and the BPTT algorithm;
[0063] In this step, when solving the model parameters of the machine learning algorithm, that is, the unconstrained optimization problem, gradient descent is one of the most commonly used methods. In the machine learning algorithm, when minimizing the loss function, the gradient descent method can be used to iterate step by step to obtain the minimized loss function. The algorithm of the gradient descent method can be represented by two methods: algebraic method and matrix method (also known as vector method). The algebraic method is easier to understand, and the matrix method is more concise. The gradient descent method includes batch gradient descent method, stochastic gradient descent method and mini-batch gradient descent method. Batch gradient descent method is the most commonly used form of gradient descent method. The specific method is to use all samples to update when updating parameters. The commonly used linear regression gradient descent algorithm is batch gradient descent method. For training speed, the stochastic gradient descent method uses only one sample to iterate each time, so the training speed is very fast, but the stochastic gradient descent method uses only one sample to determine the gradient direction, so the solution is likely not to be optimal. As for the convergence speed, since the stochastic gradient descent method iterates one sample at a time, the iteration direction changes greatly and cannot converge to the local optimal solution quickly. The update formula of the stochastic gradient descent method is: The mini-batch gradient descent method is a compromise between the batch gradient descent method and the stochastic gradient descent method. The present invention can use the gradient descent method described above to solve the gradient of θ=(W, U, B).
[0064] The BPTT algorithm is a commonly used back propagation over time algorithm in neural networks. An exemplary algorithm is shown below:
[0065] 1:for t from T downto 1 do
[0066] 2: do t ←g′(o t )·dL(z t ;y t ) / dz t
[0067] 3:db o ←db o +do t
[0068] 4:
[0069] 5:
[0070] 6:dz t ←e′(zt )·dh t
[0071] 7:
[0072] 8:db h ←db h +dz t
[0073] 9:
[0074] 10:
[0075] 11: end for
[0076] 12: Return dθ = [dW hv , dW hh ,dW oh ,db h ,db o ,dh0].
[0077] With the data obtained in steps S1-S4, the gradient of θ=(W,U,B) can be solved according to the gradient descent method and BPTT algorithm described above.
[0078] Step S6: Determine the learning rate η and update the weight gradient W = W-η·δ W , where δ W Represents the partial derivative of the weight matrix obtained by the gradient descent method during the back propagation of the neural network;
[0079] In this step, the learning rate η is generally determined manually and adjusted according to the learning effect of the neural network. It is generally adjusted according to the learning error of the neural network. As the error gradually decreases, the learning rate decreases accordingly. For example, the next learning rate can be one tenth of the previous one.
[0080] Specifically, the learning rate can also be determined by the following formula: η = Ae -λn , where n is the number of iterations in the network training process, 1≤A≤50, 0.0001≤λ≤0.001;
[0081] Or, η(k) = e -λ (k-1), where 0.0001≤λ≤0.001 and k is the number of iterations.
[0082] Step S7: Determine whether the neural network has reached stability. If so, execute step S8; if not, return to step S2 and iteratively update the model parameters;
[0083] In this step, the stability of the neural network is generally determined by the overall error between the output of the neural network and the true label. As the number of training epochs increases (epoch means using a complete training data set once), when the error curve finally stabilizes and the error value is less than a given threshold, the neural network learning is considered to be effective.
[0084] Step S8: Output the optimized model.
[0085] In this step, when the determined objective function J is minimized, the optimal model is output.
[0086] The present invention provides a GRU-based recursive neural network multi-label learning method, which can fully utilize RNN to obtain effective feature representation of samples and improve the accuracy of multi-label classification. The GRU structure can effectively perform long-term memory. It is simpler than the LSTM structure and is not prone to the gradient disappearance problem during back propagation.
[0087] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. All equivalent structural changes made by using the contents of the present invention specification and drawings under the concept of the present invention, or directly / indirectly applied in other related technical fields are included in the patent protection scope of the present invention.
Claims
1. A GRU-based recursive neural network multi-label learning method for video classification, the multi-label learning method is applied to the field of machine learning and is performed by a GRU-based RNN multi-label classifier, characterized in that: The GRU-based RNN multi-label classifier includes a hidden layer and a softmax layer. Each component in is input into the hidden layer as the T input value of RNN at T time points, and a context vector h is obtained. T , the softmax layer uses the context vector h T Calculate sample x i Multi-label vector Finally, calculate the loss function L of the i-th sample label pair i ; Wherein, the method comprises the following steps: Step S1: Initialize system parameters θ = (W, U, B), where W represents the weight matrix between neural network layers; U represents the weight matrix when data is input into the neural network; B represents the bias matrix between neural network layers; Step S2: Input sample Calculate the hidden state h of the RNN output at each moment t , where N is the number of samples, y i is a sample x i A multi-label vector of Step S3: Calculate the context vector h T And the output layer output z i ; Step S4: Calculate the predicted output Calculate the loss L i , determine the objective function J Step S5: solving the gradient of θ=(W, U, B) according to the gradient descent method and the BPTT algorithm; Step S6: Determine the learning rate η and update the weight gradient W = W-η·δ W , where δ W Represents the partial derivative of the weight matrix obtained by the gradient descent method during the back propagation of the neural network; Step S7: Determine whether the neural network has reached stability. If so, execute step S8; if not, return to step S2 and iteratively update the model parameters; Step S8: Output the optimized model.
2. The method according to claim 1, characterized in that The step S2 further comprises: i Do the normalization so that its value is in [0, 1], and then In As the input value of RNN at time t, it is input into the hidden layer to obtain the hidden state h t .
3. The method according to claim 2, characterized in that The step S3 further comprises: Each component in is input into the hidden layer as the T input value of RNN at T time points, and a context vector h is obtained. T ,in, In the above formula, the symbol represents the dot product operation between matrices, and σ represents the standard deviation of the samples.
4. The method according to claim 1, characterized in that: The step S4 further comprises: using the context vector h T Calculate sample x i Multi-label vector Among them, the joint probability of the i-th sample label pair is: L i for: The objective function J is determined as: in express Assume that the output of the jth neuron in the output layer is The output of the softmax layer is The actual output is The value of is 0 or 1, and λ represents the scale coefficient.
5. The method according to claim 1, characterized in that The step S6 further includes determining the learning rate by the following method: η=Ae -λn , where n is the number of iterations in the network training process, 1≤A≤50, 0.0001≤λ≤0.001; or, η(k)=e -λ (k-1), where 0.0001≤λ≤0.001 and k is the number of iterations.
6. The method according to claim 1, characterized in that The step S8 further includes: when the determined objective function J is minimized, outputting the optimal model.
