Method for generating hidden state in recurrent neural network for language processing

Abstract

The invention relates to the technical field of recurrent neural networks, in particular to a method for generating a hidden state in a recurrent neural network for language processing, and the method comprises the following steps: S1, inputting and embedding: converting discrete words into continuous vector representation, providing a computable semantic basis for a model, converting abstract meanings of the words into a numerical form, and storing the numerical form; the embedding layer is constructed through pre-training or random initialization, each word is mapped to a fixed dimension vector space, and pre-training embedding captures semantic association by utilizing a large-scale corpus statistical rule; the importance of each historical position is determined by comparing a query vector with key vectors generated by all historical hidden states, specifically, each historical hidden state can be converted into a key vector and a value vector, then the dot product score of the query vector and each historical hidden state is calculated, and the attention weight is obtained after scaling and normalization. The weights reflect the association strength of the current word and each historical position.

Description

Technical Field

[0001] The present invention relates to the technical field of recurrent neural networks, and in particular to a method for generating hidden states in a recurrent neural network for language processing. Background Art

[0002] In recurrent neural networks for language processing, standard RNN hidden state generation is a key process. Based on the characteristics of sequential data, it processes inputs in chronological order. At each time step, the network combines the current input information with the hidden state of the previous moment, and generates the current hidden state through a nonlinear transformation process. Initially, the hidden state is usually set to a specific initial value, such as an all-zero vector or a random vector. As the input sequence continues to advance, the hidden state will continue to update, gradually accumulating and fusing all previous input information, thereby capturing long-term dependencies in the sequence and providing effective feature representation for subsequent language processing tasks such as text generation and sentiment analysis.

[0003] The model's understanding of semantics is mainly based on the learning of word vectors and sequence information. For pronouns, which are highly flexible and context-dependent, its semantic understanding is often not accurate enough. The referential meaning of "this" depends entirely on the context, and the model may simply infer its reference based on the co-occurrence relationship of words or the local context. It is difficult to deeply understand the semantic logic and grammatical structure of the text to determine the exact referent.

[0004] Therefore, the present invention proposes a method for generating hidden states in a recurrent neural network for language processing. By comparing the query vector with the key vectors generated from all historical hidden states, the importance of each historical position is determined. Specifically, each historical hidden state is converted into a key vector and a value vector. The dot product score between the query vector and each is then calculated, and after scaling and normalization, attention weights are obtained. These weights reflect the strength of the association between the current word and each historical position. In the sentence "The government implemented policies to reduce carbon emissions. This initiative received widespread support.", when processing "This", the model will give a higher score to the key vector corresponding to "policies", thereby generating a larger attention weight. Summary of the Invention

[0005] Technical Problems Solved: The model's understanding of semantics is primarily based on learning word vectors and sequence information. For pronouns, which are highly flexible and context-dependent, its semantic understanding is often inaccurate. The referential meaning of "this" depends entirely on the context, and the model may simply infer its referent based on word co-occurrence or local context. It is difficult to deeply understand the semantic logic and grammatical structure of the text to determine the exact referent.

[0006] In response to the deficiencies of the prior art, the present invention provides a method for generating hidden states in a recurrent neural network for language processing, thereby solving the technical problems mentioned in the background technology.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0008] A method for generating hidden states in a recurrent neural network for language processing comprises the following steps:

[0009] S1. Input embedding: Discrete words are converted into continuous vector representations, providing a computable semantic foundation for the model and translating the abstract meaning of words into numerical form. The embedding layer is constructed through pre-training or random initialization, mapping each word to a fixed-dimensional vector space. Pre-trained embeddings capture semantic associations using the statistical laws of large-scale corpora; randomly initialized embeddings are gradually adjusted through task data training. The embedding vector dimension needs to balance expressiveness and computational cost. The original text is thus converted into a high-dimensional numerical sequence suitable for neural network processing.

[0010] S2. Generate the current query: Construct a "search focus" vector as the starting point of the attention mechanism. The query vector construction depends on the input information of the current time step or the hidden state of the previous moment. Through linear transformation, the input vector or hidden state is projected into a new vector space to form the query. This is implemented by a learnable weight matrix. The matrix parameters are optimized during training. The query vector generated contains the local features of the current vocabulary and may also inherit the accumulated information of the historical hidden state.

[0011] S3. Calculating Attention Weights: Calculating attention weights is the core step in dynamically filtering historical information. The model compares the current query vector with all historical position key vectors to quantify the importance of historical elements. The key vectors are generated by linearly transforming the historical hidden states. The attention scores are measured by dot product operations to measure similarity, scaled, and then converted to probability distributions using a normalization function. The calculation process does not rely on sequence order or distance, and directly establishes global associations.

[0012] S4. Generate context vector: The context vector is the output of the attention mechanism. It is a weighted fusion of historical information. The attention weight is multiplied by the corresponding value vector and summed. The value vector is obtained by linearly transforming the historical hidden state. The context vector is generated selectively and retains features that are highly relevant to the current query.

[0013] S5. Update hidden state: Hidden state update is the core operation of recurrent neural networks. It fuses the current input, historical state, and context information through nonlinear transformation. After the introduction of the attention mechanism, the context vector is used as an additional input. The three parts of information are linearly combined through independent weight matrices and then compressed to a reasonable range by the activation function.

[0014] S6. Iterative transfer: Iterative transfer is the core mechanism of recurrent neural networks for processing sequence data. The updated hidden state serves as the initial state of the next time step and continues to participate in subsequent attention calculations and state updates. For long sequence processing, computational efficiency can be optimized by limiting the length of the historical window or adopting a sparse attention strategy. Hidden state transfer forms a vertical flow of information, and the attention mechanism enhances cross-stride information interaction.

[0015] In one possible implementation, an embedding method needs to be selected before input embedding, using the word vectors trained with the corpus to provide semantic prior knowledge.

[0016] A method for generating hidden states in a recurrent neural network for language processing comprises the following steps:

[0017] S21: Based on the previous hidden state: q t =W q ·h t-1 , using the hidden state of the previous time step to generate queries and capture the progressive logic of the sequence;

[0018] S22: Based on current input: q t =W q ·x t , taking the current word embedding as the query, emphasizing the current input scenario;

[0019] S23:W q ∈R h*h is a learnable matrix.

[0020] A method for generating hidden states in a recurrent neural network for language processing comprises the following steps:

[0021] S31: key vector k i :k i =W k ·h i , mapping historical hidden states to key spaces for matching queries;

[0022] S32: value vector vi :v i =W v ·h i , mapping historical hidden states to value space for information aggregation;

[0023] S33:W k ,W v ∈R h*h Control the semantic projection direction of keys and values respectively;

[0024] S34: The formula for calculating the attention score is:

[0025]

[0026] S35: The formula normalized to probability distribution is:

[0027]

[0028] In one possible implementation, a comprehensive context vector is obtained by weighted summing the attention weights and the corresponding value vectors.

[0029] In one possible implementation, in addition to the current input and previous hidden state, a context vector is added to the hidden state update formula of the standard RNN, and the new hidden state fuses these three pieces of information through a nonlinear activation function.

[0030] In one possible implementation, the newly generated hidden state is passed to the next time step and participates in subsequent attention calculations as part of historical information.

[0031] Beneficial effects compared with existing technologies:

[0032] 1. In this solution, attention weights are normalized into probability distributions, allowing the model to automatically focus on key information points. For example, when processing the sentence "The government implemented policies to reduce carbon emissions. This initiative received widespread support.", if the current word is "this," the model might assign a weight of 0.9 to "policies" in the previous text, while assigning only 0.05 to other secondary information such as "government," thereby accurately locating the referent.

[0033] 2. In this solution, attention weights can be combined with the rule engine to improve model robustness. For example, if a grammatical rule requires that "this" refer to the subject of the preceding context first, the weight distribution can be adjusted through post-processing to force the model to follow specific logic. If "this" is incorrectly associated with secondary information, the weight is found to be abnormally high in irrelevant positions, and the model structure or training strategy can be adjusted accordingly. When the model makes a reference error, examining the attention weight distribution can help locate the problem.

[0034] 3. In this scheme, since the calculation of the attention mechanism does not rely on chain gradient propagation, the weight calculation of long-distance information is not affected by gradient vanishing. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention with reference to the accompanying drawings.

[0036] Figure 1 Schematic diagram of the method of the present invention. DETAILED DESCRIPTION

[0037] Preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the present invention can be implemented in various forms, so the present invention is not limited to the embodiments described below.

[0038] The technical solution in the embodiments of the present application is to solve the problems of the above-mentioned background technology, and the overall idea is as follows:

[0039] Example 1:

[0040] Please refer to Figure 1 As shown, this embodiment introduces a method for generating hidden states in a recurrent neural network for language processing, including the following steps:

[0041] S1. Input embedding: Converting discrete words into continuous vector representations to capture the semantic and grammatical information of words. Input embedding is the process of converting discrete vocabulary symbols into continuous vector representations. The core goal of this step is to provide a computable semantic foundation for the model and convert the abstract meaning of words into numerical form. The embedding layer is usually constructed through pre-training or random initialization. Each word is mapped to a vector space of fixed dimension. Pre-trained embedding uses statistical laws in large-scale corpora to capture the semantic associations between words. For example, synonyms are closer in vector space. Randomly initialized embedding is gradually adjusted through the training process of task data to be more suitable for the vocabulary distribution of specific fields. The dimension of the embedding vector needs to balance expressive power and computational cost. Higher dimensions can accommodate more semantic information, but will increase the number of model parameters and training time. Through this step, the original text is converted into a high-dimensional numerical sequence suitable for neural network processing, laying the foundation for subsequent context modeling.

[0042] S2. Generate the current query: Construct a "search focus" vector to guide the model to retrieve relevant content from historical information. Generating the query vector is the starting point of the attention mechanism. Its function is to provide the model with the current focus of attention. The construction of the query vector depends on the input information of the current time step or the hidden state of the previous moment. Through linear transformation operations, the input vector or hidden state is projected into a new vector space to form a query with a clear semantic direction. This transformation process is implemented through a learnable weight matrix. The matrix parameters are gradually optimized during training to adapt to the attention mode of different tasks. The generation of the query vector not only contains the local features of the current vocabulary, but may also inherit the accumulated information of the historical hidden state, thereby dynamically adjusting the attention direction. For example, when processing pronouns, the query will tend to carry the key features of the previous event in order to retrieve related referents in the historical context. The key to this step is to encode the semantic requirements of the current processing into a comparable vector form to provide a benchmark for subsequent attention calculations;

[0043] S3. Calculate attention weights: Quantify the strength of the association between the current query and each historical position, and dynamically filter key information. Calculating attention weights is the core link in dynamically filtering historical information. The model quantifies the importance of each historical element to the current processing by comparing the current query vector with the key vectors of all historical positions. The key vector is generated by linear transformation of the historical hidden state. Its function is to convert the abstract representation of the hidden state into a semantic feature that can match the query. The attention score measures the similarity between the query and the key through dot product operation. A high score indicates a strong correlation. To avoid numerical instability, the score needs to be scaled and then converted into a probability distribution through a normalization function. This distribution reflects the model's attention intensity to historical information. Information at high-weight positions will be retained first. During the calculation process, the model does not need to rely on sequence order or distance, but directly establishes global associations, thus breaking through the limitations of traditional recursive structures. The dynamic nature of attention weights enables the model to flexibly adapt to different contexts. For example, when dealing with polysemous words, the semantic focus in different contexts can be distinguished through weight distribution.

[0044] S4. Generate context vector: Integrate historical information to form a global context representation related to the current query. The context vector is the output of the attention mechanism. Its essence is a weighted fusion of historical information. By multiplying the attention weight with the corresponding value vector and summing them, the model integrates scattered historical features into a compact global representation. The value vector is also obtained by linear transformation of the historical hidden state, which encodes the detailed information of each historical position. The generation process of the context vector is selective, retaining only features that are highly relevant to the current query and suppressing irrelevant or redundant content. This focusing mechanism enables the model to extract key fragments from long sequences, such as accurately locking the previous event in the reference resolution task. The context vector not only contains semantic content, but may also imply grammatical structure or logical relations, providing rich supplementary information for subsequent hidden state updates. Through this step, the model realizes active perception of the global context, rather than passively relying on local information transmitted recursively;

[0045] S5. Update the hidden state: Fusion of the current input, historical hidden state, and global context generates a new hidden state. Hidden state update is the core operation of recurrent neural networks. It fuses the current input, historical state, and contextual information through nonlinear transformations. Traditional RNNs rely solely on the simple superposition of the current input and the previous hidden state. After the introduction of the attention mechanism, the context vector as an additional input significantly enhances the information capacity. During the update process, the three parts of information are linearly combined through independent weight matrices and then compressed to a reasonable range through the activation function. The choice of activation function (such as the hyperbolic tangent function) ensures the smoothness and stability of the output while retaining nonlinear features. The updated hidden state not only reflects the local context of the current moment, but also carries global key information filtered by attention. This fusion mechanism enables the model to simultaneously capture the temporal characteristics of the sequence and the semantic association across strides. For example, when processing causal logic, the hidden state can simultaneously encode the current result and the distant cause.

[0046] S6. Iterative transfer: The new hidden state is transferred to the subsequent time step, and the deep representation of the sequence is gradually constructed. Iterative transfer is the core mechanism of recurrent neural networks for processing sequence data. The updated hidden state will be used as the initial state of the next time step and continue to participate in subsequent attention calculations and state updates. As the time step progresses, the model gradually builds up a contextual representation covering the entire sequence. Each hidden state accumulates the essence of historical information. For long sequence processing, the computational efficiency can be optimized by limiting the length of the historical window or adopting a sparse attention strategy, but the ability to capture some long-distance dependencies may be sacrificed. The transmission path of the hidden state forms a vertical flow of information, and the attention mechanism enhances the information interaction across strides through horizontal associations. This vertical and horizontal combination of structures enables the model to maintain the temporal nature of sequence processing while breaking through the constraints of local vision and achieving a truly global understanding. Ultimately, the hidden state sequence generated by the iterative process can be used as a high-level semantic representation for downstream tasks (such as classification or generation).

[0047] Before input embedding, you need to select an embedding method and use the word vectors trained by the corpus to provide semantic prior knowledge.

[0048] A method for generating hidden states in a recurrent neural network for language processing comprises the following steps:

[0049] S21: Based on the previous hidden state: q t =W q ·h t-1 , using the hidden state of the previous time step to generate queries and capture the progressive logic of the sequence;

[0050] S22: Based on current input: q t =W q ·x t, taking the current word embedding as the query, emphasizing the current input scenario;

[0051] S23:W q ∈R h*h is a learnable matrix that adjusts the query generation direction through training.

[0052] A method for generating hidden states in a recurrent neural network for language processing, comprising the following steps: generating a key vector and a value vector when calculating the attention weight

[0053] S31: key vector k i :k i =W k ·h i , mapping historical hidden states to key spaces for matching queries;

[0054] S32: value vector v i :v i =W v ·h i , mapping historical hidden states to value space for information aggregation;

[0055] S33:W k ,W v ∈R h*h Control the semantic projection direction of keys and values respectively;

[0056] S34: The formula for calculating the attention score is:

[0057]

[0058] Scaling Factor Prevent the dot product result from being too large and causing Softmax gradient saturation;

[0059] S35: The formula normalized to probability distribution is:

[0060]

[0061] By weighted summing the attention weights and the corresponding value vectors, a comprehensive context vector is obtained. This vector concentrates the historical information most relevant to the current word. In the hidden state update formula of the standard RNN, in addition to the current input and the previous hidden state, the context vector is also added. The new hidden state fuses these three parts of information through a nonlinear activation function. This fusion mechanism enables the hidden state to not only contain local context, but also dynamically introduce global key information. The newly generated hidden state will be passed to the next time step and participate in subsequent attention calculations as part of historical information. As the sequence processing progresses, the model gradually accumulates richer context representations. When processing long texts, the computational efficiency can be optimized by truncating the window or storing the historical hidden state in blocks, while maintaining access to key information.

[0062] Working principle:

[0063] Input statement:

[0064] "The government implemented policy stored carbon emissions. This initiative received wide spread support."

[0065] Each word is converted into a continuous vector through an embedding layer, capturing semantic and grammatical features, such as:

[0066] “government” → a vector encoding semantics such as organization and power.

[0067] “policies” → vectors encoding semantics such as rules and plans.

[0068] “carbonemissions” → A vector encoding topics such as environmental protection and pollution.

[0069] The embedding matrix maps words into high-dimensional vectors, providing numerical input for subsequent calculations.

[0070] When processing the second sentence "This", a query vector is generated to locate the referent. If based on the previous hidden state (the hidden state at the end of the previous sentence), the query vector q t Encode the overall information of the previous text.

[0071] Generate key and value:

[0072] k i =W k ·h i , v i =W v ·h i

[0073] Previous hidden state: such as the hidden states h1, h2, and h3 corresponding to "government", "policies", and "carbon emissions" in the first sentence.

[0074] Key vector: Converts the hidden state into semantic features that can match the query. For example, the key vector k2 of “policies” encodes the “policy content”.

[0075] Value vector: retains detailed information of the hidden state, such as the value vector v2 of "policies" contains the specific description of the emission reduction policy.

[0076] Calculate the attention score:

[0077]

[0078] Query for matches of "This" with a key:

[0079] The k2 dot product score with the key “policies” is high (strong semantic correlation).

[0080] The key k1 with “government” has a low score (weak semantic correlation).

[0081] Normalized weights:

[0082]

[0083] "policies" weight a t,2 ≈0.9, and the weights of other words are close to 0.

[0084] Generate context vector:

[0085] The value vector v2 of “policies” dominates the context vector due to its high weight, while the value vectors of other words (such as “government”) are suppressed due to their low weights. The context vector focuses on representing the core information of “implementing emission reduction policies”.

[0086] Update the hidden state:

[0087] The new latent state encodes the complete logic of “This initiative points to policies”.

[0088] Iterative pass:

[0089] The new hidden state is passed to the subsequent time step, and information is continuously accumulated.

[0090] When processing "received":

[0091] The hidden state already contains the association “This→policies”, ensuring that the action “received support” is correctly bound to the policy.

[0092] Global consistency:

[0093] The model gradually builds coherent semantics across sentences and ultimately understands the core logic of the entire paragraph: "Policy implementation has received support."

[0094] Finally, it should be noted that the above embodiments are merely examples for the purpose of illustrating the present invention and are not intended to limit the embodiments. Those skilled in the art will readily appreciate that other variations or modifications based on the above description are possible. It is not necessary and impossible to provide an exhaustive list of all embodiments. However, obvious variations or modifications arising therefrom remain within the scope of protection of the present invention.

Claims

1. A method for generating hidden states in a recurrent neural network for language processing, characterized in that: The following steps are involved: S1. Input embedding: Discrete words are converted into continuous vector representations, providing a computable semantic foundation for the model and translating the abstract meaning of words into numerical form. The embedding layer is constructed through pre-training or random initialization, mapping each word to a fixed-dimensional vector space. Pre-trained embeddings capture semantic associations using the statistical laws of large-scale corpora; randomly initialized embeddings are gradually adjusted through task data training. The embedding vector dimension needs to balance expressiveness and computational cost. The original text is thus converted into a high-dimensional numerical sequence suitable for neural network processing. S2. Generate the current query: Construct a "search focus" vector as the starting point of the attention mechanism. The query vector construction depends on the input information of the current time step or the hidden state of the previous moment. Through linear transformation, the input vector or hidden state is projected into a new vector space to form the query. This is implemented by a learnable weight matrix. The matrix parameters are optimized during training. The generated query vector contains the local features of the current vocabulary and may also inherit the accumulated information of the historical hidden state. S3. Calculating Attention Weights: Calculating attention weights is the core step in dynamically filtering historical information. The model compares the current query vector with all historical position key vectors to quantify the importance of historical elements. The key vectors are generated by linearly transforming the historical hidden states. The attention scores are measured by dot product operations to measure similarity, scaled, and then converted to probability distributions using a normalization function. The calculation process does not rely on sequence order or distance, and directly establishes global associations. S4. Generate context vector: The context vector is the output of the attention mechanism. It is a weighted fusion of historical information. The attention weight is multiplied by the corresponding value vector and summed. The value vector is obtained by linearly transforming the historical hidden state. The context vector is generated selectively and retains features that are highly relevant to the current query. S5. Update hidden state: Hidden state update is the core operation of recurrent neural networks. It fuses the current input, historical state, and context information through nonlinear transformation. After the introduction of the attention mechanism, the context vector is used as an additional input. The three parts of information are linearly combined through independent weight matrices and then compressed to a reasonable range through the activation function. S6. Iterative transfer: Iterative transfer is the core mechanism of recurrent neural networks for processing sequence data. The updated hidden state serves as the initial state of the next time step and continues to participate in subsequent attention calculations and state updates. For long sequence processing, computational efficiency can be optimized by limiting the length of the historical window or adopting a sparse attention strategy. Hidden state transfer forms a vertical flow of information, and the attention mechanism enhances cross-stride information interaction.

2. The method for generating hidden states in a recurrent neural network for language processing according to claim 1, wherein: Before input embedding, you need to select an embedding method and use the word vectors trained by the corpus to provide semantic prior knowledge.

3. The method for generating hidden states in a recurrent neural network for language processing according to claim 1, wherein: The following steps are involved: S21: Based on the previous hidden state: q t =W q ·h t-1 , using the hidden state of the previous time step to generate queries and capture the progressive logic of the sequence; S22: Based on current input: q t =W q ·x t , taking the current word embedding as the query, emphasizing the current input scenario; S23:W q ∈R h*h is a learnable matrix.

4. The method for generating hidden states in a recurrent neural network for language processing according to claim 1, wherein: The following steps are involved: S31: key vector k i :k i =W k ·h i , mapping historical hidden states to key spaces for matching queries; S32: value vector v i :v i =W v ·h i , mapping historical hidden states to value space for information aggregation; S33:W k ,W v ∈R h*h Control the semantic projection direction of keys and values respectively; S34: The formula for calculating the attention score is: S35: The formula normalized to probability distribution is:

5. The method for generating hidden states in a recurrent neural network for language processing according to claim 1, wherein: A comprehensive context vector is obtained by weighted summing the attention weights and the corresponding value vectors.

6. The method for generating hidden states in a recurrent neural network for language processing according to claim 1, wherein: In the hidden state update formula of the standard RNN, in addition to the current input and the previous hidden state, a context vector is also added. The new hidden state fuses these three parts of information through a nonlinear activation function.

7. The method for generating hidden states in a recurrent neural network for language processing according to claim 1, wherein: The newly generated hidden state will be passed to the next time step and participate in subsequent attention calculations as part of historical information.

Images