A method for generating hidden states in recurrent neural networks for language processing

Abstract

The application relates to the technical field of recurrent neural networks, in particular to a method for generating hidden states in a recurrent neural network for language processing, which comprises the following steps: S1, input embedding: converting discrete words into continuous vector representations, providing a computable semantic basis for the model, converting the abstract meaning of the words into numerical form, constructing the embedding layer through pre-training or random initialization, mapping each word to a fixed-dimensional vector space, and capturing semantic correlation by using the statistical rules of a large-scale corpus; by comparing the query vector with all the key vectors generated by the historical hidden states, the importance of each historical position is determined; specifically, each historical hidden state is converted into a key vector and a value vector, then the dot product score of the query vector and each is calculated, and the attention weight is obtained after scaling and normalization. The weights reflect the correlation strength between the current word and each historical position.

Description

Technical Field

[0001] This invention relates to the field of recurrent neural network technology, and in particular to a method for generating hidden states in a recurrent neural network for language processing. Background Technology

[0002] In recurrent neural networks (RNNs) for language processing, hidden state generation in standard RNNs is a crucial process. Based on the characteristics of sequential data, it processes inputs sequentially over time. At each time step, the network combines the current input information with the hidden state from the previous time step, generating the current hidden state through a non-linear transformation. Initially, the hidden state is usually set to a specific initial value, such as a zero vector or a random vector. As the input sequence progresses, the hidden state is continuously updated, gradually accumulating and fusing information from all previous inputs. This allows it to capture long-term dependencies in the sequence, providing effective feature representations for subsequent language processing tasks such as text generation and sentiment analysis.

[0003] The model's understanding of semantics is mainly based on the learning of word vectors and sequence information. For pronouns, which are highly flexible and context-dependent words, the semantic understanding is often not accurate enough. The referential meaning of "this" depends entirely on the context, and the model may simply infer its referent based on the co-occurrence relationship of words or local context, making it difficult to deeply understand the semantic logic and grammatical structure of the text to determine the accurate referent.

[0004] Therefore, this invention proposes a method for generating hidden states in recurrent neural networks for language processing. By comparing the query vector with the key vectors generated from all historical hidden states, the importance of each historical position is determined. Specifically, each historical hidden state is converted into a key vector and a value vector. Then, the dot product score between the query vector and each value vector is calculated, and after scaling and normalization, attention weights are obtained. These weights reflect the association strength between the current word and each historical position. In the sentence "The government implemented policies to reduce carbon emissions. This initiative received widespread support.", when processing "This", the model will give a higher score to the key vector corresponding to "policies", thus generating a larger attention weight. Summary of the Invention

[0005] The technical problem to be solved: The model's understanding of semantics is mainly based on the learning of word vectors and sequence information. For pronouns, which are highly flexible and context-dependent words, the semantic understanding is often not accurate enough. The referential meaning of "this" depends entirely on the context, and the model may simply infer its referent based on the co-occurrence relationship of words or local context, making it difficult to deeply understand the semantic logic and grammatical structure of the text to determine the accurate referent.

[0006] To address the shortcomings of existing technologies, this invention provides a method for generating hidden states in recurrent neural networks for language processing, thereby solving the technical problems mentioned in the background section.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A method for generating hidden states in a recurrent neural network for language processing includes the following steps:

[0009] S1. Input Embedding: This transforms discrete words into continuous vector representations, providing a computable semantic foundation for the model. It converts the abstract meaning of words into numerical form. The embedding layer is constructed through pre-training or random initialization. Each word is mapped to a fixed-dimensional vector space. Pre-trained embeddings capture semantic associations using statistical patterns from a large-scale corpus. Randomly initialized embeddings are gradually adjusted through training on task data. The dimension of the embedding vectors needs to balance expressive power and computational cost. The original text is thus transformed into a high-dimensional numerical sequence suitable for neural network processing.

[0010] S2. Generate the current query: Construct a "search focus" vector as the starting point of the attention mechanism. The query vector construction depends on the input information of the current time step or the hidden state of the previous time step. Through linear transformation, the input vector or hidden state is projected onto a new vector space to form a query, which is achieved by a learnable weight matrix. The matrix parameters are optimized during training. The generated query vector contains local features of the current word and may also inherit accumulated information from historical hidden states.

[0011] S3. Calculate attention weights: Calculating attention weights is the core step in dynamically filtering historical information. The model compares the current query vector with all historical position key vectors to quantify the importance of historical elements. Key vectors are generated from historical hidden states through linear transformation. Attention scores measure similarity through dot product operations, are scaled, and then converted into a probability distribution through a normalization function. The calculation process does not depend on sequence order or distance and directly establishes global associations.

[0012] S4. Generate Context Vector: The context vector is the output of the attention mechanism. It is a weighted fusion of historical information. The attention weights are multiplied by the corresponding value vectors and summed. The value vectors are obtained from the historical hidden states through linear transformation. The generation of the context vector is selective, retaining features that are highly relevant to the current query.

[0013] S5. Hidden State Update: Hidden state update is the core operation of recurrent neural networks. It fuses the current input, historical state and context information through nonlinear transformation. After introducing the attention mechanism, the context vector is used as an additional input. The three parts of information are linearly combined through independent weight matrices and then compressed to a reasonable range by the activation function.

[0014] S6. Iterative Propagation: Iterative propagation is the core mechanism for recurrent neural networks to process sequential data. The updated hidden state serves as the initial state for the next time step and continues to participate in subsequent attention calculations and state updates. For long sequence processing, computational efficiency can be optimized by limiting the length of the historical window or by adopting a sparse attention strategy. Hidden state propagation forms the vertical flow of information, and the attention mechanism enhances cross-step information interaction.

[0015] In one possible implementation, an embedding method needs to be selected before input embedding, utilizing the semantic prior knowledge provided by word vectors trained on the corpus.

[0016] A method for generating hidden states in a recurrent neural network for language processing includes the following steps:

[0017] S21: Based on the previous hidden state: q t =W q ·h t-1 The query is generated by utilizing the hidden state of the previous time step to capture the progressive logic of the sequence;

[0018] S22: Based on the current input: q t =W q ·x t The query is based on the current word embedding, emphasizing the context of the current input.

[0019] S23: W q ∈R h*h It is a learnable matrix.

[0020] A method for generating hidden states in a recurrent neural network for language processing includes the following steps:

[0021] S31: Key vector k i :k i =W k ·h i This maps historical hidden states to the key space for matching queries;

[0022] S32: Value vector vi :v i =W v ·h i This maps historical hidden states to the value space for information aggregation.

[0023] S33: W k W v ∈R h*h Control the semantic projection direction of the key and value respectively;

[0024] S34: The formula for calculating the attention score is:

[0025]

[0026] S35: The formula for normalizing to a probability distribution is:

[0027]

[0028] In one possible implementation, a comprehensive context vector is obtained by weighted summing of attention weights with their corresponding value vectors.

[0029] In one possible implementation, the hidden state update formula of a standard RNN includes a context vector in addition to the current input and the previous hidden state. The new hidden state integrates these three parts of information through a non-linear activation function.

[0030] In one possible implementation, the newly generated hidden state is passed to the next time step and participates in subsequent attention calculations as part of the historical information.

[0031] Beneficial effects compared to existing technologies:

[0032] 1. In this scheme, attention weights are normalized to a probability distribution, and the model can automatically focus on key information points. When processing the sentence "The government implemented policies to reduce carbon emissions. This initiative received widespread support.", if the current word is "this", the model may assign a weight of 0.9 to "policies" in the preceding text, while assigning only 0.05 to other secondary information such as "government", thereby accurately locating the referent.

[0033] 2. In this solution, attention weights can be combined with the rule engine to improve the robustness of the model. If the grammatical rules require "this" to refer to the subject of the preceding text first, the weight distribution can be adjusted through post-processing to force the model to follow specific logic. If "this" is incorrectly associated with secondary information, it is found that the weight is abnormally high in an irrelevant position, and the model structure or training strategy can be adjusted accordingly. When the model has a referential error, checking the attention weight distribution can help locate the problem.

[0034] 3. In this scheme, since the calculation of the attention mechanism does not depend on chain gradient propagation, the weight calculation of long-distance information is not affected by gradient vanishing. Attached Figure Description

[0035] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, the preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0036] Figure 1 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0037] Preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the present invention can also be implemented in various different forms, and therefore the present invention is not limited to the embodiments described below.

[0038] The technical solution in this application embodiment is to solve the problems mentioned in the background art, and the overall idea is as follows:

[0039] Example 1:

[0040] Please refer to Figure 1 As shown in the figure, this embodiment introduces a method for generating hidden states in a recurrent neural network for language processing, including the following steps:

[0041] S1. Input Embedding: This step transforms discrete words into continuous vector representations, capturing their semantic and syntactic information. The core objective of this step is to provide a computationally achievable semantic foundation for the model, converting the abstract meaning of words into numerical form. Embedding layers are typically constructed through pre-training or random initialization. Each word is mapped to a fixed-dimensional vector space. Pre-trained embeddings utilize statistical patterns from large-scale corpora to capture semantic relationships between words; for example, synonyms are closer together in the vector space. Randomly initialized embeddings are gradually adjusted through the training process of task data, making them more suitable for the vocabulary distribution of a specific domain. The dimensionality of the embedding vectors needs to balance expressive power and computational cost; higher dimensions can accommodate more semantic information but increase the number of model parameters and training time. Through this step, the original text is transformed into a high-dimensional numerical sequence suitable for neural network processing, laying the foundation for subsequent contextual modeling.

[0042] S2. Generating the Current Query: Constructing a "search focus" vector to guide the model in retrieving relevant content from historical information. Generating the query vector is the starting point of the attention mechanism, and its role is to provide the model with the current focus of attention. The construction of the query vector depends on the input information of the current time step or the hidden state of the previous time step. Through linear transformation operations, the input vector or hidden state is projected onto a new vector space to form a query with a clear semantic direction. This transformation process is achieved through a learnable weight matrix. The matrix parameters are gradually optimized during training to adapt to the attention patterns of different tasks. The generation of the query vector not only includes the local features of the current word, but may also inherit the accumulated information of the historical hidden state, thereby dynamically adjusting the direction of attention. For example, when processing pronouns, the query tends to carry the key features of the preceding event in order to retrieve the relevant referent in the historical context. The key to this step is to encode the semantic requirements of the current processing into a comparable vector form, providing a benchmark for subsequent attention calculations.

[0043] S3. Calculate Attention Weights: Quantify the correlation strength between the current query and each historical position, dynamically filter key information. Calculating attention weights is the core step in dynamically filtering historical information. The model quantifies the importance of each historical element to the current processing by comparing the current query vector with the key vectors of all historical positions. The key vector is generated by the historical hidden state through linear transformation. Its role is to transform the abstract representation of the hidden state into semantic features that can match the query. The attention score measures the similarity between the query and the key through dot product operation. A high score indicates a strong correlation. To avoid numerical instability, the score needs to be scaled and then converted into a probability distribution through a normalization function. This distribution reflects the intensity of the model's attention to historical information. Information from high-weight positions will be retained first. During the calculation process, the model does not need to rely on sequence order or distance, but directly establishes global associations, thereby breaking through the limitations of traditional recursive structures. The dynamic nature of attention weights allows the model to flexibly adapt to different contexts. For example, when dealing with polysemous words, the semantic focus in different contexts can be distinguished by weight allocation.

[0044] S4. Generating Context Vectors: Integrating historical information to form a global context representation relevant to the current query. The context vector is the output of the attention mechanism, which is essentially a weighted fusion of historical information. By multiplying the attention weights with the corresponding value vectors and summing them, the model integrates scattered historical features into a compact global representation. The value vector is also obtained by linear transformation of the historical hidden states, which encodes the detailed information of each historical position. The generation process of the context vector is selective, retaining only the features highly relevant to the current query while suppressing irrelevant or redundant content. This focusing mechanism enables the model to extract key segments from long sequences, such as accurately locking the preceding events in the referential resolution task. The context vector not only contains semantic content but may also implicitly contain grammatical structures or logical relationships, providing rich supplementary information for subsequent hidden state updates. Through this step, the model achieves proactive perception of the global context, rather than passively relying on recursively passed local information.

[0045] S5. Updating the Hidden State: The hidden state is generated by fusing the current input, the historical hidden state, and the global context. Hidden state update is the core operation of recurrent neural networks. It fuses the current input, the historical state, and the context information through nonlinear transformation. Traditional RNNs rely only on the simple superposition of the current input and the previous hidden state. However, after introducing the attention mechanism, the context vector as an additional input significantly enhances the information capacity. During the update process, the three parts of information are linearly combined through independent weight matrices and then compressed to a reasonable range by the activation function. The choice of activation function (such as the hyperbolic tangent function) ensures the smoothness and stability of the output while retaining nonlinear features. The updated hidden state not only reflects the local context at the current moment but also carries the global key information filtered by attention. This fusion mechanism enables the model to capture the temporal characteristics of the sequence and the semantic association across the stride. For example, when dealing with causal logic, the hidden state can encode the current result and the distant cause at the same time.

[0046] S6. Iterative Propagation: The new hidden state is passed to subsequent time steps to gradually build a deep representation of the sequence. Iterative propagation is the core mechanism of recurrent neural networks in processing sequence data. The updated hidden state will serve as the initial state for the next time step and continue to participate in subsequent attention calculations and state updates. As time steps advance, the model gradually builds a contextual representation covering the entire sequence. Each hidden state accumulates the essence of historical information. For long sequence processing, computational efficiency can be optimized by limiting the length of the historical window or by using a sparse attention strategy, but this may sacrifice some ability to capture long-distance dependencies. The propagation path of the hidden state forms the vertical flow of information, while the attention mechanism enhances cross-step information interaction through horizontal association. This combined vertical and horizontal structure allows the model to maintain the temporality of sequence processing while breaking through the constraints of local vision, achieving true global understanding. Finally, the hidden state sequence generated in the iterative process can serve as a high-level semantic representation for downstream tasks (such as classification or generation).

[0047] Before embedding the input, it is necessary to select an embedding method and utilize the semantic prior knowledge provided by word vectors trained on the corpus.

[0048] A method for generating hidden states in a recurrent neural network for language processing includes the following steps:

[0049] S21: Based on the previous hidden state: q t =W q ·h t-1 The query is generated using the hidden state of the previous time step to capture the progressive logic of the sequence;

[0050] S22: Based on the current input: q t =W q ·x tThe query is based on the current word embedding, emphasizing the context of the current input.

[0051] S23: W q ∈R h*h It is a learnable matrix, and the direction of query generation is adjusted through training.

[0052] A method for generating hidden states in a recurrent neural network for language processing includes the following steps: When calculating attention weights, it is necessary to generate key vectors and value vectors.

[0053] S31: Key vector k i :k i =W k ·h i This maps historical hidden states to the key space for matching queries;

[0054] S32: Value vector v i :v i =W v ·h i This maps historical hidden states to the value space for information aggregation.

[0055] S33: W k W v ∈R h*h Control the semantic projection direction of the key and value respectively;

[0056] S34: The formula for calculating the attention score is:

[0057]

[0058] scaling factor To prevent the dot product result from becoming too large and causing the Softmax gradient to saturate;

[0059] S35: The formula for normalizing to a probability distribution is:

[0060]

[0061] By weighted summing of attention weights and their corresponding value vectors, a comprehensive context vector is obtained. This vector concentrates the historical information most relevant to the current word. In the hidden state update formula of a standard RNN, in addition to the current input and the previous hidden state, a context vector is also added. The new hidden state fuses these three parts of information through a non-linear activation function. This fusion mechanism allows the hidden state to not only contain local context but also dynamically introduce global key information. The newly generated hidden state is passed to the next time step and participates in subsequent attention calculations as part of the historical information. As sequence processing progresses, the model gradually accumulates richer context representations. When processing long texts, computational efficiency can be optimized by truncating windows or storing historical hidden states in blocks, while maintaining access to key information.

[0062] Working principle:

[0063] Input statement:

[0064] "The government implemented policy stored carbon emissions. This initiative received wide spread support."

[0065] Each word is transformed into a continuous vector through an embedding layer, capturing semantic and syntactic features, for example:

[0066] "government" → a vector that encodes semantics such as institutions and power.

[0067] "policies" → a vector representing semantics such as coding rules and plans.

[0068] "carbonemissions" → Vectors that encode themes such as environmental protection and pollution.

[0069] Embedding matrices map words to high-dimensional vectors, providing numerical input for subsequent computations.

[0070] When processing "This" in the second sentence, a query vector is generated to locate the referent. If based on the previous hidden state (the hidden state at the end of the previous sentence), the query vector q... t Encode the overall information in the preceding text.

[0071] Generate key and value:

[0072] k i =W k ·h i v i =W v ·h i

[0073] Implicit states in the preceding text: such as the implicit states h1, h2, and h3 corresponding to "government," "policies," and "carbon emissions" in the first sentence.

[0074] Key vectors: Transform latent states into semantic features that can be matched with queries, such as the key vector k2 encoding "policies" for "policy content".

[0075] Value vector: Retains detailed information about hidden states, such as the value vector v2 of “policies” which contains a specific description of emission reduction policies.

[0076] Calculate attention score:

[0077]

[0078] Find the match between "This" and the key:

[0079] The key k2 dot product score with "policies" is high (strong semantic correlation).

[0080] The key k1 with "government" has a low score (weak semantic correlation).

[0081] Normalized weights:

[0082]

[0083] “policies” weight a t,2 The weight of the other words is approximately 0.9, while the weight of the other words is close to 0.

[0084] Generate context vectors:

[0085] The value vector v2 of “policies” dominates the context vector due to its high weight, while the value vectors of other words (such as “government”) are suppressed due to their low weight. The context vectors collectively represent the core information of “implementing emission reduction policies”.

[0086] Update hidden state:

[0087] The complete logic of the new hidden state encoding "This initiative points to policies".

[0088] Iterative propagation:

[0089] The new hidden state is passed to subsequent time steps, continuously accumulating information.

[0090] When processing "received":

[0091] The hidden state already includes the association "This → policies", ensuring that the action "received support" is correctly bound to the policy.

[0092] Global consistency:

[0093] The model gradually builds coherent semantics across sentences, ultimately understanding the core logic of the entire passage: "The implementation of the policy has received support."

[0094] Finally, it should be noted that the above embodiments are merely examples for clearly illustrating the present invention and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A method for generating hidden states in a recurrent neural network for language processing, characterized in that, Includes the following steps: S1. Input Embedding: This transforms discrete words into continuous vector representations, providing a computable semantic foundation for the model. It converts the abstract meaning of words into numerical form. The embedding layer is constructed through pre-training or random initialization. Each word is mapped to a fixed-dimensional vector space. Pre-trained embeddings capture semantic associations using statistical patterns from a large-scale corpus. Randomly initialized embeddings are gradually adjusted through training on task data. The dimension of the embedding vectors needs to balance expressive power and computational cost. The original text is thus transformed into a high-dimensional numerical sequence suitable for neural network processing. S2. Generate the current query: Construct a "search focus" vector as the starting point of the attention mechanism. The query vector construction depends on the input information of the current time step or the hidden state of the previous time step. Through linear transformation, the input vector or hidden state is projected onto a new vector space to form a query, which is achieved by a learnable weight matrix. The matrix parameters are optimized during training. The generated query vector contains local features of the current word and may also inherit accumulated information from the historical hidden state. S3. Calculate attention weights: Calculating attention weights is the core step in dynamically filtering historical information. The model compares the current query vector with all historical position key vectors to quantify the importance of historical elements. Key vectors are generated from historical hidden states through linear transformation. Attention scores measure similarity through dot product operations, are scaled, and then converted into a probability distribution through a normalization function. The calculation process does not depend on sequence order or distance and directly establishes global associations. S4. Generate Context Vector: The context vector is the output of the attention mechanism. It is a weighted fusion of historical information. The attention weights are multiplied by the corresponding value vectors and summed. The value vectors are obtained from the historical hidden states through linear transformation. The generation of the context vector is selective, retaining features that are highly relevant to the current query. S5. Hidden State Update: Hidden state update is the core operation of recurrent neural networks. It fuses the current input, historical state and context information through nonlinear transformation. After introducing the attention mechanism, the context vector is used as an additional input. The three parts of information are linearly combined through independent weight matrices and then compressed to a reasonable range by the activation function. S6. Iterative Propagation: Iterative propagation is the core mechanism for recurrent neural networks to process sequential data. The updated hidden state serves as the initial state for the next time step and continues to participate in subsequent attention calculations and state updates. For long sequence processing, computational efficiency can be optimized by limiting the length of the historical window or by adopting a sparse attention strategy. Hidden state propagation forms the vertical flow of information, and the attention mechanism enhances cross-step information interaction.

2. The method for generating hidden states in a recurrent neural network for language processing as described in claim 1, characterized in that, Before embedding the input, it is necessary to select an embedding method and utilize the semantic prior knowledge provided by word vectors trained on the corpus.

3. The method for generating hidden states in a recurrent neural network for language processing as described in claim 1, characterized in that, Includes the following steps: S21: Based on the previous hidden state: The query is generated by utilizing the hidden state of the previous time step to capture the progressive logic of the sequence; S22: Based on the current input: Using the current word embedding as the query, it emphasizes the context of the current input. For the first The query vector at each time step; S23: It is a learnable matrix.

4. The method for generating hidden states in a recurrent neural network for language processing as described in claim 1, characterized in that, Includes the following steps: S31: Key Vector : This maps historical hidden states to the key space for matching queries; S32: Value Vector : This maps historical hidden states to the value space for information aggregation. S33: Control the semantic projection direction of the key and value respectively; S34: The formula for calculating the attention score is: ； in, This is the scaling factor; S35: The formula for normalizing to a probability distribution is: 。 5. The method for generating hidden states in a recurrent neural network for language processing as described in claim 1, characterized in that, By summing the attention weights with their corresponding value vectors, a comprehensive context vector is obtained.

6. The method for generating hidden states in a recurrent neural network for language processing as described in claim 1, characterized in that, In the hidden state update formula of a standard RNN, in addition to the current input and the previous hidden state, a context vector is added. The new hidden state integrates these three parts of information through a non-linear activation function.

7. The method for generating hidden states in a recurrent neural network for language processing as described in claim 1, characterized in that, The newly generated hidden states will be passed to the next time step and participate in subsequent attention calculations as part of the historical information.

Images