A text feature extraction method based on an NGU language model

Abstract

The application relates to the technical field of natural language processing, and relates to a text feature extraction method based on an NGU language model. The patent is optimized and improved based on a GRU model of a recurrent neural network, a new text feature extraction model, namely an NGU language model, is proposed, a gating unit of the GRU is introduced into a normalization mechanism, a hyperbolic tangent function with a saturation zone is replaced by a layer normalization operation, and a feedforward layer neural network of a Transformer is integrated into an iteration unit to improve semantic expression capability of the model, namely data fitting capability of the model.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, specifically to a text feature extraction method based on the NGU language model. Background Technology

[0002] RNNs and Transformers, as fundamental building blocks in Natural Language Processing (NLP), play a crucial role in NLP tasks. Among them, the GRU model, belonging to the RNN family, is particularly important in entity recognition, relation extraction, and text generation. In text generation tasks, GRU units enable iterative and streaming text generation reasoning, resulting in faster text generation speeds compared to Transformer and GPT models. In entity recognition and relation extraction tasks, GRU models also perform well because their iterative mechanism incorporates information from neighboring text. However, Transformers, employing a globally interactive multi-head self-attention mechanism, possess powerful semantic representation capabilities. Therefore, current NLP tasks increasingly utilize large models based on Transformer variants. These models achieve excellent results on large datasets but require significant computational resources. To combine the advantages of both Transformers and GRUs, this patent proposes the NGU language model, an improvement upon the GRU model. Summary of the Invention

[0003] This patent optimizes and improves the GRU model based on recurrent neural networks, proposing a new text feature extraction model, the NGU language model. It introduces a normalization mechanism into the gating unit of GRU, replaces the hyperbolic tangent function with a layer normalization operation with the hyperbolic tangent function that has a saturation region, and integrates the feedforward layer neural network of Transformer into the iterative unit to improve the model's semantic representation ability, that is, the model's ability to fit data. This patent defines it as the NGU language model.

[0004] To address the problems existing in current technologies and to combine the advantages of both GRU and Transformer, a text feature extraction method based on the NGU language model is provided, comprising the following steps:

[0005] S1. Construct training dataset: Collect and organize the training dataset related to the task and put it into train.txt. The maximum length of the text input to the NGU language model is 1000. When the text length is less than 1000, use [PAD] to pad to the maximum length of 1000.

[0006] S2. Construct a mapping from characters to IDs: For the training set train.txt in S1, count the characters and denote them as token_list. Then, build a dictionary Dict_token based on the characters in token_list. The first key of Dict_token is the character index number, and the second value of Dict_token is the specific single character. [PAD] is the padding character when the text is not long enough.

[0007] S3. Training Data and Model Adaptation: In the training dataset obtained in step S1, if the text samples are less than the maximum length of 1000, the list is filled with [PAD] to reach the maximum length of 1000. Then, it is mapped to an index number list through the dictionary Dict_token and becomes the tensor X input to the model. The batch size of the NGU language model is 128, so the size of X is [128, 1000].

[0008] S4.NGU Language Model Text Feature Extraction: The original GRU network model iterative formula is as follows:

[0009]

[0010] f is the equivalent formula for a GRU gated cyclic unit. The detailed formula for f is as follows:

[0011]

[0012] The proposed NGU iterative formula is as follows:

[0013]

[0014] The sigmoid function exhibits a saturation region when x is far from 0. After passing through a fully connected layer, information is lost due to the sigmoid function. To address this, a layer normalization operation is introduced to normalize the embedding representation dimension. Then, the sigmoid function is used to effectively preserve the text representation information.

[0015] when middle When the value moves far from 0 and reaches the saturation region, the output tends to a stable value, losing a lot of semantic information. Layer normalization replaces the hyperbolic tangent function tanh in GRU. The layer normalization operation is as follows: normalization is performed on the semantic representation dimension d_model. Assuming the current word representation matrix is ​​T, and the matrix size is [1, d_model], the values ​​in the second dimension are as follows:

[0016]

[0017] Their mean values ​​are as follows:

[0018]

[0019] Its variance is:

[0020]

[0021] Each input to the layer normalization is:

[0022]

[0023] The layer normalization operation only translates and shrinks the data without losing semantic information. At the same time, this operation normalizes the embedding representation dimension to near 0, making the model training more stable. The feedforward neural network layer in the Transformer is transferred to the NGU model. The feedforward neural network contains two linear transformations and one nonlinear transformation GeLU activation function, and then passes through the residual network and the layer normalization operation.

[0024]

[0025] The character embedding dimension is 256, and the hidden layer dimension of the feedforward neural network is 2048. For the model input size in step S3, which is [128, 1000], each character in the text data is first embedded using token_embedding. The shape of the embedding matrix for each of the 1000 characters is [128, 256], where 128 is the batch size during training and 256 is the character embedding dimension in the text. Then, the embedding of each character in the text data X is sequentially input into the aforementioned NGU iterative unit, and then at each time step... spliced ​​together The dimensions are [128, 1000, 256]. Text features are extracted using the NGU language model. Each word of each sentence in a batch of data is represented as a 256-dimensional string using the NGU language model.

[0026] S5. NGU Language Model Application: The proposed NGU language model is a non-pre-trained language model. Its parameters are trained according to specific natural language processing tasks. The text representation tensor size obtained from S4 is...

[0027] [128, 1000, 256], this tensor is fed into the neural networks for subsequent text classification, relation extraction, text generation, and entity recognition for training.

[0028] The beneficial effect of this invention is that the GRU's sigmoid activation function has a saturation region, and the fully connected layers in the middle will compress the text representation data information in the saturation region. This patent introduces layer normalization after the fully connected layers in the activation function, shifting the data center to near 0 and preserving information. To further preserve text information, that is, to avoid activation functions with saturation regions as much as possible, this patent directly replaces the hyperbolic tangent function in the GRU with layer normalization operations. Because of the memory gating effect of the sigmoid function itself, the model training stability is not affected. The normalization operation only shifts the values ​​and normalizes the variance, preserving the richness of function values ​​as much as possible, which will improve the model's ability to fit the amount of data. The Transformer's feedforward neural network is a very powerful module with strong fitting capabilities that performs fully connected operations in the embedding representation dimension. The NGU language model of this patent effectively combines the advantages of GRU and Transformer and can be applied to various natural language processing tasks. Although the idea for this patent originates from GRU, it replaces the hyperbolic tangent function in GRU, introduces layer normalization operations in multiple places, and incorporates a feedforward fully connected neural network in the iterative unit, making it significantly different from the GRU model. This patent defines it as the NGU language model. Attached Figure Description

[0029] Figure 1 This is a flowchart illustrating the overall text feature extraction process of this invention.

[0030] Figure 2 Plot a graph of GRU using the sigmoid function;

[0031] Figure 3 Plot the tanh function and its derivative. Detailed Implementation

[0032] The following is a further detailed explanation using specific examples:

[0033] This invention specifically relates to a text feature extraction method based on the NGU language model, comprising the following steps:

[0034] S1. Constructing the training dataset: For specific tasks in the project, such as text classification, relation extraction, entity recognition, text search, text writing, text completion, and other natural language processing tasks, collect and organize the task-related training datasets and put them into train.txt. This patent stipulates that the maximum length of the text input to the NGU language model is 1000 characters. When the text length is less than 1000 characters, it is padded to the maximum length of 1000 characters using [PAD].

[0035] S2. Construct the mapping from characters to IDs: For the train.txt in the training set in S1, count the characters in it and record them as token_list. Then, based on the characters in token_list, establish a dictionary Dict_token. The former key of Dict_token is the character index number, and the latter value of Dict_token is the specific single character, that is, Dict_token={0:”中”,1:”华“,2:”唐”,3:”人”,4:”民”,5:”[PAD]”.......}, where [PAD] is the padding character when the text is not long enough to reach the maximum length;

[0036] S3. Adapt the training data to the model: Suppose a text sample in the training dataset obtained in step S1 is "With the development of deep learning, artificial intelligence has received increasing attention in the fields of finance, healthcare, and education.", map this text to a list, which is ['随', '着', '深', '度', '学', '习', '的','发', '展', ',', '人', '工', '智', '能', '在', '金', '融', '领', '域', '、', '医', '疗', '领', '域', '、', '教', '育', '领', '域', ',', '得', '到', '了', '愈', '加', '广', '泛', '的', '重', '视', '。']. This text is not long enough to reach the maximum length of 1000, so fill the list with [PAD] to the maximum length of 1000 of this text, then map it to a list of index numbers through the dictionary Dict_token, and turn it into the tensor X input to the model. At this time, if the batch size batch_size of the input NGU language model is 128, the size of X is [128, 1000].

[0037] S4. Extract text features of the NGU language model: The iterative formula of the original GRU network model is as follows:

[0038]

[0039] f is the equivalent formula of the GRU gated recurrent unit, and the detailed formula of f is as follows:

[0040]

[0041] The specific NGU iterative formula proposed in this patent is as follows:

[0042]

[0043] GRU is plotted in the sigmoid function as Figure 2 As shown.

[0044] The sigmoid function exhibits a saturation region when x is far from 0. After passing through a fully connected layer, information is lost due to the sigmoid function. This patent introduces a layer normalization operation to normalize the embedding representation dimension, and then uses the sigmoid function to effectively preserve the richness of the text representation information.

[0045] In GRU, the tanh function also exhibits saturation when x is far from 0, because the derivative becomes 0, causing the gradient to vanish. The tanh function and its derivative are plotted as follows: Figure 3 As shown.

[0046] Among them, when middle When the value moves far from 0 and reaches the saturation region, the output tends to a stable value, losing a lot of semantic information. This patent replaces the hyperbolic tangent function tanh in GRU with a layer normalization operation. The layer normalization operation of this patent is as follows: normalization is performed on the semantic representation dimension d_model. Assuming the current word representation matrix is ​​T, and the matrix size is [1, d_model], the values ​​in the second dimension are as follows:

[0047]

[0048] Their mean values ​​are as follows:

[0049]

[0050] Its variance is:

[0051]

[0052] Each input to the layer normalization is:

[0053]

[0054] As can be seen, the layer normalization operation merely shifts and shrinks the data without losing semantic information. Simultaneously, this operation normalizes the embedding representation dimension to near 0, making model training more stable. This patent introduces layer normalization operations throughout the gating unit, replacing the hyperbolic tangent function in the GRU with layer normalization operations, resulting in faster model training and excellent semantic representation capabilities. To further improve the data fitting ability of this patent, the feedforward neural network layer in the Transformer is transferred to the NGU model. The feedforward neural network contains two linear transformations and one nonlinear transformation (GeLU activation function), followed by a residual network and layer normalization operations.

[0055]

[0056] In this patent, the character embedding dimension is 256, and the hidden layer dimension of the feedforward neural network is 2048. For the model input size in step S3, which is [128, 1000], each character in the text data is first embedded using token_embedding. At this point, the matrix shape of the embedding representation of each character in the 1000 characters is [128, 256], where 128 is the batch size during training and 256 is the character embedding dimension in the text. Then, the embedding of each character in the text data X is sequentially input into the aforementioned NGU iterative unit. Then, at each time step... spliced ​​together The dimensions are [128, 1000, 256], which means that the text features are extracted through the NGU language model. In other words, each word of each sentence in a batch of data is represented by the NGU language model with 256 dimensions.

[0057] S5. NGU Language Model Application: The NGU language model proposed in this patent is a non-pre-trained language model, requiring parameter training based on specific natural language processing tasks. The text representation tensor size obtained from S4 is...

[0058] [128, 1000, 256], this tensor is fed into the neural network for subsequent text classification, relation extraction, text generation, and entity recognition for training, thereby completing the application of this patent.

[0059] This patent, through theoretical analysis, proposes the NGU language model based on improvements to the GRU model. The main improvements include: introducing layer normalization operations before the sigmoid function operations in both the GRU reset and update gates; and replacing the hyperbolic tangent function in the GRU with a layer normalization operation. Furthermore, a feedforward neural network from the Transformer is introduced into NGU to further enhance the model's semantic representation complexity, i.e., its ability to fit data. This patented NGU language model effectively combines the advantages of both GRU and Transformer, and can be applied to various natural language processing tasks.

Claims

1. A text feature extraction method based on the NGU language model, characterized in that... The steps include the following: S1. Construct training dataset: Collect and organize the training dataset related to the task and put it into train.txt. The maximum length of the text input to the NGU language model is 1000. When the text length is less than 1000, use [PAD] to pad to the maximum length of 1000. S2. Construct a mapping from characters to IDs: For the training set train.txt in S1, count the characters in it and denote it as token_list. Then, build a dictionary Dict_token based on the characters in token_list. The first key of Dict_token is the character index number, and the second value of Dict_token is the specific single character. [PAD] is the padding character when the text is not long enough. S3. Training Data and Model Adaptation: In the training dataset obtained in step S1, if the text samples are less than the maximum length of 1000, the list is filled with [PAD] to reach the maximum length of 1000. Then, it is mapped to an index number list through the dictionary Dict_token and becomes the tensor X input to the model. The batch size of the NGU language model is 128, so the size of X is [128, 1000]. S4 NGU Language Model Text Feature Extraction: The original GRU network model iterative formula is as follows: f is the equivalent formula for a GRU gated cyclic unit. The detailed formula for f is as follows: The proposed NGU iterative formula is as follows: The sigmoid function exhibits a saturation region when x is far from 0. After passing through a fully connected layer, information is lost due to the sigmoid function. To address this, a layer normalization operation is introduced to normalize the embedding representation dimension. Then, the sigmoid function is used to effectively preserve the text representation information. when middle When the value moves away from 0 and reaches the saturation region, the output tends to a stable value, losing a lot of semantic information; The hyperbolic tangent function tanh in GRU is replaced by layer normalization. The layer normalization operation is as follows: normalization is performed on the semantic representation dimension d_model. Assuming the current word representation matrix is ​​T and the matrix size is [1, d_model], the values ​​of the second dimension are as follows: Their mean values ​​are as follows: Its variance is: Each input to the layer normalization is: The layer normalization operation only translates and shrinks the data without losing semantic information. At the same time, this operation normalizes the embedding representation dimension to near 0, making the model training more stable. The feedforward neural network layer in the Transformer is transferred to the NGU model. The feedforward neural network contains two linear transformations and one nonlinear transformation GeLU activation function, and then passes through the residual network and the layer normalization operation. The character embedding dimension is 256, and the hidden layer dimension of the feedforward neural network is 2048. For the model input size in step S3, which is [128, 1000], each character in the text data is first embedded using token_embedding. The shape of the embedding matrix for each of the 1000 characters is [128, 256], where 128 is the batch size during training and 256 is the character embedding dimension in the text. Then, the embedding of each character in the text data X is sequentially input into the aforementioned NGU iterative unit, and then at each time step... spliced ​​together The dimensions are [128, 1000, 256]. Text features are extracted using the NGU language model. Each word of each sentence in a batch of data is represented as a 256-dimensional string using the NGU language model. S5. NGU Language Model Application: The proposed NGU language model is a non-pre-trained language model. The parameters are trained according to the specific natural language processing task. The text representation tensor obtained in S4 has a size of [128, 1000, 256]. This tensor is then fed into the neural networks for subsequent text classification, relation extraction, text generation, and entity recognition for training.

Images