Transform-based natural language processing method

Abstract

The invention relates to the technical field of natural language processing, and particularly provides a natural language processing method based on transformer. The method comprises the following steps: inputting processed data into an input layer of a transformer-based natural language processing model, and converting characters in the data into vectors through the input layer; inputting the vector into a language encoder combined with a sparse attention mechanism so as to improve the modeling capability of a long text and obtain a hidden sequence; the hidden sequence is input into the output layer, output related to the input text is generated, the ability of understanding the long text is enhanced, and the requirement of a local window mechanism for a video memory is reduced, so that common hardware can support a larger input scale, and better experience feeling is brought to a user.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular, to a natural language processing method based on Transformer. Background Art

[0002] In recent years, with the rapid development of deep learning technology, intelligent dialogue systems based on natural language processing (NLP) have been widely used in various fields, including scenarios such as intelligent customer service, virtual assistants, medical consultations, and educational tutoring. The core technologies of these dialogue systems rely on the generation ability and context understanding ability of language models to achieve high-quality generation of natural language. However, when processing ultra-long texts, such as legal documents and scientific research papers, the demand for video memory by the local window mechanism makes it impossible for ordinary hardware to support a larger input scale, resulting in a poor user experience. Summary of the Invention

[0003] In view of this, the present invention provides a natural language processing method based on Transformer to enhance the understanding ability of long texts, reduce the demand for video memory by the local window mechanism, so that ordinary hardware can support a larger input scale and bring a better user experience.

[0004] In a first aspect, the present invention provides a natural language processing method based on Transformer, and the method includes: Step 1: Input the processed data into the input layer of the natural language processing model based on Transformer, and convert the text in the data into a vector hidden_state_llminput through the input layer; Step 2: Input the vector hidden_state_llminput into a language encoder combined with a sparse attention mechanism to improve the modeling ability of long texts and obtain a hidden sequence hidden_state_llmout; Step 3: Input the hidden sequence hidden_state_llmout into the output layer to generate an output related to the input text.

[0005] Optionally, the Step 1 includes: Step 11: For the dialogue dataset S, perform data preprocessing on the dataset S to obtain a preprocessed dialogue dataset I; Step 12: Input the i-th data in the preprocessed dialogue dataset I into the input layer. Through serialization, indexing, and embedding, the data will obtain a real-number vector hidden_state. Then, through positional encoding to add the position information of sequence elements, and finally through a mask matrix to make the attention scores of invalid information approach 0, the output vector hidden_state_llminput of the input layer is finally obtained. The input layer includes five stages, which are, in sequence: serialization, indexing, embedding, mask matrix, and positional encoding.

[0006] Optionally, step 2 includes: Step 21: Use the vector hidden_state_llminput as the input of the language encoder and input it into the language encoder. The language encoder includes the first stage to the twenty-sixth stage, and each stage includes three consecutive parts, which are, in sequence: fully connected layer, sparse attention mechanism, and forward propagation layer. The modules and parameters of each stage are exactly the same, and the input of the latter stage is the output of the previous stage. After passing through the language encoder, hidden_state_llminput will obtain a hidden sequence hidden_state_llmout.

[0007] Optionally, step 3 includes: Step 31: Input the hidden sequence hidden_state_llmout into the output layer to generate the probability distribution of the next word. The output layer includes a fully connected layer and an activation function GELU.

[0008] Optionally, step 11 includes: Crop all samples included in the dialogue dataset, crop the text length greater than 4096 characters, and then perform text normalization, including: converting to lowercase, removing special characters, handling extra spaces, and removing spelling mistakes, to obtain the preprocessed dialogue dataset.

[0009] Optionally, step 12 includes: Serialization: Tokenize according to the dictionary of the tokenizer. The wordpiece tokenization method is adopted. The basic principle is to apply the input text to a pre-given vocabulary and split the given text into basic unit word pieces according to probability. Indexing: Use the Int32 subscript as the id of the basic unit word piece to obtain an integer array. Embedding: After the text is tokenized, embedding continues to map each input to a real-number vector, which is the embedding vector hidden_state. Positional Encoding: For each position after tokenization, a positional encoding vector is added to provide information about the position in the integer array; positional encoding is to distinguish words at different positions and provide information about the context relationship for the model.

[0010] Mask Matrix: Used to control the participation of positions in the calculation of the attention mechanism. In the model, the mask matrix affects the calculation of attention scores, thereby constraining what the model focuses on.

[0011] Optionally, the sparse attention mechanism in step 21 includes: Using a local window as the sparse pattern, and its attention calculation formula is as follows: ; where Q is the query matrix, K is the key matrix, V is the value matrix, M is the local window mask matrix used to represent valid positions, and d k is the feature dimension of the key / query; For the local window size w, the mask matrix M is defined as: ; In the calculation, only the positions (i, j) that satisfy are subjected to the attention operation; When the input vector hidden_state_llminput enters the first stage, after passing through the fully connected layer and the sparse attention mechanism, a real number vector demo is obtained. The weighted sum of demo and hidden_state_llminput is input into the forward propagation, and a real number vector after feature extraction can be obtained through the forward propagation layer ; is used as the input of the second stage, and then is output; is then used as the input of the third stage, and so on until the twenty-sixth stage. ~ are used as the inputs of the second stage to the twenty-sixth stage respectively. ~ are the outputs of the first stage to the twenty-fifth stage respectively, and the output of the twenty-sixth stage is the hidden sequence hidden_state_llmout.

[0012] Optionally, step 31 includes: Performing a weighted sum of the final output hidden_state_llmout of the language encoder and hidden_state to obtain the matrix ; The input passes through the fully connected layer and the activation function GELU to obtain the output token; Perform a weighted sum with the token, then update the input_ids. At the same time, judge that if the token sequence reaches the preset maximum length or the last token = eos_token, the iteration ends; otherwise, continue the iteration. In each iteration, the value of this token will be concatenated with the hidden_state for the next iteration.

[0013] In a second aspect, an embodiment of the present invention provides a computer-readable storage medium, where the computer-readable storage medium includes a stored program. When the program runs, it controls the device where the computer-readable storage medium is located to execute the transformer-based natural language processing method in the first aspect or any possible implementation manner of the first aspect.

[0014] In a third aspect, an embodiment of the present invention provides an electronic device, including: one or more processors; a memory; and one or more computer programs, where the one or more computer programs are stored in the memory, and the one or more computer programs include instructions. When the instructions are executed by the device, the device is caused to execute the transformer-based natural language processing method in the first aspect or any possible implementation manner of the first aspect.

[0015] In the technical solution provided by the present invention, the method includes inputting the processed data into the input layer of a transformer-based natural language processing model, converting the text in the data into vectors through the input layer; inputting the vectors into a language encoder combined with a sparse attention mechanism to improve the modeling ability for long texts and obtain a hidden sequence; inputting the hidden sequence into the output layer to generate an output related to the input text. This method enhances the understanding ability of long texts, reduces the demand for video memory by the local window mechanism, enables ordinary hardware to support a larger input scale, and brings a better experience to users. Description of the Drawings

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0017] Figure 1 It is a flowchart of the transformer-based natural language processing method provided by the embodiment of the present invention; Figure 2 It is a schematic diagram of the transformer-based natural language processing model provided by the embodiment of the present invention; Figure 3Schematic diagram of the language encoder provided by an embodiment of the present invention; Figure 4 Schematic diagram of an electronic device provided by an embodiment of the present invention. Detailed implementation manners

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0019] It should be clear that the described embodiments are only some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0020] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "said", and "the" used in the embodiments of the present invention are also intended to include the plural forms unless the context clearly indicates otherwise.

[0021] It should be understood that the term " / and / " used herein is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " herein generally represents an "or" relationship between the associated objects before and after.

[0022] Depending on the context, the word "if" as used herein can be interpreted as "when" or "while" or "in response to determining" or "in response to detecting". Similarly, depending on the context, the phrase "if determined" or "if detecting (stated condition or event)" can be interpreted as "when determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)".

[0023] Figure 1 Flowchart of the natural language processing method based on transformer provided by an embodiment of the present invention, as Figure 1 shown, the method includes: Step 1: Input the processed data into the input layer of the natural language processing model based on Transformer. Through the input layer, the text in the data is converted into a vector hidden_state_llminput; In the embodiment of the present invention, Step 1 includes: Step 11: For the dialogue dataset S, perform data preprocessing on the dataset S to obtain the preprocessed dialogue dataset I.

[0024] Step 12: Input the i-th data in the preprocessed dialogue dataset I into the input layer. Through serialization, indexing, and embedding, the data will obtain a real number vector hidden_state; then through positional encoding to add the position information of sequence elements, and finally through a mask matrix to make the attention scores of invalid information approach 0, and finally obtain the output vector hidden_state_llminput of the input layer; where the input layer includes five stages, namely: serialization, indexing, embedding, mask matrix, and positional encoding.

[0025] In the embodiment of the present invention, Step 11 includes: Crop all samples included in the dialogue dataset with the text length greater than 4096 characters cropped off, and then perform text normalization, including: converting to lowercase, removing special characters, handling redundant spaces, and removing spelling mistakes, to obtain the preprocessed dialogue dataset .

[0026] In the embodiment of the present invention, as Figure 2 shown, Step 12 includes: Serialization: Tokenize according to the dictionary of the tokenizer. The wordpiece tokenization method is adopted. The basic principle is to apply the input text to a pre-given vocabulary and split the given text into basic unit word pieces according to probability; In some embodiments, for example: Input: {Before my bed a pool of light, I wonder if it's frost aground. Looking up, I find the moon bright; Bowing, in homesickness I'm drowned}; Serialization: [‘Bos’, ‘bed’, ‘before’, ‘bright moon’, ‘light’,..... ‘’, ‘bowing head’, ‘thinking’, ‘hometown’].

[0027] Indexing: Use the Int32 subscript as the id of the basic unit word piece to obtain an integer array; In some embodiments, for example: Serialization: ['Bos', 'bed', 'front','moonlight', 'light',..... '', 'lower head', 'think', 'hometown']; Indexing: ['Bos', '10', '3', '5755', '809',..... '', '1354', '564', '155'].

[0028] Embedding: After the text is tokenized, the embedding continues to map each input to a real - valued vector, which is the embedding vector hidden_state; Position Encoding: For each position after tokenization, a position encoding vector is added to provide information about the position in the integer array; Position encoding is to distinguish words in different positions and provide information about the context relationship for the model.

[0029] In some embodiments, the position encoding is, for example: .

[0030] Mask Matrix: Used to control the participation of positions in the attention mechanism. In the model, the mask matrix affects the calculation of attention scores, thereby constraining what the model focuses on.

[0031] Step 2: Input the vector hidden_state_llminput into the language encoder combined with the sparse attention mechanism to improve the modeling ability for long texts, and obtain the hidden sequence hidden_state_llmout; In the embodiments of the present invention, as Figure 2 and Figure 3 shown, Step 2 includes: Step 21: Take the vector hidden_state_llminput as the input of the language encoder and input it into the language encoder; The language encoder includes the first stage to the twenty - sixth stage, and each stage includes three consecutive parts, which are in turn: a fully - connected layer, a sparse attention mechanism, and a forward - propagation layer; The modules and parameters of each stage are exactly the same, and the input of the latter stage is the output of the previous stage; After passing through the language encoder, hidden_state_llminput will obtain a hidden sequence hidden_state_llmout.

[0032] In the embodiments of the present invention, the sparse attention mechanism in Step 21 includes: Using a local window as the sparse pattern, and its attention calculation formula is as follows: ; Among them, Q is the query matrix, K is the key matrix, V is the value matrix, and M is the local window mask matrix, which is used to represent valid positions, and d k is the feature dimension of the key / query; For the local window size w, the mask matrix M is defined as: ; In the calculation, only the positions (i, j) that satisfy are subjected to the attention operation; When the input vector hidden_state_llminput enters the first stage, after passing through the fully connected layer and the sparse attention mechanism, a real number vector demo will be obtained. The weighted sum of demo and hidden_state_llminput is input into the forward propagation, and a real number vector after feature extraction can be obtained through the forward propagation layer ; is used as the input of the second stage, and then is output; is then used as the input of the third stage, and so on until the twenty-sixth stage, ~ are used as the inputs of the second stage to the twenty-sixth stage respectively, ~ are the outputs of the first stage to the twenty-fifth stage respectively, and the output of the twenty-sixth stage is the hidden sequence hidden_state_llmout.

[0033] Step 3: Input the hidden sequence hidden_state_llmout into the output layer to generate an output related to the input text.

[0034] In the embodiment of the present invention, step 3 includes: Step 31: Input the hidden sequence hidden_state_llmout into the output layer to generate the probability distribution of the next word; where the output layer includes a fully connected layer and the activation function GELU.

[0035] In the embodiment of the present invention, as Figure 2 shown, step 31 includes: Perform a weighted sum of the final output hidden_state_llmout of the language encoder and hidden_state to obtain the matrix ; The input passes through the fully connected layer and the activation function GELU to obtain the output token; the matrix Perform a weighted sum with the token, then update the input_ids. At the same time, judge that if the token sequence reaches the preset maximum length or the last token = eos_token, the iteration ends; otherwise, continue the iteration. In each iteration, the value of the token will be concatenated with the hidden_state for the next iteration.

[0036] In the embodiment of the present invention, when asking questions to the model, such as inputting "How many ethnic groups are there in China?", this input passes through the output layer and is converted into a real number vector hidden_state_llminpu. hidden_state_llminpu then enters the encoder as an input. After passing through the encoder, a capture matrix hidden_state_llmout will be obtained. hidden_state_llmout then passes through the output layer to obtain the output: "There are 56 ethnic groups in China. Among them, the Han ethnic group has the largest population, accounting for about 91.11% of the total population, and the remaining 55 ethnic minorities account for 8.89% of the total population."

[0037] The present invention adopts a sparse attention mechanism, which is a technology that optimizes the traditional global attention mechanism and achieves more efficient performance by reducing the number of attention points calculated. Compared with the global attention, the sparse attention mechanism has significant advantages in long sequence modeling, resource utilization, and feature capture.

[0038] In the technical solution provided by the present invention, the method includes inputting the processed data into the input layer of a natural language processing model based on transformer, converting the text in the data into vectors through the input layer; inputting the vectors into a language encoder combined with a sparse attention mechanism to improve the long text modeling ability and obtain a hidden sequence; inputting the hidden sequence into the output layer to generate an output related to the input text. This method enhances the understanding ability of long texts, reduces the demand for video memory of the local window mechanism, enables ordinary hardware to support a larger input scale, and brings a better experience to users.

[0039] Each step of the embodiment of the present invention can be executed by an electronic device. Among them, the electronic device includes but is not limited to mobile phones, tablet computers, portable PCs, desktop computers, etc.

[0040] The embodiment of the present invention provides a computer-readable storage medium. The computer-readable storage medium includes a stored program. Among them, when the program runs, it controls the electronic device where the computer-readable storage medium is located to execute the embodiment of the above-mentioned natural language processing method based on transformer.

[0041] Figure 4 It is a schematic diagram of an electronic device provided for the embodiment of the present invention, such as Figure 4As shown, the electronic device 21 includes: a processor 211, a memory 212, and a computer program 213 stored in the memory 212 and executable on the processor 211. When the computer program 213 is executed by the processor 211, it implements the transformer-based natural language processing method in the embodiments. To avoid repetition, details are not described here one by one.

[0042] The electronic device 21 includes, but is not limited to, a processor 211 and a memory 212. Those skilled in the art can understand that Figure 4 These are merely examples of the electronic device 21 and do not constitute a limitation on the electronic device 21. It may include more or fewer components than shown in the figure, or combine certain components, or have different components. For example, the electronic device may also include input / output devices, network access devices, buses, etc.

[0043] The so-called processor 211 may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0044] The memory 212 may be an internal storage unit of the electronic device 21, such as the hard disk or memory of the electronic device 21. The memory 212 may also be an external storage device of the electronic device 21, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 21. Further, the memory 212 may also include both the internal storage unit and the external storage device of the electronic device 21. The memory 212 is used to store the computer program and other programs and data required by the network device. The memory 212 may also be used to temporarily store data that has been output or will be output.

[0045] Those skilled in the art can clearly understand that for the sake of convenience and brevity of description, the specific working processes of the above-described systems, devices, and units can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated here.

[0046] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A natural language processing method based on transformer, characterized in that: The method comprises: Step 1: Input the processed data into the input layer of the transformer-based natural language processing model, and convert the text in the data into a vector hidden_state_llminput through the input layer; Step 2: Input the vector hidden_state_llminput into the language encoder combined with the sparse attention mechanism to improve the modeling ability of long texts and obtain the hidden sequence hidden_state_llmout; Step 3: Input the hidden sequence hidden_state_llmout into the output layer to generate output related to the input text.

2. The method according to claim 1, characterized in that The step 1 comprises: Step 11: dialogue data set S, preprocessing the data set S to obtain a preprocessed dialogue data set I; Step 12: Input the i-th data in the preprocessed dialogue dataset I into the input layer. The data will be serialized, indexed and embedded to obtain a real vector hidden_state; then, through position encoding, the position information of the sequence elements is added, and finally, through the mask matrix, the attention score of the invalid information is approached to 0, and finally the output vector hidden_state_llminput of the input layer is obtained; the input layer includes five stages, namely: serialization, indexing, embedding, mask matrix and position encoding.

3. The method according to claim 1, characterized in that The step 2 comprises: Step 21. Use the vector hidden_state_llminput as the input of the language encoder and input it into the language encoder; the language encoder includes the first stage to the twenty-sixth stage, each stage includes three consecutive parts, which are: fully connected layer, sparse attention mechanism and forward propagation layer; the modules and parameters of each stage are exactly the same, and the input of the latter stage is the output of the previous stage; after hidden_state_llminput passes through the language encoder, a hidden sequence hidden_state_llmout will be obtained.

4. The method according to claim 1, characterized in that: The step 3 comprises: Step 31: Input the hidden sequence hidden_state_llmout to the output layer to generate the probability distribution of the next word; the output layer includes a fully connected layer and an activation function GELU.

5. The method according to claim 1, characterized in that: The step 11 comprises: All samples included in the dialogue dataset are trimmed, and texts with a length greater than 4096 characters are trimmed. Then, text normalization is performed, including: converting to lowercase, removing special characters, processing extra spaces, and removing spelling errors, to obtain the preprocessed dialogue dataset.

6. The method according to claim 1, characterized in that The step 12 comprises: Serialization: Segment words according to the dictionary of the word segmenter. The mode adopts the wordpiece segmentation method. The basic principle is to apply the input text to a pre-given vocabulary and divide the given text into basic unit word pieces according to probability; Indexing: Use the subscript of Int32 as the id of the basic unit word fragment to get an integer array; Embedding: After the text is segmented, embedding continues to map each input into a real number vector, which is the embedded vector hidden_state; Position encoding: For each position after word segmentation, a position encoding vector is added to provide information about the position in the integer array; position encoding is used to distinguish words in different positions and provide the model with contextual information. Mask matrix: used to control the position participation calculation in the attention mechanism. In the model, the mask matrix affects the calculation of the attention score, thereby constraining the content that the model focuses on.

7. The method according to claim 1, characterized in that The sparse attention mechanism in step 21 includes: Using the local window as the sparse mode, the attention calculation formula is as follows: ； Among them, Q is the query matrix, K is the key matrix, V is the value matrix, M is the local window mask matrix, which is used to represent the valid position, d k is the feature dimension of the key / query; For a local window size w, the mask matrix M is defined as: ； In the calculation, only Perform attention operation at the position (i, j); When the input vector hidden_state_llminput enters the first stage, it passes through the fully connected layer and the sparse attention mechanism to obtain a real vector demo. The weighted sum of demo and hidden_state_llminput is input into the forward propagation. Through the forward propagation layer, a real vector after feature extraction can be obtained. ;Will As the input of the second stage, the output ; then As the input of the third stage, this is repeated until the twenty-sixth stage. ~ As the input of the second stage to the twenty-sixth stage respectively, ~ They are respectively used as the outputs of the first stage to the twenty-fifth stage, and the output of the twenty-sixth stage is the hidden sequence hidden_state_llmout.

8. The method according to claim 1, characterized in that The step 31 comprises: The final output of the language encoder, hidden_state_llmout and hidden_state, are weighted and summed to obtain the matrix ; The input passes through the fully connected layer and the activation function GELU to get the output token; the matrix Perform weighted summation with token, then update input_ids. At the same time, if the token sequence reaches the preset maximum length or the last token = eos_token, the iteration ends; otherwise, continue to iterate, and each iteration token value will be concatenated with hidden_state for the next iteration.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the transformer-based natural language processing method according to any one of claims 1 to 8.

10. An electronic device, characterized in that: include: one or more processors; Memory; And one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the device, enable the device to perform the transformer-based natural language processing method described in any one of claims 1 to 8.

Images