A method to improve the accuracy of language processing models

Abstract

The present invention relates to the field of natural language processing technology, and in particular, provides a method for improving the accuracy of a language processing model. The method comprises obtaining a conversation data set S, performing data preprocessing on the data set S, and obtaining a preprocessed conversation data set I; constructing a LittleLlama network structure, the LittleLlama network structure comprising: an input layer, a Transformer encoder, an output layer, and converting the i-th item I in the preprocessed conversation data set I into a i The input is into the LittleLlama network structure to obtain the output text. This method effectively processes time series data. On the basis of maintaining causality, it optimizes the prediction performance of the model, avoids the interference of future information on current predictions, reduces prediction errors, and meets the needs of practical applications.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a method for improving the accuracy of a language processing model. Background Art

[0002] In many time series tasks, such as language modeling, time series forecasting, and dynamic system control, it is crucial to ensure that the model only has access to information from previous time steps when predicting the output at the current time step.

[0003] Traditional sequence models, such as recurrent neural networks and long short-term memory networks, often face the problem of information leakage when processing time series data, that is, the model may accidentally use future input information when making predictions. The non-causal nature of this information may cause the model to produce inaccurate prediction results, especially in application scenarios that require strict adherence to time order. Summary of the invention

[0004] In view of this, the present invention provides a method for improving the accuracy of language processing models to effectively process time series data, optimize the prediction performance of the model while maintaining causality, avoid interference of future information on current predictions, and reduce prediction errors to meet the needs of practical applications.

[0005] In a first aspect, the present invention provides a method for improving the accuracy of a language processing model, the method comprising:

[0006] Step 1: Get the conversation dataset , Represents the first Items of data, , perform data preprocessing on the data set S to obtain the preprocessed dialogue data set ;

[0007] Step 2: Construct a LittleLlama network structure, which includes an input layer, a Transformer encoder, and an output layer. Input into the LittleLlama network structure and get the output text.

[0008] Optionally, step 1 includes:

[0009] The conversation dataset All samples included in the preprocessed dialogue dataset are trimmed, and texts with a length greater than 4096 characters are trimmed. Then, text normalization is performed, including: converting to lowercase, removing special characters, processing extra spaces, and removing spelling errors. .

[0010] Optionally, the step 2 includes:

[0011] Step 21, input the i-th data in the conversation data set S into the input layer to obtain a real vector logger; the input layer includes four stages, namely: serialization, indexing, embedding and rotation position encoding;

[0012] Step 22, pass the real vector logger through the Transformer encoder to obtain the hidden state sequence X; wherein the Transformer encoder includes the first stage to the eighteenth stage, each stage includes five consecutive parts, which are: fused root mean square normalization layer, causal attention mechanism layer, regularized dropout layer, multi-layer perceptron layer and regularized dropout layer, the composition order and parameters of each stage are exactly the same, and the input of the latter stage is the output of the previous stage;

[0013] Step 23: Input the hidden sequence X into the output layer to generate the probability distribution of the next word; wherein the output layer includes a fused RMS normalization layer and a fully connected layer.

[0014] Optionally, the step 21 includes:

[0015] Serialization: After a piece of text is input, it needs to be tokenized and divided into words or characters to form a token sequence. When segmenting, the llama tokenizer is used;

[0016] Indexing: After obtaining the token sequence, the text is mapped into an input form that the model understands, and the text sequence is converted into an integer index sequence, where the index is the index of the word or character in the corpus;

[0017] Embedding: After the text is indexed, embedding continues to map each token into a real number vector, which is the embedding vector.

[0018] Rotational position encoding: For each position in the token sequence, add a position encoding vector to provide information about the position of the token in the sequence; position encoding is used to distinguish tokens at different positions and provide the model with contextual information.

[0019] Optionally, the step 22 includes:

[0020] The steps of fusion RMS normalization are as follows:

[0021] (1) Calculate the root mean square RMS. Calculate the root mean square for each feature of the input tensor x. The formula is:

[0022] ;

[0023] Where N is the dimension of the feature, is a very small constant used to prevent division by zero errors;

[0024] (2) Normalization: Use the calculated RMS to normalize the input so that each feature of the output has the same scale. The formula is:

[0025] ;

[0026] (3) Scaling and offset: adding scaling parameters and offset parameters , the formula is:

[0027] ;

[0028] in, and is a learning parameter with the same feature dimension;

[0029] The causal attention mechanism adopts a multi-head attention mechanism fused with a multi-head attention mechanism. Assume that the input sequence is X, and its dimension is (N, T, d), where N is the batch size, T is the sequence length, and d is the feature dimension; first, the input X is linearly transformed into query Q, key K, and value V, and its formula is:

[0030] ;

[0031] in, , , is the weight matrix;

[0032] Divide the query Q, key K and value V into h heads, and the formula is:

[0033] ;

[0034] ;

[0035] ;

[0036] For each head i, the attention weight is calculated as follows:

[0037] ;

[0038] Where M is the causal masking matrix, ensuring that the jth position only pays attention to the previous j positions;

[0039] The weighted sum is calculated using the attention weights, and the formula is:

[0040] ;

[0041] The output of all heads is concatenated, and the formula is:

[0042] ;

[0043] Finally, the concatenated result is linearly transformed, and the formula is:

[0044] ;

[0045] in, is the output weight matrix, with dimension ;

[0046] When the input vector passes through all layers of the first stage, a real number vector after feature extraction is obtained ;Will As the input of the second stage, the output ; then As the input of the third stage, this is repeated until the eighteenth stage. ~ As the input of the second stage to the eighteenth stage, ~ As the output of the first to eighteenth stages respectively, the final output is obtained .

[0047] Optionally, the step 23 includes:

[0048] The final output of the encoder is input into the fusion RMS normalization function to obtain the matrix ;

[0049] The resulting matrix , input to the fully connected layer, and get the matrix prediction result, that is, output a text.

[0050] In a second aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute a method for improving the accuracy of a language processing model in the first aspect or any possible implementation of the first aspect.

[0051] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the device, enable the device to execute a method for improving the accuracy of a language processing model in the first aspect or any possible implementation of the first aspect.

[0052] In the technical solution provided by the present invention, the method includes obtaining a conversation data set S, performing data preprocessing on the data set S to obtain a preprocessed conversation data set I; constructing a LittleLlama network structure, the LittleLlama network structure includes: an input layer, a Transformer encoder, an output layer, and converting the i-th item I in the preprocessed conversation data set I into a i The input is into the LittleLlama network structure to obtain the output text. This method effectively processes time series data. On the basis of maintaining causality, it optimizes the prediction performance of the model, avoids the interference of future information on current predictions, reduces prediction errors, and meets the needs of practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 A flow chart of a method for improving the accuracy of a language processing model provided by an embodiment of the present invention;

[0055] Figure 2 A schematic diagram of a LittleLlama network model provided by an embodiment of the present invention;

[0056] Figure 3 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] It should be clear that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention are also intended to include plural forms, unless the context clearly indicates other meanings.

[0060] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0061] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0062] Figure 1 A flowchart of a method for improving the accuracy of a language processing model provided by an embodiment of the present invention, such as Figure 1 As shown, the method includes:

[0063] Step 1: Get the conversation dataset , Represents the first Items of data, , perform data preprocessing on the data set S to obtain the preprocessed dialogue data set .

[0064] In the embodiment of the present invention, step 1 includes:

[0065] The conversation dataset All samples included in the preprocessed dialogue dataset are trimmed, and texts with a length greater than 4096 characters are trimmed. Then, text normalization is performed, including: converting to lowercase, removing special characters, processing extra spaces, and removing spelling errors. .

[0066] The preprocessed dialogue data set I is divided for training. The specific operation is as follows: the first 50% of the data set is divided for model training; the remaining 50% of the question data is used as a test set, and the answer data is used as a validation set.

[0067] Step 2: Construct a LittleLlama network structure, which includes an input layer, a Transformer encoder, and an output layer. i Input into the LittleLlama network structure and get the output text.

[0068] In the embodiment of the present invention, Figure 2 As shown, step 2 includes:

[0069] Step 21: Input the i-th data in the conversation data set S into the input layer to obtain a real vector logger; the input layer includes four stages, namely: serialization, indexing, embedding and rotation position encoding.

[0070] In the embodiment of the present invention, step 21 includes:

[0071] Serialization: After a piece of text is input, it needs to be tokenized and divided into words or characters to form a token sequence. When segmenting, the llama tokenizer is used;

[0072] In some embodiments, for example:

[0073] enter:

[0074] {Have you not seen that the water of the Yellow River comes from the sky, rushes to the sea and never returns};

[0075] Serialization:

[0076] ['Bos', 'Jun', 'don't see', 'Yellow River', 'zhi', ..... '', 'not', 'return', 'return'].

[0077] Indexing: After obtaining the token sequence, the text is mapped into an input form that the model understands, and the text sequence is converted into an integer index sequence, where the index is the index of the word or character in the corpus;

[0078] In some embodiments, for example:

[0079] Serialization:

[0080] ['Bos', 'Jun', 'not seen', 'Yellow River', 'zhi', ..... '', 'not', 'return', 'return'];

[0081] Indexing:

[0082] ['Bos', '10', '3', '5755', '809', .....'', '1354', '564', '155'].

[0083] Embedding: After the text is indexed, embedding continues to map each token into a real number vector, which is the embedding vector.

[0084] In some embodiments, the embedding vector is, for example:

[0085] .

[0086] Rotational position encoding: For each position in the token sequence, add a position encoding vector to provide information about the position of the token in the sequence; position encoding is used to distinguish tokens at different positions and provide the model with contextual information.

[0087] In some embodiments, the position code is, for example:

[0088] .

[0089] Step 22: Pass the real vector logger through the Transformer encoder to obtain the hidden state sequence X; the Transformer encoder includes the first stage to the eighteenth stage, each stage includes five consecutive parts, which are: fused root mean square normalization layer, causal attention mechanism layer, regularized dropout layer, multi-layer perceptron layer and regularized dropout layer, the composition order and parameters of each stage are exactly the same, and the input of the latter stage is the output of the previous stage.

[0090] In the embodiment of the present invention, step 22 includes:

[0091] The steps of fusion RMS normalization are as follows:

[0092] (1) Calculate the root mean square RMS. Calculate the root mean square for each feature of the input tensor x. The formula is:

[0093] ;

[0094] Where N is the dimension of the feature, is a very small constant used to prevent division by zero errors;

[0095] (2) Normalization: Use the calculated RMS to normalize the input so that each feature of the output has the same scale. The formula is:

[0096] ;

[0097] (3) Scaling and offset: adding scaling parameters and offset parameters , the formula is:

[0098] ;

[0099] in, and is a learning parameter with the same feature dimension;

[0100] The causal attention mechanism adopts a multi-head attention mechanism fused with a multi-head attention mechanism. Assume that the input sequence is X, and its dimension is (N, T, d), where N is the batch size, T is the sequence length, and d is the feature dimension; first, the input X is linearly transformed into query Q, key K, and value V, and its formula is:

[0101] ;

[0102] in, , , is the weight matrix;

[0103] Divide the query Q, key K and value V into h heads, and the formula is:

[0104] ;

[0105] ;

[0106] ;

[0107] For each head i, the attention weight is calculated as follows:

[0108] ;

[0109] Where M is the causal masking matrix, ensuring that the jth position only pays attention to the previous j positions;

[0110] The weighted sum is calculated using the attention weights, and the formula is:

[0111] ;

[0112] The output of all heads is concatenated, and the formula is:

[0113] ;

[0114] Finally, the concatenated result is linearly transformed, and the formula is:

[0115] ;

[0116] in, is the output weight matrix, with dimension ;

[0117] When the input vector passes through all layers of the first stage, a real number vector after feature extraction is obtained ;Will As the input of the second stage, and then output ; then As the input of the third stage, this is repeated until the eighteenth stage. ~ As the input of the second stage to the eighteenth stage, ~ As the output of the first to eighteenth stages respectively, the final output is obtained .

[0118] Step 23: Input the hidden sequence X into the output layer to generate the probability distribution of the next word; wherein the output layer includes a fused RMS normalization layer and a fully connected layer.

[0119] In the embodiment of the present invention, step 23 includes:

[0120] The final output of the encoder is input into the fusion RMS normalization function to obtain the matrix ;

[0121] The resulting matrix , input to the fully connected layer, and get the matrix prediction result, that is, output a text.

[0122] In the embodiment of the present invention, when the model is asked a question, the input is 'Who is the writer and director of Star Wars: Episode IX?'. This input is converted into a real vector logger through the output layer. The logger then enters the encoder as an input. After passing through the encoder, a capture matrix W is obtained. 18 , W 18 After passing through the output layer, the output is 'Liam Neeson.'

[0123] The present invention adopts an attention mechanism that combines the causal attention mechanism and the multi-head attention mechanism. This model ensures that when predicting the output of the current time step, only the previous time step information can be accessed. This is necessary for many time series tasks (such as language modeling) and avoids the interference of future information on the current prediction.

[0124] Each step of the embodiment of the present invention may be performed by an electronic device, which includes but is not limited to a mobile phone, a tablet computer, a portable PC, a desktop computer, etc.

[0125] In the technical solution provided by the present invention, the method includes obtaining a conversation data set S, performing data preprocessing on the data set S to obtain a preprocessed conversation data set I; constructing a LittleLlama network structure, the LittleLlama network structure includes: an input layer, a Transformer encoder, an output layer, and converting the i-th item I in the preprocessed conversation data set I into a i The input is into the LittleLlama network structure to obtain the output text. This method effectively processes time series data. On the basis of maintaining causality, it optimizes the prediction performance of the model, avoids the interference of future information on current predictions, reduces prediction errors, and meets the needs of practical applications.

[0126] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the electronic device where the computer-readable storage medium is located is controlled to execute an embodiment of the above-mentioned method for improving the accuracy of a language processing model.

[0127] Figure 3 A schematic diagram of an electronic device provided by an embodiment of the present invention, such as Figure 3 As shown, the electronic device 21 includes: a processor 211, a memory 212, and a computer program 213 stored in the memory 212 and executable on the processor 211. When the computer program 213 is executed by the processor 211, an embodiment of a method for improving the accuracy of a language processing model in an embodiment is implemented. To avoid repetition, they are not described one by one here.

[0128] The electronic device 21 includes, but is not limited to, a processor 211 and a memory 212. Those skilled in the art will appreciate that Figure 3 It is only an example of the electronic device 21 and does not constitute a limitation of the electronic device 21. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0129] The processor 211 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0130] The memory 212 may be an internal storage unit of the electronic device 21, such as a hard disk or memory of the electronic device 21. The memory 212 may also be an external storage device of the electronic device 21, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card (FlashCard), etc. equipped on the electronic device 21. Further, the memory 212 may also include both an internal storage unit of the electronic device 21 and an external storage device. The memory 212 is used to store computer programs and other programs and data required by network devices. The memory 212 may also be used to temporarily store data that has been output or is to be output.

[0131] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0132] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for improving the accuracy of a language processing model, characterized in that: The method comprises: Step 1: Get the conversation dataset , Represents the first Items of data, , perform data preprocessing on the data set S to obtain the preprocessed dialogue data set ; Step 2: Construct a LittleLlama network structure, which includes an input layer, a Transformer encoder, and an output layer. Input into the LittleLlama network structure and get the output text; The step 2 comprises: Step 21, input the i-th data in the conversation data set S into the input layer to obtain a real vector logger; the input layer includes four stages, namely: serialization, indexing, embedding and rotation position encoding; Step 22, pass the real vector logger through the Transformer encoder to obtain the hidden state sequence X; wherein the Transformer encoder includes the first stage to the eighteenth stage, each stage includes five consecutive parts, which are: fused root mean square normalization layer, causal attention mechanism layer, regularized dropout layer, multi-layer perceptron layer and regularized dropout layer, the composition order and parameters of each stage are exactly the same, and the input of the latter stage is the output of the previous stage; Step 23: input the hidden sequence X into the output layer to generate the probability distribution of the next word; wherein the output layer includes a fused RMS normalization layer and a fully connected layer; The step 21 comprises: Serialization: After a piece of text is input, it needs to be tokenized and divided into words or characters to form a token sequence. When segmenting, the llama tokenizer is used; Indexing: After obtaining the token sequence, the text is mapped into an input form that the model understands, and the text sequence is converted into an integer index sequence, where the index is the index of the word or character in the corpus; Embedding: After the text is indexed, embedding continues to map each token into a real number vector, which is the embedding vector. Rotational position encoding: For each position in the token sequence, add a position encoding vector to provide information about the position of the token in the sequence; position encoding is used to distinguish tokens at different positions and provide the model with contextual information.

2. The method according to claim 1, characterized in that The step 1 comprises: The conversation dataset All samples included in the preprocessed dialogue dataset are trimmed, and texts with a length greater than 4096 characters are trimmed. Then, text normalization is performed, including: converting to lowercase, removing special characters, processing extra spaces, and removing spelling errors. .

3. The method according to claim 1, characterized in that The step 22 comprises: The steps of fusion RMS normalization are as follows: (1) Calculate the root mean square RMS. Calculate the root mean square for each feature of the input tensor x. The formula is: ； Among them, N` is the dimension of the feature, is a very small constant used to prevent division by zero errors; (2) Normalization: Use the calculated RMS to normalize the input so that each feature of the output has the same scale. The formula is: ； (3) Scaling and offset: adding scaling parameters and offset parameters , the formula is: ； in, and is a learning parameter with the same feature dimension; The causal attention mechanism adopts a multi-head attention mechanism fused with a multi-head attention mechanism. Assume that the input sequence is X, and its dimension is (N``, T, d), where N`` is the batch size, T is the sequence length, and d is the feature dimension; first, the input X is linearly transformed into query Q, key K, and value V, and its formula is: ； in, , , is the weight matrix; Divide the query Q, key K and value V into h heads, and the formula is: ； ； ； For each head , calculate the attention weight, the formula is: ； Where M is the causal masking matrix; The weighted sum is calculated using the attention weights, and the formula is: ； The output of all heads is concatenated, and the formula is: ； Finally, the concatenated result is linearly transformed, and the formula is: ； in, is the output weight matrix, with dimension ; When the input vector passes through all layers of the first stage, a real number vector after feature extraction is obtained ;Will As the input of the second stage, the output ; then As the input of the third stage, this is repeated until the eighteenth stage. ~ As the input of the second stage to the eighteenth stage, ~ As the output of the first to eighteenth stages respectively, the final output is obtained .

4. The method according to claim 1, characterized in that: The step 23 comprises: The final output of the encoder is input into the fusion RMS normalization function to obtain the matrix ; The resulting matrix , input to the fully connected layer, and get the matrix prediction result, that is, output a text.

5. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute a method for improving the accuracy of a language processing model as described in any one of claims 1 to 4.

6. An electronic device, characterized in that: include: one or more processors; Memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the device, enable the device to execute a method for improving the accuracy of a language processing model as described in any one of claims 1 to 4.

Images