Natural language processing method based on improved Llama model

Abstract

The present invention relates to the technical field of natural language processing, and in particular, provides a natural language processing method based on an improved Llama model. The method comprises obtaining conversation data S, preprocessing data S to obtain preprocessed conversation data M, and constructing an improved Llama model comprising an input layer, a language encoder, and an output layer. The preprocessed conversation data M is input into the improved Llama model to obtain output text. By utilizing the efficient and lightweight pre-trained language model Llama, the method reduces training costs and inference complexity while maintaining powerful language generation capabilities, thereby improving model stability.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a natural language processing method based on an improved Llama model. Background Art

[0002] In recent years, the field of natural language processing (NLP) has made significant progress with the development of deep learning technology. Large-scale pre-trained language models have become an important technical means to promote the performance of NLP tasks.

[0003] Early NLP systems relied on hand-designed rules and finite state machines, such as parsers and word segmenters. This approach was effective in specific domains, but lacked generalizability and was costly to develop. With the introduction of statistical language models (such as N-grams) and machine learning algorithms, NLP systems began to utilize data-driven approaches to model language. However, these approaches have limited performance in high-dimensional contexts. Summary of the Invention

[0004] In view of this, the present invention provides a natural language processing method based on an improved Llama model, which is used to reduce training costs and reasoning complexity and improve model stability while maintaining powerful language generation capabilities.

[0005] In a first aspect, the present invention provides a natural language processing method based on an improved Llama model, the method comprising:

[0006] Step 1: Acquire conversation data S, preprocess the data S, and obtain preprocessed conversation data M;

[0007] Step 2: Construct an improved Llama model, wherein the improved Llama model includes an input layer, a language encoder, and an output layer. The preprocessed conversation data M is input into the improved Llama model to obtain output text.

[0008] Optionally, step 1 includes:

[0009] The conversation data S is preprocessed, including converting to lowercase, removing special characters, processing extra spaces, and removing spelling errors, to obtain preprocessed conversation data M.

[0010] Optionally, step 2 includes:

[0011] Step 21: Input the preprocessed conversation data M into the input layer to obtain a real number vector ; The input layer consists of two stages: word segmentation layer and embedding layer;

[0012] Step 22: Real vector Obtain hidden state sequence through language encoder The encoder consists of the first to twentieth stages, each of which consists of four consecutive parts: global and local fusion attention mechanism, normalization, forward propagation, and normalization. The modules and parameters of each stage are exactly the same, and the input of the next stage is the output of the previous stage.

[0013] Step 23: Hidden state sequence Input to the output layer to generate the probability distribution of the next word; the output layer includes the projection layer and the de-tagged layer.

[0014] Optionally, step 21 includes:

[0015] Word segmentation layer: After a piece of text is input, it is tokenized and divided into words or characters to form a token sequence. After obtaining the token sequence, the text is mapped into an input form understood by the model, and the text sequence is converted into an integer index sequence, where the index is the index of the word or character in the corpus;

[0016] Embedding: After the text passes through the word segmentation layer, embedding continues to map each token into a real number vector, which is the embedding vector .

[0017] Optionally, step 22 includes:

[0018] Using the global and local fusion attention mechanism, the input is First, Enter the global attention mechanism and local attention mechanism respectively;

[0019] The calculation formula of the global attention mechanism is:

[0020] ;

[0021] Among them, Q, K, and V are query, key, and value matrices respectively; is the dimension of the key, used as a scaling factor;

[0022] The calculation formula of the local attention mechanism is:

[0023] ;

[0024] in, Represents a local window range, selecting only the key K and value V at adjacent positions in the sequence for calculation;

[0025] Computational complexity: From the global Reduced to , where w is the window size and m is the sequence length;

[0026] The fusion formula is:

[0027] ;

[0028] in is the output weight matrix;

[0029] Take n_1 as the input of the first layer of the encoder, n_1 is obtained through the global and local attention mechanism , then Normalize to get V1, input V1 into the forward propagation layer, get output V2, V2 is normalized to get V3, perform weighted summation on V3 and V1, and reassign the obtained matrix to ,Will As the final output of the first stage;

[0030] The normalization steps are as follows:

[0031] (1) Calculate the root mean square, the formula is:

[0032] ;

[0033] (2) Normalize each data point using the root mean square value, the formula is:

[0034] ;

[0035] in is each value in the data, and n is the number of data points;

[0036] As the output of the first stage, As the input of the second stage, stage two goes through the same process as stage one and then outputs ; then As the input of the third stage, this is repeated until the twentieth stage. ~ As inputs for the second to twentieth stages respectively, ~ As the output of the first to tenth stages respectively, the final output is obtained .

[0037] Optionally, step 23 includes:

[0038] The final output of the encoder Input the projection layer, which can convert the matrix into a text ID sequence, and then input the obtained ID sequence into the de-labeler, which converts the ID sequence into text, and finally obtains a text output.

[0039] In a second aspect, an embodiment of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the natural language processing method based on the improved Llama model in the first aspect or any possible implementation of the first aspect.

[0040] In a third aspect, an embodiment of the present invention provides an electronic device comprising: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the device, enable the device to execute the natural language processing method based on the improved Llama model in the first aspect or any possible implementation of the first aspect.

[0041] In the technical solution provided by the present invention, the method includes obtaining conversation data S, preprocessing the data S to obtain preprocessed conversation data M; constructing an improved Llama model, the improved Llama model includes: an input layer, a language encoder, and an output layer, inputting the preprocessed conversation data set M into the improved Llama model to obtain output text. This method uses an efficient and lightweight pre-trained language model Llama to reduce training costs and inference complexity while maintaining powerful language generation capabilities, thereby improving model stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 A flowchart of a natural language processing method based on an improved Llama model provided in an embodiment of the present invention;

[0044] Figure 2 A schematic diagram of an improved Llama model provided in an embodiment of the present invention;

[0045] Figure 3 A schematic diagram of a speech encoder layer provided by an embodiment of the present invention;

[0046] Figure 4A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0048] It should be understood that the embodiments described are only a portion of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative work are within the scope of protection of the present invention.

[0049] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "the" and "the" used in the embodiments of the present invention are also intended to include plural forms, unless the context clearly indicates other meanings.

[0050] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. Furthermore, the character " / " in this document generally indicates an "or" relationship between the associated objects.

[0051] The word "if," as used herein, may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0052] Figure 1 A flowchart of a natural language processing method based on an improved Llama model provided in an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method includes:

[0053] Step 1: Acquire conversation data S, preprocess the data S, and obtain preprocessed conversation data M;

[0054] In the embodiment of the present invention, step 1 includes:

[0055] The conversation data S is preprocessed, including converting to lowercase, removing special characters, processing extra spaces, and removing spelling errors, to obtain preprocessed conversation data M.

[0056] One million pre-processed conversation data were collected and divided for training. The specific operation was as follows: the first 50% of the data set was divided for model pre-training; the remaining 50% of the question data was used as the test set, and the answer data was used as the validation set.

[0057] Step 2: Construct an improved Llama model, wherein the improved Llama model includes an input layer, a language encoder, and an output layer. The preprocessed conversation data M is input into the improved Llama model to obtain output text.

[0058] In the embodiment of the present invention, Figure 2 and Figure 3 As shown, step 2 includes:

[0059] Step 21: Input the preprocessed conversation data M into the input layer to obtain a real number vector ; The input layer consists of two stages: word segmentation layer and embedding layer;

[0060] In the embodiment of the present invention, step 21 includes:

[0061] Word segmentation layer: After a piece of text is input, it is tokenized and divided into words or characters to form a token sequence. After obtaining the token sequence, the text is mapped into an input form understood by the model, and the text sequence is converted into an integer index sequence, where the index is the index of the word or character in the corpus;

[0062] When segmenting words, llama's word segmenter is used. In some embodiments, for example:

[0063] enter:

[0064] {The sun sets behind the mountains, the Yellow River flows into the sea. To see a thousand miles, one must climb to a higher level};

[0065] Token sequence:

[0066] ['Bos', 'white', 'sun', 'depending on', 'mountain', .... ', 'up', 'a floor', 'floor'].

[0067] Index sequence:

[0068] ['Bos', '10', '3', '5755', '809', .....'', '1354', '564', '155'].

[0069] Embedding: After the text passes through the word segmentation layer and is converted into an index sequence, embedding continues to map each token into a real number vector, which is the embedding vector n_1;

[0070] In some embodiments, the embedding vector n_1 is such as:

[0071] .

[0072] Step 22: Real vector Obtain hidden state sequence through language encoder The encoder consists of the first to twentieth stages, each of which consists of four consecutive parts: global and local fusion attention mechanism, normalization, forward propagation, and normalization. The modules and parameters of each stage are exactly the same, and the input of the next stage is the output of the previous stage.

[0073] In the embodiment of the present invention, step 22 includes:

[0074] Using the global and local fusion attention mechanism, the input is First, Enter the global attention mechanism and local attention mechanism respectively;

[0075] The calculation formula of the global attention mechanism is:

[0076] ;

[0077] Among them, Q, K, and V are query, key, and value matrices respectively; is the dimension of the key, used as a scaling factor;

[0078] The calculation formula of the local attention mechanism is:

[0079] ;

[0080] in, Represents a local window range, selecting only the key K and value V at adjacent positions in the sequence for calculation;

[0081] Computational complexity: From the global Reduced to , where w is the window size and m is the sequence length;

[0082] The fusion formula is:

[0083] ;

[0084] in is the output weight matrix;

[0085] Take n_1 as the input of the first layer of the encoder, n_1 is obtained through the global and local attention mechanism , then Normalize to get V1, input V1 into the forward propagation layer, get output V2, V2 is normalized to get V3, perform weighted summation on V3 and V1, and reassign the obtained matrix to ,Will As the final output of the first stage.

[0086] The normalization steps are as follows:

[0087] (1) Calculate the root mean square, the formula is:

[0088] ;

[0089] (2) Normalize each data point using the root mean square value, the formula is:

[0090] ;

[0091] in is each value in the data, and n is the number of data points;

[0092] As the output of the first stage, As the input of the second stage, stage two goes through the same process as stage one and then outputs ; then As the input of the third stage, this is repeated until the twentieth stage. ~ As inputs for the second to twentieth stages respectively, ~ As the output of the first to tenth stages respectively, the final output is obtained .

[0093] In the embodiment of the present invention, step 23 includes:

[0094] The final output of the encoder Input the projection layer, which can convert the matrix into a text ID sequence, and then input the obtained ID sequence into the de-labeler, which converts the ID sequence into text, and finally obtains a text output.

[0095] In this embodiment of the present invention, when the model is asked a question, the input is "What is your favorite movie?" This input is converted into a real number vector through the output layer. , Then it enters the encoder as an input, and after passing through the encoder, a capture matrix is ​​obtained ,matrix After passing through the output layer, the output is "My favorite movie genres are science fiction and horror movies, such as "Interstellar" and "Avengers 4: Endgame". In addition, I also like to watch suspense movies and comedies, such as "Mad Max 4" and "Detective Chinatown 3".

[0096] This invention aims to balance global and local information, avoiding the limitations of a single attention model while significantly reducing complexity in long-sequence tasks while maintaining high performance. Furthermore, the research will explore the integration of various plug-ins to further enhance the model's functionality and flexibility. Ultimately, experiments will verify the effectiveness of the proposed method in improving inference efficiency and accuracy, providing new insights and practical experience for the field of natural language processing.

[0097] Each step of the embodiment of the present invention may be performed by an electronic device, including but not limited to a mobile phone, a tablet computer, a portable PC, a desktop computer, etc.

[0098] In the technical solution provided by the present invention, the method includes obtaining conversation data S, preprocessing the data S to obtain preprocessed conversation data M; constructing an improved Llama model, the improved Llama model includes: an input layer, a language encoder, and an output layer, inputting the preprocessed conversation data M into the improved Llama model to obtain output text. This method uses an efficient and lightweight pre-trained language model Llama to reduce training costs and inference complexity while maintaining powerful language generation capabilities, thereby improving model stability.

[0099] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program. When the program runs, the electronic device where the computer-readable storage medium is located is controlled to execute the above-mentioned embodiment of the natural language processing method based on the improved Llama model.

[0100] Figure 4 A schematic diagram of an electronic device provided by an embodiment of the present invention is shown in FIG. Figure 4 As shown, the electronic device 21 includes: a processor 211, a memory 212, and a computer program 213 stored in the memory 212 and executable on the processor 211. When the computer program 213 is executed by the processor 211, the natural language processing method based on the improved Llama model in the embodiment is implemented. To avoid repetition, they are not described here one by one.

[0101] The electronic device 21 includes, but is not limited to, a processor 211 and a memory 212. Those skilled in the art will understand that Figure 4It is only an example of the electronic device 21 and does not constitute a limitation of the electronic device 21. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0102] The processor 211 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0103] The memory 212 can be an internal storage unit of the electronic device 21, such as the hard drive or memory of the electronic device 21. The memory 212 can also be an external storage device of the electronic device 21, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the electronic device 21. Furthermore, the memory 212 can include both the internal storage unit of the electronic device 21 and an external storage device. The memory 212 is used to store computer programs and other programs and data required by the network device. The memory 212 can also be used to temporarily store data that has been output or is about to be output.

[0104] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0105] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A natural language processing method based on an improved Llama model, characterized in that: The method comprises: Step 1: Acquire conversation data S, preprocess the data S, and obtain preprocessed conversation data M; Step 2: construct an improved Llama model, wherein the improved Llama model includes: an input layer, a language encoder, and an output layer, and input the preprocessed conversation data M into the improved Llama model to obtain output text; The step 2 includes: Step 21: Input the preprocessed conversation data M into the input layer to obtain a real number vector ; The input layer consists of two stages: word segmentation layer and embedding layer; Step 22: Real vector Obtain hidden state sequence through language encoder The encoder consists of the first to twentieth stages, each of which consists of four consecutive parts: global and local fusion attention mechanism, normalization, forward propagation, and normalization. The modules and parameters of each stage are exactly the same, and the input of the next stage is the output of the previous stage. Step 23: Hidden state sequence Input to the output layer to generate the probability distribution of the next word; the output layer includes the projection layer and the de-tokenizer; The step 22 includes: Using the global and local fusion attention mechanism, the input is First, Enter the global attention mechanism and local attention mechanism respectively; The calculation formula of the global attention mechanism is: ； Among them, Q, K, and V are query, key, and value matrices respectively; is the dimension of the key, used as a scaling factor; The calculation formula of the local attention mechanism is: ； in, Represents a local window range, selecting only the key K and value V at adjacent positions in the sequence for calculation; Computational complexity: From the global Reduced to , where w is the window size and m is the sequence length; The fusion formula is: ； in is the output weight matrix; Take n_1 as the input of the first layer of the encoder, n_1 is obtained through the global and local attention mechanism , then Normalize to get V1, input V1 into the forward propagation layer, get output V2, V2 is normalized to get V3, perform weighted summation on V3 and V1, and reassign the obtained matrix to ,Will As the final output of the first stage; As the output of the first stage, As the input of the second stage, stage two goes through the same process as stage one and then outputs ; then As the input of the third stage, this is repeated until the twentieth stage. ~ As inputs for the second to twentieth stages respectively, ~ As the output of the first to tenth stages respectively, the final output is obtained .

2. The method according to claim 1, characterized in that The step 1 comprises: The conversation data S is preprocessed, including converting to lowercase, removing special characters, processing extra spaces, and removing spelling errors, to obtain preprocessed conversation data M.

3. The method according to claim 1, characterized in that The step 21 includes: Word segmentation layer: After a piece of text is input, it is tokenized and divided into words or characters to form a token sequence. After obtaining the token sequence, the text is mapped into an input form understood by the model, and the text sequence is converted into an integer index sequence, where the index is the index of the word or character in the corpus; Embedding: After the text passes through the word segmentation layer, embedding continues to map each token into a real number vector as the embedding vector n_1.

4. The method according to claim 1, wherein The normalization steps are as follows: (1) Calculate the root mean square, the formula is: ； (2) Normalize each data point using the root mean square value, the formula is: ； in is each value in the data, and n is the number of data points.

5. The method according to claim 1, characterized in that The step 23 includes: The final output of the encoder Input the projection layer, which can convert the matrix into a text ID sequence, and then input the obtained ID sequence into the de-labeler, which converts the ID sequence into text, and finally obtains a text output.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the natural language processing method based on the improved Llama model according to any one of claims 1 to 5.

7. An electronic device, characterized in that: include: one or more processors; Memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the device, enable the device to perform the natural language processing method based on the improved Llama model described in any one of claims 1 to 5.

Images