User intention recognition system and method based on LLM large model

Abstract

The invention discloses a user intention recognition system and method based on an LLM large model, and the system comprises an intention pre-classification module which is used for carrying out the preprocessing of user intention input, and finding out an intention function set which is closer to the user intention input; a cue word project: performing text description of functions and parameters on the intention function, and constructing cue words; splicing and integrating cue words constructed based on the intention function and user intention input to form LLM large model input; and the LLM large model module is used for performing model reasoning to obtain an output result, large model output is a character string text which comprises a name and parameters of an intention function, and analyzing the character string text to obtain a user intention. According to the method, the user intention is recognized by adopting an LLM large model technology, a 2B model with a small parameter quantity is selected as a base model, meanwhile, the intention pre-classification module is designed to reduce the system response time, and the method has high recognition accuracy and expandability in the aspect of recognizing the user intention.

Description

Technical Field

[0001] The present invention relates to the fields of data analysis, intelligent interaction and artificial intelligence, and in particular to a user intention recognition system and method based on an LLM large model. Background Art

[0002] User intent recognition is often used in the field of human-computer interaction and is an important part of it. With the development of intelligence, the application scope of human-computer interaction is becoming wider and wider, such as smart speakers, cars, smart homes, etc.

[0003] Intent recognition is an important task in the field of natural language processing. It mainly determines the intention or purpose expressed in the sentence input by the user in order to better answer the user's questions or provide related services. Traditional practices include rule matching, entity naming and slot matching. The rule matching method identifies the intent by comparing the similarity between the user's input and the preset rule template; other practices are to first perform entity naming recognition on the user's input, and then use slot filling to identify the intent. These two methods also have many disadvantages. They are not flexible, and new intents need to be designed by themselves and then manually updated. It is difficult to capture the context and the semantic understanding is insufficient, resulting in low recognition accuracy. Summary of the invention

[0004] In order to solve the deficiencies in the prior art, the purpose of the present invention is to provide a user intention recognition system and method based on the LLM large model.

[0005] To achieve the purpose of the present invention, the technical solution adopted by the present invention is: A user intention recognition system based on the LLM large model, including an intention pre-classification module, a prompt word engineering, and an LLM large model module; The intent pre-classification module is used to pre-process the user's intent input and find a set of intent functions that are close to the user's intent input; Prompt word engineering: Provide text descriptions of the functions and parameters of the intent function and construct prompt words; splice and integrate the prompt words constructed based on the intent function and the user intent input to form the LLM large model input; The LLM large model module performs model reasoning to obtain output results. The large model output is a string text, which contains the name and parameters of the intent function. It is parsed to obtain the user intent.

[0006] Furthermore, the intent pre-classification module is composed of a text extraction feature model AC-Net, which vectorizes the user intent input; each intent function in the intent function library corresponds to a text description of its function, and the text extraction feature model AC-Net is used to extract features from the text and vectorize it; then, the cosine similarity between the vector representation of the user intent input and the vector representation of the intent function library is calculated, sorted, and the top-k intent functions with the highest similarity to the user intent input are found.

[0007] Furthermore, the substructure of the AC-Net model consists of a grouped multi-head attention mechanism, a normalization layer, a residual connection, and a feed-forward layer. The feed-forward layer consists of a fully connected layer. The substructures are repeatedly stacked to form AC-Net. Improve the multi-head attention mechanism and adopt a grouped multi-head attention mechanism. That is, the Q in Q, K, and V of the multi-head attention mechanism is divided into different groups, and each group shares a K and V. Use data to train and tune the AC-Net model.

[0008] Furthermore, the intent function is described to form a string description. A script is written to use the Top-k intent functions, find the corresponding intent function description in the intent function library, and then merge it with the user's intent input text to form the LLM large model input.

[0009] Furthermore, the LLM large model substructure consists of RMS normalization, multi-head attention mechanism, feedforward layer, and residual connection. The feedforward layer consists of a fully connected layer. Data is constructed to fine-tune the model, and a LORA-based fine-tuning method is used to update some parameters of the model.

[0010] A user intention recognition method based on the LLM large model includes the following steps: Step 1: Use the intent pre-classification module to pre-process the user intent input and find out the intent function set that is close to the user intent input; Step 2: Provide a text description of the functions and parameters of the intent function and construct prompt words; integrate the prompt words constructed based on the intent function and the user's intent input to form the LLM large model input; Step 3: Send the large model input to the LLM large model for model reasoning, analyze the output of the large model, and obtain the user intention.

[0011] Furthermore, in step 1, the text extraction feature model AC-Net is used to vectorize the user intention input; each intention function in the intention function library corresponds to a text description of its function, and the text extraction feature model AC-Net is used to extract features from the text and vectorize it; then, the cosine similarity between the vector representation of the user intention input and the vector representation of the intention function library is calculated, sorted, and the top-k intention functions with the highest similarity to the user intention input are found.

[0012] Furthermore, in step 2, the intent function is described to form a string description; a script is written to use the Top-k intent functions, the corresponding intent function description is found in the intent function library, and then it is merged with the user's intent input text to form the LLM large model input.

[0013] Furthermore, the substructure of the AC-Net model consists of a grouped multi-head attention mechanism, a normalization layer, a residual connection, and a feed-forward layer. The feed-forward layer consists of a fully connected layer. The substructures are repeatedly stacked to form AC-Net. Improve the multi-head attention mechanism and adopt a grouped multi-head attention mechanism. That is, the Q in Q, K, and V of the multi-head attention mechanism is divided into different groups, and each group shares a K and V. Use data to train and tune the AC-Net model.

[0014] Furthermore, in step 3, the LLM large model substructure consists of RMS normalization, multi-head attention mechanism, feedforward layer, and residual connection, and the feedforward layer consists of a fully connected layer; and data is constructed to fine-tune the model, and a LORA-based fine-tuning method is used to update some parameters of the model.

[0015] The beneficial effect of the present invention is that, compared with the prior art, the present invention adopts LLM large model technology to identify user intentions. Human-computer interaction is basically deployed on the terminal side, with high performance requirements. A 2B model with a small number of parameters is selected as the base model, and an intention pre-classification module is designed to reduce the system response time. In order to make the model have knowledge related to intent recognition, parameters are fine-tuned based on this base model. The model after parameter fine-tuning cannot directly recognize intent. A prompt word template is specially designed for the large model to identify intent. The model is universal and can recognize different intents. It has high flexibility. When adding new intents, there is no need to retrain the model like entity naming. In terms of identifying user intentions, it has high recognition accuracy and scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic diagram of the user intention recognition system based on the LLM large model of the present invention; Figure 2 It is a schematic diagram of the substructure of the AC-Net model; Figure 3 It is a schematic diagram of the substructure of the LLM large model. DETAILED DESCRIPTION

[0017] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of this application.

[0018] like Figure 1 As shown, the user intention recognition system based on the LLM large model described in the present invention solves the complex intention recognition that traditional methods in human-computer interaction cannot solve, and is mainly composed of three parts: intention pre-classification module, prompt word engineering, and LLM large model module.

[0019] The intent pre-classification module is used to pre-process the user intent input and find a set of intent functions that are closer to the user intent input. The intent pre-classification module is mainly composed of the text extraction feature model AC-Net, which vectorizes the user intent input. Each intent function in the intent function library corresponds to a text that simply describes its function. The text extraction feature model AC-Net is used to extract features from the text and vectorize it. Then, the cosine similarity between the vector representation of the user intent input and the vector representation of the intent function library is calculated, and then sorted to find the top-k intent functions with the highest similarity to the user intent input, in preparation for further screening.

[0020] Prompt word engineering: Provide text descriptions of the functions and parameters of the intent function and construct prompt words; splice and integrate the prompt words constructed based on the intent function and the user's intent input to form the LLM large model input. Describe the intent function to form a string description; write a script to use the Top-k intent functions, find the corresponding intent function description in the intent function library, and then merge it with the user's intent input text to form the LLM large model input.

[0021] The LLM large model module performs model reasoning, parses the output of the large model, and obtains the user's intent. The output of the large model is a string text, which contains the name and parameters of the intent function. The text is parsed and the parsed function name and parameter analysis are verified. If they do not meet the requirements, the type of error is given, and then the multi-round dialogue capability of the large model is used to send the cause of the error to the large model so that it can re-reason.

[0022] The present invention also provides a method for identifying user intention based on the LLM large model, which specifically comprises the following steps: Step 1: Use the intent pre-classification module to pre-process the user intent input and find out the intent function set that is close to the user intent input; The intention function library consists of many intention functions, including functional text descriptions of the intention functions. For example, the window control is described as "opening or closing the car window."

[0023] The intent pre-classification module is mainly composed of the text extraction feature model AC-Net, which vectorizes the user intent input; each intent function in the intent function library corresponds to a text that simply describes its function, and the text extraction feature model AC-Net is used to extract features from the text and vectorize it; then, the cosine similarity between the vector representation of the user intent input and the vector representation of the intent function library is calculated, and then sorted to find the top-k intent functions with the highest similarity to the user intent input, in preparation for further screening.

[0024] The AC-Net model is improved based on the BERT model. The specific substructure is as follows Figure 2 As shown in the figure, the substructure consists of a grouped multi-head attention mechanism, a normalization layer, a residual connection, and a feedforward layer. The feedforward layer is mainly composed of a fully connected layer. The substructure is repeatedly stacked to form an AC-Net. In order to reduce the inference time, the multi-head attention mechanism is improved and a grouped multi-head attention mechanism is adopted. That is, the Q in the Q, K, and V of the multi-head attention mechanism is divided into different groups, and each group shares a K and V. The data is used to train and tune the AC-Net model.

[0025] Step 2: Provide a text description of the functions and parameters of the intent function and construct prompt words; integrate the prompt words constructed based on the intent function and the user's intent input to form the LLM large model input; The prompt word is mainly a text description of the intended function, including a detailed description of the function and parameters. The user's input text is added to the end of the prompt word sentence, and the two form a long text.

[0026] Describe the intent function, such as opening the window, playing music, etc. For the intent function of opening the window, simply describe it as "the function of opening or closing the window". Parameters also need to be defined. If there are two parameters, one is parameter A that controls the opening or closing of the window, and the other is parameter B that indicates the position of the window. Parameters A and B are described in text to form a string description.

[0027] The big model itself does not have the ability to recognize intent. If you want to perform intent recognition, you need to design prompt words, mainly to tell the big model that intent recognition is needed, analyze the matching relationship between the user's intent input and the intent function library, find the best matching intent function, and fill in the parameters of the intent function based on the user's intent input.

[0028] Write a script using the Top-k intent functions, find the corresponding intent function description in the intent function library, and then merge it with the user's intent input text to form the LLM large model input.

[0029] Step 3: Send the large model input to the LLM large model for model reasoning, analyze the large model output results, and obtain the user intention; The LLM large model is a model with a parameter amount of about 2B. Its intent recognition ability is weak when it is not fine-tuned. It cannot meet the requirements in many scenarios, and there will be problems such as incorrect filling of intent function parameters. In order to solve these problems, the present invention constructs data, fine-tunes the model, and adopts a LORA-based fine-tuning method to update some parameters of the model.

[0030] The LLM model is mainly composed of a multi-head attention mechanism, with substructures such as Figure 3 As shown in the figure, the substructure mainly consists of RMS normalization, multi-head attention mechanism, feedforward layer, and residual connection. The feedforward layer is mainly composed of a fully connected layer.

[0031] The output of the big model is a string text, which contains the name and parameters of the intended function. The text is parsed and verified. If the parsed function name and parameter parsing do not meet the requirements, the type of error is given. Then, using the multi-round dialogue capability of the big model, the cause of the error is sent to the big model so that it can re-reason.

[0032] The present invention is analyzed through a specific embodiment below, taking opening the main driving window as the user's intention input. The intention function library has many intention functions, such as window control, light control, air conditioning control, playing music, etc.

[0033] Step 1.1, provide a text description for each intent function in the intent function library, such as window control, which is described as "open or close the car window"; Step 1.2: Use AC-Net to extract features from the user’s intention input “open the driver’s window” and use a 768-dimensional vector C as its feature representation; Step 1.3: Use AC-Net to extract features from the text description of the intent function library to obtain a (N, 768) matrix D. Calculate the cosine similarity between C and D, and then continue to sort to find the top-k intent functions that are close to the user's intent input. Step 1.4, describe each intention function in the intention function library, such as window control, use text to describe its function, its parameter A is the switch state of the window, parameter B is the position of the window to be controlled, forming a string description; Step 1.5, write a script to use the Top-k intent function results, find the corresponding intent function description in the intent function library, and then merge it with the text "Open the driver's window", and then add some prompt words, such as you need to select the corresponding intent function according to the user input and fill in the parameters, etc., and the design of the prompt word project is completed; Step 1.6, send the processed text to the LLM model for reasoning; Step 1.7, the output result of the LLM large model is in string format, which needs to be parsed to find the name of the intent function and the value of the parameters of the intent function, and then analyze whether the parameters are correct. If they are not correct, explain the reasons, and then return them to the LLM large model for re-inference.

[0034] The beneficial effect of the present invention is that, compared with the prior art, the present invention adopts LLM large model technology to identify user intentions. Human-computer interaction is basically deployed on the terminal side, with high performance requirements. A 2B model with a small number of parameters is selected as the base model, and an intention pre-classification module is designed to reduce the system response time. In order to make the model have knowledge related to intent recognition, parameters are fine-tuned based on this base model. The model after parameter fine-tuning cannot directly recognize intent. A prompt word template is specially designed for the large model to identify intent. The model is universal and can recognize different intents. It has high flexibility. When adding new intents, there is no need to retrain the model like entity naming. In terms of identifying user intentions, it has high recognition accuracy and scalability.

[0035] The applicant of the present invention has made a detailed explanation and description of the implementation examples of the present invention in conjunction with the drawings in the specification. However, those skilled in the art should understand that the above implementation examples are only preferred implementation schemes of the present invention, and the detailed description is only to help readers better understand the spirit of the present invention, but not to limit the scope of protection of the present invention. On the contrary, any improvements or modifications based on the inventive spirit of the present invention should fall within the scope of protection of the present invention.

Claims

1. A user intention recognition system based on LLM large model, characterized in that: Including intent pre-classification module, prompt word engineering, and LLM large model module; The intent pre-classification module is used to pre-process the user's intent input and find a set of intent functions that are close to the user's intent input; Prompt word engineering: Provide text description of the functions and parameters of the intent function and construct prompt words; The prompt words built based on the intention function and the user intention input are spliced ​​and integrated to form the LLM large model input; The LLM large model module performs model reasoning to obtain output results. The large model output is a string text, which contains the name and parameters of the intent function. It is parsed to obtain the user intent.

2. The user intention recognition system based on the LLM large model according to claim 1 is characterized in that: The intent pre-classification module is composed of a text extraction feature model AC-Net, which vectorizes the user intent input; each intent function in the intent function library corresponds to a text description of its function, and the text extraction feature model AC-Net is used to extract features from the text and vectorize it; then, the cosine similarity between the vector representation of the user intent input and the vector representation of the intent function library is calculated, sorted, and the top-k intent functions with the highest similarity to the user intent input are found.

3. The user intention recognition system based on the LLM large model according to claim 2 is characterized in that: The substructure of the AC-Net model consists of a grouped multi-head attention mechanism, a normalization layer, a residual connection, and a feed-forward layer. The feed-forward layer consists of a fully connected layer. The substructures are repeatedly stacked to form AC-Net. Improve the multi-head attention mechanism and adopt a grouped multi-head attention mechanism. That is, the Q in Q, K, and V of the multi-head attention mechanism is divided into different groups, and each group shares a K and V. Use data to train and tune the AC-Net model.

4. The user intention recognition system based on the LLM large model according to claim 1 is characterized in that: Describe the intent function to form a string description; write a script using the Top-k intent functions, find the corresponding intent function description in the intent function library, and then merge it with the user's intent input text to form the LLM large model input.

5. The user intention recognition system based on the LLM large model according to claim 1 is characterized in that: The LLM large model substructure consists of RMS normalization, multi-head attention mechanism, feedforward layer, and residual connection. The feedforward layer consists of a fully connected layer. Data is constructed to fine-tune the model, and a LORA-based fine-tuning method is used to update some parameters of the model.

6. A method for identifying user intention based on the LLM large model, characterized in that: The following steps are included: Step 1: Use the intent pre-classification module to pre-process the user intent input and find out the intent function set that is close to the user intent input; Step 2: Provide a text description of the functions and parameters of the intended function and construct a prompt word; The prompt words built based on the intention function and the user intention input are spliced ​​and integrated to form the LLM large model input; Step 3: Send the large model input to the LLM large model for model reasoning, analyze the output of the large model, and obtain the user intention.

7. The method for identifying user intention based on the LLM large model according to claim 6 is characterized in that: In step 1, the text extraction feature model AC-Net is used to vectorize the user intent input; each intent function in the intent function library corresponds to a text description of its function, and the text extraction feature model AC-Net is used to extract features from the text and vectorize it; then, the cosine similarity between the vector representation of the user intent input and the vector representation of the intent function library is calculated, sorted, and the top-k intent functions with the highest similarity to the user intent input are found.

8. The method for identifying user intention based on the LLM large model according to claim 6 is characterized in that: In step 2, describe the intent function to form a string description; write a script using the Top-k intent functions, find the corresponding intent function description in the intent function library, and then merge it with the user's intent input text to form the LLM large model input.

9. The method for identifying user intention based on the LLM large model according to claim 7 is characterized in that: The substructure of the AC-Net model consists of a grouped multi-head attention mechanism, a normalization layer, a residual connection, and a feed-forward layer. The feed-forward layer consists of a fully connected layer. The substructures are repeatedly stacked to form AC-Net. Improve the multi-head attention mechanism and adopt a grouped multi-head attention mechanism. That is, the Q in Q, K, and V of the multi-head attention mechanism is divided into different groups, and each group shares a K and V. Use data to train and tune the AC-Net model.

10. The method for identifying user intention based on the LLM large model according to claim 6, characterized in that: In step 3, the LLM large model substructure consists of RMS normalization, multi-head attention mechanism, feedforward layer, and residual connection. The feedforward layer consists of a fully connected layer. Data is constructed to fine-tune the model, and some parameters of the model are updated using a LORA-based fine-tuning method.

Images