User intent recognition system and method based on LLM large model

Abstract

The application discloses a user intention recognition system and method based on an LLM large model, an intention pre-classification module, which is used for pre-processing user intention input, finding an intention function set close to the user intention input, a prompt word engineering, which is used for performing text description on the functions and parameters of the intention function, and constructing a prompt word, splicing and integrating the prompt word constructed based on the intention function and the user intention input to form LLM large model input, an LLM large model module, which is used for performing model reasoning to obtain an output result, and the large model output being a string text containing the name and parameters of the intention function, and the user intention being obtained by analyzing the string text. The application adopts the LLM large model technology to recognize the user intention, selects a 2B model with a smaller parameter quantity as a base model, and simultaneously designs the intention pre-classification module to reduce the system response time, and in the aspect of recognizing the user intention, the application has higher recognition accuracy and scalability.

Description

Technical Field

[0001] The present invention relates to the fields of data analysis, intelligent interaction, and artificial intelligence, and in particular to a user intention recognition system and method based on an LLM large model. Background Art

[0002] User intent recognition is often used in the field of human-computer interaction and is an important part of the field. With the development of intelligence, the application scope of human-computer interaction is becoming wider and wider, such as smart speakers, cars, smart homes, etc.

[0003] Intent recognition is a critical task in natural language processing. Its primary purpose is to determine the intent or purpose expressed in user input, in order to better answer user questions or provide relevant services. Traditional approaches include rule matching, entity naming, and slot matching. Rule matching identifies intent by comparing user input with pre-set rule templates. Other approaches first perform entity naming recognition on user input and then use slot filling to identify intent. Both approaches have significant drawbacks, including limited flexibility. New intents require custom rules to be designed and manually updated, making it difficult to capture context and lacking semantic understanding, resulting in low recognition accuracy. Summary of the Invention

[0004] In order to solve the deficiencies in the prior art, the purpose of the present invention is to provide a user intention recognition system and method based on the LLM large model.

[0005] To achieve the purpose of the present invention, the technical solution adopted by the present invention is:

[0006] A user intention recognition system based on the LLM large model, including an intention pre-classification module, a prompt word engineering, and an LLM large model module;

[0007] The intent pre-classification module is used to pre-process the user's intent input and find a set of intent functions that are close to the user's intent input;

[0008] Prompt word engineering: Provide textual descriptions of the intent function's functions and parameters to construct prompt words. Prompt words constructed based on the intent function are combined with user intent input to form the LLM large model input.

[0009] The LLM large model module performs model inference to obtain output results. The large model output is a string text containing the name and parameters of the intent function. It is parsed to obtain the user intent.

[0010] Furthermore, the intent pre-classification module consists of a text extraction feature model AC-Net, which vectorizes the user intent input. Each intent function in the intent function library corresponds to a text description of its function, and the text extraction feature model AC-Net is used to extract features from the text and vectorize it. Then, the cosine similarity between the vector representation of the user intent input and the vector representation of the intent function library is calculated and sorted to find the top-k intent functions with the highest similarity to the user intent input.

[0011] Furthermore, the AC-Net model substructure consists of a grouped multi-head attention mechanism, a normalization layer, a residual connection, and a feedforward layer. The feedforward layer consists of a fully connected layer. The substructures are repeatedly stacked to form AC-Net.

[0012] Improve the multi-head attention mechanism and adopt a grouped multi-head attention mechanism. That is, the Q in the Q, K, and V of the multi-head attention mechanism is divided into different groups, and each group shares the same K and V. Use data to train and tune the AC-Net model.

[0013] Furthermore, the intent function is described to form a string description. A script is written using the Top-k intent functions to find the corresponding intent function description in the intent function library, and then merged with the user's intent input text to form the LLM large model input.

[0014] Furthermore, the LLM large model substructure consists of RMS normalization, multi-head attention mechanism, feedforward layer, and residual connection. The feedforward layer consists of a fully connected layer. Data is constructed to fine-tune the model, and a LORA-based fine-tuning method is used to update some model parameters.

[0015] A user intention recognition method based on the LLM large model includes the following steps:

[0016] Step 1: Use the intent pre-classification module to pre-process the user intent input and find a set of intent functions that are close to the user intent input;

[0017] Step 2: Provide a textual description of the intent function's functions and parameters, and construct prompt words. Combine the prompt words constructed based on the intent function and the user's intent input to form the LLM large model input.

[0018] Step 3: Send the large model input to the LLM large model for model inference, analyze the output results of the large model, and obtain the user intention.

[0019] Furthermore, in step 1, the text extraction feature model AC-Net is used to vectorize the user intent input; each intent function in the intent function library corresponds to a text description of its function, and the text extraction feature model AC-Net is used to extract features from the text and vectorize it; then, the cosine similarity between the vector representation of the user intent input and the vector representation of the intent function library is calculated, and the two are sorted to find the top-k intent functions with the highest similarity to the user intent input.

[0020] Furthermore, in step 2, the intent function is described to form a string description; a script is written using the Top-k intent functions, the corresponding intent function description is found in the intent function library, and then it is merged with the user's intent input text to form the LLM large model input.

[0021] Furthermore, the AC-Net model substructure consists of a grouped multi-head attention mechanism, a normalization layer, a residual connection, and a feedforward layer. The feedforward layer consists of a fully connected layer. The substructures are repeatedly stacked to form AC-Net.

[0022] Improve the multi-head attention mechanism and adopt a grouped multi-head attention mechanism. That is, the Q in the Q, K, and V of the multi-head attention mechanism is divided into different groups, and each group shares the same K and V. Use data to train and tune the AC-Net model.

[0023] Furthermore, in step 3, the LLM large model substructure consists of RMS normalization, multi-head attention mechanism, feedforward layer, and residual connection, and the feedforward layer consists of a fully connected layer; and data is constructed to fine-tune the model, and a LORA-based fine-tuning method is used to update some parameters of the model.

[0024] The beneficial effect of the present invention is that, compared with the prior art, the present invention adopts LLM large model technology to identify user intentions, and human-computer interaction is basically deployed on the end side, with high performance requirements. The 2B model with a smaller parameter amount is selected as the base model, and an intention pre-classification module is designed to reduce the system response time. In order to make the model have knowledge related to intent recognition, parameters are fine-tuned based on this base model. The model after parameter fine-tuning cannot directly identify intent. A prompt word template is specially designed for the large model to identify intent. The model is universal and can recognize different intents. It has high flexibility. When adding new intents, there is no need to retrain the model like entity naming. In terms of identifying user intentions, it has high recognition accuracy and scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 Schematic diagram of the user intention recognition system based on the LLM large model of the present invention;

[0026] Figure 2 It is a schematic diagram of the substructure of the AC-Net model;

[0027] Figure 3 It is a schematic diagram of the substructure of the LLM large model. DETAILED DESCRIPTION

[0028] The technical solution of the present invention will be further described below in conjunction with the accompanying drawings and embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present invention and are not intended to limit the scope of protection of this application.

[0029] like Figure 1 As shown, the user intention recognition system based on the LLM large model described in the present invention solves the complex intention recognition that traditional methods in human-computer interaction cannot solve. It mainly consists of three parts: intention pre-classification module, prompt word engineering, and LLM large model module.

[0030] The intent pre-classification module is used to preprocess user intent input and identify a set of intent functions that are closest to the user intent input. The intent pre-classification module is mainly composed of the text extraction feature model AC-Net, which vectorizes the user intent input. Each intent function in the intent function library corresponds to a text that simply describes its function. The text extraction feature model AC-Net is used to extract features from the text and vectorize it. Then, the cosine similarity between the vector representation of the user intent input and the vector representation of the intent function library is calculated, and then sorted to identify the top-k intent functions with the highest similarity to the user intent input, preparing for subsequent further screening.

[0031] The prompt word project describes the intent function's functionality and parameters in text format and constructs prompt words. The prompt words constructed based on the intent function are then combined with the user's intent input to form the LLM model input. The intent function is described as a string. A script is written to use the top-k intent functions, find the corresponding intent function description in the intent function library, and then merge it with the user's intent input text to form the LLM model input.

[0032] The LLM large model module performs model reasoning, parses the output of the large model, and obtains the user intent. The output of the large model is a string text containing the name and parameters of the intent function. The parsed function name and parameters are parsed and verified. If they do not meet the requirements, the type of error is given. Then, using the large model's multi-round dialogue capability, the cause of the error is also sent to the large model, allowing it to re-infer.

[0033] The present invention also provides a method for identifying user intention based on the LLM model, which specifically includes the following steps:

[0034] Step 1: Use the intent pre-classification module to pre-process the user intent input and find a set of intent functions that are close to the user intent input;

[0035] The intent function library consists of many intent functions, including functional text descriptions of the intent functions. For example, window control is described as "opening or closing the car window."

[0036] The intent pre-classification module mainly consists of the text extraction feature model AC-Net, which vectorizes the user intent input. Each intent function in the intent function library corresponds to a text that briefly describes its function. The text extraction feature model AC-Net is used to extract features from the text and vectorize it. Then, the cosine similarity between the vector representation of the user intent input and the vector representation of the intent function library is calculated, and then sorted to find the top-k intent functions with the highest similarity to the user intent input, preparing for subsequent further screening.

[0037] The AC-Net model is improved based on the BERT model. The specific substructure is as follows Figure 2 As shown in the figure, the substructure consists of a grouped multi-head attention mechanism, a normalization layer, a residual connection, and a feedforward layer. The feedforward layer is mainly composed of a fully connected layer. The substructures are repeatedly stacked to form AC-Net. In order to reduce the inference time, the multi-head attention mechanism is improved and a grouped multi-head attention mechanism is adopted. That is, the Q in the Q, K, and V of the multi-head attention mechanism is divided into different groups. Each group shares the same K and V, and the data is used to train and tune the AC-Net model.

[0038] Step 2: Provide a textual description of the intent function's functions and parameters, and construct prompt words. Combine the prompt words constructed based on the intent function and the user's intent input to form the LLM large model input.

[0039] The prompt word is mainly a text description of the intended function, including a detailed description of the function and parameters. The user's input text is added to the end of the prompt word sentence, and the two form a long text.

[0040] Describe the intent function, such as opening the car window, playing music, etc. For the intent function of opening the car window, it is simply described as "the function of opening or closing the car window". Parameters also need to be defined. If there are two parameters, one is parameter A that controls the opening or closing of the car window, and the other is parameter B that indicates the position of the car window, parameters A and B are described in text to form a string description.

[0041] The large model itself does not have the ability to recognize intent. If you want to perform intent recognition, you need to design prompt words, mainly to tell the large model that intent recognition is needed, analyze the matching relationship between the user's intent input and the intent function library, find the most matching intent function, and fill in the parameters of the intent function based on the user's intent input.

[0042] Write a script using the Top-k intent functions, find the corresponding intent function description in the intent function library, and then merge it with the user's intent input text to form the LLM large model input.

[0043] Step 3: Send the large model input to the LLM large model for model inference, analyze the output of the large model, and obtain the user intention;

[0044] The LLM large model is a model with a parameter size of about 2B. Its intent recognition capability is weak when it is not fine-tuned. It cannot meet the requirements in many scenarios, and problems such as incorrect filling of intent function parameters will occur. In order to solve these problems, the present invention constructs data, fine-tunes the model, and adopts a LORA-based fine-tuning method to update some parameters of the model.

[0045] The LLM model is mainly composed of a multi-head attention mechanism, with substructures such as Figure 3 As shown in the figure, the substructure mainly consists of RMS normalization, multi-head attention mechanism, feedforward layer, and residual connection. The feedforward layer mainly consists of fully connected layers.

[0046] The output of the large model is a string of text containing the name and parameters of the intended function. The text is parsed and verified. If the parsed function name and parameters do not meet the requirements, the type of error is given. The large model's multi-round dialogue capability is then used to send the cause of the error to the large model, allowing it to re-reason.

[0047] The present invention is analyzed through a specific embodiment below. Opening the main driving window is used as the user intention input. The intention function library has many intention functions, such as window control, light control, air conditioning control, music playing, etc.

[0048] Step 1.1, provide a text description for each intent function in the intent function library. For example, for window control, the description is "open or close the car window";

[0049] In step 1.2, AC-Net is used to extract features from the user intent input "open the driver's window" and a 768-dimensional vector C is used as its feature representation.

[0050] In step 1.3, AC-Net is used to extract features from the text descriptions of the intent function library to obtain a (N, 768) matrix D. Cosine similarity is calculated between C and D, and then sorted to find the top-k intent functions that are close to the user's intended input.

[0051] Step 1.4: Describe each intent function in the intent function library. For example, for window control, use text to describe its function. Parameter A is the window switch state, and parameter B is the window position to be controlled, forming a string description.

[0052] In step 1.5, write a script to use the top-k intent function results, find the corresponding intent function description in the intent function library, and merge it with the text "Open the driver's window". Then add some prompt words, such as selecting the corresponding intent function based on user input and filling in parameters. This completes the prompt word project design.

[0053] Step 1.6: Send the processed text to the LLM model for reasoning;

[0054] In step 1.7, the output result of the LLM model is in string format. It needs to be parsed to find the name of the intent function and the value of the parameters of the intent function. Then, analyze whether the parameters are correct. If they are not correct, explain the reasons and return them to the LLM model for re-reasoning.

[0055] The beneficial effect of the present invention is that, compared with the prior art, the present invention adopts LLM large model technology to identify user intentions, and human-computer interaction is basically deployed on the end side, with high performance requirements. The 2B model with a smaller parameter amount is selected as the base model, and an intention pre-classification module is designed to reduce the system response time. In order to make the model have knowledge related to intent recognition, parameters are fine-tuned based on this base model. The model after parameter fine-tuning cannot directly identify intent. A prompt word template is specially designed for the large model to identify intent. The model is universal and can recognize different intents. It has high flexibility. When adding new intents, there is no need to retrain the model like entity naming. In terms of identifying user intentions, it has high recognition accuracy and scalability.

[0056] The applicant of the present invention has made a detailed explanation and description of the implementation examples of the present invention in conjunction with the drawings in the specification. However, those skilled in the art should understand that the above implementation examples are only preferred implementation plans of the present invention, and the detailed description is only to help readers better understand the spirit of the present invention, and is not a limitation on the scope of protection of the present invention. On the contrary, any improvements or modifications based on the inventive spirit of the present invention should fall within the scope of protection of the present invention.

Claims

1. A user intention recognition system based on LLM large model, characterized by: Includes intent pre-classification module, prompt word engineering, and LLM large model module; The intent pre-classification module is used to pre-process the user's intent input and find a set of intent functions that are close to the user's intent input; The intent pre-classification module, comprised of an AC-Net text feature extraction model, vectorizes user intent input. Each intent function in the intent function library corresponds to a textual description of its function. The AC-Net text feature extraction model extracts features from this text and vectorizes it. The module then calculates the cosine similarity between the vector representation of the user intent input and the vector representation of the intent function library, ranking them to identify the top-k intent functions with the highest similarity to the user intent input. Prompt word engineering: Provide text descriptions of the functions and parameters of the intent function and construct prompt words; The prompt words built based on the intent function and the user's intention input are spliced ​​and integrated to form the LLM large model input; The LLM large model module performs model reasoning to obtain output results. The large model outputs a string text containing the name and parameters of the intent function. It is parsed to obtain the user intent. Specifically, the output result of the LLM large model is in string format and needs to be parsed to find the name of the intent function and the value of the parameters of the intent function. Then, analyze whether the parameters are correct. If they are incorrect, explain the reasons and return them to the LLM large model for re-reasoning.

2. The user intention recognition system based on the LLM large model according to claim 1 is characterized in that: The AC-Net model substructure consists of a grouped multi-head attention mechanism, a normalization layer, a residual connection, and a feedforward layer. The feedforward layer consists of a fully connected layer. The substructures are repeatedly stacked to form AC-Net. Improve the multi-head attention mechanism and adopt a grouped multi-head attention mechanism. That is, the Q in the Q, K, and V of the multi-head attention mechanism is divided into different groups, and each group shares the same K and V. Use data to train and tune the AC-Net model.

3. The user intention recognition system based on the LLM large model according to claim 1 is characterized in that: Describe the intent function to form a string description; write a script using the Top-k intent functions, find the corresponding intent function description in the intent function library, and then merge it with the user's intent input text to form the LLM large model input.

4. The user intention recognition system based on the LLM model according to claim 1 is characterized in that: The LLM large model substructure consists of RMS normalization, multi-head attention mechanism, feedforward layer, and residual connection. The feedforward layer consists of a fully connected layer. It also constructs data, fine-tunes the model, and uses a LORA-based fine-tuning method to update some model parameters.

5. A user intention recognition method based on LLM large model, characterized in that: The following steps are included: Step 1: Use the intent pre-classification module to pre-process the user intent input and find a set of intent functions that are close to the user intent input; The intent pre-classification module, comprised of an AC-Net text feature extraction model, vectorizes user intent input. Each intent function in the intent function library corresponds to a textual description of its function. The AC-Net text feature extraction model extracts features from this text and vectorizes it. The module then calculates the cosine similarity between the vector representation of the user intent input and the vector representation of the intent function library, ranking them to identify the top-k intent functions with the highest similarity to the user intent input. Step 2: Provide a text description of the intended function's functions and parameters, and construct prompt words; The prompt words built based on the intent function and the user's intention input are spliced ​​and integrated to form the LLM large model input; Step 3: Send the large model input to the LLM large model for model inference, analyze the output of the large model, and obtain the user intention; Specifically, the output result of the LLM large model is in string format, which needs to be parsed to find the name of the intent function and the value of the parameters of the intent function. Then, analyze whether the parameters are correct. If they are not correct, explain the reasons, and then return them to the LLM large model for re-inference.

6. The user intention recognition method based on the LLM large model according to claim 5 is characterized in that: In step 2, describe the intent function to form a string description; write a script using the Top-k intent functions, find the corresponding intent function description in the intent function library, and then merge it with the user's intent input text to form the LLM large model input.

7. The user intention recognition method based on the LLM model according to claim 5 is characterized in that: The AC-Net model substructure consists of a grouped multi-head attention mechanism, a normalization layer, a residual connection, and a feedforward layer. The feedforward layer consists of a fully connected layer. The substructures are repeatedly stacked to form AC-Net. Improve the multi-head attention mechanism and adopt a grouped multi-head attention mechanism. That is, the Q in the Q, K, and V of the multi-head attention mechanism is divided into different groups, and each group shares the same K and V. Use data to train and tune the AC-Net model.

8. The user intention identification method based on the LLM model according to claim 5 is characterized in that: In step 3, the LLM large model substructure consists of RMS normalization, multi-head attention mechanism, feedforward layer, and residual connection. The feedforward layer consists of a fully connected layer. In addition, data is constructed to fine-tune the model, and some parameters of the model are updated using a LORA-based fine-tuning method.

Images