Intention analysis system based on large language model

Abstract

The invention provides an intention recognition system based on a large language model, which takes the large language model as a basic model, and through continuous optimization of a training model and the intention recognition system, the system has a deep semantic understanding capability when processing a natural language understanding task, so that the intention recognition efficiency is improved. Complicated or ambiguous user intentions can be recognized more accurately, intention recognition tasks in various languages and professional fields can be processed, and the method has high universality and expansibility.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of data analysis, and particularly relates to an intention analysis system based on a large language model. BACKGROUND

[0002] In the field of modern natural language processing (NLP), understanding human language and parsing its meaning is a crucial task. This includes identifying the user's goal or requirement from their utterance, which is what we call "intention recognition". For example, from the sentence "I want to book a ticket to Beijing", we can identify the user's intention as "booking a ticket". In the past, we usually relied on pre-set rules and patterns, or used machine learning methods to learn from labeled training data, to complete such tasks. However, rule-based intention recognition techniques often do not perform well when dealing with complex or ambiguous intentions, and require manual setting of rules and patterns, which requires a high level of professional knowledge and experience. While machine learning methods can learn from a large amount of labeled training data, the demand for labeled data is large, and it is often difficult to obtain a large amount of labeled training data in real-world applications, making it difficult to accurately identify the user's intention when dealing with complex intention recognition tasks.

[0003] With the development of deep learning technology, especially the emergence of pre-trained language models (such as BERT, which stands for Bidirectional Encoder Representations from Transformers), we have made significant progress in various NLP tasks. BERT learns rich language representations by pre-training on large amounts of unlabeled text data, and then fine-tunes on specific tasks, so it can handle more complex intentions and has relatively less demand for labeled data.

[0004] However, when we face some specific professional fields, such as the aviation field, the user's requests may contain multiple intentions, and the information to be recognized is more complex. For example, for the sentence "Dongfang 6454, correct runway 21, cleared to land", we need to recognize two intentions, namely "confirmation" and "landing clearance", and at the same time, we also need to extract key information in the professional field such as "call sign", "flight number", "runway number", etc. In this case, traditional methods may not be able to accurately parse such complex and multi-intent expressions due to their inherent limitations and lack of specific domain knowledge. Although pre-trained models such as BERT have achieved remarkable results in many NLP tasks, they still face many challenges when dealing with these complex and multi-intent expressions. On the one hand, although these models can learn rich language knowledge from large-scale unlabeled data, how to effectively apply this knowledge to specific tasks remains a difficult problem. On the other hand, how to design and train models so that they can make the most of the capabilities of pre-trained models is also a difficult problem. In addition, how to handle problems in specific applications, such as data sparsity, model interpretability, etc., is also something we need to consider. In short, although we have made some progress in the intent recognition task, when dealing with some complex expressions in the aviation field, especially those with multiple intentions, the existing technologies still cannot meet the requirements of the natural language processing field for efficient and accurate intent recognition. Therefore, a new technical solution is needed to effectively solve the above problems. Summary of the Invention

[0005] The object of the present invention is to develop an intent recognition system based on a large language model, which has deep semantic understanding ability when dealing with natural language understanding tasks, can more accurately recognize complex or ambiguous user intentions. At the same time, such a system can not only reduce the dependence on labeled data, but also handle intent recognition tasks in multiple languages and professional fields, and has strong versatility and scalability.

[0006] To achieve the object of the invention, the present invention provides the following technical solutions: The present invention proposes an intent recognition system based on a large language model, which is characterized in that the main construction method of the system is as follows: S1. Collect a small amount of diverse and labeled data, and generate a data set for training the model through python; S2. Use a large language model for pre-training, and the large language model includes but is not limited to ERNIE Bot, Baichuan, Tongyi Qianwen, LLAMA; S3. Add an example library module, a text embedding module, and a cosine similarity calculation module to the model; the example library module uses the embedding text embedding module to convert the user input text and the data in the example library into a one-dimensional vector, and then calculates the example in the example vector library that is most similar to the user input text vector according to the cosine similarity calculation module, and outputs the recognition result based on this. S4. Use the dataset generated in step S1 to train the large language model, so that the system can better understand and process the intent recognition task; S5. Use the trained model to identify the intent and key information of the user request text: The model puts the obtained user request text into the embedding module to convert it into a one-dimensional vector, and then uses the cosine similarity calculation module to calculate the example data with the highest similarity between this vector and the example vectors in the example library. The model will then perform word segmentation and determine the key information and intent of the user request text based on the example data with the highest similarity. S6. Determine whether the similarity between the requested text vector and the example vectors in the example library reaches a specified threshold: a. When the similarity reaches a specified threshold, the model will process and output the request text based on the key information and intent of the most similar example; b. When the highest similarity does not reach the specified threshold, the system provides the function of manually adding new examples. The new examples added by the user will be sent to the embedding module for vectorization and compared with the request text vector by cosine similarity. If the vector similarity between the new example and the request text reaches the specified threshold, the model will process the request text according to the key information and intent of the new example and add the new example to the example library. If the vector similarity between the new example and the request text does not reach the specified threshold, the user can fine-tune the new example to improve its similarity to the level that meets the specified threshold.

[0007] This invention proposes an intent recognition system based on a large language model. The key feature is that the large language model can be continuously fine-tuned and learned, enabling the intent recognition system to continuously optimize. Through further training and fine-tuning, the system can adapt to more intents and key information, thereby accurately identifying key information and user intent in user requests.

[0008] The fine-tuning methods for the large language model include, but are not limited to, adding new examples, updating the embedding module, and adjusting the similarity threshold.

[0009] This system also has strong scalability, and can continuously optimize and adapt to new tasks and data to meet user needs and environmental changes. Furthermore, because the large language model has outstanding text understanding capabilities, the training time will be greatly reduced, which greatly improves work efficiency.

[0010] The beneficial effects of this invention are as follows: The intent recognition system based on a large language model provided by this invention uses a large language model that, when trained on a small amount of labeled text data, can reach the level of professionals in the aviation or other technical fields. In addition, the large language model has a powerful multi-intent pattern processing capability, which can accurately identify multiple user intents from complex sentence structures. This system can accurately identify key information and user intents in user requests, thereby effectively handling intent recognition tasks in the aviation or other technical fields.

[0011] Because large language models can be continuously fine-tuned and learned to adapt to new tasks and data, intent recognition systems can be continuously optimized to cope with ever-changing user needs and environments.

[0012] Due to its powerful scalability, the system can continuously optimize and adapt to new tasks and data to meet user needs and environmental changes. Furthermore, because the large language model has outstanding text understanding capabilities, the training time will be greatly reduced, significantly improving work efficiency.

[0013] Thanks to the powerful language understanding and generation capabilities of large language models, intent recognition systems based on large language models can efficiently and accurately identify user intent, thereby providing a better user experience and service. Whether a user's query is a simple information search or a complex, multi-layered, multi-objective request, the system can understand and provide an answer that meets the user's needs.

[0014] In summary, intent recognition systems based on large language models can provide deep semantic understanding, accurately identify user intent, adapt to intent recognition tasks in multiple languages, and have strong versatility and scalability. This makes them promising for applications in natural language processing. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating the construction process of an intent recognition system based on a large language model, as described in this invention, when applied to the aviation field. Detailed Implementation

[0016] The present invention’s intent recognition system is illustrated using its application in the aviation field as an example. The invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0017] like Figure 1 As shown, the specific construction steps of the intent recognition system based on a large language model provided by this invention when applied to the aviation field are as follows: S1. Collect a small amount of diverse labeled aviation data, express the aviation data using regular expressions, and then use Python to reverse the regular expressions to generate aviation data text, providing high-quality data for subsequent training of large language models.

[0018] S2. Pre-training using a large language model The system utilizes large language models, including but not limited to Wenxin Yiyan, Baichuan, Tongyi Qianwen, and LLAMA, as its foundational language models. These models possess excellent language understanding capabilities, enabling them to comprehend various complex sentence structures. Through extensive pre-training, these models have acquired rich linguistic knowledge and the ability to understand complex sentence structures. This allows them to accurately understand and recognize user requests when handling intent recognition tasks, even making accurate judgments in complex and ambiguous contexts.

[0019] S3. Before model training, add an example library module, a text embedding module, and a cosine similarity calculation module to the model. The example library module uses the embedding text embedding module to convert the user input text and the data in the example library into a one-dimensional vector, and then calculates the example in the example vector library that is most similar to the user input text vector based on the cosine similarity calculation module, and outputs the recognition result based on this.

[0020] S3.1 Add an example library module to the model prompt word section of the system. The example library contains a variety of examples, each corresponding to a specific user input text. The example library helps the system more accurately recognize user input, thereby improving the system's recognition accuracy. The example library is not merely a simple dataset, but a knowledge base containing rich semantic and contextual information. It is crucial for the model to understand and parse user input information and is the foundation for the model to accurately respond to user requests.

[0021] S3.2 Add an embedding module Embedding is the process of mapping discrete variables (such as words) to a vector space, where these vectors represent the semantics of these discrete variables. In natural language processing, word embedding is the most common form, such as Word2Vec and GloVe. Word embedding captures the semantic information of words; words with similar meanings are closer together in the vector space. This model can convert user input text and data from a sample library into one-dimensional vectors. A one-dimensional vector is a data format that can be used to represent textual information; it converts textual information into numerical values, facilitating subsequent processing and analysis.

[0022] S3.3 Use cosine similarity to find the example that is most similar to the user input text vector.

[0023] Cosine similarity is a measure of how similar the directions of two vectors are. It is calculated by multiplying vector A by vector B and dividing by the product of the magnitudes of A and B. Cosine similarity values ​​range from -1 to 1. A value closer to 1 indicates that the two vectors are more similar in direction; 0 indicates that the two vectors are orthogonal; and -1 indicates that the two vectors are completely opposite in direction. In natural language processing, cosine similarity is often used to measure the semantic similarity between two pieces of text. This means that we can find the best-matching example by comparing the cosine similarity between the user's input text vector and every example vector in the example library.

[0024] S4. Training the model The dataset generated in step S1 is used to train the large language model. The large language model is trained with a small amount of diverse labeled data, so that the system can better understand and process the intent recognition task in the aviation field.

[0025] S5. Use the trained model to identify the intent and key information in the user request text. The specific steps are as follows: S5.1. The obtained user request is used as the first text. The model first puts the first text into the embedding module to convert it into a one-dimensional vector. Then, the model uses the cosine similarity calculation module to calculate the example data with the highest similarity between this vector and the example vectors in the example library. Then, the model performs word segmentation and determines key information and intent based on this example data. S5.2. Since the same word segmentation may have different key information and intention expressions in different text contexts, the model assigns key information and intentions to the word segmentation on the first text based on the example data with the highest similarity. For example, there are currently two pieces of example data: "Ah, then head for 110, turn left, two three eight turns to the east" and "Uh, can we climb to 110? South, three nine zero six". The key information in "Ah, then head for 110, turn left, two three eight turns to the east" includes "call sign - east, flight number - two three eight turns, turn - left, heading - 110", and the intention is "modify / keep heading track - then head for 110 and turn left". The key information in "Uh, can we climb to 110? South, three nine zero six" includes "call sign - south, flight number - three nine zero six, climb / descend - climb, request altitude - 110", and the intention is "apply for altitude adjustment / keep - can we climb to 110?". When the content of the first text is "Dongfang 100, can we climb to 110?", the model converts the first text into a one-dimensional vector, and then calculates the cosine similarity between the one-dimensional vectors of the above two examples and this vector respectively. It is calculated that the similarity with "Uh, can we climb to 110? South, three nine zero six" is higher. Then, the key information in the first text can be obtained as "call sign - east, flight number - 100, climb / descend - climb, request altitude - 110", and the intention is "apply for altitude adjustment / keep - can we climb to 110?". S5.3. After our tests, the large language model also performs outstandingly in dealing with multiple intentions and in some semantically complex scenarios. For example, for "Dongfang 110, head for 110, climb to 110, speed 110", the output result of the model is: key information "call sign - east, flight number - 100, heading - 110, climb / descend - climb, request altitude - 110, speed - 110", and intentions "modify / keep heading track - head for 110, modify / keep altitude - climb to 110, modify / keep speed - speed 110".

[0026] S6. Determine whether the similarity between the request text vector and the example vectors in the example library reaches the specified threshold: (1) When the similarity reaches the specified threshold, process and output the request text according to the example with the highest similarity; (2) When the similarity does not reach the specified threshold, the model will prompt the user to manually add a new example for the request text. The added new example will be sent to the embedding module for vectorization and compared with the request text vector through cosine similarity; [[ID=拾]] If the vector similarity between the new example and the request text reaches the specified threshold, the model will process the request text according to the key information and intentions of the new example and add the new example to the example library. Conversely, if the vector similarity between the new example and the request text does not reach the specified threshold, the model will return the similarity value between the new example and the request text, so that the user can fine-tune the new example according to this similarity value to make its similarity reach the level that meets the specified threshold.

[0027] S7. Continuous optimization and fine-tuning of the system: S7.1 Continuous Optimization and Fine-tuning: In practical applications, models may encounter various problems, such as misidentification, missed identification, and poor handling of new tasks and data. To address these issues, our system possesses the capability for continuous optimization and fine-tuning. Based on user feedback and system performance, we can continuously optimize and fine-tune the model to adapt to new tasks and data. This may include, but is not limited to, adding new examples, updating embedding modules, and adjusting similarity thresholds. Through such continuous optimization, our system can better meet user needs and provide higher-quality services.

[0028] S7.2 Adapting to Intent and Key Information within the Aviation Domain: The aviation domain is highly specialized, and the intent and key information within it may differ from regular text information. Therefore, through further training and fine-tuning, our system can optimize its performance, enabling it to better identify and process intent and key information within the aviation domain. This allows our system to provide high-quality service whether processing air traffic control information or other aviation-related information.

[0029] S7.3 System Scalability: System scalability is crucial when facing constantly changing user needs and environments. Our system possesses excellent scalability, enabling continuous optimization and adaptation to new tasks and data. Whether facing new tasks or new data, our system can continuously optimize and fine-tune to meet the ever-changing needs of users and the environment.

[0030] The above embodiments are merely illustrative of the present invention. Those skilled in the art can make many changes based on the technical solutions disclosed in the present invention. Any technical solutions obtained by replacing the same or equivalent technical means are within the scope of protection of the claims of the present invention.

Claims

1. An intent recognition system based on a large language model, characterized in that, The main construction method of this system is as follows: S1. Collect a small amount of diverse labeled data and generate a dataset for training the model using Python; S2. Pre-train using a large language model, which includes, but is not limited to, Wenxin Yiyan, Baichuan, Tongyi Qianwen, and LLAMA; S3. Add an example library module, a text embedding module, and a cosine similarity calculation module to the model; the example library module uses the embedding text embedding module to convert the user input text and the data in the example library into a one-dimensional vector, and then calculates the example in the example vector library that is most similar to the user input text vector according to the cosine similarity calculation module, and outputs the recognition result based on this. S4. Use the dataset generated in step S1 to train the large language model, so that the system can better understand and process the intent recognition task; S5. Use the trained model to identify the intent and key information of the user request text: The model puts the obtained user request text into the embedding module to convert it into a one-dimensional vector, and then uses the cosine similarity calculation module to calculate the example data with the highest similarity between this vector and the example vectors in the example library. The model will then perform word segmentation and determine the key information and intent of the user request text based on the example data with the highest similarity. S6. Determine whether the similarity between the requested text vector and the example vectors in the example library reaches a specified threshold: a. When the similarity reaches a specified threshold, the model will process and output the request text based on the key information and intent of the most similar example; b. When the highest similarity does not reach the specified threshold, the system provides the function of manually adding new examples. The new examples added by the user will be sent to the embedding module for vectorization and compared with the request text vector by cosine similarity. If the vector similarity between the new example and the request text reaches the specified threshold, the model will process the request text according to the key information and intent of the new example and add the new example to the example library. If the vector similarity between the new example and the request text does not reach the specified threshold, the user can fine-tune the new example to improve its similarity to the level that meets the specified threshold.

2. The intent recognition system according to claim 1, characterized in that, The large language model can be continuously fine-tuned and learned, enabling the intent recognition system to have continuous optimization capabilities.

3. The intent recognition system according to claim 2, characterized in that, Fine-tuning of the large language model includes, but is not limited to, adding new examples, updating the embedding module, and adjusting the similarity threshold.

4. The intent recognition system according to claim 1, characterized in that, The example library is a knowledge base containing rich semantic and contextual information. It is the key to the model's understanding and parsing of user input information and the foundation for the model to accurately respond to user requests.

5. The intent recognition system according to claim 1, characterized in that, The specific operation method of step S1 is as follows: collect a small amount of diverse labeled aviation data, express the aviation data using regular expressions, and then use Python to reverse the regular expressions to generate aviation data text.

Images