Method for intention understanding in large language model through interaction table

Abstract

The invention discloses a method for performing intention understanding in a large language model through an interaction table, which comprises the following steps: a table interaction design stage: S1, an initial generation step: judging fuzziness of a user instruction by a first large language model; s2, a supervision step: a second large language model evaluates the interactive table according to a rule, and proposes an improvement suggestion; s3, a regeneration step: regenerating an interaction table by the first large language model according to the improvement suggestion; s4, in the step of simulating the user response, the third language model simulates the user response; s5, an intention summarization step: summarizing the intention of the user by the first large language model according to the reply of the user in the interaction table; in the proprietary model training stage, the step S1 of initial generation, the step S2 of supervision, the step S3 of regeneration and the step S5 of intention summarization are executed, and the step S1, the step S2 and the step S3 are sequentially completed through one-time calling.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a method for understanding intent in a large language model through an interaction table. Background Art

[0002] Recent advances in large language models (LLMs), such as GPT, LLaMA, ChatGLM, and Mistral, have significantly improved their performance in a variety of language understanding and generation tasks. The enhancements to these LLMs enable them to provide natural, plausible responses across a wide range of topics, making them well suited to meet user needs. They can have fluent conversations with users, gradually meeting their requests until they complete the entire task.

[0003] While focusing on user experience helps achieve full interaction, existing large-scale language models face a key problem: lack of understanding of user intent. These solutions (such as ChatGPT) often find it difficult to get close to the essence of user intent, but only produce specious results, leading to "false success". This has become the most common reason for user dissatisfaction. Figure 1 As shown in (a), when the user instructs “Please recommend a good book to me”, the generated result “Educated” is too broad and cannot meet the user’s requirements. This casual response greatly affects the efficiency and effect, and may even overwhelm the advantages of large language models.

[0004] Therefore, improving the ability of large language models to understand user intent has become an important research area. Current research on aligning large language models with human preferences seems to address this problem, where manually crafted instructions are used to fine-tune large language models based on human preferences. However, without understanding the user's intent, these efforts can only generate human-like answers. Although multi-round dialogues can enrich the details of the results, users need to actively state their needs during the interaction. To explore this issue in depth, Mistral-Interact first explored user intent and focused on the understanding of implicit intent. Figure 1 As shown in (b), facing the same problem, Mistral-Interact attempts to clarify the implicit intention behind the user's instructions through the design of the intention model, and provides supplementary queries through each question-answer pair to help users complete the task. As a result, Sapiens: A Brief History of Humankind is clearly customized according to personal preferences.

[0005] Although Mistral-Interact is a pioneering exploration, it does not clearly point out the main challenges faced by this task. First, intent understanding requires efficiency, and the user's task should be solved in a limited number of rounds rather than endless interactions. Figure 1As shown in (b), each round of interaction only processes a question-answer pair, which is part of the understanding of the intent. Second, accuracy determines the quality of intent understanding and ensures a close connection between user intent and large language model generation. Without external supervision, repeatedly guiding large language models to interpret question intent may not be sufficient to ensure accuracy. Summary of the invention

[0006] In view of this, in order to meet these challenges, considering that when facing a specific task, humans are more inclined to brainstorm related elements to form a basic strategy and then improve its quality. Inspired by this mechanism, this application proposes a novel Interaction Table-based Intention Understanding (ITIU) framework to effectively understand users by building an intention table and interact with users to achieve their real goals. In addition, this application also designs a rule-based supervision mechanism to improve the accuracy of the content.

[0007] Specifically, when faced with ambiguous instructions, ITIU uses a parallel deep analysis approach to delve into the underlying intent, thereby "brainstorming" relevant details and constructing an interactive intent table. Figure 1 As shown in (c), the table can promote information exchange and improve interaction efficiency. In addition, in order to ensure the accuracy of the table content, this application designs a rule-based supervision mechanism to enable ITIU to self-enhance its understanding ability. In this process, the empirical rules derived from the training data can be used as external knowledge to provide constructive improvement suggestions. In order to speed up the intent understanding process, this application trains a smaller proprietary model (LLaMA-3-8B) to replace the internal steps in the original interaction design. After the interaction, the user intent summary forms a more precise execution strategy, resulting in a more satisfactory result "Gun of August".

[0008] This application conducts a comprehensive experimental evaluation of the performance of ITIU from two key perspectives: intent understanding and intent execution. The evaluation results of intent understanding show that ITIU reduces the average number of user-model interaction rounds to 1.24 rounds, and the query coverage rate exceeds 70%, thus confirming its SOTA efficiency and accuracy in intent understanding. In terms of intent execution, the results show that whether using the open source LLaMA-3 or powerful models such as GPT-4 and Claude-3, integrating ITIU as an upstream intent understanding module can significantly improve the quality of execution results. This shows that when ITIU is combined with popular large language models, it has high compatibility and effectiveness.

[0009] Specifically, the method for understanding intent in a large language model through an interaction table disclosed in the present application is characterized by comprising the following steps:

[0010] Receive instructions from users;

[0011] Table interaction design stage: S1 initial generation step: the first language model determines the ambiguity of the user's instructions; for clear instructions, it directly outputs, and for other instructions, it generates intent clarification and interactive tables, and the interactive table contains multiple queries; S2 supervision step: the second language model evaluates the interactive table according to the rules and makes improvement suggestions to improve the accuracy of intent understanding; S3 regeneration step: according to the improvement suggestions, the first language model regenerates the interactive table; S4 simulated user response step: the third language model simulates user response; S5 intent summary step: the first language model summarizes the user's intention based on the user's reply in the interactive table; this summary will be used as the next round of user instructions, and will continue until the first language model determines that the instruction is clear;

[0012] The proprietary model training phase executes the above-mentioned S1 initial generation step, S2 supervision step, S3 regeneration step and S5 intention summary step, and one call completes steps S1, S2 and S3 in sequence;

[0013] Output the intent in the user command.

[0014] Furthermore, in the initial generation step, the first large language model determines the ambiguity of the instruction; intent clarification includes reformulating the user instruction into a format that is easier for the large language model to understand, including correcting any spelling or grammatical errors, and improving the ability of the large language model to accurately interpret the instruction; the interactive form is designed to reveal any missing intentions related to the user instruction, including three attributes: (1) user query, a targeted query designed to obtain detailed information from the user; (2) reference options, providing various options for the user to refer to, providing context or examples; and (3) user response, specifying a blank space for the user to record a response to the query.

[0015] Furthermore, in the supervision step, the second language model summarizes the rules that reflect user preferences through contrastive learning: the IN3 intent understanding dataset is used as the source data; for each user instruction t i , the manually written queries in IN3 are integrated into one table as a high-quality interaction table h i , and use the interaction table generated by LLaMA-2-7B-Chat as the low-quality interaction table l i ; High-quality interactive table i and low quality interaction table l i Form a comparative gradient; specifically, a two-stage strategy is used to summarize the rules:

[0016] (1) Pattern extraction: For each user instruction t i , assign a task to GPT-4: convert high-quality interaction tables h iInteraction table with low quality i Contrasts are performed to extract interpretable patterns, which are classified into favorable P i and unfavorable P′ i There are two categories, and each pattern point has a text description; this process is formalized as:

[0017]

[0018] in, represents all possible favorable modes, p ∞ It is the mode with the highest number among the favorable modes. represents all possible adverse modes, p ′ ∞ It is the mode with the highest number among the unfavorable modes;

[0019] (2) Rule summary: Instruct GPT-4 to select from the favorable mode P = {P1, P2, ..., P n} and unfavorable mode P′={P′1,P′2,…,P′ n}, and this summary process is formalized as follows:

[0020]

[0021] in, represents all possible favorable rules, r ∞ is the rule with the highest number among the favorable rules. represents all possible unfavorable rules, r′ m is the rule with the largest number among the unfavorable rules; favorable rule R and unfavorable rule R′ reflect the user's preferences; by adopting these rules to guide the improvement of the initial interaction table, it is ensured that the interaction table is more in line with the user's preferences.

[0022] Furthermore, in the step of simulating user responses, the third language model simulates different user tones, including:

[0023] Formal: Provide a professional response;

[0024] Not serious: Provide short and direct responses;

[0025] Enthusiasm: Provide detailed responses.

[0026] Furthermore, the proprietary model is constructed as follows:

[0027] Constructing a training dataset: A new table-based interaction dataset TIN was constructed to train LLaMA-3-8B-Instruct as a proprietary model; Specifically, the BPO instruction dataset was used as the instruction source of the dataset; The interaction records for each instruction were collected to form multiple records; These interaction records were annotated: Three users made preliminary annotations to these interaction records, and they were carefully reviewed and improved based on their insights; Review and consensus: Each record was checked by another annotator until a consensus was reached on the final annotation;

[0028] Integrate each interaction record in the TIN training set into the training sample template, and accumulate and connect the contents of the S1 initial generation step, the S2 supervision step, and the S3 step in series to generate a whole in one model call, in order to reduce the frequency of model calls;

[0029] Training: Select LLaMA-3-8b-Instruct as the default base model and use HuggingfaceTransformers for full parameter fine-tuning.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] For the first time, we comprehensively identify the main challenges facing intent understanding in large language models and propose solutions;

[0032] A novel framework ITIU for intent understanding is proposed, which includes two innovative stages: table interaction design and specialized model training;

[0033] Experiments demonstrate the feasibility and effectiveness of this application in intent understanding and intent execution evaluation, especially in terms of efficiency and accuracy, thus promoting a new paradigm for intent understanding in large language models. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 : Comparison of user command execution: (a) direct execution, (b) execution after question-answer pair interaction, and (c) execution after table interaction.

[0035] Figure 2 The overall structure of this application includes two stages: (I) table interaction design stage and (II) proprietary model training stage;

[0036] Figure 3 Templates used to convert all interaction records for training, (a) training example templates of clear instructions or (b) ambiguous instructions;

[0037] Figure 4: The time cost and number of model calls during the interaction between the user and the model. The model time refers to the average time required for the model to generate text during the user instruction interaction process, the user time refers to the average time required for the user to respond, and the total time is the sum of the two;

[0038] Figure 5 : A case study on the intent understanding process, comparing the user model interaction process, intent summary, and execution results of four methods: (a) LLaMA-Interact, (b) ITIU, (c) BPO, and (d) Original (direct execution of user instructions). Accurately captured user intent is highlighted. DETAILED DESCRIPTION

[0039] The present invention is further described below in conjunction with the accompanying drawings, but the present invention is not limited in any way. Any changes or substitutions made based on the teachings of the present invention belong to the protection scope of the present invention.

[0040] Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0041] Natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0042] The technical solutions provided in the embodiments of the present application involve technologies such as machine learning and natural language processing of artificial intelligence, which are specifically introduced and explained through the following embodiments.

[0043] Before introducing the embodiments of the present application, some terms involved in the present application are first explained.

[0044] 1. Human Preference Alignment: Human preference alignment aims to train or fine-tune large language models to generate texts that are more in line with human intentions and preferences. Most of the existing alignment studies follow the paradigm proposed by Ouyang et al., which includes two main stages: supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF). SFT adjustment enables large language models to have preliminary intent tracking capabilities, which mainly relies on a large amount of high-quality training data. RLHF alignment aims to further align large language models with scalable feedback through two steps: reward modeling and policy training. Some alternative methods explore how to minimize the dependence on a large amount of training data, such as Sun et al. who explored iterative SFT alignment using only the model itself, and DPO directly introduced feedback into the loss function. However, human preference alignment largely enables the model to predict user intent based on prior knowledge rather than truly understand user intent. Therefore, this method has problems such as low accuracy of intent understanding and high training cost. .

[0045] 2. Hint Optimization: Studies have shown that hints can significantly affect the performance of downstream tasks for online learners. Therefore, some research has focused on enhancing intent understanding of large language models through hint optimization. Unlike human preference tuning that tunes large language models to follow user intent, hint optimization aims to improve input hints without modifying the large language model itself. However, hint optimization mainly involves the model predicting user intent rather than truly understanding user intent.

[0046] 3. User-model interaction design: In order to better understand personalized user intentions, some studies focus on enhancing this aspect of large language models through effective user-model interaction design. For example, Shi et al. proposed a self-seed-based intent understanding framework, which generates dialogue seeds to purposefully guide the model to interact with users. Similarly, Qian et al. developed an interactive model, Mistral-Interact, which can actively obtain missing details from users through question-answer pairs. These methods enable large language models to understand personalized user intentions rather than simply predicting user intentions. However, due to the lack of efficient interaction patterns and robust supervision mechanisms, current user-model interaction designs often lead to low efficiency and accuracy in intent understanding, which has a negative impact on the overall user interaction experience.

[0047] In view of these shortcomings, this application focuses on improving the efficiency and accuracy of intent understanding during the interaction process, aiming to significantly improve the user interaction experience. The following will first introduce the task definition of intent understanding in large language models, and then outline the ITIU framework (such as Figure 2 The design of the table interaction design phase and the proprietary model training phase are introduced in detail.

[0048] Based on the challenges faced by intent understanding, this application describes the research problem as follows. Define the initial user instruction as t, define the execution model as function f, and define the execution result as r (i.e., r = f(t)). The intent understanding process is a "buffer" between t and f: when t is clear, the process should directly transfer t to the execution stage. On the contrary, if t is unclear, interaction with the user will be added to refine t into a more specific intent t user (Right now ). We expect that, compared with f(t), f(t user ) will perform better and be closer to the user's actual intention.

[0049] Table interaction design

[0050] The table interaction design phase includes five key steps, each with its own specific role. The process begins with (a) the initial generation step, where Intention-GPT (GPT-4) judges the ambiguity of the user's instructions. Clear instructions are directly output, while ambiguous instructions lead to the generation of intent clarification and an interactive table. The table contains multiple queries designed to improve the efficiency of the interaction. In the (b) supervision step, Supervisor-GPT (another GPT-4) evaluates the interactive table according to the pre-summarized rules and makes constructive suggestions to improve the accuracy of intent understanding. Based on these suggestions, Intention-GPT regenerates the interactive table in the (c) regeneration step. In the (d) simulated user response step, User-GPT (a third GPT-4) simulates user responses. Finally, in the (e) intention summary step, Intention-GPT summarizes the user's intention based on the user's response in the interactive table. This summary will serve as the user instruction for the next round and will continue until Intention-GPT judges that the instruction is clear. Subsequently, we will discuss the role and mechanism of these steps in detail.

[0051] Initial generation. Figure 2 In (a), the step of processing user instructions begins with Intention-GPT judging the ambiguity of the instructions and providing a thinking process.

[0052] The ambiguity of instructions is determined by the first language model according to the given criteria. Ambiguity includes: the user's task is too general, lacks some important details necessary to understand the user's intention, or lacks some preference details that can better help the user achieve the task goal. Clarity includes: the user is clear enough about the task and provides enough details about the task goal, personal preferences, etc.

[0053] The thinking process is the thinking basis for the first language model to determine whether an instruction is a model, which helps to improve the accuracy of instruction ambiguity judgment.

[0054] If the instruction is clear, it is directly output (i.e., t-user = t). If the instruction is ambiguous, Intention-GPT generates intent clarification and interactive tables. Intent clarification involves reformulating the user instruction into a format that is easier for large language models to understand. This includes correcting any spelling or grammatical errors, thereby improving the ability of large language models to accurately interpret instructions. The interactive table aims to reveal any missing intentions related to the user instruction, which includes three attributes: (1) user query, a targeted query designed to obtain detailed information from the user; (2) reference options, providing various options for the user to refer to, providing context or examples; and (3) user response, specifying a blank space for the user to record a response to the query. With the interactive table, users can participate in multiple queries in one round and choose responses based on their interests, which greatly improves the efficiency of the interaction.

[0055] This process can be formalized as:

[0056] {C i ,T i}=Intention―GPT(q i )

[0057] Among them, q i is the command entered by the user, C i is the generated intent clarification, T i is the generated interactive table. With the interactive table, users can participate in multiple queries in one round and choose replies according to their interests, which greatly improves the interaction efficiency.

[0058] Supervision. In the interactive forms initially generated, relying solely on the natural language generation capabilities of large language models cannot ensure that the generated queries can effectively capture or reflect the user's intent. For example, the query "Can you name some books you liked in the past?" Figure 2 (a)) may not match the user’s interests. This mismatch can lead to poor interaction quality and affect the accuracy of intent understanding. To address this challenge, we propose to implement a rule-based supervision mechanism. Figure 2 As shown in (b), this application instructs Supervisor-GPT to score the relevance and usefulness of the initially generated interactive form according to a set of pre-summarized rules. Supervisor-GPT then makes constructive suggestions for improving the form. Through this reflective process, we hope that these suggestions will make the interactive form more in line with the user's preferences, thereby ensuring the accuracy of intent understanding.

[0059] This process can be formalized as:

[0060]

[0061] Where R, R′ represent favorable rules and unfavorable rules respectively, q i is the command input by the user, T i is the initially generated interaction table, represent the score of favorable rules and the score of unfavorable rules respectively, S i Represents the generated improvement suggestions.

[0062] In order to formulate rules that accurately reflect user preferences, this application instructs GPT-4 to summarize these rules through comparative learning. The IN3 intent understanding dataset is used as the source data. For each user instruction t i , the manually written queries in IN3 are integrated into one table as a high-quality interaction table h i , and use the interaction table generated by LLaMA-2-7B-Chat as the low-quality interaction table l i This setting provides results of varying quality for large language models, forming a comparative gradient. Specifically, we adopt a two-stage strategy to summarize the rules.

[0063] (1) Pattern extraction. For each user instruction t i , will assign a task to GPT-4: to convert high-quality interactive tables into i Interaction table with low quality i The goal is to extract meaningful and interpretable patterns and classify them into favorable P i and unfavorable P′ i There are two types, and each mode point has a text description. This process can be formalized as:

[0064]

[0065]

[0066] in, represents all possible favorable modes, Represents all possible adverse modes.

[0067] (2) Rule summary. Instruct GPT-4 to select the favorable mode P = {P1, P2, ..., P n}, Pn} and unfavorable mode P′={P′1,P′2,… , P′ n}. This summary process can be formalized as:

[0068]

[0069] in, represents all possible favorable rules, Represents all possible unfavorable rules. The favorable rule R and unfavorable rule R′ accurately reflect the user's preferences. By adopting these rules to guide the improvement of the initial interaction table, it is ensured that the interaction table is more in line with the user's preferences.

[0070] Regenerate. Figure 2 As shown in (c), this application instructs Intention-GPT to refine the initially generated interactive form based on specialized suggestions. These suggestions come from rules specifically designed to reflect user preferences. This process can be formalized as:

[0071]

[0072] where q i is the command input by the user, T i is the initially generated interaction table, S i represents the generated improvement suggestions, It is an improved interactive table.

[0073] Through this guided refinement, the query and reference options in the improved interaction table are more consistent with the user's preferences, thus promoting higher quality and more satisfying interactions between the user and the model.

[0074] Simulate user responses. Figure 2 As shown in (d), in order to establish a complete interaction record, the application instructs User-GPT to simulate the user's response in the interaction table. In order to improve the applicability and robustness of the model trained based on this data, the application prompts User-GPT to simulate different user tones, including:

[0075] -Formal. Assume the user is polite and provide a professional response to cater to scenarios where professionalism and etiquette are extremely important.

[0076] -Not serious. Assume that the user is impatient and provide short and direct responses, which is suitable for fast and efficient communication scenarios.

[0077] - Enthusiasm. Assume that the user is articulate and provides detailed, emotional responses.

[0078] By simulating various tones, specialized models trained on these generated interaction transcripts can better handle different communication styles, thereby improving their effectiveness in various user interaction scenarios.

[0079] This process can be formalized as:

[0080]

[0081] Among them, q i is the command entered by the user. is the improved interactive table. is the intent table with user responses.

[0082] Summary of intent. Figure 2 As shown in (e), this application prompts Intention-GPT to aggregate all user-provided intentions from an interactive table containing user responses. This process can be formalized as:

[0083]

[0084] Among them, q i is the command entered by the user. is the intent table with the user's responses, It is a summary of the user’s intent.

[0085] This process aims to provide a more reasonable, comprehensive and clear summary of user intent. The intent summary will serve as the user instruction for the next round of interaction. If the instruction is judged to be ambiguous in the next round, a new round of interaction will be initiated. On the contrary, if the instruction is judged to be clear, it will be output as t-usr and can be directly applied to the downstream execution model, expressed as f(t-user). This procedure promotes seamless integration between the intent understanding model and the execution model.

[0086] Proprietary model training

[0087] In the table interaction design stage, using GPT-4 to generate high-quality interactions may hinder computational efficiency and response speed. These problems mainly stem from the need to repeatedly call GPT-4 during each round of interaction, which has a negative impact on the user experience. To overcome these problems, this application trains a smaller specialized model for interactive intent understanding. Figure 2 As shown in (II), the specialized model can perform steps such as (a) initial generation, (b) supervision, (c) regeneration, and (e) intent summarization while maintaining a small size. In addition, it simplifies the process and generates the content of steps (a), (b), and (c) with just one call, significantly reducing the number of model calls. This improvement improves computational efficiency and response speed. Subsequently, the construction of the training dataset and the training details of the specialized model will be further discussed.

[0088] Constructing a training dataset. We constructed a new table-based interaction dataset, TIN, to train LLaMA-3-8B-Instruct as a proprietary model. Specifically, we utilized the BPO instruction dataset as the instruction source for the dataset. During the table interaction design phase, we collected interaction records for each instruction, initially forming 2,683 records. To ensure reliability, we conducted a rigorous annotation process on these interaction records: (1) Initial annotation. Three users made preliminary annotations on these interaction records, and carefully reviewed and improved them based on their insights. (2) Review and consensus. Each record was then thoroughly reviewed by another annotator. Any differences triggered discussions until consensus was reached on the final annotation. Of the 2,683 records, we allocated 2,483 records to the training set and 200 records to the test set. Table 1 lists the statistical comparison between the TIN dataset and the existing IN3 intent understanding dataset. The analysis shows several noteworthy differences: the TIN dataset has a larger capacity, contains more queries, and provides a wider range of reference options. These features can enhance the training of the model's intent understanding capabilities, thereby improving its practicality in real-world scenarios.

[0089] Table 1: Detailed statistics of the training and testing parts of the TIN and IN3 datasets. The average number of queries is calculated by the fuzzy instruction counts, and the average number of options is calculated by the query counts.

[0090]

[0091] like Figure 3 As shown in Figure 1, we integrate each interaction record in the TIN training set into the training sample template, and accumulate and concatenate the content generated in steps (a), (b), and (c) to generate a whole in one model call, with the aim of reducing the frequency of model calls. Specifically, we first introduce system prompts containing pre-summarized rules, and then connect user instructions. For clear instructions, such as Figure 3 As shown in (a), the training model directly generates thoughts and judges the ambiguity of user instructions. For ambiguous instructions, such as Figure 3 As shown in (b), the training model generates an intent description and an interaction table, provides ratings and improvement suggestions, and completes the interaction table. Subsequently, the application connects the user response, and the training model summarizes the user intent and determines the ambiguity of the summary. If the summary is still unclear, we will continue to integrate the next round of interaction data until a clear intent summary is obtained.

[0092] Training details. In one embodiment, this application selects LLaMA-3-8b-Instruct as the default base model of ITIU, and uses Huggingface Transformers to perform 5 rounds of full parameter fine-tuning on two 80GB A100s. During the training phase, the AdamW optimizer is used, where 1 / 41=0.9, 1 / 42=0.999. The learning rate was set to 1e-6, and the ratio of the warmup step and linear decay was 0.1. The maximum input length was set to 4,096 tokens, and the training batch size was set to 4. The entire training process was completed in 4.5 hours, and we used the final checkpoint in subsequent evaluations.

[0093] The ability to understand user intent in a large language model can be directly evaluated by the quality of the user's interaction with the model, or indirectly evaluated by the performance of the downstream large language model in performing tasks. Direct evaluation focuses on intent understanding itself, gaining insight into how the target model understands and interprets user intent. Indirect evaluation emphasizes the ultimate goal of intent understanding: ensuring that the output of a large language model is more consistent with user intent.

[0094] To comprehensively evaluate the effectiveness of ITIU, in one embodiment, the present application divides the experiment into two parts: (1) intent understanding, which evaluates the ability of the target model to understand the user's intent during the interaction process; and (2) intent execution, which evaluates the performance of the downstream large language model in executing the task after the user interacts with the model.

[0095] Intent Understanding Assessment

[0096] Intent understanding evaluation does not involve the execution of any tasks. We directly evaluate the target model itself during interaction to assess its ability as an upstream intent understanding module.

[0097] In order to perform a comprehensive evaluation, in one embodiment, two test sets of intent understanding datasets are selected for evaluation.

[0098] -IN3-test is the test set of the IN3 dataset, containing 108 instances, which are manually annotated with the help of GPT-4.

[0099] -TIN-test is the test set of the TIN dataset, which contains 200 instances. It is characterized by a large number of queries, which means that the model needs a stronger intent understanding ability to discover more user intents.

[0100] For each instruction, the user is allowed to have an open interaction with the target model, during which the target model actively asks the user's intention. To ensure diversity of opinions and answers, this application recruited three users to participate in the conversation and provide answers. The user first outlines the intention based on his or her own ideas, and then expresses his or her intention under the guidance of the model's prompts. The entire interaction process will be recorded and then evaluated according to specific indicators.

[0101] This application conducts a comparative analysis of ITIU through two configurations: (1) ITIU-SMT, which only uses table interaction design without dedicated model training, using Mistral-7B-Instruct-v0.27, LLaMA-3-8B-Instruct, and GPT-4-turbo as baselines; (2) ITIU, which fully implements the ITIU framework. Given that Mistral-Interact is the previous SOTA model for interactive intent understanding via question-answer pairs, we choose Mistral-Interact as a strong competitor. To ensure a fair comparison between ITIU and Mistral-Interact that is not affected by different underlying models, we equip them with the same baseline models, namely Mistral-7B-Instruct-v0.2 and LLaMA-3-8B-Instruct.

[0102] This application uses the measurement method developed by Qian et al. to convert subjective user intentions in the interaction between users and user models into objective numerical values, thereby simplifying data analysis and comparison.

[0103] Main indicators. These indicators directly reflect the model's ability to understand user intent.

[0104] -Fuzzy judgment accuracy: This metric calculates the percentage of the model's instruction ambiguity judgments that are consistent with the ground truth.

[0105] -Query coverage: It measures the percentage of ground truth queries covered by queries generated by the model during interaction.

[0106] -Average number of interactions: It calculates the average number of interaction rounds for each instruction.

[0107] -Average number of queries per round: This metric determines the average number of queries per round of interaction.

[0108] -Average number of queries per instruction: This metric measures the average number of times the model queries in each instruction.

[0109] Other metrics. These metrics provide a more comprehensive assessment of the details of the interaction.

[0110] -Option Presentation Rate: This metric measures the percentage of queries that come with potential reference options.

[0111] -Option rationality rate: It records the percentage of options that are considered reasonable among the reference options provided by the model.

[0112] -Average number of options per query: This represents the average number of reference options provided by the model for each query.

[0113] - Intent summary coverage: This ratio quantifies the percentage of user-provided intents that the model explicitly summarizes in t-user.

[0114] For all indicators, this application uses direct statistical calculations or GPT-4 assisted measurements. In addition, only the macro-average results of all test indicators are listed, because the micro-average results of different models usually show similar trends.

[0115] Table 1: Intention understanding evaluation results for different metrics on IN3 and TIN test sets. Comparisons include ITIU-SMT (ITIU without dedicated model training), Interact series, and ITIU, which use different baseline model configurations. Arrows indicate higher (↑) or lower (↓), better

[0116]

[0117] Based on the intent understanding evaluation results listed in Table 1, we analyze ITIU as follows.

[0118] Higher interaction efficiency. Our analysis shows that compared with the Interact series of models, ITIU achieves higher interaction efficiency, which is mainly reflected in two aspects: (1) Lower average number of interactions. Users can complete the intent understanding process with fewer interaction rounds. (2) More average number of queries per round. Users can exchange more information with the model in each round. The improvement in ITIU's efficiency is mainly attributed to its innovative deployment of the user-model interaction table, which significantly improves the overall interaction efficiency.

[0119] Better interaction quality. ITIU outperforms its competitors in terms of interaction quality, as reflected in the following key findings: (1) Accuracy in predicting user intent. ITIU excels in predicting the ambiguity of user instructions, with approximately 70% of manually annotated label queries consistent with those generated by ITIU (LLaMA-3-8B). This high consistency with user preferences highlights ITIU’s accuracy in intent understanding. (2) Enhanced user experience metrics. ITIU excels in metrics that contribute to user satisfaction during the interaction process, particularly in providing more reference options per query and maintaining a reasonable reference option rate. (3) Robust performance across different environments. On the challenging TIN test set, ITIU maintains strong performance, while competitors such as Mistral-Interact experience a significant drop in performance. This robustness highlights the effectiveness of ITIU across different operating environments. These results can be attributed to the rule-based supervision mechanism, which greatly improves the relevance and effectiveness of the interaction data used for training.

[0120] Comparable performance to closed-source GPT-4. The results show that ITIU (LLaMA-3-8B) performs very close to ITIU-SMT (GPT-4) in all indicators, and even exceeds GPT-4 in terms of ambiguity judgment accuracy, average number of interaction rounds, and option presentation rate. This shows that smaller-scale specialized models can perform comparable to general large-scale models in intent understanding tasks. ITIU provides a more cost-effective solution while maintaining comparable performance to GPT-4.

[0121] To further demonstrate the efficiency of our approach in terms of interaction time, Figure 4 We compared the user-model interaction time and model call count of ITIU-SMT (GPT-4), LLaMA-Interact, and ITIU on the TIN test set. Our findings include (1) the total interaction time of ITIU-SMT (GPT-4) is too high, mainly due to the large number of model calls and the long duration of each call. (2) In contrast, ITIU significantly reduces the total user-model interaction time to less than one minute, due to the reduction in the number of model calls, which shortens the generation time. In addition, unlike LLaMA-Interact, which uses a question-answer pair model, ITIU uses a table model for interaction, allowing users to handle multiple queries at the same time and respond selectively, thereby greatly reducing user response time.

[0122] Intention Execution Assessment

[0123] In order to evaluate the effectiveness of intent understanding in large language models from the perspective of intent execution, this application integrates specialized models as upstream intent understanding modules into various large language models. Subsequently, the experimental results are verified by comparing the quality of intent execution and conducting case studies.

[0124] This application selects test sets from IN3 and TIN for evaluation. For each user instruction t and its corresponding intent summary t derived by the intent understanding model user , this application instructs the large language model to execute t and t respectively as the downstream execution model user This will produce two outputs: r = f(t) and r user =f(t user ). We evaluate r and r user The quality of the intention is used to assess the effectiveness of the execution of the intention.

[0125] This application uses LLaMA-3-8B-Instruct, LLaMA-3-70B-Instruct8, GPT-4-turbo, and Claude-3-opus as basic execution models. We compare ITIU with LLaMA-Interact and BPO. BPO is an intent understanding method that does not involve user-model interaction. It uses a dedicated prompt optimization model to refine user instructions.

[0126] Existing research has shown that powerful large-scale language models can effectively act as evaluators. According to the research of Cheng et al., this application uses GPT-4-turbo and Claude-3-opus as evaluation models to evaluate the quality of intention execution results. The evaluation process is as follows: For each user instruction, the evaluation model receives the instruction, the intent outline provided by the user, and a pair of execution results obtained from the comparison method. The task of the evaluation model is then to determine which result is better, or whether the pair of results are the same. The hint for the GPT-4-turbo score comes from MT-bench, and the hint for the Claude-3-opus score comes from Alpaca Eval. In order to reduce position bias and reduce costs, this application randomly adjusts the position of the intention execution results in each evaluation.

[0127] Table 2 presents the results of the intention-to-execute assessment.

[0128]

[0129] The results of the study on ITIU are summarized as follows: (1) Higher win rate. Using ITIU as the upstream interactive intent understanding model achieves a higher win rate on both datasets. Notably, using LLaMA-3-8B as the execution model with ITIU, the average win rate is 25.2% higher than the original LLaMA-3-70B, which shows that ITIU has the potential to enhance the performance of smaller models and thus surpass larger models. (2) SOTA performance. Compared with BPO and LLaMA-Interact, adopting ITIU as the upstream intent understanding model can significantly improve the quality of intent execution results. This shows that ITIU has higher accuracy and effectiveness in understanding user intent. (3) Consistent gains across different model sizes. The observations show that the improvements of ITIU are consistent from smaller open source models (such as LLaMA-3-8B) to more powerful large models (such as GPT-4 and Claude-3). This highlights the strong versatility of ITIU in models of different capabilities.

[0130] In order to clearly explain the role of ITIU, Figure 5 An implementation example is shown in Figure 1. A user instruction is randomly selected, and three models: (a) LLaMA-Interact, (b) ITIU, and (c) BPO are used to understand the user intent and generate an intent summary. Subsequently, GPT-4 is used as the execution model to implement the user intent. For a fair comparison, the same user participated in the entire interaction process and provided consistent answers to similar queries. The results show that ITIU has the following advantages: (1) Improved interaction efficiency. Compared with LLaMA-Interact, ITIU significantly improves the interaction efficiency by increasing the amount of information exchanged per round and reducing the number of rounds, thanks to its table interaction mode. (2) Higher intent understanding accuracy. In the intent summary comparison, the light-colored phrases indicate that ITIU accurately captures most of the user intent. This is due to its effective supervision mechanism, which improves the accuracy of intent understanding. (3) Personalized execution results. ITIU provides the most personalized results. In contrast, the results of BPO and Original are too general and do not match the user's actual intent, which may be due to their lack of direct interaction with the user.

[0131] This application deeply studies the understanding of intent in large language models, especially mining deeper user intent through table interaction; the ITIU framework is introduced, which includes two innovative stages: table interaction design and proprietary model training. In the table interaction design stage, ITIU adopts a table mode for user-model interaction to improve efficiency. In addition, a rule-based supervision mechanism is adopted to improve the accuracy of intent understanding. In the proprietary model training stage, this application cooperates to develop a table-based interaction dataset TIN through GPT-4 and manual annotation. ITIU is used to train a specialized interaction model, which can be integrated into a large language model as an upstream intent understanding module. This specialized model can effectively judge ambiguity in user instructions, promote interaction with users in table mode, and comprehensively summarize user intent. The specialized model is small in size and fast in generation, which further improves the overall interaction efficiency. The comprehensive experimental results of intent understanding and execution show that ITIU is superior to the current mainstream methods, and has achieved SOTA results compared with the open source LLaMA series and advanced large language models such as GPT-4 and Claude-3.

[0132] As used herein, the word "preferred" is intended to be used as an example, instance, or illustration. Any aspect or design described herein as "preferred" is not necessarily to be construed as being more advantageous than other aspects or designs. On the contrary, the use of the word "preferred" is intended to present concepts in a specific way. The term "or" as used in this application is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless otherwise specified or clear from the context, "X uses A or B" means any one of the naturally included permutations. That is, if X uses A; X uses B; or X uses both A and B, then "X uses A or B" is satisfied in any of the foregoing examples.

[0133] Moreover, although the present disclosure has been shown and described with respect to one or implementations, those skilled in the art will think of equivalent variations and modifications based on the reading and understanding of this specification and the accompanying drawings. The present disclosure includes all such modifications and variations, and is limited only by the scope of the appended claims. In particular, with respect to the various functions performed by the above-mentioned components (such as elements, etc.), the terms used to describe such components are intended to correspond to any component (unless otherwise indicated) that performs the specified function of the component (such as it is functionally equivalent), even if the structure is not equivalent to the disclosed structure of the function in the exemplary implementation of the present disclosure shown herein. In addition, although the specific features of the present disclosure have been disclosed with respect to only one of several implementations, such features can be combined with one or other features of other implementations that may be desired and advantageous for a given or specific application. Moreover, insofar as the terms "including", "having", "containing" or their variations are used in specific embodiments or claims, such terms are intended to be included in a manner similar to the term "comprising".

[0134] The functional units in the embodiments of the present invention may be integrated into a processing module, or each unit may exist physically separately, or multiple or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc. The above-mentioned devices or systems may execute the storage method in the corresponding method embodiment.

[0135] To sum up, the above embodiment is an implementation mode of the present invention, but the implementation mode of the present invention is not limited by the embodiment. Any other changes, modifications, substitutions, combinations, and simplifications that deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A method for understanding intent in a large language model through an interaction table, characterized in that: The following steps are involved: Receive instructions from users; Table interaction design stage: S1 initial generation step: the first language model determines the ambiguity of the user's instructions; for clear instructions, it directly outputs, and for other instructions, it generates intent clarification and interactive tables, and the interactive table contains multiple queries; S2 supervision step: the second language model evaluates the interactive table according to the rules and makes improvement suggestions to improve the accuracy of intent understanding; S3 regeneration step: according to the improvement suggestions, the first language model regenerates the interactive table; S4 simulated user response step: the third language model simulates user response; S5 intent summary step: the first language model summarizes the user's intention based on the user's reply in the interactive table; this summary will be used as the next round of user instructions, and will continue until the first language model determines that the instruction is clear; The proprietary model training phase executes the above-mentioned S1 initial generation step, S2 supervision step, S3 regeneration step and S5 intention summary step, and one call completes steps S1, S2 and S3 in sequence; Output the intent in the user command.

2. The method for understanding intent in a large language model through an interaction table according to claim 1, characterized in that: In the initial generation step, the first large language model determines the ambiguity of the instruction; intent clarification includes reformulating the user instruction into a format that is easier for the large language model to understand, including correcting any spelling or grammatical errors and improving the ability of the large language model to accurately interpret the instruction; the interactive form is designed to reveal any missing intentions related to the user instruction, including three attributes: (1) user query, a targeted query designed to obtain detailed information from the user; (2) reference options, providing various options for the user to refer to, providing context or examples; and (3) user response, specifying a blank space for the user to record a response to the query.

3. The method for understanding intent in a large language model through an interaction table according to claim 1, characterized in that: In the supervision step, the second language model summarizes the rules that reflect user preferences through contrastive learning: the IN3 intent understanding dataset is used as the source data; for each user instruction t i , the manually written queries in IN3 are integrated into one table as a high-quality interaction table h i , and use the interaction table generated by LLaMA-2-7B-Chat as the low-quality interaction table l i ; High-quality interactive table i and low quality interaction table l i Form a comparative gradient; specifically, a two-stage strategy is used to summarize the rules: (1) Pattern extraction: For each user instruction t i , assign a task to GPT-4: convert high-quality interaction tables h i Interaction table with low quality i Contrasts are performed to extract interpretable patterns, which are classified into favorable P i and unfavorable P′ i There are two categories, and each pattern point has a text description; this process is formalized as: in, represents all possible favorable modes, p ∞ It is the mode with the highest number among the favorable modes. represents all possible unfavorable modes, p′ ∞ It is the mode with the highest number among the unfavorable modes; (2) Rule summary: Instruct GPT-4 to select from the favorable mode P = {P1, P2, ..., P n } and unfavorable mode P′={P′1,P′2,…,P′ n }, and this summary process is formalized as follows: in, represents all possible favorable rules, r ∞ is the rule with the highest number among the favorable rules. represents all possible unfavorable rules, r′ m is the rule with the largest number among the unfavorable rules; favorable rule R and unfavorable rule R′ reflect the user's preferences; by adopting these rules to guide the improvement of the initial interaction table, it is ensured that the interaction table is more in line with the user's preferences.

4. The method for understanding intent in a large language model through an interaction table according to claim 1, characterized in that: In the step of simulating user responses, the third language model simulates different user tones, including: Formal: Provide a professional response; Not serious: Provide short and direct responses; Enthusiasm: Provide detailed responses.

5. The method for understanding intent in a large language model through an interaction table according to claim 1, characterized in that: The proprietary model is constructed as follows: Constructing a training dataset: A new table-based interaction dataset TIN was constructed to train LLaMA-3-8B-Instruct as a proprietary model; Specifically, the BPO instruction dataset was used as the instruction source of the dataset; The interaction records for each instruction were collected to form multiple records; These interaction records were annotated: Three users made preliminary annotations to these interaction records, and they were carefully reviewed and improved based on their insights; Review and consensus: Each record was checked by another annotator until a consensus was reached on the final annotation; Integrate each interaction record in the TIN training set into the training sample template, and accumulate and connect the contents of the S1 initial generation step, the S2 supervision step, and the S3 step in series to generate a whole in one model call, in order to reduce the frequency of model calls; Training: Select LLaMA-3-8b-Instruct as the default base model and use HuggingfaceTransformers for full parameter fine-tuning.

Images