Classification method for enhancing multi-label intentions of large language model based on remote supervision algorithm
Through remote supervision algorithm and LoRA method, the large language model is fine-tuned, which solves the problem of data acquisition and labeling in vehicle intention recognition, and realizes the accuracy of multi-intention recognition of intelligent vehicle intentions and the ability to classify multi-intentions. Improves the car owner's experience.
Patent Information
- Application Number
- CN202411133549.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-19
- Publication Date
- 2024-11-26
AI Technical Summary
Traditional car machine intention recognition technology cannot effectively identify car owners' multiple intentions to use cars, and the large language model generates expressions or control instructions for wrong facts in car owners' multi-intention recognition tasks, mainly because it is difficult to obtain and label the training data.
Using a method based on remote supervision algorithm, data cleaning and manual pre-labeling are obtained by obtaining vehicle and machine voice buried data to form a seed labeling data set, and the remote supervision algorithm is used to automatically label unlabeled data, and the large language model is fine-tuned in combination with the LoRA method. Construct multi-label intention classification prompt words and perform inference to realize multi-intention recognition of smart car machines.
Improve data labeling efficiency, cover a wider range of intention categories, provide rich training data, enhance the multi-intention understanding ability of large language models, ensure the accuracy of intention recognition of smart car machines and multi-label intention classification ability, and improve the Car owner experience with a car.
Smart Images

Figure CN119025679A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of vehicle-mounted computer intention recognition, and more specifically, to a classification method for multi-label intentions of a large language model enhanced by a remote supervision algorithm. Background Art
[0002] Traditional in-car computer systems rely on vendor-provided vocabulary libraries and fixed rule templates to identify the driver's intent. This approach typically only recognizes pre-set, fixed commands and often fails to correctly interpret non-standard expressions. Furthermore, it lacks contextual understanding, making it unable to accurately identify the driver's diverse intent based on the continuity of the conversation and environmental changes. In recent years, with the advancement of intelligent in-car entertainment systems, intelligent in-car computer semantic understanding technology has enabled drivers to engage in human-computer interaction via voice. Combined with natural language processing technology, intelligent in-car computer systems can identify intent from conversations and generate commands to rapidly control the vehicle. However, due to the diversity of driver language expressions and the difficulty of labeling training data for multi-intent label classification algorithms, intent recognition algorithms are still unable to effectively identify the driver's diverse intents. The semantic understanding capabilities of intelligent in-car computer systems, such as multi-intent recognition and intent ranking based on natural language processing, still need to be improved.
[0003] Distant Supervision is a semi-supervised learning algorithm whose basic assumption is that for a triple (consisting of a pair of entities and a relationship) in an existing dataset, any sentence in external unlabeled data that contains this pair of entities reflects this relationship to some extent. Based on this assumption, the distant supervision algorithm can use a well-labeled small-scale dataset to automatically label sentences in external unlabeled data, thereby generating a large amount of training data and increasing the amount and richness of training data for downstream models. Distant supervision algorithms are widely used in natural language processing tasks such as relation extraction, named entity recognition, and sentiment analysis. Distant supervision algorithms can greatly improve the efficiency of training data labeling, saving the time and cost of manual labeling; at the same time, by extracting new rules from pre-labeled datasets, the flexibility of data labeling can be further improved.
[0004] A large language model (LLM) is a deep learning algorithm model with massive parameters that focuses on understanding and generating natural language text. The LLM can adapt to a variety of downstream natural language processing tasks through fine-tuning. The core mechanism of the LLM is to enable the model to capture the syntactic, semantic, and contextual information of the language through training on large-scale text data, and then generate natural language text that conforms to grammatical rules and has semantic coherence. Large language models are widely used in many fields, such as chatbots, intelligent service assistants, automatic translation systems, writing assistance tools, etc. They can generate corresponding responses to user input text and provide personalized suggestions and services. In short, as a key natural language processing technology, the LLM has shown broad application prospects and huge development potential.
[0005] While large language models have seen numerous successful applications, they remain inaccessible to direct application in the field of multi-intent natural language understanding for intelligent vehicle systems because they lack customized pre-training for the task of identifying multiple driver intents. Specifically, large language models can generate erroneous statements or control commands because they fail to understand the driver's true intent. Summary of the Invention
[0006] In response to the problems in the existing technology of high difficulty in obtaining and labeling training data for fine-tuning large language models and poor intention recognition capabilities of traditional vehicle computers, the present invention provides a classification method for multi-label intents of large language models enhanced by remote supervision algorithms. Relying on the powerful text comprehension capabilities of large language models and the efficient automatic labeling of remote supervision algorithms, accurate multi-intent recognition functions of intelligent vehicle computers are realized, thereby improving the multi-label intent classification capabilities of intelligent vehicle computers.
[0007] In order to achieve the above object, the present invention adopts the following technical solutions:
[0008] A classification method for multi-label intent based on a remote supervision algorithm to enhance a large language model. The method obtains user input from vehicle-mounted voice data, cleans the user input, and randomly selects a portion for manual pre-labeling to form a seed labeled dataset. The remaining user input that has not been manually pre-labeled constitutes an unlabeled dataset.
[0009] A distant supervision algorithm is used to extract rule templates from the seed labeled dataset, and user input in the unlabeled dataset is automatically labeled based on the rule templates. The automatically labeled and manually pre-labeled user input together constitute the large language model fine-tuning training set;
[0010] Send the large language model fine-tuning training set to the large language model fine-tuning interface in a certain format, and use the LoRA method to fine-tune the large language model;
[0011] Construct a prompt word project and fill the current user input into the reserved slots for prompt words to form multi-label intent classification prompt words;
[0012] The multi-label intent classification prompt words are input into the fine-tuned large language model for inference, and the corresponding multi-label intent list is output.
[0013] The seed annotation dataset is obtained in the following way:
[0014] Obtain user input from the vehicle-mounted voice data, clean the text content of the user input, and generate cleaned text data;
[0015] Downsample the cleaned text data to form a seed dataset to be labeled;
[0016] Manual data annotation is performed on the seed dataset to be annotated, and after the annotation is completed, the data is summarized to obtain the seed annotated dataset.
[0017] The large language model fine-tuning training set is obtained specifically by the following method:
[0018] The manually pre-labeled entity keywords and corresponding action keywords in the obtained seed labeled dataset are sent to the unlabeled dataset, and a keyword matching operation is performed on each user input in the unlabeled dataset to filter out user inputs that hit both the entity keywords and the corresponding action keywords;
[0019] For user input that hits both entity keywords and corresponding action keywords, each time a keyword match occurs, the one or more intents associated with the keyword match are used as multi-label intents for data annotation of the user input. Finally, the above operation is performed on all user inputs that hit to complete the multi-label intent data annotation.
[0020] Manually annotate the keyword matches of new entity keywords and action keywords and the corresponding one or more intents from the hit user input, add them to the seed annotation dataset and perform multi-label intent data annotation. Repeat the keyword matching, add the corresponding intent to the seed annotation dataset and perform multi-label intent data annotation process until the data volume reaches the data scale required for the large language model fine-tuning training set. Finally, summarize the annotation results to obtain the large language model fine-tuning training set.
[0021] The LoRA method is used to fine-tune the large language model as follows:
[0022] A new path is added next to the original pre-trained large language model with an intermediate dimension of r. The rank is simulated by multiplying the dimensionality reduction matrix A and the dimensionality increase matrix B. During training, only matrices A and B are trained. When outputting, BA is superimposed with the original parameters of the pre-trained large language model.
[0023] The data cleaning rules are as follows:
[0024] Remove stop words from the user input text, including special symbols, spaces, line breaks, brackets, underscores, etc., then convert traditional Chinese characters in the text to simplified Chinese characters, and finally convert all uppercase English characters that appear in the user input text to lowercase English characters to form the cleaned text data.
[0025] Wherein, the downsampling method is:
[0026] First, N random numbers are generated, and then the user input of the index in the table formed by the cleaned text data corresponding to the generated random numbers is selected.
[0027] Preferably, when N random numbers are generated, N is equal to 8%-20% of the total number of data items input by users.
[0028] Among them, the manual data annotation rules are:
[0029] Label each user input with entity keywords, action keywords, and one or more intent tags corresponding to the user input.
[0030] Wherein, the dimension reduction matrix A is initialized with a random Gaussian distribution.
[0031] The dimension-raising matrix B matrix is initialized with a 0 matrix.
[0032] The beneficial effects brought by the present invention are:
[0033] The present invention fully utilizes the powerful natural language understanding ability of the large language model in the field of NLP. The present invention utilizes a remote supervision algorithm combined with a large language model to solve the problems of high difficulty in obtaining and labeling training data for fine-tuning the large language model, and realizes the multi-intention recognition function of the owner's car use in the field of intelligent vehicle machines, which can accurately identify the owner's various car use intentions. This method greatly improves the efficiency of data annotation, can cover a wider range of intent categories, and provides a richer data foundation for subsequent large language model training. At the same time, it enables the large language model to fully learn the diversity of the owner's natural language car use intention descriptions, and extracts and outputs various real car use intentions from them, thereby providing a basis for calling the corresponding car machine function. It can effectively avoid the large language model outputting incorrect car use intentions, improve the owner's car use experience through accurate intelligent vehicle machine intention recognition capabilities, and realize the intelligent vehicle machine multi-label intention classification capability. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 Develop a flow chart for the technical solution implementation;
[0035] Figure 2 Schematic diagram of the remote supervision algorithm;
[0036] Figure 3 Schematic diagram of the LoRA fine-tuning method principle. DETAILED DESCRIPTION
[0037] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.
[0038] It should be pointed out that, according to the needs of implementation, the various steps described in this application can be split into more steps, or two or more steps or parts of steps can be combined into new steps to achieve the purpose of the present invention.
[0039] The present invention provides a multi-label intent classification method for a large language model based on a remote supervision algorithm, which aims to solve the problems of high difficulty in obtaining and labeling training data for fine-tuning large language models and poor intent recognition capabilities of traditional vehicle computers. Figure 1 As shown, the specific implementation process of the technical solution of this method is as follows:
[0040] S1: Obtain user input from the vehicle-mounted voice data, first clean the user input and randomly select a portion for manual pre-labeling to form a seed labeled dataset; the remaining user input that has not been manually pre-labeled constitutes an unlabeled dataset.
[0041] The steps of S1 can be divided into:
[0042] S11: Obtain user input from the vehicle-mounted voice tracking data, and perform data cleaning on the text content of the user input to generate cleaned text data;
[0043] The data cleaning rules are as follows: first remove stop words in the user input text, including special symbols, spaces, line breaks, brackets, underscores, etc., then convert traditional Chinese characters in the text into simplified Chinese characters, and finally convert all uppercase English characters that appear in the user input text into lowercase English characters to form the cleaned text data.
[0044] S12: Downsampling is performed from the cleaned text data to form a seed dataset to be labeled;
[0045] The downsampling method is as follows: N random numbers are generated, and then N random numbers are selected to correspond to the user input indexed in the cleaned text data. In this embodiment, N is equal to 10% of the total number of data items. Other values can be used depending on the amount of data, such as 8%-20% of the total number of data items. In principle, for efficiency reasons, the value is not too large.
[0046] S13: Manually label the seed dataset to be labeled, and summarize the data after labeling to obtain the seed labeled dataset.
[0047] The rule for manual data labeling is to label each user input with entity keywords, action keywords, and one or more intent tags corresponding to the user input.
[0048] In this example, we cleaned the user input text in the car voice tracking data, and obtained a total of 1,000 data items. We sorted the manually pre-labeled labels into entities, actions, and intentions. Figure 3 The data set construction example is shown in Table 1:
[0049] Table 1. Example of seed annotation dataset construction
[0050]
[0051]
[0052] S2: After obtaining the seed labeled dataset, a remote supervision algorithm is used to extract rule templates from the seed labeled dataset, and user input in the unlabeled dataset is automatically labeled based on the rule templates. The automatically labeled and manually pre-labeled user inputs together constitute the large language model fine-tuning training set.
[0053] The specific steps of S2 are as follows:
[0054] S21: sending the manually pre-annotated entity keywords and corresponding action keywords in the obtained seed annotated dataset into the unannotated dataset, performing a keyword matching operation on each user input in the unannotated dataset, and screening out user inputs that hit both the entity keywords and the corresponding action keywords;
[0055] S22: For user input that hits both the entity keyword and the corresponding action keyword, each time a keyword match occurs, one or more intents associated with the keyword match are used as multi-label intents for data annotation of the user input. Finally, the above operation is performed on all the user inputs that hit the keyword to complete the multi-label intent data annotation.
[0056] S23: Manually mark keyword matches of new entity keywords and action keywords and corresponding one or more intents from the hit user input, add them to the seed annotation dataset and perform multi-label intent data annotation, repeat the keyword matching, add the corresponding intent to the seed annotation dataset and perform multi-label intent data annotation process until the data volume reaches the data scale required for the large language model fine-tuning training set, and finally summarize the annotation results to obtain the large language model fine-tuning training set.
[0057] In this embodiment, we use a remote supervision algorithm to automatically label unlabeled data. The schematic diagram of the remote supervision algorithm is as follows: Figure 2 As shown, the process of keyword matching, adding the corresponding intent to the seed annotation dataset, and performing multi-label intent data annotation is repeated. The number of executions can be considered according to the actual situation, and the execution cutoff condition can also be set according to the actual situation. In this embodiment, after repeating the execution five times, 15,243 data items were obtained. This amount of data reaches the data scale required for the large language model fine-tuning training set. The annotation results are summarized, and the final large language model fine-tuning training set example is shown in Table 2:
[0058] Table 2 Examples of large language model fine-tuning training sets
[0059]
[0060] S3: Send the large language model fine-tuning training set to the large language model fine-tuning interface in a certain format, and use the LoRA method to fine-tune the large language model.
[0061] We process the large language model fine-tuning training set obtained in S22 into a format that conforms to LoRA fine-tuning, and send it to the large language model fine-tuning interface position for fine-tuning. In this embodiment, based on the large language model fine-tuning training set obtained in S2, the generative pre-training model is fine-tuned by the LoRA method. LoRA, namely low-rank adaptation, is a technology that decomposes a large matrix into a small matrix by matrix multiplication, thereby reducing the number of parameters and increasing the training speed. The LoRA fine-tuning method is to add a new path next to the original pre-trained large language model, with an intermediate dimension of r. The rank is simulated by multiplying the dimensionality reduction matrix A and the dimensionality increase matrix B. The dimensionality reduction matrix A is initialized with a random Gaussian distribution, and the dimensionality increase matrix B is initialized with a 0 matrix. During training, only matrices A and B are trained. When outputting, BA is superimposed and output with the original parameters of the pre-trained large language model. The schematic diagram of its principle is shown as follows Figure 3 The advantage of the LoRA fine-tuning method is that it can efficiently switch tasks by replacing the reduced-dimensional matrix A and the increased-dimensional matrix B. At the same time, the LoRA fine-tuning method allows merging with the weights of a pre-trained large language model at deployment time. Compared with the fully fine-tuned model, by construction, no inference delay is introduced.
[0062] S4: Construct a prompt word project and fill the current user input into the reserved slot of the prompt word to form a multi-label intent classification prompt word.
[0063] The prompt word template used in the prompt word construction project in this embodiment is: "Please help me analyze the intent contained in the following vehicle owner input: Please enter one or more vehicle use intentions. \nPlease return the results in list format, without other content: \nThe vehicle owner input is: <user input>. \nPlease provide a result list of the vehicle owner's vehicle use intentions:" After constructing the template, the user input text content is filled into the reserved space, thus forming a multi-label intent classification prompt word.
[0064] S5: Input the multi-label intent classification prompt words into the fine-tuned large language model for inference and output the corresponding multi-label intent list.
[0065] After data cleaning of the user input text, this example obtained a total of 624 test set data. According to the prompt word template constructed in S4, the user input was filled into the reserved slots to obtain multi-label intent classification prompt words. The obtained prompt words were then input into the fine-tuned large language model. The final answer generated by the fine-tuned large language model is shown in Table 3:
[0066] Table 3. Examples of responses generated by the large language model after fine-tuning
[0067]
[0068]
[0069] The results show that the multi-label intent classification method based on a remotely supervised algorithm-enhanced large language model accurately outputs vehicle usage intention labels that are very similar to the actual vehicle owner's vehicle usage intentions, demonstrating outstanding effectiveness. This result demonstrates the feasibility of this method based on remotely supervised algorithms to enhance large language models. By leveraging the powerful text understanding capabilities of large language models and the efficient automated labeling capabilities of remotely supervised algorithms, this method achieves accurate multi-intent recognition for intelligent vehicle systems, improves their multi-label intent classification capabilities, and effectively addresses the challenges of obtaining and labeling training data for fine-tuning large language models.
[0070] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all such improvements and changes should fall within the scope of protection of the appended claims of the present invention.
Claims
1. A classification method for multi-label intent of a large language model enhanced by a remote supervision algorithm, characterized in that: Obtain user input from the vehicle-mounted voice embedding data, perform data cleaning on the user input and randomly select a portion for manual pre-labeling to form a seed labeled data set; other user inputs that are not manually pre-labeled constitute an unlabeled data set; A rule template is extracted from the seed labeled data set using a remote supervision algorithm, and user input in the unlabeled data set is automatically labeled based on the rule template. The automatically labeled and manually pre-labeled user input together constitute a large language model fine-tuning training set; Send the large language model fine-tuning training set to the large language model fine-tuning interface in a certain format, and use the LoRA method to fine-tune the large language model; Construct a prompt word project and fill the current user input into the prompt word reserved slot to form a multi-label intent classification prompt word; The multi-label intent classification prompt words are input into the fine-tuned large language model for inference, and the corresponding multi-label intent list is output.
2. According to claim 1, the classification method of multi-label intent based on remote supervision algorithm enhanced large language model is characterized in that: The seed annotation dataset is obtained specifically in the following way: Obtain user input from the vehicle computer voice buried data, perform data cleaning on the text content input by the user, and generate cleaned text data; Downsample the cleaned text data to form a seed data set to be labeled; Manual data annotation is performed on the seed data set to be annotated, and after the annotation is completed, it is summarized to obtain the seed annotated data set.
3. According to claim 1, the classification method of multi-label intent based on remote supervision algorithm enhanced large language model is characterized in that: The large language model fine-tuning training set is specifically obtained in the following way: The manually pre-labeled entity keywords and corresponding action keywords in the obtained seed labeling data set are sent to the unlabeled data set, and a keyword matching operation is performed on each user input in the unlabeled data set to filter out user inputs that hit both the entity keywords and the corresponding action keywords; For user input that hits both entity keywords and corresponding action keywords, each time a keyword match is hit, one or more intents associated with the keyword match are used as multi-label intents for data annotation of the user input. Finally, the above operation is performed on all hit user inputs to complete the multi-label intent data annotation. New entity keywords and action keywords are manually annotated from the hit user input, along with keyword matches and one or more corresponding intents, added to the seed annotation dataset, and multi-label intent data annotation is performed. The keyword matching, adding the corresponding intents to the seed annotation dataset, and multi-label intent data annotation process are repeated until the data volume reaches the data scale required for the large language model fine-tuning training set. Finally, the annotation results are summarized to obtain the large language model fine-tuning training set.
4. According to claim 1, the classification method of multi-label intent based on remote supervision algorithm enhanced large language model is characterized in that: The LoRA method is used to fine-tune the large language model as follows: A new path is added next to the original pre-trained large language model with an intermediate dimension of r. The rank is simulated by multiplying the reduced-dimensional matrix A and the increased-dimensional matrix B. Only matrices A and B are trained during training. When outputting, BA is superimposed with the original parameters of the pre-trained large language model.
5. According to claim 2, the classification method of multi-label intent based on remote supervision algorithm enhanced large language model is characterized in that: The data cleaning rules are: Remove stop words in the user input text, including special symbols, spaces, line breaks, brackets, underscores, etc., then convert the traditional Chinese characters in the text into simplified Chinese characters, and finally convert all uppercase English characters that appear in the user input text into lowercase English characters to form the cleaned text data.
6. The classification method of multi-label intent based on remote supervision algorithm enhanced large language model according to claim 2 is characterized in that: The downsampling method is: First, N random numbers are generated, and then the user input indexed in the table formed by the cleaned text data corresponding to the generated random numbers is selected.
7. The classification method of multi-label intent based on remote supervision algorithm enhanced large language model according to claim 2 is characterized in that: The manual data annotation rules are: Label each user input with entity keywords, action keywords, and one or more intent tags corresponding to the user input.
8. The classification method of multi-label intent based on remote supervision algorithm enhanced large language model according to claim 4 is characterized in that: The dimension reduction matrix A is initialized with a random Gaussian distribution.
9. The classification method of multi-label intent based on remote supervision algorithm enhanced large language model according to claim 4 is characterized in that: The dimension-raising matrix B matrix is initialized with a 0 matrix.
10. The classification method of multi-label intent based on remote supervision algorithm enhanced large language model according to claim 6, characterized in that: In the generation of N random numbers, N is equal to 8%-20% of the total number of data items input by users.
Citation Information
Patent Citations
Remote supervision relationship extraction method for large-scale unstructured text data
CN111241303A
Semantic matching model generation method and system based on remote supervision
CN115563512A
Voice interaction method, server and computer readable storage medium
CN117373456A
Stream tuning question and answer template generation method based on large language model instruction fine tuning
CN117574919A
Cited By
Domain knowledge fusion data enhancement method for text classification
CN119646223A
Domain Knowledge Fusion Data Augmentation Method for Text Classification
CN119646223B
Text multi-label classification method and device, electronic equipment and storage medium
CN119807417A
