A multi-intention dialogue language understanding method and system based on decoupled pre-training
Through the decoupled pre-training method, using named body recognition and single-intention dialogue data for pre-training, the data scarcity and high cost of the multi-intention dialogue language understanding model is solved, and wider application and higher recognition accuracy are achieved Spend.
Patent Information
- Application Number
- CN202410658392.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-27
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-05-27
AI Technical Summary
The existing multi-intention dialogue language understanding model faces data scarcity and high cost when deployed, which limits its wide application.
Using a decoupled pre-training method, task-independent ability pre-training is performed by naming body recognition training data, single-intention dialogue language understanding data is used to perform task-specific ability pre-training, and multi-intention dialogue language understanding fine-tuning data adjustment model to obtain multi-intention Dialogue language understanding model.
It effectively alleviates the scarcity and high cost of data in the field of multi-intention dialogue language understanding, broadens application scenarios, promotes the development of task-based dialogue systems, and improves the accuracy of multi-intention recognition and slot filling of the model.
Smart Images

Figure CN118504587B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of intelligent dialogue technology, and in particular, relates to a multi-intention dialogue language understanding method and system based on decoupling pre-training. Background Art
[0002] With the widespread application of intelligent dialogue systems in various industries, such as smart homes, customer service robots, and medical consultation, the complexity of user-system interactions has also increased. In real life, users often express multiple requests or intentions in a single sentence when interacting with the system. For example, "I want to listen to a piece of light music and check tomorrow's weather forecast." This covers two intentions - playing music and getting weather information. Traditional single-intent dialogue language understanding task-based dialogue systems cannot effectively handle such complex dialogues with multiple intentions, so their utility in practical applications is greatly limited.
[0003] In order to meet the needs of real-world scenarios, task-based dialogue systems with multi-intent dialogue language understanding capabilities have become a research hotspot. Its goal is to simultaneously identify multiple intents and their corresponding domain information from the user's multi-intent utterances, so as to more accurately grasp the user's complex needs and provide more personalized and precise services.
[0004] Although models with multi-intent conversational language understanding capabilities have significant potential in both theory and application, existing multi-intent conversational language understanding models face serious data scarcity issues when deployed in practice. These models rely on a large amount of precisely annotated multi-intent data for training, which is not only costly but also time-consuming and labor-intensive to obtain. In the real world, the lack of such data has become a major obstacle to the widespread application of multi-intent conversational language understanding technology. Summary of the invention
[0005] In order to remedy the deficiencies of the prior art, the present invention provides a method and system for multi-intent dialogue language understanding based on decoupled pre-training.
[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:
[0007] In a first aspect, a multi-intent dialogue language understanding method based on decoupled pre-training is provided, including:
[0008] Obtain named entity recognition training data, single-intent dialogue language understanding training data, and multi-intent dialogue language understanding fine-tuning data;
[0009] Use the named body recognition training data to pre-train the preset dialogue language model for task-independent capabilities to obtain the first-stage model;
[0010] The second-stage model is obtained by pre-training the first-stage model with task-specific capabilities using the single-intent dialogue language understanding training data;
[0011] The second-stage model is adjusted using the multi-intent dialogue language understanding fine-tuning data to obtain a multi-intent dialogue language understanding model.
[0012] The multi-intent conversation request to be analyzed is processed based on the multi-intent conversation language understanding model to obtain multiple intents and slots.
[0013] Furthermore, the task-independent capability pre-training of the preset dialogue language model is performed using the named body recognition training data to obtain the first-stage model, including:
[0014] Pre-training the pre-built conversational language model for entity range prediction tasks;
[0015] Pre-train the preset dialogue language model for the task of predicting the number of entities;
[0016] The preset dialogue language model is jointly trained with the named entity recognition task pre-training and the entity quantity prediction task pre-training using the named entity recognition training data to obtain the first-stage model.
[0017] Furthermore, the preset dialogue language model is pre-trained for the entity range prediction task, including:
[0018] Given input X = {[CLS], x1, x2, ..., x n}, CLS represents a special tag used to capture the overall semantics of a sentence;
[0019] Use the preset conversational language model for X = {[CLS], x1, x2, ..., x n} is encoded to obtain the hidden state vector H = {h[CLS], h1, h2, ..., h n};
[0020] The entity range prediction task is pre-trained as:
[0021] H = Encoder(X),
[0022] e j =softmax(W E H j +b),
[0023] Encoder represents the encoder, e j ∈{BX, IX, O}, BX represents the beginning of entity X, IX represents the inside of entity X, O represents non-entity, softmax represents the activation layer used to output the final classification result of the word, W Eand b are learnable parameters, H j represents the hidden state representation of the j-th word.
[0024] Furthermore, the preset dialogue language model is pre-trained for the entity quantity prediction task, including:
[0025] Given an input sentence h [CLS] , CLS represents a special tag used to capture the overall semantics of a sentence
[0026] The pre-training for the entity quantity prediction task is:
[0027] m=softmax{Wh [CLS] +b),
[0028] m represents the predicted value of the maximum number of entities in the input sentence, softmax represents the activation layer used to output the classification result of the number of entities, and W and b are learnable parameters.
[0029] Furthermore, the method also includes:
[0030] The loss function for pre-training task-independent capabilities is:
[0031]
[0032]
[0033]
[0034] represents the training loss for the entity range prediction task, represents the training loss of the entity number prediction task, represents the training loss of the task-independent ability pre-training stage, α1 and α2 are learnable weight parameters, and n refers to the number of named entities in the sentence. represents the predicted value of the jth named entity, e j represents the true label of the jth named entity, |m| refers to the maximum number of entities in the input sentence, represents the predicted value of the number of entities in the sentence, and m represents the true value of the number of entities in the sentence.
[0035] Furthermore, the task-specific capabilities of the first-stage model are pre-trained using the single-intent dialogue language understanding training data to obtain the second-stage model, including:
[0036] Pre-set the interactive relationship between slot filling and intent recognition;
[0037] Based on the interaction relationship, the first-stage model is pre-trained for task-specific capabilities using single-intent dialogue language understanding training data to obtain the second-stage model.
[0038] Further, task-specific capability pre-training includes:
[0039] Use a sentence-level encoder to process the input sentence X and obtain the corresponding hidden representation H of the sentence;
[0040] The new sentence representation obtained by the interaction of intent filling and intent recognition is:
[0041]
[0042]
[0043] Based on the entire sentence Semantic representation of , performing intent recognition tasks to classify sentence intents:
[0044]
[0045] I represents the predicted intent classification, softmax represents the activation layer used to output the intent classification result, and W and b are learnable parameters;
[0046] Representation based on the entire sentence Perform slot filling task prediction Slots in:
[0047]
[0048] S represents the result of predicting slot filling.
[0049] Furthermore, the method also includes:
[0050] The loss function for task-specific capability pre-training is:
[0051]
[0052]
[0053]
[0054] represents the training loss for single-intent intent recognition, represents the training loss for single intent slot filling, represents the training loss of the task-specific capability pre-training phase, β1 and β2 are learnable weight parameters, and n I refers to the total number of intent labels, represents the predicted value of the i-th intention, y I i represents the true value of the i-th intention, n SRefers to the total number of slot labels. represents the predicted value of the i-th slot, y S i Represents the true value of the i-th slot.
[0055] Secondly, a multi-intention dialogue language understanding system based on decoupled pre-training is provided, including:
[0056] The data acquisition module is used to acquire named entity recognition training data, single-intent dialogue language understanding training data, and multi-intent dialogue language understanding fine-tuning data;
[0057] The decoupled pre-training module is used to pre-train the preset dialogue language model for task-independent capabilities using the named body recognition training data to obtain the first-stage model; the first-stage model is pre-trained for task-specific capabilities using the single-intent dialogue language understanding training data to obtain the second-stage model; the second-stage model is adjusted using the multi-intent dialogue language understanding fine-tuning data to obtain the multi-intent dialogue language understanding model;
[0058] The multi-intent dialogue language understanding module is used to process the multi-intent dialogue request to be analyzed based on the multi-intent dialogue language understanding model to obtain multiple intents and slots.
[0059] The beneficial effects achieved by the present invention are:
[0060] Task-independent capability pre-training and task-specific capability pre-training make full use of a large amount of named body recognition training data and single-intent dialogue language understanding data. Named body recognition training data is non-dialogue language understanding data, which can greatly alleviate the data scarcity and high cost problems in the field of multi-intent dialogue language understanding in practical applications, broaden the application scenarios of multi-intent dialogue language understanding, and promote the development of task-based dialogue systems. BRIEF DESCRIPTION OF THE DRAWINGS
[0061] Figure 1 It is a flow chart of the multi-intent dialogue language understanding method based on decoupled pre-training of the present invention;
[0062] Figure 2 is a flow chart of the process of forming the corpus to be analyzed of the present invention;
[0063] Figure 3 It is a supervised paragraph topic model diagram based on defect complaint metadata of the present invention;
[0064] Figure 4 This is a structural diagram of the multi-intent dialogue language understanding system based on decoupled pre-training of the present invention. DETAILED DESCRIPTION
[0065] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.
[0066] like Figure 1 As shown, an embodiment of the present invention provides a multi-intent dialogue language understanding method based on decoupled pre-training, including:
[0067] 101, obtaining named entity recognition training data, single-intent dialogue language understanding training data, and multi-intent dialogue language understanding fine-tuning data;
[0068] Among them, the decoupled pre-training framework (DPF) involved in this embodiment is designed to improve the ability of the dialogue system to handle complex user queries. Before model training, named entity recognition training data, single-intent dialogue language understanding training data, and multi-intent dialogue language understanding fine-tuning data are obtained. In order to enhance the model's ability to recognize and understand semantic entities, the widely used named entity recognition (NER) is used for pre-training.
[0069] 102, using the named body recognition training data to pre-train the preset dialogue language model for task-independent capabilities to obtain a first-stage model;
[0070] Among them, task-independent capability pre-training is divided into two sub-tasks: entity range prediction and entity quantity prediction, so as to capture the general knowledge of dialogue language understanding at the word level and sentence level respectively. Through such a preliminary pre-training method, non-multi-intent dialogue language understanding data (named body recognition training data) can be used to achieve preliminary learning of multi-intent dialogue language understanding capabilities, thereby reducing dependence on multi-intent dialogue data;
[0071] The process of pre-training task-irrelevant abilities is as follows: Figure 2 As shown, specifically:
[0072] 201, pre-training the preset dialogue language model for entity range prediction tasks;
[0073] Entity scope prediction is treated as a word-level classification problem;
[0074] Given input X = {[CLS], x1, x2, ..., x n}, CLS represents a special tag used to capture the overall semantics of a sentence;
[0075] Use the preset conversational language model for X = {[CLS], x1, x2, ..., x n} is encoded to obtain the hidden state vector H = {h [CLS] ,h1,h2,...,h n};
[0076] The entity range prediction task is pre-trained as:
[0077] H = Encoder(X),
[0078] e j =softmax(W E H j +b),
[0079] Encoder represents the encoder, e j ∈{BX, IX, O}, BX represents the beginning of entity X, IX represents the inside of entity X, O represents non-entity, softmax represents the activation layer used to output the final classification result of the word, W E and b are learnable parameters, H j represents the hidden state representation of the jth word;
[0080] Through task training for entity range prediction, the model can learn word-level text features, thereby enhancing the model's fine-grained feature learning capabilities in the process of multi-intent dialogue language understanding, which is beneficial to improving the accuracy of slot filling tasks in multi-intent dialogues.
[0081] 202, pre-training the preset dialogue language model for the entity quantity prediction task;
[0082] Entity number prediction is a sentence-level classification problem;
[0083] Given an input sentence h [CLS] ,CLS represents a special tag used to capture the overall semantics of a sentence;
[0084] The pre-training for the entity quantity prediction task is:
[0085] m=softmax(Wh [CLS ]+b),
[0086] m represents the predicted value of the maximum number of entities in the input sentence, softmax represents the activation layer used to output the classification result of the number of entities, and W and b are learnable parameters;
[0087] The entity number prediction task aims to predict the number of entities contained in the entire sentence, thereby enhancing the model's understanding of sentence-level structure, which is beneficial to improving and enhancing the intent recognition task in multi-intent dialogue language understanding;
[0088] 203 , using the named entity recognition training data, jointly train the preset dialogue language model for entity range prediction task pre-training and entity quantity prediction task pre-training to obtain a first-stage model.
[0089] In order to enable the model to better learn both word-level and sentence-level text features, we use the existing named entity recognition data to jointly train the entity range prediction task and the entity quantity prediction task to capture general capabilities that are unrelated to the dialogue language understanding task and obtain the first-stage model.
[0090] The loss function for pre-training task-independent capabilities is:
[0091]
[0092]
[0093]
[0094] represents the training loss for the entity range prediction task, represents the training loss of the entity number prediction task, represents the training loss of the task-independent ability pre-training stage, α1 and α2 are learnable weight parameters, and n refers to the number of named entities in the sentence. represents the predicted value of the jth named entity, e j represents the true label of the jth named entity, |m| refers to the maximum number of entities in the input sentence, represents the predicted value of the number of entities in the sentence, and m represents the true value of the number of entities in the sentence;
[0095] In step 102, the training process of the first-stage model makes good use of non-multi-intent conversation data to preliminarily train the model's word-level and sentence-level language comprehension capabilities, thereby improving the model's comprehension capabilities while avoiding dependence on multi-intent conversation language comprehension data.
[0096] 103, using the single-intent dialogue language understanding training data to pre-train the first-stage model for task-specific capabilities to obtain the second-stage model;
[0097] Among them, the interactive relationship between intent filling and intent recognition is preset; Figure 3 As shown in Figure 3 is the pre-training process diagram. Figure 3 (b) has an interaction module, which is used to enhance the interaction between the intent recognition task and the slot filling task;
[0098] Based on the interaction relationship, the single-intent dialogue language understanding training data is used to pre-train the first-stage model for task-specific capabilities to obtain the second-stage model. The specific pre-training process is as follows:
[0099] Use a sentence-level encoder to process the input sentence X and obtain the corresponding hidden representation H of the sentence;
[0100] The new sentence representation obtained by the interaction of intent filling and intent recognition is:
[0101]
[0102]
[0103] Based on the entire sentence Semantic representation of , performing intent recognition tasks to classify sentence intents:
[0104]
[0105] I represents the predicted intent classification, softmax represents the activation layer used to output the intent classification result, and W and b are learnable parameters;
[0106] Representation based on the entire sentence Perform slot filling task prediction Slots in:
[0107]
[0108] S represents the result of predicting slot filling;
[0109] The loss function for task-specific capability pre-training is:
[0110]
[0111]
[0112]
[0113] represents the training loss for single-intent intent recognition, represents the training loss for single intent slot filling, represents the training loss of the task-specific capability pre-training phase, β1 and β2 are learnable weight parameters, and n I refers to the total number of intent labels, represents the predicted value of the i-th intention, y I i represents the true value of the i-th intention, n S Refers to the total number of slot labels. represents the predicted value of the i-th slot, y S i Represents the true value of the i-th slot;
[0114] The training process of the second-stage model obtained above makes full use of the single-intent dialogue language understanding data to further train the model's dialogue language understanding task-related intent recognition and slot filling capabilities. Compared with multi-intent dialogue language understanding data, single-intent dialogue language understanding data is more abundant. This strategy can not only improve the task-related capabilities of the model dialogue language understanding, but also alleviate the problem caused by the scarcity of multi-intent dialogue language data.
[0115] 104, adjusting the second-stage model through the multi-intent dialogue language understanding fine-tuning data to obtain a multi-intent dialogue language understanding model;
[0116] The second-stage model is fine-tuned on a specific multi-intent conversation language understanding dataset, and a multi-task learning strategy is adopted to take multi-intent prediction and slot filling as training objectives, which are optimized together to achieve accurate parsing and response to complex conversation intents and entities.
[0117] The fine-tuning process only requires a small amount of multi-intent dialogue language understanding fine-tuning data to fine-tune the model to achieve SOTA performance. It only requires relatively abundant and easily available non-multi-intent dialogue language understanding data (named body recognition training data), single-intent dialogue language understanding data, and a small amount of multi-intent dialogue language understanding fine-tuning data to achieve better multi-intent dialogue language understanding results, which greatly alleviates the data scarcity problem faced by multi-intent dialogue language understanding research.
[0118] 105. Based on the multi-intent dialogue language understanding model, the multi-intent dialogue request to be analyzed is processed to obtain multiple intents and slots.
[0119] The beneficial effects of the embodiments of the present invention are:
[0120] Task-independent capability pre-training and task-specific capability pre-training can make full use of a large amount of non-dialogue language understanding data (named body recognition training data) and single-intent dialogue language understanding data, which can greatly alleviate the data scarcity and high cost problems in the field of multi-intent dialogue language understanding in practical applications, broaden the application scenarios of multi-intent dialogue language understanding, and promote the development of task-based dialogue systems.
[0121] In the above Figure 1 In the embodiment shown, a multi-intention dialogue language understanding method based on decoupled pre-training is described. Figure 3 The pre-training process diagram shown in the figure is used to describe the application example:
[0122] Figure 3(c) The user makes the following request to the smart assistant through voice command: "Add Despacito to POPplaylist and play it", which contains two intents: play music (PlayMusic) and add to playlist (AddToPlaylist);
[0123] First, the dialogue system captures the user's dialogue input and converts it into text form through speech recognition technology. The input is the user's specific words: "Add Despacito to POP playlist and play it";
[0124] In order to obtain a multi-intent dialogue language understanding system, in the task-independent capability pre-training stage, the named entity recognition dataset is used to jointly train the entity range and entity number to enhance its ability to recognize and understand semantic entities; for example, in the non-multi-intent dialogue "John works at Google in California", the system learns to recognize that "Google" is a company name (B-Company), "California" is a location (B-Location), and the sentence contains 3 entities, that is, the number of entities is 3, such as Figure 3 As shown in (a);
[0125] Entering the task-specific capability pre-training stage, further training is carried out through single-intent dialogue language understanding data, and an additional interaction module for intent recognition and slot filling encoding representation is added, thereby improving the model's ability to understand and predict intents and slots; in this stage, the relationship between user queries and intents and slots is simulated to strengthen the model's ability to perform specific intent recognition and slot filling tasks, such as Figure 2 (b) as shown;
[0126] After pre-training of task-independent and task-specific capabilities, the downstream task fine-tuning phase begins. At this point, the model is fine-tuned on a multi-intent dialogue dataset for a specific application based on the learning results of the previous phase, so that it can handle multiple intents at the same time and generate accurate dialogue responses in actual applications.
[0127] After the above three stages of pre-training and fine-tuning, we get the desired multi-intent dialogue language understanding module.
[0128] Finally, in the request made by the user, the multi-intent dialogue language understanding module was able to successfully identify two intents and fill the slots; finally, based on the analysis results, the system output the correct multiple intents: "Play Music" and "Add to Playlist" and the corresponding slots: "Despacito--B-Music, POP--B-Playlist", and performed the corresponding operations: played Despacito music and added it to the user's POP playlist.
[0129] In combination with the multi-intent conversation language understanding method based on decoupled pre-training described in the above embodiments, a multi-intent conversation language understanding system based on decoupled pre-training is described below through an embodiment.
[0130] like Figure 4 As shown, an embodiment of the present invention provides a multi-intent dialogue language understanding system based on decoupled pre-training, including:
[0131] The data acquisition module 401 is used to acquire a pre-training data set, where the pre-training data set includes named entity recognition training data, single-intent dialogue language understanding training data, and multi-intent dialogue language understanding training data;
[0132] The decoupled pre-training module 402 is used to perform task-independent capability pre-training on the preset dialogue language model using the named body recognition training data to obtain a first-stage model; perform task-specific capability pre-training on the first-stage model using the single-intent dialogue language understanding training data to obtain a second-stage model; and adjust the second-stage model using the multi-intent dialogue language understanding training data to obtain a multi-intent dialogue language understanding model;
[0133] The multi-intent dialogue language understanding module 403 is used to process the multi-intent dialogue request to be analyzed based on the multi-intent dialogue language understanding model to obtain multiple intents and slots.
[0134] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0135] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0136] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0137] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0138] The above are merely embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are included in the scope of the claims of the present invention to be approved.
Claims
1. A multi-intent dialogue language understanding method based on decoupled pre-training, characterized in that: include: Obtain named entity recognition training data, single-intent dialogue language understanding training data, and multi-intent dialogue language understanding fine-tuning data; Using the named body recognition training data to perform task-independent capability pre-training on a preset dialogue language model to obtain a first-stage model; Using the single-intent dialogue language understanding training data, pre-training the first-stage model for task-specific capabilities to obtain a second-stage model; The second-stage model is adjusted using the multi-intent dialogue language understanding fine-tuning data to obtain a multi-intent dialogue language understanding model; The multi-intent dialogue request to be analyzed is processed based on the multi-intent dialogue language understanding model to obtain multiple intents and slots; The step of using the named body recognition training data to perform task-independent capability pre-training on a preset dialogue language model to obtain a first-stage model includes: Pre-training the pre-built conversational language model for entity range prediction tasks; Pre-training the preset dialogue language model for an entity quantity prediction task; Using the named entity recognition training data, the preset dialogue language model is jointly trained with the entity range prediction task pre-training and the entity quantity prediction task pre-training to obtain a first-stage model; The pre-training of the preset dialogue language model for the entity range prediction task includes: Given input X = {[CLS], x1, x2, ..., x n }, the CLS represents a special tag used to capture the overall semantics of a sentence; the n refers to the number of named entities in the sentence; Use the preset dialogue language model to n } is encoded to obtain the hidden state vector H = {h [CLS] ,h1,h2,...,h n }; The entity range prediction task is pre-trained as: H=Encoder(X), e j =softmax(W E H j +b), The Encoder represents an encoder, the e j ∈{BX,IX,O}, where BX represents the beginning of entity X, IX represents the interior of entity X, O represents a non-entity, softmax represents an activation layer for outputting the final classification result of a word, and W E and b is a learnable parameter, the H j represents the hidden state representation of the jth word; The pre-training of the preset dialogue language model for the entity quantity prediction task includes: Given an input sentence h [CLS] ; The pre-training for the entity quantity prediction task is: m=softmax(Wh [CLS] +b), The m represents the predicted value of the maximum number of entities in the input sentence, the softmax represents an activation layer for outputting the classification result of the number of entities, and the W and b are learnable parameters; The step of pre-training the first-stage model for task-specific capabilities using the single-intent dialogue language understanding training data to obtain the second-stage model includes: Pre-set the interactive relationship between slot filling and intent recognition; Based on the interaction relationship, pre-training the first-stage model for task-specific capabilities using the single-intent dialogue language understanding training data to obtain a second-stage model; The task-specific capability pre-training includes: Use a sentence-level encoder to process the input sentence X and obtain the corresponding hidden representation H of the sentence; The new sentence representation obtained by the interaction of intent filling and intent recognition is: I={I [CLS] ,I1,I2,...,I n }, The I = Interaction-Module (H); Based on the entire sentence I [CLS] Semantic representation of , performing intent recognition tasks to classify sentence intents: I=softmax(W l [CLS] +b), The I represents the predicted intent classification, the softmax represents an activation layer for outputting the intent classification result, and the W and b are learnable parameters; Based on the representation l of the entire sentence, a slot filling task is performed to predict the slots in l: S = softmax(Wl+b), The S represents the result of predicting slot filling.
2. The multi-intention dialogue language understanding method based on decoupled pre-training according to claim 1 is characterized in that: The method further comprises: The loss function of the task-independent capability pre-training is: Said represents the training loss of the entity range prediction task, represents the training loss of the entity quantity prediction task, represents the training loss in the pre-training stage of task-independent capabilities, α1 and α2 are learnable weight parameters, and represents the predicted value of the jth named entity, the e j represents the true label of the jth named entity, the |m| refers to the maximum number of entities in the input sentence, the Represents the predicted value of the number of entities in the sentence, and m represents the true value of the number of entities in the sentence.
3. The multi-intention dialogue language understanding method based on decoupled pre-training according to claim 1 is characterized in that: The method further comprises: The loss function of the task-specific capability pre-training is: Said represents the training loss for single-intent intent recognition, represents the training loss for single intent slot filling, represents the training loss of the task-specific capability pre-training stage, the β1 and β2 are learnable weight parameters, and the n I Refers to the total number of intent labels. represents the predicted value of the i-th intention, the y I i represents the true value of the i-th intent, the n S Refers to the total number of slot labels. represents the predicted value of the i-th slot, the y S i Represents the true value of the i-th slot.
4. A multi-intention dialogue language understanding system based on decoupled pre-training, characterized in that: include: The data acquisition module is used to acquire named entity recognition training data, single-intent dialogue language understanding training data, and multi-intent dialogue language understanding fine-tuning data; A decoupled pre-training module is used to use the named body recognition training data to perform task-independent capability pre-training on the preset dialogue language model to obtain a first-stage model; use the single-intent dialogue language understanding training data to perform task-specific capability pre-training on the first-stage model to obtain a second-stage model; and adjust the second-stage model through the multi-intent dialogue language understanding fine-tuning data to obtain a multi-intent dialogue language understanding model; A multi-intent dialogue language understanding module, used to process the multi-intent dialogue request to be analyzed based on the multi-intent dialogue language understanding model to obtain multiple intents and slots; The decoupling pre-training module is specifically used to perform the following steps: Given input X = {[CLS], x1, x2, ..., x n }, the CLS represents a special tag used to capture the overall semantics of a sentence; the n refers to the number of named entities in the sentence; Use the preset dialogue language model to n } is encoded to obtain the hidden state vector H = {h [CLS] ,h1,h2,...,h n }; The entity range prediction task is pre-trained as: H=Encoder(X), e j =softmax(W E H j +b), The Encoder represents an encoder, the e j ∈{BX,IX,O}, where BX represents the beginning of entity X, IX represents the interior of entity X, O represents a non-entity, softmax represents an activation layer for outputting the final classification result of a word, and W E and b is a learnable parameter, the H j represents the hidden state representation of the jth word; Given an input sentence h [CLS] ; The pre-training for the entity quantity prediction task is: m=softmax(Wh [CLS] +b), The m represents the predicted value of the maximum number of entities in the input sentence, the softmax represents an activation layer for outputting the classification result of the number of entities, and the W and b are learnable parameters; Using the named entity recognition training data, the preset dialogue language model is jointly trained with the entity range prediction task pre-training and the entity quantity prediction task pre-training to obtain a first-stage model; Pre-set the interactive relationship between slot filling and intent recognition; Based on the interaction relationship, pre-training the first-stage model for task-specific capabilities using the single-intent dialogue language understanding training data to obtain a second-stage model; The task-specific capability pre-training includes: Use a sentence-level encoder to process the input sentence X and obtain the corresponding hidden representation H of the sentence; The new sentence representation obtained by the interaction of intent filling and intent recognition is: I={I [CLS] ,I1,I2,...,I n }, The I = Interaction-Module (H); Based on the entire sentence I [CLS] Semantic representation of , performing intent recognition tasks to classify sentence intents: I=softmax(Wl [CLS] +b), The I represents the predicted intent classification, the softmax represents an activation layer for outputting the intent classification result, and the W and b are learnable parameters; Based on the representation l of the entire sentence, a slot filling task is performed to predict the slots in l: S = softmax(Wl+b), The S represents the result of predicting slot filling.
