Pre-training model training method for dialogue state tracking, electronic device, storage medium and computer program product

Abstract

A pre-training model training method for dialogue state tracking, an electronic device, a storage medium and a computer program product are disclosed, wherein the method comprises: training a pre-training model on a plurality of auxiliary tasks respectively, wherein the plurality of auxiliary tasks are label-aware auxiliary tasks with a denoising target, and for each auxiliary task, a dialogue history or a dialogue state is added with noise, and the target of each auxiliary task is to restore the damaged content with the help of the remaining context; and fine-tuning the trained pre-training model on a dialogue state tracking task.

Description

Technical Field

[0001] This application relates to the field of dialogue state tracking technology, and in particular to a pre-trained model training method, electronic device, storage medium, and computer program product for dialogue state tracking. Background Technology

[0002] In related technologies, task-oriented dialogues (TOD) are very common in our daily lives, such as when we book hotels or restaurants. Dialogue State Tracking (DST) is crucial in current TOD systems. Its goal is to understand the user's needs based on the dialogue history within a domain. For example, Figure 1 shows a restaurant reservation. The system should identify the user's requirements for the restaurant. The belief state is the domain-slot-value triplet in the dialogue history.

[0003] With the development of pre-trained language models (PLMs), many research works have applied PLMs to DST tasks. Many of these works attempt to leverage large amounts of external dialogue corpora to assist downstream DST tasks. For example, TOD-BERT, based on the BERT architecture, was pre-trained on 1.4 million dialogues. Furthermore, SOLOIST further pre-trained GPT2 on 766K dialogues. PPTOD uses T5 as its backbone model and utilizes over 2.3 million dialogues. GALAXY, on the other hand, trained UniLM on approximately 36 million dialogues.

[0004] These models typically perform better on DST tasks after being pre-trained on large external dialogue corpora. Therefore, many researchers are exploring how to leverage more external data. Summary of the Invention

[0005] This invention provides a pre-training model training method, electronic device, and storage medium for dialogue state tracking, which at least solves one of the above-mentioned technical problems.

[0006] In a first aspect, embodiments of the present invention provide a method for training a pre-trained model for dialogue state tracking, comprising: training the pre-trained model on multiple auxiliary tasks, wherein the multiple auxiliary tasks are label-aware auxiliary tasks with denoising objectives, for each auxiliary task, adding noise to the dialogue history or dialogue state, wherein the objective of each auxiliary task is to recover the damaged content with the help of the remaining context; and fine-tuning the trained pre-trained model on the dialogue state tracking task.

[0007] In a second aspect, embodiments of the present invention provide an electronic device comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to execute any of the above-described pre-trained model training methods for dialogue state tracking of the present invention.

[0008] Thirdly, embodiments of the present invention provide a storage medium storing one or more programs including execution instructions, the execution instructions being readable and executable by electronic devices (including but not limited to computers, servers, or network devices, etc.) to perform any of the above-described pre-trained model training methods for dialogue state tracking of the present invention.

[0009] Fourthly, embodiments of the present invention also provide a computer program product, the computer program product including a computer program stored on a storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to execute any of the above-mentioned pre-trained model training methods for dialogue state tracking.

[0010] This application proposes a Label-Aware Auxiliary Learning for Dialogue State Tracking (LALDST) method, which focuses on leveraging the rich internal information of the target DST dataset to improve performance. The inventors designed label-aware auxiliary tasks in which a noise function is applied to dialogue history or dialogue state labels, and these labels are concatenated as input. The goal of each task is to recover the corrupted context. During training, the inventors first further train a large pre-trained language model on the auxiliary tasks, and then fine-tune it on the DST. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is an example of dialogue state tracking in existing technology; Figure 2 A flowchart illustrating a pre-trained model training method for dialogue state tracking, provided in an embodiment of the present invention; Figure 3 This is a data example of a tag-aware auxiliary task provided in an embodiment of the present invention; Figure 4 and Figure 5 The results of DST on MultiWOZ2.0 and WOZ provided in an embodiment of the present invention; Figure 6 The average improvement result of JGA relative to the T5 benchmark on MultiWOZ2.0 and WOZ is provided for an embodiment of the present invention. Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] Please refer to Figure 2 The diagram illustrates a flowchart of a pre-trained model training method for dialogue state tracking provided by an embodiment of the present invention.

[0015] like Figure 2 As shown, in step 101, the pre-trained model is trained on multiple auxiliary tasks, wherein the multiple auxiliary tasks are label-aware auxiliary tasks with denoising objectives. For each auxiliary task, the dialogue history or dialogue state is denoised. The objective of each auxiliary task is to recover the damaged content with the help of the remaining context. In step 102, the pre-trained model is fine-tuned on the dialogue state tracking task.

[0016] This application proposes a label-aware auxiliary learning method for dialogue state tracking, which focuses on leveraging the rich internal information of the target DST dataset to improve performance. Furthermore, multiple label-aware auxiliary tasks are designed, in which the inventors apply a noise function to dialogue history or dialogue state labels and concatenate them as input. The goal of each task is to recover the disrupted context. During training, the inventors first further train a large pre-trained language model on the auxiliary tasks and then fine-tune it on the dialogue state tracking tasks.

[0017] In some alternative embodiments, the input to the model is a concatenation of the dialogue history and the dialogue state, and the output of the model is the original data of the corrupted portion.

[0018] In some alternative embodiments, the model is a sequence-to-sequence generative pre-trained language model.

[0019] In some optional embodiments, the plurality of auxiliary tasks include dialogue state masking, dialogue history masking, user utterance generation, front-wheel system response generation, user dialogue arrangement, and system response arrangement.

[0020] In some optional embodiments, for the dialogue state masking, the domains, slots and values ​​in the dialogue state are randomly masked, while the dialogue history remains unchanged, so that the model outputs the original dialogue state. For the dialogue history masking, the domains, slots and values ​​appearing in the dialogue history are randomly masked while the dialogue state remains unchanged, so that the model outputs the complete dialogue history. For the generation of user utterances, one user utterance in the dialogue history is randomly masked, so that the model outputs the masked sentence based on the dialogue state and the rest of the dialogue history. For the generation of the previous round system response, the system response of a previous round is randomly masked, so that the model generates the masked response based on the remaining dialogue history and dialogue state; For the user dialogue arrangement, the user utterances in the dialogue history are randomly shuffled, while the order of the system responses in the sequence remains unchanged, so that the model outputs the original dialogue sequence; The system responses are randomly shuffled while the user's utterances remain unchanged. The goal of the task is to restore the original sentence order so that the model can output the original dialogue history.

[0021] In some optional embodiments, the dialogue state masking can help the model better understand the dialogue context and domain knowledge; the dialogue history masking, the user utterance generation, and the front-wheel system response generation can enhance the model's understanding of the dialogue context and domain knowledge; the user dialogue arrangement and the system response arrangement can enhance the model's understanding of the dialogue context.

[0022] In related technologies, dialogue state tracking is an important task in natural language processing related to dialogue systems. In a dialogue system, the goal of dialogue state tracking is to accurately capture the user's intent and needs during the conversation, so that the system can provide more accurate and targeted responses.

[0023] The current mainstream approach for dialogue state tracking involves fine-tuning a pre-trained language model. Fine-tuning based on a pre-trained language model generally includes the following steps: First, the output format of the dialogue state tracking task is processed into a token sequence suitable for model generation. Then, external dialogue data is used to further train the pre-trained model, making it transferable to the dialogue domain. Finally, the dialogue history is used as input, and the dialogue state as output, for direct fine-tuning on the pre-trained model.

[0024] The aforementioned methods suffer from at least the following drawbacks: First, they consume enormous computational resources. Training on a large amount of external dialogue data requires collecting high-quality dialogue data and then training the pre-trained model for a sufficient period, which consumes significant computational resources. Second, they do not adequately mine the datasets for downstream tasks. Dialogue state labels correspond to rich information contained in the dialogue history, but previous methods merely used them as outputs for the model to predict, without fully exploring the internal information of the existing labels. This is mainly due to the following reasons: On the one hand, there is an overemphasis on the impact of external dialogue data on downstream tasks: attempting to accumulate data volume and computational cost to achieve good results. On the other hand, the characteristics of the dialogue state tracking task itself are ignored: dialogue state labels contain rich content information, but previous methods merely used them as outputs for the model to predict, without attempting to further mine the internal information of the existing labels.

[0025] To address these shortcomings, most people skilled in the art would want to design a variety of tasks when pre-training using external dialogue data, while neglecting the target task itself. This is because the structured labels used in dialogue state tracking make it difficult to design auxiliary tasks to extract internal information.

[0026] Previous research has rarely focused on how to utilize the internal information of the DST dataset itself. Each dataset has corresponding dialogue state labels, which are equivalent to summaries of the dialogue history. The inventors hypothesize that enhancing the interaction between dialogue states and dialogue history during training can improve the model's understanding of annotation information, thereby improving the performance of DST. Therefore, the inventors focus on how to utilize the internal information of the DST dataset in this application.

[0027] Unlike previous methods, this application proposes a Label-Aware Auxiliary Learning for DST (LALDST), which focuses on improving performance by leveraging the rich internal information of the target DST dataset. The inventors designed label-aware auxiliary tasks in which a noise function is applied to dialogue history or dialogue state labels and concatenated as input. The goal of each task is to recover the corrupted context. During training, the inventors first further train a large pre-trained language model on the auxiliary tasks and then fine-tune it on the DST. Experimental results empirically demonstrate the performance improvement of LAL-DST on MultiWOZ2.0 and WOZ.

[0028] Models pre-trained on large-scale external dialogue corpora typically perform better on dialogue state tracking tasks. Therefore, many researchers are exploring how to leverage more external data. However, previous work has rarely focused on utilizing the internal information of the dialogue state tracking datasets themselves. Each dataset has its corresponding dialogue state labels, equivalent to a summary of the dialogue history. The inventors hypothesize that enhancing the interaction between dialogue state and dialogue history during training can improve the model's understanding of the labeled information, thereby improving dialogue state tracking performance. Therefore, embodiments of this application focus on mining the internal information of dialogue state tracking datasets.

[0029] The inventors used a sequence-to-sequence generative pre-trained language model as the backbone model. Based on this, they designed label-aware auxiliary tasks with a denoising objective. For each auxiliary task, noise was added to the dialogue history or dialogue state. The goal of each task was to recover the corrupted content with the help of the remaining context. During training, the model was first further trained on each auxiliary task, and then fine-tuned on a dialogue state tracking task.

[0030] Please refer to the following example, which shows a flowchart of a specific implementation of an embodiment of this application.

[0031] Step 1: A label-aware auxiliary task with a denoising objective was designed. In each task, the dialogue history or dialogue state was corrupted using a specific noise function and reconstructed with the help of the remaining context. The dialogue history and dialogue state were concatenated as input. The model output was the original data of the corrupted portion.

[0032] Task 0: Dialogue State Masking: The inventor will randomly mask the domains, slots, and values ​​in the dialogue state. The dialogue history remains unchanged. The model should output the original dialogue state.

[0033] Task 1: Dialogue History Masking: Randomly mask the fields, slots, and values ​​that appear in the dialogue history. The dialogue state remains unchanged. The output is the complete dialogue history.

[0034] Task 2: User Speech Generation: Randomly mask one instance of a user speech from the dialogue history. The model should output the masked sentence based on the dialogue state and the rest of the dialogue history.

[0035] Task 3: Generation of Previous Round System Responses: The inventor randomly masks the system responses from a previous round. The model should generate the masked responses based on the remaining dialogue history and dialogue state.

[0036] Task 4: User Dialogue Rearrangement: User utterances in the dialogue history are randomly shuffled. The order of system responses in the sequence remains unchanged. The output is the original dialogue sequence.

[0037] Task 5: System Response Rearrangement: System responses are randomly shuffled, while the user's utterances remain unchanged. The goal of this task is to restore the original sentence order. The model should output the original dialogue history.

[0038] Step 2: Train the pre-trained model on each auxiliary task separately. Step 3: Fine-tune the model obtained in Step 2 on the dialogue state tracking task. Task 0 involves predicting masked domain, slot, and value labels based on the context and dialogue history of the unmasked dialogue state, thereby enhancing the model's understanding of dialogue state representation and domain knowledge. This helps the model learn the dependencies between masked labels and other labels in their respective domain-slot-value triples. Simultaneously, the model reinforces its understanding of structured output formats. Therefore, this model can improve the understanding of dialogue state representation and domain knowledge.

[0039] Task 1 helps the model better understand dialogue context and domain knowledge. In this task, the model should predict masked dialogue state labels in the dialogue history. This training enables the model to learn the contextual relationships between parts of the dialogue history related to the dialogue states, ultimately improving its understanding of dialogue context and domain knowledge.

[0040] Tasks 2 and 3 further enhance the model's understanding of dialogue context and domain knowledge. Each sentence in a dialogue sequence typically contains partial information about the dialogue state. Therefore, the dialogue state helps generate masked sentences. The model needs to predict whether the masked sentences contain which domain slot triples and reconstruct the dialogue based on this.

[0041] Tasks 4 and 5 enhance the model's understanding of dialogue context. Changes in sentence order affect the final dialogue state. For example, a user might change their needs during the conversation. If the sentence expressing the final need is moved to the beginning and the initial need to the end, the dialogue state changes. Therefore, dialogue state is related to sentence order. By reordering sentences using dialogue state, the model improves its understanding of dialogue context.

[0042] In the process of realizing this application, the inventors also adopted the following approach: when training on auxiliary tasks, training can be performed on multiple auxiliary tasks simultaneously, rather than on a single task. The advantage of this approach is that it can fully utilize the potential of the auxiliary tasks; the disadvantage is that it is difficult to train.

[0043] The solution in this application mainly involves designing tasks for information mining based on the target task of dialogue state tracking, and designing auxiliary tasks around dialogue state tags.

[0044] The following provides specific experiments and related data to enable those skilled in the art to better understand the solution of this application.

[0045] This application proposes LAL-DST to leverage the rich internal information associated with dialogue state labels in the DST dataset. Since seq2seq PLM has become mainstream across various fields, the inventors adopted a T5-based pre-trained seq2seq model as the backbone model. The seq2seq DST model is expected to understand dialogue context, dialogue state representation, and domain knowledge well. To enhance the model's understanding of these by leveraging these elements, the inventors designed label-aware auxiliary tasks aimed at denoising. For each auxiliary task, a noise function is applied to the dialogue history or dialogue state. The goal of each task is to recover the corrupted content with the help of other contexts.

[0046] During training, the inventors first further trained the T5-base model on each auxiliary task. Then, they fine-tuned the model on DST.

[0047] Figure 3This example shows data for a label-aware assisted task. All of this data is derived from the same data sample, which contains three rounds of dialogue history and dialogue states consisting of only a domain-slot-value triplet. [SEP] is a special separator. [MASK] indicates masked content. U i * and R i * These are some blocked sentences related to the dialogue state.

[0048] Compared to previous methods, the inventors did not utilize external large datasets for pre-training, thus saving computational resources and training time. Finally, the inventors demonstrated the improvements of LAL-DST over MultiWOZ2.0 and WOZ.

[0049] method In the DST task, the inventors generate the dialogue state B based on the dialogue history H of each round. For the t-th round, H... t User speech U from the first t-1 rounds i and system reply R i And the current round's U t Composition, that is H t = [U1,R1,...,U t-1 ,R t-1 U t (1) Dialogue state B is a sequence of domain-slot-value triples. For round t... B t = [(d1,s1,v1),...,(d nt ,s nt ,v nt (2) Where, d i s i and v i These refer to the field, slot, and value of the i-th triple, respectively, where nt is B. t The number of domain-slot-value triples. The inventors describe DST as a seq2seq generation task. B t = F(H t (3) Here, F represents the generative model. Both the dialogue history and dialogue states are flattened into a sequence of tokens. During training, the model optimizes for the negative log probability of serializing dialogue states. During inference, the model directly generates all possible domain-slot-value triples given the dialogue history. To achieve good performance on DST, sequence-to-sequence models must have a strong understanding of the following: - The dialogue context model should be able to understand the dialogue context, including the domain and the user's intent and requirements.

[0050] - A structured representation model of dialogue state should be able to generate dialogue state in a structured format.

[0051] - Domain knowledge: By fully understanding the domain knowledge specific to the task, including the possible slots and values ​​in each domain, the model can understand the dialogue context and output the dialogue state more accurately.

[0052] 2.2. Label-aware assisted task The data in DST consists of two parts: dialogue history and dialogue state labels. To help the model understand these three parts, the inventors designed a label-aware auxiliary task aimed at denoising, in order to further utilize the connection between these two parts.

[0053] In each task, the inventors disrupt the dialogue history or dialogue state using a specific noise function and then reconstruct it with the help of other contexts. When input into the model, the dialogue history and dialogue state are concatenated, with a portion of them disrupted. The output is the original data containing the disrupted portion. Below is an example of the label-aware task designed by the inventors. Figure 6 As shown.

[0054] Task 0: Dialogue State Masking: The inventor will randomly mask the domains, slots, and values ​​in the dialogue state. The dialogue history remains unchanged. The model should output the original dialogue state.

[0055] Task 1: Dialogue History Masking: Randomly mask the fields, slots, and values ​​that appear in the dialogue history. The dialogue state remains unchanged. The output is the complete dialogue history.

[0056] Task 2: User Speech Generation: Randomly mask one instance of a user speech from the dialogue history. The model should output the masked sentence based on the dialogue state and the rest of the dialogue history.

[0057] Task 3: Generation of Previous Round System Responses: The inventor randomly masks the system responses from a previous round. The model should generate the masked responses based on the remaining dialogue history and dialogue state.

[0058] Task 4: User Dialogue Rearrangement: User utterances in the dialogue history are randomly shuffled. The order of system responses in the sequence remains unchanged. The output is the original dialogue sequence.

[0059] Task 5: System Response Rearrangement: System responses are randomly shuffled, while the user's utterances remain unchanged. The goal of this task is to restore the original sentence order. The model should output the original dialogue history.

[0060] Design Concept In the embodiments of this application, the inventors will discuss the effects of each label perception assistance task. Specifically, the inventors will discuss in what aspects each task enhances the model's understanding.

[0061] Task 0 enhances the model's understanding of "structured representations of dialogue states" and "domain knowledge" by predicting masked domain, slot, and value labels based on the context of the unmasked dialogue state and the dialogue history. This helps the model learn the dependencies between masked labels and other labels in their respective domain-slot-value triples. Simultaneously, the model strengthens its understanding of structured output formats. Therefore, the model can improve its understanding of dialogue state representations and domain knowledge.

[0062] Task 1 helps the model better understand dialogue history and domain knowledge. In this task, the model needs to predict masked dialogue state labels in the dialogue history. Through this training, the model can learn the contextual content of parts of the dialogue history related to the dialogue states, ultimately improving its understanding of dialogue context and domain knowledge.

[0063] Tasks 2 and 3 also enhance the model's understanding of dialogue context and domain knowledge. Each sentence in a dialogue sequence typically contains partial information about the dialogue state. Therefore, the dialogue state helps generate masked sentences. The model needs to predict whether the masked sentences contain which domain point-value triples and reconstruct the dialogue accordingly.

[0064] Tasks 4 and 5 can enhance the model's understanding of dialogue context. Changes in sentence order can affect the final dialogue state. For example, a user might change their request during the conversation. If the sentence expressing the user's final request is moved to the beginning, while the sentence expressing their initial request is moved to the end, the user's dialogue state will change. Therefore, dialogue state and sentence order are related. By reordering sentences with the help of dialogue state, the model has the potential to improve its understanding of dialogue context.

[0065] Figure 4The DST results on the MultiWOZ2.0 test set are shown. √ and × indicate whether the method used an external dialogue corpus. The English and Chinese translations are as follows: Model; External Data.

[0066] Training process Before training, the inventors generate new data based on the original dataset according to the definition of the auxiliary task for each label. To test the effectiveness of the auxiliary task, the inventors first further train a pre-trained large language model T5-base on each task. Then, the inventors fine-tune the model on DST and compare its performance with a baseline that is fine-tuned directly on T5-base on DST.

[0067] Experimental Dataset MultiWOZ 2.0 is a large-scale, multi-domain task-oriented dialogue dataset containing over 10,000 annotated dialogues. It spans multiple domains, including trains, hotels, restaurants, hospitals, tourist attractions, and taxis. WOZ is a dialogue dataset focused on single-domain tasks, covering restaurant search.

[0068] Evaluate The primary evaluation metric for DST is Joint Objective Accuracy (JGA). For each data sample, a prediction is considered correct only if it completely contains all domain-slot-value triples of the actual dialogue state.

[0069] Figure 5 The results of DST on the WOZ test set are shown. √ and × indicate whether the method uses an external dialogue corpus.

[0070] result Figure 4 and Figure 5 These are the results of DST on MultiWOZ2.0 and WOZ. Figure 4 and Figure 5 The results are from DST on MultiWOZ2.0 and WOZ. The following section presents the inventors' experiments. First, the inventors list the baseline results of directly fine-tuning T5base on DST. Then, they demonstrate the performance on label-aware auxiliary tasks. For MultiWOZ2.0 and WOZ, the inventors observe that the model outperforms the baseline in 5 out of 6 auxiliary tasks, demonstrating the effectiveness of the tasks.

[0071] For MultiWOZ2.0, Task 3 achieves the highest Joint Goal Accuracy (JGA) score, which is 0.99% higher than the baseline. In addition, it improves the JGA by 0.82% compared with previous methods.

[0072] In terms of WOZ, Task 2 - User Utterance Generation performs the best, and its JGA is 1% higher than the baseline. Besides, it improves the JGA by 0.2% compared with previous methods.

[0073] Figure 6 It shows the average improvement result of JGA relative to the T5 baseline on MultiWOZ2.0 and WOZ. The Chinese and English comparison is as follows: Improvement of JGA,Improvement of JGA.

[0074] Analysis Effect of label-aware tasks. The inventor empirically demonstrates the effect of the label-aware task through the improvement of the label-aware task relative to the baseline. In addition, the performance of LAL-DST is better than previous methods using external dialogue corpora on both MultiWOZ2.0 and WOZ. The performance of LAL-DST is also better than those more complex methods that do not use external dialogue corpora, such as L4P4K2-DSGraph and Seq2SeqDU. This shows that the inventor indeed better develops and utilizes rich internal information.

[0075] Generalization ability of Task 0 and Task 3. Figure 6 It shows the average improvement result of JGA relative to the T5 baseline on two datasets. It can be seen that Task 0 and Task 3 are the two best auxiliary tasks. They perform well on both datasets and show better generalization ability compared with other tasks. For Task 0 "Dialogue State Masking", the inventor推测 that its strong generalization performance is due to its being very similar to DST in form. For Task 3 "Previous Response Generation", the inventor believes that training on this task can effectively improve the model's understanding of "dialogue context" and "domain knowledge", because system responses usually contain rich dialogue state information. Conjecture on the poor performance of Task 1. From Figure 6 it can be seen that Task 1 - Dialogue Masking performs the worst among all tasks. The main reason is that the inventor did not consider changes in linguistic forms when masking labels. For example, the value "cheap" can have other expressions, such as "affordable", "low-priced", etc. But the inventor only masked the label "cheap" in the dialogue history. Omitting variants will lead to insufficient utilization of the entire dialogue history.

[0076] The advantages of learning knowledge in a specific field. Figure 6 As we can see, tasks that attempt to enhance the understanding of domain knowledge perform well (except for Task 1). This suggests that direct fine-tuning may lead to insufficient integration of domain knowledge into the model, thus highlighting the necessity of improving model performance from this perspective.

[0077] in conclusion In this embodiment, the inventors leveraged the rich internal information of the original target DST dataset by constructing label-aware auxiliary tasks. In each label-aware task, a noise function was applied to the dialogue history or dialogue state. The goal of these tasks was to recover corrupted content. During training, the inventors first trained the model on each auxiliary task and then fine-tuned it on the DST tasks. The inventors demonstrated the effectiveness of LAL-DST through improvements achieved on multi-domain and single-domain datasets.

[0078] In other embodiments, the present invention also provides a non-volatile computer storage medium storing computer-executable instructions that can execute the pre-trained model training method for dialogue state tracking in any of the above method embodiments. In one embodiment, the non-volatile computer storage medium of the present invention stores computer-executable instructions, which are configured as follows: The pre-trained model is trained on multiple auxiliary tasks, wherein the multiple auxiliary tasks are label-aware auxiliary tasks with denoising objectives. For each auxiliary task, noise is added to the dialogue history or dialogue state. The objective of each auxiliary task is to recover the damaged content with the help of the remaining context. Fine-tune the pre-trained model after training on the dialogue state tracking task.

[0079] Non-volatile computer-readable storage media may include a stored program area and a stored data area, wherein the stored program area may store an operating system, an application program required for at least one function, and the stored data area may store data created based on the use of a pre-trained model training device for dialogue state tracking. Furthermore, the non-volatile computer-readable storage medium may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include memory remotely configured relative to a processor, which can be connected via a network to a pre-trained model training device for dialogue state tracking. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0080] This invention also provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium. The computer program includes program instructions, which, when executed by a computer, cause the computer to perform any of the above-described pre-trained model training methods for dialogue state tracking.

[0081] Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiment of the present invention, such as... Figure 7 As shown, the device includes: one or more processors 710 and memory 720. Figure 7 Taking a processor 710 as an example, the device for training a pre-trained model for dialogue state tracking may further include an input device 730 and an output device 740. The processor 710, memory 720, input device 730, and output device 740 can be connected via a bus or other means. Figure 7 Taking a bus connection as an example, the memory 720 is the aforementioned non-volatile computer-readable storage medium. The processor 710 executes various server functions and data processing by running non-volatile software programs, instructions, and modules stored in the memory 720, thereby implementing the pre-trained model training method for dialogue state tracking described in the above method embodiment. The input device 730 can receive input numeric or character information and generate key signal inputs related to user settings and function control of the pre-trained model training device for dialogue state tracking. The output device 740 may include a display screen or other display device.

[0082] The above-described product can execute the method provided in the embodiments of the present invention, and has the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in the embodiments of the present invention.

[0083] In one implementation, the above-described electronic device is used in a pre-trained model training apparatus for dialogue state tracking, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to: The pre-trained model is trained on multiple auxiliary tasks, wherein the multiple auxiliary tasks are label-aware auxiliary tasks with denoising objectives. For each auxiliary task, noise is added to the dialogue history or dialogue state. The objective of each auxiliary task is to recover the damaged content with the help of the remaining context. Fine-tune the pre-trained model after training on the dialogue state tracking task.

[0084] The electronic devices described in this application exist in various forms, including but not limited to: (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include: smartphones (e.g., iPhones), multimedia phones, feature phones, and low-end phones, etc.

[0085] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as the iPad.

[0086] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes: audio and video players (e.g., iPods), handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0087] (4) Server: A device that provides computing services. The components of a server include a processor, hard disk, memory, system bus, etc. Servers are similar to general computer architectures, but because they need to provide highly reliable services, they have higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0088] (5) Other electronic devices with data interaction functions.

[0089] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0090] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for training a pre-trained model for dialogue state tracking, wherein, The model is a sequence-to-sequence generative pre-trained language model, and the method includes: The pre-trained model is trained on multiple auxiliary tasks, wherein the multiple auxiliary tasks are label-aware auxiliary tasks with denoising objectives. For each auxiliary task, the dialogue history or dialogue state is denoised. The objective of each auxiliary task is to recover the damaged content with the help of the remaining context. The multiple auxiliary tasks include dialogue state masking, dialogue history masking, user speech generation, previous system response generation, user dialogue arrangement, and system response arrangement. Fine-tune the pre-trained model after training on the dialogue state tracking task; Specifically, for the dialogue state masking, domains, slots, and values ​​in the dialogue state are randomly masked, while the dialogue history remains unchanged, allowing the model to output the original dialogue state. For the dialogue history masking, domains, slots, and values ​​appearing in the dialogue history are randomly masked, while the dialogue state remains unchanged, allowing the model to output the complete dialogue history. For the user utterance generation, one user utterance in the dialogue history is randomly masked, allowing the model to output the masked sentence based on the dialogue state and the rest of the dialogue history. For the previous round system response generation, a previous round system response is randomly masked, allowing the model to generate the masked response based on the remaining dialogue history and dialogue state. For the user dialogue arrangement, user utterances in the dialogue history are randomly shuffled, while the order of system responses in the sequence remains unchanged, allowing the model to output the original dialogue sequence. For the system response arrangement, system responses are randomly shuffled, while user utterances remain unchanged. The goal of the task is to restore the original sentence order, allowing the model to output the original dialogue history.

2. The method according to claim 1, wherein, The input to the model is the concatenation of the dialogue history and the dialogue state, and the output of the model is the original data of the destroyed part.

3. The method according to claim 1, wherein, The dialogue state masking helps the model better understand the dialogue context and domain knowledge; The dialogue history masking, user utterance generation, and front-wheel system response generation can enhance the model's understanding of dialogue context and domain knowledge; The arrangement of user dialogues and the arrangement of system responses can enhance the model's understanding of the dialogue context.

4. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1 to 3.

5. A storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 3.

6. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 3.

Images