Dialogue data augmentation method, electronic device and storage medium

Abstract

The present application discloses a method for augmenting conversation data, an electronic device and a storage medium. The method for augmenting conversation data comprises: obtaining conversation template prompt information; calling a pre-trained language model whose parameter scale is greater than a parameter quantity threshold, and applying the pre-trained language model to independently perform conversation data augmentation processing based on the conversation template prompt information. The method for augmenting conversation data of the present application can effectively improve the augmentation efficiency of open-end conversation data, reduce the complexity of the augmentation process, improve the augmentation scale and generalization ability of open-end conversation data, and thus improve the effectiveness and reliability of model training based on augmented conversation data.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a conversation data augmentation method, an electronic device, and a storage medium. Background Art

[0002] Current pre-trained dialogue models usually require a large amount of scenario-specific dialogue data for training in open-ended dialogue downstream tasks (such as emotional support). Since this part of data is generally not directly available, crowdsourcing is used to construct dialogue data in actual applications. However, crowdsourcing is costly, time-consuming, and has complex quality control. In addition, crowdsourced dialogues are small in scale and have limited topics. Models trained based on crowdsourced data are difficult to handle unseen topics. Therefore, non-crowdsourced dialogue data augmentation technology has become a research focus.

[0003] At present, the existing non-crowdsourcing dialogue data augmentation methods mainly focus on task-oriented dialogue data augmentation, that is, user speech needs to be based on preset user goals, system speech needs to be based on preset knowledge bases and dialogue behavior sets, etc., which is difficult to migrate to open-end dialogue data augmentation. In addition, the existing dialogue data augmentation methods mainly generate pseudo dialogue data through simulated interactions between two independent models, but when the model scale increases, the time spent on the two models alternately generating replies increases greatly, making it difficult to augment dialogue data on a large scale, and the generalization ability is poor, which limits the efficiency, scale and generalization ability of open-end dialogue data augmentation. Summary of the invention

[0004] The purpose of the present application is to provide a method for augmenting conversational data, an electronic device, and a storage medium. In order to have a basic understanding of some aspects of the disclosed embodiments, a simple summary is given below. This summary is not a general review, nor is it intended to identify key / important components or describe the scope of protection of these embodiments. Its only purpose is to present some concepts in a simple form as a preface to the detailed description that follows.

[0005] According to one aspect of an embodiment of the present application, a method for augmenting conversation data is provided, comprising:

[0006] Get the dialog template prompt information;

[0007] A pre-trained language model having a parameter scale greater than a parameter amount threshold is called, and the pre-trained language model is applied to independently perform dialog data augmentation processing based on the dialog template prompt information.

[0008] In some implementations of the present application, before the pre-trained language model is applied to independently perform dialog data augmentation processing based on the dialog template prompt information, the process further includes:

[0009] The pre-trained language model is fine-tuned using dialogue samples whose number is less than a sample threshold.

[0010] In some implementations of the present application, the obtaining of the dialog template prompt information includes:

[0011] Obtaining a starting sentence corresponding to a preset target scene and introduced by a first preset word;

[0012] Based on the description information of the current target scene, the starting sentence introduced by the first preset word, and the second preset word used to prompt the pre-trained language model to continue writing, the dialogue template prompt information corresponding to the target scene is generated.

[0013] In some embodiments of the present application, obtaining a starting sentence corresponding to a preset target scene and introduced by a first preset word includes:

[0014] Extracting a dialogue start sentence from an open source dialogue dataset corresponding to the target scenario and / or a shared text for the target scenario;

[0015] Based on the first preset word and the dialogue initiating sentence, an initiating sentence introduced by the first preset word is generated.

[0016] In some implementations of the present application, the applying the pre-trained language model to independently perform dialogue data augmentation processing based on the dialogue template prompt information includes:

[0017] The dialogue template prompt information is written into a preset dialogue template, so that the pre-trained language model continues to write dialogue data in the dialogue template starting with the second preset word to complete the augmentation processing of the dialogue data.

[0018] In some embodiments of the present application, the target scenario includes: an emotional support dialogue scenario;

[0019] Correspondingly, fine-tuning the pre-trained language model using dialogue samples whose number is less than the sample threshold includes:

[0020] In the emotion support dialogue dataset ESConv under the framework of the emotion support task ESC, select dialogue samples whose number is less than the sample threshold;

[0021] The pre-trained language model is fine-tuned using the conversation sample.

[0022] In some embodiments of the present application, the target scenario includes: an emotional support dialogue scenario;

[0023] Correspondingly, the open source dialogue dataset includes: the open source empathetic dialogue dataset EmpatheticDialogues; the shared text includes: mental health text content in the public social platform Reddit.

[0024] In some embodiments of the present application, the pre-trained language model includes: GPT-J.

[0025] According to another aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the above-mentioned methods for augmenting conversation data.

[0026] According to another aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored, and the program is executed by a processor to implement any of the above-mentioned methods for augmenting conversation data.

[0027] The technical solution provided by one aspect of the embodiments of the present application may have the following beneficial effects:

[0028] Compared with the prior art, the conversation data augmentation method provided in the embodiment of the present application (1) effectively improves the augmentation efficiency of the open-end conversation data by using a pre-trained language model to independently perform conversation data augmentation processing, thereby avoiding the problem of long time consumption caused by simulated interaction between two models; (2) by pre-setting and using conversation template prompt information, the complexity of the augmentation process can be effectively reduced, and steps such as collecting basic information of users through fixed questions, extracting some questions from a preset question library according to a strategy, constructing a main question stream to ask users, classifying user replies, and selecting corresponding reply sentences as responses to users can be effectively omitted, thereby further improving the augmentation efficiency of the open-end conversation data and reducing data redundancy; (3) by calling a pre-trained language model with a parameter scale greater than a parameter quantity threshold, the augmentation scale and generalization ability of the open-end conversation data can be effectively improved, thereby improving the effectiveness and reliability of model training based on the augmented conversation data.

[0029] Other features and advantages of the present application will be described in the subsequent description, and some of them will become obvious from the description, or some of them can be inferred or determined unambiguously from the description, or can be understood by implementing the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0031] Figure 1A flow chart showing a method for augmenting conversation data according to some embodiments of the present application is shown;

[0032] Figure 2 A flowchart of obtaining dialog template prompt information in some embodiments of the present application is shown;

[0033] Figure 3 A flowchart of obtaining a starting sentence introduced by a first preset word corresponding to a preset target scene in some embodiments of the present application is shown;

[0034] Figure 4 A schematic diagram showing an augmented record text of English conversation data of a specific example of the present application;

[0035] Figure 5 A block diagram of an electronic device structure of some embodiments of the present application is shown;

[0036] Figure 6 A schematic diagram of a computer-readable storage medium illustrating some embodiments of the present application.

[0037] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0039] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0040] There are few technologies for data augmentation of open-ended conversations in the prior art, and they mainly focus on data augmentation of task-oriented conversations (i.e., user speech needs to be based on preset user goals, and system speech needs to be based on preset knowledge bases and conversation behavior sets). The inventors found that it is difficult to migrate the prior art to data augmentation of open-ended conversations, and its shortcomings mainly include the following two aspects:

[0041] First, the computational efficiency is low, and it is difficult to augment the conversation data on a large scale. Existing technologies generally generate pseudo conversation data through simulated interactions between models (for example, a user model and a system model chatting with each other). However, when the model scale increases, the time it takes for the two models to generate replies alternately increases greatly, making it difficult to augment the conversation data on a large scale, and also limiting the use of large pre-trained models.

[0042] Second aspect: insufficient generalization ability, difficult to expand topics. Existing technologies can generally only expand topics within the domain (i.e., covered by the conversation data used for training), and cannot generalize to topics outside the domain (i.e., not included in the training data); the crowdsourced conversation database is small in scale and has a large range of topics, which makes it difficult for models trained based on crowdsourced data to handle unseen topics. The inability of existing technologies to use large pre-trained models also further constrains the generalization ability of the method.

[0043] When considering using a pre-trained language model for conversation data augmentation, it is necessary to first collect the user's basic information through fixed questions; extract some questions from the preset question library according to the strategy, and build the main question stream to ask the user; classify the user's replies, and select the corresponding reply sentences as responses to the user; then use the pre-trained language model to generate follow-up questions related to the user's reply based on the current question and the user's reply. Although this method uses a pre-trained language model, it still requires a lot of complex work, so it cannot fundamentally solve the problem of low efficiency in existing conversation data augmentation. At the same time, the scale of pre-trained language models is large, and if they are not effectively selected, it is still impossible to improve the augmentation scale and generalization ability of open-end conversation data.

[0044] like Figure 1 As shown, an embodiment of the present application provides a method for augmenting conversation data. In some implementations of this embodiment, the method includes steps S10 to S20:

[0045] S10. Obtaining dialog template prompt information.

[0046] like Figure 2 As shown, in some implementations, obtaining the dialog template prompt information includes steps S101 to S102:

[0047] S101: Obtain an initial sentence corresponding to a preset target scene and introduced by a first preset word.

[0048] like Figure 3 As shown, in some embodiments, obtaining a starting sentence corresponding to a preset target scene and introduced by a first preset word includes:

[0049] S1011. Extracting a dialogue start sentence from an open source dialogue dataset corresponding to the target scenario and / or a shared text for the target scenario.

[0050] S1012: Based on a first preset word and the dialogue initiating sentence, generate an initiating sentence introduced by the first preset word.

[0051] A specific example of an augmented transcript of English conversation data is as follows: Figure 4 As shown, in Figure 4 In the augmented record text of the English conversation data shown, the first paragraph "The following is a conversation with an AI assistant. The assistant is helpful, empathetic, clever, and very friendly. It can use various support skills to provide emotional support to human." describes the preset target scenario. The first preset word is Human, and the starting sentence introduced by the first preset word is "Human: I moved into a new state recently, and there's a lot to do, but I don't have any friends in the new place I stay at." The second preset word is AI.

[0052] S102: Generate dialogue template prompt information corresponding to the target scene based on the description information of the current target scene, the starting sentence introduced by the first preset word, and the second preset word used to prompt the pre-trained language model to continue writing.

[0053] In the above specific example, based on the description information of the current target scene, the starting sentence introduced by the first preset word Human, and the second preset word AI used to prompt the pre-trained language model to continue writing, the dialogue template prompt information corresponding to the target scene is generated as follows:

[0054] “The following is a conversation with an AI assistant. The assistant is helpful, empathetic, clever, and very friendly. It can use various supportskills to provide emotional support to human.

[0055] Human:I moved into a new state recently,and there's a lot to do,but Idon't have any friends in the new place I stay at.

[0056] AI: ".

[0057] S20: calling a pre-trained language model whose parameter scale is greater than a parameter amount threshold, and applying the pre-trained language model to independently perform dialogue data augmentation processing based on the dialogue template prompt information.

[0058] In some embodiments, the pre-trained language model includes: GPT-J.

[0059] In some implementations, before the pre-trained language model is used to independently perform dialog data augmentation processing based on the dialog template prompt information, the method of this embodiment further includes:

[0060] The pre-trained language model is fine-tuned using dialogue samples whose number is less than a sample threshold.

[0061] In some embodiments, the applying the pre-trained language model to independently perform dialogue data augmentation processing based on the dialogue template prompt information includes:

[0062] The dialogue template prompt information is written into a preset dialogue template, so that the pre-trained language model continues to write dialogue data in the dialogue template starting with the second preset word to complete the augmentation processing of the dialogue data.

[0063] In certain embodiments, the target scenario includes: an emotional support conversation scenario;

[0064] Correspondingly, fine-tuning the pre-trained language model using dialogue samples whose number is less than the sample threshold includes:

[0065] In the emotion support dialogue dataset ESConv under the framework of the emotion support task ESC, select dialogue samples whose number is less than the sample threshold;

[0066] The pre-trained language model is fine-tuned by applying the conversation sample. Fine-tuning the pre-trained language model by applying the conversation sample can improve the quality of the conversation continued by the pre-trained language model, so that it fully retains sufficient generalization ability and is better in generalization of new topics.

[0067] Correspondingly, the open source conversation dataset includes: the open source empathetic dialogue dataset EmpatheticDialogues; the shared text includes: mental health text content in the public social platform Reddit. Using the open source empathetic dialogue dataset EmpatheticDialogues and the mental health text content in the public social platform Reddit can improve the topic diversity and richness of the triggered conversations.

[0068] In the augmented record text of the English conversation data in the above specific example, the remaining part of the text is the remaining conversation that can be continued by GPT-J given a template prompt by the autoregressive language model. The autoregressive language model can be, for example, a GPT-J model. The autoregressive language model can predict the following text based on the previous text or predict the previous text based on the following text. Using the pre-trained language model, the augmented conversation data can be processed into a language modeling task; using an informative start sentence to trigger a conversation with a variety of topics. The augmented method of language modeling only requires one generation, which greatly improves the computational efficiency compared to the method of using two independent models to simulate interaction and alternate generation. The computational efficiency is high, and the use of large pre-trained language models becomes feasible; the largest open source pre-trained language model GPT-J (containing 6 billion parameters) is used, and only a small amount of sample fine-tuning is required to generate high-quality conversation data, and the reliance on crowdsourced conversation data is less; the GPT-J model used is large in scale and has rich knowledge itself. The small amount of sample fine-tuning allows it to fully retain sufficient generalization ability and better generalization on new topics.

[0069] GPT-J has a parameter scale of 6 billion and is currently the largest open source generative pre-trained language model. Its huge scale can ensure the efficient completion of the task of continuing the conversation in the form of language modeling.

[0070] In order to improve the quality of the dialogues continued by GPT-J, it can be fine-tuned in advance on a small number (for example, 100) of dialogues of downstream tasks (for example, for emotional support dialogue scenarios, the dataset ESConv can be used). Fine-tuning on a small number of samples is to retain the original knowledge of GPT-J, prevent overfitting, and improve generalization on new topics.

[0071] Compared with the prior art, the conversation data augmentation method provided in the embodiment of the present application (1) effectively improves the augmentation efficiency of the open-end conversation data by using a pre-trained language model to independently perform conversation data augmentation processing, thereby avoiding the problem of long time consumption caused by simulated interaction between two models; (2) by pre-setting and using conversation template prompt information, the complexity of the augmentation process can be effectively reduced, and steps such as collecting basic information of users through fixed questions, extracting some questions from a preset question library according to a strategy, constructing a main question stream to ask users, classifying user replies, and selecting corresponding reply sentences as responses to users can be effectively omitted, thereby further improving the augmentation efficiency of the open-end conversation data and reducing data redundancy; (3) by calling a pre-trained language model with a parameter scale greater than a parameter quantity threshold, the augmentation scale and generalization ability of the open-end conversation data can be effectively improved, thereby improving the effectiveness and reliability of model training based on the augmented conversation data.

[0072] Another embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the conversation data augmentation method described in any of the above embodiments.

[0073] like Figure 5 As shown, the electronic device 10 may include: a processor 100, a memory 101, a bus 102 and a communication interface 103, and the processor 100, the communication interface 103 and the memory 101 are connected via the bus 102; the memory 101 stores a computer program that can be run on the processor 100, and the processor 100 executes the method provided in any of the aforementioned embodiments of the present application when running the computer program.

[0074] The memory 101 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 103 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0075] The bus 102 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 101 is used to store programs, and the processor 100 executes the programs after receiving execution instructions. The method disclosed in any implementation of the above-mentioned embodiment of the present application may be applied to the processor 100, or implemented by the processor 100.

[0076] The processor 100 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 100 or an instruction in the form of software. The above processor 100 may be a general-purpose processor, which may include a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 101, and the processor 100 reads the information in the memory 101 and completes the steps of the above method in combination with its hardware.

[0077] The electronic device provided in the embodiment of the present application and the method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, operated or implemented by them.

[0078] Another embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the conversation data augmentation method described in any of the above embodiments.

[0079] The present application also provides a computer-readable storage medium corresponding to the method provided in the above embodiment. Figure 6 The computer-readable storage medium shown is a CD 20 on which a computer program (ie, a program product) is stored. When the computer program is run by a processor, the method provided in any of the aforementioned embodiments is executed.

[0080] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0081] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the method provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

[0082] It should be noted that:

[0083] The term "module" is not intended to be limited to a specific physical form. Depending on the specific application, a module can be implemented as hardware, firmware, software, and / or a combination thereof. In addition, different modules can share common components or even be implemented by the same components. There may or may not be clear boundaries between different modules.

[0084] The algorithm and display provided herein are not inherently related to any specific computer, virtual device or other equipment. Various general devices can also be used together with examples based on this. According to the above description, it is obvious to construct the structure required for this type of device. In addition, the application is not directed to any specific programming language yet. It should be understood that various programming languages ​​can be utilized to realize the content of the application described herein, and the description made to specific languages ​​above is for the purpose of disclosing the best mode of implementation of the application.

[0085] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0086] The above-mentioned embodiments only express the implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for augmenting conversation data, characterized in that: include: Get the dialog template prompt information; Calling a pre-trained language model whose parameter scale is greater than a parameter amount threshold, and applying the pre-trained language model to independently perform dialogue data augmentation processing based on the dialogue template prompt information; The step of obtaining the dialog template prompt information includes: Obtaining a starting sentence corresponding to a preset target scene and introduced by a first preset word; Generate dialogue template prompt information corresponding to the target scene based on the description information of the current target scene, the starting sentence introduced by the first preset word, and the second preset word used to prompt the pre-trained language model to continue writing; The step of obtaining a starting sentence corresponding to a preset target scene and introduced by a first preset word includes: Extracting a dialogue start sentence from an open source dialogue dataset corresponding to the target scenario and / or a shared text for the target scenario; Based on the first preset word and the dialogue initiating sentence, an initiating sentence introduced by the first preset word is generated.

2. The method for augmenting conversation data according to claim 1, characterized in that: Before the pre-trained language model is applied to independently perform dialog data augmentation processing based on the dialog template prompt information, the method further includes: The pre-trained language model is fine-tuned using dialogue samples whose number is less than a sample threshold.

3. The method for augmenting conversation data according to claim 1, characterized in that: The applying the pre-trained language model to independently perform dialogue data augmentation processing based on the dialogue template prompt information includes: The dialogue template prompt information is written into a preset dialogue template, so that the pre-trained language model continues to write dialogue data in the dialogue template starting with the second preset word to complete the augmentation processing of the dialogue data.

4. The method for augmenting conversation data according to claim 2, characterized in that: The target scenarios include: emotional support dialogue scenarios; Correspondingly, fine-tuning the pre-trained language model using dialogue samples whose number is less than the sample threshold includes: In the emotion support dialogue dataset ESConv under the framework of the emotion support task ESC, select dialogue samples whose number is less than the sample threshold; The pre-trained language model is fine-tuned using the conversation sample.

5. The method for augmenting conversation data according to claim 1, characterized in that: The target scenarios include: emotional support dialogue scenarios; Correspondingly, the open source dialogue dataset includes: the open source empathetic dialogue dataset EmpatheticDialogues; the shared text includes: mental health text content in the public social platform Reddit.

6. The method for augmenting conversation data according to any one of claims 1 to 5, characterized in that: The pre-trained language model includes: GPT-J.

7. An electronic device, characterized in that: The device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for augmenting conversation data as claimed in any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the conversation data augmentation method as described in any one of claims 1-6.

Images