Language model training method, dialogue reply generating method and device and medium

Abstract

The invention discloses a language model training method. The method comprises the steps of obtaining initial training data and a pre-training language model; wherein the initial training data comprises first training data and second training data, the first training data comprises an instruction training text and a corresponding reply training text, the second training data comprises a first training text and a plurality of associated training texts, and the plurality of associated training texts are associated with the first training text; and training a pre-training language model based on the initial training data. The language model has the ability of thinking association through training, so that the language model obtained through training can output more meaningful reply content according to dialogue content.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, in particular to the field of natural language processing and intelligent dialogue technology, and in particular to a language model training method, a language model-based dialogue response generation method, device, electronic device and storage medium. Background Art

[0002] Artificial intelligence technology is a branch of computer science. It is a technology that achieves intelligence through computers and algorithms. Through continuous learning and evolution, it makes computers more and more "intelligent" and able to perform complex operations such as dialogue, reasoning, and learning. Research in the field of artificial intelligence includes robots, language recognition, image recognition, natural language processing, and expert systems.

[0003] Most existing dialogue systems are based on generative models, such as Transformer, Seq2Seq, etc., which learn to generate language models close to humans by learning a large number of human dialogues. This type of model is trained by constructing question-answer sequences so that it can answer user input questions in a more natural and reasonable way. Although dialogue systems have made significant progress in automated processing and natural language understanding, there are still some problems. Current dialogue systems are often insufficient when dealing with complex problems that require multi-step thinking. Current dialogue systems can learn a large number of human language models, but they still lack an understanding of the rich emotions and complex psychological states of humans. In addition, current dialogue systems often fail to give meaningful responses when dealing with problems involving complex logic or requiring deep thinking. Summary of the invention

[0004] To this end, the present invention provides a language model training method, a language model-based dialogue response generation method, device, electronic device and storage medium, in an effort to solve or at least alleviate at least one of the above problems.

[0005] According to one aspect of the present invention, a method for training a language model is provided, comprising:

[0006] Acquire initial training data and a pre-trained language model; wherein the initial training data includes first training data and second training data, the first training data includes an instruction training text and a corresponding reply training text, and the second training data includes a first training text and a plurality of associated training texts, wherein the plurality of associated training texts are related to the first training text; train the pre-trained language model based on the initial training data.

[0007] Optionally, the initial training data further includes third training data, wherein the third training data includes a plurality of first question training texts and a plurality of first answer training texts, wherein the plurality of first question training texts and the plurality of first answer training texts are texts associated in multiple rounds of dialogue.

[0008] Optionally, the first training text and the multiple associated training texts are designed based on a knowledge graph.

[0009] Optionally, training the pre-trained language model based on the initial training data specifically includes: inputting the initial training data into the pre-trained language model and outputting a first reply text; adjusting parameters of the pre-trained language model based on a first loss; the first loss represents the deviation between the first reply text and a target text, and the target text is the reply training text and one of a plurality of associated training Chinese texts.

[0010] Optionally, the first loss is a cross entropy loss.

[0011] Optionally, the training method further includes: performing reinforcement learning on the trained pre-trained language model.

[0012] Optionally, the reinforcement learning includes: obtaining fourth training data, the fourth training data including a second question training text and a second answer training text, and a correlation score between the second answer training text and the second question training text; training the first reward model based on the fourth training data to obtain a second reward model; using the second reward model to score the output of the trained pre-trained language model; and fine-tuning the trained pre-trained language model according to the above scoring to improve the output score of the fine-tuned language model.

[0013] According to another aspect of the present invention, a method for generating dialogue responses based on a language model is provided, comprising: inputting a first dialogue text into the language model; the language model determines whether an external search is required and the external search keywords based on a prompt project; if an external search is required, an external search is performed based on the external search keywords to obtain a search result, and the language model outputs a first reply text based on the first dialogue text and the search result; wherein the language model is trained according to the above-mentioned training method.

[0014] Optionally, the conversation response generation method further includes: when external retrieval is not required, the language model outputs a second response text based on the first conversation text.

[0015] According to another aspect of the present invention, a language model training device is provided, the training device comprising: an acquisition module, used to acquire initial training data and a pre-trained language model; wherein the initial training data comprises first training data and second training data, the first training data comprises an instruction training text and a corresponding reply training text, the second training data comprises a first training text and a plurality of associated training texts, the plurality of associated training texts are related to the first training text; and a training module, used to train the pre-trained language model based on the initial training data.

[0016] According to another aspect of the present invention, a device for generating dialogue responses based on a language model is provided, comprising: an input module for inputting a first dialogue text into the language model; a judgment module, wherein the language model determines whether an external search is required and the external search keywords based on a prompt project; an output module, wherein if an external search is required, an external search is performed based on the external search keywords to obtain a search result, and the language model outputs a first reply text based on the first dialogue text and the search result; wherein the language model is trained using the above-mentioned training method.

[0017] According to another aspect of the present invention, there is provided an electronic device, comprising: at least one processor; and a memory storing program instructions, wherein the program instructions are configured to be suitable for execution by the at least one processor, and the program instructions include instructions for executing the above-mentioned training method or the above-mentioned dialogue response generation method.

[0018] According to another aspect of the present invention, a readable storage medium storing program instructions is provided, and when the program instructions are read and executed by a computing device, the computing device executes the above-mentioned training method or the above-mentioned dialogue response generation method.

[0019] The training method of the language model of the embodiment of the present invention includes: obtaining initial training data and a pre-trained language model; wherein the initial training data includes first training data and second training data, the first training data includes an instruction training text and a corresponding reply training text, and the second training data includes a first training text and multiple associated training texts, and the multiple associated training texts are related to the first training text; and the pre-trained language model is trained based on the initial training data. Through training, the language model is equipped with the ability of thinking association, so that the trained language model can output more meaningful reply content according to the content of the conversation. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments recorded in the present application. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings. The drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the drawings, the same reference numerals generally represent the same components or steps.

[0021] Figure 1 is a schematic block diagram of an electronic device according to an embodiment of the present invention;

[0022] Figure 2 is a schematic flow chart of a method for training a language model according to an embodiment of the present invention;

[0023] Figure 3 is a schematic diagram of the correlation between multiple topics constructed according to an embodiment of the present invention;

[0024] Figure 4 is a schematic flow chart of a method for generating a dialogue response based on a language model according to an embodiment of the present invention;

[0025] Figure 5 is a schematic block diagram of a language model training device according to an embodiment of the present invention;

[0026] Figure 6 The invention relates to a device for generating dialogue responses based on a language model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0028] The embodiment of the present application provides a training method based on a language model, a method for generating a dialogue response, a device, an electronic device, and a storage medium. The training method includes obtaining initial training data and a pre-trained language model; wherein the initial training data includes first training data and second training data, the first training data includes an instruction training text and a corresponding reply training text, and the second training data includes a first training text and a plurality of associated training texts, and the plurality of associated training texts are related to the first training text; the pre-trained language model is trained based on the initial training data. This solves the defect that the current dialogue system cannot handle complex problems.

[0029] The language model-based training method and dialogue response generation method provided in the embodiments of the present application can be applied to electronic devices with computing and processing capabilities, which may be mobile phones, tablet computers, desktop computers, laptop computers, servers, vehicle-mounted devices, netbooks, as well as augmented reality (AR) devices, virtual reality (VR) devices, smart speakers, artificial intelligence robots, etc. The embodiments of the present application do not impose any special restrictions on the specific form of the electronic device.

[0030] For ease of understanding, the relevant terms involved in this application are first explained.

[0031] 1. Pre-training and fine-tuning

[0032] Pre-training refers to the process of training a model on large-scale unlabeled data. In Natural Language Processing (NLP), pre-training is the initial stage of language model learning. During pre-training, the model is exposed to a large amount of unlabeled text data, such as books, articles, and websites. Train a language model on a large amount of unlabeled text data. For example, a language model like GPT-3 is pre-trained on a dataset containing millions of books, articles, and websites. The goal of pre-training is to capture the underlying patterns, structures, and semantic knowledge present in the text corpus.

[0033] Fine-tuning refers to the supervised training of a model using a small amount of labeled data based on a pre-trained model. Through fine-tuning, the pre-trained model can adapt to the data and objectives of a specific task, improving the performance of the model on that task. In natural language processing, fine-tuning is the process of further training a large language model (LLM) on a specific task or domain. Specifically, fine-tuning can be seen as using a pre-trained LLM as the initial model and then training it on a labeled dataset for a specific task or domain. Fine-tuning can improve the performance of the LLM on a specific task or domain by adjusting the weights of the model to better fit the data.

[0034] Pre-training and fine-tuning techniques are widely used in the field of NLP, covering tasks such as text classification, named entity recognition, sentiment analysis, and machine translation. Through pre-training and fine-tuning, the model can learn general semantic representations from large-scale unlabeled data, allowing the model to better understand and process natural language.

[0035] Taking text classification as an example, pre-trained models such as BERT can learn rich semantic representations by pre-training on large-scale unlabeled data. Then, by fine-tuning on labeled text classification data, the model can adapt to the data and objectives of a specific classification task and improve the accuracy of classification.

[0036] Pre-training focuses on unsupervised learning from large-scale unlabeled data to acquire general language understanding. Fine-tuning builds on pre-training and uses task-specific labeled data to adapt the model to a specific task, thereby achieving specialized performance.

[0037] 2. Pre-trained language model

[0038] A pre-trained language model is a language model obtained through pre-training. The pre-trained language model adopts a two-stage learning method. First, the model is trained on a large corpus to enable the model to learn universal language representation, and then the pre-trained model is fine-tuned according to different downstream tasks.

[0039] 3. Reward Model

[0040] The reward model is derived from the reward function in reinforcement learning. It can describe the current state with a score to show how much value this state generates. The reward model in the fine-tuning of the large language model calculates a score for the input question and answer. The higher the match between the input answer and the question, the higher the score output by the reward model.

[0041] 4. Large Model

[0042] In the field of artificial intelligence technology, a large model usually refers to a deep neural network model with a large number of parameters (for example, hundreds of millions of scale). Among them, a large language model (LLM) is a deep learning model that is trained with a large amount of text data and can process natural language (for example, generate natural language text or understand the meaning of language text, etc.). The large language model can process not only text, but also images and sounds. The training text data used in the training of the large language model usually corresponds to at least one natural language task. Therefore, the large language model obtained by training has good natural language processing capabilities and can realize the generation of natural language text or the understanding of language text. The large language model has a wide range of application scenarios, including but not limited to chatbots, language translation, content creation, etc. This application does not impose any restrictions on the specific network structure of the large language model used.

[0043] 5. Prompt Engineering

[0044] In the context of artificial intelligence, a prompt is an initial input or prompt given to a model to guide the model to generate a specific output. The prompt can be a question, a text description, or even a text description with a bunch of parameters. The AI ​​model will generate the corresponding text or image based on the information provided by the prompt.

[0045] Prompt engineering is an artificial intelligence (AI) technology that improves AI performance by designing and improving AI prompts. The goal of prompt engineering is to create highly effective and controllable AI systems that can perform specific tasks accurately and reliably. Prompt engineering is used to develop and optimize prompts to help users effectively use language models for various application scenarios and research fields.

[0046] 6. Reinforcement Learning

[0047] Reinforcement learning is mainly based on environmental feedback. Through continuous interaction with the environment and trial and error, it ultimately achieves a specific goal or maximizes the overall action benefits. In reinforcement learning, the model acts on the current state through an action, so that the model has a certain probability of transferring from the current state to another state. When transferring to another state (or the state does not change), the environment will feedback a reward to the machine. The size of the reward is determined by the potential reward function.

[0048] Reinforcement learning is a goal-oriented learning method through interaction, which aims to find the optimal policy for continuous time series.

[0049] Figure 1FIG. 1 is a schematic diagram showing the structure of an electronic device 100 according to an embodiment of the present invention. The electronic device can be used to implement the methods of various embodiments of the present invention. Figure 1 As shown, in a basic configuration 102, the electronic device 100 typically includes a memory 106 and one or more processors 104. A memory bus 108 may be used for communication between the processor 104 and the memory 106.

[0050] Depending on the desired configuration, the processor 104 may be any type of processor, including but not limited to: a microprocessor, a microcontroller, a digital signal processor (DSP), or any combination thereof. The processor 104 may include one or more levels of cache, such as a primary cache 110 and a secondary cache 112, a processor core 114, and registers 116. An example processor core 114 may include an arithmetic logic unit (ALU), a floating point unit (FPU), a digital signal processing core, or any combination thereof. In some implementations, the memory controller 118 may be a separate component used with the processor 104; in some other implementations, the memory controller 118 may be an internal component of the processor 104.

[0051] Depending on the desired configuration, the memory 106 can be any type of memory, including but not limited to: volatile memory (such as RAM, etc.), non-volatile memory (such as ROM, flash memory, etc.) or any combination thereof. The physical memory in the electronic device generally refers to the volatile memory RAM, and the data in the disk needs to be loaded into the physical memory before it can be read by the processor 104. The memory 106 can store an operating system 120 and one or more applications 122. In some embodiments, the application 122 can be arranged to execute instructions on the operating system by one or more processors 104. The operating system 120 can be, for example, Linux, Windows, Android, Hongmeng, Apple iOS, etc., which includes program instructions for processing basic system services and performing hardware-dependent tasks. The application 122 includes program instructions for implementing various user-desired functions. The application 122 can be, for example, a browser, game software, business software, office software, instant messaging software, software development tools (such as integrated development environment IDE, compiler, etc.), etc., but is not limited thereto. When the application 122 is installed in the electronic device 100, a driver module can be added to the operating system 120 if necessary.

[0052] When the electronic device 100 starts running, the processor 104 reads and executes the program instructions of the operating system 120 from the memory 106. The application 122 runs on the operating system 120 and uses the interface provided by the operating system 120 and the underlying hardware to implement various functions desired by the user. When the user starts the application 122, the processor 104 reads and executes the program instructions of the application 122 from the memory 106.

[0053] The electronic device 100 may also include an interface bus 140 that facilitates communication from various interface devices (e.g., output devices 142, peripheral interfaces 144, and communication devices 146) to the basic configuration 102 via the bus / interface controller 130. Example output devices 142 include a graphics processing unit 148 and an audio processing unit 150. They can be configured to facilitate communication with various external devices such as a display or speakers via one or more A / V ports 152. Example peripheral interfaces 144 may include a serial interface controller 154 and a parallel interface controller 156, which may be configured to facilitate communication with external devices such as input devices (e.g., keyboards, mice, voice input devices, touch input devices, handwriting input devices, etc.) or other peripherals (e.g., printers, scanners, etc.) via one or more I / O ports 158. Example communication devices 146 may include a network controller 160, which may be configured to facilitate communication with one or more other electronic devices via a network communication link via one or more communication ports 164.

[0054] A network communication link can be an example of a communication medium. Communication media can generally be embodied as computer-readable instructions, data structures, program modules in a modulated data signal such as a carrier wave or other transmission mechanism, and can include any information delivery medium. As a non-limiting example, communication media can include wired media such as a wired network or a dedicated line network, and various wireless media such as sound, radio frequency (RF), microwave, infrared (IR) or other wireless media. The term computer-readable medium used herein can include both storage media and communication media.

[0055] In the electronic device 100 according to the present invention, the application 122 includes instructions for executing the language model-based training method and the dialogue response generation method of the present invention, which can instruct the processor 104 to execute the language model-based training method and the dialogue response generation method of the present invention.

[0056] It is to be understood that the structure shown in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include: Figure 1More or fewer components are shown, or some components are combined, or some components are separated, or there are different component arrangements. The components shown can be implemented in hardware, software, firmware, or any combination of the three.

[0057] Below, based on the above-mentioned hardware involved in the embodiments of the present application, other embodiments of the present application will be described in detail in conjunction with the accompanying drawings.

[0058] like Figure 2 As shown, an embodiment of the present invention provides a method for training a language model, comprising:

[0059] S210: Acquire initial training data and a pre-trained language model; wherein the initial training data includes first training data and second training data, the first training data includes an instruction training text and a corresponding reply training text, and the second training data includes a first training text and a plurality of associated training texts, wherein the plurality of associated training texts are associated with the first training text;

[0060] S220: Training the pre-trained language model based on the initial training data.

[0061] Because the pre-trained language model is trained using instruction data and associated data, the pre-trained language model after training has the ability of thought association, so that the trained language model can output more meaningful responses based on the content of the conversation.

[0062] Specifically, the instruction training text and the corresponding reply training text can guide the language model to learn how to think and associate; for example, the instruction training text can be "The user just mentioned that his mobile phone is broken, what topic should you associate with it?", and the corresponding reply training text can be "First I should comfort him, and then I need to introduce him to the latest mobile phone configuration and recommendation". The second training data is to enable the trained pre-trained language model to have more accurate multi-line thinking. For example, the first training text can be "It will rain tomorrow", and the multiple related training texts can be "My quilt has not been collected yet", "There may be traffic jams on the road today", so that the trained pre-trained language model can associate multiple related training texts through the first training text. Specifically, the first training text and the multiple related training texts are designed based on the knowledge graph. The correlation between the multiple related training texts is different, and the status in different scenarios is also different. Specifically, the multiple related training texts are marked, and the specific mark can be the correlation with the first training text. In this way, the language model finally trained will associate multiple related replies when outputting replies, and these related replies will present a network structure. The language model has the function of spreading association, and can output replies with higher correlation, thereby improving the quality of the replies.

[0063] Specifically, the pre-trained language model is obtained by pre-training the GPT large prediction model that only includes a decoder. Specifically, a large amount of unlabeled text data (including open domain dialogues, public corpora, etc.) and text data with emotion labels are used as training data in the pre-training stage to train the GPT large language model, so that the trained GPT large language model has the basic function of understanding and generating natural language. Specifically, the training data in the pre-training stage here is a large amount of unlabeled text data crawled from the Internet, which is obtained after cleaning, deduplication, and other operations. For example, text data with emotion labels can be [sad] "I broke up recently", [angry] "That scumbag deliberately angered me!". In this way, the reply text output by the trained GPT large language model has emotional labels, which improves the richness of the reply text.

[0064] In the embodiment of the present invention, in order to enable the trained pre-trained language model to have the ability to understand the context, the initial training data also includes third training data, and the third training data includes multiple first question training texts and multiple first answer training texts, and the multiple first question training texts and multiple first answer training texts are texts associated in multiple rounds of dialogue. The third training data is data related to the dialogue task, such as several questions and answers associated in multiple rounds of dialogue.

[0065] In the embodiment of the present invention, step S220 specifically includes:

[0066] S2201: Inputting initial training data into the pre-trained language model and outputting a first reply text;

[0067] S2203: Adjust parameters of the pre-trained language model based on the first loss; the first loss represents the deviation between the first reply text and the target text, and the target text is the reply training text and one of multiple associated training Chinese texts.

[0068] Specifically, the pre-trained language model may be first trained based on one of the first training data and the second training data to obtain the first pre-trained language model, and then the first pre-trained language model may be trained based on the other of the first training data and the second training data to obtain the final language model. When the initial training data also includes the third training data, one of the three training data is selected for training first, and then one of the remaining two training data is selected, and finally the remaining one training data is used for training.

[0069] Specifically, the first loss is the cross entropy loss.

[0070] In an embodiment of the present invention, the training method further includes performing reinforcement learning on the pre-trained language model after training. Specifically, the reinforcement learning includes:

[0071] S2301: Acquire fourth training data; wherein the fourth training data includes a second question training text and a second answer training text, and a correlation score between the second answer training text and the second question training text;

[0072] S2303: Training the first reward model based on the fourth training data to obtain a second reward model;

[0073] S2305: Using a second reward model to score the output of the trained pre-trained language model;

[0074] S2307: Fine-tune the trained pre-trained language model according to the above-mentioned scoring to improve the output score of the fine-tuned language model.

[0075] Specifically, the correlation score between the second answer training text and the second question training text may be manually marked. Figure 3 The figure shows the correlation between multiple topics. The arrows are marked with correlation scores between two topics. For example, the topics related to topic 1 are topic 2, topic 3, topic 4, and topic 5. The correlations between the above four topics and topic 1 are 0.5, 0.1, 0.9, and 0.2 respectively. The topic related to topic 2 is topic 3, and the correlation between topic 3 and topic 2 is 0.3. Specifically, Figure 3 The multiple topics and the correlation between them can be manually labeled and used as training data to train the first reward model. Specifically, the first reward model and the second reward model can be large language models. The second reward model can predict human preferences. After inputting question text and answer text, the second reward model can output a score, which is used to evaluate the score of this set of questions and answers (corresponding to the degree of correlation). The purpose of fine-tuning the pre-trained language model after training with the second reward model is to enable the text output by the fine-tuned language model to get a higher score, and to improve the correlation between the reply text output by the language model and the input question text.

[0076] A method for generating a dialogue response based on a language model according to an embodiment of the present invention, such as Figure 4 As shown, including:

[0077] S310: inputting the first dialogue text into the language model;

[0078] S320: The language model determines whether external search is required and external search keywords according to the prompt project;

[0079] S330: If external retrieval is required, an external retrieval is performed based on external retrieval keywords to obtain retrieval results, and the language model outputs a first reply text based on the first dialogue text and the retrieval results, wherein the language model is trained according to the above-mentioned training method.

[0080] The following describes the specific reasoning process of the language model:

[0081] After receiving the first dialogue text, the language model will parse the first dialogue text, including sentiment analysis and content understanding, and then start the multi-line thinking process, including: stimulating related associated thinking chains, that is, developing associations based on the first dialogue text and giving related content; and finding clues that may interrupt the dialogue, that is, finding the answer that can best answer the first dialogue text and meet the actual needs of the user. During the dialogue process, the language model will also start the patch search as needed, obtain the necessary information from external information sources, and then feed the obtained information back to the language model and incorporate it into the multi-line thinking process. Finally, the language model combines the above reasoning process to output the final dialogue reply text, that is, the first reply text.

[0082] The language model of the embodiment of the present invention is designed based on associative thinking, and can expand more associated information, which increases the possibility of reply texts, and can be flexibly switched, which is closer to human thinking. In addition, the language model here can retrieve and organize external information while thinking in multiple lines, making the conversation content richer and better understanding the needs of users.

[0083] Specifically, the dialogue response generation method further includes:

[0084] S340: When external retrieval is not required, the language model outputs a second reply text based on the first dialogue text.

[0085] The application of the language model in the dialogue system in the embodiment of the present invention is described with a specific application example. Assume that a user wants to buy a new mobile phone on an e-commerce platform, but does not know much about the current mobile phone age, and needs a dialogue system including a language model to help recommend.

[0086] The user's initial input may be "I want to buy a new mobile phone, but I don't know which brand to buy." After receiving the input, the language model will generate multiple possible answers based on the user's input, such as asking the user whether he prefers Android or Apple, or asking the user what budget requirements he has for the mobile phone. The dialogue system then sorts these possible answers and selects the most suitable answer, such as "Do you prefer to use Android phones or Apple's iPhone?" and outputs it to the user. At this time, the user may answer, "I prefer Android." After receiving the user's reply, the dialogue system will use associative thinking to generate new associative clues based on the user's reply, such as the user may have special requirements for price, the user may value camera functions, or the user may pay attention to mobile phones with long battery life. At the same time, the dialogue system uses the language model to determine whether external search is needed. Assuming that it is determined that external search is needed at this time, the language model will also give search keywords. The dialogue system will search for external information based on the search keywords. For example, the search keywords are best-selling, well-rated, and Android phones. At this time, the dialogue system will use external search tools to search for Android phones with good sales and high reviews in the current market and pull relevant data.

[0087] The dialogue system generates a comprehensive answer based on the search result information and related clues. For example, the answer can be "I understand that you like Android phones. Currently, the S brand G products, H brand M products, and X brand M series are highly rated in the market. Do you have a particular price range or a particular function of the phone that you particularly value?" In response to the output of the dialogue system, the user may answer, "I hope the price is around 3,000, and the camera must be good enough." The dialogue system can perform external information retrieval again based on the user's new answer feedback to find phones with good camera reviews and a price of around 3,000. At the same time, it searches for the related information between price and camera in the local data, and then forms an answer, such as "Within your budget, I recommend that you consider the X brand model 10, which has a very good camera effect."

[0088] Figure 5 FIG. 4 is a schematic block diagram of a language model training device according to an embodiment of the present invention. Figure 3 As shown, the training device 200 includes an acquisition module 201 and a training module 202 .

[0089] Among them, the acquisition module 201 is used to obtain initial training data and a pre-trained language model; wherein the initial training data includes first training data and second training data, the first training data includes an instruction training text and a corresponding reply training text, and the second training data includes a first training text and multiple associated training texts, and the multiple associated training texts are related to the first training text; the training module 202 is used to train the pre-trained language model based on the initial training data.

[0090] Figure 6 FIG. 2 is a schematic block diagram of a language model-based dialogue response generation device according to an embodiment of the present invention. Figure 6 As shown, the dialogue response generating device 300 includes an input module 301, a judgment module 302 and an output module 303.

[0091] Among them, the input module 301 is used to input the first dialogue text into the language model; the judgment module 302, the language model determines whether external search is required and the external search keywords according to the prompt project; the output module 303 is used to perform external search according to the external search keywords to obtain search results if external search is required, and the language model outputs a first reply text according to the first dialogue text and the search results; wherein the language model is trained by the training method according to claims 1-6.

[0092] The training device 200 and the dialogue response generating device 300 of the embodiment of the present invention can be Figure 1 The electronic device 100 shown is implemented by running program instructions, and can respectively execute the various steps in the training method and the dialogue response generation method according to the embodiments of the present invention.

[0093] According to an exemplary embodiment of the present invention, an electronic device is provided. The electronic device may include at least one processor and at least one memory. The memory stores program codes for implementing the corresponding steps in the training method and the dialogue response generation method according to the embodiment of the present invention. The processor is used to run the program codes stored in the memory to execute the corresponding steps in the training method and the dialogue response generation method according to the embodiment of the present invention, and to implement the various modules in the training device and the dialogue response generation device according to the embodiment of the present invention. The electronic device may adopt the aforementioned Figure 1 The electronic devices in the illustrated embodiments and any other suitable electronic devices.

[0094] In one embodiment, when the program code is executed by a processor, the following steps are performed:

[0095] A language model training method, comprising:

[0096] Acquire initial training data and a pre-trained language model; wherein the initial training data includes first training data and second training data, the first training data includes an instruction training text and a corresponding reply training text, and the second training data includes the first training text and a plurality of associated training texts, wherein the plurality of associated training texts are associated with the first training text;

[0097] The pre-trained language model is trained based on the initial training data.

[0098] In addition, when the program code is executed by the processor, the following steps are also performed:

[0099] Optionally, the initial training data further includes third training data, wherein the third training data includes a plurality of first question training texts and a plurality of first answer training texts, wherein the plurality of first question training texts and the plurality of first answer training texts are texts associated in multiple rounds of dialogue.

[0100] Optionally, the first training text and the multiple associated training texts are designed based on a knowledge graph.

[0101] Optionally, training the pre-trained language model based on the initial training data specifically includes: inputting the initial training data into the pre-trained language model and outputting a first reply text; adjusting parameters of the pre-trained language model based on a first loss; the first loss represents the deviation between the first reply text and a target text, and the target text is the reply training text and one of a plurality of associated training Chinese texts.

[0102] Optionally, the first loss is a cross entropy loss.

[0103] Optionally, the training method further includes: performing reinforcement learning on the trained pre-trained language model.

[0104] Optionally, the reinforcement learning includes: obtaining fourth training data, the fourth training data including a second question training text and a second answer training text, and a correlation score between the second answer training text and the second question training text; training the first reward model based on the fourth training data to obtain a second reward model; using the second reward model to score the output of the trained pre-trained language model; and fine-tuning the trained pre-trained language model according to the above scoring to improve the output score of the fine-tuned language model.

[0105] In another embodiment, when the program code is executed by a processor, the following steps are performed:

[0106] A method for generating dialogue responses based on a language model, comprising: inputting a first dialogue text into the language model; the language model determining whether an external search is required and the external search keywords based on a prompt project; if an external search is required, performing an external search based on the external search keywords to obtain a search result, and the language model outputting a first response text based on the first dialogue text and the search result; wherein the language model is trained according to the above-mentioned training method.

[0107] In addition, according to another embodiment of the present invention, a computer-readable storage medium is provided, on which program instructions are stored, and when the program instructions are executed by a computer or a processor, they are used to execute the corresponding steps of the training method and the dialogue response generation method of the embodiment of the present invention, and are used to implement the corresponding modules in the training device and the dialogue response generation device according to the embodiment of the present invention. The storage medium may include, for example, a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0108] In one embodiment, when the computer program instructions are executed by a computer, they can implement the various functional modules of the training device 200 and the dialogue response generation device 300 according to the embodiments of the present invention, and / or can execute the training method and the dialogue response generation method according to the embodiments of the present invention.

[0109] In one embodiment, the computer program instructions, when executed by a computer, perform the following steps:

[0110] A language model training method, comprising:

[0111] Acquire initial training data and a pre-trained language model; wherein the initial training data includes first training data and second training data, the first training data includes an instruction training text and a corresponding reply training text, and the second training data includes a first training text and a plurality of associated training texts, wherein the plurality of associated training texts are related to the first training text; train the pre-trained language model based on the initial training data.

[0112] In addition, the computer program instructions, when executed by a computer, further perform the following steps:

[0113] Optionally, the initial training data further includes third training data, wherein the third training data includes a plurality of first question training texts and a plurality of first answer training texts, wherein the plurality of first question training texts and the plurality of first answer training texts are texts associated in multiple rounds of dialogue.

[0114] Optionally, the first training text and the multiple associated training texts are designed based on a knowledge graph.

[0115] Optionally, training the pre-trained language model based on the initial training data specifically includes: inputting the initial training data into the pre-trained language model and outputting a first reply text; adjusting parameters of the pre-trained language model based on a first loss; the first loss represents the deviation between the first reply text and a target text, and the target text is the reply training text and one of a plurality of associated training Chinese texts.

[0116] Optionally, the first loss is a cross entropy loss.

[0117] Optionally, the training method further includes: performing reinforcement learning on the trained pre-trained language model.

[0118] Optionally, the reinforcement learning includes: obtaining fourth training data, the fourth training data including a second question training text and a second answer training text, and a correlation score between the second answer training text and the second question training text; training the first reward model based on the fourth training data to obtain a second reward model; using the second reward model to score the output of the trained pre-trained language model; and fine-tuning the trained pre-trained language model according to the above scoring to improve the output score of the fine-tuned language model.

[0119] In another embodiment, the computer program instructions, when executed by a computer, perform the following steps:

[0120] A method for generating dialogue responses based on a language model, comprising: inputting a first dialogue text into the language model; the language model determining whether an external search is required and the external search keywords based on a prompt project; if an external search is required, performing an external search based on the external search keywords to obtain a search result, and the language model outputting a first response text based on the first dialogue text and the search result; wherein the language model is trained according to the above-mentioned training method.

[0121] The modules of the training device 200 and the dialogue response generating device 300 according to the embodiment of the present invention can be implemented by executing computer program instructions stored in a memory by a processor of an electronic device according to the embodiment of the present invention, or can be implemented when computer instructions stored in a computer-readable storage medium according to the embodiment of the present invention are executed by a computer.

[0122] According to the training method, the method and device for generating a dialogue response, the electronic device and the storage medium of the embodiment of the present invention, the training method of the language model includes: obtaining initial training data and a pre-trained language model; wherein the initial training data includes first training data and second training data, the first training data includes an instruction training text and a corresponding reply training text, the second training data includes a first training text and a plurality of associated training texts, and the plurality of associated training texts are related to the first training text; the pre-trained language model is trained based on the initial training data. Through training, the language model is equipped with the ability of thinking association, so that the trained language model can output more meaningful reply content according to the dialogue content.

[0123] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely exemplary and are not intended to limit the scope of the present invention thereto. Various changes and modifications may be made therein by one of ordinary skill in the art without departing from the scope and spirit of the present invention. All such changes and modifications are intended to be included within the scope of the present invention as required by the appended claims.

[0124] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0125] The various techniques described herein may be implemented in combination with hardware or software, or a combination thereof. Thus, the method, apparatus, and device of the present invention, or some aspects or portions of the method, apparatus, and device of the present invention may be in the form of program codes (i.e., instructions) embedded in a tangible medium, such as a removable hard disk, a USB flash drive, a floppy disk, a CD-ROM, or any other machine-readable storage medium, wherein when the program is loaded into a machine such as a computer and executed by the machine, the machine becomes a device for practicing the present invention.

[0126] In the case where the program code is executed on a programmable computer, the electronic device generally includes a processor, a storage medium readable by the processor (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. The memory is configured to store the program code; the processor is configured to execute the remote assistance method of the present invention according to the instructions in the program code stored in the memory.

[0127] By way of example and not limitation, readable media include readable storage media and communication media. Readable storage media stores information such as computer-readable instructions, data structures, program modules or other data. Communication media generally embody computer-readable instructions, data structures, program modules or other data in modulated data signals such as carrier waves or other transmission mechanisms, and include any information transmission medium. Any combination of the above is also included in the scope of readable media. In the specification of the present application, algorithms and displays are not inherently related to any specific computer, virtual system or other device. Various general systems can also be used with the examples of the present invention. According to the above description, it is obvious to construct the structure required for such systems. In addition, the present invention is not directed to any specific programming language. It should be understood that various programming languages ​​can be used to implement the software and algorithm-related content of the present invention described herein, and the description of the specific language above is only to disclose some specific embodiments of the present invention, but not to limit the use of the specific language.

[0128] In the specification of the application, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this specification.

[0129] In the several embodiments provided in this application, it should be understood that the disclosed devices, equipment and methods can be implemented in other ways. For example, the above-described device and equipment embodiments are only illustrative, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0130] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0131] Similarly, it should be understood that in order to streamline the present invention and help understand one or more of the various inventive aspects, in the description of the exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, the method of the present invention should not be interpreted as reflecting the following intention: the claimed invention requires more features than the features explicitly stated in each claim. More specifically, as reflected in the corresponding claims, the inventive point is that the corresponding technical problem can be solved with less than all the features of a single disclosed embodiment. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present invention.

[0132] It will be understood by those skilled in the art that, except for mutually exclusive features, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this specification may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0133] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims, any one of the claimed embodiments may be used in any combination.

[0134] The components in the various embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that computers, servers, microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some modules in the equipment and devices according to embodiments of the present invention. The present invention can also be implemented as a device program (e.g., computer program and computer program product) for executing part or all of the methods described herein. Such a program implementing the present invention can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0135] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets should not be constructed as a limitation to the claims. Nouns such as "comprise", "include", "have" do not exclude the presence of elements or steps not listed in the claims. The quantifier "one" or "a" preceding an element does not exclude the presence of multiple such elements. The present invention can be implemented by means of hardware including several different elements and by means of appropriately programmed computers. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order, and these words may be interpreted as names.

[0136] In addition, in the embodiments of the present application, the expression "and / or" includes any and all combinations of the associated listed words. For example, the expression "A and / or B" may include A, may include B, or may include both A and B. Words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way. In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more.

[0137] In the description of the embodiments of the present application, when a component is referred to as being "connected" or "accessed" to other components, it should be understood that: the component is not only directly connected to or accessed to other components, but also another component may exist between the component and the other components. On the other hand, when a component is referred to as being "directly connected" or "directly accessed" to other components, it should be understood that there is no component between them. When a component is referred to as being "movably connected" to other components, it means that the positional relationship between the component and the other components is changeable, such as the relationship between a slide rail and a slider, the slider can slide on the slide rail, and the slide rail and the track are movably connected. When a component is referred to as being "fixedly connected" to other components, it means that the positional relationship between the component and the other components after being assembled together is relatively fixed.

[0138] The above is only a specific embodiment of the present invention or an explanation of a specific embodiment. The protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Although the present application is described according to a limited number of embodiments, it is understood by the technicians in the technical field that other embodiments can be envisioned within the scope of the present application described thereby, thanks to the above description. In addition, it should be noted that the language used in this specification is mainly selected for the purpose of readability and teaching, rather than for explaining or limiting the subject matter of the present application. Therefore, without departing from the scope and spirit of the attached claims, many modifications and changes are obvious to ordinary technicians in the technical field. For the scope of the present application, the disclosure made in the present application is illustrative, not restrictive, and the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for training a language model, characterized in that: include: Acquire initial training data and a pre-trained language model; wherein the initial training data includes first training data and second training data, the first training data includes an instruction training text and a corresponding reply training text, and the second training data includes the first training text and a plurality of associated training texts, wherein the plurality of associated training texts are associated with the first training text; The pre-trained language model is trained based on the initial training data.

2. The training method according to claim 1, characterized in that: The initial training data also includes third training data, where the third training data includes a plurality of first question training texts and a plurality of first answer training texts, where the plurality of first question training texts and the plurality of first answer training texts are texts associated in multiple rounds of dialogue.

3. The training method according to claim 1, characterized in that: The first training text and the multiple associated training texts are designed based on a knowledge graph.

4. The training method according to claim 1, characterized in that: Training the pre-trained language model based on the initial training data specifically includes: Inputting the initial training data into the pre-trained language model and outputting a first reply text; The parameters of the pre-trained language model are adjusted based on a first loss; the first loss represents the deviation between the first reply text and the target text, and the target text is the reply training text and one of multiple associated training Chinese texts.

5. The training method according to claim 4, characterized in that: The first loss is the cross entropy loss.

6. The training method according to claim 1, characterized in that: Also includes: Perform reinforcement learning on the trained pre-trained language model.

7. The training method according to claim 6, characterized in that: The reinforcement learning includes: Acquire fourth training data, the fourth training data including a second question training text and a second answer training text, and a correlation score between the second answer training text and the second question training text; Training the first reward model based on the fourth training data to obtain a second reward model; Using a second reward model to score the output of the trained pre-trained language model; The trained pre-trained language model is fine-tuned according to the above-mentioned scoring to improve the output score of the fine-tuned language model.

8. A method for generating dialogue responses based on a language model, characterized in that: include: Inputting the first dialogue text into the language model; The language model determines whether external search is required and external search keywords according to the prompt project; If external search is required, external search is performed according to external search keywords to obtain search results, and the language model outputs a first reply text according to the first dialogue text and the search results; Wherein, the language model is trained according to the training method according to any one of claims 1 to 7.

9. The method for generating a dialogue response according to claim 8, characterized in that: Also includes: When external retrieval is not required, the language model outputs a second reply text based on the first dialogue text.

10. A language model training device, characterized in that: The training device comprises: An acquisition module, used to acquire initial training data and a pre-trained language model; wherein the initial training data includes first training data and second training data, the first training data includes an instruction training text and a corresponding reply training text, and the second training data includes the first training text and a plurality of associated training texts, wherein the plurality of associated training texts are associated with the first training text; A training module is used to train the pre-trained language model based on the initial training data.

11. A device for generating dialogue responses based on a language model, characterized in that: include: An input module, used for inputting the first dialogue text into the language model; A judgment module, wherein the language model determines whether an external search is required and the external search keywords according to the prompt project; An output module is used for, if an external search is required, performing an external search based on external search keywords to obtain search results, and the language model outputs a first reply text based on the first dialogue text and the search results; Wherein, the language model is trained according to the training method according to any one of claims 1 to 7.

12. An electronic device, characterized in that: include: at least one processor; as well as A memory storing program instructions, wherein the program instructions are configured to be suitable for execution by the at least one processor, and the program instructions include instructions for executing the training method described in any one of claims 1-7 or the dialogue response generation method described in any one of claims 8-9.

13. A readable storage medium storing program instructions, characterized in that: When the program instructions are read and executed by a computing device, the computing device executes the training method according to any one of claims 1 to 7 or the dialogue response generation method according to any one of claims 8 to 9.

Images