Chat-type multi-round dialogue method, system, medium and device based on semantic gate

Abstract

The present invention provides a method, system, medium and device for a chat-type multi-round dialogue based on a semantic gate, comprising: step S1: collecting corresponding user corpus according to business needs and preprocessing to obtain preprocessed corpus; step S2: fine-tuning a pre-trained language model GPT2-Chinese using the preprocessed corpus to obtain a fine-tuned GPT2 generation model, so that the fine-tuned GPT2 generation model learns to switch from general natural language prediction to prediction generation that conforms to the language style of in-vehicle text; step S3: inputting text and using the fine-tuned GPT2 generation model to generate multiple replies, and sorting them from large to small according to confidence, and taking the top N replies; step S4: fine-tuning a Bert classification model using a preset database to obtain a fine-tuned Bert classification model; step S5: jointly screening replies with the fine-tuned Bert classification model to make the generated in-vehicle chat text fluent and conform to the purpose of user semantics.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence processing technology, and in particular, to a chat-type multi-round dialogue method, system, medium and device based on semantic gates, and more particularly, to a controllable method and system for natural language generation. Background Art

[0002] In the past three years, pre-trained language models have set off a wave of natural language research tasks. Language models are pre-trained on high-quality, large-scale datasets to learn and understand the representation of natural language. Researchers can directly use pre-trained language models to fine-tune downstream tasks, eliminating the tedious model training process and promoting the rapid development of various natural language processing technologies. Pre-trained language models mainly learn the contextual representation of words. According to different sequence prediction methods, they can be divided into autoencoder and autoregressive language models. Bert proposed by Google is a typical autoencoder representative. It uses Transformer to extract features and introduces Masked Language Model (MLM) and Next Sentence Prediction (NSP) pre-training objectives. It can obtain context-related bidirectional feature representations and is good at handling sentence or paragraph matching tasks. However, the inconsistency between the pre-training process and the generation process leads to poor results in generation tasks. Representatives of autoregressive language models include ELMo, GPT, and XLnet: Peters Matthew E et al. proposed that ELMo is the earliest pre-trained language model, which uses a bidirectional long short-term memory network (BiLSTM) to extract features serially. The model learns the deep contextual representation of words from left to right and from right to left in the order of text sequence decomposition, and obtains the bidirectional features of contextual information. However, ELMo is essentially the splicing of two unidirectional language models and cannot obtain contextual representation at the same time. In addition, the neural network LSTM cannot solve long-distance dependencies and has weak feature extraction capabilities. GPT is the first to use Transformer for feature extraction, which can quickly capture information in a longer range. It has been updated to the third generation. The language model auxiliary target is introduced in the fine-tuning stage of GPT1 to solve the catastrophic forgetting in the fine-tuning process. GPT2 is an improvement on GPT1, using training data with wider coverage and higher quality. It is believed that pre-training already contains a lot of information required for specific tasks. There is no fine-tuning process for specific models, and it has achieved good results in generating tasks. GPT3 uses more training data and higher-performance computing resources than GPT2 to improve the performance of the model. The XLnet model uses Transformer-XL to extract features. This model introduces Permuted Language Modeling (PLM) can learn various bidirectional context representations. The results show that XLNet performs better than BERT on 20 tasks and has been significantly improved.Through comprehensive analysis, although the XLnet language model is stronger than other pre-trained language models in learning context representation, the unidirectional autoregressive language model is more suitable for generation tasks. Considering the consistency of pre-training and fine-tuning in the generation task and the fact that the research on chat text is aimed at Chinese text generation, the present invention chooses to fine-tune the chat text generation task under the Chinese version of the pre-trained GPT2-Chinese. Although GPT3 has been released, its model requires higher computing resources and the experimental arrangement is difficult, so GPT2 is the best choice. Since the chat robot based on the generation model is built using a data-driven end-to-end method, it can respond to any question, so it has the highest flexibility, but the responses generated by the generation model often have problems such as inaccurate semantics and inconsistent context, so the accuracy is the lowest.

[0003] Patent document CN112131367A (application number: 202011018603.6) discloses a self-reviewing human-computer dialogue method, system and readable storage medium. The self-reviewing human-computer dialogue system consists of an input and output module, a dialogue reply generation module and a dialogue self-review module. The user can input any dialogue, generate multiple candidate replies through the generation model in the reply generation module, and then calculate the loss of the candidate reply relative to the historical dialogue through the MMI scoring model, and select the one with the smallest loss as the final reply. The generated reply is input into the dialogue self-review model, and the text features are first extracted using the pre-trained Bert model, and then the bidirectional LSTM model is used for classification to determine whether it belongs to a sensitive sentence. If it does not belong to a sensitive speech, the reply is output to the user; if it is judged to be a sensitive sentence, the reply in the pre-set dialogue template is output.

[0004] The present invention proposes a new method for generating multi-round small talk based on the Bert classification model. The method uses the GPT2 generation model for reasoning to obtain the generated top 5 replies, and then uses the Bert classification model to screen out the best reply or the reply that is most consistent with the previous semantics. Summary of the invention

[0005] In view of the defects in the prior art, the purpose of the present invention is to provide a method, system, medium and device for chatting multi-round dialogue based on semantic gates.

[0006] According to the present invention, a chat-type multi-round dialogue method based on semantic gates includes:

[0007] Step S1: Collect corresponding user corpus according to business needs and preprocess it to obtain preprocessed corpus;

[0008] Step S2: fine-tune the pre-trained language model GPT2-Chinese using the pre-processed corpus to obtain a fine-tuned GPT2 generation model, so that the fine-tuned GPT2 generation model learns to switch from general natural language prediction to prediction generation that conforms to the language style of the in-vehicle text;

[0009] Step S3: Generate multiple responses using the fine-tuned GPT2 generative model for the input text, sort them from high to low according to confidence, and take the top N responses;

[0010] Step S4: fine-tuning the Bert classification model using a preset database to obtain a fine-tuned Bert classification model;

[0011] Step S5: The fine-tuned Bert classification model is used to perform reply screening, so that the generated in-vehicle chat text is fluent and meets the user's semantic purpose.

[0012] Preferably, the step S2 adopts:

[0013] Step S2.1: obtaining a high-quality in-vehicle chat training set that meets preset conditions based on the preprocessed corpus;

[0014] Step S2.2: Use a high-quality in-vehicle chat training set that meets preset conditions to perform domain adaptation training on the pre-trained language model GPT2-Chinese to obtain a fine-tuned GPT2 generation model, so that the fine-tuned GPT2 generation model learns to switch from general natural language prediction to prediction generation that conforms to the in-vehicle text language style.

[0015] Preferably, the high-quality in-vehicle chat training set that meets preset conditions includes: test environment corpus and / or production environment corpus.

[0016] Preferably, the step S3 adopts: using a fine-tuned Bert classification model to perform semantic consistency analysis on the input text and N replies, and finally realizing personalized generation of in-vehicle text.

[0017] According to the present invention, a chat-type multi-round dialogue system based on semantic gates is provided, comprising:

[0018] Module M1: collects corresponding user corpus according to business needs and preprocesses it to obtain preprocessed corpus;

[0019] Module M2: Use the preprocessed corpus to fine-tune the pre-trained language model GPT2-Chinese to obtain a fine-tuned GPT2 generation model, so that the fine-tuned GPT2 generation model learns to switch from general natural language prediction to prediction generation that conforms to the language style of the in-vehicle text;

[0020] Module M3: Generate multiple replies using the fine-tuned GPT2 generative model for the input text, sort them from high to low according to confidence, and take the top N replies;

[0021] Module M4: fine-tune the Bert classification model using a preset database to obtain a fine-tuned Bert classification model;

[0022] Module M5: The fine-tuned Bert classification model is used to screen responses, so that the generated in-vehicle chat text is fluent and meets the user's semantic purpose.

[0023] Preferably, the module M2 adopts:

[0024] Module M2.1: Obtain high-quality in-vehicle chat training sets that meet preset conditions based on the preprocessed corpus;

[0025] Module M2.2: Use a high-quality in-vehicle chat training set that meets the preset conditions to perform domain adaptation training on the pre-trained language model GPT2-Chinese to obtain a fine-tuned GPT2 generation model, so that the fine-tuned GPT2 generation model can learn to switch from general natural language prediction to prediction generation that conforms to the in-vehicle text language style.

[0026] Preferably, the high-quality in-vehicle chat training set that meets preset conditions includes: test environment corpus and / or production environment corpus.

[0027] Preferably, the module M3 adopts: using a fine-tuned Bert classification model to perform semantic consistency analysis on the input text and N replies, and finally realizing personalized generation of in-vehicle text.

[0028] According to a computer-readable storage medium storing a computer program provided by the present invention, the steps of the above-mentioned method are implemented when the computer program is executed by a processor.

[0029] According to the present invention, a chat-type multi-round dialogue device based on semantic gates includes: a controller;

[0030] The controller includes the computer-readable storage medium storing a computer program, and when the computer program is executed by the processor, the steps of the above-mentioned chat-type multi-round dialogue method based on semantic gates are implemented; or, the controller includes the above-mentioned chat-type multi-round dialogue system based on semantic gates.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] 1. The present invention uses the fine-tuned Bert classification model to infer multiple replies generated by the gpt2 model respectively, and finally realizes the personalized generation of in-vehicle text, and further classifies and screens the replies generated by the gpt2 model; further, the accuracy of in-vehicle voice dialogue is improved.

[0033] 2. The present invention uses a fine-tuned pre-trained gpt2 model to make the reply more scenario-oriented and biased towards in-vehicle chat, improve the matching degree of in-vehicle voice dialogue, and enhance the user experience; BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0035] Figure 1 The figure is a flow chart of the chat-type multi-round dialogue method based on semantic gates. DETAILED DESCRIPTION

[0036] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0037] Example 1

[0038] The in-vehicle voice dialogue system includes task-based dialogue (navigation, weather, music, etc.), question-and-answer dialogue (user manual, etc.) and chat dialogue;

[0039] In the in-vehicle voice dialogue system, directly using the open source pre-trained model gpt2 will result in uncontrollable technical problems, which will affect the accuracy of the in-vehicle voice dialogue and user experience.

[0040] According to a semantic gate-based chat-type multi-round dialogue method provided by the present invention, Figure 1 As shown, including:

[0041] Step S1: Collect corresponding user corpus according to business needs and preprocess it to obtain preprocessed corpus;

[0042] Step S2: fine-tune the pre-trained language model GPT2-Chinese using the pre-processed corpus to obtain a fine-tuned GPT2 generation model, so that the fine-tuned GPT2 generation model learns to switch from general natural language prediction to prediction generation that conforms to the language style of the in-vehicle text;

[0043] Step S3: Generate multiple responses using the fine-tuned GPT2 generative model for the input text, sort them from high to low according to confidence, and take the top N responses;

[0044] Step S4: fine-tuning the Bert classification model using a preset database to obtain a fine-tuned Bert classification model;

[0045] Step S5: The fine-tuned Bert classification model is used to perform reply screening, so that the generated in-vehicle chat text is fluent and meets the user's semantic purpose.

[0046] In summary, in order to improve the matching degree of in-vehicle voice dialogue and improve the user experience, the input text is based on the GPT2 model to obtain multiple replies, and the confidence is sorted from large to small, and the top N replies are taken. However, because the confidence is unreliable, the present invention uses the fine-tuned BERT model to filter out the best reply based on multiple replies of the GPT2 model, so that the probability value of semantic matching between the reply and the above text is maximized. However, directly using the open source pre-trained model GPT2 will result in uncontrollable technical problems in the reply. Therefore, the GPT2 model is fine-tuned, and the fine-tuned pre-trained GPT2 model is used to make the reply more scenario-oriented and biased towards in-vehicle chat.

[0047] Specifically, the step S2 adopts:

[0048] Step S2.1: obtaining a high-quality in-vehicle chat training set that meets preset conditions based on the preprocessed corpus;

[0049] Step S2.2: Use a high-quality in-vehicle chat training set that meets preset conditions to perform domain adaptation training on the pre-trained language model GPT2-Chinese to obtain a fine-tuned GPT2 generation model, so that the fine-tuned GPT2 generation model learns to switch from general natural language prediction to prediction generation that conforms to the in-vehicle text language style.

[0050] Specifically, the high-quality in-vehicle chat training set that meets the preset conditions includes: test environment corpus and / or production environment corpus. The corpus is made more in line with the business scenario, for example: the corpus is more in line with the in-vehicle voice dialogue scenario. Specifically, the test environment corpus includes the test in-vehicle chat corpus obtained during the test phase before the vehicle leaves the factory; the production environment corpus includes the in-vehicle chat corpus obtained during the use phase after the vehicle leaves the factory.

[0051] Specifically, step S3 adopts: using the fine-tuned Bert classification model to perform semantic consistency analysis on the input text and N replies, and finally realizes the personalized generation of in-vehicle text. Specifically, for the fine-tuned Bert classification model, the input is the input and an output of the fine-tuned GPT2 model, and the output is the semantic consistency of the input and an output of the fine-tuned GPT2 model. When the input text matches a reply, the matching label is 1; when it does not match, the mismatching label is 0; more specifically, the fine-tuned Bert model is a binary classification.

[0052] For example, if the input text is "Hello", and the input text "Hello" is input into the fine-tuned GPT2 model, N replies will be generated, including: "Hello, master", "I haven't chatted with me for a long time, I miss you so much" and "Finally I've waited for you"; then the input text "Hello" and "Hello, master", "Hello" and "I haven't chatted with me for a long time, I miss you so much" and "Hello" and "Finally I've waited for you" are respectively input into the fine-tuned BERT model, and the fine-tuned BERT model is used to obtain the matching degree between the input text and the current reply. Match is represented by 1 and mismatch is represented by 0. The best reply can be screened according to the matching degree.

[0053] According to the present invention, a chat-type multi-round dialogue system based on semantic gates is provided, comprising:

[0054] Module M1: collects corresponding user corpus according to business needs and preprocesses it to obtain preprocessed corpus;

[0055] Module M2: Use the preprocessed corpus to fine-tune the pre-trained language model GPT2-Chinese to obtain a fine-tuned GPT2 generation model, so that the fine-tuned GPT2 generation model learns to switch from general natural language prediction to prediction generation that conforms to the language style of the in-vehicle text;

[0056] Module M3: Generate multiple replies using the fine-tuned GPT2 generative model for the input text, sort them from high to low according to confidence, and take the top N replies;

[0057] Module M4: fine-tune the Bert classification model using a preset database to obtain a fine-tuned Bert classification model;

[0058] Module M5: The fine-tuned Bert classification model is used to screen responses, so that the generated in-vehicle chat text is fluent and meets the user's semantic purpose.

[0059] In summary, in order to improve the matching degree of in-vehicle voice dialogue and improve the user experience, the input text is based on the GPT2 model to obtain multiple replies, and the confidence is sorted from large to small, and the top N replies are taken. However, because the confidence is unreliable, the present invention uses the fine-tuned BERT model to filter out the best reply based on multiple replies of the GPT2 model, so that the probability value of semantic matching between the reply and the above text is maximized. However, directly using the open source pre-trained model GPT2 will result in uncontrollable technical problems in the reply. Therefore, the GPT2 model is fine-tuned, and the fine-tuned pre-trained GPT2 model is used to make the reply more scenario-oriented and biased towards in-vehicle chat.

[0060] Specifically, the module M2 adopts:

[0061] Module M2.1: Obtain high-quality in-vehicle chat training sets that meet preset conditions based on the preprocessed corpus;

[0062] Module M2.2: Use a high-quality in-vehicle chat training set that meets the preset conditions to perform domain adaptation training on the pre-trained language model GPT2-Chinese to obtain a fine-tuned GPT2 generation model, so that the fine-tuned GPT2 generation model can learn to switch from general natural language prediction to prediction generation that conforms to the in-vehicle text language style.

[0063] Specifically, the high-quality in-vehicle chat training set that meets the preset conditions includes: test environment corpus and / or production environment corpus. The corpus is made more in line with the business scenario, for example: the corpus is more in line with the in-vehicle voice dialogue scenario. Specifically, the test environment corpus includes the test in-vehicle chat corpus obtained during the test phase before the vehicle leaves the factory; the production environment corpus includes the in-vehicle chat corpus obtained during the use phase after the vehicle leaves the factory.

[0064] Specifically, the module M3 adopts: using the fine-tuned Bert classification model to perform semantic consistency analysis on the input text and N replies, and finally realizes the personalized generation of in-vehicle text. Specifically, for the fine-tuned Bert classification model, the input is the input and an output of the fine-tuned GPT2 model, and the output is the semantic consistency of the input and an output of the fine-tuned GPT2 model. When the input text matches a reply, the matching label is 1; when it does not match, the mismatching label is 0; more specifically, the fine-tuned Bert model is a binary classification.

[0065] For example, if the input text is "Hello", and the input text "Hello" is input into the fine-tuned GPT2 model, N replies will be generated, including: "Hello, master", "I haven't chatted with me for a long time, I miss you so much" and "Finally I've waited for you"; then the input text "Hello" and "Hello, master", "Hello" and "I haven't chatted with me for a long time, I miss you so much" and "Hello" and "Finally I've waited for you" are respectively input into the fine-tuned BERT model, and the fine-tuned BERT model is used to obtain the matching degree between the input text and the current reply. Match is represented by 1 and mismatch is represented by 0. The best reply can be screened according to the matching degree.

[0066] According to a computer-readable storage medium storing a computer program provided by the present invention, the steps of the above-mentioned method are implemented when the computer program is executed by a processor.

[0067] According to the present invention, a chat-type multi-round dialogue device based on semantic gates includes: a controller;

[0068] The controller includes the computer-readable storage medium storing a computer program, and when the computer program is executed by the processor, the steps of the above-mentioned chat-type multi-round dialogue method based on semantic gates are implemented; or, the controller includes the above-mentioned chat-type multi-round dialogue system based on semantic gates.

[0069] Those skilled in the art know that, in addition to implementing the system, device and its various modules provided by the present invention in a purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers and embedded microcontrollers by logically programming the method steps. Therefore, the system, device and its various modules provided by the present invention can be considered as a hardware component, and the modules included therein for implementing various programs can also be considered as structures within the hardware component; the modules for implementing various functions can also be considered as both software programs for implementing the method and structures within the hardware component.

[0070] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A chat-type multi-round dialogue method based on semantic gates, characterized in that: include: Step S1: Collect corresponding user corpus according to business needs and preprocess it to obtain preprocessed corpus; Step S2: fine-tune the pre-trained language model GPT2-Chinese using the pre-processed corpus to obtain a fine-tuned GPT2 generation model, so that the fine-tuned GPT2 generation model learns to switch from general natural language prediction to prediction generation that conforms to the language style of the in-vehicle text; Step S3: Generate multiple responses using the fine-tuned GPT2 generative model for the input text, sort them from high to low according to confidence, and take the top N responses; Step S4: fine-tuning the Bert classification model using a preset database to obtain a fine-tuned Bert classification model; Step S5: The fine-tuned Bert classification model is used to perform reply screening, so that the generated in-vehicle chat text is smooth and meets the user's semantic purpose; The step S2 adopts: Step S2.1: obtaining a high-quality in-vehicle chat training set that meets preset conditions based on the preprocessed corpus; Step S2.2: Use a high-quality in-vehicle chat training set that meets the preset conditions to perform domain adaptation training on the pre-trained language model GPT2-Chinese to obtain a fine-tuned GPT2 generation model, so that the fine-tuned GPT2 generation model learns to switch from general natural language prediction to prediction generation that conforms to the in-vehicle text language style; The high-quality in-vehicle chat training set that meets the preset conditions includes: test environment corpus and / or production environment corpus; The step S3 includes: using the fine-tuned Bert classification model to perform semantic consistency analysis on the input text and N replies, and finally realizing personalized generation of in-vehicle text; for the fine-tuned Bert classification model, the input is the input and an output of the fine-tuned GPT2 model, and the output is the semantic consistency of the input and an output of the fine-tuned GPT2 model. When the input text matches a reply, the matching label is 1; when it does not match, the unmatched label is 0; the fine-tuned Bert model is a binary classification.

2. A chat-type multi-round dialogue system based on semantic gates, characterized in that: include: Module M1: collects corresponding user corpus according to business needs and preprocesses it to obtain preprocessed corpus; Module M2: Use the preprocessed corpus to fine-tune the pre-trained language model GPT2-Chinese to obtain a fine-tuned GPT2 generation model, so that the fine-tuned GPT2 generation model learns to switch from general natural language prediction to prediction generation that conforms to the language style of the in-vehicle text; Module M3: Generate multiple replies using the fine-tuned GPT2 generative model for the input text, sort them from high to low according to confidence, and take the top N replies; Module M4: fine-tune the Bert classification model using a preset database to obtain a fine-tuned Bert classification model; Module M5: Use the fine-tuned Bert classification model to perform reply screening, so that the generated in-car chat text is fluent and meets the user's semantic purpose; The module M2 adopts: Module M2.1: Obtain high-quality in-vehicle chat training sets that meet preset conditions based on the preprocessed corpus; Module M2.2: Use the high-quality in-vehicle chat training set that meets the preset conditions to perform domain adaptation training on the pre-trained language model GPT2-Chinese to obtain a fine-tuned GPT2 generative model, so that the fine-tuned GPT2 generative model learns to switch from general natural language prediction to prediction generation that conforms to the in-vehicle text language style; The high-quality in-vehicle chat training set that meets the preset conditions includes: test environment corpus and / or production environment corpus; The module M3 includes: using the fine-tuned Bert classification model to perform semantic consistency analysis on the input text and N replies, and finally realizing personalized generation of in-vehicle text; for the fine-tuned Bert classification model, the input is the input and an output of the fine-tuned GPT2 model, and the output is the semantic consistency of the input and an output of the fine-tuned GPT2 model. When the input text matches a reply, the matching label is 1; when it does not match, the unmatched label is 0; the fine-tuned Bert model is a binary classification.

3. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to claim 1 are implemented.

4. A chat-type multi-round dialogue device based on semantic gates, characterized in that: include: Controller; The controller comprises a computer-readable storage medium storing a computer program as described in claim 3, and when the computer program is executed by a processor, the steps of the chat-type multi-round dialogue method based on semantic gates as described in any one of claim 1 are implemented; or, the controller comprises a chat-type multi-round dialogue system based on semantic gates as described in any one of claim 2.

Images