Multi-style modular intelligent dialogue system for low-resource environment

Abstract

The application relates to a multi-style modular intelligent dialogue system for a low-resource environment and belongs to the technical field of intelligent dialogue. The system comprises a front-end module and a back-end module. The front-end module comprises a user input unit, a model selection unit and a style selection unit, and the back-end module comprises an input encoder, a decoder, a generator and an adapter unit. A user determines a model and a style used by the system through the model selection unit and the style selection unit on a front-end interface, and then inputs a text through the input unit. The back-end receives the model, the style and the input text transmitted by the front-end, generates a reply and transmits the reply back to the front-end. The front-end accepts the reply transmitted by the back-end, adds the reply to a chat record and displays the chat record on the front-end. The application greatly reduces the occupation of storage space and memory space and greatly improves the switching speed of the multi-style dialogue system between different styles.

Description

TECHNICAL FIELD

[0001] The application relates to a multi-style intelligent dialogue system with low storage space and low memory environment, and belongs to the technical field of intelligent dialogue. BACKGROUND

[0002] Traditional dialogue systems are mainly based on keyword analysis and template matching technical means, and have poor flexibility and expandability, and depend on a large number of human resources for template construction and maintenance. In 2015, the recurrent neural network and Seq2Seq architecture showed excellent performance in the field of machine translation, and was soon applied to various text generation tasks. Open-domain dialogue based on this technology has made certain progress in flexibility and no longer relies on a fixed knowledge base, but still has problems such as slow training and reasoning speed, high repetition rate of replies, and is difficult to be put into actual business scenarios.

[0003] Since 2017, represented by Bert, GPT and the like, models based on transformer can achieve excellent performance on downstream tasks through pre-training on large-scale unlabeled corpus and short-time fine-tuning. Open-domain dialogue tasks are also based on pre-trained language models, and through the pre-training-fine-tuning method, dialogue models such as DialoGPT and BlenderBot have been developed, which are close to human performance.

[0004] Open-domain dialogue tasks do not have specific scenarios, and there are no artificial constraints except general indicators such as fluency and logic. However, from the perspective of application scenarios, there are a large number of needs for style dialogue, such as dialogue showing specific emotions and dialogue imitating the tone of specific characters. According to the pre-training-fine-tuning paradigm introduced above, whenever there is a demand for a certain style, a model needs to be fine-tuned again. Since the pre-training model based on the transformer structure is large in size (generally between hundreds of MB and hundreds of GB), when there is a demand for multiple styles, the storage space occupied by the model will grow rapidly, which puts a lot of pressure on the equipment for deploying dialogue models. If it is necessary to quickly switch between different styles, due to the slow loading speed of large-parameter models, different style models need to be pre-loaded into the memory, and the required memory space also exceeds the conditions of many deployment environments.

[0005] Adapter technology is a training method that can apply the model to the downstream task without changing the parameters of the pre-trained model itself. The main idea is to freeze the original model parameters (mainly the transformer layer with large parameters) and insert a small parameter projection layer between the frozen layers. In the training process, the forward propagation process will use all the parameters of the original model and the projection layer, so as to fully utilize the knowledge of the pre-trained model itself. And the backward propagation process only updates the parameters of the projection layer, which not only speeds up the training, but also does not change the original model. In this way, adapter can be inserted into the pre-trained model as a modular component to adapt to different downstream tasks. When there are multiple downstream tasks, only one pre-trained model needs to be saved locally, and multiple small-sized adapter components are needed. Not only does it greatly save storage space, but it can also switch adapters at any time, and only one model needs to be loaded in memory, which greatly improves the switching speed and saves memory usage.

[0006] However, adapter technology has almost no application in the direction of text generation. Multi-style dialogue is a scenario that adapter is applicable to. Through adapter technology, not only can the demand for storage / memory space be reduced, but also the style switching speed can be accelerated, improving user experience. SUMMARY

[0007] The purpose of the present application is to address the problems of using multiple models in existing multi-style dialogue systems, high storage space and memory space occupation, and slow style switching speed, and creatively proposes a modular adapter technology for a low-resource environment multi-style intelligent dialogue system, which can avoid the high storage / memory space occupation and slow style switching caused by multi-style.

[0008] The present application adopts the following technical solutions.

[0009] A multi-style modular intelligent dialogue system for a low-resource environment includes a front-end module and a back-end module. The front-end module includes a user input unit, a model selection unit, and a style selection unit. The back-end module includes an input encoder, a decoder, a generator, and an adapter unit.

[0010] The user input unit provides a graphical interface for users to input text.

[0011] The model selection unit is used for users to select a basic model for dialogue generation, which is a drop-down selection box that can provide two options: T5 and BlenderBot.

[0012] The style selection unit is used for users to select the emotional style of dialogue generation, which is a drop-down selection box that provides different types of emotional style options.

[0013] The encoder is used to convert the user input text into a vector representation, which is a multi-layer structure transformer model.

[0014] The decoder is used to convert the vector given by the encoder into an output vector representing the probability distribution of each word, which is a multi-layer structure transformer model.

[0015] The generator is used to convert the word probability distribution given by the decoder into specific words (which can be achieved by beam search method), and splice into reply text.

[0016] The adapter unit is a pluggable modular structure that can be inserted into the encoder and decoder units to change the emotional style of the final generated dialogue.

[0017] The connection relationship of each component of the system is as follows:

[0018] The output ends of the model selection unit and the style selection unit are connected to the input ends of the backend module; the backend module loads the corresponding encoder, decoder and adapter according to the information transmitted by the model selection unit and the style selection unit; the output end of the user input unit is connected to the input end of the encoder to generate a vector representation of the user input text; the output end of the encoder is connected to the input end of the decoder to generate a probability vector of each word of the reply text; the output end of the decoder is connected to the input end of the generator to generate the final reply text; the output end of the generator is connected to the input end of the front-end module to display the reply text to the graphical interface.

[0019] Further, the working method of the system of the present application comprises the following steps:

[0020] Step 1: The user determines the model and style used by the system through the model selection unit and the style selection unit on the front-end interface, and then inputs the text through the input unit.

[0021] Step 2: The backend receives the model, style and input text transmitted by the front end, generates a reply and transmits it back to the front end.

[0022] Specifically, it can include the following steps:

[0023] Step 2.1: The backend selects the corresponding model and adapter according to the parameters transmitted by the front end, and loads the adapter into the model.

[0024] Step 2.2: Encode the input text into a vector representation.

[0025] Specifically, it can include the following steps:

[0026] Step 2.2.1: The input text is divided into words or subwords using tokenizer tool in transformers library.

[0027] Step 2.2.2: Each word / subword is mapped to a fixed-length vector, and concatenated in the original order to form a word vector matrix.

[0028] Step 2.2.3: The word vector matrix is passed to the next module for further processing.

[0029] Step 2.3: The model encoder after loading the adapter encodes the word vector matrix to form an intermediate matrix.

[0030] Step 2.4: The model decoder after loading the adapter decodes the intermediate vector to generate an output matrix. The length of the matrix is the length of the sentence, and each position is a vector containing the probability distribution of all words.

[0031] Step 2.5: The generator selects the final output word for each position based on the probability distribution of each word contained in the output matrix through the beam search method, and concatenates it into a complete reply, which is returned to the front end.

[0032] Step 3: The front end accepts the reply passed in by the back end, adds it to the chat record, and displays it on the front end.

[0033] Preferably, the aforementioned model includes T5 and BlenderBot. Both are pre-trained language models based on large-scale unsupervised corpus, whose task is to predict the probability of the next word from the input sequence. This task is modeled as predicting the conditional probability of a word at each position in the sequence.

[0034] Specifically, given an input sequence of length n, the pre-trained model predicts the conditional probability of the next word in turn through autoregressive.

[0035] Preferably, the adapter mentioned in the foregoing steps uses the compactor architecture. Generally, a general adapter introduces one or more projection layers (including a dimension reduction matrix, a nonlinear connection, a dimension increase matrix, connected by a residual structure) in each transformer layer of the pre-trained model. Compacter is a lightweight modular adapter that replaces the linear dimension reduction / increase matrix with a parameterized hypercomplex multiplication layer (PHM layer). Unlike linear layers, PHM layers construct a weight matrix through two smaller matrices, thereby reducing the number of parameters. These matrices can be decomposed and shared among all adapter layers. Compared with traditional adapters, Compacter reduces the number of parameters of the adapter model and improves the efficiency of the model.

[0036] The training of the model can use a general supervised training method to train a separate adapter for each style. The training process includes data set construction, model building, training and optimization, and model evaluation.

[0037] Wherein:

[0038] Data set construction: The system training can use two stylized data sets: Empathetic Dialogues and Blended Skill Talk. Empathetic Dialogues is a dialogue data set containing dozens of emotional labels, and Blended Skill Talk is a dialogue data set containing multiple skills (specifically, including the environment, emotions, etc. of the dialogue).

[0039] Since a separate adapter needs to be trained for each style, each style in the two data sets needs to be extracted separately, processed into a unified format, and a separate sub-data set is constructed. Specifically, the input of a single training data is concatenated with the style text using a specified format before the original input, and the output is not changed.

[0040] Model building, training and optimization: A general supervised training method for text generation can be used to abstract the problem as mapping an input sequence to an output sequence. Model construction includes loading a pre-trained model (i.e. the above T5 or BlenderBot), inserting an adapter layer, and defining an optimizer and a loss function.

[0041] During training, the model generates an output sequence based on the input sequence and calculates the cross-entropy loss, and updates the model parameters through backpropagation.

[0042] In addition, in order to optimize the model performance, the hyperparameters of the model, such as the learning rate, batch size, iteration number, etc., need to be adjusted to further improve the performance of the model.

[0043] Model evaluation: Various evaluation methods can be used to comprehensively evaluate the performance of the model, such as n-gram similarity-based indicators (BLEU, METEOR, etc.), diversity-based indicators, model prediction accuracy-based indicators (perplexity, etc.), and accuracy-based indicators for evaluating the generation style.

[0044] Advantages

[0045] Compared with the prior art, the present application has the following advantages:

[0046] 1. The present application greatly reduces the occupation of storage space and memory space on the basis of providing similar performance. Specifically, the resources occupied by a single adapter are less than 1% of the basic model. Compared with the model volume, the size of the adapter can be ignored. As the number of tasks increases, the space occupied by the traditional method grows linearly, while the space occupied by the present application always maintains the size of one basic model.

[0047] 2. The present application greatly improves the speed of switching between different styles of multi-style dialogue systems on the basis of providing similar performance. Specifically, it takes about 1-5 seconds for modern machines used to deploy deep learning models to load the basic model used by the present application from the local. For the traditional method, a basic model needs to be loaded every time the style is switched; while in the present application, the same basic model can always be loaded, and the loading or switching of the style corresponding adapter is almost instantaneous (about 0.1 seconds). BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 The overall architecture of the system of the present application is shown in the figure;

[0049] Figure 2 The principle of the adapter technology used in the present application is shown in the figure;

[0050] Figure 3 The generation effect evaluation result of the system of the present application is shown in the figure;

[0051] Figure 4 The space occupation of the system of the present application is shown in the figure.

[0052] Figure 5 The front-end interface of the system of the present application is shown in the figure. DETAILED DESCRIPTION

[0053] The technical solutions of the present application will be further described in detail below with reference to the accompanying drawings.

[0054] As Figure 1 shown, a multi-style modular intelligent dialogue system for low-resource environments includes a front-end module and a back-end module. The front-end module includes a user input unit, a model selection unit, and a style selection unit. The back-end module includes an input encoder, a decoder, a generator, and an adapter unit.

[0055] The system works as follows:

[0056] Step 1: The user determines the model and style used by the system through the model selection unit and style selection unit on the front-end interface, and then inputs the text through the input unit.

[0057] Step 2: The back-end receives the generation parameters (including model, style) and input text from the front-end, generates a reply, and transmits it back to the front-end.

[0058] Specifically, step 2 includes the following steps:

[0059] Step 2.1: The back-end selects the corresponding model and adapter according to the parameters transmitted by the front-end, and loads the adapter into the model. As Figure 2 shown, the adapter is loaded by inserting an additional adapter layer after the multi-head attention layer and the feed forward layer in each transformer layer of the original model. For the training process, as Figure 2 shown on the right, the multi-head attention layer and the feed forward layer will be frozen, and only the parameters of the adapter layer will be trained. In this way, adapters corresponding to multiple different styles can be trained on the same model.

[0060] The specific mechanism of the adapter is to use a small parameter layer to simulate a large parameter layer in the original model, which performs dimension reduction, nonlinear transformation, and dimension increase on the output of the multi-head attention layer and the feed forward layer. By learning the target dataset through the additional adapter layer, the pre-trained model's original knowledge can be fully utilized, and it can also be adapted to downstream tasks.

[0061] Adapters have various architectures and implementations; this article uses the compactor architecture. A typical adapter introduces one or more projection layers (including a dimensionality reduction matrix, a non-linear connection, and a dimensionality increase matrix connected via a residual structure) into each transformer layer of the pre-trained model. The compactor is a lightweight, modular adapter that replaces the linear dimensionality reduction / incrementing matrices with a PHM (parameterized hypercomplex multiplication layer). Unlike linear layers, the PHM layer constructs a weight matrix using two smaller matrices, thus reducing the number of parameters. These matrices can be decomposed and shared across all adapter layers. Compared to traditional adapters, the compactor reduces the number of parameters in the adapter model, improving model efficiency. Its specific working principle is as follows:

[0062] A l (x)=U l (Gelu(D l (x))+x

[0063] Where x is the parameter of layer l in the original model after which an adapter needs to be added, and D l and U l These are the dimension reduction and dimension increase matrices, respectively. Previously, a non-linear transformation was introduced using the GeLU activation function, and the final sum of these matrices yields the residual structure. The parameter x undergoes dimension reduction, activation function, dimension increase, and residual connection to ultimately obtain the output A of the Adapter layer. l (x).

[0064] By training the adapter on a dataset corresponding to the style, and then loading it during the usage phase, the model can achieve stylized dialogue. The following datasets are used in this invention:

[0065] 1. The Empathetic Dialogues dataset is a publicly available conversational dataset for studying human-computer emotional interaction, jointly developed by researchers at Stanford University and Columbia University. This dataset primarily contains conversations of emotional interaction between humans and machines, aiming to help machine learning models learn to understand human interaction styles and emotional needs, thereby generating more natural, flexible, and human-like responses. The Empathetic Dialogues dataset includes dialogues from three different domains: family, health, and psychological counseling. Each domain contains 1000 conversations, each with 3 to 4 dialogue rounds, performed by a simulated character and a human-computer user role-playing. The simulated character is designed to help solve problems or provide support by answering questions posed by the user.

[0066] 2. Blended Skill Talk Dataset A dataset developed by a research team at Facebook for dialogue system research, designed to help machine learning models learn to extract information from multiple skill domains and combine them into satisfactory dialogue responses. The dataset contains a series of scenarios and tasks, each involving an interaction between two characters. The dialogue includes an open dialogue phase and a specific task phase. Each task phase requires participants to use multiple skill domains (e.g. sports, movies, books, etc.) for dialogue and extract information from multiple knowledge sources. The Blended Skill Talk dataset aims to simulate real-world conversations, aiming to improve the naturalness, diversity, and intelligence levels of dialogue systems. At the same time, the dataset also provides some additional challenges, such as switching between different skill domains, extracting information from multiple contexts, and generating more personalized and fluent responses. The dataset was collected using Amazon Mechanical Turk and has been widely used in dialogue system research and evaluation, such as fast response, high-quality response, and diversity.

[0067] Step 2.2: Encode the input text into a vector representation. Specifically, it includes the following steps:

[0068] Step 2.2.1: Use the tokenizer tool in the transformers library to divide the input text into words or subwords.

[0069] Step 2.2.2: Each word or subword is mapped to a fixed-length vector, and the vectors are concatenated in the original order to form a word vector matrix.

[0070] Step 2.2.3: Pass the word vector matrix to the next module for further processing.

[0071] Step 2.3: Load the adapter-based model encoder to encode the word vector matrix into an intermediate matrix.

[0072] Step 2.4: Load the adapter-based model decoder to decode the intermediate vector into an output matrix.

[0073] Step 2.5: The generator selects the final output word for each position based on the probability distribution of each word contained in the output matrix using the beam search method, and concatenates it into a complete response and returns it to the front end. The process of processing text and generating responses in the back end is as follows:

[0074] p n = model(w1, w2,..., w n-1 )

[0075] where p n is a vector of size V representing the probability distribution of the nth word in the output text, and V is the size of the vocabulary; w n represents the nth input word.

[0076] In the generation phase, the model employs the beam search method. The beam search method is a method for decoding the probability distribution of the model input to obtain the final text. Here, beam refers to a "beam", that is, a set of candidate decoding sequences. In the decoding process, according to the probability distribution of the current input, select the k most likely operations from all possible next operations to generate k new candidate decoding sequences, and sort them according to their scores (i.e. probabilities). Then, select the highest score from the k candidate sequences as the next output. Then, this output becomes the new input, and the above steps are repeated until the maximum output length is reached or all sequences are terminated.

[0077] Specifically, the beam search method includes the following three steps:

[0078] 1. Initialization: The initial state of the model and a starting symbol are input, all possible first-step outputs are calculated, and the top k are selected as the initial candidate decoding sequences.

[0079] 2. Expansion: For each candidate sequence, calculate all possible next-step outputs in the current state. Here, a greedy algorithm can be used to select the top k operations and add them to the end of each candidate sequence to generate k new candidate decoding sequences.

[0080] 3. Pruning: Evaluate all candidate decoding sequences and keep the top k as candidate sequences for the next round of processing. If any sequence reaches the end state, it will be removed from the candidate list.

[0081] This loop is iterated until the termination condition (such as reaching the maximum output length) is met, or all sequences have stopped, and the highest-scoring sequence is selected as the final text output. Through the beam search method, the optimal output sequence can be efficiently decoded given the model's output probability distribution.

[0082] Step 3: When the backend sends a reply, the frontend adds it to the chat record and displays it on the interface. The invention uses a local file to save the chat record, enabling real-time read and write during user conversation and providing the frontend with real-time display.

[0083] As Figure 3As shown, experiments were conducted on three base models—T5-base, BlenderBot-90M, and BlenderBot-400M—to evaluate the dialogue generation performance of the proposed method. In the figure, "single" represents the performance after training a single model on data of all styles; "avg" represents the average performance after training a separate model for each style; and "adapter" represents the method used in this invention, indicating the average performance after training a separate adapter module for each style using a single model.

[0084] Figure 3 The evaluation metrics used include:

[0085] 1. BLEU: Evaluates the similarity between the generated text and the target text; the higher the better.

[0086] 2. Diversity: Assess the diversity of the generated text; higher is better.

[0087] 3. METEOR: Evaluates the similarity between the generated text and the target text; the higher the better.

[0088] 4. Perplexity: Evaluates the uncertainty of the generated text; the lower the better.

[0089] 5. Accuracy: Evaluates the accuracy of the model in generating target sentiment; the higher the better.

[0090] like Figure 3 As shown, when using a single model + adapter, the present invention outperforms the method using a single model in all aspects, and performs very close to the method using multiple models (the difference is less than 2%).

[0091] like Figure 4 As shown, this invention evaluated storage and memory usage on the T5-base model. It can be seen that with four tasks, the resource usage of this invention is only about 25% of that of the traditional method. As the number of tasks increases, the resource usage of the traditional method using multiple models rises rapidly, while the method used in this invention remains almost unchanged.

[0092] according to Figure 3 and Figure 4 The evaluation results demonstrate that this invention achieves multi-style dialogue performance very close to traditional methods using only a fraction of the space required. Furthermore, as the number of tasks (styles) increases, the resource consumption of this invention remains almost constant, exhibiting scalability far superior to traditional methods.

[0093] The system of this invention mainly uses the streamlit library to build the front-end module, and the effect is as follows: Figure 5Streamlit is a library for building web applications in the field of data science, machine learning, etc. In addition, the present invention uses PyTorch and transformers library for reading, training, evaluation and deployment of deep learning models; uses adapter-transformers library for building, reading and loading adapters.

Claims

1. A multi-style modular intelligent dialogue system for low-resource environments, characterized in that, The front-end module and the back-end module are connected. The front-end module includes a user input unit, a model selection unit, and a style selection unit, and the back-end module includes an input encoder, a decoder, a generator, and an adapter unit. The user input unit provides a graphical interface for the user to input text. The model selection unit allows the user to select a base model for dialogue generation, and is a drop-down selection box that provides options including T5 and BlenderBot. The style selection unit allows the user to select an emotional style for dialogue generation, and is a drop-down selection box that provides different types of emotional style options. The encoder is a multi-layer transformer model that converts the user input text into a vector representation. The decoder is a multi-layer transformer model that converts the vector output by the encoder into an output vector representing the probability distribution of each word. The generator converts the word probability distribution output by the decoder into specific words and concatenates them into a reply text. The adapter unit is a pluggable modular structure that can be inserted into the encoder and decoder units to change the emotional style of the generated dialogue. The connection between the various components of the system is as follows: The output of the model selection unit and the style selection unit is connected to the input of the back-end module.

2. The multi-style modular intelligent dialog system for low-resource environment of claim 1, wherein, The back-end module loads the corresponding encoder, decoder, and adapter unit based on the information transmitted by the model selection unit and the style selection unit. The output of the user input unit is connected to the input of the encoder, which generates a vector representation of the user input text. The output of the encoder is connected to the input of the decoder, which generates a probability vector for each word in the reply text. The output of the decoder is connected to the input of the generator, which generates the final reply text. The output of the generator is connected to the input of the front-end module, which displays the reply text on the graphical interface. The working method of the system includes the following steps: Step 1: The user determines the model and style used by the system through the model selection unit and the style selection unit on the front-end interface, and then inputs the text through the input unit. Step 2: The back-end receives the model, style, and input text transmitted by the front-end, generates a reply, and transmits it back to the front-end. Step 2.1: The back-end selects the corresponding model and adapter based on the parameters transmitted by the front-end, and loads the adapter into the model.

3. The multi-style modular intelligent dialog system for low-resource environment of claim 2, wherein, Step 2.2: The input text is encoded into a vector representation. Step 2.3: The model encoder with the loaded adapter encodes the word vector matrix to form an intermediate matrix. Step 2.4: The model decoder with the loaded adapter decodes the intermediate vector to generate an output matrix. Step 2.5: The generator selects the final output word for each position based on the probability distribution of each word contained in the output matrix, and concatenates it into a complete reply, which is transmitted back to the front-end. Step 3: The front-end accepts the reply transmitted by the back-end, adds it to the chat record, and displays it on the front-end. Step 2.2 includes the following steps: Step 2.2.1: Use tokenizer tool in transformers library to split input text into words or subwords; Step 2.2.2: Each word / subword is mapped to a fixed-length vector, concatenated in the original order to form a word vector matrix; Step 2.2.3: Pass the word vector matrix to the next module for further processing.

4. The multi-style modular intelligent dialog system for low-resource environment of claim 2, wherein, The model includes T5 and BlenderBot, and the task is to predict the probability of the next word from the input sequence, which is modeled as the conditional probability of predicting the word at each position in the sequence; Given an input sequence of length n, the pre-trained model predicts the conditional probability of the next word in an autoregressive manner.

5. The multi-style modular intelligent dialog system for low-resource environment according to any one of claims 1-4, wherein, The adapter uses the compactor architecture.

6. The multi-style modular intelligent dialog system for low-resource environment of claim 2, wherein, The model training process includes dataset construction, model building, training and tuning, and model evaluation, which includes: Dataset construction: The system training uses two stylized datasets: Empathetic Dialogues and BlendedSkillTalk; Since an adapter needs to be trained for each style, each style in the two datasets is extracted separately and processed into a unified format to build separate sub-datasets; The input of a single training data is concatenated with the specified format before the original input, and the output is not changed; Model building, training and tuning: Use supervised training method, abstract the problem as mapping input sequence to output sequence; Model building includes pre-training model loading, adapter layer insertion, and optimizer and loss function definition; During training, the model generates the output sequence according to the input sequence and calculates the cross-entropy loss, and updates the model parameters through backpropagation; Model evaluation: Evaluate model performance using evaluation methods, including n-gram similarity-based indicators, diversity-based indicators, model prediction accuracy-based indicators, and accuracy-based indicators for evaluating generated style.

7. The multi-style modular intelligent dialog system for low resource environment of claim 2, wherein, In step 2.5, the process of processing text and generating replies in the backend is: p n = model(w1, w2,..., wn) = model(w1, w2,..., wn) n-1 ) where p n is a vector of size V representing the probability distribution of the n-th word in the output text, V is the size of the vocabulary; w n represents the n-th input word.

Images