A method and apparatus for encoding a dialogue utterance

Abstract

The application discloses a dialogue utterance encoding method and device, and relates to the technical field of artificial intelligence. A specific embodiment of the method comprises the following steps: constructing a data set, wherein the data set comprises positive samples and negative samples, the positive samples comprise dialogue utterances and their respective fields, and the negative samples comprise dialogue utterances and their non-respective fields; adjusting parameters of a pre-trained language model based on a contrast learning algorithm and by using the data set, thereby obtaining an encoder; and encoding a target dialogue utterance by using the encoder. The embodiment can solve the technical problem that there is no semantic corresponding relationship between fields and dialogue utterances.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and apparatus for encoding conversational utterances. Background Technology

[0002] Task-oriented dialogue systems aim to help users achieve specific goals and have a wide range of applications. For example, virtual assistants like Siri and Alexa can help users complete many daily tasks, such as ordering food and booking hotels. A typical task-oriented dialogue system consists of four key components: Natural Language Understanding (NLU), Dialogue State Tracking (DST), Dialogue Policy Learning (DPL), and Natural Language Generation (NLG). DST aims to track the user's intent in each round of dialogue and represent it as a list of triplets (domain, slot, slot value). Since DPL and NLG rely on the results of DST to select the next round of system action and generate the latest round of system response, accurately predicting the dialogue state is crucial for improving the overall performance of the dialogue system.

[0003] Large-scale pre-trained language models have achieved state-of-the-art performance on many NLP (Natural Language Processing) tasks. For example, BERT (a large-scale pre-trained language model) uses a transformer (a sequence-to-sequence network structure) as its core component, which has advantages over other methods in semantic feature extraction. Given the successful applications of pre-trained language models in other fields, some researchers have also applied them to DST (Discussion and Scripting) tasks with good results. For instance, they use pre-trained language models as encoders of domains, slots, slot values, and dialogue utterances before predicting dialogue states.

[0004] Existing dialogue state tracking models typically use pre-trained language models as encoders. However, this approach has limitations: there is no semantic correspondence between the domain and the dialogue utterances, meaning that most dialogue utterances do not contain domain-specific text, resulting in inaccurate predictions of the dialogue state. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a method and apparatus for encoding dialogue discourse to solve the technical problem of no semantic correspondence between domains and dialogue discourse.

[0006] To achieve the above objectives, according to one aspect of the present invention, a method for encoding dialogue utterances is provided, comprising:

[0007] Construct a dataset that includes positive samples and negative samples. The positive samples include dialogue utterances and their respective domains, and the negative samples include dialogue utterances and their non-domains.

[0008] The encoder is obtained by adjusting the parameters of a pre-trained language model based on a contrastive learning algorithm and using the dataset.

[0009] The encoder is used to encode the target dialogue utterances.

[0010] Optionally, an encoder is obtained by tuning the parameters of a pre-trained language model based on a contrastive learning algorithm and using the dataset, including:

[0011] The dialogue utterances are input into the first pre-trained language model, thereby outputting the dialogue utterance embedding vector; at the same time, the domain and non-domain of the dialogue utterances are input into the second pre-trained language model, thereby outputting the domain embedding vector.

[0012] The dialogue utterance embedding vector and the domain embedding vector are respectively subjected to pooling and normalization processing to obtain the dialogue utterance feature vector and the domain feature vector;

[0013] Based on the contrastive learning algorithm and the cross-entropy loss function, and using the dialogue utterance feature vector and the domain feature vector, the first pre-trained language model and the second pre-trained language model are optimized to obtain the encoder.

[0014] Optionally, the first pre-trained language model and the second pre-trained language model have the same model structure and share parameters.

[0015] Optionally, the dialogue utterance embedding vector and the domain embedding vector are respectively subjected to pooling and normalization processing to obtain the dialogue utterance feature vector and the domain feature vector, including:

[0016] The average embedding vector of the dialogue utterance is obtained by summing the embedding vectors of each word in the dialogue utterance embedding vector and averaging them.

[0017] The average embedding vector of the dialogue utterance is normalized to obtain the dialogue utterance feature vector;

[0018] The average embedding vector of the domain is obtained by summing the embedding vectors of each word in the domain embedding vector and averaging them.

[0019] The average embedding vector of the domain is normalized to obtain the domain feature vector.

[0020] Optionally, in the process of optimizing the pre-trained language model, the optimization is carried out with the goal of reducing the distance between the dialogue utterance and its domain in the feature space, and increasing the distance between the dialogue utterance and the non-domain in the feature space.

[0021] Optionally, the cross-entropy loss function is as follows:

[0022]

[0023] Where Sim(·) represents the cosine distance, z i Represents the feature vector of dialogue discourse. e represents the domain feature vector of the domain. - j This represents the domain feature vector that does not belong to the domain, τ represents the temperature coefficient, and k represents the number of domains.

[0024] Optionally, the pre-trained language model is selected from one of the following:

[0025] BERT, XLNet, RoBERTa, ALBERT.

[0026] Furthermore, according to another aspect of the present invention, a device for encoding dialogue utterances is provided, comprising:

[0027] A construction module is used to construct a dataset, which includes positive samples and negative samples. The positive samples include dialogue utterances and their respective domains, and the negative samples include dialogue utterances and their non-domains.

[0028] The training module is used to adjust the parameters of a pre-trained language model based on a contrastive learning algorithm and using the dataset to obtain the encoder;

[0029] The encoding module is used to encode the target dialogue utterances using the encoder.

[0030] Optionally, the training module is further configured to:

[0031] The dialogue utterances are input into the first pre-trained language model, thereby outputting the dialogue utterance embedding vector; at the same time, the domain and non-domain of the dialogue utterances are input into the second pre-trained language model, thereby outputting the domain embedding vector.

[0032] The dialogue utterance embedding vector and the domain embedding vector are respectively subjected to pooling and normalization processing to obtain the dialogue utterance feature vector and the domain feature vector;

[0033] Based on the contrastive learning algorithm and the cross-entropy loss function, and using the dialogue utterance feature vector and the domain feature vector, the first pre-trained language model and the second pre-trained language model are optimized to obtain the encoder.

[0034] Optionally, the first pre-trained language model and the second pre-trained language model have the same model structure and share parameters.

[0035] Optionally, the training module is further configured to:

[0036] The average embedding vector of the dialogue utterance is obtained by summing the embedding vectors of each word in the dialogue utterance embedding vector and averaging them.

[0037] The average embedding vector of the dialogue utterance is normalized to obtain the dialogue utterance feature vector;

[0038] The average embedding vector of the domain is obtained by summing the embedding vectors of each word in the domain embedding vector and averaging them.

[0039] The average embedding vector of the domain is normalized to obtain the domain feature vector.

[0040] Optionally, in the process of optimizing the pre-trained language model, the optimization is carried out with the goal of reducing the distance between the dialogue utterance and its domain in the feature space, and increasing the distance between the dialogue utterance and the non-domain in the feature space.

[0041] Optionally, the cross-entropy loss function is as follows:

[0042]

[0043] Where Sim(·) represents the cosine distance, z i Represents the feature vector of dialogue discourse. e represents the domain feature vector of the domain. - j This represents the domain feature vector that does not belong to the domain, τ represents the temperature coefficient, and k represents the number of domains.

[0044] Optionally, the pre-trained language model is selected from one of the following:

[0045] BERT, XLNet, RoBERTa, ALBERT.

[0046] According to another aspect of the present invention, an electronic device is also provided, comprising:

[0047] One or more processors;

[0048] Storage device for storing one or more programs.

[0049] When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any of the above embodiments.

[0050] According to another aspect of the present invention, a computer-readable medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the methods described in any of the above embodiments.

[0051] According to another aspect of the present invention, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods described in any of the above embodiments.

[0052] One embodiment of the above invention has the following advantages or beneficial effects: By constructing a dataset including positive and negative samples (positive samples include dialogue utterances and their respective domains, negative samples include dialogue utterances and non-domains), and then using a contrastive learning algorithm and the dataset to adjust the parameters of a pre-trained language model to obtain the encoder, the technical means of overcoming the technical problem in existing technologies where there is no semantic correspondence between domains and dialogue utterances. This embodiment of the invention uses a contrastive learning algorithm to fit the semantic gap between dialogue utterances and their domains, establishing a precise semantic mapping relationship between dialogue utterances and domains. This allows for fine-tuning of the pre-trained language model, making it easier for the dialogue state tracking model to establish the relevance between dialogue utterances and their respective domains, thus helping to improve the accuracy of dialogue state tracking.

[0053] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0055] Figure 1 This is a schematic diagram of the main flow of the dialogue utterance encoding method according to an embodiment of the present invention;

[0056] Figure 2 This is a schematic diagram of a Siamese network for a pre-trained language model M according to an embodiment of the present invention;

[0057] Figure 3 This is a schematic diagram of the main flow of a dialogue utterance encoding method according to a possible embodiment of the present invention;

[0058] Figure 4 This is a schematic diagram of the main flow of a method for encoding dialogue utterances according to another applicable embodiment of the present invention;

[0059] Figure 5 This is a schematic diagram of the main modules of a dialogue encoding device according to an embodiment of the present invention;

[0060] Figure 6This is an exemplary system architecture diagram in which embodiments of the present invention can be applied;

[0061] Figure 7 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation

[0062] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0063] Figure 1 This is a schematic diagram illustrating the main flow of a dialogue utterance encoding method according to an embodiment of the present invention. As one embodiment of the present invention, such as... Figure 1 As shown, the encoding method for the dialogue utterances may include:

[0064] Step 101: Construct a dataset, which includes positive samples and negative samples. The positive samples include dialogue utterances and their respective domains, and the negative samples include dialogue utterances and their non-domains.

[0065] Positive and negative samples can be extracted from the MultiWOZ dataset to construct the dataset. Specifically, each turn of dialogue utterance in the MultiWOZ dataset, its domain, and several non-domain utterances are extracted to construct positive and negative samples for use by subsequent contrastive learning algorithms.

[0066] Step 101 is the data preparation stage. In the MultiWOZ dataset, each round of dialogue utterance corresponds to a specific domain. Therefore, a round of dialogue utterance and its corresponding domain naturally form a positive sample, while the same round of dialogue utterance and the other domains form a negative sample. For example, the dataset is constructed by extracting each round of dialogue utterance, its corresponding domain, and five non-corresponding domains from the MultiWOZ dataset. Alternatively, the dataset can be constructed by extracting each round of dialogue utterance, its corresponding domain, and eight non-corresponding domains from the MultiWOZ dataset.

[0067] Step 102: Based on the contrastive learning algorithm and using the dataset, the parameters of the pre-trained language model are adjusted to obtain the encoder.

[0068] Contrastive learning is a supervised algorithm that reduces the distance between similar instances in feature space and increases the distance between dissimilar instances. It improves the expressive power of a pre-trained language model by bringing similar samples closer together and dissimilar samples further apart.

[0069] In this step, the pre-trained language model M extracts the feature representations of the positive and negative samples respectively. Then, the cross-entropy loss function of the contrastive learning framework is followed to optimize the model parameters of the language model M, thereby fine-tuning the parameters of the language model M.

[0070] Optionally, the pre-trained language model is selected from one of the following: BERT, XLNet, RoBERTa, and ALBERT. These pre-trained language models can be fine-tuned using the methods provided in the embodiments of the present invention. The fine-tuned model can accurately encode dialogue utterances, thereby improving the accuracy of the prediction results of dialogue states.

[0071] Optionally, step 102 may include: inputting the dialogue utterances into a first pre-trained language model to output dialogue utterance embedding vectors; simultaneously, inputting the domain and non-domain of the dialogue utterances into a second pre-trained language model to output domain embedding vectors; performing pooling and normalization processing on the dialogue utterance embedding vectors and the domain embedding vectors respectively to obtain dialogue utterance feature vectors and domain feature vectors; optimizing the first pre-trained language model and the second pre-trained language model based on a contrastive learning algorithm and a cross-entropy loss function, and using the dialogue utterance feature vectors and the domain feature vectors, to obtain an encoder. Optionally, the first pre-trained language model and the second pre-trained language model have the same model structure and share parameters. Figure 2 As shown, this embodiment of the invention is based on a Siamese network using a pre-trained language model M (M can be BERT, XLNet, RoBERTa, or ALBERT), which consists of symmetrical upper and lower branches (a first pre-trained language model and a second pre-trained language model) with shared parameters. This embodiment adds pooling and normalization operations after the output layer of the pre-trained language model M to calculate the dialogue utterance feature vector and the domain language feature vector. To optimize the pre-trained language model M, the Siamese network is used to update the weights, ensuring that the resulting sentence embedding vectors are semantically meaningful and can be calculated in pairs using cosine similarity.

[0072] Optionally, in optimizing the pre-trained language model, the optimization objective is to reduce the distance between the dialogue utterance and its domain in the feature space, and to increase the distance between the dialogue utterance and its non-domain in the feature space. For example, the upper branch (first pre-trained language model M1) inputs a round of dialogue utterance, and the lower branch (second pre-trained language model M2) inputs the domain and non-domain of the dialogue utterance. If it is a positive sample, the similarity between the dialogue utterance feature vector and the domain feature vector is higher; if it is a negative sample, the similarity between the dialogue utterance feature vector and the domain feature vector is lower.

[0073] Optionally, the cross-entropy loss function is as follows:

[0074]

[0075] Where Sim(·) represents the cosine distance, z i Represents the feature vector of dialogue discourse. e represents the domain feature vector of the domain. - j This represents the domain feature vector that does not belong to the domain, τ represents the temperature coefficient, and k represents the number of domains.

[0076] The model was trained for one epoch (a complete training run using all the data in the training set, called "epoch training") with a learning rate of 3e-5, and evaluated on the MultiWOZ 2.1 and MultiWOZ 2.4 datasets for every 250 training batches. Evaluations on the validation and test datasets were averaged based on the best checkpoint. The temperature hyperparameter in the loss function was set to 0.05.

[0077] Optionally, pooling and normalization are performed on the dialogue utterance embedding vector and the domain embedding vector respectively to obtain the dialogue utterance feature vector and the domain feature vector, including: averaging the embedding vectors of each word in the dialogue utterance embedding vector to obtain the average dialogue utterance embedding vector; normalizing the average dialogue utterance embedding vector to obtain the dialogue utterance feature vector; averaging the embedding vectors of each word in the domain embedding vector to obtain the average domain embedding vector; and normalizing the average domain embedding vector to obtain the domain feature vector. In this embodiment of the invention, pooling and normalization operations are added after the output layer of the pre-trained language model M. Specifically, for each round of dialogue utterances, the embedding vectors of each word in the dialogue utterance embedding vectors output by the pre-trained language model M are summed and averaged, and then normalized to map the embedding vectors onto the unit hypersphere. Similarly, for each round of dialogue utterances belonging to and not belonging to the domain, the embedding vectors of each word in the domain embedding vectors output by the pre-trained language model M are summed and averaged, and then normalized to map the embedding vectors onto the unit hypersphere. This allows for a better representation of the text content.

[0078] Contrastive learning frameworks can fine-tune pre-trained language models to better bridge the semantic gap between dialogue context and domain, providing high-quality embeddings for both. This addresses the performance degradation of dialogue state tracking models caused by errors resulting from the lack of relevant domain information in the dialogue utterance. This invention employs a contrastive learning algorithm to fit the semantic gap between dialogue utterance and domain, thereby obtaining a language model with better expressive power.

[0079] Step 103: Encode the target dialogue utterance using the encoder.

[0080] After fine-tuning the pre-trained language model, an encoder is obtained. This encoder then replaces the encoder in the existing dialogue state tracking model. Before inputting the dialogue utterances into the dialogue state tracking model, the encoder encodes the utterances, and then the encoded result is input into the dialogue state tracking model to track the dialogue state.

[0081] It should be noted that the encoder in this embodiment of the invention can be used for different dialogue state tracking models.

[0082] Based on the various embodiments described above, it can be seen that the embodiments of the present invention solve the technical problem in the prior art of lacking a semantic correspondence between domains and dialogue utterances by constructing a dataset, which includes positive samples and negative samples. Positive samples include dialogue utterances and their respective domains, while negative samples include dialogue utterances and non-domain-specific utterances. Then, based on a contrastive learning algorithm and using the dataset, the parameters of a pre-trained language model are adjusted to obtain the encoder. The embodiments of the present invention fit the semantic gap between dialogue utterances and their domains using a contrastive learning algorithm, establishing a precise semantic mapping relationship between dialogue utterances and domains. This allows for fine-tuning of the pre-trained language model, making it easier for the dialogue state tracking model to establish the relevance between dialogue utterances and their respective domains, thus helping to improve the accuracy of dialogue state tracking.

[0083] Figure 3 This is a schematic diagram illustrating the main flow of a dialogue utterance encoding method according to a possible embodiment of the present invention. As another embodiment of the present invention, such as... Figure 3 As shown, the encoding method for the dialogue utterances may include:

[0084] Step 301: Construct a dataset, which includes positive samples and negative samples. The positive samples include dialogue utterances and their respective domains, and the negative samples include dialogue utterances and their non-domains.

[0085] It is possible to extract each round of dialogue utterances from the MultiWOZ dataset, the domain to which each round of dialogue utterances belong, and several non-domains, such as 1 belonging domain and 6 non-domains, or 1 belonging domain and 10 non-domains, thereby constructing positive and negative samples for use by subsequent contrastive learning algorithms.

[0086] Step 302: Input the dialogue utterance into the first pre-trained language model to output the dialogue utterance embedding vector; at the same time, input the domain and non-domain of the dialogue utterance into the second pre-trained language model to output the domain embedding vector.

[0087] To optimize the pre-trained language model M, a Siamese network is used to update the weights, ensuring that the generated sentence embedding vectors are semantically meaningful and can be calculated in pairs using cosine similarity. The parameters can be fine-tuned using a Siamese network based on the pre-trained language model M (M can be BERT, XLNet, RoBERTa, or ALBERT). The upper and lower branches of the Siamese network (the first and second pre-trained language models) are symmetrical and share parameters. The upper branch (the first pre-trained language model M1) receives a round of dialogue utterances, while the lower branch (the second pre-trained language model M2) receives the domain and non-domain information of the dialogue utterances.

[0088] Step 303: Perform pooling and normalization processing on the dialogue utterance embedding vector and the domain embedding vector respectively to obtain the dialogue utterance feature vector and the domain feature vector.

[0089] Specifically, the average value of the embedding vectors of each word in the dialogue utterance embedding vector is calculated by summing the embedding vectors of each word in the dialogue utterance embedding vector; the average embedding vector of the dialogue utterance is normalized to obtain the dialogue utterance feature vector; the average embedding vector of the domain utterance embedding vector is calculated by summing the embedding vectors of each word in the domain embedding vector; the average embedding vector of the domain utterance embedding vector is normalized to obtain the domain feature vector.

[0090] Step 304: Based on the contrastive learning algorithm and the cross-entropy loss function, and using the dialogue utterance feature vector and the domain feature vector, the pre-trained language model is optimized to obtain the encoder.

[0091] The pre-trained language model M extracts the feature representations of positive and negative samples respectively. Then, the cross-entropy loss function of the contrastive learning framework is followed to optimize the model parameters of the language model M, thereby fine-tuning the parameters of the language model M.

[0092] In optimizing the pre-trained language model, the optimization aims to reduce the distance between the dialogue utterance and its domain in the feature space, and to increase the distance between the dialogue utterance and the non-domain utterance in the feature space. That is, for positive samples, the goal is to increase the similarity between the dialogue utterance feature vector and the domain feature vector, and for negative samples, the goal is to decrease the similarity between the dialogue utterance feature vector and the domain feature vector.

[0093] Step 305: Encode the target dialogue utterance using the encoder.

[0094] Before inputting the dialogue utterances into the dialogue state tracking model, an encoder is used to encode the dialogue utterances, and then the encoded results are input into the dialogue state tracking model to track the dialogue state.

[0095] Furthermore, the specific implementation details of the dialogue encoding method in one of the reference embodiments of the present invention have been described in detail in the dialogue encoding method described above, so the details will not be repeated here.

[0096] Figure 4 This is a schematic diagram of the main flow of a dialogue utterance encoding method according to another possible embodiment of the present invention. As another embodiment of the present invention, such as... Figure 4 As shown, the encoding method for the dialogue utterances may include:

[0097] Step 401: Define the dialogue discourse and its domain and non-domain.

[0098] Specifically, a conversation session containing T rounds is defined as follows:

[0099] X t ={(R1,U1),…,(R t U t ),…,(R T U T )}

[0100] Among them, R t and U t Let these represent the system discourse and user discourse in round t, respectively.

[0101] Define the domain to which the T-round dialogue belongs, and represent it as:

[0102] B = {B1,…,B} t ,…,B T},

[0103] Define the non-domains of T-round dialogue utterances, which have N domains in total, represented as follows:

[0104] D = {D1, ..., D} N}

[0105] Step 402: Construct a dataset, which includes positive samples and negative samples. The positive samples include dialogue utterances and their respective domains, and the negative samples include dialogue utterances and their non-domains.

[0106] Step 403: Input the dialogue utterance into the first BERT model to output the dialogue utterance embedding vector; at the same time, input the domain and non-domain of the dialogue utterance into the second BERT model to output the domain embedding vector.

[0107] The discourse in round t is represented as follows: This represents the sequence concatenation operation.

[0108] Dialogue utterance embedding vector H t As shown below:

[0109] H t =M([CLS]X t [SEP]) (1)

[0110] in, This is the dialogue utterance embedding vector output by the first BERT model, where L is the length of the dialogue sequence and k is the encoder hidden size. The [CLS] tag is used to aggregate the representations of all words in a sentence, and [SEP] is used to mark the end of the sentence.

[0111] Domain Embedding Vector As shown below:

[0112]

[0113] in, This is the domain embedding vector output by the second BERT model, where L is the length of the domain sequence and k is the encoder hidden size. The [CLS] tag is used to aggregate the representations of all words in a sentence, and [SEP] is used to mark the end of the sentence.

[0114] Step 404: The average value of the embedding vectors of each word in the dialogue utterance embedding vector is calculated by summing the embedding vectors of each word in the dialogue utterance embedding vector. The average embedding vector of the dialogue utterance is then normalized to obtain the dialogue utterance feature vector. At the same time, the average value of the embedding vectors of each word in the domain embedding vector is calculated by summing the embedding vectors of each word in the domain embedding vector. The average embedding vector of the domain embedding vector is then normalized to obtain the domain feature vector.

[0115] The dialogue utterance feature vectors after pooling and normalization are shown below:

[0116] z t =LN(AVG(H t (3)

[0117] in, is the feature vector of the t-th round of dialogue, AVG(·) refers to the average of the sum of the embedding vectors of each word in the sentence, and LN(·) is the normalization function.

[0118] The neighborhood feature vectors after pooling and normalization are shown below:

[0119]

[0120]

[0121] in, It is a sentence embedding of domain j.

[0122] Step 405: Based on the contrastive learning algorithm and the cross-entropy loss function, and using the dialogue utterance feature vector and the domain feature vector, optimize the first BERT model and the second BERT model to obtain the encoder.

[0123] The purpose of contrastive learning is to shorten the distance between semantically similar samples and widen the distance between dissimilar samples, thereby learning high-quality embedding representations. The goal is to map dialogue utterances and domain information to the same representation space.

[0124] Suppose a set of sample pairs z here i and It is semantically related, for z i and It is semantically irrelevant.

[0125] The embodiments of this invention follow the cross-entropy loss function of the contrastive learning framework:

[0126]

[0127] Where Sim(·) represents the cosine distance, z i Represents the feature vector of dialogue discourse. e represents the domain feature vector of the domain. - j This represents the domain feature vector that does not belong to the domain, τ represents the temperature coefficient, and k represents the number of domains.

[0128] In the optimization process, the goal is to aggregate semantically similar dialogue utterances and domain pairs in the vector space, and push away dissimilar vectors.

[0129] Step 406: Encode the target dialogue utterance using the encoder.

[0130] Finally, the BERT (i.e., the encoder) with fine-tuned parameters was obtained. The BERT with fine-tuned parameters was used in the dialogue state tracking model as the encoder to complete the dialogue state tracking task.

[0131] Furthermore, the specific implementation details of the dialogue encoding method in another reference embodiment of the present invention have been described in detail in the dialogue encoding method described above, so the details will not be repeated here.

[0132] Figure 5 This is a schematic diagram of the main modules of a dialogue encoding device according to an embodiment of the present invention. Figure 5 As shown, the dialogue utterance encoding device 500 includes a construction module 501, a training module 502, and an encoding module 503; wherein, the construction module 501 is used to construct a dataset, the dataset including positive samples and negative samples, the positive samples including dialogue utterances and their respective domains, and the negative samples including dialogue utterances and their non-domains; the training module 502 is used to adjust the parameters of a pre-trained language model based on a contrastive learning algorithm and using the dataset, thereby obtaining an encoder; the encoding module 503 is used to encode the target dialogue utterance using the encoder.

[0133] Optionally, the training module 502 is further configured to:

[0134] The dialogue utterances are input into the first pre-trained language model, thereby outputting the dialogue utterance embedding vector; at the same time, the domain and non-domain of the dialogue utterances are input into the second pre-trained language model, thereby outputting the domain embedding vector.

[0135] The dialogue utterance embedding vector and the domain embedding vector are respectively subjected to pooling and normalization processing to obtain the dialogue utterance feature vector and the domain feature vector;

[0136] Based on the contrastive learning algorithm and the cross-entropy loss function, and using the dialogue utterance feature vector and the domain feature vector, the first pre-trained language model and the second pre-trained language model are optimized to obtain the encoder.

[0137] Optionally, the first pre-trained language model and the second pre-trained language model have the same model structure and share parameters.

[0138] Optionally, the training module 502 is further configured to:

[0139] The average embedding vector of the dialogue utterance is obtained by summing the embedding vectors of each word in the dialogue utterance embedding vector and averaging them.

[0140] The average embedding vector of the dialogue utterance is normalized to obtain the dialogue utterance feature vector;

[0141] The average embedding vector of the domain is obtained by summing the embedding vectors of each word in the domain embedding vector and averaging them.

[0142] The average embedding vector of the domain is normalized to obtain the domain feature vector.

[0143] Optionally, in the process of optimizing the pre-trained language model, the optimization is carried out with the goal of reducing the distance between the dialogue utterance and its domain in the feature space, and increasing the distance between the dialogue utterance and the non-domain in the feature space.

[0144] Optionally, the cross-entropy loss function is as follows:

[0145]

[0146] Where Sim(·) represents the cosine distance, z i Represents the feature vector of dialogue discourse. e represents the domain feature vector of the domain. - j This represents the domain feature vector that does not belong to the domain, τ represents the temperature coefficient, and k represents the number of domains.

[0147] Optionally, the pre-trained language model is selected from one of the following:

[0148] BERT, XLNet, RoBERTa, ALBERT.

[0149] It should be noted that the specific implementation of the dialogue encoding device described in this invention has been described in detail in the dialogue encoding method described above, so the content will not be repeated here.

[0150] Figure 6 An exemplary system architecture 600 is shown, in which a method or apparatus for encoding dialogue utterances according to embodiments of the present invention can be applied.

[0151] like Figure 6 As shown, system architecture 600 may include terminal devices 601, 602, and 603, a network 604, and a server 605. Network 604 serves as the medium for providing communication links between terminal devices 601, 602, and 603 and server 605. Network 604 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0152] Users can use terminal devices 601, 602, and 603 to interact with server 605 via network 604 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 601, 602, and 603, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0153] Terminal devices 601, 602, and 603 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0154] Server 605 can be a server that provides various services, such as a backend management server that supports shopping websites browsed by users using terminal devices 601, 602, and 603 (this is just an example). The backend management server can analyze and process data such as received item information query requests, and then feed the processing results back to the terminal devices.

[0155] It should be noted that the dialogue encoding method provided in this embodiment of the invention is generally executed by server 605, and correspondingly, the dialogue encoding device is generally located in server 605. The dialogue encoding method provided in this embodiment of the invention can also be executed by terminal devices 601, 602, and 603, and correspondingly, the dialogue encoding device can be located in terminal devices 601, 602, and 603.

[0156] It should be understood that Figure 6The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0157] The following is for reference. Figure 7 It shows a schematic diagram of the structure of a computer system 700 suitable for implementing a terminal device of the present invention. Figure 7 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0158] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 702 or programs loaded from storage section 708 into random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the system 700. The CPU 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0159] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0160] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 709, and / or installed from removable medium 711. When the computer program is executed by central processing unit (CPU) 701, it performs the functions defined above in the system of this invention.

[0161] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0162] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer programs according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0163] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be housed in a processor; for example, a processor can be described as including a construction module, a training module, and an encoding module, wherein the names of these modules do not necessarily limit the module itself.

[0164] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, implement the following method: constructing a dataset including positive samples and negative samples, wherein the positive samples include dialogue utterances and their respective domains, and the negative samples include dialogue utterances and their non-domains; adjusting the parameters of a pre-trained language model based on a contrastive learning algorithm and using the dataset to obtain an encoder; and encoding the target dialogue utterance using the encoder.

[0165] In another aspect, embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, implements the methods described in any of the above embodiments.

[0166] According to the technical solution of this invention, by constructing a dataset including positive and negative samples, where positive samples include dialogue utterances and their respective domains, and negative samples include dialogue utterances and their non-domains, and then using a contrastive learning algorithm and the dataset to adjust the parameters of a pre-trained language model to obtain the encoder, the technical means overcome the technical problem in the prior art where there is no semantic correspondence between domains and dialogue utterances. This invention, through a contrastive learning algorithm, fits the semantic gap between dialogue utterances and their domains, establishing a precise semantic mapping relationship between dialogue utterances and domains, thereby fine-tuning the pre-trained language model. This makes it easier for the dialogue state tracking model to establish the relevance between dialogue utterances and their respective domains, contributing to improved accuracy in dialogue state tracking.

[0167] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for encoding dialogue utterances, characterized in that, include: Construct a dataset that includes positive samples and negative samples. The positive samples include dialogue utterances and their respective domains, and the negative samples include dialogue utterances and their non-domains. A round of dialogue utterances and a corresponding domain naturally form a positive sample, and the same round of dialogue utterances and other domains form a negative sample. Based on the contrastive learning algorithm, the goal is to reduce the distance between the dialogue utterance and its domain in the feature space, and to increase the distance between the dialogue utterance and the non-domain in the feature space. The dataset is used to adjust the parameters of the pre-trained language model to obtain the encoder. The encoder is used to encode the target dialogue utterances, and the encoding result is used to input into the dialogue state tracking model to track the dialogue state.

2. The method according to claim 1, characterized in that, The encoder is obtained by adjusting the parameters of a pre-trained language model based on a contrastive learning algorithm and using the aforementioned dataset, including: The dialogue utterances are input into the first pre-trained language model, thereby outputting the dialogue utterance embedding vector; at the same time, the domain and non-domain of the dialogue utterances are input into the second pre-trained language model, thereby outputting the domain embedding vector. The dialogue utterance embedding vector and the domain embedding vector are respectively subjected to pooling and normalization processing to obtain the dialogue utterance feature vector and the domain feature vector; Based on the contrastive learning algorithm and the cross-entropy loss function, and using the dialogue utterance feature vector and the domain feature vector, the first pre-trained language model and the second pre-trained language model are optimized to obtain the encoder.

3. The method according to claim 2, characterized in that, The first pre-trained language model and the second pre-trained language model have the same model structure and share parameters.

4. The method according to claim 2, characterized in that, The dialogue utterance embedding vector and the domain embedding vector are respectively subjected to pooling and normalization processing to obtain the dialogue utterance feature vector and the domain feature vector, including: The average embedding vector of the dialogue utterance is obtained by summing the embedding vectors of each word in the dialogue utterance embedding vector and averaging them. The average embedding vector of the dialogue utterance is normalized to obtain the dialogue utterance feature vector; The average embedding vector of the domain is obtained by summing the embedding vectors of each word in the domain embedding vector and averaging them. The average embedding vector of the domain is normalized to obtain the domain feature vector.

5. The method according to claim 2, characterized in that, In optimizing the pre-trained language model, the optimization aims to reduce the distance between the dialogue utterance and its domain in the feature space, and to increase the distance between the dialogue utterance and the non-domain in the feature space.

6. The method according to claim 5, characterized in that, The cross-entropy loss function is as follows: Where Sim(·) represents the cosine distance, Represents the feature vector of dialogue discourse. This represents the domain feature vector. This represents a domain feature vector that does not belong to the domain. It is represented as a temperature coefficient, and k represents the number of regions.

7. The method according to claim 1, characterized in that, The pre-trained language model is selected from one of the following: BERT, XLNet, RoBERTa, ALBERT.

8. A device for encoding dialogue, characterized in that, include: A construction module is used to construct a dataset, which includes positive samples and negative samples. The positive samples include dialogue utterances and their corresponding domains, and the negative samples include dialogue utterances and their non-domains. A turn of dialogue utterance and a corresponding domain naturally form a positive sample, and the same turn of dialogue utterance and other domains form a negative sample. The training module is used to adjust the parameters of the pre-trained language model based on the contrastive learning algorithm, with the goal of reducing the distance between the dialogue utterance and its domain in the feature space and increasing the distance between the dialogue utterance and the non-domain in the feature space, and to obtain the encoder by using the dataset. The encoding module is used to encode the target dialogue utterance using the encoder, and the encoding result is used to input into the dialogue state tracking model to track the dialogue state.

9. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7.

10. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.

Images