Retrieval type multi-round dialogue method and device, storage medium and electronic equipment

Abstract

The application discloses a retrieval type multi-round dialogue method and device, a storage medium and an electronic device, and belongs to the natural language and artificial intelligence fields. The technical problem to be solved by the application is how to effectively utilize the last sentence in the historical dialogue to complete semantic matching and information screening on the remaining historical dialogue sequence, and simultaneously perform information screening on the candidate response, so that the prediction accuracy of the multi-round dialogue model is improved. The technical scheme is as follows: the method specifically comprises the following steps: acquiring a multi-round dialogue data set; constructing a multi-round dialogue model; for each piece of data in the multi-round dialogue data set, each sentence in the historical dialogue is sequentially marked according to the dialogue order to construct a historical dialogue sequence, and a candidate response is constructed for each historical dialogue sequence; each sentence in the historical dialogue sequence and the candidate response are respectively subjected to encoding processing, so that the encoding representation of each sentence in the historical dialogue and the candidate response encoding representation are obtained; and the multi-round dialogue model is trained.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing and artificial intelligence, specifically to a retrieval-based multi-turn dialogue method and apparatus, storage medium, and electronic device. Background Technology

[0002] Statistics show that the number of people suffering from various mental illnesses has exceeded 100 million, including over 6.4 million with schizophrenia and 1.1 million with bipolar disorder. However, the diagnosis and treatment of mental illnesses are severely inadequate, with a treatment rate of less than 10%. Mental health experts point out that the greatest harm is not the mental illness itself, but people's attitudes towards it. Simply put, people with mental illnesses are often unwilling to acknowledge or face their condition, and are even more worried about others knowing about it. In this situation, patients often don't confide in anyone, leading to a worsening of their condition. With the rapid development of internet technology, interactive question-and-answer technology has become increasingly sophisticated. If this technology can be used for patient consultation, it can effectively reduce patients' aversion to seeing psychiatrists and avoid the stigma associated with discussing mental health issues in front of a real person.

[0003] Dialogue question answering technology includes single-turn and multi-turn dialogues. Since patients generally lack specialized medical knowledge and are often unable to describe their problems professionally in one go, multi-turn dialogue technology is needed to understand their descriptions and predict their symptoms. In multi-turn dialogue tasks, the last sentence in the historical dialogue often has special value for the entire dialogue. On the one hand, it is the sentence that leads to all the historical dialogue sequences, a result of the historical dialogue sequence, meaning it connects to the preceding text; on the other hand, it plays the greatest role in selecting the correct response in terms of logical consistency and semantic coherence, meaning it inspires the following text. Because the last sentence in the historical dialogue sequence acts as a dividing point, connecting the preceding and following text, it should play a greater role in filtering information about the historical dialogue and candidate responses. However, existing methods have not paid enough attention to this issue, resulting in unsatisfactory performance of existing models.

[0004] Therefore, how to effectively utilize the last sentence in the historical dialogue to complete semantic matching and information filtering of the remaining historical dialogue sequence, and at the same time filter the candidate responses to improve the prediction accuracy of the multi-turn dialogue model, is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] The technical objective of this invention is to provide a retrieval-based multi-turn dialogue method, apparatus, storage medium, and electronic device to address the problem of how to effectively utilize the last sentence in historical dialogues to perform semantic matching and information filtering on the remaining historical dialogue sequences, while simultaneously filtering candidate responses, thereby improving the prediction accuracy of multi-turn dialogue models.

[0006] The technical objective of this invention is achieved as follows: a retrieval-based multi-turn dialogue method, the specific method of which is as follows:

[0007] Obtain a multi-turn dialogue dataset: Use web scraping technology to crawl dialogue data from online public medical Q&A platforms to obtain a multi-turn dialogue knowledge base; construct positive and negative example data for each dialogue data, and divide the training and test sets according to a set ratio;

[0008] Constructing a multi-turn dialogue model: For each piece of data in the multi-turn dialogue dataset, each sentence in the historical dialogue is sequentially labeled according to the dialogue order to construct a historical dialogue sequence, and candidate responses are constructed for each historical dialogue sequence. Each sentence and candidate response in the historical dialogue sequence is encoded to obtain the encoded representation of each sentence in the historical dialogue and the encoded representation of the candidate response. Then, taking the boundary sentence as the boundary, semantic matching and information filtering are performed on the encoded representation of each sentence in the remaining dialogue sentences. At the same time, information filtering is performed on the encoded representation of the candidate response. The information filtering operation is completed on itself, and the dialogue semantic representation is obtained through aggregation. The dialogue semantic representation is mapped to a floating-point data in a specified interval. This floating-point data is used as the matching degree between the candidate response and the historical dialogue. The matching degrees of different candidate responses are compared, and the candidate response with the highest matching degree is taken as the correct response.

[0009] Training a multi-turn dialogue model: Use the training set to train a multi-turn dialogue model, and construct loss and optimization functions to predict candidate responses.

[0010] As a preferred approach, the multi-turn dialogue model is constructed as follows:

[0011] Constructing Input Data: For each data point in the dataset, construct a historical dialogue sequence for each sentence in the historical dialogue according to the dialogue order, denoted as h1, h2, ..., h... n Select one response from multiple candidate responses as the current response, formalized as r; determine the label of the data based on whether the response is correct, i.e., if the response is correct, record it as 1; otherwise, record it as 0; the historical dialogue sequence, the current response, and the label together constitute a single input data, in the form: (h1, h2, ..., h n (r, tag);

[0012] Encoding Processing: The pre-trained language model BERT is used to encode the input data, resulting in the encoded representation of each sentence in the historical dialogue and the encoded representation of the candidate responses, denoted as... and in, It is a delimiter statement encoding representation; The candidate response is encoded and represented using the following formula:

[0013]

[0014]

[0015] Where h1, h2, ..., h n-1 ,h n `r` represents statement 1, statement 2, ..., statement n-1, statement n in the historical dialogue; `r` represents the candidate response.

[0016] Boundary matching and filtering: Taking the boundary statement as the boundary, semantic matching calculations are performed on the encoded representations of each sentence in the remaining dialogue statements, and the information filtering process is completed according to the matching degree between each sentence and the boundary statement; at the same time, the cross-attention mechanism is used to complete the information filtering process of the candidate response encoded representations of the boundary statement, and the self-attention mechanism is used to complete the filtering process of its own information; finally, the dialogue semantic representation is obtained through aggregation operation.

[0017] Label prediction: The semantic representation of the dialogue is processed through a fully connected network with a dimension of 1 and an activation function of Sigmoid to obtain the probability that the current response is the correct response;

[0018] Among these measures, when the multi-turn dialogue model has not yet been trained, it is trained to optimize the parameters of the multi-turn dialogue model.

[0019] Once the multi-turn dialogue model has been trained, it predicts which of the candidate responses is the correct response.

[0020] More specifically, the boundary matching filter is as follows:

[0021] The boundary statement encoding representation is joined with encoding representation 1, and then a fully connected network Dense with a sigmoid activation function is used to map the join result, thus obtaining the matching degree λ1 between encoding representation 1 and the boundary statement encoding representation. Next, a multiplication operation is performed on the matching degree λ1 of the boundary statement encoding representation and encoding representation 1 to obtain the filtered representation 1, denoted as . The formula is as follows:

[0022]

[0023]

[0024] in, This indicates the encoding representation of delimited statements; The encoding represents 1;

[0025] The boundary statement encoding representation is joined with encoding representation 2, and then a fully connected network Dense with a sigmoid activation function is used to map the join result, thus obtaining the matching degree λ2 between encoding representation 2 and the boundary statement encoding representation. Next, the matching degree λ2 of the boundary statement encoding representation is multiplied by encoding representation 2 to obtain the filtered representation 2, denoted as... The formula is as follows:

[0026]

[0027]

[0028] in, This indicates the encoding representation of delimited statements; The encoding represents 2;

[0029] Join the delimiter statement encoding representation and encoding representation 3. The subsequent operation is similar to obtaining filter representation 2, thus obtaining filter representation 3, denoted as... This process continues until the delimiter statement's encoded representation is joined with encoded representation n-1, and an operation similar to obtaining filter representation 2 is performed to obtain filter representation n-1, denoted as... The formula is as follows:

[0030]

[0031]

[0032] in, This indicates the encoding representation of delimited statements; The encoding represents n-1;

[0033] The filtering representations 1 through n-1, the key information representation, and the candidate response filtering representation are aggregated by addition to obtain the dialogue semantic representation, denoted as . The formula is as follows:

[0034]

[0035] in, Let them represent filter representation 1, filter representation 2, ..., filter representation n-1, respectively. This indicates the representation of key information; This indicates the candidate response filtering method.

[0036] As a preferred option, the multi-turn dialogue model is trained as follows:

[0037] Constructing the loss function: Cross-entropy is used as the loss function, as shown in the following formula:

[0038]

[0039] Among them, y true For real labels; y pred This represents the correct probability output by the multi-turn dialogue model.

[0040] Constructing the optimization function: After testing various optimization functions, AdamW was finally selected as the optimization function. Except for its learning rate, which was set to 2e-5, all other hyperparameters of AdamW were set to the default values ​​in PyTorch.

[0041] A retrieval-based multi-turn dialogue device, the device comprising:

[0042] The dataset acquisition unit is used to crawl dialogue data from online public medical question-and-answer platforms using web crawling technology to obtain a multi-turn dialogue knowledge base; positive and negative examples are constructed for each dialogue data, and the training and test sets are divided according to a set ratio;

[0043] The multi-turn dialogue model construction unit is used to construct a historical dialogue sequence for each piece of data in the multi-turn dialogue dataset by sequentially labeling each sentence in the historical dialogue according to the dialogue order, and constructing candidate responses for each historical dialogue sequence. Each sentence and candidate response in the historical dialogue sequence is encoded to obtain the encoded representation of each sentence in the historical dialogue and the encoded representation of the candidate response. Then, using the boundary sentence as a boundary, semantic matching and information filtering are performed on the encoded representation of each sentence in the remaining dialogue sentences. Simultaneously, information filtering is performed on the encoded representation of the candidate response, completing the information filtering operation on itself. Through aggregation, the dialogue semantic representation is obtained and mapped to a floating-point number in a specified range. This floating-point number serves as the matching degree between the candidate response and the historical dialogue. The matching degrees of different candidate responses are then compared, and the candidate response with the highest matching degree is taken as the correct response.

[0044] The multi-turn dialogue model training unit is used to train a multi-turn dialogue model using the training set, and to construct loss and optimization functions to predict candidate responses.

[0045] Preferably, the multi-turn dialogue model construction unit includes:

[0046] The input module is used to preprocess the raw dataset and construct the input data;

[0047] The encoding module is used to encode the input data using the pre-trained language model BERT, thereby obtaining the encoded representation of each sentence in the historical dialogue and the encoded representation of the candidate response.

[0048] The boundary matching and filtering module is used to perform semantic matching calculations on the encoded representations of each sentence in the remaining dialogue statements, with the boundary statement as the boundary, and complete the information filtering process according to the matching degree between each sentence and the boundary statement; at the same time, the cross-attention mechanism is used to complete the information filtering process of the candidate response encoded representations of the boundary statement; finally, the dialogue semantic representation is obtained through aggregation operation.

[0049] The label prediction module is used to determine whether the current response is correct based on the semantic representation of the dialogue.

[0050] Preferably, the implementation process of the boundary matching and filtering module is as follows:

[0051] (1) Connect the boundary statement encoding representation and encoding representation 1, and then use a fully connected network Dense with Sigmoid activation function to map the connection result, thereby obtaining the matching degree λ1 between encoding representation 1 and boundary statement encoding representation; then perform a multiplication operation on the matching degree λ1 between boundary statement encoding representation and encoding representation 1 to obtain the filtered representation 1, denoted as The formula is as follows:

[0052]

[0053]

[0054] in, This indicates the encoding representation of delimited statements; The encoding represents 1;

[0055] (2) Connect the boundary statement encoding representation and encoding representation 2, and then use a fully connected network Dense with a sigmoid activation function to map the connection result, thereby obtaining the matching degree λ2 between encoding representation 2 and the boundary statement encoding representation; then perform a multiplication operation on the matching degree λ2 of the boundary statement encoding representation and encoding representation 2 to obtain the filtered representation 2, denoted as The formula is as follows:

[0056]

[0057]

[0058] in, This indicates the encoding representation of delimited statements; The encoding represents 2;

[0059] (3) Connect the delimiter statement encoding representation and encoding representation 3. The subsequent operation process is similar to obtaining filter representation 2, thus obtaining filter representation 3, denoted as... This process continues until the delimiter statement's encoded representation is joined with encoded representation n-1, and an operation similar to obtaining filter representation 2 is performed to obtain filter representation n-1, denoted as... The formula is as follows:

[0060]

[0061]

[0062] in, This indicates the encoding representation of delimited statements; The encoding represents n-1;

[0063] (4) By adding the filtering representations 1 to n-1, the key information representation, and the candidate response filtering representation, the dialogue semantic representation is obtained, denoted as . The formula is as follows:

[0064]

[0065] in, Let them represent filter representation 1, filter representation 2, ..., filter representation n-1, respectively. This indicates the representation of key information; This indicates the candidate response filtering method.

[0066] Preferably, the multi-turn dialogue model training unit includes:

[0067] The loss function building module is used to calculate the error between the prediction results and the actual data using the cross-entropy loss function.

[0068] The optimization function building module is used to train and adjust the parameters during model training to reduce prediction error.

[0069] An electronic device includes: a memory and at least one processor;

[0070] The memory contains computer programs;

[0071] The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the retrieval-based multi-turn dialogue method as described above.

[0072] A computer-readable storage medium storing a computer program that can be executed by a processor to implement the retrieval-based multi-turn dialogue method described above.

[0073] The term "boundary statement" in this invention refers to the last statement sequence in the historical dialogue sequence during a multi-turn dialogue task. Because this object will be used and mentioned multiple times in this invention, and its description is quite long, it is named "boundary statement."

[0074] Remaining Dialogue Statements: In this invention, "remaining dialogue statements" refers to all historical dialogue statement sequences in a multi-turn dialogue task, excluding the last statement sequence in the historical dialogue sequence. Because this object will be used and mentioned multiple times in this invention, and its description is too long, it is named "remaining dialogue statements."

[0075] The retrieval-based multi-turn dialogue method, apparatus, storage medium, and electronic device of the present invention have the following advantages:

[0076] (i) This invention can effectively utilize the last sentence in the historical dialogue to complete semantic matching and information filtering of the remaining historical dialogue sequence, thereby obtaining effective historical semantic information; at the same time, it can filter the candidate responses to obtain accurate candidate response semantic features, thereby improving the prediction accuracy of the multi-turn dialogue model.

[0077] (ii) This invention can capture the encoded features in historical dialogues and candidate responses through encoding processing, thereby obtaining a richer and more accurate encoded representation;

[0078] (III) This invention can effectively utilize the last sentence in the historical dialogue to complete semantic matching and information filtering of the remaining historical dialogue sequence through boundary matching and filtering, thereby obtaining an effective representation of historical semantic information;

[0079] (iv) The present invention can effectively filter candidate responses by boundary matching, thereby obtaining accurate semantic feature representations of candidate responses;

[0080] (v) The present invention, combined with boundary matching screening, can effectively improve the prediction accuracy of multi-turn dialogue models. Attached Figure Description

[0081] The invention will be further described below with reference to the accompanying drawings.

[0082] Appendix Figure 1 The flowchart is for a retrieval-based multi-turn dialogue method.

[0083] Appendix Figure 2 Flowchart for building a multi-turn dialogue model;

[0084] Appendix Figure 3 A flowchart for training a multi-turn dialogue model;

[0085] Appendix Figure 4 This is a block diagram of a retrieval-based multi-turn dialogue device;

[0086] Appendix Figure 5 This is a schematic diagram illustrating the implementation process of the boundary matching and filtering module;

[0087] Appendix Figure 6 This is a schematic diagram of the framework for the boundary matching and filtering module. Detailed Implementation

[0088] The following detailed description of the retrieval-based multi-turn dialogue method, apparatus, storage medium, and electronic device of the present invention is based on the accompanying drawings and specific embodiments.

[0089] Example 1:

[0090] As attached Figure 1 As shown, this embodiment provides a retrieval-based multi-turn dialogue method, which is as follows:

[0091] S1. Obtain the multi-turn dialogue dataset, as follows:

[0092] S101. Use web crawlers to obtain data: Write a web crawler program to crawl dialogue data from online public medical Q&A platforms to obtain a multi-turn dialogue knowledge base;

[0093] For example, public medical Q&A sharing platforms on the internet contain a large amount of Q&A data, all of which is open to the public. Therefore, based on the characteristics of these platforms, a corresponding web crawler program can be designed to obtain multi-turn dialogue data, thereby creating a multi-turn dialogue knowledge base. An example of a multi-turn dialogue from a medical Q&A platform is shown in the table below:

[0094]

[0095] S102. Construct a multi-turn dialogue dataset: For each dialogue data, construct positive example data and negative example data, and divide the training set and test set according to a fixed ratio;

[0096] For example, taking the data shown in step S101 as an example, S1 to S5 are treated as historical dialogue sequences, and S6 is treated as a correct response, marked as 1; then, three dialogue statements are randomly selected from the other data and treated as incorrect responses for each statement, marked as 0. The data format in this dataset is shown in the table below:

[0097]

[0098]

[0099] The dataset was divided into training and test sets in a 4:1 ratio.

[0100] S2. Constructing a multi-turn dialogue model: For each piece of data in the multi-turn dialogue dataset, each sentence in the historical dialogue is labeled sequentially according to the dialogue order to construct a historical dialogue sequence, and candidate responses are constructed for each historical dialogue sequence. Each sentence and candidate response in the historical dialogue sequence is encoded to obtain the encoded representation of each sentence in the historical dialogue and the encoded representation of the candidate response. Then, taking the boundary sentence as the boundary, semantic matching and information filtering are performed on the encoded representation of each sentence in the remaining dialogue sentences. At the same time, information filtering is performed on the encoded representation of the candidate response. The information filtering operation is completed on itself, and the dialogue semantic representation is obtained through aggregation. The dialogue semantic representation is mapped to a floating-point data in a specified interval. This floating-point data is used as the matching degree between the candidate response and the historical dialogue. The matching degrees of different candidate responses are compared, and the candidate response with the highest matching degree is taken as the correct response.

[0101] S3. Training a multi-turn dialogue model: Use the training set to train a multi-turn dialogue model, and construct loss and optimization functions to predict candidate responses.

[0102] As attached Figure 2 As shown, the construction of the multi-turn dialogue model in this embodiment is as follows:

[0103] S201. Constructing Input Data: For each piece of data in the dataset, construct a historical dialogue sequence for each sentence in the historical dialogue according to the dialogue order, denoted as h1, h2, ..., h... n Select one response from multiple candidate responses as the current response, formalized as r; determine the label of the data based on whether the response is correct, i.e., if the response is correct, record it as 1; otherwise, record it as 0; the historical dialogue sequence, the current response, and the label together constitute a single input data, in the form: (h1, h2, ..., h n (r, tag);

[0104] For example, using the data shown in S102 as a sample, we can construct an input data set. The result is shown below:

[0105] (h1: My child is a senior in high school and will be taking the college entrance exam soon. My child gets nervous during exams and I would like to get some counseling. h2: Hello, pre-exam anxiety is very common. Are there any mental health hospitals in your area? h3: Yes, how many counseling sessions are needed before the exam? h4: I don't recommend last-minute cramming. Pre-exam anxiety is often caused by cognitive biases. You need to change your cognitive patterns regularly. Generally, it takes 8-12 sessions, but it depends on the child's situation. h5: What should I do next? r: You should take your child to a psychologist for an evaluation. They will use some psychological testing scales and follow the doctor's advice for psychotherapy or other treatments. 1).

[0106] S202. Encoding Processing: The pre-trained language model BERT is used to encode the input data, thereby obtaining the encoded representation of each sentence in the historical dialogue and the encoded representation of the candidate response, denoted as... and in, It is a delimiter statement encoding representation; The candidate response is encoded and represented using the following formula:

[0107]

[0108]

[0109] Where h1, h2, ..., h n-1 ,h n `r` represents statement 1, statement 2, ..., statement n-1, statement n in the historical dialogue; `r` represents the candidate response.

[0110] For example, when implementing this embodiment on the Ubuntu Dialogue Corpus dataset, the pre-trained language model BERT is invoked to complete the module's operations, with all settings following the default settings for BERT in PyTorch. In PyTorch, the code implementation described above is as follows:

[0111]

[0112] Here, h_embed_list represents each statement in the history of the dialogue, r is the candidate response, h_encoder_list represents the encoded representation of each statement in the history of the dialogue, response represents the sequence of options, and response_embed is the encoded representation of the candidate response.

[0113] S203, Boundary Matching and Filtering: Using the boundary statement as the boundary, semantic matching calculations are performed on the encoded representations of each sentence in the remaining dialogue statements, and the information filtering process is completed according to the matching degree between each sentence and the boundary statement; at the same time, the cross-attention mechanism is used to complete the information filtering process of the bounding statement on the encoded representations of candidate responses, and the self-attention mechanism is used to complete the filtering process of its own information; finally, the dialogue semantic representation is obtained through aggregation operation.

[0114] S204, Label Prediction: The semantic representation of the dialogue is processed through a fully connected network with a dimension of 1 and an activation function of Sigmoid to obtain the probability that the current response is the correct response;

[0115] Among these measures, when the multi-turn dialogue model has not yet been trained, it is trained to optimize the parameters of the multi-turn dialogue model.

[0116] Once the multi-turn dialogue model has been trained, it predicts which of the candidate responses is the correct response.

[0117] As attached Figure 5 and 6 As shown, the boundary matching and filtering in step S203 of this embodiment is as follows:

[0118] S20301. Connect the boundary statement encoding representation and encoding representation 1, then use a fully connected network Dense with a sigmoid activation function to map the connection result, thus obtaining the matching degree λ1 between encoding representation 1 and the boundary statement encoding representation; then perform a multiplication operation on the matching degree λ1 of the boundary statement encoding representation and encoding representation 1 to obtain the filtered representation 1, denoted as... The formula is as follows:

[0119]

[0120]

[0121] in, This indicates the encoding representation of delimited statements; The encoding represents 1;

[0122] S20302. Connect the boundary statement encoding representation and encoding representation 2, then use a fully connected network Dense with a sigmoid activation function to map the connection result, thus obtaining the matching degree λ2 between encoding representation 2 and the boundary statement encoding representation; then perform a multiplication operation on the matching degree λ2 of the boundary statement encoding representation and encoding representation 2 to obtain the filtered representation 2, denoted as... The formula is as follows:

[0123]

[0124]

[0125] in, This indicates the encoding representation of delimited statements; The encoding represents 2;

[0126] S20303, Connecting the delimiter statement encoding representation and encoding representation 3, the subsequent operation process is similar to obtaining filter representation 2, thus obtaining filter representation 3, denoted as This process continues until the delimiter statement's encoded representation is joined with encoded representation n-1, and an operation similar to obtaining filter representation 2 is performed to obtain filter representation n-1, denoted as... The formula is as follows:

[0127]

[0128]

[0129] in, This indicates the encoding representation of delimited statements; The encoding represents n-1;

[0130] The cross-attention mechanism is used to complete the information filtering process of the candidate response encoding representation by the delimiter statement, thereby obtaining the candidate response filtering representation, denoted as . The formula is as follows:

[0131]

[0132] in, This indicates that interactive computation between the delimited statement and the candidate response encoding representation is achieved through dot product multiplication operations; This indicates the encoding representation of delimited statements; This represents the candidate response encoding. This represents the dot product multiplication operation;

[0133]

[0134] This indicates that the attention weight α is obtained through a normalization operation. i , i and i' represent the indexes of the elements in the corresponding input tensor; l represents the input tensor The number of elements in;

[0135]

[0136] This indicates the use of the already obtained attention weight α. i The feature selection of the candidate response encoding representation is completed, thus obtaining the candidate response selection representation; l represents and α i The number of elements in;

[0137] Using a self-attention mechanism to encode the boundary statements completes the semantic information filtering process, thereby obtaining the key information representation, denoted as . Specific formula:

[0138]

[0139] This indicates that the boundary statement performs interactive calculations on itself through the dot product operation. This indicates the encoding representation of delimited statements. This represents the dot product multiplication operation;

[0140]

[0141] This indicates that the attention weight α is obtained through a normalization operation, where i and i' represent the indices of the corresponding elements in the input tensor; l represents the input tensor. The number of elements in;

[0142]

[0143] This indicates the use of the already obtained attention weight α. i The feature selection process for encoding the delimited statements is completed, thereby obtaining the key information representation; l represents and α i The number of elements in;

[0144] S20304. By adding the filtering representations 1 to n-1, the key information representation, and the candidate response filtering representation, the dialogue semantic representation is obtained, denoted as... The formula is as follows:

[0145]

[0146] in, Let them represent filter representation 1, filter representation 2, ..., filter representation n-1, respectively. This indicates the representation of key information; This indicates the candidate response filtering method.

[0147] For example: Due to the current lack of multi-turn dialogue datasets for public mental health conditions, this embodiment is implemented on another multi-turn dialogue dataset, Ubuntu Dialogue Corpus; in PyTorch, the code implementation described above is as follows:

[0148]

[0149]

[0150] Here, history_embed_list represents a list of all utterance encodings in the history of the dialogue; response_embed represents the candidate response encodings; and Q represents the dialogue semantic representation.

[0151] As attached Figure 3 As shown, the specific steps for training the multi-turn dialogue model in step S3 of this embodiment are as follows:

[0152] S301. Constructing the loss function: Cross-entropy is used as the loss function, as shown in the following formula:

[0153]

[0154] Among them, y true For real labels; y pred This represents the correct probability output by the multi-turn dialogue model.

[0155] For example, in PyTorch, the code implementation described above would be as follows:

[0156] #Calculate the error between the predicted value and the label using the cross-entropy loss function.

[0157] loss_fct = CrossEntropyLoss()

[0158] loss=loss_fct(logits.view(-1,self.num_labels),labels.view(-1))

[0159] Here, labels are the true labels, and logits are the correct probabilities of the model output.

[0160] S302. Constructing the optimization function: After testing various optimization functions, AdamW was finally selected as the optimization function. Except for its learning rate being set to 2e-5, all other hyperparameters of AdamW were set to the default values ​​in PyTorch.

[0161] For example, in PyTorch, the code implementation described above would be as follows:

[0162] #Optimize model parameters using the AdamW optimizer

[0163] optimizer=AdamW(optimizer_grouped_parameters,lr=2e-5)

[0164] Here, optimizer_grouped_parameters are the parameters to be optimized, which by default are all the parameters in the automatic question answering model.

[0165] If the model has not yet been trained, step S3 needs to be executed to train it and optimize the model's parameters; once the model has been trained, step S204 can be executed to predict which of the candidate responses is the correct response.

[0166] Example 2:

[0167] As attached Figure 4 As shown in this embodiment, a retrieval-based multi-turn dialogue device includes:

[0168] The dataset acquisition unit is used to crawl dialogue data from online public medical question-and-answer platforms using web crawling technology to obtain a multi-turn dialogue knowledge base; positive and negative examples are constructed for each dialogue data, and the training and test sets are divided according to a set ratio;

[0169] The multi-turn dialogue model construction unit is used to construct a historical dialogue sequence for each piece of data in the multi-turn dialogue dataset by sequentially labeling each sentence in the historical dialogue according to the dialogue order, and constructing candidate responses for each historical dialogue sequence. Each sentence and candidate response in the historical dialogue sequence is encoded to obtain the encoded representation of each sentence in the historical dialogue and the encoded representation of the candidate response. Then, using the boundary sentence as a boundary, semantic matching and information filtering are performed on the encoded representation of each sentence in the remaining dialogue sentences. Simultaneously, information filtering is performed on the encoded representation of the candidate response, completing the information filtering operation on itself. Through aggregation, the dialogue semantic representation is obtained and mapped to a floating-point number in a specified range. This floating-point number serves as the matching degree between the candidate response and the historical dialogue. The matching degrees of different candidate responses are then compared, and the candidate response with the highest matching degree is taken as the correct response.

[0170] The multi-turn dialogue model training unit is used to train a multi-turn dialogue model using the training set, and to construct loss and optimization functions to predict candidate responses.

[0171] The multi-turn dialogue model construction unit in this embodiment includes:

[0172] The input module is used to preprocess the raw dataset and construct the input data;

[0173] The encoding module is used to encode the input data using the pre-trained language model BERT, thereby obtaining the encoded representation of each sentence in the historical dialogue and the encoded representation of the candidate response.

[0174] The boundary matching and filtering module is used to perform semantic matching calculations on the encoded representations of each sentence in the remaining dialogue statements, with the boundary statement as the boundary, and complete the information filtering process according to the matching degree between each sentence and the boundary statement; at the same time, the cross-attention mechanism is used to complete the information filtering process of the candidate response encoded representations of the boundary statement; finally, the dialogue semantic representation is obtained through aggregation operation.

[0175] The label prediction module is used to determine whether the current response is correct based on the semantic representation of the dialogue.

[0176] As attached Figure 5 and 6 As shown, the implementation process of the boundary matching and filtering module in this embodiment is as follows:

[0177] (1) Connect the boundary statement encoding representation and encoding representation 1, and then use a fully connected network Dense with Sigmoid activation function to map the connection result, thereby obtaining the matching degree λ1 between encoding representation 1 and boundary statement encoding representation; then perform a multiplication operation on the matching degree λ1 between boundary statement encoding representation and encoding representation 1 to obtain the filtered representation 1, denoted as The formula is as follows:

[0178]

[0179]

[0180] in, This indicates the encoding representation of delimited statements; The encoding represents 1;

[0181] (2) Connect the boundary statement encoding representation and encoding representation 2, and then use a fully connected network Dense with a sigmoid activation function to map the connection result, thereby obtaining the matching degree λ2 between encoding representation 2 and the boundary statement encoding representation; then perform a multiplication operation on the matching degree λ2 of the boundary statement encoding representation and encoding representation 2 to obtain the filtered representation 2, denoted as The formula is as follows:

[0182]

[0183]

[0184] in, This indicates the encoding representation of delimited statements; The encoding represents 2;

[0185] (3) Connect the delimiter statement encoding representation and encoding representation 3. The subsequent operation process is similar to obtaining filter representation 2, thus obtaining filter representation 3, denoted as... This process continues until the delimiter statement's encoded representation is joined with encoded representation n-1, and an operation similar to obtaining filter representation 2 is performed to obtain filter representation n-1, denoted as... The formula is as follows:

[0186]

[0187]

[0188] in, This indicates the encoding representation of delimited statements; The encoding represents n-1;

[0189] The cross-attention mechanism is used to complete the information filtering process of the candidate response encoding representation by the delimiter statement, thereby obtaining the candidate response filtering representation, denoted as . The formula is as follows:

[0190]

[0191] in, This indicates that interactive computation between the delimited statement and the candidate response encoding representation is achieved through dot product multiplication operations; This indicates the encoding representation of delimited statements; This represents the candidate response encoding. This represents the dot product multiplication operation;

[0192]

[0193] This indicates that the attention weight α is obtained through a normalization operation. i , i and i' represent the indexes of the elements in the corresponding input tensor; l represents the input tensor The number of elements in;

[0194]

[0195] This indicates the use of the already obtained attention weight α. i The feature selection of the candidate response encoding representation is completed, thus obtaining the candidate response selection representation; l represents and α i The number of elements in;

[0196] Using a self-attention mechanism to encode the boundary statements completes the semantic information filtering process, thereby obtaining the key information representation, denoted as . Specific formula:

[0197]

[0198] This indicates that the boundary statement performs interactive calculations on itself through the dot product operation. This indicates the encoding representation of delimited statements. This represents the dot product multiplication operation;

[0199]

[0200] This indicates that the attention weight α is obtained through a normalization operation, where i and i' represent the indices of the corresponding elements in the input tensor; l represents the input tensor. The number of elements in;

[0201]

[0202] This indicates the use of the already obtained attention weight α. i The feature selection process for encoding the delimited statements is completed, thereby obtaining the key information representation; l represents and α i The number of elements in;

[0203] (4) By adding the filtering representations 1 to n-1, the key information representation, and the candidate response filtering representation, the dialogue semantic representation is obtained, denoted as . The formula is as follows:

[0204]

[0205] in, Let them represent filter representation 1, filter representation 2, ..., filter representation n-1, respectively. This indicates the representation of key information; This indicates the candidate response filtering method.

[0206] The multi-turn dialogue model training unit in this embodiment includes:

[0207] The loss function building module is used to calculate the error between the prediction results and the actual data using the cross-entropy loss function.

[0208] The optimization function building module is used to train and adjust the parameters during model training to reduce prediction error.

[0209] Example 3:

[0210] This embodiment also provides an electronic device, including: a memory and a processor;

[0211] The memory stores the instructions executed by the computer.

[0212] The processor executes computer execution instructions stored in the memory, causing the processor to perform the retrieval-based multi-turn dialogue method in any embodiment of the present invention.

[0213] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can be a microprocessor or any conventional processor.

[0214] Memory is used to store computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory. Memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, at least one application program required for a function, etc.; the data storage area can store data created based on the use of the terminal, etc. In addition, memory can also include high-speed random access memory, and can also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart memory cards (SMC), secure digital cards (SD cards), flash memory cards, at least one disk storage device, flash memory devices, or other volatile solid-state storage devices.

[0215] Example 4:

[0216] This embodiment also provides a computer-readable storage medium storing a plurality of instructions, which are loaded by a processor to cause the processor to execute the retrieval-based multi-turn dialogue method in any embodiment of the present invention. Specifically, a system or apparatus equipped with a storage medium may be provided, on which software program code implementing the functions of any of the above embodiments is stored, and the computer (or CPU or MPU) of the system or apparatus may read and execute the program code stored in the storage medium.

[0217] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.

[0218] Storage media embodiments for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.

[0219] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0220] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion unit connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion unit execute some and all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0221] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A retrieval-based multi-turn dialogue method, characterized in that, The method is as follows: Obtain a multi-turn dialogue dataset: Use web scraping technology to crawl dialogue data from online public medical Q&A platforms to obtain a multi-turn dialogue knowledge base; construct positive and negative example data for each dialogue data, and divide the training and test sets according to a set ratio; Constructing a multi-turn dialogue model: For each piece of data in the multi-turn dialogue dataset, each sentence in the historical dialogue is labeled in the order of the dialogue to construct a historical dialogue sequence, and candidate responses are constructed for each historical dialogue sequence; Each statement and candidate response in the historical dialogue sequence is encoded separately to obtain the encoded representation of each statement and the encoded representation of each candidate response in the historical dialogue. Then, taking the delimited statement as the boundary, semantic matching and information filtering are performed on the encoded representation of each statement in the remaining dialogue statements; at the same time, information filtering is performed on the encoded representation of the candidate response, and the information filtering operation is completed on itself. The dialogue semantic representation is obtained through aggregation operation, and the dialogue semantic representation is mapped to a floating-point data in a specified interval. This floating-point data is used as the matching degree between the candidate response and the historical dialogue. Then, the matching degrees of different candidate responses are compared, and the candidate response with the highest matching degree is taken as the correct response. Training a multi-turn dialogue model: Use the training set to train a multi-turn dialogue model, and construct loss and optimization functions to predict candidate responses; The specific details of the boundary matching filter are as follows: The boundary statement encoding representation is joined with encoding representation 1, and then a fully connected network Dense with sigmoid activation function is used to map the join result, thus obtaining the matching degree between encoding representation 1 and the boundary statement encoding representation. Then, the delimiter statement is encoded to represent the matching degree. Multiplying the encoded representation 1 by the result of the filtered representation 1 is denoted as . The formula is as follows: ； ； in, This indicates the encoding representation of delimited statements; The encoding represents 1; The boundary statement encoding representation is joined with encoding representation 2, and then a fully connected network Dense with sigmoid activation function is used to map the joining result, thus obtaining the matching degree between encoding representation 2 and the boundary statement encoding representation. Then, the delimiter statement is encoded to represent the matching degree. Multiplying the encoded representation 2 by the result of the filtered representation 2 is denoted as . The formula is as follows: ； ； in, This indicates the encoding representation of delimited statements; The encoding represents 2; Join the delimiter statement encoding representation and encoding representation 3. The subsequent operation is similar to obtaining filter representation 2, thus obtaining filter representation 3, denoted as... This process continues until the delimiter statement's encoded representation is joined with encoded representation n-1, and an operation similar to obtaining filter representation 2 is performed to obtain filter representation n-1, denoted as... The formula is as follows: ； ； in, This indicates the encoding representation of delimited statements; The encoding represents n-1; The filtering representations 1 through n-1, the key information representation, and the candidate response filtering representation are aggregated by addition to obtain the dialogue semantic representation, denoted as . The formula is as follows: ； in, , … Let them represent filter representation 1, filter representation 2, ..., filter representation n-1, respectively. This indicates the representation of key information; This indicates the candidate response filtering method.

2. The retrieval-based multi-turn dialogue method according to claim 1, characterized in that, The construction of a multi-turn dialogue model is as follows: Constructing Input Data: For each data point in the dataset, construct a historical dialogue sequence for each sentence in the historical dialogue according to the dialogue order, denoted as h1, h2, ..., h... n Select one response from multiple candidate responses as the current response, formalized as r; determine the label of the data based on whether the response is correct, i.e., if the response is correct, record it as 1; otherwise, record it as 0; the historical dialogue sequence, the current response, and the label together constitute a single input data, in the form of: (h1, h2, ..., h n (r, tag); Encoding Processing: The pre-trained language model BERT is used to encode the input data, resulting in the encoded representation of each sentence in the historical dialogue and the encoded representation of the candidate responses, denoted as... , , ..., , and ;in, It is a delimiter statement encoding representation; The candidate response is encoded and represented using the following formula: ； ； Where h1, h2, …, h n-1 , h n `r` represents statement 1, statement 2, ..., statement n-1, statement n in the historical dialogue; `r` represents the candidate response. Boundary matching and filtering: Taking the boundary statement as the boundary, semantic matching calculations are performed on the encoded representations of each sentence in the remaining dialogue statements, and the information filtering process is completed according to the matching degree between each sentence and the boundary statement; at the same time, the cross-attention mechanism is used to complete the information filtering process of the candidate response encoded representations of the boundary statement, and the self-attention mechanism is used to complete the filtering process of its own information; finally, the dialogue semantic representation is obtained through aggregation operation. Label prediction: The semantic representation of the dialogue is processed through a fully connected network with a dimension of 1 and an activation function of Sigmoid to obtain the probability that the current response is the correct response; Among these measures, when the multi-turn dialogue model has not yet been trained, it is trained to optimize the parameters of the multi-turn dialogue model. Once the multi-turn dialogue model has been trained, it predicts which of the candidate responses is the correct response.

3. The retrieval-based multi-turn dialogue method according to claim 1, characterized in that, The training of the multi-turn dialogue model is as follows: Constructing the loss function: Cross-entropy is used as the loss function, as shown in the following formula: ; Among them, y true For real labels; y pred This represents the correct probability output by the multi-turn dialogue model. Constructing the optimization function: After testing various optimization functions, AdamW was finally selected as the optimization function. Except for its learning rate, which was set to 2e-5, all other hyperparameters of AdamW were set to the default values ​​in PyTorch.

4. A retrieval-based multi-turn dialogue device, characterized in that, The device includes: The dataset acquisition unit is used to crawl dialogue data from online public medical question-and-answer platforms using web crawling technology to obtain a multi-turn dialogue knowledge base; positive and negative examples are constructed for each dialogue data, and the training and test sets are divided according to a set ratio; The multi-turn dialogue model construction unit is used to construct a historical dialogue sequence for each piece of data in the multi-turn dialogue dataset by sequentially labeling each sentence in the historical dialogue according to the dialogue order, and constructing candidate responses for each historical dialogue sequence. Each sentence and candidate response in the historical dialogue sequence is encoded to obtain the encoded representation of each sentence in the historical dialogue and the encoded representation of the candidate response. Then, using the boundary sentence as a boundary, semantic matching and information filtering are performed on the encoded representation of each sentence in the remaining dialogue sentences. Simultaneously, information filtering is performed on the encoded representation of the candidate response, completing the information filtering operation on itself. Through aggregation, the dialogue semantic representation is obtained and mapped to a floating-point number in a specified range. This floating-point number serves as the matching degree between the candidate response and the historical dialogue. The matching degrees of different candidate responses are then compared, and the candidate response with the highest matching degree is taken as the correct response. The multi-turn dialogue model training unit is used to train the multi-turn dialogue model using the training set, and to construct the loss function and optimization function to predict the candidate response; The implementation process of the boundary matching and filtering module is as follows: (1) Connect the boundary statement encoding representation and encoding representation 1, and then use a fully connected network Dense with Sigmoid activation function to map the connection result, thereby obtaining the matching degree between encoding representation 1 and boundary statement encoding representation. Then, the delimiter statement is encoded to represent the matching degree. Multiplying the encoded representation 1 by the result of the filtered representation 1 is denoted as . The formula is as follows: ； ； in, This indicates the encoding representation of delimited statements; The encoding represents 1; (2) Connect the boundary statement encoding representation and encoding representation 2, and then use a fully connected network Dense with Sigmoid activation function to map the connection result, thereby obtaining the matching degree between encoding representation 2 and boundary statement encoding representation. Then, the delimiter statement is encoded to represent the matching degree. Multiplying the encoded representation 2 by the result of the filtered representation 2 is denoted as . The formula is as follows: ； ； in, This indicates the encoding representation of delimited statements; The encoding represents 2; (3) Connect the delimiter statement encoding representation and encoding representation 3. The subsequent operation is similar to the operation of obtaining filter representation 2, thus obtaining filter representation 3, denoted as This process continues until the delimiter statement's encoded representation is joined with encoded representation n-1, and an operation similar to obtaining filter representation 2 is performed to obtain filter representation n-1, denoted as... The formula is as follows: ； ； in, This indicates the encoding representation of delimited statements; The encoding represents n-1; (4) By adding the filtering representations 1 to n-1, the key information representation, and the candidate response filtering representation, the dialogue semantic representation is obtained, denoted as . The formula is as follows: ； in, , … Let them represent filter representation 1, filter representation 2, ..., filter representation n-1, respectively. This indicates the representation of key information; This indicates the candidate response filtering method.

5. The retrieval-based multi-turn dialogue device according to claim 4, characterized in that, The multi-turn dialogue model construction unit includes: The input module is used to preprocess the raw dataset and construct the input data; The encoding module is used to encode the input data using the pre-trained language model BERT, thereby obtaining the encoded representation of each sentence in the historical dialogue and the encoded representation of the candidate response. The boundary matching and filtering module is used to perform semantic matching calculations on the encoded representations of each sentence in the remaining dialogue statements, with the boundary statement as the boundary, and complete the information filtering process according to the matching degree between each sentence and the boundary statement; at the same time, the cross-attention mechanism is used to complete the information filtering process of the candidate response encoded representations of the boundary statement; finally, the dialogue semantic representation is obtained through aggregation operation. The label prediction module is used to determine whether the current response is correct based on the semantic representation of the dialogue.

6. The retrieval-based multi-turn dialogue device according to claim 4, characterized in that, The multi-turn dialogue model training unit includes: The loss function building module is used to calculate the error between the prediction results and the actual data using the cross-entropy loss function. The optimization function building module is used to train and adjust the parameters during model training to reduce prediction error.

7. An electronic device, characterized in that, include: Memory and at least one processor; The memory contains computer programs; The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the retrieval-based multi-turn dialogue method as described in any one of claims 1 to 3.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed by a processor to implement the retrieval-based multi-turn dialogue method as described in any one of claims 1 to 3.

Images