Dialogue system testing method and device, sentence rewriting method

Abstract

Disclosed are a method and device for testing a dialogue system, and a sentence rewriting method. The sentence structure information of multiple preset questions in a set of preset questions with the same semantics corresponding to the dialogue system is mapped to a random vector distribution, and the corresponding random vector distribution parameters are obtained. Based on the random vector distribution parameters, sampling is performed from the random vector distribution to obtain a random vector. The preset question and the random vector are input into a sequence model to obtain a rewritten question corresponding to the preset question. The rewritten question is input into the dialogue system to obtain a response output. It is determined whether the response output is consistent with the preset response of the dialogue system corresponding to the preset question. Therefore, according to the scheme disclosed in the present invention, by automatically generating synonymous question expressions, the test of the dialogue system can be automated, which can greatly reduce the manual workload and greatly improve the test efficiency.

Description

Technical Field

[0001] The present disclosure relates to dialog system simulation testing and text processing. Background Art

[0002] A dialogue system (or also called a "question-answering system") refers to a system that uses a machine or intelligent system to automatically answer inquiries from users.

[0003] In customer service scenarios, dialogue systems (or "question-and-answer systems") play an increasingly important role. For some simple scenarios, dialogue systems can greatly save the workload of manual customer service.

[0004] Many dialogue systems in customer service scenarios are based on the QA (Question Answering) model.

[0005] In simple terms, configure {(q1,r1), (q2,r2),..., (q n ,r n )} such qr (question-answer, question-answer / answer) pairs, n is the number of qr pairs in the database.

[0006] Receive user input (query) and perform preprocessing. Then use DeepQA model or other QA models to match, from the database {(q1,r1),(q2,r2),...,(q n ,r n )} select the query that is closest to the query semantics m and the corresponding response r m , 1≤m≤n. m After post-processing, it is returned to the user.

[0007] In order to ensure the accuracy of QA matching, multiple questions are often configured for the same semantics. For example, if a user wants to inquire about the waterproof performance of a certain mobile phone, the following questions may be asked:

[0008] Is it waterproof?

[0009] What is the waterproof rating of this phone?

[0010] Will it be damaged if it gets wet?

[0011] ·......

[0012] In this way, on the one hand, it is expected that the dialogue system can give correct responses to various questions. On the other hand, it is also hoped that testers can input various test scenarios (test questions) that users may ask under a certain semantics into the dialogue system during the test phase, so as to detect whether the answers returned by the system can better answer the user's questions.

[0013] By rewriting the text, the existing questions in the dialogue system can be rewritten into different forms, which can help solve the above problems. Text rewriting means rewriting the text at the word, phrase or semantic level to ensure semantic consistency with the original sentence.

[0014] One existing solution is to expand the question by manually rewriting it. For multiple questions with the same semantics (indicated by the same subscript a), according to the existing question method [q 1 a ,q 2 a ,…,q n a ]Manually rewrite some questions and input the rewritten questions into the dialogue system to determine whether the output of the dialogue system can be returned to the database as an existing question [q 1 a ,q 2 a ,…,q n a ]Configured response / answer.

[0015] However, under this solution, testers need to manually construct new questions based on the existing knowledge base question formats, which is heavily dependent on manual labor and involves a very large amount of manual work.

[0016] Moreover, it is difficult to cover all possible forms of questions through manual rewriting, which means that the test may miss some situations and it is difficult to cover all scenarios.

[0017] Another possible solution is to rewrite based on rules, which may include the following rewriting methods:

[0018] 1. Word order structure transformation:

[0019] Can this phone be returned without reason? -->Can this phone be returned? Without reason

[0020] 2. Word / Phrase Level Transformation:

[0021] Why can't I open the box? -->Why can't I open the box?

[0022] However, the writing of rewriting rules still relies heavily on manual work. Moreover, rule-based rewriting is also difficult to cover all scenarios, lacks diversity, and is difficult to reflect user problems in real scenarios.

[0023] Therefore, a testing scheme or text rewriting method is still expected, which can reduce the workload of testers, cover as many test scenarios as possible, and detect problems in the dialogue system in advance, so as to ensure that users' questions can be better answered after going online. Summary of the invention

[0024] A technical problem to be solved by the present disclosure is to provide a dialogue system testing solution and a sentence rewriting solution, which can reduce the related manual workload.

[0025] According to a first aspect of the present disclosure, a dialogue system testing method is provided, comprising: mapping sentence structure information of a plurality of preset questions in a set of preset questions of the same semantics corresponding to the dialogue system to a random vector distribution, and obtaining corresponding random vector distribution parameters; sampling from the random vector distribution based on the random vector distribution parameters to obtain a random vector; inputting the preset questions and the random vectors into a sequence model to obtain a rewritten question corresponding to the preset question; inputting the rewritten question into the dialogue system to obtain a response output; and determining whether the response output is consistent with a preset response of the dialogue system corresponding to the preset question.

[0026] Optionally, the step of mapping the sentence structure information of multiple preset questions of the dialogue system corresponding to the same semantics to a random vector distribution includes: training a syntactic embedding model and a sequence-to-sequence model based on the multiple preset questions of the dialogue system corresponding to the same semantics, the syntactic embedding model being used to map the sentence structure information of multiple preset questions of the same semantics to a random vector distribution, and obtaining corresponding random vector distribution parameters.

[0027] Optionally, the step of training the syntactic embedding model and the sequence-to-sequence model based on multiple preset questions corresponding to the same semantics of the dialogue system includes: inputting the multiple preset questions into the syntactic embedding model to obtain training random vector distribution parameters; using at least one pair of different questions from the multiple preset questions as training data, using one question in the training data as the original training question, and the other question as the rewritten training question, inputting the original training question into the encoder of the sequence-to-sequence model to obtain a training context vector; sampling from the corresponding training random vector distribution based on the training random vector distribution parameters to obtain a training random vector; inputting the training random vector and the training context vector into the decoder of the sequence-to-sequence model, combining the training rewritten question, and calculating the loss function; and optimizing the parameters of the syntactic embedding model and the sequence-to-sequence model to reduce the loss function.

[0028] Optionally, the step of inputting a preset question and a random vector into a sequence model to obtain a rewritten question corresponding to the preset question may include: inputting the preset question into an encoder of the sequence model to obtain a hidden vector and a context vector; and inputting the random vector, the hidden vector and the context vector into a decoder of the sequence model to obtain a rewritten question corresponding to the preset question.

[0029] Optionally, the method may further include: taking the inverse document frequency of each word in the preset question as a weight, performing weighted summation on the word vector representation of each word in the preset question to obtain a first vector, taking the inverse document frequency of each word in the rewritten question as a weight, performing weighted summation on the word vector representation of each word in the rewritten question to obtain a second vector, determining a rewriting accuracy index based on the similarity between the first vector and the second vector; and / or determining a rewriting diversity index based on a machine translation evaluation index between the preset question and the rewritten question.

[0030] Optionally, the method may further include: sorting and / or screening a plurality of rewritten questions corresponding to the same semantics based on a rewriting accuracy index and / or a rewriting diversity index.

[0031] Optionally, the method may further include: adding the rewritten question sentence to a preset question sentence set.

[0032] According to a second aspect of the present disclosure, a method for testing a dialogue system is provided, comprising: inputting a preset question of the dialogue system into a sentence rewriting model to obtain a rewritten question corresponding to the preset question, the sentence rewriting model rewriting the input sentence into an output sentence having the same semantics as the input sentence; inputting the rewritten question into the dialogue system to obtain a response output; and determining whether the response output is consistent with a preset response of the dialogue system corresponding to the preset question.

[0033] According to a third aspect of the present disclosure, a method for expanding preset questions in a dialogue system is provided, comprising: inputting a preset question of the dialogue system into a sentence rewriting model to obtain a rewritten question corresponding to the preset question, the sentence rewriting model rewriting the input sentence into an output sentence having the same semantics as the input sentence; and adding the rewritten sentence to a set of synonymous questions corresponding to the preset question in the dialogue system.

[0034] According to a fourth aspect of the present disclosure, a test paper generation method is provided, comprising: inputting existing test questions in a question bank into a sentence rewriting model to obtain rewritten test questions corresponding to the existing test questions, the sentence rewriting model rewriting the input sentences into output sentences having the same semantics as the input sentences; and adding the rewritten test questions to the test paper.

[0035] According to a fifth aspect of the present disclosure, a method for evaluating the work of customer service personnel is provided, wherein a correct answer corresponding to a preset question has been provided to the customer service personnel, comprising: inputting the preset question into a sentence rewriting model to obtain a rewritten question corresponding to the preset question, the sentence rewriting model rewriting the input sentence into an output sentence having the same semantics as the input sentence; providing the rewritten question to the customer service personnel; receiving a response from the customer service personnel in response to the rewritten question; and determining whether the response is consistent with the correct response.

[0036] According to a sixth aspect of the present disclosure, a sentence rewriting method is provided, including: based on multiple preset sentences corresponding to the same semantics, training a syntactic embedding model and a sequence-to-sequence model, the syntactic embedding model being used to map sentence structure information of the multiple preset sentences corresponding to the same semantics to a random vector distribution, and obtaining corresponding random vector distribution parameters; based on the random vector distribution parameters, sampling from the random vector distribution to obtain a random vector; inputting the preset sentence into a sequence into an encoder of a sequence model to obtain a hidden vector and a context vector; inputting the random vector, the hidden vector and the context vector into a sequence into a decoder of a sequence model to obtain a rewritten sentence corresponding to the preset sentence.

[0037] According to a seventh aspect of the present disclosure, a sentence rewriting method is provided, comprising: mapping sentence structure information of multiple preset sentences corresponding to the same semantics to a random vector distribution, and obtaining corresponding random vector distribution parameters; based on the random vector distribution parameters, sampling from the random vector distribution to obtain a random vector; and inputting a sequence of preset sentences and random vectors into a sequence model to obtain a rewritten sentence corresponding to the preset sentence.

[0038] According to an eighth aspect of the present disclosure, a text rewriting method is provided, including: receiving a first text in a first language style; inputting the first text into a style conversion model to obtain a second text in a second language style, the style conversion model being used to convert the text in the first language style into text in the second language style; and outputting the second text.

[0039] According to a ninth aspect of the present disclosure, a paragraph rewriting method is provided, comprising: receiving a first text paragraph; analyzing a first language style of the first text paragraph; inputting the first text paragraph sentence by sentence into a style conversion model to obtain sentences in a second language style, wherein the style conversion model is used to convert text in the first language style into text in the second language style; concatenating the obtained sentences in the second language style to obtain a second text paragraph; and outputting the second text paragraph.

[0040] According to a tenth aspect of the present disclosure, a method for changing a novel text into a script is provided, comprising: inputting the novel text sentence by sentence or paragraph by paragraph; analyzing the content type of the input novel text; inputting the novel text into a text rewriting model suitable for the content type to obtain a corresponding script text, the text rewriting model being used to rewrite the novel text of the content type into a script text of the corresponding type; and splicing the obtained script texts to obtain a script.

[0041] According to an eleventh aspect of the present disclosure, a method for writing text based on film and television content is provided, including: dividing the film and television content into a plurality of film and television clips; inputting the background sound of the film and television clips into a background sound analysis model to obtain a background sound description text; inputting the speech in the film and television clips into a speech analysis model to obtain a dialogue text and / or a narration text; inputting the video image of the film and television clips into a scene analysis model to obtain a scene description text; intercepting a video clip of a character image area from the film and television clip, inputting the intercepted video clip into a character analysis model to obtain a character description text; and combining the background sound description text, the dialogue text and / or the narration text, the scene description text, and the character description text to obtain a film and television rewritten text.

[0042] According to the twelfth aspect of the present disclosure, a homework generation method is provided, including: inputting learning materials; dividing the learning materials to obtain multiple knowledge point texts; inputting the knowledge point texts into an exercise generation model to obtain corresponding exercises; and combining multiple exercises to obtain homework.

[0043] According to the thirteenth aspect of the present disclosure, a conversation analysis method is provided, comprising: extracting sentences and their contexts containing high-frequency words and / or specified keywords from conversation records; inputting the sentences and their contexts into a summary generation model to obtain summary texts; and comprehensively analyzing multiple summary texts to obtain conversation analysis results.

[0044] According to the fourteenth aspect of the present disclosure, a dialogue system testing device is provided, including: a mapping device, used to map the sentence structure information of multiple preset questions in a preset question set corresponding to the same semantics of the dialogue system to a random vector distribution, and obtain corresponding random vector distribution parameters; a sampling device, used to sample from the random vector distribution based on the random vector distribution parameters to obtain a random vector; a rewriting device, used to input the preset question and the random vector into a sequence model to obtain a rewritten question corresponding to the preset question; a question-answering device, used to input the rewritten question into the dialogue system to obtain an answer output; and a judging device, used to judge whether the answer output is consistent with a preset answer of the dialogue system corresponding to the preset question.

[0045] According to the fifteenth aspect of the present disclosure, a computing device is provided, comprising: a processor; and a memory on which executable codes are stored, and when the executable codes are executed by the processor, the processor executes the methods described in the first to thirteenth aspects above.

[0046] According to the sixteenth aspect of the present disclosure, a non-temporary machine-readable storage medium is provided, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor executes the method described in the first to thirteenth aspects above.

[0047] Therefore, according to the solution disclosed in the present invention, by automatically generating synonymous question expressions, the test of the QA-based dialogue system is automated, which can greatly reduce the manual workload of testers and greatly improve the test efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components in the exemplary embodiments of the present disclosure.

[0049] Figure 1 This is a simplified diagram of the dialogue system.

[0050] Figure 2 The diagram schematically shows a scenario in which a test system tests a dialog system.

[0051] Figure 3 It is a schematic diagram of the model training phase according to the present disclosure.

[0052] Figure 4 is a schematic flow chart of the model training phase according to the present disclosure.

[0053] Figure 5 It is a schematic diagram of the model inference stage according to the present disclosure.

[0054] Figure 6 It is a schematic flow chart of the dialogue system testing method according to the present disclosure.

[0055] Figure 7 is a schematic block diagram of a dialog system testing device according to the present disclosure.

[0056] Figure 8 A schematic diagram of the structure of a computing device that can be used to implement the method of the present disclosure is shown. DETAILED DESCRIPTION

[0057] The preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0058] According to the dialogue system test scheme disclosed in the present invention, the questions (preset questions) configured in the knowledge base are rewritten by synonyms to obtain automatically rewritten questions (rewritten questions). The dialogue system is tested using the rewritten questions. The dialogue system is evaluated based on whether the returned response result is consistent with the preset response configured in the original knowledge base (database).

[0059] First, refer to Figure 1 Briefly describe the QA-based dialogue system.

[0060] The preprocessing module performs data preprocessing on the input user query to obtain Q.

[0061] The DeepQA module matches Q with various queries in the database and finds the query q that is closest in meaning. a , output the corresponding r a .

[0062] Here, a question with the same meaning can have multiple ways of asking (preset questions), corresponding to one answer (preset response). For example, the set of preset synonymous questions Q for question a is a = {q 1 a ,···,q n a} corresponds to the preset response r a , synonymous question set Q b = {q 1 b ,···} corresponds to the preset response r b .

[0063] Post-processing module a After data post-processing, the data is output to the user as a response to the user's query.

[0064] In the technical solution of the present disclosure, for multiple questions (preset questions) corresponding to a preset answer in the database, for example, the preset question set Q of question a a = {q 0 a ,q 1 a ,···,q n a}, the set of pre-set questions Q for question b b = {q 0 b ,q 1 b ,···,q n b}, perform synonymous rewriting.

[0065] Figure 2 The diagram schematically shows a scenario in which a test system tests a dialog system.

[0066] like Figure 2 As shown, the database records various ways of asking questions corresponding to various semantics (questions). For example, for question a, the database records the preset answer ra and the corresponding preset question sentences:

[0067] Q a = {q 1 a ,q 2 a ,···,q n a},

[0068] n is the number of pre-set questions corresponding to question a.

[0069] These different preset questions are taken out from the database and input into the rewriting module (sentence rewriting model). The rewriting module (sentence rewriting model) is used to rewrite the input sentence into an output sentence with the same semantics as the input sentence.

[0070] The rewriting module is used to rewrite these different preset questions to obtain a set of rewritten questions:

[0071]

[0072] m is the number of rewritten questions.

[0073] Input the rewritten question into the dialogue system to be tested and obtain the system response output set:

[0074] R p a = {r 1p a ,r 2p a ,···,r mp a},

[0075] p represents the answer to the rewritten question during the test.

[0076] The judgment module judges these system responses and outputs r ip aThe preset answer r corresponding to the preset question sentence configured in the database of the dialogue system a Whether they are consistent, the evaluation results of the dialogue system to be tested are given.

[0077] Figure 2 The rewriting module in can be implemented in many ways. Here is a rewriting solution adapted to, for example, a QA-based dialogue system.

[0078] First, refer to Figure 3 and Figure 4 Describe the model training process of the present disclosure.

[0079] Figure 3 It is a schematic diagram of the model training phase according to the present disclosure.

[0080] The sentence rewriting model used in the present disclosure may involve a syntax embedding model (Syntax Embedding Model) and a sequence to sequence model (Seq2SeqModel).

[0081] The syntactic embedding model is used to map the sentence structure information of multiple preset questions corresponding to the same semantics into random vector distribution and obtain the corresponding random vector distribution parameters.

[0082] The sequence-to-sequence model consists of an encoder and a decoder, which converts one sentence (sequence of words) into another sentence (sequence of words).

[0083] The present disclosure trains a syntactic embedding model and the sequence-to-sequence model based on a dialogue system corresponding to multiple preset questions with the same semantics.

[0084] Figure 4 is a schematic flow chart of the model training phase according to the present disclosure.

[0085] like Figure 4 As shown, after the model training starts, in step S410, a plurality of preset questions are input into the syntactic embedding model to obtain training random vector distribution parameters μ, σ.

[0086] Here, if Figure 3 As shown, all sentences with different expressions under the same semantics are:

[0087] Q a = {q 1 a ,q 2 a ,···,q n a},

[0088] Input into the syntactic embedding model to map the sentence structure information of different expressions under the same semantics to the probability distribution N(μ,σ).

[0089] Here, the input of the syntactic embedding model is all sentences corresponding to the same semantics (preset questions).

[0090] The output is two vectors μ and σ, where μ and σ represent the mean and variance of the high-dimensional normal distribution N(μ,σ) respectively.

[0091] Then, in step S420, at least one pair of different questions among the plurality of preset questions is used as training data, one question in the training data is used as the original training question, and the other question is used as the rewritten training question, and the original training question is input into the encoder of the sequence model to obtain a training context vector c. Here, "training" is added before the term to indicate that the term is used to represent the corresponding concept involved in the training stage.

[0092] For example, with the preset question set Q a Any question in q j a is the target question (training rewritten question), set Q a Divide by q j a All other preset questions except the above can be used as original training questions.

[0093] Thus, the input of the encoder can be the set Q a Divide by q j a A collection of other pre-set questions other than Any of the pre-set questions q i a , i≠j, 1≤i≤n, 1≤j≤n.

[0094] The input of the encoder is the sentence that needs to be rewritten (the original training question) That is, q in the following formula (1) s , the output is the training context vector c.

[0095] When training based on a semantic (question) a, various combinations of i and j can be used to train q i a ,q j a (at least one pair of different questions) as training data to perform multiple trainings.

[0096] In step S430, based on the training random vector distribution parameters μ, σ, random sampling is performed from the corresponding training random vector distribution N(μ, σ) to obtain a training random vector (also referred to as a "training random sampling variable") z.

[0097] In step S440, the training random vector z and the training context vector c (for example, c and z can be concatenated) are input into the sequence decoder of the sequence model. The decoder is used to decode the final sentence output. In the training phase, the question q is rewritten in combination with the training j a , which is q in the following formula (1) t , calculate the loss function LOSS.

[0098] In step S450, it is determined whether the loss function LOSS is less than a preset threshold.

[0099] If LOSS is less than the preset threshold, it means that the model parameters have been fully optimized and the training is completed.

[0100] If LOSS is not less than the preset threshold, it indicates that the model parameters need to be further optimized, and the process goes to step S460 to optimize the parameters of the syntactic embedding model and the sequence-to-sequence model to reduce the loss function.

[0101] Here, the total LOSS function of the syntactic embedding model and the sequence-to-sequence model is defined as:

[0102]

[0103] Among them, q φ (z|Q) represents the syntactic embedding model, where φ is the parameter of the syntactic embedding model. When training for a certain semantics (question) a, its input is the set Q of all synonymous sentences (preset questions) of the semantics (question) a. a , the output is the parameters μ, σ of the high-dimensional distribution N(μ,σ). Random sampling from this distribution can produce a random variable z.

[0104] p θ (q t |z,q s ) represents a Seq2Seq model consisting of an encoder and a decoder, where θ represents the parameters of the encoder and the decoder.

[0105] p(z) is a hypothetical prior distribution, which is the above-mentioned N(μ,σ). Gaussian distribution is used here.

[0106] Below, the meaning of each item of the loss function LOSS is introduced in detail:

[0107] - is the variational lower bound, which involves the common form of the VAE (variational autoencoder) model and is used to ensure that the original question q s Able to accurately rewrite the target question q t .

[0108] KL(q φ(z|Q)||p(z)) is the KL distance (Kullback-Leibler Divergence), also called relative entropy, which is used to measure the posterior distribution q φ The difference between (z|Q) and the prior distribution p(z).

[0109] Through iterative training, the parameters of the syntactic embedding model and the Seq2Seq model are continuously optimized until the loss function LOSS is less than the preset threshold.

[0110] During training, original training questions and rewritten training questions can be extracted for training based on all sentences (preset questions) corresponding to various semantics (questions) a, b, ...

[0111] If time and computing power allow, all semantics (questions) involved in the database can be trained. For each semantics (question), all i, j combinations of all corresponding synonymous sentences (preset questions) are also used as training data pairs for training original questions and training rewritten questions for training.

[0112] Below, reference Figures 5 to 7 , describing the model inference process.

[0113] Figure 5 It is a schematic diagram of the model inference stage according to the present disclosure.

[0114] Figure 6 It is a schematic flow chart of the dialogue system testing method according to the present disclosure.

[0115] Figure 7 is a schematic block diagram of a dialog system testing device according to the present disclosure.

[0116] like Figure 5 As shown, unlike the model training phase, the inference phase no longer requires the syntactic embedding model and its input.

[0117] See also Figure 6 and Figure 7 In step S610, for example, by means of a mapping device 710, sentence structure information of a plurality of preset questions in a preset question set corresponding to the same semantics of the dialogue system is mapped to a random vector distribution, and corresponding random vector distribution parameters are obtained.

[0118] In the training phase, the random vector distribution N(μ,σ) and its parameters μ,σ after model training optimization are finally obtained. The random vector distribution parameters μ,σ corresponding to the semantics can be stored for use in later model inference.

[0119] In addition, if some semantics (questions) are not trained in the model training stage, or the corresponding random vector distribution parameters μ, σ are not stored after the training, the trained syntactic embedding model can be used to obtain the corresponding random vector distribution N(μ, σ) and its parameters μ, σ based on multiple preset questions (synonymous sentences) corresponding to the semantics.

[0120] In step S620, for example, through the sampling device 720, sampling is performed from the random vector distribution N(μ, σ) (prior distribution) based on the random vector distribution parameters μ, σ to obtain a random vector (also referred to as a "random sampling variable") z.

[0121] In step S630, for example, the preset question q is changed by the rewriting device 730. s And the random vector z is input into the sequence model to obtain the rewritten question q corresponding to the preset question t .

[0122] Here, we can use the preset question q s Input the sequence into the encoder of the sequence model to obtain the hidden vector h and the context vector c.

[0123] Then, the random vector z, hidden vector h and context vector c are concatenated, for example, and the sequence is input into the decoder of the sequence model to obtain the corresponding preset question q s Rewrite the question q t .

[0124] In step S640, for example, the question answering device 740 rewrites the question q t Input the dialogue system and get the response output.

[0125] In step S650, for example, the judging device 750 judges whether the answer output from the dialogue system corresponds to the preset question q s The preset response is consistent.

[0126] In addition, we can also rewrite the question q t Add to the preset question set Q corresponding to the corresponding semantics, so as to expand the preset question set Q and further improve the user experience when using the dialogue system.

[0127] Therefore, the dialogue system can be tested by using the rewritten questions automatically rewritten based on the preset questions.

[0128] As mentioned above, after rewriting for synonymous semantics, we can get various forms of rewritten questions:

[0129]

[0130] To make the rewritten question as similar as possible to the preset question q s ∈Q a and be precisely synonymous, the generated results can be evaluated for quality.

[0131] For example, the generated rewritten questions can also be sorted and / or filtered based on rewriting accuracy metrics and / or diversity metrics, etc.

[0132] Regarding the rewriting accuracy metric, for example, the similarity between the preset question and the rewritten question can be calculated as the rewriting accuracy metric. For example, it can be represented by Cosine-IDF.

[0133] Here, the inverse document frequency idf w can be used to reflect the importance of each word in the question:

[0134]

[0135] Among them, idf w represents the inverse document frequency of the word w in a sentence. N represents the total number of sentences in the corpus (the number of preset questions), and N w represents the number of sentences in the corpus that contain this word w.

[0136] The inverse document frequency idf w measures the category discrimination ability of the word w. The larger the idf, the more representative the term is of a document. And the IDF scores of some general words such as "了" and "是" are very low.

[0137] Taking the inverse document frequency idf s of each word w in the preset question q w as the weight, the word vector representation v s of each word w in the preset question q w is weighted and summed to obtain the first vector v s .

[0138]

[0139] Taking the inverse document frequency idf t of each word w in the rewritten question q w as the weight, the word vector representation v of each word w in the rewritten question w is weighted and summed to obtain the second vector v t .

[0140]

[0141] Then, a rewriting accuracy index may be determined based on the similarity between the first vector and the second vector.

[0142] For example, the cosine of the angle between the first vector and the second vector can be used to calculate the rewriting accuracy indicator, namely the Cosine-IDF score:

[0143] Score sim = cos(v s ,v t ).

[0144] Regarding the rewriting diversity index (diversity), the word overlap can be used to determine the diversity evaluation index. The more words the rewritten sentence (rewritten question) overlaps with the synonymous reference sentence (preset question), the lower the diversity can be considered.

[0145] Here, the rewriting diversity index can be determined based on the machine translation evaluation index BLEU between the preset question and the rewritten question:

[0146]

[0147] Is the statement q s and rewrite the statement The subscript 1 indicates the segmentation of normal words. For example, “Can this mobile phone be returned without reason?” is segmented into “Can this ||mobile phone|| be returned||without||reason||?”.

[0148] The final ranking and / or screening index may be a combination of multiple indexes.

[0149] For example, multiple rewritten questions corresponding to the same semantics may be ranked and / or screened based on a rewriting accuracy index, or based on a rewriting diversity index, or based on a combination of the rewriting accuracy index and the rewriting diversity index.

[0150] Above, the dialog system testing scheme according to the present disclosure has been described in detail.

[0151] According to the test solution disclosed in the present invention, by automatically generating synonymous question expressions, the test of the QA-based dialogue system is automated, which can greatly reduce the manual workload of testers and greatly improve the test efficiency.

[0152] Furthermore, the test scheme according to the present disclosure can generate multiple synonymous expressions, covering scenarios not covered by manual testing.

[0153] In addition, for a preset question, multiple different random vectors (random sampling variables) z can be sampled. Based on different random vectors z, different synonymous rewritten questions can be generated. This means that all scenarios can be covered to a certain extent.

[0154] By using the synonym rewriting model, we can compare the performance of different dialogue system models horizontally.

[0155] On the other hand, according to some embodiments of the present disclosure, a method for expanding preset questions in a dialogue system may also be implemented.

[0156] The preset question of the dialogue system is input into the sentence rewriting model to obtain a rewritten question corresponding to the preset question. As mentioned above, the sentence rewriting model rewrites the input sentence into an output sentence having the same semantics as the input sentence.

[0157] Then, the rewritten sentence can be added to the synonymous question set corresponding to the preset question in the dialogue system, thereby expanding the preset question set.

[0158] On the other hand, according to some embodiments of the present disclosure, a test paper generating method may also be implemented.

[0159] The existing test questions in the question bank are input into the sentence rewriting model to obtain the rewritten test questions corresponding to the existing test questions. As described above, the sentence rewriting model rewrites the input sentence into an output sentence having the same semantics as the input sentence.

[0160] The rewrite question can then be added to the test paper.

[0161] In this way, the expression of test questions can be enriched.

[0162] On the other hand, according to some embodiments of the present disclosure, a customer service personnel work evaluation method can also be implemented.

[0163] The customer service personnel have been provided with correct answers to the preset questions in advance, for example, through training, etc. Here, the work of the customer service personnel needs to be evaluated.

[0164] The preset question is input into the sentence rewriting model to obtain a rewritten question corresponding to the preset question. As described above, the sentence rewriting model rewrites the input sentence into an output sentence having the same semantics as the input sentence.

[0165] The rewritten question is provided to the customer service personnel, and the customer service personnel's response to the rewritten question is received. Then, it can be determined whether the response is consistent with the correct response, thereby evaluating the work of the customer service personnel.

[0166] On the other hand, according to some embodiments of the present disclosure, a statement rewriting method may also be implemented.

[0167] Based on multiple preset sentences corresponding to the same semantics, the syntactic embedding model and the sequence-to-sequence model are trained. The syntactic embedding model is used to map the sentence structure information of multiple preset sentences corresponding to the same semantics to random vector distribution and obtain the corresponding random vector distribution parameters.

[0168] Based on the random vector distribution parameters, a random vector is sampled from the random vector distribution to obtain a random vector.

[0169] The preset sentence is input into the sequence into the encoder of the sequence model to obtain the hidden vector and context vector.

[0170] The random vector, hidden vector and context vector are input into the sequence to the decoder of the sequence model to obtain the rewritten sentence corresponding to the preset sentence.

[0171] On the other hand, according to some embodiments of the present disclosure, a statement rewriting method may also be implemented.

[0172] The sentence structure information of multiple preset sentences corresponding to the same semantics is mapped to random vector distribution, and the corresponding random vector distribution parameters are obtained.

[0173] Based on the random vector distribution parameters, a random vector is sampled from the random vector distribution to obtain a random vector.

[0174] The preset sentence and the random vector are input into the sequence model to obtain the rewritten sentence corresponding to the preset sentence.

[0175] On the other hand, according to some embodiments of the present disclosure, a text rewriting method may also be implemented to rewrite a text in one language style into a text in another language style.

[0176] The language styles here can be, for example, humorous, serious; masculine style, feminine style; children's style, adult style; official style, folk style, self-media style, and so on.

[0177] A first text in a first language is received.

[0178] The first text is input into the style transfer model to obtain a second text in the second language style. Here, the style transfer model is used to convert the text in the first language style into the text in the second language style.

[0179] Output the second text.

[0180] On the other hand, according to some embodiments of the present disclosure, a paragraph rewriting method may also be implemented.

[0181] A first paragraph of text is received.

[0182] Analyze the first language style of the first text paragraph.

[0183] The first text paragraph is input into the style transfer model sentence by sentence to obtain sentences in the second language style. Here, the style transfer model is used to convert text in the first language style into text in the second language style.

[0184] The obtained second language style sentences are concatenated to obtain a second text paragraph.

[0185] Output the second text paragraph.

[0186] On the other hand, according to some embodiments of the present disclosure, a method for changing a novel text into a script may also be implemented.

[0187] Enter your novel text sentence by sentence or paragraph by paragraph.

[0188] Analyze the content type of the input novel text.

[0189] For example, the content types of a novel text may include environment description, scene description, character image description, character action description, dialogue, and so on.

[0190] Input a novel text into a text rewriting model suitable for the content type to obtain a corresponding script text. Here, the text rewriting model is used to rewrite the novel text of the content type into a script text of the corresponding type.

[0191] The obtained script texts are spliced ​​together to obtain the script.

[0192] On the other hand, according to some embodiments of the present disclosure, a method for writing text based on film and television content can also be implemented.

[0193] Divide the video content into multiple video clips.

[0194] The background sound of the film and television clip is input into the background sound analysis model to obtain the background sound description text.

[0195] The speech in the film and television clip is input into the speech analysis model to obtain the dialogue text and / or narration text.

[0196] The video images of the film and television clips are input into the scene analysis model to obtain the scene description text.

[0197] A video clip of a character image area is captured from a film or TV clip, and the captured video clip is input into a character analysis model to obtain a character description text.

[0198] The background sound description text, the dialogue text and / or the narration text, the scene description text, and the character description text are combined to obtain a film and television adaptation text.

[0199] For example, the background sound description text, dialogue text and / or narration text, scene description text, and character description text may be input into the text synthesis model to obtain the above-mentioned film and television rewritten text.

[0200] On the other hand, according to some embodiments of the present disclosure, a job generation method may also be implemented.

[0201] Input learning materials, such as texts, study materials, examples, existing exercises, etc.

[0202] Divide the learning materials to obtain multiple knowledge point texts.

[0203] The knowledge point text is input into the exercise generation model to obtain the corresponding exercise. The exercise generation model can rewrite the knowledge point text to obtain the exercise text.

[0204] For example, the exercise generation model can extract keywords from the exercise text to obtain fill-in-the-blank questions.

[0205] Alternatively, the exercise generation model can also modify individual words in the exercise text to change its semantics, thereby obtaining judgment questions.

[0206] Alternatively, the exercise generation model may also combine multiple exercise texts, modify individual words of zero or more exercise texts to change their semantics, and thus obtain multiple-choice questions.

[0207] Combine multiple exercises to get homework.

[0208] On the other hand, according to some embodiments of the present disclosure, a conversation analysis method may also be implemented.

[0209] Extract sentences and their contexts containing high-frequency words and / or specified keywords from the conversation records.

[0210] The conversation record may be, for example, a conversation record in a chat software, or may be a conversation record manually recorded or obtained through voice recognition.

[0211] The sentence and its context are input into the summary generation model to obtain the summary text.

[0212] By comprehensively analyzing multiple summary texts, we can obtain the results of conversation analysis.

[0213] The results of the conversation analysis can be presented in the form of a brief report.

[0214] Figure 8 A schematic diagram of the structure of a computing device that can be used to implement the above method according to an embodiment of the present invention is shown.

[0215] See also Figure 8 , the computing device 800 includes a memory 810 and a processor 820 .

[0216] The processor 820 may be a multi-core processor or may include multiple processors. In some embodiments, the processor 820 may include a general-purpose main processor and one or more special coprocessors, such as a graphics processing unit (GPU), a digital signal processor (DSP), etc. In some embodiments, the processor 820 may be implemented using a customized circuit, such as an application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0217] The memory 810 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. Among them, ROM can store static data or instructions required by the processor 820 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose the stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In some other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as a dynamic random access memory. The system memory may store some or all instructions and data required by the processor at run time. In addition, the memory 810 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory 810 may include a readable and / or writable removable storage device, such as a laser disc (CD), a read-only digital versatile disc (e.g., DVD-ROM, double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. The computer-readable storage medium does not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0218] The memory 810 stores executable codes, and when the executable codes are processed by the processor 820 , the processor 820 can execute the method described above.

[0219] The dialog system testing scheme and sentence rewriting scheme according to the present invention have been described in detail above with reference to the accompanying drawings.

[0220] In addition, the method according to the present invention may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing the above steps defined in the above method of the present invention.

[0221] Alternatively, the present invention may also be implemented as a non-temporary machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) on which executable code (or computer program, or computer instruction code) is stored. When the executable code (or computer program, or computer instruction code) is executed by a processor of an electronic device (or computing device, server, etc.), the processor executes the various steps of the above-mentioned method according to the present invention.

[0222] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or a combination of both.

[0223] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system and method according to multiple embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or code, and the part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0224] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for testing a dialogue system, comprising: Mapping sentence structure information of multiple preset questions in a set of preset questions with the same semantics corresponding to the dialogue system to a random vector distribution, and obtaining corresponding random vector distribution parameters; Based on the random vector distribution parameter, sampling is performed from the random vector distribution to obtain a random vector; Inputting a preset question sentence and the random vector into a sequence model to obtain a rewritten question sentence corresponding to the preset question sentence; Inputting the rewritten question into the dialogue system to obtain a response output; as well as Determine whether the response output is consistent with a preset response of the dialogue system corresponding to the preset question.

2. The method according to claim 1, wherein: The step of mapping sentence structure information of a plurality of preset questions corresponding to the same semantics of the dialogue system to random vector distribution comprises: Based on a plurality of preset questions corresponding to the same semantics of the dialogue system, a syntactic embedding model and the sequence-to-sequence model are trained. The syntactic embedding model is used to map the sentence structure information of the plurality of preset questions corresponding to the same semantics to a random vector distribution, and obtain corresponding random vector distribution parameters.

3. The method according to claim 2, wherein: The steps of training the syntactic embedding model and the sequence-to-sequence model based on multiple preset questions corresponding to the same semantics of the dialogue system include: Inputting the plurality of preset questions into the syntactic embedding model to obtain training random vector distribution parameters; Using at least one pair of different questions from the plurality of preset questions as training data, using one question in the training data as an original training question and the other question as a rewritten training question, inputting the original training question into an encoder of a sequence model to obtain a training context vector; Based on the training random vector distribution parameter, sampling is performed from the corresponding training random vector distribution to obtain a training random vector; Inputting the training random vector and the training context vector into a sequence into a decoder of a sequence model, combining the training rewritten question, and calculating a loss function; and Optimizing parameters of the syntactic embedding model and the sequence-to-sequence model to reduce the loss function.

4. The method according to claim 1, wherein: The step of inputting the preset question sentence and the random vector into a sequence model to obtain a rewritten question sentence corresponding to the preset question sentence comprises: Input the preset question sequence into the encoder of the sequence model to obtain a hidden vector and a context vector; and The random vector, the hidden vector and the context vector are input into a sequence of a decoder of a sequence model to obtain a rewritten question sentence corresponding to the preset question sentence.

5. The method according to claim 1, further comprising: Taking the inverse document frequency of each word in the preset question as a weight, weighted summing the word vector representation of each word in the preset question is performed to obtain a first vector, taking the inverse document frequency of each word in the rewritten question as a weight, weighted summing the word vector representation of each word in the rewritten question is performed to obtain a second vector, and determining a rewriting accuracy index based on the similarity between the first vector and the second vector; and / or A rewriting diversity index is determined based on a machine translation evaluation index between the preset question and the rewritten question.

6. The method according to claim 5, further comprising: Based on the rewriting accuracy index and / or the rewriting diversity index, multiple rewritten questions corresponding to the same semantics are sorted and / or screened.

7. The method according to claim 1, further comprising: The rewritten question sentence is added to the preset question sentence set.

8. A method for testing a dialogue system, comprising: Inputting a preset question of the dialogue system into a sentence rewriting model to obtain a rewritten question corresponding to the preset question, wherein the sentence rewriting model rewrites the input sentence into an output sentence having the same semantics as the input sentence; Inputting the rewritten question into the dialogue system to obtain a response output; as well as Determining whether the response output is consistent with a preset response of the dialogue system corresponding to the preset question; Inputting a preset question sentence of the dialogue system into a sentence rewriting model to obtain a rewritten question sentence corresponding to the preset question sentence includes: The sentence structure information of multiple preset sentences corresponding to the same semantics is mapped to random vector distribution through the syntactic embedding model, and the corresponding random vector distribution parameters are obtained; Based on the random vector distribution parameter, sampling is performed from the random vector distribution to obtain a random vector; The preset question sentence and the random vector are input into a sequence model to obtain a rewritten question sentence corresponding to the preset question sentence.

9. A method for expanding preset questions in a dialogue system, comprising: Inputting a preset question of the dialogue system into a sentence rewriting model to obtain a rewritten question corresponding to the preset question, wherein the sentence rewriting model rewrites the input sentence into an output sentence having the same semantics as the input sentence; as well as Adding the rewritten question sentence to a set of synonymous questions corresponding to the preset question sentence in the dialogue system; Inputting a preset question sentence of the dialogue system into a sentence rewriting model to obtain a rewritten question sentence corresponding to the preset question sentence includes: The sentence structure information of multiple preset sentences corresponding to the same semantics is mapped to random vector distribution through the syntactic embedding model, and the corresponding random vector distribution parameters are obtained; Based on the random vector distribution parameter, sampling is performed from the random vector distribution to obtain a random vector; The preset question sentence and the random vector are input into a sequence model to obtain a rewritten question sentence corresponding to the preset question sentence.

10. A test paper generation method, comprising: Inputting existing test questions in the question bank into a sentence rewriting model to obtain a rewritten test question corresponding to the existing test question, wherein the sentence rewriting model rewrites the input sentence into an output sentence having the same semantics as the input sentence; as well as adding the rewritten test questions to the test paper; Inputting existing test questions in the question bank into the sentence rewriting model to obtain rewritten test questions corresponding to the existing test questions includes: The sentence structure information of multiple preset sentences corresponding to the same semantics is mapped to random vector distribution through the syntactic embedding model, and the corresponding random vector distribution parameters are obtained; Based on the random vector distribution parameter, sampling is performed from the random vector distribution to obtain a random vector; The existing test questions and the random vector are input into a sequence model, corresponding to the rewritten test questions of the existing test questions.

11. A customer service staff work evaluation method, wherein: Customer service personnel have been provided with the correct responses to the preset questions, including: Inputting a preset question into a sentence rewriting model to obtain a rewritten question corresponding to the preset question, wherein the sentence rewriting model rewrites the input sentence into an output sentence having the same semantics as the input sentence; Providing the rewritten question to the customer service staff; receiving a response from the customer service staff in response to the rewritten question; and Determining whether the response is consistent with the correct response; Inputting a preset question into a sentence rewriting model to obtain a rewritten question corresponding to the preset question includes: The sentence structure information of multiple preset sentences corresponding to the same semantics is mapped to random vector distribution through the syntactic embedding model, and the corresponding random vector distribution parameters are obtained; Based on the random vector distribution parameter, sampling is performed from the random vector distribution to obtain a random vector; The preset question sentence and the random vector are input into a sequence model to obtain a rewritten question sentence corresponding to the preset question sentence.

12. A sentence rewriting method, comprising: Based on multiple preset sentences corresponding to the same semantics, a syntactic embedding model and a sequence-to-sequence model are trained, wherein the syntactic embedding model is used to map sentence structure information of the multiple preset sentences corresponding to the same semantics to a random vector distribution, and obtain corresponding random vector distribution parameters; Based on the random vector distribution parameter, sampling is performed from the random vector distribution to obtain a random vector; Input the preset sentence sequence into the encoder of the sequence model to obtain the hidden vector and context vector; The random vector, the hidden vector and the context vector are input into a sequence into a decoder of a sequence model to obtain a rewritten sentence corresponding to the preset sentence.

13. A sentence rewriting method, comprising: Mapping sentence structure information of multiple preset sentences corresponding to the same semantics to random vector distribution, and obtaining corresponding random vector distribution parameters; Based on the random vector distribution parameter, sampling from the random vector distribution to obtain a random vector; and The preset sentence and the random vector are input into a sequence model to obtain a rewritten sentence corresponding to the preset sentence.

14. A dialogue system testing device, comprising: A mapping device, used to map sentence structure information of a plurality of preset questions in a preset question set corresponding to the same semantics of the dialogue system to a random vector distribution, and obtain corresponding random vector distribution parameters; A sampling device, used for sampling from the random vector distribution based on the random vector distribution parameters to obtain a random vector; A rewriting device, used for inputting a preset question sentence and the random vector into a sequence model to obtain a rewritten question sentence corresponding to the preset question sentence; A question-answering device, used for inputting the rewritten question into the dialogue system to obtain an answer output; as well as The judging device is used to judge whether the response output is consistent with the preset response of the dialogue system corresponding to the preset question.

15. A computing device comprising: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to perform the method according to any one of claims 1 to 13.

16. A non-transitory machine-readable storage medium having executable codes stored thereon, which, when executed by a processor of an electronic device, causes the processor to execute the method according to any one of claims 1 to 13.

Images