Dialogue annotation method, device, computer equipment, storage medium and product

Abstract

The present application provides a dialogue annotation method, device, computer equipment, storage medium and product, which belongs to the field of Internet technology. The method includes: obtaining a first dialogue corpus, which includes multiple groups of dialogue information; determining multiple slot values, a first label of each slot value and dependency information of each slot value in the multiple groups of dialogue information, the first label is used to indicate the name of the first category to which the slot value belongs, and the dependency information is semantically associated with the slot value; based on the first labels and dependency information of the multiple slot values, determining the second labels of the multiple slot values, the second label is used to indicate the name of the second category to which the slot value belongs; based on the multiple slot values ​​and the second labels of the multiple slot values, annotating the first dialogue corpus to obtain a second dialogue corpus, and the second dialogue corpus is used to train a dialogue model. This method makes the annotation information of the dialogue corpus more detailed and rich, thereby improving the efficiency of model training based on the annotated dialogue corpus.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to a conversation annotation method, device, computer equipment, storage medium and product. Background Art

[0002] With the development of science and technology, human-computer interaction technology has been applied in many scenarios. In human-computer interaction technology, in order to realize the human-computer dialogue function, a large amount of dialogue corpus is usually collected for model training, and then the trained model can answer the user's questions to generate dialogue. Among them, the efficiency of model training depends to a large extent on the annotation information of the dialogue corpus. Therefore, how to annotate the dialogue corpus is very important to improve the training efficiency of the model. Summary of the invention

[0003] The embodiments of the present application provide a method, apparatus, computer device, storage medium and product for annotating dialogues, which can make the annotation information of dialogue corpus more detailed and rich, thereby improving the efficiency of model training based on the annotated dialogue corpus. The technical solution is as follows:

[0004] In one aspect, a conversation annotation method is provided, the method comprising:

[0005] Acquire a first dialogue corpus, wherein the first dialogue corpus includes multiple groups of dialogue information;

[0006] Determine multiple slot values ​​in the multiple groups of dialog information, a first label of each slot value, and dependency information of each slot value, wherein the first label is used to indicate a name of a first category to which the slot value belongs, and the dependency information is semantically associated with the slot value;

[0007] Determine, based on the first labels and dependency information of the plurality of slot values, second labels of the plurality of slot values, where the second labels are used to indicate the name of a second category to which the slot value belongs;

[0008] Based on the multiple slot values ​​and the second labels of the multiple slot values, the first dialogue corpus is labeled to obtain a second dialogue corpus, where the second dialogue corpus is used to train a dialogue model.

[0009] In one implementation, determining the second labels of the plurality of slot values ​​based on the first labels of the plurality of slot values ​​and the dependency information includes:

[0010] Obtaining a first confidence, where the first confidence is an average of confidences of first labels of multiple slot values ​​included in multiple dialogue corpora in a dialogue corpus set, and the dialogue prediction set includes the first dialogue prediction;

[0011] Based on the first confidence, the first labels of the plurality of slot values, and the dependency information of the plurality of slot values, second labels of the plurality of slot values ​​are determined.

[0012] In one implementation, determining the second labels of the plurality of slot values ​​based on the first confidence, the first labels of the plurality of slot values, and the dependency information of the plurality of slot values ​​includes:

[0013] When the first confidence level is greater than a first preset value, classifying the multiple slot values ​​based on the first labels and dependency information of the multiple slot values ​​to obtain multiple categories of slot values;

[0014] For each slot value in each type of slot value, the dependency information of the slot value and the first label are combined to obtain a second label of the slot value.

[0015] In one implementation, the classifying the multiple slot values ​​based on the first labels and dependency information of the multiple slot values ​​to obtain multiple types of slot values ​​includes at least one of the following:

[0016] Acquire multiple slot values ​​in the dialogue corpus set, and for multiple slot values ​​corresponding to the same first label, classify the multiple slot values ​​into one category when the number of same dependency information corresponding to the multiple slot values ​​is greater than a preset threshold;

[0017] Acquire multiple slot values ​​in the dialogue corpus set, and for multiple slot values ​​corresponding to the same first label, classify the multiple slot values ​​based on the dependency information of each slot value when the number of identical dependency information corresponding to the multiple slot values ​​is not greater than a preset threshold, wherein the dependency information corresponding to multiple slot values ​​in each category of slot values ​​is identical;

[0018] Obtain multiple slot values ​​in the dialogue corpus set, and for multiple slot values ​​corresponding to the same first label, determine the similarity between the dependency information of the multiple slot values ​​when the number of identical dependency information corresponding to the multiple slot values ​​is not greater than a preset threshold, and classify the multiple slot values ​​based on the similarity, wherein the similarity between the dependency information corresponding to the multiple slot values ​​in each category of slot values ​​is greater than a preset similarity.

[0019] In one implementation, determining the second labels of the plurality of slot values ​​based on the first confidence, the first labels of the plurality of slot values, and the dependency information of the plurality of slot values ​​includes:

[0020] When the first confidence level is not greater than a first preset value, obtaining multiple slot values ​​in the dialogue corpus set, and clustering the multiple slot values ​​based on dependency information of the multiple slot values ​​to obtain multiple categories of slot values;

[0021] Obtaining a second confidence level for each category of slot values, where the second confidence level is an average of confidence levels of a plurality of slot values ​​in each category of slot values ​​that belong to the category to which they belong;

[0022] Based on the second confidence of each type of slot value, second labels of multiple slot values ​​in each type of slot value are determined, and second labels of each slot value corresponding to each type of slot value in the multiple groups of dialogue information are obtained.

[0023] In one implementation, determining second labels of multiple slot values ​​in each category of slot values ​​based on the second confidence of each category of slot values ​​includes:

[0024] For each type of slot value, when the second confidence is greater than a second preset value, for each slot value therein, combining the dependency information of the slot value and the first label to obtain a second label of the slot value;

[0025] For each type of slot value, when the second confidence is not greater than the second preset value, for multiple slot values ​​therein, determine the target sentences associated with the sentence to which each slot value belongs, determine the target words with the highest frequency in the target sentences associated with the multiple slot values, combine the dependency information of the slot value and the target words, and obtain second labels for the multiple slot values.

[0026] In one implementation, clustering the multiple slot values ​​based on dependency information of the multiple slot values ​​includes:

[0027] Determine first semantic information of each slot value and second semantic information of dependency information of each slot value respectively;

[0028] For each slot value, performing a weighted summation on the first semantic information and the second semantic information to obtain target semantic information of the slot value;

[0029] The multiple slot values ​​are clustered based on target semantic information respectively corresponding to the multiple slot values.

[0030] In one implementation, the labeling the first dialogue corpus based on the multiple slot values ​​and the second labels of the multiple slot values ​​includes:

[0031] Based on the multiple slot values ​​and the second labels of the multiple slot values, determining a dialog task of the first dialog corpus from a dialog mapping relationship, wherein the dialog mapping relationship is a mapping relationship between the multiple slot values, the second labels of the multiple slot values, and the dialog task;

[0032] The first dialogue corpus is labeled based on a plurality of slot values ​​of the first dialogue corpus, a second label of each slot value, and the dialogue task.

[0033] In one implementation, the method further includes:

[0034] Determining action information of the first dialogue data;

[0035] The second dialogue corpus is labeled based on the action information to obtain a third dialogue corpus, and the third dialogue corpus is used to train a dialogue model.

[0036] In one implementation, the first dialogue corpus is a dialogue corpus without action information labeled;

[0037] The determining the action information of the first dialogue data includes:

[0038] Based on the dialogue corpus with marked action information, model training is performed to obtain an action recognition model, wherein the action recognition model is used to recognize action information in the dialogue corpus;

[0039] The first dialogue corpus is input into the action recognition model, and action information of the first dialogue corpus is output.

[0040] In one implementation, determining a plurality of slot values ​​in the plurality of groups of dialog information includes:

[0041] Determine a first slot value and a second slot value in the multiple groups of dialogue information to obtain multiple slot values ​​in the multiple groups of dialogue information, wherein the first slot value is a necessary slot value for marking the first dialogue material, and the second slot value is used to assist in marking the first dialogue material.

[0042] In another aspect, a conversation annotation device is provided, the device comprising:

[0043] An acquisition module, configured to acquire a first dialogue corpus, wherein the first dialogue corpus includes multiple groups of dialogue information;

[0044] A first determination module, configured to determine a plurality of slot values ​​in the plurality of groups of dialog information, a first label of each slot value, and dependency information of each slot value, wherein the first label is used to indicate a name of a first category to which the slot value belongs, and the dependency information is semantically associated with the slot value;

[0045] A second determining module, configured to determine second labels of the plurality of slot values ​​based on the first labels of the plurality of slot values ​​and the dependency information, wherein the second labels are used to indicate the name of a second category to which the slot value belongs;

[0046] The first labeling module is used to label the first dialogue corpus based on the multiple slot values ​​and the second labels of the multiple slot values ​​to obtain a second dialogue corpus, where the second dialogue corpus is used to train the dialogue model.

[0047] In one implementation, the second determining module includes:

[0048] an acquisition unit, configured to acquire a first confidence, wherein the first confidence is an average of confidences of first labels of multiple slot values ​​included in multiple dialogue corpora in a dialogue corpus set, wherein the dialogue corpus set includes the first dialogue corpus;

[0049] A determining unit is configured to determine second labels of the plurality of slot values ​​based on the first confidence, the first labels of the plurality of slot values, and dependency information of the plurality of slot values.

[0050] In one implementation, the determining unit includes:

[0051] a classification subunit, configured to classify the plurality of slot values ​​based on the first labels and dependency information of the plurality of slot values ​​to obtain a plurality of categories of slot values ​​when the first confidence level is greater than a first preset value;

[0052] The combining subunit is used to combine the dependency information of each slot value in each type of slot value with the first label to obtain a second label of the slot value.

[0053] In one implementation, the classification subunit is used to:

[0054] Acquire multiple slot values ​​in the dialogue corpus set, and for multiple slot values ​​corresponding to the same first label, classify the multiple slot values ​​into one category when the number of same dependency information corresponding to the multiple slot values ​​is greater than a preset threshold;

[0055] Acquire multiple slot values ​​in the dialogue corpus set, and for multiple slot values ​​corresponding to the same first label, classify the multiple slot values ​​based on the dependency information of each slot value when the number of identical dependency information corresponding to the multiple slot values ​​is not greater than a preset threshold, wherein the dependency information corresponding to multiple slot values ​​in each category of slot values ​​is identical;

[0056] Obtain multiple slot values ​​in the dialogue corpus set, and for multiple slot values ​​corresponding to the same first label, determine the similarity between the dependency information of the multiple slot values ​​when the number of identical dependency information corresponding to the multiple slot values ​​is not greater than a preset threshold, and classify the multiple slot values ​​based on the similarity, wherein the similarity between the dependency information corresponding to the multiple slot values ​​in each category of slot values ​​is greater than a preset similarity.

[0057] In one implementation, the determining unit includes:

[0058] a clustering subunit, configured to obtain, when the first confidence level is not greater than a first preset value, a plurality of slot values ​​in the dialogue corpus, and cluster the plurality of slot values ​​based on dependency information of the plurality of slot values ​​to obtain a plurality of categories of slot values;

[0059] An acquisition subunit, used for acquiring a second confidence level of each type of slot value, where the second confidence level is an average of confidence levels of a plurality of slot values ​​in each type of slot value that belong to the type to which they belong;

[0060] The determination subunit is used to determine the second labels of multiple slot values ​​in each category of slot values ​​based on the second confidence of each category of slot values, and obtain the second label of each slot value corresponding to each category of slot values ​​in the multiple groups of dialogue information.

[0061] In one implementation, the determining subunit includes:

[0062] For each type of slot value, when the second confidence is greater than a second preset value, for each slot value therein, combining the dependency information of the slot value and the first label to obtain a second label of the slot value;

[0063] For each type of slot value, when the second confidence is not greater than the second preset value, for multiple slot values ​​therein, determine the target sentences associated with the sentence to which each slot value belongs, determine the target words with the highest frequency in the target sentences associated with the multiple slot values, combine the dependency information of the slot value and the target words, and obtain second labels for the multiple slot values.

[0064] In one implementation, the clustering subunit includes:

[0065] Determine first semantic information of each slot value and second semantic information of dependency information of each slot value respectively;

[0066] For each slot value, performing a weighted summation on the first semantic information and the second semantic information to obtain target semantic information of the slot value;

[0067] The multiple slot values ​​are clustered based on target semantic information respectively corresponding to the multiple slot values.

[0068] In one implementation, the first marking module includes:

[0069] Based on the multiple slot values ​​and the second labels of the multiple slot values, determining a dialog task of the first dialog corpus from a dialog mapping relationship, wherein the dialog mapping relationship is a mapping relationship between the multiple slot values, the second labels of the multiple slot values, and the dialog task;

[0070] The first dialogue corpus is labeled based on a plurality of slot values ​​of the first dialogue corpus, a second label of each slot value, and the dialogue task.

[0071] In one implementation, the device further includes:

[0072] A third determining module, used to determine the action information of the first dialogue data;

[0073] The second annotation module is used to annotate the second dialogue corpus based on the action information to obtain a third dialogue corpus, and the third dialogue corpus is used to train the dialogue model.

[0074] In one implementation, the first dialogue corpus is a dialogue corpus without action information labeled;

[0075] The third determining module is used to:

[0076] Based on the dialogue corpus with marked action information, model training is performed to obtain an action recognition model, wherein the action recognition model is used to recognize action information in the dialogue corpus;

[0077] The first dialogue corpus is input into the action recognition model, and action information of the first dialogue corpus is output.

[0078] In one implementation, the first determining module is configured to:

[0079] Determine a first slot value and a second slot value in the multiple groups of dialogue information to obtain multiple slot values ​​in the multiple groups of dialogue information, wherein the first slot value is a necessary slot value for marking the first dialogue material, and the second slot value is used to assist in marking the first dialogue material.

[0080] On the other hand, a computer device is provided, comprising one or more processors and one or more memories, wherein at least one program code is stored in the one or more memories, and the at least one program code is loaded and executed by the one or more processors to implement the conversation annotation method described in any of the above implementations.

[0081] On the other hand, a computer-readable storage medium is provided, in which at least one program code is stored. The at least one program code is loaded and executed by a processor to implement the conversation annotation method described in any of the above implementations.

[0082] On the other hand, a computer program product is provided, which includes a computer program code, wherein the computer program code is stored in a computer-readable storage medium, a processor of a computer device reads the computer program code from the computer-readable storage medium, and the processor executes the computer program code, so that the computer device executes the conversation annotation method described in any of the above-mentioned implementations.

[0083] An embodiment of the present application provides a method for annotating dialogues. This method can not only obtain slot values ​​in dialogue corpus, but also divide the slot values ​​more finely based on dependency information of the slot values ​​and a first label of a first category to which the slot values ​​belong, thereby obtaining a second label of a second category to which the slot values ​​belong. The dialogue corpus can then be annotated based on the slot values ​​and the second labels of the slot values, thereby making the annotation information of the dialogue corpus more detailed and rich, thereby improving the efficiency of model training based on the annotated dialogue corpus. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0085] Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present application;

[0086] Figure 2 is a flow chart of a conversation annotation method provided in an embodiment of the present application;

[0087] Figure 3 is a flow chart of a conversation annotation method provided in an embodiment of the present application;

[0088] Figure 4 is a schematic diagram of a dialogue corpus for marking slot values ​​provided in an embodiment of the present application;

[0089] Figure 5 is a schematic diagram of a dependency word relationship in a sentence provided by an embodiment of the present application;

[0090] Figure 6 is a schematic diagram of a dialogue corpus for marking action information provided by an embodiment of the present application;

[0091] Figure 7 is a schematic diagram of a second dialogue corpus provided in an embodiment of the present application;

[0092] Figure 8 is a schematic diagram of a conversation transition probability provided by an embodiment of the present application;

[0093] Fig. 9 is a flow chart of a conversation annotation method provided in an embodiment of the present application;

[0094] Fig.10 is a schematic diagram of a dialogue corpus for marking action information provided by an embodiment of the present application;

[0095] Fig.11is a block diagram of a conversation annotation device provided in an embodiment of the present application;

[0096] Fig.12 is a block diagram of a terminal provided in an embodiment of the present application;

[0097] Fig.13 This is a block diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0098] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0099] It should be noted that the user information involved in this application (including but not limited to user device information, user personal information, etc.) is information authorized by the user or fully authorized by all parties.

[0100] The terms "first", "second", "third" and "fourth" etc. in the specification and claims of the present application and the drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0101] The present application embodiment provides an implementation environment of a conversation annotation method, see Figure 1 , the implementation environment includes a first computer device 10 and a second computer device 20, which are connected by wire or wirelessly. The first computer device 10 is used to obtain a large amount of first dialogue corpus without labeled information, annotate the first dialogue corpus to obtain annotated second dialogue corpus, and send the second dialogue corpus to the second computer device 20; the second computer device 20 is provided with a dialogue model, and the second computer device 20 is used to train the dialogue model based on the annotated second dialogue corpus, so as to realize human-computer dialogue based on the trained dialogue model; optionally, the dialogue model is used in dialogue scenarios such as ordering air tickets and purchasing goods, but is not limited thereto.

[0102] Among them, the first computer device 10 and the second computer device 10 can both be provided as terminals or servers; in the embodiment of the present application, the first computer device 10 and the second computer device 20 are both provided as servers for illustration; optionally, the first computer device 10 is provided as a first server, and the second computer device 20 is provided as a second server, and the first server and the second server are connected by wire or wirelessly. The first server is used to obtain a large amount of first dialogue corpus without labeled information, annotate the first dialogue corpus to obtain annotated second dialogue corpus, and send the second dialogue corpus to the second server; the second server is provided with a dialogue model, and the second server is used to train the dialogue model based on the annotated second dialogue corpus.

[0103] The present application embodiment provides a method for annotating a conversation, the execution subject of which is a computer device, see Figure 2 , methods include:

[0104] Step 201: Obtain a first dialogue corpus, where the first dialogue corpus includes multiple groups of dialogue information.

[0105] Step 202: Determine multiple slot values ​​in multiple groups of dialog information, a first label of each slot value, and dependency information of each slot value, wherein the first label is used to indicate the name of a first category to which the slot value belongs, and the dependency information is semantically associated with the slot value.

[0106] Step 203: Based on the first labels and dependency information of the plurality of slot values, second labels of the plurality of slot values ​​are determined, where the second labels are used to indicate the name of the second category to which the slot value belongs.

[0107] Step 204: Based on the multiple slot values ​​and the second labels of the multiple slot values, the first dialogue corpus is labeled to obtain a second dialogue corpus, and the second dialogue corpus is used to train the dialogue model.

[0108] In one implementation, determining second labels of the plurality of slot values ​​based on the first labels and dependency information of the plurality of slot values ​​includes:

[0109] Obtaining a first confidence, where the first confidence is an average of confidences of first labels of multiple slot values ​​included in multiple dialogue corpora in the dialogue corpus set, and the dialogue prediction set includes the first dialogue prediction;

[0110] Based on the first confidence, the first labels of the plurality of slot values, and the dependency information of the plurality of slot values, second labels of the plurality of slot values ​​are determined.

[0111] In one implementation, determining second labels of the plurality of slot values ​​based on the first confidence, the first labels of the plurality of slot values, and dependency information of the plurality of slot values ​​includes:

[0112] When the first confidence is greater than a first preset value, classifying the multiple slot values ​​based on the first labels and dependency information of the multiple slot values ​​to obtain multiple categories of slot values;

[0113] For each slot value in each category of slot values, the dependency information of the slot value and the first label are combined to obtain a second label of the slot value.

[0114] In one implementation, based on the first labels and dependency information of the multiple slot values, the multiple slot values ​​are classified to obtain multiple categories of slot values, including at least one of the following:

[0115] Acquire multiple slot values ​​in the dialogue corpus set, and for multiple slot values ​​corresponding to the same first label, classify the multiple slot values ​​into one category when the number of same dependency information corresponding to the multiple slot values ​​is greater than a preset threshold;

[0116] Acquire multiple slot values ​​in the dialogue corpus set, and for multiple slot values ​​corresponding to the same first label, classify the multiple slot values ​​based on the dependency information of each slot value when the number of identical dependency information corresponding to the multiple slot values ​​is not greater than a preset threshold, wherein the dependency information corresponding to the multiple slot values ​​in each category of slot values ​​is identical;

[0117] Obtain multiple slot values ​​in a dialogue corpus set, and for multiple slot values ​​corresponding to the same first label, determine the similarity between the dependency information of the multiple slot values ​​when the number of identical dependency information corresponding to the multiple slot values ​​is not greater than a preset threshold, and classify the multiple slot values ​​based on the similarity, wherein the similarity between the dependency information corresponding to the multiple slot values ​​in each category of slot values ​​is greater than a preset similarity.

[0118] In one implementation, determining second labels of the plurality of slot values ​​based on the first confidence, the first labels of the plurality of slot values, and dependency information of the plurality of slot values ​​includes:

[0119] When the first confidence level is not greater than a first preset value, a plurality of slot values ​​in the dialogue corpus are obtained, and based on dependency information of the plurality of slot values, the plurality of slot values ​​are clustered to obtain a plurality of categories of slot values;

[0120] Obtaining a second confidence level for each type of slot value, where the second confidence level is an average of confidence levels of multiple slot values ​​in each type of slot value that belong to the same type;

[0121] Based on the second confidence of each type of slot value, second labels of multiple slot values ​​in each type of slot value are determined, and second labels of each slot value corresponding to each type of slot value in multiple groups of dialogue information are obtained.

[0122] In one implementation, determining second labels of multiple slot values ​​in each category of slot values ​​based on the second confidence of each category of slot values ​​includes:

[0123] For each type of slot value, when the second confidence is greater than a second preset value, for each slot value therein, combining the dependency information of the slot value and the first label to obtain a second label of the slot value;

[0124] For each type of slot value, when the second confidence is not greater than the second preset value, for multiple slot values ​​therein, determine the target sentences associated with the sentence to which each slot value belongs, determine the target words with the highest frequency in the target sentences associated with the multiple slot values, combine the dependency information and the target words of the slot values, and obtain second labels for the multiple slot values.

[0125] In one implementation, clustering the multiple slot values ​​based on dependency information of the multiple slot values ​​includes:

[0126] Determine first semantic information of each slot value and second semantic information of dependency information of each slot value respectively;

[0127] For each slot value, performing a weighted summation on the first semantic information and the second semantic information to obtain the target semantic information of the slot value;

[0128] Based on the target semantic information respectively corresponding to the multiple slot values, the multiple slot values ​​are clustered.

[0129] In one implementation, labeling the first dialogue corpus based on the plurality of slot values ​​and the second labels of the plurality of slot values ​​includes:

[0130] Based on the multiple slot values ​​and the multiple second labels of the slot values, determining a dialog task of the first dialog corpus from a dialog mapping relationship, where the dialog mapping relationship is a mapping relationship between the multiple slot values, the multiple second labels of the slot values, and the dialog task;

[0131] The first dialogue corpus is labeled based on multiple slot values ​​of the first dialogue corpus, a second label of each slot value, and a dialogue task.

[0132] In one implementation, the method further includes:

[0133] Determine the action information of the first dialogue data;

[0134] The second dialogue corpus is labeled based on the action information to obtain a third dialogue corpus, and the third dialogue corpus is used to train the dialogue model.

[0135] In one implementation, the first dialogue corpus is a dialogue corpus without action information labeled;

[0136] Determine the action information of the first dialogue data, including:

[0137] Based on the dialogue corpus with annotated action information, a model is trained to obtain an action recognition model, which is used to recognize action information in the dialogue corpus;

[0138] The first dialogue corpus is input into the action recognition model, and the action information of the first dialogue corpus is output.

[0139] In one implementation, determining multiple slot values ​​in multiple sets of dialog information includes:

[0140] Determine the first slot value and the second slot value in the multiple groups of dialogue information to obtain multiple slot values ​​in the multiple groups of dialogue information, wherein the first slot value is a necessary slot value for marking the first dialogue corpus, and the second slot value is used to assist in marking the first dialogue corpus.

[0141] An embodiment of the present application provides a method for annotating dialogues. This method can not only obtain slot values ​​in dialogue corpus, but also divide the slot values ​​more finely based on dependency information of the slot values ​​and a first label of a first category to which the slot values ​​belong, thereby obtaining a second label of a second category to which the slot values ​​belong. The dialogue corpus can then be annotated based on the slot values ​​and the second labels of the slot values, thereby making the annotation information of the dialogue corpus more detailed and rich, thereby improving the efficiency of model training based on the annotated dialogue corpus.

[0142] This application embodiment provides a method for annotating a conversation. Figure 3 , methods include:

[0143] Step 301: A computer device obtains a first dialogue corpus, where the first dialogue corpus includes multiple groups of dialogue information.

[0144] Among them, the first dialogue data is a dialogue data without annotation information; the first dialogue data is obtained from a dialogue data set, and the dialogue data set includes multiple dialogue data of the same business type as the first dialogue data, such as dialogue data for ordering air tickets. In the embodiment of the present application, no specific limitation is made to this.

[0145] Step 302: The computer device determines multiple slot values ​​in multiple groups of dialog information, a first label of each slot value, and dependency information of each slot value, wherein the first label is used to indicate the name of a first category to which the slot value belongs, and the dependency information is semantically associated with the slot value.

[0146] In one implementation, the slot value in the dialogue information is a named entity in the dialogue, and the first label is a named entity label; optionally, the computer device processes the first dialogue corpus using Spacy (a natural language processing tool) natural language processing tool to obtain multiple slot values ​​and the first label of each slot value. For example, for the sentence "I want to leave place A on the 13th and I will travel to place B", the computer device recognizes that the slot values ​​in the sentence are 13th, place A and place B, and their corresponding first labels are date, place and place respectively.

[0147] In one implementation, a computer device determines a first slot value and a second slot value in multiple groups of dialogue information to obtain multiple slot values ​​in the multiple groups of dialogue information, wherein the first slot value is a necessary slot value for marking a first dialogue corpus, and the second slot value is used to assist in marking the first dialogue corpus.

[0148] It should be noted that, since the dialogue information may contain content other than the necessary first slot value that the user is concerned about, by determining the optional auxiliary slot values ​​provided in the dialogue information, the slot value information annotated in the dialogue corpus can be enriched. Since the more annotated slot value information is, the more conducive it is to model training, the efficiency of model training based on dialogue corpus annotated with multiple slot values ​​can be improved. Figure 4 ,by Figure 4 Take the first dialogue data in the example, Figure 4 On the right are the types of multiple slot values ​​in the multiple groups of dialogue information in the first dialogue corpus marked. The destination, airline, departure place, departure date, and number of passengers in the solid-line box are the necessary types of the first slot value, and the airport, departure time, and refund in the dotted-line box are the optional types of the second slot value.

[0149] In some embodiments, the computer device obtains the shortest target number of dialogue corpus in the dialogue corpus set for named entity recognition, and determines the intersection of slot value types appearing in the multiple groups of dialogue corpus as the necessary slot value type for such dialogue; conversely, the union of slot value types appearing in the multiple groups of dialogue corpus minus the type of the first slot value is the type of the optional second slot value for such dialogue. Since the shortest dialogue generally completes the dialogue task in the most concise way, and the shortest dialogue only contains the necessary first slot value, the type of the first slot value and the type of the second slot value determined in the above manner are highly reliable.

[0150] Optionally, the dependency information is a dependency word. In the embodiments of the present application, the dependency information is a dependency word as an example for explanation. It should be noted that due to the large granularity of named entity recognition, many slot values ​​need to be further distinguished at the named entity level. For example, in the dialogue task of ordering air tickets, the named entity recognizes the place name and uniformly marks it as a place name. However, in the actual dialogue, the place names in this dialogue task should be further distinguished into departure and destination, and are generally distinguished by words that depend on the slot value; therefore, optionally, the computer device uses the Spacy natural language processing tool to perform dependency analysis on the sentence where the slot value is located, and obtains each dependency word that depends on the slot value, and then the slot value can be further distinguished in detail based on the dependency word. It should be noted that some special slot values ​​cannot be accurately distinguished by only one dependency analysis, such as slot values ​​such as time and date, so the computer device performs two dependency analyses on this type of slot value. For example, see Figure 5, some dependent words corresponding to slot values are prepositions such as "at" and "of", which have no practical meaning. Therefore, these slot values need to be dependency parsed twice. For example, for the first dependent word of "No. 8" in the figure is "at", and "at" has no practical meaning, so it is necessary to perform dependency parsing on "at" again to obtain the final dependent word "leave" of "No. 8". The dependent word obtained the second time has practical meaning and can be used to distinguish slot values. In this way, highly reliable dependent words of slot values can be obtained through multiple dependency parsings, ensuring the accuracy of the obtained dependent words.

[0151] Step 303: The computer device obtains a first confidence level, which is the average of the confidence levels of the first labels of multiple slot values included in multiple dialogue corpora in the dialogue corpus set. The dialogue corpus set includes a first dialogue corpus.

[0152] In one implementation, each slot value corresponds to an entropy for its first label respectively. The computer device obtains the confidence level of each slot value based on the entropy of each slot value, and then obtains the average first confidence level of multiple slot values. In another implementation, the first confidence level is obtained from the average entropy of multiple entropies of each slot value belonging to its corresponding first label; the smaller the average entropy, the higher the first confidence level, indicating the higher the reliability of the first label. Optionally, after the computer device determines multiple slot values and the first label of each slot value in the dialogue corpus set, it determines the first confidence level, which is convenient for directly obtaining and using the first confidence level subsequently.

[0153] Step 304: The computer device determines the second labels of multiple slot values based on the first confidence level, the first labels of multiple slot values, and the dependency information of multiple slot values.

[0154] Among them, the second label is used to represent the name of the second category to which the slot value belongs, that is, the slot name. Step 304 includes the following two cases:

[0155] The first case: When the first confidence level is greater than the first preset value, the computer device classifies multiple slot values based on the first labels and dependency information of multiple slot values to obtain multiple categories of slot values; for each slot value in each category of slot values, the computer device combines the dependency information and the first label of the slot value to obtain the second label of the slot value.

[0156] Among them, the size of the first preset value can be set and changed as needed, and this is not specifically limited in the present application. It should be noted that, since the first confidence level can be used to represent the reliability of the first label, the second label of the slot value determined based on the first confidence level can obtain an accurate second label on the basis of ensuring the accuracy of the first label. And when the first confidence level is greater than the first preset value, it means that the first label is reliable, and since the first label has a certain generalization for the slot value, in an embodiment of the present application, the second label is obtained by combining the dependent word and the first label, so that the obtained second label is accurate and reliable. Optionally, for a slot value of the departure place, the computer device combines the dependent word and the first label to determine that the second label of the slot value is the departure place, and for a slot value of the destination, the computer device combines the dependent word and the first label to determine that the second label of the slot value is the destination.

[0157] The computer device classifies the multiple slot values ​​based on the first labels and dependency information of the multiple slot values ​​to obtain multiple types of slot values, including the following implementation methods:

[0158] (1) The computer device obtains multiple slot values ​​in the dialogue corpus set, and for multiple slot values ​​corresponding to the same first label, when the number of identical dependency information corresponding to the multiple slot values ​​is greater than a preset threshold, the multiple slot values ​​are classified into one category.

[0159] Optionally, the computer device determines a preset threshold based on the number of slot values ​​of the same type corresponding to the same first label; if the preset threshold is 60% of the number of slot values ​​of this type, in the embodiment of the present application, no specific limitation is made to this. It should be noted that when the number of multiple dependent words corresponding to multiple slot values ​​of the same first label exceeds the preset threshold, it means that the slot values ​​of this type do not need to be further subdivided, and the multiple slot values ​​are classified into one category, which saves time and effort, thereby improving the efficiency of classifying the slot values ​​of this type.

[0160] (2) The computer device obtains multiple slot values ​​in the dialogue corpus set, and for multiple slot values ​​corresponding to the same first label, classifies the multiple slot values ​​based on the dependency information of each slot value when the number of identical dependency information corresponding to the multiple slot values ​​is not greater than a preset threshold; wherein the dependency information corresponding to the multiple slot values ​​in each category of slot values ​​is the same.

[0161] It should be noted that this implementation corresponds to slot values ​​that are easier to distinguish, such as locations. For example, in the business of booking air tickets, the slot values ​​of the location slot value with the dependent word "from" are divided into one category, and the slot values ​​with the dependent word "to" are divided into another category, which are respectively classified into the departure category and the destination category.

[0162] (3) The computer device obtains multiple slot values ​​in the dialogue corpus. For multiple slot values ​​corresponding to the same first label, when the number of identical dependency information corresponding to the multiple slot values ​​is not greater than a preset threshold, the computer device determines the similarity between the dependency information of the multiple slot values. The computer device classifies the multiple slot values ​​based on the similarity. The similarity between the dependency information corresponding to the multiple slot values ​​in each category of slot values ​​is greater than a preset similarity. The categories of slot values ​​corresponding to the first label and the similarity in the multiple groups of dialogue information are obtained.

[0163] It should be noted that this implementation corresponds to special slot values, such as the case where the dependent words of the slot value of time or place are multiple verbs. In one implementation, the computer device inputs all dependent words into the pre-trained BERT (Bidirectional Encoder Representation from Transformers) model, and outputs the similarity between the dependent words; optionally, the similarity is cosine similarity. For example, for the time slot value, the dependent words are leave, depart, departing, back, return, etc., which are difficult to distinguish. The computer device calculates based on the similarity and divides the dependent words into two categories, leave, depart, departing, etc. into one category, and back, return, etc. into one category, corresponding to the departure time category and the return time category, respectively. In this way, through this implementation, the slot value can be divided into a more fine-grained manner.

[0164] The second situation: when the first confidence is not greater than the first preset value, the computer device obtains multiple slot values ​​in the dialogue corpus set, and clusters the multiple slot values ​​based on the dependency information of the multiple slot values ​​to obtain multiple categories of slot values; the computer device obtains a second confidence for each category of slot values, and the second confidence is the average confidence of the multiple slot values ​​in each category of slot values ​​that belong to their respective categories; the computer device determines second labels for the multiple slot values ​​in each category of slot values ​​based on the second confidence for each category of slot values, and obtains the second label for each slot value corresponding to each category of slot values ​​in multiple groups of dialogue information.

[0165] It should be noted that when the first confidence is not greater than the first preset value, it means that the first label is unreliable. Therefore, in this case, the slot values ​​are no longer divided based on the first label, but the slot values ​​are clustered based on the dependency information, so as to ensure the accuracy of clustering. Optionally, the computer device obtains the second confidence based on the cluster entropy of each type of slot value. The smaller the cluster entropy, the higher the second confidence, indicating that the accuracy of the slot value of this type belongs to its category is higher. And in the embodiment of the present application, since the second confidence is the average of the confidences of multiple slot values ​​belonging to their categories, determining the second label of the slot value based on the second confidence can ensure that the second label is obtained on the basis of accurate clustering of the slot value, thereby improving the accuracy of the obtained second label.

[0166] The computer device determines the second labels of the plurality of slot values ​​in each type of slot value based on the second confidence level of each type of slot value, including the following two implementation methods:

[0167] In one implementation, for each type of slot value, when the second confidence is greater than the second preset value, the computer device combines the dependency information of the slot value and the first label for each slot value therein to obtain a second label for the slot value. In this implementation, since the second confidence is greater than the second preset value, it means that the clustering result is reliable, and then for each type of slot value, combining the dependency information and the first label can obtain a second label with high accuracy.

[0168] In another implementation, for each type of slot value, when the second confidence is not greater than the second preset value, the computer device determines, for multiple slot values ​​therein, the target sentences associated with the sentence to which each slot value belongs, determines the target words with the highest frequency in the target sentences associated with the multiple slot values, combines the dependency information and the target words of the slot value, and obtains second labels for the multiple slot values.

[0169] Optionally, the target sentence associated with the sentence to which each slot value belongs is the question sentence of the previous sentence; wherein, since the generalized words generally appear in the question sentence of the previous sentence of the sentence where the slot value is located, the reliability of using the question sentence of the previous sentence as the target sentence is high, and then the noun adverbs in the question sentence of the previous sentence of the sentence where the slot value is located are used as the words of the second label in the subsequent process, so that the accuracy of the obtained second label is high. For example, for the slot value "A place" in the sentence "It is A place", the question sentence of the previous sentence associated with the sentence where it is located is "Where are you from", and the word frequency of "where" in the sentence is the highest, then "where" can be used as the target word, and then combined with the dependent word "from" of the slot value, the second label of A place is confirmed as "where you come from", that is, the departure place. In this implementation, when the second confidence is not greater than the second preset value, it means that the category obtained by clustering is unreliable and the first label is no longer reliable; since the generalized words for the slot value generally appear in the target sentence associated with the slot value, in this implementation, the words associated with the slot value in the associated target sentence are combined with the dependent words as the second label, thereby improving the accuracy of the determined second label.

[0170] In one implementation, a computer device clusters multiple slot values ​​based on dependency information of the multiple slot values, including: the computer device respectively determines first semantic information of each slot value and second semantic information of dependency information of each slot value; for each slot value, the computer device performs weighted summation of the first semantic information and the second semantic information to obtain target semantic information of the slot value; the computer device clusters the multiple slot values ​​based on the target semantic information respectively corresponding to the multiple slot values.

[0171] Optionally, the computer device determines the first semantic information of the slot value and the second semantic information of the dependency information based on the BERT model, and the first semantic information and the second semantic information are both semantic vectors; the computer device weights and sums the first semantic vector and the second semantic vector to obtain a target semantic vector, and then the computer device clusters the slot values ​​based on the target semantic vector; optionally, the computer device clusters based on the similarity between multiple target semantic vectors. Optionally, the computer device determines the number of clusters based on the BIC (Bayesian Information Criterions)-AIC (Akaikeinformation criterion) indicator, which is not specifically limited in the embodiments of the present application.

[0172] It should be noted that, when the first confidence is not greater than the first preset value, it means that the first label is unreliable, and in this case, the slot values ​​are no longer distinguished based on the first label, but are classified based on the dependency information, thereby improving the accuracy of classification. And in the embodiment of the present application, since the second confidence is the average of the confidences of multiple slot values ​​belonging to their categories, the second label of the slot value is determined based on the second confidence, which can ensure that the second label is obtained on the basis of accurate clustering of the slot values, thereby improving the accuracy of the obtained second label.

[0173] Step 305: The computer device labels the first dialogue corpus based on the multiple slot values ​​and the second labels of the multiple slot values ​​to obtain a second dialogue corpus, and the second dialogue corpus is used to train the dialogue model.

[0174] In one implementation, a computer device determines a dialogue task of a first dialogue material from a dialogue mapping relationship based on multiple slot values ​​and second labels of the multiple slot values, where the dialogue mapping relationship is a mapping relationship between multiple slot values, second labels of the multiple slot values, and dialogue tasks; the computer device labels the first dialogue material based on the multiple slot values ​​of the first dialogue material, the second label of each slot value, and the dialogue task.

[0175] Optionally, the computer device determines the dialogue task based on the set of multiple first slot values ​​and the set of multiple second labels in each group of dialogue data, that is, the dialogue task is determined by the necessary first slot values ​​involved, and the task can be defined only when the dialogue data includes all the necessary first slot values ​​that can define the task for the dialogue data. In this implementation, by further annotating the dialogue tasks of the dialogue data, the annotation of the dialogue data is made more detailed and accurate, thereby improving the efficiency of subsequent model training based on the dialogue data.

[0176] In one implementation, a computer device determines action information of a first dialogue corpus; the computer device annotates a second dialogue corpus based on the action information to obtain a third dialogue corpus, and the third dialogue corpus is used to train a dialogue model. In this implementation, by further annotating the action information of the dialogue corpus, the annotation of the dialogue corpus is made more detailed and accurate, thereby improving the efficiency of subsequent model training based on the dialogue corpus.

[0177] In some embodiments, the first dialogue corpus is a dialogue corpus without action information labeled. Then, the computer device determines the action information of the first dialogue corpus in the following manner: the computer device performs model training based on the dialogue corpus with action information labeled to obtain an action recognition model, and the action recognition model is used to recognize action information in the dialogue corpus. The computer device inputs the first dialogue corpus into the action recognition model and outputs the action information of the first dialogue corpus.

[0178] Optionally, the dialogue corpus with annotated action information is a dialogue corpus in a dialogue corpus set that includes the first dialogue corpus. Optionally, the action recognition model is a FastText (fast text classifier) ​​classification model, which is not specifically limited in the present application. In an embodiment of the present application, an action recognition model is obtained by training based on dialogue corpus with annotated action information, and then the first dialogue corpus without annotated action information in the dialogue corpus set can be identified based on the action recognition model, thereby improving the efficiency of determining the action information of the first dialogue corpus, and then improving the efficiency of subsequent annotation of the first dialogue corpus; for example, see Figure 6 , Figure 6 For dialogue materials marked with action information, the information before the # sign represents the action information.

[0179] In one implementation, a computer device obtains multiple Figure 7 The second dialogue corpus with annotated information is shown, and then based on the dialogue structures of a large number of such dialogue corpus, the transition probability between such dialogue structures is determined, such as Bigram probability; see Figure 8 , Figure 8 The transition probability diagram between the first slot value types necessary for ordering air tickets obtained by the computer device can clearly understand the dialogue structure and the transition relationship between them, which can play a certain auxiliary role in building dialogue strategies and drawing dialogue business diagrams, thereby improving the efficiency of building dialogue strategies and drawing dialogue business diagrams.

[0180] See also Fig. 9 , Fig. 9 The overall flow chart of the dialogue annotation method provided in the embodiment of the present application. The computer device first determines the slot value, the first label and the dependent word, and also distinguishes the necessary first slot value and the optional second slot value for the slot value. Then the computer device determines the first confidence, analyzes the dependent word when the first confidence is large, classifies the slot value using different methods based on the number of dependent words in the same slot value, and then combines the dependent word and the first label to determine the second label of the slot value. When the first confidence is small, the computer device clusters multiple slot values ​​to obtain the second confidence of each type of slot value. When the second confidence is large, the dependent word and the first label are combined to determine the second label of the slot value; when the second confidence is small, the high-frequency word and the dependent word are combined to determine the second label of the slot value. After the computer device obtains the second label, if the action information of the dialogue corpus is not obtained, a dialogue corpus annotated with the slot value and the second label is obtained; if the action information of the dialogue corpus is obtained, a dialogue corpus annotated with the slot value, the second label and the action information is obtained, and the second label is the slot name; see Fig.10 , Fig.10This is a dialogue corpus marked with action information obtained in the embodiment of the present application. The inquiries, answers, confirmations, etc. in the figure are all action information.

[0181] In some embodiments, in the single domain of sgd (Schema-Guided Dialogue dataset), the average clustering accuracy of 29 tasks in the data is 64.35%; in some papers, the accuracy of the slot value clustering algorithm on four slot value types is 70%, and the method provided by the embodiments of the present application has an average clustering accuracy of more than 70% in 13 of the 29 tasks of the sgd dataset. It can be seen that the clustering accuracy of the method provided by the present application is high.

[0182] In some embodiments, precision and recall are used to evaluate the method provided in the embodiments of the present application. Precision refers to the proportion of the second label named correctly in the identified slot values. Recall refers to the proportion of slot values ​​mined and correctly named from the original corpus to the actual total slot values, where manual evaluation is used to determine whether the naming is correct. For example, referring to the evaluation results on the dialog task of ordering a one-way flight ticket shown in Table 1, it can be seen that both precision and recall are high, indicating that the accuracy and reliability of annotating the dialog corpus by the method provided in the embodiments of the present application are high.

[0183] Table 1

[0184] Second label Accuracy Recall Departure 97％ 76％ destination 97％ 78％ Departure Date 90％ 58％ airline 100％ 88％ Number of votes 99％ 69％ average value 97％ 75％

[0185] In some embodiments, the method provided in the embodiments of the present application is applied to intelligent customer service. The specific application scenario is to automatically analyze and process a large amount of dialogue corpus without labeled information, generate second labels for slot values ​​and the transition probabilities between them, help build and improve the intelligent customer service system, assist in understanding dialogue business, and save manpower and financial resources.

[0186] It should be noted that the solution provided in the embodiment of the present application can not only parse the dialogue features, but also classify the dialogue features and determine the second label to obtain the slot name, which can be used to build and improve the human-computer dialogue system; and this solution uses different methods to process dialogue materials in different fields and of different difficulty levels; this solution not only focuses on the dialogue features themselves, but also conducts a quantitative analysis of the transition probabilities between features; when providing annotation information, this solution can also mine and analyze dialogue action information. It can be seen that the method provided in the embodiment of the present application can improve the richness and accuracy of the annotation information, avoid manual annotation, and further improve the efficiency of annotating dialogue materials.

[0187] An embodiment of the present application provides a method for annotating dialogues. This method can not only obtain slot values ​​in dialogue corpus, but also divide the slot values ​​more finely based on dependency information of the slot values ​​and a first label of a first category to which the slot values ​​belong, thereby obtaining a second label of a second category to which the slot values ​​belong. The dialogue corpus can then be annotated based on the slot values ​​and the second labels of the slot values, thereby making the annotation information of the dialogue corpus more detailed and rich, thereby improving the efficiency of model training based on the annotated dialogue corpus.

[0188] The present application also provides a dialogue annotation device, see Fig.11 , the device comprises:

[0189] An acquisition module 1101 is used to acquire a first dialogue corpus, where the first dialogue corpus includes multiple groups of dialogue information;

[0190] A first determination module 1102, configured to determine multiple slot values ​​in multiple groups of dialog information, a first label of each slot value, and dependency information of each slot value, wherein the first label is used to indicate the name of the first category to which the slot value belongs, and the dependency information is semantically associated with the slot value;

[0191] A second determining module 1103, configured to determine second labels of the plurality of slot values ​​based on the first labels and dependency information of the plurality of slot values, wherein the second labels are used to indicate the name of a second category to which the slot values ​​belong;

[0192] The first labeling module 1104 is used to label the first dialogue corpus based on the multiple slot values ​​and the second labels of the multiple slot values ​​to obtain the second dialogue corpus, and the second dialogue corpus is used to train the dialogue model.

[0193] In one implementation, the second determining module 1103 includes:

[0194] An acquisition unit, configured to acquire a first confidence, where the first confidence is an average of confidences of first labels of multiple slot values ​​included in multiple dialogue corpora in a dialogue corpus set, where the dialogue corpus set includes the first dialogue corpus;

[0195] The determining unit is used to determine the second labels of the plurality of slot values ​​based on the first confidence, the first labels of the plurality of slot values ​​and the dependency information of the plurality of slot values.

[0196] In one implementation, the determining unit includes:

[0197] A classification subunit, configured to classify the multiple slot values ​​based on the first labels and dependency information of the multiple slot values ​​to obtain multiple categories of slot values ​​when the first confidence is greater than a first preset value;

[0198] The combining subunit is used to combine the dependency information of the slot value and the first label for each slot value in each type of slot value to obtain a second label of the slot value.

[0199] In one implementation, the classification subunit is used to:

[0200] Acquire multiple slot values ​​in the dialogue corpus set, and for multiple slot values ​​corresponding to the same first label, classify the multiple slot values ​​into one category when the number of same dependency information corresponding to the multiple slot values ​​is greater than a preset threshold;

[0201] Acquire multiple slot values ​​in the dialogue corpus set, and for multiple slot values ​​corresponding to the same first label, classify the multiple slot values ​​based on the dependency information of each slot value when the number of identical dependency information corresponding to the multiple slot values ​​is not greater than a preset threshold, wherein the dependency information corresponding to the multiple slot values ​​in each category of slot values ​​is identical;

[0202] Obtain multiple slot values ​​in a dialogue corpus set, and for multiple slot values ​​corresponding to the same first label, determine the similarity between the dependency information of the multiple slot values ​​when the number of identical dependency information corresponding to the multiple slot values ​​is not greater than a preset threshold, and classify the multiple slot values ​​based on the similarity, wherein the similarity between the dependency information corresponding to the multiple slot values ​​in each category of slot values ​​is greater than a preset similarity.

[0203] In one implementation, the determining unit includes:

[0204] A clustering subunit, configured to obtain, when the first confidence level is not greater than a first preset value, a plurality of slot values ​​in the dialogue corpus, and cluster the plurality of slot values ​​based on dependency information of the plurality of slot values ​​to obtain a plurality of categories of slot values;

[0205] An acquisition subunit is used to acquire a second confidence level of each type of slot value, where the second confidence level is an average value of confidence levels of multiple slot values ​​in each type of slot value that belong to the type to which they belong;

[0206] The determination subunit is used to determine the second labels of multiple slot values ​​in each category of slot values ​​based on the second confidence of each category of slot values, and obtain the second label of each slot value corresponding to each category of slot values ​​in multiple groups of dialogue information.

[0207] In one implementation, determining a subunit includes:

[0208] For each type of slot value, when the second confidence is greater than a second preset value, for each slot value therein, combining the dependency information of the slot value and the first label to obtain a second label of the slot value;

[0209] For each type of slot value, when the second confidence is not greater than the second preset value, for multiple slot values ​​therein, determine the target sentences associated with the sentence to which each slot value belongs, determine the target words with the highest frequency in the target sentences associated with the multiple slot values, combine the dependency information and the target words of the slot values, and obtain second labels for the multiple slot values.

[0210] In one implementation, the clustering subunit includes:

[0211] Determine first semantic information of each slot value and second semantic information of dependency information of each slot value respectively;

[0212] For each slot value, performing a weighted summation on the first semantic information and the second semantic information to obtain the target semantic information of the slot value;

[0213] Based on the target semantic information respectively corresponding to the multiple slot values, the multiple slot values ​​are clustered.

[0214] In one implementation, the first labeling module 1104 includes:

[0215] Based on the multiple slot values ​​and the multiple second labels of the slot values, determining a dialog task of the first dialog corpus from a dialog mapping relationship, where the dialog mapping relationship is a mapping relationship between the multiple slot values, the multiple second labels of the slot values, and the dialog task;

[0216] The first dialogue corpus is labeled based on multiple slot values ​​of the first dialogue corpus, a second label of each slot value, and a dialogue task.

[0217] In one implementation, the apparatus further includes:

[0218] A third determining module, used to determine the action information of the first dialogue corpus;

[0219] The second annotation module is used to annotate the second dialogue corpus based on the action information to obtain a third dialogue corpus, and the third dialogue corpus is used to train the dialogue model.

[0220] In one implementation, the first dialogue corpus is a dialogue corpus without action information labeled;

[0221] The third determination module is used to:

[0222] Based on the dialogue corpus with annotated action information, a model is trained to obtain an action recognition model, which is used to recognize action information in the dialogue corpus;

[0223] The first dialogue corpus is input into the action recognition model, and the action information of the first dialogue corpus is output.

[0224] In one implementation, the first determining module 1102 is configured to:

[0225] Determine the first slot value and the second slot value in the multiple groups of dialogue information to obtain multiple slot values ​​in the multiple groups of dialogue information, wherein the first slot value is a necessary slot value for marking the first dialogue corpus, and the second slot value is used to assist in marking the first dialogue corpus.

[0226] Fig.12 The structure block diagram of a terminal 1200 provided by an exemplary embodiment of the present application is shown. The terminal 1200 may be a portable mobile terminal, such as a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer or a desktop computer. The terminal 1200 may also be called a user device, a portable terminal, a laptop terminal, a desktop terminal or other names.

[0227] Typically, the terminal 1200 includes: a processor 1201 and a memory 1202 .

[0228] The processor 1201 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1201 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1201 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1201 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1201 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0229] The memory 1202 may include one or more computer-readable storage media, which may be non-transitory. The memory 1202 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1202 is used to store at least one program code, which is used to be executed by the processor 1201 to implement the conversation annotation method provided in the method embodiment of the present application.

[0230] In some embodiments, the terminal 1200 may also optionally include: a peripheral device interface 1203 and at least one peripheral device. The processor 1201, the memory 1202 and the peripheral device interface 1203 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 1203 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 1204, a display screen 1205, a camera assembly 1206, an audio circuit 1207, a positioning assembly 1208 and a power supply 1209.

[0231] The peripheral device interface 1203 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1201 and the memory 1202. In some embodiments, the processor 1201, the memory 1202, and the peripheral device interface 1203 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1201, the memory 1202, and the peripheral device interface 1203 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0232] The radio frequency circuit 1204 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1204 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1204 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 1204 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 1204 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1204 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0233] The display screen 1205 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 1205 is a touch display screen, the display screen 1205 also has the ability to collect touch signals on the surface or above the surface of the display screen 1205. The touch signal can be input to the processor 1201 as a control signal for processing. At this time, the display screen 1205 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 1205 can be one, set on the front panel of the terminal 1200; in other embodiments, the display screen 1205 can be at least two, respectively set on different surfaces of the terminal 1200 or in a folding design; in other embodiments, the display screen 1205 can be a flexible display screen, set on a curved surface or a folding surface of the terminal 1200. Even, the display screen 1205 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 1205 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0234] The camera assembly 1206 is used to capture images or videos. Optionally, the camera assembly 1206 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 1206 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0235] The audio circuit 1207 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 1201 for processing, or input them into the radio frequency circuit 1204 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 1200. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 1201 or the radio frequency circuit 1204 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 1207 may also include a headphone jack.

[0236] The positioning component 1208 is used to locate the current geographical location of the terminal 1200 to implement navigation or LBS (Location Based Service). The positioning component 1208 can be a positioning component based on the US GPS (Global Positioning System), China's Beidou system or Russia's Galileo system.

[0237] The power supply 1209 is used to power various components in the terminal 1200. The power supply 1209 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 1209 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery charged through a wired line, and a wireless rechargeable battery is a battery charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0238] In some embodiments, the terminal 1200 further includes one or more sensors 1210 , including but not limited to: an acceleration sensor 1211 , a gyroscope sensor 1212 , a pressure sensor 1213 , a fingerprint sensor 1214 , an optical sensor 1215 , and a proximity sensor 1216 .

[0239] The acceleration sensor 1211 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal 1200. For example, the acceleration sensor 1211 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 1201 can control the display screen 1205 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 1211. The acceleration sensor 1211 can also be used to collect motion data of games or users.

[0240] The gyro sensor 1212 can detect the body direction and rotation angle of the terminal 1200, and the gyro sensor 1212 can cooperate with the acceleration sensor 1211 to collect the user's 3D actions on the terminal 1200. The processor 1201 can implement the following functions based on the data collected by the gyro sensor 1212: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0241] The pressure sensor 1213 can be set on the side frame of the terminal 1200 and / or the lower layer of the display screen 1205. When the pressure sensor 1213 is set on the side frame of the terminal 1200, the user's holding signal of the terminal 1200 can be detected, and the processor 1201 performs left and right hand recognition or shortcut operations according to the holding signal collected by the pressure sensor 1213. When the pressure sensor 1213 is set on the lower layer of the display screen 1205, the processor 1201 controls the operability controls on the UI interface according to the user's pressure operation on the display screen 1205. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0242] The fingerprint sensor 1214 is used to collect the user's fingerprint, and the processor 1201 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 1214, or the fingerprint sensor 1214 identifies the user's identity based on the collected fingerprint. When the user's identity is identified as a trusted identity, the processor 1201 authorizes the user to perform relevant sensitive operations, including unlocking the screen, viewing encrypted information, downloading software, paying, and changing settings. The fingerprint sensor 1214 can be set on the front, back, or side of the terminal 1200. When a physical button or a manufacturer logo is set on the terminal 1200, the fingerprint sensor 1214 can be integrated with the physical button or the manufacturer logo.

[0243] The optical sensor 1215 is used to collect the ambient light intensity. In one embodiment, the processor 1201 can control the display brightness of the display screen 1205 according to the ambient light intensity collected by the optical sensor 1215. Specifically, when the ambient light intensity is high, the display brightness of the display screen 1205 is increased; when the ambient light intensity is low, the display brightness of the display screen 1205 is reduced. In another embodiment, the processor 1201 can also dynamically adjust the shooting parameters of the camera assembly 1206 according to the ambient light intensity collected by the optical sensor 1215.

[0244] The proximity sensor 1216, also called a distance sensor, is usually arranged on the front panel of the terminal 1200. The proximity sensor 1216 is used to collect the distance between the user and the front of the terminal 1200. In one embodiment, when the proximity sensor 1216 detects that the distance between the user and the front of the terminal 1200 is gradually decreasing, the processor 1201 controls the display screen 1205 to switch from the screen-on state to the screen-off state; when the proximity sensor 1216 detects that the distance between the user and the front of the terminal 1200 is gradually increasing, the processor 1201 controls the display screen 1205 to switch from the screen-off state to the screen-on state.

[0245] Those skilled in the art will understand that Fig.12 The structure shown in the figure does not constitute a limitation on the terminal 1200, and the terminal 1200 may include more or fewer components than those shown in the figure, or combine certain components, or adopt a different component arrangement.

[0246] Fig.13 It is a block diagram of a server provided by an embodiment of the present disclosure. The server 1300 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 1301 and one or more memories 1302, wherein the memory 1302 is used to store executable program codes, and the processor 1301 is configured to execute the above executable program codes to implement the conversation annotation methods provided by the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server may also include other components for implementing device functions, which will not be described in detail here.

[0247] In an exemplary embodiment, a storage medium including a program code is also provided, such as a memory 1302 including the program code, and the program code can be executed by the processor 1301 of the server 1300 to complete the above-mentioned conversation annotation method. Optionally, the storage medium can be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium can be a ROM (Read-Only Memory), a RAM (Random Access Memory), a CD-ROM (Compact Disc Read-Only Memory), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0248] An embodiment of the present application also provides a computer-readable storage medium, in which at least one program code is stored, and the at least one program code is loaded and executed by a processor to implement the conversation annotation method of any of the above implementation modes.

[0249] An embodiment of the present application also provides a computer program product, which includes a computer program code. The computer program code is stored in a computer-readable storage medium. A processor of a computer device reads the computer program code from the computer-readable storage medium, and the processor executes the computer program code, so that the computer device executes the conversation annotation method of any of the above-mentioned implementation methods.

[0250] In some embodiments, the computer program product involved in the embodiments of the present application may be deployed and executed on a computer device, or on multiple computer devices located at one location, or on multiple computer devices distributed at multiple locations and interconnected by a communication network. Multiple computer devices distributed at multiple locations and interconnected by a communication network may constitute a blockchain system.

[0251] An embodiment of the present application provides a method for annotating dialogues. This method can not only obtain slot values ​​in dialogue corpus, but also divide the slot values ​​more finely based on dependency information of the slot values ​​and a first label of a first category to which the slot values ​​belong, thereby obtaining a second label of a second category to which the slot values ​​belong. The dialogue corpus can then be annotated based on the slot values ​​and the second labels of the slot values, thereby making the annotation information of the dialogue corpus more detailed and rich, thereby improving the efficiency of model training based on the annotated dialogue corpus.

[0252] The above are only optional embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A conversation annotation method, characterized in that: The method comprises: Acquire a first dialogue corpus, wherein the first dialogue corpus includes multiple groups of dialogue information; Determine multiple slot values ​​in the multiple groups of dialog information, a first label of each slot value, and dependency information of each slot value, wherein the first label is used to indicate a name of a first category to which the slot value belongs, and the dependency information is semantically associated with the slot value; Determine, based on the first labels and dependency information of the plurality of slot values, second labels of the plurality of slot values, where the second labels are used to indicate the name of a second category to which the slot value belongs; Based on the multiple slot values ​​and the second labels of the multiple slot values, label the first dialogue corpus to obtain a second dialogue corpus, where the second dialogue corpus is used to train a dialogue model; The determining, based on the first labels and dependency information of the plurality of slot values, second labels of the plurality of slot values ​​comprises: Obtaining a first confidence, where the first confidence is an average of confidences of first labels of multiple slot values ​​included in multiple dialogue corpora in a dialogue corpus set, where the dialogue corpus set includes the first dialogue corpus; Determine second labels for the plurality of slot values ​​based on the first confidence, the first labels for the plurality of slot values, and dependency information for the plurality of slot values; The determining, based on the first confidence, the first labels of the plurality of slot values, and the dependency information of the plurality of slot values, second labels of the plurality of slot values ​​comprises: When the first confidence level is not greater than a first preset value, obtaining multiple slot values ​​in the dialogue corpus set, and clustering the multiple slot values ​​based on dependency information of the multiple slot values ​​to obtain multiple categories of slot values; Obtaining a second confidence level for each category of slot values, where the second confidence level is an average of confidence levels of a plurality of slot values ​​in each category of slot values ​​that belong to the category to which they belong; Based on the second confidence of each type of slot value, second labels of multiple slot values ​​in each type of slot value are determined, and second labels of each slot value corresponding to each type of slot value in the multiple groups of dialogue information are obtained.

2. The method according to claim 1, characterized in that: The determining, based on the first confidence, the first labels of the plurality of slot values, and the dependency information of the plurality of slot values, second labels of the plurality of slot values ​​comprises: When the first confidence level is greater than a first preset value, classifying the multiple slot values ​​based on the first labels and dependency information of the multiple slot values ​​to obtain multiple categories of slot values; For each slot value in each type of slot value, the dependency information of the slot value and the first label are combined to obtain a second label of the slot value.

3. The method according to claim 2, characterized in that The classifying the multiple slot values ​​based on the first labels and dependency information of the multiple slot values ​​to obtain multiple types of slot values ​​includes at least one of the following: Acquire multiple slot values ​​in the dialogue corpus set, and for multiple slot values ​​corresponding to the same first label, classify the multiple slot values ​​into one category when the number of same dependency information corresponding to the multiple slot values ​​is greater than a preset threshold; Acquire multiple slot values ​​in the dialogue corpus set, and for multiple slot values ​​corresponding to the same first label, classify the multiple slot values ​​based on the dependency information of each slot value when the number of identical dependency information corresponding to the multiple slot values ​​is not greater than a preset threshold, wherein the dependency information corresponding to multiple slot values ​​in each category of slot values ​​is identical; Obtain multiple slot values ​​in the dialogue corpus set, and for multiple slot values ​​corresponding to the same first label, determine the similarity between the dependency information of the multiple slot values ​​when the number of identical dependency information corresponding to the multiple slot values ​​is not greater than a preset threshold, and classify the multiple slot values ​​based on the similarity, wherein the similarity between the dependency information corresponding to the multiple slot values ​​in each category of slot values ​​is greater than a preset similarity.

4. The method according to claim 1, characterized in that The determining, based on the second confidence of each type of slot value, second labels of a plurality of slot values ​​in each type of slot value comprises: For each type of slot value, when the second confidence is greater than a second preset value, for each slot value therein, combining the dependency information of the slot value and the first label to obtain a second label of the slot value; For each type of slot value, when the second confidence is not greater than the second preset value, for multiple slot values ​​therein, determine the target sentences associated with the sentence to which each slot value belongs, determine the target words with the highest frequency in the target sentences associated with the multiple slot values, combine the dependency information of the slot value and the target words, and obtain second labels for the multiple slot values.

5. The method according to claim 1, characterized in that The clustering of the multiple slot values ​​based on the dependency information of the multiple slot values ​​includes: Determine first semantic information of each slot value and second semantic information of dependency information of each slot value respectively; For each slot value, a weighted sum is performed on the first semantic information and the second semantic information to obtain target semantic information of the slot value; and the multiple slot values ​​are clustered based on the target semantic information respectively corresponding to the multiple slot values.

6. The method according to claim 1, characterized in that The step of labeling the first dialogue corpus based on the multiple slot values ​​and the second labels of the multiple slot values ​​includes: Based on the multiple slot values ​​and the second labels of the multiple slot values, determining a dialog task of the first dialog corpus from a dialog mapping relationship, wherein the dialog mapping relationship is a mapping relationship between the multiple slot values, the second labels of the multiple slot values, and the dialog task; The first dialogue corpus is labeled based on a plurality of slot values ​​of the first dialogue corpus, a second label of each slot value, and the dialogue task.

7. The method according to claim 1, characterized in that The method further comprises: Determining action information of the first dialogue data; The second dialogue corpus is labeled based on the action information to obtain a third dialogue corpus, and the third dialogue corpus is used to train a dialogue model.

8. The method according to claim 1, characterized in that: The first dialogue corpus is a dialogue corpus without action information labeled; The determining the action information of the first dialogue data includes: Based on the dialogue corpus with marked action information, model training is performed to obtain an action recognition model, wherein the action recognition model is used to recognize action information in the dialogue corpus; The first dialogue corpus is input into the action recognition model, and action information of the first dialogue corpus is output.

9. The method according to claim 1, wherein determining a plurality of slot values ​​in the plurality of sets of dialog information comprises: Determine a first slot value and a second slot value in the multiple groups of dialogue information to obtain multiple slot values ​​in the multiple groups of dialogue information, wherein the first slot value is a necessary slot value for marking the first dialogue material, and the second slot value is used to assist in marking the first dialogue material.

10. A conversation annotation device, characterized in that: The device comprises: An acquisition module, configured to acquire a first dialogue corpus, wherein the first dialogue corpus includes multiple groups of dialogue information; A first determination module, configured to determine a plurality of slot values ​​in the plurality of groups of dialog information, a first label of each slot value, and dependency information of each slot value, wherein the first label is used to indicate a name of a first category to which the slot value belongs, and the dependency information is semantically associated with the slot value; A second determining module is configured to determine second labels of the plurality of slot values ​​based on the first labels and dependency information of the plurality of slot values, wherein the second labels are used to indicate a name of a second category to which the slot value belongs. The determining the second labels of the plurality of slot values ​​based on the first labels and dependency information of the plurality of slot values ​​includes: Obtaining a first confidence, where the first confidence is an average of confidences of first labels of multiple slot values ​​included in multiple dialogue corpora in a dialogue corpus set, where the dialogue corpus set includes the first dialogue corpus; Determine second labels for the plurality of slot values ​​based on the first confidence, the first labels for the plurality of slot values, and dependency information for the plurality of slot values; The determining, based on the first confidence, the first labels of the plurality of slot values, and the dependency information of the plurality of slot values, second labels of the plurality of slot values ​​comprises: When the first confidence level is not greater than a first preset value, obtaining multiple slot values ​​in the dialogue corpus set, and clustering the multiple slot values ​​based on dependency information of the multiple slot values ​​to obtain multiple categories of slot values; Obtaining a second confidence level for each category of slot values, where the second confidence level is an average of confidence levels of a plurality of slot values ​​in each category of slot values ​​that belong to the category to which they belong; Based on the second confidence of each type of slot value, determine the second labels of multiple slot values ​​in each type of slot value, and obtain the second label of each slot value corresponding to each type of slot value in the multiple groups of dialogue information; The first labeling module is used to label the first dialogue corpus based on the multiple slot values ​​and the second labels of the multiple slot values ​​to obtain a second dialogue corpus, where the second dialogue corpus is used to train the dialogue model.

11. A computer device, characterized in that: The computer device includes one or more processors and one or more memories, wherein at least one program code is stored in the one or more memories, and the at least one program code is loaded and executed by the one or more processors to implement the conversation annotation method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The storage medium stores at least one program code, and the at least one program code is loaded and executed by the processor to implement the conversation annotation method according to any one of claims 1 to 9.

13. A computer program product, characterized in that The computer program product includes a computer program code, which is stored in a computer-readable storage medium. A processor of a computer device reads the computer program code from the computer-readable storage medium, and the processor executes the computer program code, so that the computer device executes the conversation annotation method according to any one of claims 1 to 9.

Images