Corpus annotation method, device, computer storage medium and electronic device
The corpus annotation method uses semantic compression and proximity algorithms to automatically annotate unannotated corpora, addressing the inefficiencies of manual annotation and expanding training data, thereby improving the accuracy and stability of task-based dialogue systems.
Patent Information
- Application Number
- CN201910482496.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-06-04
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2039-08-01
AI Technical Summary
The manual annotation of corpora for task-based dialogue systems is labor-intensive, time-consuming, and limits the amount of available training data, hindering the efficiency and accuracy of these systems.
A corpus annotation method that utilizes semantic compression and proximity algorithms to automatically annotate intent and slot information in unannotated corpora, leveraging a semantic representation model to convert text into vectors and determine semantic similarity, thereby expanding the annotated corpus in a semi-supervised manner.
This approach reduces annotation costs and increases the amount of annotated data, improving the accuracy and stability of task-based dialogue systems, enhancing user experience.
Smart Images

Figure CN112036186B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a corpus annotation method, a corpus annotation device, a computer storage medium and an electronic device. Background Art
[0002] With the gradual development of artificial intelligence, it has gradually been applied to various fields of life, such as paying bills through facial recognition, identifying lesions through image recognition, navigating through voice recognition, playing games or chess and card games with machines, etc. It can be said that artificial intelligence has provided great convenience for people's lives.
[0003] In the field of dialogue, task-based dialogue is one of the main research focuses. The purpose of a task-based dialogue system is to help users achieve certain specific goals accurately and efficiently in the form of natural language. At present, the corpus used to train task-based dialogue systems is basically manually annotated, resulting in a very limited amount of corpus. In addition, annotating a large amount of corpus requires a lot of manpower, time and cost, which is very unfavorable for the training of task-based dialogue systems.
[0004] In view of this, there is an urgent need to develop a new corpus annotation method in this field.
[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present application, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention
[0006] The embodiments of the present invention provide a corpus annotation method, a corpus annotation device, a computer storage medium and an electronic device, which can improve the efficiency of corpus annotation, increase the amount of corpus, facilitate the training of dialogue systems, and improve the accuracy of dialogue systems, at least to a certain extent.
[0007] Other features and advantages of the present invention will become apparent from the following detailed description, or may be learned in part by practice of the present invention.
[0008] According to one aspect of an embodiment of the present invention, a corpus annotation method is provided, including: obtaining an annotated corpus and an unannotated corpus; semantically compressing the annotated corpus and the unannotated corpus respectively to obtain a first vector corresponding to the annotated corpus and a second vector corresponding to the unannotated corpus; obtaining intention information of the unannotated corpus according to the first vector and the second vector; obtaining an annotated slot value in the annotated corpus and a candidate slot value in the unannotated corpus, and semantically compressing the annotated slot value and the candidate slot value respectively to obtain a third vector corresponding to the annotated slot value and a fourth vector corresponding to the candidate slot value; and obtaining slot information of the unannotated corpus according to the third vector and the fourth vector.
[0009] According to one aspect of an embodiment of the present invention, a corpus annotation device is provided, including: a corpus acquisition module, used to acquire annotated corpus and unannotated corpus; a first vectorization module, used to semantically compress the annotated corpus and the unannotated corpus according to the semantic representation model, so as to acquire a first vector corresponding to the annotated corpus and a second vector corresponding to the unannotated corpus; an intention information acquisition module, used to acquire intention information of the unannotated corpus according to the first vector and the second vector; a second vectorization module, used to acquire annotated slot values in the annotated corpus and candidate slot values in the unannotated corpus, and semantically compress the annotated slot values and the candidate slot values, respectively, so as to acquire a third vector corresponding to the annotated slot value and a fourth vector corresponding to the candidate slot value; a slot information acquisition module, used to acquire slot information of the unannotated corpus according to the third vector and the fourth vector.
[0010] In some embodiments of the present invention, based on the aforementioned scheme, the corpus annotation device also includes: a training module, used to obtain unannotated corpus samples, and train the semantic representation model to be trained based on the unannotated corpus samples, so that the semantic representation model performs semantic compression on the annotated corpus and the unannotated corpus respectively.
[0011] In some embodiments of the present invention, based on the aforementioned scheme, the first vectorization module is configured to: input the annotated corpus and the unannotated corpus into the semantic representation model respectively, and map the words in the annotated corpus and the unannotated corpus into a vector space of a first preset dimension through the semantic representation model to obtain the first vector and the second vector.
[0012] In some embodiments of the present invention, based on the above-mentioned solution, the intention information acquisition module includes: a first calculation unit, used to obtain the intention information of the unlabeled corpus according to the first vector and the second vector and based on a proximity algorithm.
[0013] In some embodiments of the present invention, the number of the first vectors and the number of the second vectors are both multiple; based on the above scheme, the first calculation unit is configured to: calculate the distance between any second vector and each first vector; arrange the distances in order from small to large to form a sequence, and obtain a first preset number of distances in the sequence according to a first preset rule; determine the target annotated corpus based on the first preset number of distances, and obtain the target intent based on the target annotated corpus; count the main intention in the target intent, and use the main intention as the intent information of the unannotated corpus.
[0014] In some embodiments of the present invention, based on the above solution, the first calculation unit may also be configured to: compare the distance with a first preset threshold to obtain a target distance that is less than the first preset threshold.
[0015] In some embodiments of the present invention, based on the aforementioned scheme, the corpus annotation device also includes: a syntactic analysis module, used to perform syntactic analysis on the unlabeled corpus to obtain nouns and noun phrases in the unlabeled corpus, and use the nouns and / or the noun phrases as the candidate slot values.
[0016] In some embodiments of the present invention, based on the aforementioned scheme, the second vectorization module is configured to: input the marked slot value and the candidate slot value into the semantic representation model respectively, and map the marked slot value and the candidate slot value into a vector space of a second preset dimension through the semantic representation model to obtain the third vector and the fourth vector.
[0017] In some embodiments of the present invention, based on the above-mentioned solution, the slot information acquisition module includes: a second calculation unit, used to obtain the slot information of the unlabeled corpus according to the third vector and the fourth vector and based on a proximity algorithm.
[0018] In some embodiments of the present invention, the number of the marked slot values and the number of the candidate slot values are both multiple; based on the aforementioned scheme, the second calculation unit is configured to: calculate the distance between any of the fourth vectors and each of the third vectors; arrange the distances in order from small to large to form a sequence, and obtain a second preset number of distances in the sequence according to a second preset rule; determine the target marked slot value based on the second preset number of distances; count the main slot values in the target marked slot values, and obtain the target slot information corresponding to the main slot values, so as to use the target slot information as the slot information of the unlabeled corpus.
[0019] In some embodiments of the present invention, based on the above solution, the annotated corpus is manually annotated corpus, and the amount of the manually annotated corpus is less than the amount of the unannotated corpus.
[0020] According to one aspect of an embodiment of the present invention, a computer storage medium is provided, on which a computer program is stored, characterized in that when the computer program is executed by a processor, the corpus annotation method described in the above embodiment is implemented.
[0021] According to one aspect of an embodiment of the present invention, there is provided an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the corpus annotation method as described in the above embodiments.
[0022] In the technical solutions provided by some embodiments of the present invention, semantic compression is performed on the annotated corpus and the unannotated corpus respectively to obtain a first vector and a second vector, and then the intent information of the unannotated corpus is obtained based on the first vector and the second vector; then semantic compression is performed on the annotated slot values in the annotated corpus and the candidate slot values in the unannotated corpus respectively to obtain a third vector and a fourth vector, and then the slot information of the unannotated corpus is obtained based on the third vector and the fourth vector. On the one hand, the technical solution of the present invention can obtain the intent information and slot information of the unannotated corpus in a semi-supervised manner, avoiding manual annotation, reducing annotation costs, and improving annotation efficiency and the number of annotated corpora; on the other hand, by training the task-based dialogue system with a large amount of annotated corpus, the accuracy and stability of the task-based dialogue system can be improved, further enhancing the user experience.
[0023] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] The drawings herein are incorporated into and constitute a part of the specification, showing embodiments consistent with the present invention, and together with the specification, are used to explain the principles of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:
[0025] Figure 1 A schematic diagram showing an exemplary system architecture to which the technical solution of an embodiment of the present invention can be applied;
[0026] Figure 2 A schematic diagram of a process flow of a corpus annotation method according to an embodiment of the present invention is shown;
[0027] Figure 3 A schematic diagram of a process flow of intention annotation according to an embodiment of the present invention is shown;
[0028] Figure 4 A schematic diagram of a process of slot information labeling according to an embodiment of the present invention is shown;
[0029] Figure 5 A schematic diagram showing the structure of a traditional task-based dialogue system according to an embodiment of the present invention is shown;
[0030] Figure 6 A schematic diagram of the structure of an end-to-end task-based dialogue system according to an embodiment of the present invention is shown;
[0031] Figure 7 A block diagram schematically shows a corpus annotation device according to an embodiment of the present invention;
[0032] Figure 8 A block diagram schematically shows a corpus annotation device according to an embodiment of the present invention;
[0033] Fig. 9 A block diagram schematically shows a corpus annotation device according to an embodiment of the present invention;
[0034] Fig.10 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present invention is shown. DETAILED DESCRIPTION
[0035] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the present invention will be more comprehensive and complete and fully convey the concept of the example embodiments to those skilled in the art.
[0036] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present invention. However, those skilled in the art will appreciate that the technical solution of the present invention can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring various aspects of the present invention.
[0037] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.
[0038] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.
[0039] Figure 1 A schematic diagram showing an exemplary system architecture to which the technical solution of an embodiment of the present invention can be applied.
[0040] like Figure 1 As shown, the system architecture 100 may include a terminal device (such as Figure 1 The embodiment of the present invention is a schematic diagram of a mobile phone 101, a tablet computer 102, a portable computer 103, a desktop computer, etc., a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the terminal device and the server 105. The network 104 may include various connection types, such as a wired communication link, a wireless communication link, etc.
[0041] It should be understood that Figure 1 The number of terminals, networks and servers in the embodiment is only for illustration. Any number of terminals, networks and servers may be provided as required. For example, the server 105 may be a server cluster composed of multiple servers.
[0042] In one embodiment of the present invention, the terminal device 101, which may also be the terminal devices 102 and 103, after acquiring the annotated corpus and the unannotated corpus, sends them to the server 105 through the network 104. The server 105 may be provided with a machine learning model, which may specifically be a semantic representation model. The semantic representation model trained based on large-scale unannotated corpus can perform semantic compression on the received annotated corpus and the unannotated corpus, and obtain a first vector corresponding to the annotated corpus and a second vector corresponding to the unannotated corpus. According to the first vector and the second vector, the intention information of the unannotated corpus can be obtained; at the same time, the semantic representation model can also be used to perform semantic compression on the annotated slot values in the annotated corpus and the candidate slot values in the unannotated corpus, and obtain a third vector corresponding to the annotated slot value and a fourth vector corresponding to the candidate slot value. According to the third vector and the fourth vector, the slot information of the unannotated corpus can be obtained, thereby realizing the annotation of the unannotated corpus, which is convenient for training the dialogue system. On the one hand, the technical solution of the embodiment of the present invention can obtain the intent information and slot information of unlabeled corpus in a semi-supervised manner, avoiding manual labeling, reducing labeling costs, and improving labeling efficiency and the number of labeled corpora; on the other hand, by training the task-based dialogue system with a large amount of labeled corpus, the accuracy and stability of the task-based dialogue system can be improved, further improving the user experience.
[0043] It should be noted that the corpus annotation method provided in the embodiment of the present invention is generally executed by the server 105, and accordingly, the corpus annotation device is generally arranged in the server 105. However, in other embodiments of the present invention, the terminal may also have similar functions as the server, so as to execute the corpus annotation solution provided in the embodiment of the present invention.
[0044] In view of the problems existing in the related art, the embodiment of the present invention first proposes a corpus annotation method. The corpus annotation method in the embodiment of the present invention can annotate any corpus. The implementation details of the technical solution of the embodiment of the present invention are described in detail below:
[0045] Figure 2 A flowchart of a corpus annotation method according to an embodiment of the present invention is schematically shown. The corpus annotation method can be executed by a server, which can be Figure 1 Refer to the server shown in . Figure 2 As shown, the corpus annotation method at least includes steps S210 to S250, which are described in detail as follows:
[0046] In step S210, annotated corpus and unannotated corpus are obtained.
[0047] In one embodiment of the present invention, a large amount of dialogue corpus can be obtained through human-computer dialogue, a large amount of text can be obtained as corpus through the Internet, and of course, corpus can be obtained based on communication between people in real life, etc., which is not specifically limited in the embodiment of the present invention. The annotated corpus is manually annotated corpus. After obtaining a large amount of corpus, part of the corpus can be annotated by manual annotation, such as annotating the intention, slot type, slot value and other information of the sentence in the corpus. Since the dialogue system mainly performs natural language processing on the sentences input by the user, obtains the intention of the user, identifies the slots in the sentences, and determines the strategy based on the intention information and slot information, and then generates natural language based on the dialogue state, database query information and the determined strategy to conduct a dialogue with the user, in the embodiment of the present invention, only the intention and slot information of the unannotated corpus can be annotated.
[0048] In one embodiment of the present invention, the annotated corpus can be a directly acquired manually annotated corpus, for example, a corpus can be directly acquired as the annotated corpus, or an unannotated corpus can be acquired and annotated to form an annotated corpus. Furthermore, the annotated corpus and the unannotated corpus can be acquired simultaneously or separately, which is not specifically limited in the embodiment of the present invention.
[0049] In step S220, semantic compression is performed on the annotated corpus and the unannotated corpus respectively to obtain a first vector corresponding to the annotated corpus and a second vector corresponding to the unannotated corpus.
[0050] In one embodiment of the present invention, after obtaining the annotated corpus and the unannotated corpus, the annotated corpus and the unannotated corpus can be processed to vectorize the annotated corpus and the unannotated corpus. In the embodiment of the present invention, the annotated corpus and the unannotated corpus can be semantically compressed by a machine learning model. Specifically, the machine learning model can be a semantic representation model, which can convert the corpus from the form of natural language into word vectors, and then can annotate the unannotated corpus according to the vector corresponding to the corpus to obtain a large amount of dialogue training corpus.
[0051] In one embodiment of the present invention, before using a semantic representation model to process annotated corpus and unannotated corpus, the semantic representation model can be trained based on a large number of corpus samples. The semantic representation model can be word2vec or a pre-trained language processing model, such as an ELMo (Embeddings from Language Models) model or a BERT (Bidirectional Encoder Representations from Transformers). By inputting the corpus sample into the semantic representation model, the words in the corpus sample can be vectorized and the natural language can be converted into vectors. Since the word2vec model does not consider the order between words, the features obtained by this model are discrete and sparse. In the ELMo model, the representation of each word is a function of the entire input sentence. Specifically, a bidirectional LSTM model is first trained on a large corpus with the language model as the target, and then the LSTM is used to generate word representations. Therefore, the word representation determined by the ELMo model can consider the context of the text context, and the output result is more accurate. In addition, when processing a word, the BERT model can consider the information of the words before and after the word, so as to obtain the semantics of the context. Therefore, compared with the word2vec model, using the ELMo model or the BERT model can obtain more accurate word representations. For pre-trained language processing models, unlabeled corpus samples can be used to train them.
[0052] In one embodiment of the present invention, after completing the training of the semantic representation model, the annotated corpus and the unannotated corpus obtained in step S210 can be semantically compressed by the trained semantic representation model to vectorize the annotated corpus and the unannotated corpus. Specifically, the semantic representation model can be used to map the words in the annotated corpus and the unannotated corpus to a vector space of a first preset dimension to obtain a first vector corresponding to the annotated corpus and a second vector corresponding to the unannotated corpus. The first preset dimension is a parameter set when the semantic representation model is trained, for example, it can be set to 64 dimensions, 128 dimensions, etc., and the embodiment of the present invention does not specifically limit this.
[0053] In step S230, the intention information of the unlabeled corpus is obtained according to the first vector and the second vector.
[0054] In one embodiment of the present invention, the number of annotated corpora and the number of unannotated corpora are both multiple, and thus after converting the annotated corpora and the unannotated corpora from natural language forms into vectors, the number of first vectors and second vectors are also multiple. Furthermore, after obtaining the first vector and the second vector, the intent of the unannotated corpus can be annotated by calculating semantic similarity.
[0055] In one embodiment of the present invention, the semantic similarity can be calculated based on a proximity algorithm.
[0056] Figure 3 The flowchart of intent annotation is shown in Figure 1. Figure 3 As shown, the process diagram of the intention annotation mainly includes S301-S304, specifically:
[0057] In step S301, the distance between any second vector and each first vector is calculated;
[0058] In one embodiment of the present invention, when calculating semantic similarity, the comparison can be performed by calculating the distance between two vectors, and the distance can be any type of distance, such as Euclidean distance, cosine distance, Mahalanobis distance, etc., which is not specifically limited in the embodiment of the present invention. In addition, since the unlabeled corpus is labeled with intent, and the labeled corpus contains intent information, the distance between the second vector corresponding to each unlabeled corpus and the first vector corresponding to all labeled corpora can be calculated, and then the labeled corpus similar to the unlabeled corpus is determined based on the similarity, so that each unlabeled corpus is labeled with intent based on the intent information of the similar labeled corpus.
[0059] In step S302, the distances are arranged in ascending order to form a sequence, and a first preset number of distances in the sequence are obtained according to a first preset rule.
[0060] In one embodiment of the present invention, after obtaining the distance between each second vector and all first vectors, all distances corresponding to each second vector can be arranged in order from small to large to form a sequence, and a first preset number of distances in the sequence can be obtained according to a first preset rule. For distances obtained according to different distance algorithms, the first preset rule is different. For example, for cosine distance, when the cosine distance is closer to 1, it means that the angle between the two vectors is closer to 0°, that is, the two vectors are similar; when the cosine distance is closer to -1, it means that the angle between the two vectors is closer to 180°, that is, the two vectors are not similar. Therefore, for a sequence formed by sorting the cosine distance from small to large, the first preset rule can be to obtain the first preset number of distances in reverse order to obtain a first vector with a higher similarity to the second vector; for Euclidean distance or Mahalanobis distance, the values are not negative, so for a sequence formed by sorting the distance from small to large, the first preset rule can be to obtain a preset number of distances in positive order to obtain a first vector with a higher similarity to the second vector. It is worth noting that the first preset number can be a number set according to actual needs, for example, selecting the first 10 or 20 second vectors with high similarity to the second vector. The embodiment of the present invention does not specifically limit the specific number of the preset number.
[0061] In step S303, a target annotated corpus is determined according to the first preset number of distances, and a target intent is acquired according to the target annotated corpus.
[0062] In one embodiment of the present invention, after determining the first preset number of distances, the target annotated corpus corresponding to the first preset number of distances can be obtained, and the target intent corresponding to the target annotated corpus can be obtained at the same time. For example, the unannotated corpus is "What is the weather like in Beijing tomorrow?", by calculating the distance between the unannotated corpus and multiple annotated corpora, four target annotated corpora with high similarity can be obtained, including "What is the weather like in Beijing tomorrow?", "The weather tomorrow is cloudy.", "Will it rain in Beijing tomorrow?", "Which route to Beijing tomorrow?", etc., and then the target intent can be determined according to the target annotated corpus, which are "Weather query", "Weather query", "Weather query", "Route query" respectively.
[0063] In step S304, main intentions in the target intention are counted, and the main intentions are used as intention information of the unlabeled corpus.
[0064] In one embodiment of the present invention, after determining the target intent, all target intents can be counted to obtain the target intent with the largest proportion, which is the main intent among all target intents. Taking the example in step S303 as an example, "weather query" accounts for 75% and "route query" accounts for 25%, so "weather query" is the main intent. After determining the main intent, the main intent can be assigned to the unlabeled corpus as the intent information of the unlabeled corpus, that is, for the unlabeled corpus "What is the weather like in Beijing tomorrow?", the corresponding intent information is "weather query".
[0065] In one embodiment of the present invention, after obtaining the distance between any second vector and all first vectors, all distances may be compared with a first preset threshold to obtain a target distance less than the first preset threshold, and then the target distances are sorted from small to large to form a sequence, and a first preset number of distances are obtained from the sequence according to a first preset rule. By comparing all distances with the first preset threshold, the amount of sorted data may be reduced and data processing efficiency may be improved.
[0066] In step S204, the annotated slot value in the annotated corpus and the candidate slot value in the unannotated corpus are obtained, and semantic compression is performed on the annotated slot value and the candidate slot value respectively to obtain a third vector corresponding to the annotated slot value and a fourth vector corresponding to the candidate slot value.
[0067] In one embodiment of the present invention, after marking the intent information of the unlabeled corpus, the slot information of the unlabeled corpus also needs to be marked, and the slot information is the information category and information value related to the task in the sentence. In an embodiment of the present invention, the slot information of the unlabeled corpus can be marked based on the slot information in the labeled corpus. Before marking, candidate slot values can be obtained from the unlabeled corpus, and then the accurate slot value and the corresponding information category can be selected from the candidate slot values. Specifically, the unlabeled corpus can be subjected to syntactic analysis, and the parts of speech of the words in the unlabeled corpus can be marked to obtain nouns and noun phrases in the unlabeled corpus. The slot values are usually nouns and noun phrases in the sentence, so the obtained nouns and / or noun phrases can be used as candidate slot values for subsequent screening and marking.
[0068] In one embodiment of the present invention, the annotated slot values in the annotated corpus can be mapped to a vector space of a second preset dimension through a trained semantic representation model to obtain a third vector corresponding to the annotated slot values, and the candidate slot values of the unannotated corpus can be mapped to a vector space of a second preset dimension to obtain a fourth vector corresponding to the candidate slot values. Then, the similarity between the annotated slot values and the candidate slot values can be calculated based on a proximity algorithm, and then the slot values and slot types of the unannotated corpus can be labeled. It is worth noting that the second preset dimension is also a parameter that is set when the semantic representation model is trained, which can be the same as the first preset dimension or different, and the embodiment of the present invention does not specifically limit this.
[0069] In one embodiment of the present invention, the number of the marked slot value and the number of the candidate slot value are both multiple. Figure 4 A schematic diagram of the process of slot information annotation is shown, such as Figure 4 As shown, the process of slot information marking mainly includes steps S401-S404, specifically:
[0070] In step S401, the distance between any of the fourth vectors and each of the third vectors is calculated.
[0071] In one embodiment of the present invention, similar to step S301, the distance between the fourth vector corresponding to each candidate slot value and the third vector corresponding to all the annotated slot values can be calculated, and whether the candidate slot value is similar to the annotated slot value can be determined based on the distance. If the distance between the candidate slot value and some annotated slot values is small and is within a preset range, it can be considered that the candidate slot value is similar to some annotated slot values. Similarly, the distance calculated in step S401 can also be any one of the Euclidean distance, cosine distance, Mahalanobis distance, etc., and the distance type in step S401 can be the same as the distance type in step S301, or it can be different. The embodiment of the present invention does not specifically limit this.
[0072] In step S402, the distances are arranged in ascending order to form a sequence, and a second preset number of distances in the sequence are obtained according to a second preset rule.
[0073] In one embodiment of the present invention, after obtaining the distance between each fourth vector and all third vectors, all distances can be arranged in order from small to large to form a sequence, and a second preset number of distances in the sequence can be obtained according to a second preset rule. Similar to the description in step S302, for different distance types, the second preset rule is also different accordingly, but the second preset number of distances obtained are all distances that are relatively close to the fourth vector distance and are within a preset range. It is worth noting that the second preset number can be a value set according to actual needs, and the second preset number and the first preset number can be the same or different, and the embodiment of the present invention does not specifically limit this.
[0074] In step S403, the target marked slot value is determined according to the second preset number of distances.
[0075] In one embodiment of the present invention, after obtaining the second preset number of distances, the slot value of the annotated corpus corresponding to the second preset number of distances can be determined, and the slot value of the annotated corpus is the target annotated slot value. Taking the example in step S303 as an example, the nouns and noun phrases in the unannotated corpus, that is, the candidate slot values are: tomorrow, Beijing and weather, and the annotated slot values in the four annotated corpora with higher semantic similarity are: tomorrow, weather and Beijing, tomorrow, weather and cloudy, Beijing and tomorrow, tomorrow, route and Beijing. By calculating the distance between the candidate slot value and the annotated slot value, the target annotated slot value can be determined to be tomorrow, weather, Beijing, tomorrow, weather, cloudy, Beijing, tomorrow, tomorrow and Beijing.
[0076] In step S404, main slot values in the target annotated slot values are counted, and target slot information corresponding to the main slot values is obtained, so as to use the target slot information as slot information of the unannotated corpus.
[0077] In one embodiment of the present invention, after determining the target annotation slot value, all target annotation slot values can be counted to obtain the target annotation slot value whose proportion meets the preset conditions, and the target annotation slot value is the main slot value among all target annotation slot values. Continuing with the above example, "tomorrow" has the largest proportion, "Beijing" has the second largest proportion, "weather" has the third largest proportion, and "cloudy" has the fourth largest proportion. If the preset condition is that the top three proportions are the main slot values, then the main slot values can be determined to be tomorrow, Beijing, and weather, respectively. Further, the corresponding slot categories can be determined according to the main slot values, specifically time attributes, location attributes, and weather attributes. The slot information of the unlabeled corpus can be determined according to the determined main slot value and target slot information.
[0078] Similarly, after obtaining the distance between any fourth vector and all third vectors, all distances may be compared with the second preset threshold to obtain a target distance less than the second preset threshold, and then the target distances may be sorted from small to large to form a sequence, and a preset number of distances may be obtained from the sequence according to the second preset rule. By comparing all distances with the second preset threshold, the amount of sorted data may be reduced, and data processing efficiency may be improved.
[0079] The embodiment of the present invention can determine the intent information and slot information of unlabeled corpus through a semi-supervised method. Specifically, based on a small-scale manually labeled corpus, a large-scale unlabeled corpus is labeled using a semantic representation model and a similarity judgment method. Through this corpus labeling method, the dialogue training corpus can be expanded and the cost of obtaining the corpus can be reduced, thereby improving the performance of the existing dialogue system and making the dialogue system more robust and universal.
[0080] In one embodiment of the present invention, after obtaining a large amount of annotated corpus, the dialogue system can be trained using the annotated corpus. The dialogue system is mainly a task-based dialogue system. Currently, the task-based dialogue system is mainly divided into two types: traditional task-based dialogue systems and end-to-end task-based dialogue systems.
[0081] Figure 5 shows a schematic diagram of the structure of a traditional task-based dialogue system, such as Figure 5As shown, the traditional task-based dialogue system 500 includes a natural language understanding module 501, a dialogue state tracking module 502, a strategy learning module 503 and a natural language generation module 504, wherein the natural language understanding module 501 will perform intent recognition, that is, recognize the task that the current user wants to perform, such as weather query, route navigation, etc. For example, if the user inputs the sentence "Please tell me the weather in Los Angeles tomorrow", then the user's intention is weather query, and the slot information contained in the sentence input by the user has a time attribute, and the attribute value is tomorrow; a location attribute, and the attribute value is Los Angeles; and the required information is a weather attribute. The dialogue state tracking module 502 can track the status of multiple rounds of dialogues, and update the status of the current dialogue according to the analysis results of the natural language understanding module 501 on each sentence input by the user, including information provided by the system, information required by the system, information provided by the user, and information required by the user. The system will obtain the corresponding query results from the database according to the current information, and send the query results to the strategy learning module 503. The strategy learning module 503 can select a strategy according to the received information, such as asking for information, providing information, or query failure. The natural language generation module 504 generates input in natural language form according to the strategy selected by the strategy learning module 503, the current dialogue state and the database query results. For example, for a user input such as "Please tell me the weather tomorrow.", the dialogue strategy is to ask for information. In the current dialogue state, the information provided by the user includes a time attribute with an attribute value of tomorrow. The information required by the user is a weather attribute. If there are multiple matching results in the database query results, the system will generate a reply in the form of "Excuse me, where is the weather you are asking about?" to guide the user to provide more information to obtain accurate query results.
[0082] Figure 6 A schematic diagram showing the structure of an end-to-end task-based dialogue system is shown in FIG. Figure 6 As shown, the system is a system based on a neural network model. After receiving the natural language sentence input by the user, the system extracts features of the natural language sentence, interacts directly with the database through the attention mechanism, and finally outputs it through the neural network.
[0083] By inputting a large amount of annotated corpus obtained by the corpus annotation method according to an embodiment of the present invention into a traditional task-based dialogue system or an end-to-end task-based dialogue system for training, the processing efficiency and accuracy of the dialogue system can be improved, thereby improving the user experience.
[0084] In one embodiment of the present invention, the above embodiment describes that the corpus annotation method is performed by the server 105. Similarly, the corpus annotation method can also be performed by a terminal device, which can be Figure 1The terminal device 101 shown in the figure may also be the terminal device 102 or the terminal device 103. Accordingly, a semantic representation model is provided in the terminal device 101. After receiving the annotated corpus and the unannotated corpus, the terminal device 101 may respectively perform semantic compression on the annotated corpus and the unannotated corpus or the slot values therein to obtain corresponding vectors, and judge the semantic similarity according to the distance between the vectors. Furthermore, the intention information and slot information of the unannotated corpus may be determined according to the intention information and slot information of the semantically similar annotated corpus, thereby realizing semi-supervised corpus annotation of the unannotated corpus, avoiding the consumption of a large amount of manpower and cost for annotating a large amount of unannotated corpus, improving the quantity and rate of obtaining annotated corpus, and further improving the accuracy of the dialogue system.
[0085] The following describes an apparatus embodiment of the present invention, which can be used to execute the corpus annotation method in the above-mentioned embodiment of the present invention. For details not disclosed in the apparatus embodiment of the present invention, please refer to the above-mentioned embodiment of the corpus annotation method of the present invention.
[0086] Figure 7 The block diagram of a corpus annotation device according to an embodiment of the present invention is schematically shown.
[0087] Reference Figure 7 As shown, a corpus annotation device 700 according to an embodiment of the present invention includes: a corpus acquisition module 701, a first vectorization module 702, an intention information acquisition module 703, a second vectorization module 704 and a slot information acquisition module 705.
[0088] Among them, the corpus acquisition module 701 is used to acquire annotated corpus and unannotated corpus; the first vectorization module 702 is used to perform semantic compression on the annotated corpus and the unannotated corpus respectively to obtain a first vector corresponding to the annotated corpus and a second vector corresponding to the unannotated corpus; the intention information acquisition module 703 is used to acquire the intention information of the unannotated corpus according to the first vector and the second vector; the second vectorization module 704 is used to acquire the annotated slot value in the annotated corpus and the candidate slot value in the unannotated corpus, and perform semantic compression on the annotated slot value and the candidate slot value respectively to acquire a third vector corresponding to the annotated slot value and a fourth vector corresponding to the candidate slot value; the slot information acquisition module 705 is used to acquire the slot information of the unannotated corpus according to the third vector and the fourth vector.
[0089] Reference Figure 8As shown, according to an embodiment of the present invention, the corpus annotation device 700 also includes: a training module 706, which is used to obtain unannotated corpus samples, and train the semantic representation model to be trained through the unannotated corpus samples, so that the semantic representation model can semantically compress the annotated corpus and the unannotated corpus respectively.
[0090] In one embodiment of the present invention, the first vectorization module 702 is configured to: input the annotated corpus and the unannotated corpus into the semantic representation model respectively, and map the words in the annotated corpus and the unannotated corpus into a vector space of a first preset dimension through the semantic representation model to obtain the first vector and the second vector.
[0091] In one embodiment of the present invention, the intention information acquisition module 703 includes: a first calculation unit, configured to acquire the intention information of the unlabeled corpus according to the first vector and the second vector and based on a proximity algorithm.
[0092] In one embodiment of the present invention, the number of the first vectors and the number of the second vectors are both multiple; the first calculation unit is configured to: calculate the distance between any second vector and each first vector; arrange the distances in order from small to large to form a sequence, and obtain a first preset number of distances in the sequence according to a first preset rule; determine the target annotated corpus based on the first preset number of distances, and obtain the target intent based on the target annotated corpus; count the main intention in the target intent, and use the main intention as the intent information of the unannotated corpus.
[0093] In one embodiment of the present invention, the first calculation unit may also be configured to: compare the distance with a first preset threshold to obtain a target distance that is less than the first preset threshold.
[0094] Reference Fig. 9 As shown, according to an embodiment of the present invention, the corpus annotation device 700 further includes: a syntactic analysis module 707, which is used to perform syntactic analysis on the unlabeled corpus to obtain nouns and noun phrases in the unlabeled corpus, and use the nouns and / or the noun phrases as the candidate slot values.
[0095] In one embodiment of the present invention, the second vectorization module 704 is configured to: input the marked slot value and the candidate slot value into the semantic representation model respectively, and map the marked slot value and the candidate slot value into a vector space of a second preset dimension respectively through the semantic representation model to obtain the third vector and the fourth vector.
[0096] In one embodiment of the present invention, the slot information acquisition module 705 includes: a second calculation unit, configured to acquire the slot information of the unlabeled corpus according to the third vector and the fourth vector and based on a proximity algorithm.
[0097] In one embodiment of the present invention, the number of the marked slot values and the number of the candidate slot values are both multiple; the second calculation unit is configured to: calculate the distance between any of the fourth vectors and each of the third vectors; arrange the distances in order from small to large to form a sequence, and obtain a second preset number of distances in the sequence according to a second preset rule; determine the target marked slot value based on the second preset number of distances; count the main slot values in the target marked slot values, and obtain the target slot information corresponding to the main slot values, so as to use the target slot information as the slot information of the unlabeled corpus.
[0098] In one embodiment of the present invention, the second calculation unit may also be configured to: compare the distance with a second preset threshold to obtain a target distance that is less than the second preset threshold.
[0099] In one embodiment of the present invention, the annotated corpus is manually annotated corpus, and the amount of the manually annotated corpus is less than the amount of the unannotated corpus.
[0100] Fig.10 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present invention is shown.
[0101] It should be noted that Fig.10 The computer system 1000 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.
[0102] like Fig.10 As shown, the computer system 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1002 or the program loaded from the storage part 1008 to the random access memory (RAM) 1003. Various programs and data required for system operation are also stored in the RAM 1003. The CPU 1001, the ROM 1002 and the RAM 1003 are connected to each other through a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.
[0103] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed so that a computer program read therefrom is installed into the storage section 1008 as needed.
[0104] In particular, according to an embodiment of the present invention, the process described below with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part 1009, and / or installed from a removable medium 1011. When the computer program is executed by a central processing unit (CPU) 1001, various functions defined in the system of the present application are executed.
[0105] It should be noted that the computer-readable medium shown in the embodiment of the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.
[0106] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0107] The units involved in the embodiments of the present invention may be implemented by software or hardware, and the units described may also be arranged in a processor. The names of these units do not, in some cases, limit the units themselves.
[0108] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the above embodiment.
[0109] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.
[0110] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation of the present invention.
[0111] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art that are not disclosed in the present invention.
[0112] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.
Claims
1. A corpus annotation method, characterized in that: include: Obtain annotated and unannotated corpora; Semantically compressing the annotated corpus and the unannotated corpus respectively to obtain a first vector corresponding to the annotated corpus and a second vector corresponding to the unannotated corpus; Acquire intent information of the unlabeled corpus according to the first vector and the second vector; Obtaining a marked slot value in the marked corpus and a candidate slot value in the unmarked corpus, and performing semantic compression on the marked slot value and the candidate slot value respectively to obtain a third vector corresponding to the marked slot value and a fourth vector corresponding to the candidate slot value; Acquire slot information of the unlabeled corpus according to the third vector and the fourth vector; wherein the slot information of the unlabeled corpus is acquired according to the third vector and the fourth vector and based on a proximity algorithm; The number of the first vector and the number of the second vector are both plural, and obtaining the intention information of the unlabeled corpus according to the first vector and the second vector includes: Calculate the distance between any of the second vectors and each of the first vectors; Arrange the distances in ascending order to form a first sequence, and obtain a first preset number of distances in the first sequence according to a first preset rule; Determine a target annotated corpus according to the first preset number of distances, and obtain a target intent according to the target annotated corpus; The main intentions in the target intention are counted, and the main intentions are used as the intention information of the unlabeled corpus.
2. The corpus annotation method according to claim 1, characterized in that: Before semantically compressing the annotated corpus and the unannotated corpus respectively, the method further includes: An unlabeled corpus sample is obtained, and a semantic representation model to be trained is trained based on the unlabeled corpus sample, so that the semantic representation model performs semantic compression on the labeled corpus and the unlabeled corpus respectively.
3. The corpus annotation method according to claim 2, characterized in that: Semantically compressing the annotated corpus and the unannotated corpus respectively to obtain a first vector corresponding to the annotated corpus and a second vector corresponding to the unannotated corpus, including: The annotated corpus and the unannotated corpus are respectively input into the semantic representation model, and the words in the annotated corpus and the unannotated corpus are mapped into a vector space of a first preset dimension through the semantic representation model to obtain the first vector and the second vector.
4. The corpus annotation method according to claim 1, characterized in that: Before arranging the distances in ascending order to form a first sequence, and obtaining a first preset number of distances in the first sequence according to a first preset rule, the method further includes: The distance is compared with a first preset threshold to obtain a target distance that is less than the first preset threshold.
5. The corpus annotation method according to claim 1, characterized in that: Before obtaining the annotated slot value in the annotated corpus and the candidate slot value in the unannotated corpus, the method further includes: The unlabeled corpus is subjected to syntactic analysis to obtain nouns and noun phrases in the unlabeled corpus, and the nouns and / or the noun phrases are used as the candidate slot values.
6. The corpus annotation method according to claim 2, characterized in that: Obtaining a marked slot value in the marked corpus and a candidate slot value in the unmarked corpus, and semantically compressing the marked slot value and the candidate slot value respectively to obtain a third vector corresponding to the marked slot value and a fourth vector corresponding to the candidate slot value, including: The marked slot value and the candidate slot value are respectively input into the semantic representation model, and the marked slot value and the candidate slot value are respectively mapped into a vector space of a second preset dimension through the semantic representation model to obtain the third vector and the fourth vector.
7. The corpus annotation method according to claim 1, characterized in that: Acquiring slot information of the unlabeled corpus according to the third vector and the fourth vector includes: The slot information of the unlabeled corpus is obtained according to the third vector and the fourth vector and based on a proximity algorithm.
8. The corpus annotation method according to claim 7, characterized in that: The number of the marked slot value and the number of the candidate slot value are both multiple; Acquiring slot information of the unlabeled corpus according to the third vector and the fourth vector and based on a proximity algorithm includes: Calculating the distance between any of the fourth vectors and each of the third vectors; Arrange the distances between any of the fourth vectors and each of the third vectors in ascending order to form a second sequence, and obtain a second preset number of distances in the second sequence according to a second preset rule; Determine the target marking slot value according to the second preset number of distances; The main slot values in the target annotated slot values are counted, and the target slot information corresponding to the main slot value is obtained, so as to use the target slot information as the slot information of the unannotated corpus.
9. The corpus annotation method according to claim 8, characterized in that: Before arranging the distances between any of the fourth vectors and each of the third vectors in ascending order to form a second sequence, and obtaining a second preset number of distances in the second sequence according to a second preset rule, the method further includes: The distance between any one of the fourth vectors and each of the third vectors is compared with a second preset threshold to obtain a target distance less than the second preset threshold.
10. The corpus annotation method according to claim 1, characterized in that: The annotated corpus is manually annotated corpus, and the amount of the manually annotated corpus is less than the amount of the unannotated corpus.
11. A corpus annotation device, characterized in that: include: Corpus acquisition module, used to acquire annotated corpus and unannotated corpus; A first vectorization module, used for performing semantic compression on the annotated corpus and the unannotated corpus respectively, so as to obtain a first vector corresponding to the annotated corpus and a second vector corresponding to the unannotated corpus; An intention information acquisition module, configured to acquire the intention information of the unlabeled corpus according to the first vector and the second vector; a second vectorization module, configured to obtain a marked slot value in the marked corpus and a candidate slot value in the unmarked corpus, and to perform semantic compression on the marked slot value and the candidate slot value, respectively, to obtain a third vector corresponding to the marked slot value and a fourth vector corresponding to the candidate slot value; A slot information acquisition module, configured to acquire slot information of the unlabeled corpus according to the third vector and the fourth vector; wherein the slot information of the unlabeled corpus is acquired according to the third vector and the fourth vector based on a proximity algorithm; The number of the first vector and the number of the second vector are both plural, and obtaining the intention information of the unlabeled corpus according to the first vector and the second vector includes: Calculate the distance between any of the second vectors and each of the first vectors; Arrange the distances in ascending order to form a first sequence, and obtain a first preset number of distances in the first sequence according to a first preset rule; Determine a target annotated corpus according to the first preset number of distances, and obtain a target intent according to the target annotated corpus; The main intentions in the target intention are counted, and the main intentions are used as the intention information of the unlabeled corpus.
12. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the corpus tagging method according to any one of claims 1 to 10 is implemented.
13. An electronic device, characterized in that: include: one or more processors; A storage device, used to store one or more programs, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement the corpus tagging method according to any one of claims 1 to 10.
14. A computer program product, characterized in that The invention comprises a computer program carried on a computer-readable medium, wherein when the computer program is executed by a processor, the corpus annotation method according to any one of claims 1 to 10 is implemented.
Citation Information
Patent Citations
Web page searching method and device
CN107220307A
Task-type intent classification method and device
CN108334891A
Service method, service device, application open platform and storage medium
CN108876527A
Cross-domain dialogue strategy migration method and device, equipment and readable storage medium
CN109739965A
