A word slot extraction device and method for multi-round dialogue

Abstract

The present invention discloses a device and method for extracting word slots from multi-round conversations. The device comprises an input processing unit, a feature extraction unit, a masked language unit, a context-sensitive word prediction unit, an entity type recognition unit, an entity position recognition unit, and an output unit. The masked language unit and context-sensitive word prediction unit are used to increase feature diversity; the entity type recognition unit and the entity position recognition unit are used to identify the location and type of entities based on input text; and the output unit is used to analyze and evaluate the entire word slot model. The model and method described in the present invention can significantly improve accuracy in human-computer conversations, enhance user experience, and reduce pre-annotation costs.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and in particular to a word slot extraction device and method for multi-round dialogue. Background Art

[0002] Slot extraction is a key technology in multi-round human-computer dialogue. This technology is used in human-computer dialogue to convert the robot's initial understanding of the user's intention from the user's speech into clear user instruction information. In actual applications, the current conversation content is not the only way to obtain slots. It is usually closely related to the contextual text and conversation scene. Therefore, the technical difficulty of slot extraction is relatively high and the accuracy is relatively low.

[0003] Existing technical solutions for word slot extraction usually require a large amount of pre-annotated text corpus to be implemented, but real-time conversations often contain a lot of unannotated corpus. If existing technical solutions are used to process these special nouns, many errors will occur or the process may be impossible. Therefore, there are certain limitations when using existing technical solutions to process real-time human-computer conversations. Word slot extraction is imperfect, or a large amount of manually annotated corpus is required. Summary of the Invention

[0004] In order to solve the above-mentioned defects in the prior art, the present invention proposes a word slot extraction device and method for multi-round human-computer dialogue.

[0005] To achieve the above objectives, the present invention proposes a slot extraction device for multi-round dialogue, comprising an input processing unit, a feature extraction unit, an entity position recognition unit, an entity type recognition unit, a context prediction unit, a masking language unit, and an output unit, wherein:

[0006] The input processing unit is used to analyze the input text of the mixed unlabeled text and labeled text to obtain the semantic vector of the input text;

[0007] The feature extraction unit is used to analyze the semantic vector of the input text to obtain a feature vector set of the input text;

[0008] Entity location recognition unit: used to predict the start and end positions of the recognized entity from the input context;

[0009] Entity type recognition unit: used to predict the entity type based on each word entity;

[0010] Context prediction unit: used to perform context analysis on the input word and predict the completed text information;

[0011] Masked language unit: used to improve the generalization ability of samples and the diversity of features;

[0012] The output unit performs weighted summation based on the output of the aforementioned modules to analyze and evaluate the entire word slot model.

[0013] Furthermore, the input processing unit analyzes the input text including:

[0014] Get the word vector, text vector and position vector of the input text;

[0015] Get the word vector set of the input text;

[0016] Obtain an enhanced semantic vector set for the word vector set.

[0017] Furthermore, the input processing unit includes a 12-layer BERT model and / or a 6-layer BERT model.

[0018] Furthermore, the feature extraction unit includes a 6-layer transformer network and a fully connected layer neural network.

[0019] Furthermore, the entity position recognition unit, the entity type recognition unit, the context prediction unit, and the masking language unit respectively use four independent loss function units, and the loss function units are respectively connected to the output units.

[0020] Furthermore, the specific implementation of the four independent loss function units is:

[0021] The loss function unit used by the entity position recognition unit is the addition of two binary cross entropy loss functions to predict the start and end positions of the recognition respectively;

[0022] The entity type identification unit adopts a cross entropy loss function as a loss function unit to identify the type of the entity;

[0023] The upper and lower word prediction unit adopts cross entropy loss as the loss function unit, the input is the feature of the previous word, and the label is the next word;

[0024] The masked language unit adopts cross entropy loss as the loss function unit, the input is the feature of the masked word, and the label is the masked word itself.

[0025] Furthermore, the output unit performs a weighted summation of the outputs of the four loss function units.

[0026] The present invention also discloses a word slot extraction method for multi-round dialogue, comprising:

[0027] Providing training samples to train the word slot extraction device and generate trained vector parameters;

[0028] Convert the user's current input voice into natural language text;

[0029] Inputting the current natural language text mixed with the historical tag text into the word slot extraction device;

[0030] The word slot extraction device loads the trained vector parameters and extracts the corresponding word slot according to the current input;

[0031] Converting the extracted results into natural language text or speech to form a round of dialogue;

[0032] Waiting for the next round of user input of natural language text for the next round of conversation;

[0033] It is characterized in that the word slot extraction device is any one of the word slot extraction devices described above.

[0034] The present invention also discloses a word slot extraction method for multi-round dialogue, comprising:

[0035] Providing training samples to train the word slot extraction device and generate trained vector parameters;

[0036] Receive the natural language text input by the current user, and input the current natural language text mixed with the historical tag text into the word slot extraction device;

[0037] The word slot extraction device loads the trained vector parameters and extracts the corresponding word slot according to the current input;

[0038] Converting the extracted results into natural language text or speech to form a round of dialogue;

[0039] Waiting for the next round of user input of natural language text for the next round of conversation;

[0040] It is characterized in that the word slot extraction device is any one of the word slot extraction devices described above.

[0041] The present invention also discloses an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the storage medium communicate through the bus, and the processor executes the machine-readable instructions to perform the word slot extraction method for multi-round dialogue.

[0042] The present invention also discloses a storage medium, on which a computer program is stored. When the computer program is run by a processor, the method for extracting word slots from multiple rounds of dialogue is executed.

[0043] In actual applications, the modules described in the method and system disclosed in the present invention can be deployed in multiple modules on one target server, or each module can be independently deployed on a different target server. In particular, in order to provide more powerful computing processing capabilities, the modules can also be deployed on cluster target servers as needed.

[0044] It can be seen that the word slot extraction device of the present invention adopts a semi-supervised learning method. When there is a large amount of unlabeled text and a small amount of labeled text in the dialogue information, it can also learn good context features through training, so that word slots can be extracted more accurately in human-computer dialogue, obtaining a better human-computer dialogue experience, and significantly reducing the word slot extraction cost of human-computer dialogue.

[0045] In order to have a clearer and more comprehensive understanding of the present invention, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] To more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present application. Those skilled in the art can also derive other drawings based on these drawings without inventive effort.

[0047] Figure 1 This is a structural diagram of a word slot extraction device for multi-round dialogue according to an embodiment of the present application.

[0048] Figure 2 This is a schematic diagram of the specific implementation structure of a word slot extraction device for multi-round dialogue in an embodiment of the present application.

[0049] Among them: input processing unit 1, feature extraction unit 2, entity position recognition unit 3, entity category recognition unit 4, upper and lower word prediction unit 5, masking language unit 6, output unit 7. DETAILED DESCRIPTION

[0050] See also Figure 1 , Figure 1 A schematic structural diagram of a word slot extraction device for multi-round dialogue is shown.

[0051] The present application creates a novel word slot extraction device for multi-round dialogue. When the input is a mixture of unlabeled text and a small amount of labeled text, the semantic vector of the input text is obtained through processing by the input processing unit, and then the semantic vector of the input text is input into the feature extraction unit for feature extraction to obtain a feature vector set of the input text. The labeled data in the feature vector set is sent to the entity position recognition unit to predict whether the input is an entity, a backup of the feature vector set is sent to the entity type recognition unit to predict the type of each character entity, a backup of the feature vector set is sent to the upper and lower character prediction unit and the masking language unit to obtain the completion feature information of each character, and finally the accuracy of the word slot extraction is evaluated through summary analysis by the output unit.

[0052] The word slot extraction device constructed as described above mainly adopts an unsupervised mode. In the absence of labeled text, the model can be trained to learn good context features, so that the user's information can be more accurately extracted from multiple rounds of conversations, and the user's semantic intention can be further accurately identified.

[0053] The following introduces the input, output and functional principles of each module.

[0054] As an implementation method, the word slot extraction device for multi-round dialogue in an embodiment of the present application includes an input processing unit, a feature extraction unit, an entity position recognition unit, an entity type recognition unit, a context prediction unit, a masking language unit, and an output unit, wherein:

[0055] The input processing unit is used to analyze the input text of the mixed unlabeled text and labeled text, obtain the word vector, text vector and position vector of the input text, further obtain the word vector set of the input text, and continue to learn to obtain the semantic vector of the input text.

[0056] The input text refers to the data input by the user to the chatbot or intelligent customer system, which can be voice data or natural speech text data.

[0057] As a preferred embodiment, the input processing unit adopts a 12-layer BERT model and / or a 6-layer BERT model.

[0058] The feature extraction unit is used to analyze the semantic vector of the input text to obtain a feature vector set of the input text.

[0059] As a preferred implementation, the feature extraction module is composed of several dual encoder modules stacked together, using a dual encoder that shares 4 layers, each layer adopts a self-attention mechanism, and the structure of each dual encoder consists of a self-attention layer, a normalization layer, a feedforward neural network layer, and a normalization layer connected in sequence.

[0060] In this embodiment, the feature extraction module uses the self-attention layer of the self-attention mechanism as part of its technical solution. The self-attention mechanism can fully consider the semantic and grammatical connections between different words in a sentence. The word vector calculated in this way can further consider the relationship between the context. For example, in the sentence "The bird can fly because it has wing", in this sentence, the machine can associate "it" with "bird", and in a multi-round dialogue system, it can better understand the semantics of the context.

[0061] A normalization layer is set after the self-attention layer and the feedforward neural network layer. The advantage of normalization is that the features are distributed in a smaller controllable value range space, reducing the search range, which not only speeds up the training speed, but also improves the stability of the training, allowing the language model of this application to converge faster and have higher accuracy.

[0062] This enables the application to implement a lightweight design system. Even when the sample size is small, the language model of the application can well understand the context and make predictions.

[0063] As a preferred embodiment, the feature extraction unit adopts a 6-layer transformer network and a fully connected layer neural network.

[0064] As a preferred embodiment, the entity position recognition unit adopts the addition of two binary cross entropy loss functions to respectively predict the start and end positions of the recognition.

[0065] As a preferred embodiment, the entity type identification unit adopts a cross entropy loss function to identify the entity type;

[0066] As a preferred embodiment, the masked language unit adopts a cross-entropy loss function. The input is the feature of the masked word, and the label is the masked word itself. It is used to randomly mask the backup of the word feature vector set, and then complete and replace the masked part. The purpose of designing this loss is to obtain more feature samples to improve the generalization ability of the samples and the diversity of features.

[0067] As a preferred implementation, the upper and lower character prediction unit uses a cross-entropy loss function, using the feature vector of the previous character extracted by the multi-layer transformer as input, and the label is the next character. This loss function is designed to enable the model to learn the temporal patterns of language, thereby increasing feature diversity and improving sample generalization.

[0068] For example, in the sentence “The bird can fly because it has wings”, the feature of the word “The” is a 768-dimensional vector, and the label is “bird” instead of “The”. The features and labels are then input into the cross-entropy loss function to calculate the loss.

[0069] The output unit performs weighted summation based on the output of the aforementioned modules to analyze and evaluate the entire word slot model.

[0070] See also Figure 2 As a preferred embodiment, the entity position recognition unit, entity type recognition unit, upper and lower word prediction unit, and masking language unit in the embodiment of the present application respectively include four loss function units, and the loss function units are respectively connected to the output units. The output unit of the system is a weighted superposition of the outputs of these four loss function units.

[0071] Loss function unit 1 is used to improve the accuracy of entity recognition.

[0072] Loss function unit 2 is used to improve the prediction accuracy of entity category recognition.

[0073] The loss function unit 3 is used to improve the diversity of the output features of the masked language unit.

[0074] The loss function unit 4 is used to improve the diversity of the output features of the upper and lower word prediction units.

[0075] Based on the multi-round dialogue word slot extraction device of the above embodiment, the present application also discloses a multi-round dialogue word slot extraction method, which comprises the following steps:

[0076] Providing training samples to train the word slot extraction device and generate trained vector parameters;

[0077] Convert the user's current input voice into natural language text;

[0078] Inputting the current natural language text mixed with the historical tag text into the word slot extraction device;

[0079] The word slot extraction device loads the trained vector parameters and extracts the corresponding word slot according to the current input;

[0080] Converting the extracted results into natural language text or speech to form a round of dialogue;

[0081] Waiting for the next round of user input of natural language text for the next round of conversation;

[0082] The word slot extraction device used is the word slot extraction device of the above embodiment.

[0083] In addition, a variation of the multi-round dialogue slot extraction method according to the above embodiment includes:

[0084] Providing training samples to train the word slot extraction device and generate trained vector parameters;

[0085] Receive the natural language text input by the current user, and input the current natural language text mixed with the historical tag text into the word slot extraction device;

[0086] The word slot extraction device loads the trained vector parameters and extracts the corresponding word slot according to the current input;

[0087] Converting the extracted results into natural language text or speech to form a round of dialogue;

[0088] Waiting for the next round of user input of natural language text for the next round of conversation;

[0089] The word slot extraction device used is the word slot extraction device of the above embodiment.

[0090] The present application also provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the storage medium communicate through the bus, and the processor executes the machine-readable instructions to perform the method described in the above embodiment.

[0091] The present application also provides a storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described in the above embodiment is executed.

[0092] It should be noted that, those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the storage medium may include but is not limited to: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0093] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A word slot extraction device for multi-round dialogue, characterized by: It includes an input processing unit, a feature extraction unit, an entity position recognition unit, an entity type recognition unit, a context prediction unit, a masking language unit, and an output unit, wherein: The input processing unit includes a 12-layer BERT model or / and a 6-layer BERT model, which is used to analyze the input text of the mixed unlabeled text and labeled text, obtain the semantic vector of the input text, further obtain the word vector set of the input text, and continue to learn to obtain the semantic vector of the input text, wherein the input text refers to the voice data or natural speech text data input by the user to the chat robot or intelligent customer system. The input processing unit analyzes the input text, including: Get the word vector, text vector and position vector of the input text; Get the word vector set of the input text; Obtaining an enhanced semantic vector set of the word vector set; The feature extraction unit includes a 6-layer transformer network and a fully connected neural network, which is used to analyze the semantic vector of the input text and obtain the feature vector set of the input text; Entity location recognition unit: used to predict the start and end positions of the recognized entity from the input context; Entity type recognition unit: used to predict the entity type based on each word entity; Context prediction unit: used to perform context analysis on the input word and predict the completed text information; Masked language unit: used to improve the generalization ability of samples and the diversity of features; The output unit performs weighted summation based on the outputs of the entity position recognition unit, entity type recognition unit, context prediction unit, and masking language unit to analyze and evaluate the entire word slot model.

2. The word slot extraction device according to claim 1, wherein: The entity position recognition unit, the entity type recognition unit, the upper and lower word prediction unit, and the masking language unit respectively adopt four independent loss function units, and the loss function units are respectively connected to the output units.

3. The word slot extraction device according to claim 2, wherein: The loss function unit used by the entity position recognition unit is the addition of two binary cross entropy loss functions, which respectively predict the start and end positions of the recognition.

4. The word slot extraction device according to claim 2, wherein: The entity type identification unit adopts a cross entropy loss function as a loss function unit to identify the type of the entity.

5. The word slot extraction device according to claim 2, wherein: The upper and lower word prediction unit adopts cross entropy loss as the loss function unit, the input is the feature of the previous word, and the label is the next word.

6. The word slot extraction device according to claim 2, wherein: The masked language unit adopts cross entropy loss as the loss function unit, the input is the feature of the masked word, and the label is the masked word itself.

7. The word slot extraction device according to claim 2, wherein: The output unit performs a weighted summation of the outputs of the four loss function units.

8. A method for extracting word slots from a multi-turn conversation, comprising: Providing training samples to train the word slot extraction device and generate trained vector parameters; Convert the user's current input voice into natural language text; Inputting the current natural language text mixed with the historical tag text into the word slot extraction device; The word slot extraction device loads the trained vector parameters and extracts the corresponding word slot according to the current input; Convert the extracted results into natural language text or speech to form a round of dialogue; Waiting for the next round of user input of natural language text for the next round of conversation; It is characterized in that the word slot extraction device is the word slot extraction device according to any one of claims 1-7.

9. A method for extracting word slots from a multi-round conversation, comprising: Providing training samples to train the word slot extraction device and generate trained vector parameters; Receive the natural language text input by the current user, and input the current natural language text mixed with the historical tag text into the word slot extraction device; The word slot extraction device loads the trained vector parameters and extracts the corresponding word slot according to the current input; Convert the extracted results into natural language text or speech to form a round of dialogue; Waiting for the next round of user input of natural language text for the next round of conversation; It is characterized in that the word slot extraction device is the word slot extraction device according to any one of claims 1-7.

10. An electronic device, characterized in that: include: A processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the storage medium communicate via the bus, and the processor executes the machine-readable instructions to perform the word slot extraction method for multi-round dialogue as described in claim 8 or 9.

11. A storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, executes the word slot extraction method for multi-round dialogue according to claim 8 or 9.

Images