Dialogue reasoning recognition model training method, device and system and medium
By constructing input vectors containing text, role and position encoding and performing attention interaction processing, the problems of inferring multiple rounds of inference efficiency and text length limitation in task-based dialogue systems are solved, and the context interaction and recognition model between different roles is realized The recognition rate is improved.
Patent Information
- Application Number
- CN202411065732.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-05
- Publication Date
- 2024-11-15
AI Technical Summary
The existing task-based dialogue system is inefficient in multiple rounds of multi-intention reasoning scenarios, and cannot effectively conduct contextual interactions between different roles. Due to the limitation of input text length and data imbalance, the recognition rate of the recognition model is affected.
By constructing an input vector containing text vectors, character paragraph vectors and position coded vectors, input them into the encoder that recognizes the model, use attention mechanism to perform interactive processing, and strengthen model training through full connection mapping and regularization layers to generate category vectors To identify intent categories, solve text length limitations and data imbalance problems, and achieve contextual interaction between different roles.
The recognition rate of the recognition model is improved, allowing it to work effectively in both single and multi-round reasoning scenarios, and improving the recognition accuracy of the dialogue process.
Smart Images

Figure CN118964612A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of information processing, and in particular to a method, device, system and medium for training a dialogue reasoning recognition model. Background Art
[0002] Intelligent voice assistants have a wide range of application scenarios. The current task-based dialogue system has poor working efficiency in multi-round multi-intent reasoning scenarios. It only models the text of the current time step, and does not achieve contextual interaction between paragraphs and paragraphs, roles and roles. It is accurate in single-round reasoning, but inefficient in multi-round reasoning. Due to resource limitations, it is not possible to simply concatenate the previous round of degree and the current round of user input directly into text. When inputting into the model, it will be limited by the length of the text. In the cold start phase of the system, there is a lack of labeled data and there is an extreme data imbalance, which affects the recognition rate of the model. Summary of the invention
[0003] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a method, device, system and medium for training a dialogue reasoning recognition model that can be applied to financial technology or other related fields, aiming to solve the problem of limited input text length in recognition, so as to carry out contextual interaction between different roles in the dialogue process, so that the recognition model can perform both single-round reasoning and multi-round reasoning, thereby improving the recognition rate of the recognition model.
[0004] The technical solution of the present invention is as follows:
[0005] A method for training a dialogue reasoning recognition model, the method comprising:
[0006] Collecting conversation data, and constructing an input vector of a recognition model according to the conversation data; the input vector includes a text vector, a role paragraph vector, and a position encoding vector;
[0007] Inputting the input vector into an encoder of a recognition model to obtain a text representation vector;
[0008] Mapping the text representation vector to the category through full connection to obtain a category vector;
[0009] The intent category hit by the conversation data is identified according to the category vector and the category threshold, and the training of the recognition model is completed.
[0010] In one embodiment, the collecting of the conversation data and constructing an input vector of the recognition model according to the conversation data includes:
[0011] Extracting text data from the conversation data and preprocessing the text data;
[0012] Convert the preprocessed text data into text vectors.
[0013] In one embodiment, the collecting of the conversation data and constructing an input vector of the recognition model according to the conversation data includes:
[0014] Acquire dialogue text data including role identification from the dialogue data;
[0015] Determining the number of dialogue roles in the dialogue text data; the dialogue roles have corresponding role identifiers;
[0016] If the number of roles is 2, the segment identifier is allocated according to the role identifier;
[0017] If the number of roles is greater than 2, the segment identifiers are assigned according to the dialogue order of the roles;
[0018] The text paragraph of each role is converted into a corresponding role paragraph vector according to the role identifier and the segment identifier.
[0019] In one embodiment, the collecting of the conversation data and constructing an input vector of the recognition model according to the conversation data includes:
[0020] Determine the maximum sequence length of the recognition model input sequence;
[0021] Processing the conversation data according to the maximum sequence length to obtain a data sequence;
[0022] A position encoding vector is calculated for each position in the data sequence.
[0023] In one embodiment, inputting the input vector into an encoder of a recognition model to obtain a text representation vector includes:
[0024] Pruning the encoder of the recognition model to obtain an encoder with a preset number of layers;
[0025] Inputting the input vector to an encoder of a preset number of layers;
[0026] Attention interaction processing is performed on each layer of the encoder with the preset number of layers according to the attention mechanism to obtain a text representation vector.
[0027] In one embodiment, the step of identifying the intent category hit by the conversation data according to the category vector and the category threshold, and completing the training of the recognition model, includes:
[0028] If the category vector is greater than the category threshold, the intention is hit;
[0029] If the category vector is less than the category threshold, the intent is not hit.
[0030] In one embodiment, it further includes:
[0031] Add regularization layers to each encoder layer of the recognition model;
[0032] The alleged conversation data is processed through a self-learning algorithm to obtain pseudo-label data;
[0033] The recognition model is trained based on the regularization layer and the pseudo-label data.
[0034] A dialog reasoning recognition model training device, comprising:
[0035] A data collection module, used to collect conversation data and construct an input vector of a recognition model based on the conversation data;
[0036] A first vector module, used for inputting the input vector into an encoder of a recognition model to obtain a text representation vector;
[0037] A second vector module is used to map the text representation vector to a category through full connection to obtain a category vector;
[0038] The model training module is used to identify the intent category hit by the conversation data according to the category vector and the category threshold, and complete the training of the recognition model.
[0039] A dialog reasoning recognition model training system, the system comprising at least one processor; and
[0040] a memory communicatively connected to the at least one processor; wherein,
[0041] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned method for training the dialogue reasoning recognition model.
[0042] A non-volatile computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the one or more processors can execute the above-mentioned method for training a dialogue reasoning recognition model.
[0043] Beneficial effects: The present invention discloses a method, device, system and medium for training a dialogue reasoning recognition model. Compared with the prior art, the embodiments of the present invention solve the problem of limited input text length in recognition, thereby performing contextual interaction between different roles in the dialogue process, so that the recognition model can perform both single-round reasoning and multi-round reasoning, thereby improving the recognition rate of the recognition model. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:
[0045] Figure 1 A flow chart of a method for training a dialog reasoning recognition model provided by an embodiment of the present invention;
[0046] Figure 2 A schematic diagram of a method for training a dialog reasoning recognition model provided by an embodiment of the present invention;
[0047] Figure 3 A schematic diagram of the functional modules of a dialog reasoning recognition model training device provided by an embodiment of the present invention;
[0048] Figure 4 A schematic diagram of the hardware structure of a dialog reasoning recognition model training system provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0049] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail below. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. The embodiments of the present invention are described below in conjunction with the accompanying drawings.
[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0051] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.
[0052] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0053] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0054] See also Figure 1 , Figure 1 A flowchart of an embodiment of the method for training a dialog reasoning recognition model provided by the present invention. The method for training a dialog reasoning recognition model provided in this embodiment is applied to a system consisting of a terminal device, a network and a server, wherein the network is a medium for directly providing a communication link between the terminal device and the server, which may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.; the operating system on the terminal device may include a handheld device operating system (iPhone operating system, iOS system), Android system or other operating system, and the terminal device is connected to the server through the network to achieve interaction, thereby performing operations such as receiving or sending data, and specifically may be various electronic devices with a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, portable computers and desktop servers, etc.
[0055] like Figure 1 As shown, Figure 1 A method for training a dialog reasoning recognition model provided by an embodiment of the present invention specifically comprises the following steps:
[0056] S100, collecting dialogue data, and constructing an input vector of a recognition model according to the dialogue data; the input vector includes a text vector, a role paragraph vector, and a position encoding vector;
[0057] S200, inputting the input vector into an encoder of a recognition model to obtain a text representation vector;
[0058] S300, mapping the text representation vector to a category through full connection to obtain a category vector;
[0059] S400: Identify the intent category hit by the conversation data according to the category vector and the category threshold, and complete the training of the recognition model.
[0060] In this embodiment, dialogue data is collected and the input vector of the recognition model is constructed based on the dialogue data. The input vector includes a text vector, a role paragraph vector and a position encoding vector. The text vector, the role paragraph vector and the position encoding vector are fused into a model input vector through an algorithm. Such a model input vector incorporates single-round semantic information and multi-round semantic information, which is conducive to completing the information interaction between different role dialogues and improving the accuracy of the final model recognition.
[0061] The input vector is input into the optimized 6-layer encoder structure, and the character vectors inside the input vector are allowed to interact with each other. After the 6-layer encoder is updated and iterated, each paragraph text representation vector is obtained. Compared with the input vector, the text representation vector can better represent the text information because each token (character) is no longer independent, but has enhanced attention between roles and lexical semantics. The text representation vector is used as input, multiplied by the matrix of the fully connected mapping, and mapped to several (u) categories, that is, a total of several (u) intents, to obtain a category vector of size 1*u. Each value in the vector represents the score of the category at this position. According to the category vector and category threshold, the intent category hit by the dialogue data is identified to complete the training of the recognition model. By improving the encoding method and model structure, the limitation of text length is solved; the role information is marked, and interactions between different roles are carried out, the input source is changed, and multiple rounds of semantic information are learned to improve the model recognition rate.
[0062] In one embodiment, the collecting of the conversation data and constructing an input vector of the recognition model according to the conversation data includes:
[0063] Extracting text data from the conversation data and preprocessing the text data;
[0064] Convert the preprocessed text data into text vectors.
[0065] In this embodiment, text data is extracted from the conversation data, and the conversations with timestamps and conversations without timestamps in the text data are distinguished. By connecting each message in the conversation and using appropriate delimiters (such as periods, line breaks, etc.), multiple rounds of conversations with timestamps are spliced into single-sentence texts, and single-sentence texts without timestamps are maintained. The text is divided into multiple segments ("Segment" usually refers to a part or paragraph of text, which can be a sentence, phrase, or any semantically meaningful text unit). Each segment contains a certain number of sentences or text lines. Each segment is tokenized and decomposed into a sequence of tokens, and the text is divided into basic token units by spaces, punctuation marks, or other language-specific delimiters. A vocabulary is built based on the segmented text to ensure that each token has a unique index in the vocabulary. Select a pre-trained word embedding model, such as Word2Vec, GloVe, BERT, etc., or train a custom word embedding model. Each token in the sentence is represented by a fixed-dimensional vector through Token-embedding ("Token-embedding" refers to the process of converting each Token (word, character or other symbol) in the text into a vector representation of a fixed dimension) to obtain a text vector. Each token (character) represents a 768 vector. The maximum number of tokens in a single sentence is set to 512^2, so the size of the text vector is 512^2*(1*768). The original BERT model structure ideally has a maximum number of tokens of only 512, but in reality it can only be around 250 due to the constraints of machine computing power. The trained recognition model can process texts of 512^2 in length, converting text data into a numerical form that the recognition model can understand to solve the limitation of text length.
[0066] In one embodiment, the collecting of the conversation data and constructing an input vector of the recognition model according to the conversation data includes:
[0067] Acquire dialogue text data including role identification from the dialogue data;
[0068] Determining the number of dialogue roles in the dialogue text data; the dialogue roles have corresponding role identifiers;
[0069] If the number of roles is 2, the segment identifier is allocated according to the role identifier;
[0070] If the number of roles is greater than 2, the segment identifiers are assigned according to the dialogue order of the roles;
[0071] The text paragraph of each role is converted into a corresponding role paragraph vector according to the role identifier and the segment identifier.
[0072] In this embodiment, the conversation text data including the role identifier is obtained from the conversation data. The number of roles of the conversation roles in the conversation text data is determined, wherein the conversation roles have corresponding role identifiers. If the number of roles in the conversation data is 2, the segment identifier is assigned according to the role identifier; if the number of roles in the conversation data is greater than 2, the segment identifier is assigned according to the conversation order of the roles. The text paragraph of each role is converted into a corresponding role paragraph vector according to the role identifier and the segment identifier. They can be converted into vector form by One-Hot Encoding. Secondly, the role identifier can be a number (such as 0, 1, 2, etc.) or a string (such as "user", "customer_service", etc.). For example, 0 and 1 are used to distinguish the role information (such as 1 for customer and 0 for customer service). A mapping table is established to map the identifier of each role to an index. For example, if there are two roles, user and customer service, they can be mapped to indexes 0 and 1 respectively. Use One-Hot Encoding to convert the index of each role into a vector form. Then the role paragraph of each token is encoded as a 1*768 vector of all 0 or all 1. Concatenate the character paragraph encodings of 512^2 tokens to get a character paragraph vector of size 512^2*(1*768).
[0073] Specifically, one-hot encoding is a method of converting categorical data into numerical data, where each category is represented by a binary vector that is 1 at the index position of the corresponding category and 0 at the remaining positions. For each character paragraph in the conversation, use one-hot encoding to generate the corresponding character paragraph vector based on its character identifier. Enhance the model's understanding of conversation data in terms of distinguishing the semantics and emotions of different character conversations. By determining all possible categories of categorical variables in the data. For each category, create a binary vector whose length is equal to the total number of categories. Each element in the vector is 0, except for the element corresponding to the current category, which is set to 1. For each sample, select the corresponding vector from the encoding matrix based on its category. Repeat the above steps to generate a one-hot encoded vector for each sample in the dataset.
[0074] In one embodiment, the collecting of the conversation data and constructing an input vector of the recognition model according to the conversation data includes:
[0075] Determine the maximum sequence length of the recognition model input sequence;
[0076] Processing the conversation data according to the maximum sequence length to obtain a data sequence;
[0077] A position encoding vector is calculated for each position in the data sequence.
[0078] In this embodiment, the position encoding of the BERT pre-trained model is fixed and has the same dimension as the hidden layer of the model. BERT uses relative position encoding instead of absolute position encoding, which enables the model to process longer sequences than when trained. However, the input sequence length limit of BERT is usually 512. Therefore, the position encoding vector is obtained by tree optimization of the position encoding of the open source BERT pre-trained model, which solves the text length limitation problem (512->512^2) caused by the original fixed position encoding without retraining. Specifically, Figure 2 As shown: layer a (root layer) represents the original coding sequence, b is the trunk layer, and c is the leaf layer of the new coding sequence obtained in the end. 1, 2..k of the original layer a represent the coding numbers of the 1st, 2nd, ...kth positions (k equals 512), and the vector mapped to each number is fixed. For example, let a1, a2, ..ak (k = 512), and more position vectors can be obtained through k vectors, that is, c1, c2, ..ct can be generated (t is much larger than k). Now generate a fixed-size group for each number in layer a (the group size is set to k, and the specific setting is based on actual conditions).
[0079] c(i-1)*k+j=g*mi+(1-g)mj
[0080] mi=(ai-g*a1) / (1-g),i=1,…,k
[0081] The new (c layer) encoding vector of the (i-1)*k+j position is actually the encoding of the i-th position of the a layer and the j-th position of the b layer. g is a constant belonging to [0,1] and not equal to 0.5. It is a weight hyperparameter. Through experiments, g takes 0.25 for the best effect. The above improvements have the following properties: a1=c1,…,ak=ck,.. and the encoding range is expanded from [1,k] to [1,k^2]. Then, the position encoding of 512^2 tokens can be directly obtained through the optimized position encoding vector and concatenated as the position encoding vector.
[0082] Specifically, different models may have different maximum sequence length limits. For example, the original BERT model is limited to 512 tokens. Determine the maximum length of the input sequence, and pad shorter sequences or truncate longer sequences to make all text sequences of the same length. If the sequence length exceeds the maximum length, truncation can be used to reduce the number of tokens in the sequence. If the sequence length is less than the maximum length, a specific padding token can be used to pad the sequence to make it of the same length. Generate or adjust the position encoding based on the determined maximum length. For models with fixed position encoding, use the maximum length as the dimension of the position encoding. For models with dynamic position encoding, position encodings need to be generated based on the actual sequence length. Take the processed sequence (including word embeddings and position encodings) as the input of the model. For each position in the sequence, calculate the position encoding vector. Add the word embedding vector of each token to its corresponding position encoding vector to generate the final position encoding vector. Position encodings are commonly used in RNN or Transformer models to provide position information for each element in the sequence. In the Transformer model, the position encoding is a vector of the same dimension as the word embedding, whose value is calculated based on the position. This ensures that the model can effectively process sequences of different lengths while capturing the sequential information in the sequence using positional encoding.
[0083] In one embodiment, inputting the input vector into an encoder of a recognition model to obtain a text representation vector includes:
[0084] Pruning the encoder of the recognition model to obtain an encoder with a preset number of layers;
[0085] Inputting the input vector to an encoder of a preset number of layers;
[0086] Attention interaction processing is performed on each layer of the encoder with the preset number of layers according to the attention mechanism to obtain a text representation vector.
[0087] In this embodiment, first, a 6-layer encoder structure is initialized, and each layer contains a self-attention mechanism and a feedforward network. The initial weights of the 6-layer encoder do not need to be pre-trained. The open source BERT model is directly called to prune the 12-layer encoder. The implementation method is to connect the output of the 5th layer encoder directly to the input of the 12th layer encoder, and then save the latest pre-trained model structure. Without reducing the accuracy of reasoning, the model experiment can be quickly compressed (from 12 layers to 6 layers), and the reasoning speed is accelerated. Secondly, each layer of encoder will perform an attention interaction, and the first layer is:
[0088]
[0089] The rest of the layers are shown in the formula:
[0090]
[0091] Ei represents the output vector of the encoder of the i-th layer. The input vector of the first layer E1 is X, and the input of the remaining layers Ei (i=2,..6) is the output Ei-1 (i=1,2,..6) of the previous layer. Conv_2 represents a bidirectional convolution operation, the purpose of which is to reduce the dimension of the input vector and extract important local features. Since it is a bidirectional convolution, feature extraction is performed on the dimension of each token of the input vector and all paragraph dimensions at the same time. The purpose of doing this is to reduce the computing power of each calculation, so as to increase the batch_size (batch size) or increase the length of max_len (sequence), and solve the problem of limited text length from the perspective of computing power. Wqi (Query Weight Matrix, multiplied by the input Query vector to generate the Query representation for calculating the attention score), Wki (Key Weight Matrix, multiplied by the input Key vector to generate the Key representation for calculating the attention score), and Wvi (Value Weight Matrix, multiplied by the input Value vector to generate the Value representation for weighted summation) are the weight parameter matrices of the i-th layer encoder. The initial random assignment and the matrix size are not fixed, but they need to be able to meet the basic matrix operations. Wqi, Wki, and Wvi can be obtained by multiplying the text feature representation vector Conv_2(X) or Ei-1 corresponding to this layer to obtain the query matrix, key matrix, and value matrix. These three matrices can be understood as the projections of Conv_2(X) or Ei-1 on three different spaces. It is precisely because of this projection in different spaces that its expressive power is increased. Softmax represents the softmax operation. dk is a smoothing factor (constant empirical value) to prevent the numerator in the formula from being too large, causing the gradient to disappear. Finally, by pruning and splicing the original BERT model and making full use of the underlying lexical and semantic features of the Transformer, the model is quickly compressed and the actual reasoning speed of the system is accelerated.
[0092] In each layer of the encoder, a self-attention mechanism is used to allow the vector of each token to interact with other tokens in the sequence. Through the layer-by-layer processing of the 6-layer encoder, each layer will update the representation of the token and gradually extract deeper features. At each layer, a residual connection is used to add the output of the current layer to the input to help the gradient flow and prevent the gradient vanishing problem in deep networks. After processing by the 6-layer encoder, the text representation vector of each paragraph is obtained. The dimension of the text representation vector is the same as the input vector, but the content has been enhanced with deeper semantics and role relationships. Finally, the vectors of all tokens (128) in the 6th layer are avg-operated to obtain the final text representation vector 1*128avg_L as the entire paragraph.
[0093] In one embodiment, the step of identifying the intent category hit by the conversation data according to the category vector and the category threshold, and completing the training of the recognition model, includes:
[0094] If the category vector is greater than the category threshold, the intention is hit;
[0095] If the category vector is less than the category threshold, the intent is not hit.
[0096] In this embodiment, each value in the category vector represents the score of the category at this position. A category threshold (such as 0) is set to determine whether the score of each category is greater than 0. For each category, if the score is greater than the category threshold, the category is considered to be hit (the value is 1), otherwise it is not hit (the value is 0). If the score>0, the intent is hit (that is, the value is 1), if the score<0, the intent is not hit (that is, the value is 0). In the end, you can know which intents this text hits in u categories.
[0097] Furthermore, in the training phase, for the choice of loss function, the conventional approach of averaging after u sigmoid+binary_crossentropy operations is abandoned. Instead, based on the advantages of logsumexp of softmax, circle_loss is used as the loss function. In the softmax function, when the values of the input features differ greatly, direct calculation may encounter numerical underflow or overflow problems. Using logsumexp, these values can be stably calculated in logarithmic space. circle_loss improves the distinction between different categories by creating circular boundaries in the feature space. circle_loss allows samples of different categories to have clearer boundaries in the feature space, thereby improving the robustness of the model. That is, each target class score of u categories is greater than the non-target class score, without summing or averaging the loss functions of u categories and then minimizing them. circle_loss is a new loss function designed for deep metric learning, which can make samples of the same type more compactly clustered together, while samples of different types are more dispersed. Use logsumexp to stabilize the output of the softmax layer, and then apply circle_loss as the loss function to further optimize the model. This essentially solves the problem of inaccurate model prediction caused by sample imbalance.
[0098] In one embodiment, it further includes:
[0099] Add regularization layers to each encoder layer of the recognition model;
[0100] The alleged conversation data is processed through a self-learning algorithm to obtain pseudo-label data;
[0101] The recognition model is trained based on the regularization layer and the pseudo-label data.
[0102] In this embodiment, during the training phase, the tail data (data with a small proportion of categories in the corpus) is enhanced by adding dropout (regularization technology) to the feed-forward calculation process of each layer of the encoder. In each layer of the encoder, determine the location to add Dropout. A common location is at the output of the feed-forward network, that is, after the self-attention mechanism and before the input of the next layer. Select a Dropout ratio, usually between 0 and 0.5, to prevent overfitting. A higher Dropout ratio may cause the network training speed to slow down, but it may also improve the generalization ability of the model. Apply Dropout at the selected location. During training, a certain proportion of neuron outputs are randomly discarded, and during inference, all neurons are retained, but the output is multiplied by a scaling factor (usually the inverse of the Dropout ratio) to keep the expected value unchanged. The specific approach is: Dropout sets the activation output of a part of the neurons to zero, such as 0.01, and randomly sets the values obtained by some feed-forward operations to 0. The proportion of discarding is determined by the hyperparameter p. For example, p=0.5 means that there is a 50% probability of discarding the output of a certain neuron. Or randomly delete some neurons or. Dropout does not really remove neurons from the network, but temporarily cuts off some network connections during the training process. Since the neurons discarded are different each time during training, Dropout reduces the problem of over-dependence or "co-adaptation" between neurons. In this way, for the same input vector in 1, different text representation vectors can be obtained each time during the training phase. For categories with unbalanced samples, an input vector is executed n times to obtain n different sample representation text representation vectors for this category. Similarly, a random dropout operation of 0.01 is also added to the dense (fully connected layer) layer. The Dropout layer is used in the model to prevent overfitting and improve the generalization ability of the model. Through the randomness of Dropout, different network structures can be obtained in each training iteration, similar to the effect of data enhancement.
[0103] Furthermore, based on the self-learning (Semi-Supervised Self-Learning, semi-self-learning) method, unlabeled data is fully utilized. In the cold start phase, when the amount of labeled data is insufficient and unbalanced, the recall rate of the model for the recognition of tail data (such data is small) is very low, but the accuracy rate is very high. That is, the data that is actually the tail category will be mistaken for the head, but if the model predicts that a data is the tail category, then the probability of its correct prediction is very high. Using this property, we perform inference tests on the unlabeled data for our trained model, and find the data predicted as the tail category in the test set by obtaining a 1*u vector (value is 1 or 0), 1 means hitting this category. Add these as data with solid_label (pseudo-label) to the original training set, load the model weights saved in the previous round, and train again. Use the trained model to predict the unlabeled data and generate pseudo-labels. Use the data with pseudo-labels together with the labeled data for further training of the model. Iterate this process until the data category ratio is no longer distributed in a long-tail distribution. The model weights saved at this time are the final model weights. During the model training phase, the data distribution ratio is improved based on dropout and self-learning methods. By performing data enhancement without increasing the cost of manual labeling, the recognition rate of intent with less data volume is improved.
[0104] Another embodiment of the present invention provides a dialog reasoning recognition model training device, which is used to execute any embodiment of the above-mentioned dialog reasoning recognition model training method. Specifically, refer to Figure 3 , Figure 3 A schematic diagram of the functional modules of a dialog reasoning recognition model training device provided by an embodiment of the present invention. The dialog reasoning recognition model training device can be configured in a management server.
[0105] like Figure 3 As shown, the device comprises:
[0106] A data collection module 11 is used to collect conversation data and construct an input vector of a recognition model based on the conversation data;
[0107] A first vector module 12, used for inputting the input vector into an encoder of a recognition model to obtain a text representation vector;
[0108] A second vector module 13, used for mapping the text representation vector to a category through full connection to obtain a category vector;
[0109] The model training module 14 is used to identify the intent category hit by the conversation data according to the category vector and the category threshold, and complete the training of the recognition model.
[0110] like Figure 3As shown, the data collection module 11, the first vector module 12, the second vector module 13 and the model training module 14 are connected in sequence. The module referred to in the present invention refers to a series of computer program instruction segments that can complete specific functions, which is more suitable for describing the execution process of dialogue reasoning recognition model training than the program. For the specific implementation of each module, please refer to the corresponding method embodiment above, which will not be repeated here.
[0111] In one embodiment, the data collection module 11 includes:
[0112] A text processing unit, used to extract text data from the conversation data and pre-process the text data;
[0113] The vector conversion unit is used to convert the preprocessed text data into text vectors.
[0114] In one embodiment, the data collection module 11 includes:
[0115] A role identification unit, used to obtain dialogue text data containing role identification from the dialogue data;
[0116] A role number unit, used to determine the role number of dialogue roles in the dialogue text data; the dialogue roles have corresponding role identifiers;
[0117] A first segment identification unit, configured to allocate the segment identification according to the role identification if the number of roles is 2;
[0118] A second segment identification unit, configured to assign the segment identifications according to the dialogue sequence of the roles if the number of the roles is greater than 2;
[0119] The vector conversion unit is used to convert the text paragraph of each role into a corresponding role paragraph vector according to the role identifier and the segment identifier.
[0120] In one embodiment, the data collection module 11 includes:
[0121] A sequence length unit is used to determine the maximum sequence length of the input sequence of the recognition model;
[0122] A data sequence unit, used for processing the conversation data according to the maximum sequence length to obtain a data sequence;
[0123] The encoding vector unit is used to calculate a position encoding vector for each position in the data sequence.
[0124] In one embodiment, the first vector module 12 includes:
[0125] A pruning unit, used for pruning the encoder of the recognition model to obtain an encoder with a preset number of layers;
[0126] An input unit, used for inputting the input vector into an encoder of a preset number of layers;
[0127] The attention interaction unit is used to perform attention interaction processing on each layer of the encoder with the preset number of layers according to the attention mechanism to obtain a text representation vector.
[0128] In one embodiment, the second vector module 13 includes:
[0129] A first intention unit, configured to hit the intention if the category vector is greater than a category threshold;
[0130] The second intention unit is used to not hit the intention if the category vector is smaller than the category threshold.
[0131] In one embodiment, the dialog reasoning recognition model training device further includes:
[0132] Add regularization layers to each encoder layer of the recognition model;
[0133] The alleged conversation data is processed through a self-learning algorithm to obtain pseudo-label data;
[0134] The recognition model is trained based on the regularization layer and the pseudo-label data.
[0135] Another embodiment of the present invention provides a dialogue reasoning recognition model training system, such as Figure 4 As shown, the system 10 includes:
[0136] One or more processors 110 and memory 120, Figure 4 A processor 110 is used as an example for description. The processor 110 and the memory 120 may be connected via a bus or other means. Figure 4 The example of connecting through bus is taken in the following.
[0137] The processor 110 is used to complete various control logics of the system 10, and it can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISC Machine) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination of these components. In addition, the processor 110 can also be any traditional processor, microprocessor or state machine. The processor 110 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP and / or any other such configuration.
[0138] The memory 120, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions corresponding to the method for training a dialog reasoning recognition model in the embodiment of the present invention. The processor 110 executes various functional applications and data processing of the system 10 by running the non-volatile software programs, instructions and units stored in the memory 120, that is, implements the method for training a dialog reasoning recognition model in the above method embodiment.
[0139] The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required by at least one function; the data storage area may store data created according to the use of the system 10, etc. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 120 may optionally include a memory remotely arranged relative to the processor 110, and these remote memories may be connected to the system 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0140] One or more units are stored in the memory 120, and when executed by one or more processors 110, the following steps are implemented:
[0141] Collecting conversation data, and constructing an input vector of a recognition model according to the conversation data; the input vector includes a text vector, a role paragraph vector, and a position encoding vector;
[0142] Inputting the input vector into an encoder of a recognition model to obtain a text representation vector;
[0143] Mapping the text representation vector to the category through full connection to obtain a category vector;
[0144] The intent category hit by the conversation data is identified according to the category vector and the category threshold, and the training of the recognition model is completed.
[0145] In one embodiment, the collecting of the conversation data and constructing an input vector of the recognition model according to the conversation data includes:
[0146] Extracting text data from the conversation data and preprocessing the text data;
[0147] Convert the preprocessed text data into text vectors.
[0148] In one embodiment, the collecting of the conversation data and constructing an input vector of the recognition model according to the conversation data includes:
[0149] Acquire dialogue text data including role identification from the dialogue data;
[0150] Determining the number of dialogue roles in the dialogue text data; the dialogue roles have corresponding role identifiers;
[0151] If the number of roles is 2, the segment identifier is allocated according to the role identifier;
[0152] If the number of roles is greater than 2, the segment identifiers are assigned according to the dialogue order of the roles;
[0153] The text paragraph of each role is converted into a corresponding role paragraph vector according to the role identifier and the segment identifier.
[0154] In one embodiment, the collecting of the conversation data and constructing an input vector of the recognition model according to the conversation data includes:
[0155] Determine the maximum sequence length of the recognition model input sequence;
[0156] Processing the conversation data according to the maximum sequence length to obtain a data sequence;
[0157] A position encoding vector is calculated for each position in the data sequence.
[0158] In one embodiment, inputting the input vector into an encoder of a recognition model to obtain a text representation vector includes:
[0159] Pruning the encoder of the recognition model to obtain an encoder with a preset number of layers;
[0160] Inputting the input vector to an encoder of a preset number of layers;
[0161] Attention interaction processing is performed on each layer of the encoder with the preset number of layers according to the attention mechanism to obtain a text representation vector.
[0162] In one embodiment, the step of identifying the intent category hit by the conversation data according to the category vector and the category threshold, and completing the training of the recognition model, includes:
[0163] If the category vector is greater than the category threshold, the intention is hit;
[0164] If the category vector is less than the category threshold, the intent is not hit.
[0165] In one embodiment, it further includes:
[0166] Add regularization layers to each encoder layer of the recognition model;
[0167] The alleged conversation data is processed through a self-learning algorithm to obtain pseudo-label data;
[0168] The recognition model is trained based on the regularization layer and the pseudo-label data.
[0169] An embodiment of the present invention provides a non-volatile computer-readable storage medium, which stores computer-executable instructions. The computer-executable instructions are executed by one or more processors, wherein the computer program implements the following steps when executed by the processor: collecting conversation data, constructing an input vector of a recognition model based on the conversation data; the input vector includes a text vector, a role paragraph vector, and a position encoding vector; inputting the input vector into an encoder of the recognition model to obtain a text representation vector; mapping the text representation vector to a category through full connection to obtain a category vector; identifying the intent category hit by the conversation data based on the category vector and the category threshold, and completing the training of the recognition model.
[0170] As an example, non-volatile storage media can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) as external cache memory. By way of illustration and not limitation, RAM can be obtained in many forms such as synchronous RAM (SRAM), dynamic RAM, (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The disclosed memory components or memories of the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.
[0171] In summary, the method, device, system and medium for training a dialogue reasoning recognition model disclosed in the present invention include: collecting dialogue data, constructing an input vector of a recognition model according to the dialogue data; the input vector includes a text vector, a role paragraph vector and a position encoding vector; inputting the input vector into the encoder of the recognition model to obtain a text representation vector; mapping the text representation vector to a category through full connection to obtain a category vector; identifying the intent category hit by the dialogue data according to the category vector and the category threshold, and completing the training of the recognition model. The present invention solves the problem of limited input text length in recognition by changing the encoding method and model structure, thereby performing contextual interaction between different roles in the dialogue process, so that the recognition model can perform both single-round reasoning and multi-round reasoning, thereby improving the recognition rate of the recognition model.
[0172] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the above-described equipment, devices and units can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. Those of ordinary skill in the art can appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.
[0173] In the several embodiments provided by the present invention, it should be understood that the disclosed equipment, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. Units with the same function may also be combined into one unit. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices or units, or may be an electrical, mechanical or other form of connection.
[0174] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present invention.
[0175] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0176] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a computer-readable storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the specification described in each embodiment of the present invention. The aforementioned computer-readable storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a magnetic disk or an optical disk.
[0177] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium, and the computer program can include the processes of the above-mentioned method embodiments when executed. The storage medium can be a memory, a disk, a floppy disk, a flash memory, an optical storage device, etc.
[0178] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.
Claims
1. A method for training a dialogue reasoning recognition model, characterized in that: include: Collecting conversation data, and constructing an input vector of a recognition model according to the conversation data; the input vector includes a text vector, a role paragraph vector, and a position encoding vector; Inputting the input vector into an encoder of a recognition model to obtain a text representation vector; Mapping the text representation vector to the category through full connection to obtain a category vector; The intent category hit by the conversation data is identified according to the category vector and the category threshold, and the training of the recognition model is completed.
2. The method for training a dialog reasoning recognition model according to claim 1, characterized in that: The collecting of the conversation data and constructing an input vector of the recognition model according to the conversation data includes: Extracting text data from the conversation data and preprocessing the text data; Convert the preprocessed text data into text vectors.
3. The method for training a dialog reasoning recognition model according to claim 2, characterized in that: The collecting of the conversation data and constructing an input vector of the recognition model according to the conversation data includes: Acquire dialogue text data including role identification from the dialogue data; Determining the number of dialogue roles in the dialogue text data; the dialogue roles have corresponding role identifiers; If the number of roles is 2, the segment identifier is allocated according to the role identifier; If the number of roles is greater than 2, the segment identifiers are assigned according to the dialogue order of the roles; The text paragraph of each role is converted into a corresponding role paragraph vector according to the role identifier and the segment identifier.
4. The method for training a dialog reasoning recognition model according to claim 3, characterized in that: The collecting of the conversation data and constructing an input vector of the recognition model according to the conversation data includes: Determine the maximum sequence length of the recognition model input sequence; Processing the conversation data according to the maximum sequence length to obtain a data sequence; A position encoding vector is calculated for each position in the data sequence.
5. The method for training a dialog reasoning recognition model according to claim 4, characterized in that: The step of inputting the input vector into an encoder of a recognition model to obtain a text representation vector comprises: Pruning the encoder of the recognition model to obtain an encoder with a preset number of layers; Inputting the input vector to an encoder of a preset number of layers; Attention interaction processing is performed on each layer of the encoder with the preset number of layers according to the attention mechanism to obtain a text representation vector.
6. The method for training a dialog reasoning recognition model according to claim 5, characterized in that: The step of identifying the intent category hit by the conversation data according to the category vector and the category threshold comprises: If the category vector is greater than the category threshold, the intention is hit; If the category vector is less than the category threshold, the intent is not hit.
7. The method for training a dialog reasoning recognition model according to claim 6, characterized in that: Also includes: Add regularization layers to each encoder layer of the recognition model; The alleged conversation data is processed through a self-learning algorithm to obtain pseudo-label data; The recognition model is trained based on the regularization layer and the pseudo-label data.
8. A dialogue reasoning recognition model training device, characterized in that: include: A data collection module, used to collect conversation data and construct an input vector of a recognition model based on the conversation data; A first vector module, used for inputting the input vector into an encoder of a recognition model to obtain a text representation vector; A second vector module is used to map the text representation vector to a category through full connection to obtain a category vector; The model training module is used to identify the intent category hit by the conversation data according to the category vector and the category threshold, and complete the training of the recognition model.
9. A dialogue reasoning recognition model training system, characterized in that: The system includes at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for training a dialogue reasoning recognition model as described in any one of claims 1-7.
10. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores computer-executable instructions, which, when executed by one or more processors, enable the one or more processors to execute the method for training a dialogue reasoning recognition model as described in any one of claims 1-7.
