Intention recognition model training method and device, and intention recognition method and device

Abstract

The application discloses a training method and device of an intention recognition model and an intention recognition method and device. A specific embodiment of the method comprises: processing an obtained text data set to obtain a word set in text data in the text data set; selecting a plurality of sub-sets comprising a first preset number of words from the word set, and for each word in each sub-set, selecting a second preset number of text data comprising the word from the text data set to obtain a plurality of training sets corresponding to the plurality of sub-sets one by one; for a training set in the plurality of training sets, taking text data in which a corresponding word is masked in the training set as input and taking the masked word as expected output to train an initial intention recognition model to obtain a trained intention recognition model. The application provides a training method of an intention recognition model based on self-supervision, and improves the diversity of a training task and the flexibility of model training.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of computer, in particular to a training method and device of an intent recognition model and an intent recognition method and device. BACKGROUND

[0002] The essence of the intent recognition task is a classification task, the input of which is a dialogue context containing a current question, and the output is an intent of the current question. The intent recognition task can be completed by a classification model. The best classification model at present is mainly a classification model based on a deep neural network. The general process is to extract features by using a deep neural network and classify the features by using a classifier. A deep neural network usually needs a large amount of labeled data for training. In actual application scenarios, the cost of obtaining a large amount of labeled data is high. A small sample learning method can be used to train a model by using a small amount of samples. Typically, a supervised meta-learning method is used to sample a data set of a subset of classes from a data set of a fixed task with a large number of classes to generate different meta-learning tasks.

[0003] The existing supervised meta-learning method still needs a certain amount of labeled data set to sample data from it to construct a training task, and has the problems of limited diversity of training tasks and lack of flexibility in training. SUMMARY

[0004] Embodiments of the present application provide a training method and device of an intent recognition model and an intent recognition method and device.

[0005] In a first aspect, a training method of an intent recognition model is provided, including: processing an obtained text data set to obtain a word set in text data in the text data set; selecting a plurality of sub-sets each including a first preset number of words from the word set, and for each word in each sub-set, selecting a second preset number of text data including the word from the text data set to obtain a plurality of training sets corresponding to the plurality of sub-sets one by one; for a training set in the plurality of training sets, taking text data in which a corresponding word is masked in the training set as input and taking the masked word as expected output, training an initial intent recognition model to obtain a trained intent recognition model.

[0006] In some embodiments, the training of the initial intent recognition model based on the training set in the plurality of training sets, and the obtaining of the trained intent recognition model, include: adding a classification layer at the end of the initial language model to obtain the initial intent recognition model; performing the following operations on the training set in the plurality of training sets until the intent recognition model is obtained: inputting the text data in which the corresponding words are masked in the training set as input, and inputting the masked words as expected output, optimizing a first parameter set representing parameters of the initial intent recognition model; updating a second parameter set based on the first parameter set, wherein the second parameter set represents parameters in a function of generating the classification layer according to the output of the initial language model.

[0007] In some embodiments, the first parameter set includes: parameters of all feedforward neural network layers between self-attention layers of the initial language model; parameters of a multi-layer perceptron in the initial intent recognition model; parameters representing weights and biases in the classification layer; the second parameter set includes: parameters of network structures in the language model except the feedforward neural network layers; parameters in the function of generating the classification layer according to the output of the initial language model; and a learning rate of the training process of the initial intent recognition model for the training set.

[0008] In some embodiments, the method further includes: determining the words to be masked in the text data in the supervision set selected from the text data set according to the received selection instruction; and the training of the initial intent recognition model based on the training set in the plurality of training sets, and the obtaining of the trained intent recognition model, include: after the initial intent recognition model completes the training of the last combined data set, determining, based on a preset sampling rate, a combined data set including part of the supervision set and part of the training set to be used to train the initial intent recognition model, inputting the text data in which the corresponding words are masked in the combined data set as input, and inputting the masked words as expected output, training the initial intent recognition model to obtain the trained intent recognition model.

[0009] In some embodiments, the method further includes: fine-tuning the intent recognition model according to a specific scenario corresponding to the text to be recognized, so as to process the text to be recognized by using the fine-tuned intent recognition model to obtain the intent expressed by the text to be recognized.

[0010] In some embodiments, the processing of the obtained text data set to obtain the word set in the text data in the text data set includes: filtering out words in the text data in the text data set whose word frequency is lower than a preset word frequency threshold to obtain the word set in the text data in the text data set.

[0011] Secondly, embodiments of this application provide an intent recognition method for natural language, comprising: acquiring a text to be recognized representing natural language; processing the text to be recognized through an intent recognition model to obtain the intent expressed by the text to be recognized, wherein the intent recognition model is trained according to the training method described in any implementation of the first aspect.

[0012] Thirdly, embodiments of this application provide a training apparatus for an intent recognition model, comprising: a processing unit configured to process an acquired text dataset to obtain a set of words in the text data of the text dataset; a selection unit configured to select multiple subsets including a first preset number of words from the word set, and for each word in each subset, select a second preset number of text data including that word from the text dataset to obtain multiple training sets corresponding one-to-one with the multiple subsets; and a training unit configured to train an initial intent recognition model for the training sets in the multiple training sets, using text data in the training set with the corresponding words masked as input and the masked words as the desired output, to obtain a trained intent recognition model.

[0013] In some embodiments, the training unit is further configured to: add a classification layer to the end of the initial language model to obtain an initial intent recognition model; for training sets in multiple training sets, perform the following operations until an intent recognition model is obtained: take the text data in the training set with the corresponding words masked as input and the masked words as the expected output to optimize a first set of parameters representing the parameters of the initial intent recognition model; update a second set of parameters based on the first set of parameters, wherein the second set of parameters represents the parameters in the function that generates the classification layer according to the output of the initial language model.

[0014] In some embodiments, the first set of parameters includes: parameters of all feedforward neural network layers between the self-attention layers of the initial language model; parameters of the multilayer perceptron in the initial intent recognition model; parameters representing weights and biases in the classification layer; the second set of parameters includes: parameters of the network structure in the language model excluding the feedforward neural network layers; parameters in the function that generates the classification layer based on the output of the initial language model; and the learning rate of the initial intent recognition model for the training process of the training set.

[0015] In some embodiments, the apparatus further includes: a determining unit configured to determine, according to a received selection instruction, the words to be masked in the text data of the supervision set selected from the text dataset; and a training unit further configured to: after the initial intent recognition model has completed training on the previous combined dataset, based on a preset sampling rate, determine the combined dataset that will be used to train the initial intent recognition model, which includes a partial supervision set and a partial training set, with the text data of the masked words in the combined dataset as input and the masked words as the expected output, to train the initial intent recognition model and obtain the trained intent recognition model.

[0016] In some embodiments, the above-described apparatus further includes: a fine-tuning unit configured to fine-tune the intent recognition model according to the specific scenario corresponding to the text to be recognized, so as to process the text to be recognized through the fine-tuned intent recognition model and obtain the intent expressed by the text to be recognized.

[0017] In some embodiments, the processing unit is further configured to: filter out words in the text data of the text dataset whose word frequency is lower than a preset word frequency threshold, to obtain a set of words in the text data of the text dataset.

[0018] Fourthly, embodiments of this application provide an intent recognition device for natural language, comprising: an acquisition unit configured to acquire a text to be recognized representing natural language; and a recognition unit configured to process the text to be recognized through an intent recognition model to obtain the intent expressed by the text to be recognized, wherein the intent recognition model is trained according to the training method described in any implementation of the first aspect.

[0019] Fifthly, embodiments of this application provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, it implements the method as described in any implementation of the first aspect.

[0020] In a sixth aspect, embodiments of this application provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect.

[0021] The intention recognition model training method and apparatus provided in this application embodiment process an acquired text dataset to obtain a set of words in the text data of the text dataset; select multiple subsets including a first preset number of words from the word set, and for each word in each subset, select a second preset number of text data including that word from the text dataset to obtain multiple training sets corresponding one-to-one with the multiple subsets; for the training sets in the multiple training sets, use the text data of the corresponding words in the training set with the words covered as input, and use the covered words as the expected output to train an initial intention recognition model to obtain a trained intention recognition model, thereby providing a training method for a self-supervised intention recognition model, improving the diversity of training tasks and the flexibility of model training. Attached Figure Description

[0022] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0023] Figure 1 This is an exemplary system architecture diagram in which one embodiment of this application can be applied;

[0024] Figure 2 This is a flowchart of an embodiment of the training method for the intent recognition model according to this application;

[0025] Figure 3 This is a schematic diagram of the intent recognition model according to this embodiment;

[0026] Figure 4 This is a schematic diagram illustrating an application scenario of the training method for the intent recognition model according to this embodiment;

[0027] Figure 5 This is a flowchart of yet another embodiment of the training method for the intent recognition model according to this application;

[0028] Figure 6 This is a flowchart of an embodiment of the intent recognition method according to this application;

[0029] Figure 7 This is a structural diagram of an embodiment of a training apparatus for an intent recognition model according to this application;

[0030] Figure 8 This is a structural diagram of an embodiment of the intent recognition device according to this application;

[0031] Figure 9 This is a schematic diagram of the structure of a computer system suitable for implementing the embodiments of this application. Detailed Implementation

[0032] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0033] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0034] Figure 1 An exemplary architecture 100 is shown, which can be used to train the intent recognition model of this application.

[0035] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. The communication connections between terminal devices 101, 102, and 103 form a network topology. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0036] Terminal devices 101, 102, and 103 can be hardware or software that supports network connectivity for data interaction and processing. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices that support network connectivity, information acquisition, interaction, display, and processing functions, including but not limited to smartphones, tablets, e-book readers, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices. They can be implemented as, for example, multiple software programs or software modules to provide distributed services, or as a single software program or software module. No specific limitations are imposed here.

[0037] Server 105 can be a server providing various services, such as a backend processing server that acquires text to be recognized sent by users through terminal devices 101, 102, and 103, and determines the intent expressed by the text through an intent recognition model. Furthermore, the server can also train the intent recognition model using a self-supervised approach. Optionally, the server can process the recognized intent to determine the target information the user needs and feed that information back to the terminal device. As an example, server 105 can be a cloud server.

[0038] It should be noted that a server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules (such as software programs or software modules used to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0039] It should also be noted that the training method and consciousness recognition method of the intent recognition model provided in the embodiments of this application can be executed by a server, by a terminal device, or by a combination of both. Accordingly, the training device of the intent recognition model and the various parts (e.g., various units) included in the intent recognition device can all be located in the server, all in the terminal device, or separately in the server and the terminal device.

[0040] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Any number of terminal devices, networks, and servers can be included depending on implementation needs. When the electronic devices on which the intent recognition model training method runs do not require data transmission with other electronic devices, the system architecture may consist only of the electronic devices (e.g., servers or terminal devices) on which the intent recognition model training method runs.

[0041] Continue to refer to Figure 2 The flowchart 200 illustrates an embodiment of a training method for an intent recognition model, including the following steps:

[0042] Step 201: Process the obtained text dataset to obtain the word set in the text data of the text dataset.

[0043] In this embodiment, the execution entity of the training method for the intent recognition model (e.g.) Figure 1 The server can retrieve text datasets from a remote location or from a local location via wired or wireless connections, and process the retrieved text datasets to obtain a set of words from the text data in the text dataset.

[0044] The text data included in the text dataset can be text information expressed in natural language in any scenario. As an example, in a customer service dialogue scenario, the aforementioned agent can pre-collect corpus involving real customer service dialogues. Customer service dialogues are conducted in a session dimension, where the user and customer service representative alternately send messages within a session. Using the sequence {q1, a1, ..., q...} n ,a n} represents a session. Where q i Indicates a user message iLet represent customer service messages, where 'i' represents the current round (an integer between 1 and n), and 'n' represents the total number of conversation rounds. The execution entity extracts user messages from each session, resulting in a text dataset denoted as {q1,…,q}. n}

[0045] After obtaining the text dataset, the aforementioned execution entity can determine the words in each text data in the text dataset, thereby obtaining the word set in each text data in the text dataset.

[0046] Here, the words in the word set are unique, and each word has a unique ID (Identity Document). After the aforementioned execution entity determines the words in each text data in the text dataset, it filters out duplicate words to obtain the word set.

[0047] In this embodiment, the aforementioned execution entity can determine the individual characters in each text data in the text dataset on a single-character basis, thereby obtaining a word set including the individual characters in each text data in the text dataset; alternatively, it can determine the words in each text data in the text dataset on a word-word basis, thereby obtaining a word set including the words in each text data in the text dataset. It can be understood that when using words as the unit, words include single words, compound words, and phrases (also known as word groups).

[0048] In some optional implementations of this embodiment, the execution entity can filter out words in the text data of the text dataset whose word frequency is lower than a preset word frequency threshold, thereby obtaining a set of words in the text data of the text dataset.

[0049] The preset word frequency threshold can be set according to the actual situation and is not limited here. When the word frequency in the text data is low, there is less text data in the text dataset that includes that word. Filtering it can improve the flexibility of randomly selecting corresponding text data in subsequent steps.

[0050] Step 202: Select multiple subsets from the word set that include a first preset number of words, and for each word in each subset, select a second preset number of text data from the text dataset that include that word, so as to obtain multiple training sets that correspond one-to-one with the multiple subsets.

[0051] In this embodiment, the execution entity can select multiple subsets from the word set, including a first preset number of words, and for each word in each subset, select a second preset number of text data from the text dataset, including that word, to obtain multiple training sets that correspond one-to-one with the multiple subsets.

[0052] The first and second preset quantities can be set according to the actual situation, and are not limited here.

[0053] It should be noted that the words in the subsets are randomly selected from the word set, and different subsets may contain the same words. For each word in a subset, multiple text data points containing that word are randomly selected from the text dataset to obtain multiple training sets corresponding to the multiple subsets. It is understandable that different training sets may contain the same text data.

[0054] In this embodiment, the text data in each training set can be divided into a training subset and a validation subset. The intent recognition model is fitted using training samples from the training subset, and the hyperparameters of the intent recognition model are adjusted using training samples from the validation subset, and the capabilities of the intent recognition model are initially evaluated.

[0055] Step 203: For the training set of multiple training sets, take the text data of the corresponding words in the training set with the words covered as input and the covered words as the expected output to train the initial intent recognition model and obtain the trained intent recognition model.

[0056] In this embodiment, for training sets in multiple training sets, the aforementioned execution entity uses the text data of the corresponding words masked in the training set as input and the masked words as the expected output to train the initial intent recognition model, thereby obtaining the trained intent recognition model.

[0057] In this process, the text data masks the corresponding word representations: when a word is selected as a training sample in the training set based on the text data, that word is masked.

[0058] In this embodiment, the intent recognition model can be any neural network language model with intent recognition function, including but not limited to BERT (Bidirectional Encoder Representations from Transformers), LSTM (Long Short Term Memory networks), RNN (Recurrent Neural Networks), etc.

[0059] Specifically, the sequence of text data X is represented as X = {x1, ..., x...} l}. Where the length of the text sequence is l, x iLet represent the i-th word in text data X, where i is an integer between 1 and 1. Based on the word set, the aforementioned execution entity converts each word in X into its corresponding ID in the word set, obtaining the numbered representation of the text sequence: W = {w1, ..., w...} l Taking the BERT model as an example of intent recognition, the above-mentioned execution entity will add the characters [CLS] and [SEP] to the beginning and end of the sequence respectively, and the final sequence is W = {w CLS w1,...,w l ,w SEP}

[0060] Regarding the numbered representations corresponding to the text sequence, the aforementioned execution entity masks the corresponding word numbers using a preset identifier to obtain the input for the initial intent recognition model. The preset identifier can be a pre-specified identifier. For example, the preset identifier is [m], and the sequence is W = {w}. CLS ,w1,…,w l ,w SEP If the corresponding masking word in} is the word corresponding to w1, then the masked sequence is W = {w1} CLS ,m,…,w l ,w SEP}

[0061] In this embodiment, the aforementioned execution entity can determine the loss value based on the intent output by the initial intent recognition model and the words masked in the input text data, and then use the stochastic gradient descent algorithm and the backpropagation algorithm to optimize the network parameters of the initial intent recognition model.

[0062] As an example, when the training process reaches a preset termination condition, the training process of the initial intent recognition model is considered complete, and the intent recognition model is determined to be obtained. The preset termination condition could be, for example, that the loss value is less than a preset loss value threshold, the training time exceeds a preset time threshold, or the number of training iterations exceeds a preset number of iterations threshold.

[0063] In some optional implementations of this embodiment, the execution entity can perform step 203 as follows:

[0064] First, a classification layer is added to the end of the initial language model to obtain the initial intent recognition model.

[0065] In this implementation, the initial language model can be any neural network language model, including but not limited to BERT, LSTM, and RNN models. The classification layer can be a classification layer composed of fully connected layers and softmax layers.

[0066] Second, for training sets from multiple training sets, perform the following operations until the intent recognition model is obtained:

[0067] First, the first parameter set representing the parameters of the initial intent recognition model is optimized by taking the text data of the training set with the corresponding words covered as input and the covered words as the expected output.

[0068] In this implementation, the training process of the initial intent recognition model includes an inner loop process and an outer loop process. The inner loop process represents the iterative process of training the initial intent recognition model using training samples in the same training set, while the outer loop process represents the iterative process of training the initial intent recognition model using different training sets.

[0069] The inner loop process optimizes the first set of parameters that characterize the initial intent recognition model. The outer loop process optimizes the parameters in the second set of parameters.

[0070] Then, based on the first parameter set, the second parameter set is updated. The second parameter set represents the parameters in the function that generates the classification layer based on the output of the initial language model.

[0071] As an example, for a BERT model with parameter π, using the symbol f π (X) represents the output of its last layer. For a training task t corresponding to a training set, there are M categories, each category corresponding to a word. The k samples in the support set of category m (one of the M categories) are represented as: {X} 1m ,…,X km The generation process of the softmax layer is as follows:

[0072]

[0073] p(y|X)=Softmax{W t h φ (f π (X))+b t}

[0074] in, Characteristic weights, The representation biases are all represented by vectors for each class. It is assembled from multiple parts, and the output dimension is d-dimensional; g ψ This represents the function that generates the classification layer based on the output of the initial language model, where ψ represents the parameters of this function; h φ This represents a multilayer perceptron with parameter φ. The output f of the initial language model... π (X) is first transformed by a multilayer perceptron, and then classified by a classification layer to determine the classification information about the intent. y represents the label of the input text data, that is, the masked words.

[0075] In some optional implementations of this embodiment, to accelerate the convergence speed of the model during training, the parameters of the initial language model, i.e., π, are split into two parts for training. The two parts of parameters are then sequentially assigned to the first parameter set and the second parameter set mentioned above. One part consists of the parameters of all feedforward neural network layers between the attention layer and the other part consists of the remaining parameters in the initial language model, i.e., the parameters of the network structure excluding the feedforward neural network layers.

[0076] In this implementation, the first parameter set includes: parameters of all feedforward neural network layers between the self-attention layers of the initial language model; parameters of the multilayer perceptron in the initial intent recognition model; and parameters representing weights and biases in the classification layer. The second parameter set includes: parameters of the network structure in the language model excluding the feedforward neural network layers; parameters in the function that generates the classification layer based on the output of the initial language model; and the learning rate of the initial intent recognition model during training on the training set.

[0077] The parameter optimization process for the initial intent recognition model is as follows:

[0078]

[0079]

[0080] Where θ:={π1,φ,W t ,b t} represents the first set of parameters, π1, φ, W t ,b t The parameters of all feedforward neural network layers between the initial language model's attention layer and the initial intent recognition model's multilayer perceptron parameters are represented sequentially. The parameters representing the weights and biases in the classification layer are also represented sequentially. Θ:={π2,ψ,α} represents the second parameter set, where π2,ψ,α represent the parameters of the network structure in the language model excluding the feedforward neural network layers. The parameters in the function that generates the classification layer are generated based on the output of the initial language model. The learning rate of the initial intent recognition model during the training process on the training set is also represented.

[0081] In addition, D tr D val The training subset and the validation subset are represented in that order, respectively. Indicates task T i The supervised loss is β; β represents the learning rate during training. This indicates that the average value is calculated.

[0082] See also Figure 3The diagram shows a schematic of the network structure of an intent recognition model 300. The intent recognition model includes a Transformer network 301, a parameter generator 302, a fully connected layer 303, and a softmax layer 304. The training process of the intent recognition model includes an inner loop and an outer loop. Each inner loop can update and optimize the parameters in the initial parameter set of the intent recognition model using a batch of training samples. After each inner loop, the outer loop optimizes the parameters in the second parameter set. Before the inner loop, a batch of training samples is input into the Transformer network 301, and the parameter generator 302 generates the network parameters for the softmax layer 304. The training samples can be flexibly divided into training data and validation data; the loss obtained based on the training data is the training loss, and the loss obtained based on the validation data is the validation loss.

[0083] See also Figure 4 , Figure 4 This is a schematic diagram 400 illustrating an application scenario of the training method for the intent recognition model according to this embodiment. Figure 4 In the application scenario, server 401 first collects a text dataset A = {a1, a2, a3, ... a...} Q There are Q text data sets in total. Then, the text dataset A is processed to obtain a word set B = {b1, b2, b3, ..., b} from the text data in the text dataset A. P The word set B contains P words. Then, server 401 selects multiple subsets B1 = {b1, b5, b6, b7, b8, b9, b1, b1, b1, b2, b3, b4, b5, b6, b7, b8, b9, b1, b1, b1, b2, b3 ...2, b3, b1, b2, b3, b1, b2, b 10}, B2={b3,b7,b 30}, B3 = {b5, b7, b 20}; and for each word in each subset, a second predetermined number of text data points containing that word are selected from the text dataset to obtain multiple training sets L1, L2, L3 corresponding one-to-one with the multiple subsets. Taking L1 as an example, the text data in it includes: 5 text data points a1, a8, a... 14 a 32 a 43 This includes five text data points: a2, a3, a5, a6, a7, a8, a9, a1, a2, a3, a4, a5, a5, a6, a7, a8, a9 ... 15 a 23 a 25 , including b 10 Five text data points: a7, a9, a 16 a 29 a 37There are a total of 15 (3*5) text data points. Finally, for the training set of multiple training sets, the initial intent recognition model is trained by taking the text data with the corresponding words masked in the training set as input and the masked words as the expected output, thus obtaining the trained intent recognition model. As an example, the text data a2, a3, a4, and a5 in the training set L1 are used. 15 a 23 a 25 Mask b5 and use it as input data, with b5 as the expected output, to train the initial intent recognition model.

[0084] The method provided in the above embodiments of this application processes the acquired text dataset to obtain a set of words in the text data of the text dataset; selects multiple subsets including a first preset number of words from the word set, and for each word in each subset, selects a second preset number of text data including that word from the text dataset to obtain multiple training sets corresponding one-to-one with the multiple subsets; for the training sets in the multiple training sets, the text data of the corresponding words in the training set with the words covered are used as input, and the covered words are used as the expected output to train an initial intent recognition model to obtain a trained intent recognition model, thereby providing a training method for a self-supervised intent recognition model, improving the diversity of training tasks and the flexibility of model training.

[0085] In some optional implementations of this embodiment, the intent recognition model is fine-tuned according to the specific scenario corresponding to the text to be recognized, so as to process the text to be recognized through the fine-tuned intent recognition model and obtain the intent expressed by the text to be recognized.

[0086] Continue to refer to Figure 5 The illustration 500 shows an embodiment of a training method for an intent recognition model according to this application, including the following steps:

[0087] Step 501: Process the obtained text dataset to obtain the word set in the text data of the text dataset.

[0088] Step 502: Select multiple subsets from the word set that include a first preset number of words, and for each word in each subset, select a second preset number of text data from the text dataset that include that word, so as to obtain multiple training sets that correspond one-to-one with the multiple subsets.

[0089] In this embodiment, the above-mentioned execution entity can execute the above-mentioned steps 501-502 in the manner of steps 201-202, which will not be repeated here.

[0090] Step 503: Based on the received selection instructions, determine the words to be masked in the text data of the supervised set selected from the text dataset.

[0091] In this embodiment, the execution entity of the training method for the intent recognition model (e.g.) Figure 1 The server in the dataset can determine the words to be masked from the text data in the supervised set selected from the text dataset based on the received selection instructions.

[0092] In this embodiment, text data and words to be masked in the text data can be manually selected to form a supervision set.

[0093] Step 504: After the initial intent recognition model completes the training of the previous combined dataset, based on the preset sampling rate, determine the combined dataset that will be used to train the initial intent recognition model, which includes a part of the supervision set and a part of the training set. Take the text data of the combined dataset with the corresponding words covered as input and the covered words as the expected output to train the initial intent recognition model and obtain the trained intent recognition model.

[0094] In this embodiment, the aforementioned execution entity can determine multiple combined datasets, including partial supervision sets and partial training sets, based on a preset sampling rate. For each combined dataset, the initial intent recognition model is trained by using the text data with the corresponding words masked in the combined dataset as input and the masked words as the expected output, thus obtaining the trained intent recognition model.

[0095] The preset sampling rate is a value between 0 and 1, which can be set according to the actual situation and is not limited here.

[0096] As an example, if the preset sampling rate is 0.5, then the above-mentioned execution entity obtains a portion (specifically, half) of the training set from the training set at a sampling rate of 0.5, and obtains a portion (specifically, half) of the supervision set from the supervision set at a sampling rate of 0.5 (1-0.5), thus obtaining the combined dataset.

[0097] It should be noted that, without conflict, the training process of the initial intent recognition model based on the above combined training set can adopt or combine any of the implementation methods in step 203 to obtain more implementation methods.

[0098] As an example, during the training process of the initial intent recognition model based on the above combined training set, the parameters to be optimized can still be divided into a first data set and a second data set. The initial intent recognition model is trained by using an inner loop process to optimize the first parameter set representing the parameters of the initial intent recognition model, and an outer loop process to optimize the parameters in the second parameter set.

[0099] As can be seen from this embodiment, with Figure 2 Compared with the corresponding embodiments, the process 500 of the training method of the intent recognition model in this embodiment specifically describes the process of training the initial intent recognition model based on the combined dataset obtained by the preset sampling rate, which further improves the flexibility of model training.

[0100] Continue to refer to Figure 6 The flowchart 600 illustrates an embodiment of a training method for an intent recognition model, including the following steps:

[0101] Step 601: Obtain the text to be recognized that represents natural language.

[0102] In this embodiment, the execution subject of the intent identification method (e.g.) Figure 1 The server in the middle can obtain the text to be recognized, which represents natural language.

[0103] The text to be identified can be any text representing natural language. As an example, in a customer service dialogue scenario, the text to be identified is the information sent by the user.

[0104] Step 602: Process the text to be recognized using the intent recognition model to obtain the intent expressed by the text to be recognized.

[0105] In this embodiment, the aforementioned execution entity can process the text to be recognized using an intent recognition model to obtain the intent expressed by the text. The intent recognition model is trained according to the training methods described in embodiments 200 and 500.

[0106] In some optional implementations of this embodiment, after determining the intent expressed by the text to be identified, the aforementioned execution entity can perform information processing to determine the target information that the user needs to know.

[0107] Continuing with the customer service dialogue scenario as an example, the user's text to be recognized is "Height 175cm, weight 70KG, what size pants would fit?" After determining that the user's intent is "size", the aforementioned execution entity can process the information, determine the pants size corresponding to the user's height and weight information, generate target information about the size, and feed it back to the user's terminal device.

[0108] The method provided in the above embodiments of this application obtains the text to be identified, which represents natural language; processes the text to be identified through an intent recognition model, and obtains the intent expressed by the text to be identified, thereby providing an intent recognition method and enriching the intent recognition methods for natural language.

[0109] Continue to refer to Figure 7As an implementation of the methods shown in the above figures, this application provides an embodiment of a training device for an intent recognition model, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0110] like Figure 7 As shown, the training device for the intent recognition model includes: a processing unit 701 configured to process the acquired text dataset to obtain a set of words in the text data of the text dataset; a selection unit 702 configured to select multiple subsets including a first preset number of words from the word set, and for each word in each subset, select a second preset number of text data including that word from the text dataset to obtain multiple training sets corresponding one-to-one with the multiple subsets; and a training unit 703 configured to train an initial intent recognition model for the training sets in the multiple training sets, using the text data in the training set with the corresponding words covered as input and the covered words as the expected output, to obtain a trained intent recognition model.

[0111] In some optional implementations of this embodiment, the training unit 703 is further configured to: add a classification layer to the end of the initial language model to obtain an initial intent recognition model; for training sets in multiple training sets, perform the following operations until an intent recognition model is obtained: take the text data of the corresponding words masked in the training set as input, take the masked words as the expected output, and optimize the first parameter set representing the parameters of the initial intent recognition model; based on the first parameter set, update the second parameter set, wherein the second parameter set represents the parameters in the function that generates the classification layer according to the output of the initial language model.

[0112] In some optional implementations of this embodiment, the first parameter set includes: parameters of all feedforward neural network layers between the self-attention layers of the initial language model; parameters of the multilayer perceptron in the initial intent recognition model; parameters representing weights and biases in the classification layer; the second parameter set includes: parameters of the network structure in the language model excluding the feedforward neural network layers; parameters in the function that generates the classification layer based on the output of the initial language model; and the learning rate of the initial intent recognition model for the training process of the training set.

[0113] In some optional implementations of this embodiment, the above apparatus further includes: a determining unit (not shown in the figure), configured to determine, according to the received selection instruction, the words to be masked in the text data of the supervision set selected from the text dataset; and a training unit, further configured to: after the initial intent recognition model completes the training of the previous combined dataset, based on a preset sampling rate, determine the combined dataset that trains the initial intent recognition model, including a part of the supervision set and a part of the training set, using the text data of the masked words in the combined dataset as input and the masked words as the expected output, to train the initial intent recognition model and obtain the trained intent recognition model.

[0114] In some optional implementations of this embodiment, the above-mentioned device further includes: a fine-tuning unit (not shown in the figure), configured to fine-tune the intent recognition model according to the specific scenario corresponding to the text to be recognized, so as to process the text to be recognized through the fine-tuned intent recognition model and obtain the intent expressed by the text to be recognized.

[0115] In some embodiments, the processing unit 701 is further configured to: filter out words in the text data of the text dataset whose word frequency is lower than a preset word frequency threshold, to obtain a set of words in the text data of the text dataset.

[0116] In this embodiment, the processing unit in the training device of the intent recognition model processes the acquired text dataset to obtain a set of words in the text data of the text dataset; the selection unit selects multiple subsets including a first preset number of words from the word set, and for each word in each subset, selects a second preset number of text data including that word from the text dataset to obtain multiple training sets corresponding one-to-one with the multiple subsets; the training unit trains the initial intent recognition model for the training sets in the multiple training sets, using the text data of the training set with the corresponding words covered as input and the covered words as the expected output, to obtain the trained intent recognition model, thereby providing a training device for a self-supervised intent recognition model, improving the diversity of training tasks and the flexibility of model training.

[0117] Continue to refer to Figure 8 As an implementation of the methods shown in the above figures, this application provides an embodiment of a training device for an intent recognition model, which is similar to... Figure 6 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0118] like Figure 8As shown, the intention recognition device for natural language includes: an acquisition unit 801 configured to acquire a text to be recognized representing natural language; and a recognition unit 802 configured to process the text to be recognized through an intention recognition model to obtain the intention expressed by the text to be recognized. The intention recognition model is trained according to the training methods described in embodiments 200 and 500.

[0119] In this embodiment, the acquisition unit in the intent recognition device acquires the text to be recognized, which represents natural language; the recognition unit processes the text to be recognized through the intent recognition model to obtain the intent expressed by the text to be recognized, thereby providing an intent recognition device and enriching the intent recognition methods for natural language.

[0120] The following is for reference. Figure 9 It illustrates a device suitable for implementing embodiments of this application (e.g., Figure 1 The diagram shows the structure of the computer system 900 of the devices 101, 102, 103, and 105. Figure 9 The device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0121] like Figure 9 As shown, the computer system 900 includes a processor (e.g., CPU, Central Processing Unit) 901, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 902 or programs loaded from storage section 908 into random access memory (RAM) 903. The RAM 903 also stores various programs and data required for the operation of the system 900. The processor 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0122] The following components are connected to I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to I / O interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 910 as needed so that computer programs read from it can be installed into storage section 908 as needed.

[0123] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 909, and / or installed from removable medium 911. When the computer program is executed by processor 901, it performs the functions defined in the methods of this application.

[0124] It should be noted that the computer-readable medium of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0125] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the client computer, partially on the client computer, as a standalone software package, partially on the client computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the client computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0126] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0127] The units described in the embodiments of this application can be implemented in software or hardware. The described units can also be housed in a processor; for example, they can be described as: a processor including a processing unit, a selection unit, and a training unit; or as: a processor including an acquisition unit and a recognition unit. The names of these units do not necessarily limit the unit itself. For example, a training unit can also be described as "a unit that, for a training set in multiple training sets, uses text data with corresponding words masked in the training set as input, and uses the masked words as the desired output, trains an initial intent recognition model to obtain a trained intent recognition model."

[0128] In another aspect, this application also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the computer device to: process an acquired text dataset to obtain a set of words in the text data of the text dataset; select multiple subsets from the word set including a first preset number of words, and for each word in each subset, select a second preset number of text data including that word from the text dataset to obtain multiple training sets corresponding one-to-one with the multiple subsets; for each training set in the multiple training sets, using text data in the training set with the corresponding words masked as input and the masked words as the desired output, train an initial intent recognition model to obtain a trained intent recognition model. The computer device also causes to: acquire text to be recognized representing natural language; process the text to be recognized through the intent recognition model to obtain the intent expressed by the text to be recognized.

[0129] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A method for training an intent recognition model, comprising: The acquired text dataset is processed to obtain a set of words in the text data of the text dataset; Multiple subsets containing a first preset number of words are selected from the word set, and for each word in each subset, a second preset number of text data containing that word are selected from the text dataset to obtain multiple training sets that correspond one-to-one with the multiple subsets; For the training set in the multiple training sets, the initial intent recognition model is trained by taking the text data of the corresponding words in the training set with the words covered as input and the covered words as the expected output, to obtain the trained intent recognition model, including: For the training sets in the multiple training sets, perform the following operations until the intent recognition model is obtained: During the inner loop, the text data of the corresponding words in the training set that are covered up is used as input, and the covered words are used as the expected output to optimize the first parameter set that represents the parameters of the initial intent recognition model. During the outer loop, the second parameter set is updated based on the first parameter set, wherein the second parameter set represents the parameters in the function of the classification layer generated based on the output of the initial language model.

2. The method according to claim 1, wherein, The step of training an initial intent recognition model using the text data with the corresponding words masked in the training sets as input and the masked words as the expected output, to obtain a trained intent recognition model, further includes: A classification layer is added to the end of the initial language model to obtain the initial intent recognition model.

3. The method according to claim 1, wherein, The first parameter set includes: The parameters of all feedforward neural network layers between the self-attention layers of the initial language model; The parameters of the multilayer perceptron in the initial intent recognition model; The parameters representing weights and biases in the classification layer; The second parameter set includes: The parameters of the network structure in the language model, excluding the feedforward neural network layer; The parameters in the function of the classification layer are generated based on the output of the initial language model. The initial intent recognition model's learning rate for the training process of the training set.

4. The method according to claim 1, wherein, Also includes: Based on the received selection instructions, determine the words to be masked in the text data of the supervision set selected from the text dataset; as well as The process of training an initial intent recognition model using the text data with the corresponding words masked in the training sets as input and the masked words as the expected output, to obtain a trained intent recognition model, includes: After the initial intent recognition model completes training on the previous combined dataset, based on a preset sampling rate, a combined dataset, including a portion of the supervision set and a portion of the training set, is determined to be used to train the initial intent recognition model. The initial intent recognition model is trained by taking the text data with the corresponding words masked in the combined dataset as input and the masked words as the expected output, thus obtaining the trained intent recognition model.

5. The method according to claim 1, wherein, Also includes: Based on the specific scenario corresponding to the text to be identified, the intent recognition model is fine-tuned so that the text to be identified can be processed through the fine-tuned intent recognition model to obtain the intent expressed by the text to be identified.

6. The method according to claim 1, wherein, The process of processing the acquired text dataset to obtain a set of words in the text data of the text dataset includes: Filter out words in the text data of the text dataset whose word frequency is lower than a preset word frequency threshold to obtain the word set in the text data of the text dataset.

7. A method for intent recognition in natural language, comprising: Obtain the text to be identified, representing natural language; The intent recognition model is used to process the text to be recognized to obtain the intent expressed by the text to be recognized, wherein the intent recognition model is trained by the training method according to any one of claims 1-6.

8. A training device for an intent recognition model, comprising: The processing unit is configured to process the acquired text dataset to obtain a set of words in the text data of the text dataset; The selection unit is configured to select multiple subsets from the word set, including a first preset number of words, and for each word in each subset, select a second preset number of text data including that word from the text dataset to obtain multiple training sets corresponding one-to-one with the multiple subsets; The training unit is configured to train an initial intent recognition model for the training sets in the plurality of training sets, using text data of the corresponding words in the training sets with the corresponding words masked as input and the masked words as the expected output, to obtain a trained intent recognition model. This includes: for the training sets in the plurality of training sets, performing the following operations until the intent recognition model is obtained: in the inner loop, using text data of the corresponding words in the training sets with the corresponding words masked as input and the masked words as the expected output, optimizing a first parameter set representing the parameters of the initial intent recognition model; in the outer loop, updating a second parameter set based on the first parameter set, wherein the second parameter set represents the parameters in the function of the classification layer generated according to the output of the initial language model.

9. The apparatus according to claim 8, wherein, The training unit is further configured as follows: A classification layer is added to the end of the initial language model to obtain the initial intent recognition model.

10. The apparatus according to claim 9, wherein, The first parameter set includes: The parameters of all feedforward neural network layers between the self-attention layers of the initial language model; The parameters of the multilayer perceptron in the initial intent recognition model; The parameters representing weights and biases in the classification layer; The second parameter set includes: The parameters of the network structure in the language model, excluding the feedforward neural network layer; The parameters in the function of the classification layer are generated based on the output of the initial language model. The initial intent recognition model's learning rate for the training process of the training set.

11. The apparatus according to claim 8, wherein, Also includes: The determining unit is configured to determine, based on the received selection instruction, the words to be masked in the text data of the supervision set selected from the text dataset; as well as The training unit is further configured as follows: After the initial intent recognition model completes the training of the previous combined dataset, based on a preset sampling rate, a combined dataset including a partial supervision set and a partial training set is determined to be used to train the initial intent recognition model. The text data with the corresponding words covered in the combined dataset is used as input, and the covered words are used as the expected output. The initial intent recognition model is then trained to obtain the trained intent recognition model.

12. The apparatus according to claim 8, wherein, Also includes: The fine-tuning unit is configured to fine-tune the intent recognition model according to the specific scenario corresponding to the text to be recognized, so as to process the text to be recognized through the fine-tuned intent recognition model and obtain the intent expressed by the text to be recognized.

13. The apparatus according to claim 8, wherein, The processing unit is further configured to: Filter out words in the text data of the text dataset whose word frequency is lower than a preset word frequency threshold to obtain the word set in the text data of the text dataset.

14. An intent recognition device for natural language, comprising: The acquisition unit is configured to acquire the text to be recognized, which represents natural language; The recognition unit is configured to process the text to be recognized through an intent recognition model to obtain the intent expressed by the text to be recognized, wherein the intent recognition model is trained by the training method according to any one of claims 1-6.

15. A computer-readable medium having a computer program stored thereon, wherein, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.

16. An electronic device comprising: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7.

Images