Text matching model training method, intent recognition method and device

Abstract

The embodiment of the application discloses a kind of training method of text matching model, intent identification method and device, method includes: based on the standard question sentence in knowledge base constructs training sample set, each training sample in training sample set includes: standard question sentence, similar sample, dissimilar sample;During the iteration training process of diversity stage using training sample set to initial text matching model, each training sample is input initial text matching model after being carried out mixed coding processing based on linear interpolation, and the first distance between standard question sentence and similar sample is obtained, and the second distance between standard question sentence and dissimilar sample;According to first distance, second distance and loss function, adjust the model parameter of text matching model, until loss function meets set condition, obtain the text matching model after training of diversity stage.The application is conducive to improving the accuracy of text matching.

Description

Technical Field

[0001] This application relates to the field of machine learning technology, and in particular to a method for training a text matching model, an intent recognition method, and an apparatus. Background Technology

[0002] A voice robot dialogue system is a type of question-and-answer (QA) system. Typically, a customer poses a question to the voice robot, which then provides the answer. Accurate intent recognition of the customer's question is a prerequisite for providing high-quality answers and also represents a significant challenge in building such a system.

[0003] With the development of deep learning, several methods suitable for text matching have been proposed, among which two classic approaches are representation-based and interaction-based. Representation-based methods encode each of the two text segments to obtain their respective feature vectors, and then use a similarity calculation function or a correlation structure to obtain the final matching relationship. Interaction-based methods involve interacting with the two text segments at different granularities (word-level, phrase-level, etc.), and then aggregating the matching results at each granularity using a structure to obtain a single feature vector as the final matching relationship.

[0004] Regardless of the text matching method used, a typical voice chatbot system needs to recognize various customer intents. In a cold start scenario, the voice chatbot can only train its model based on similar questions entered by customer service personnel in a knowledge base. This training data is typically limited, consisting of only a few to a few dozen data points, insufficient to support training and resulting in low text matching accuracy. Furthermore, the voice chatbot's input relies on upstream speech recognition results, which often contain numerous errors such as misspellings and typos, further reducing text matching accuracy and exacerbating the difficulty of intent recognition. How to fully utilize limited training data to improve text matching accuracy and reduce the adverse effects of inaccurate speech recognition results on intent recognition remains a significant technical challenge for the industry. Summary of the Invention

[0005] The purpose of this application is to provide a method for training a text matching model, as well as an intent recognition method and apparatus, to improve the accuracy of text matching and intent recognition.

[0006] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0007] Firstly, a training method for a text matching model is provided, including:

[0008] A training sample set is constructed based on standard question statements in the knowledge base. Each training sample in the training sample set includes: the standard question statement, similar samples of the standard question statement, and dissimilar samples of the standard question statement. The standard question statement is a statement manually entered as a standard question. The similar samples are used to indicate statements that are similar to the standard question statement, and the dissimilar samples are used to indicate statements that are not similar to the standard question statement.

[0009] During the iterative training of the initial text matching model in the diversity stage using the training sample set, each training sample is processed by hybrid encoding based on linear interpolation and then input into the initial text matching model. The output is the first distance between the standard question statement and the sample of the same type, and the second distance between the standard question statement and the sample of the opposite type.

[0010] The model parameters of the text matching model are adjusted based on the output first distance, second distance, and loss function of the text matching model until the loss function of the text matching model meets the set conditions, thus obtaining the text matching model trained in the diversity stage.

[0011] Secondly, an intent recognition method is provided, including:

[0012] Obtain the speech recognition statement corresponding to the speech to be recognized;

[0013] The speech recognition statement and the standard question statement in the knowledge base are input into a pre-trained text matching model, and the matching degree between the speech recognition statement and the standard question statement is output; wherein, the text matching model is trained according to the text matching model training method described in the first aspect;

[0014] Based on the matching degree between the speech recognition statement and each standard question statement, the standard question statement that matches the speech to be recognized is determined as the intent recognition result.

[0015] Thirdly, a training device for a text matching model is provided, comprising:

[0016] The construction module is used to construct a training sample set based on standard question statements in the knowledge base. Each training sample in the training sample set includes: the standard question statement, similar samples of the standard question statement, and dissimilar samples of the standard question statement. The standard question statement is a statement manually entered as a standard question. The similar samples are used to indicate statements similar to the standard question statement, and the dissimilar samples are used to indicate statements dissimilar to the standard question statement.

[0017] The diversity training module is used to perform hybrid encoding processing based on linear interpolation on each training sample and input it into the initial text matching model during the iterative training of the initial text matching model in the diversity stage using the training sample set, and output the first distance between the standard question statement and the sample of the same type, and the second distance between the standard question statement and the sample of the opposite type.

[0018] The first adjustment module is used to adjust the model parameters of the text matching model according to the output first distance, second distance and loss function of the text matching model until the loss function of the text matching model meets the set conditions, so as to obtain the text matching model trained in the diversity stage.

[0019] Fourthly, an intent recognition device is provided, comprising:

[0020] The acquisition module is used to acquire the speech recognition statement corresponding to the speech to be recognized;

[0021] The text matching module is used to input the speech recognition statement and the standard question statement in the knowledge base into a pre-trained text matching model, and output the matching degree between the speech recognition statement and the standard question statement; wherein, the text matching model is trained according to the training method of the text matching model as described in the first aspect;

[0022] The intent recognition module is used to determine the standard question statement that matches the speech to be recognized as the intent recognition result based on the matching degree between the speech recognition statement and each standard question statement.

[0023] Fifthly, an electronic device is provided, comprising:

[0024] processor;

[0025] Memory used to store the processor's executable instructions;

[0026] The processor is configured to execute the instructions to implement the method as described in the first aspect or the second aspect.

[0027] A sixth aspect provides a computer-readable storage medium that, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform the method as described in the first or second aspect.

[0028] The text matching model training scheme provided in this application addresses the problem of limited training data, which can only be trained based on similar question data in a knowledge base. It augments each training sample in the training sample set by adding dissimilar samples of the standard question statement in addition to standard question statements and their similar samples. During iterative training in the diversity phase using the training sample set, the standard question statements, similar samples, and dissimilar samples in the training samples are augmented using hybrid encoding based on linear interpolation before being input into the initial text matching model. The model parameters are adjusted by comparing samples. By augmenting each training sample in the training sample set, and considering the diverse range of questions raised by clients, the hybrid encoding based on linear interpolation enhances the diversity of the model's input data on a limited training basis. This, in turn, effectively improves the accuracy of the text matching model through iterative training in the diversity phase.

[0029] The intent recognition scheme provided in this application uses a text matching model trained by the above-mentioned text matching model training method to perform text matching between the speech recognition statement corresponding to the speech to be recognized and the standard question statement in the knowledge base. Because the training method of the text matching model addresses the problem of limited training data, it expands the training sample set by adding annotated heterogeneous samples of question statements in addition to standard question statements and similar samples. Furthermore, it employs hybrid encoding processing based on linear interpolation for data augmentation during the diversity stage, thereby increasing the diversity of the model's input data. This improves the accuracy of the text matching model, making the trained text matching model suitable for intent recognition scenarios where the speech to be recognized has diverse meanings. Based on the accurate text matching of the text matching model for various speech to be recognized, it can accurately identify the true customer intent expressed by the speech to be recognized, thus improving the accuracy of intent recognition. Attached Figure Description

[0030] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0031] Figure 1 A flowchart illustrating a training method for a text matching model provided in one embodiment of this application;

[0032] Figure 2 A schematic diagram of a triplet network structure is provided as an embodiment of this application;

[0033] Figure 3aA schematic diagram of model iterative training during the baseline phase provided in one embodiment of this application;

[0034] Figure 3b A schematic diagram of model iterative training in a diversity phase provided for one embodiment of this application;

[0035] Figure 3c A schematic diagram of progressive stage model iterative training provided for one embodiment of this application;

[0036] Figure 4 A schematic diagram of a hybrid encoding process based on linear interpolation is provided for one embodiment of this application;

[0037] Figure 5 A flowchart of an intent recognition method provided as an embodiment of this application;

[0038] Figure 6 A flowchart illustrating the processing involved in a voice robot dialogue, provided as an embodiment of this application;

[0039] Figure 7 A schematic diagram of the structure of a training device for a text matching model provided in one embodiment of this application;

[0040] Figure 8 A schematic diagram of an intent recognition device provided for one embodiment of this application;

[0041] Figure 9 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this document.

[0043] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0044] As mentioned earlier, in the business scenario of voice robot dialogue, suitable methods for text matching include representation-based methods and interaction-based methods. Recent research on representation-based methods has mainly focused on two aspects: firstly, using different network structures to strengthen the encoder to obtain better feature vector representations; and secondly, using different similarity calculation functions for modeling. Interaction-based methods typically focus on the interaction methods of text pairs, enabling the model to extract more effective interaction information. However, representation-based methods do not pay sufficient attention to the key semantic information in the text and fail to capture the core meaning. While interaction-based methods can cater to the text matching characteristics of specific data or scenarios, the lack of large-scale pre-trained models to support basic semantics leads to a decrease in the question matching success rate of the voice robot dialogue system over time.

[0045] In business scenarios involving voice robot dialogue, the text matching models used by voice robots suffer from several problems. First, training a text matching model is a typical few-shot learning task, as the knowledge base of each type of voice robot typically contains limited data on similar questions. Few-shot learning refers to scenarios with many categories but limited training data for each category. By learning from a small amount of training data, the model gains the ability to learn and generalize. Existing technologies often use data augmentation to mitigate this situation, but they fail to consider the differences between augmented and original data, as well as the varying effects of different data augmentation methods. They treat original and augmented data equally, directly using both for training, which negatively impacts the accuracy of text matching. Second, the data input to the voice robot during prediction comes from speech recognition results. Audio Speech Recognition (ASR) is a technology that converts human speech into text. Speech recognition results often include misspellings and typos, which reduces the accuracy of text matching and increases the difficulty of intent recognition. Existing technologies do not consider this factor.

[0046] In view of this, the embodiments of this application aim to provide a training method for a text matching model, as well as an intent recognition method and apparatus, to solve the problem of low accuracy of text matching caused by limited training data; and further solve the problem of inaccurate text matching and thus difficulty in intent recognition caused by inaccurate input speech recognition results.

[0047] It should be understood that the text matching model training method and intent recognition method provided in the embodiments of this application can both be executed by an electronic device or by software installed in an electronic device, specifically by a terminal device or a server device. The text matching model training method and intent recognition method can be executed by the same electronic device, or they can be executed by different electronic devices.

[0048] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0049] Please refer to Figure 1 The following is a flowchart illustrating a method for training a text matching model, provided as an embodiment of this application. The method may include:

[0050] S101. Construct a training sample set based on standard question statements in the knowledge base. Each training sample in the training sample set includes: the standard question statement, similar samples of the standard question statement, and dissimilar samples of the standard question statement.

[0051] Knowledge bases typically contain raw text data pre-entered by humans (customer service personnel), including a certain number of standard question statements and a certain number of similar sentences for each standard question statement. As mentioned earlier, in existing technologies, voice robots can only train models based on similar question data in the knowledge base, resulting in limited training data. To address this issue, this application proposes adding a certain number of dissimilar sentences (or irrelevant sentences) for each standard question statement to the knowledge base, and constructing a training sample set for iterative model training based on these three types of sentences. In this embodiment, each training sample consists of three sentences: a standard question statement, a similar sentence to the standard question statement, and an unrelated sentence to the standard question statement. The standard question statement, also known as the anchor sentence, serves as a reference sample. The similar sentences to the standard question statement serve as positive samples, and the dissimilar sentences to the standard question statement serve as negative samples. For ease of distinction, in this embodiment, the training sample composed of the standard question statement, the similar sentences to the standard question statement, and the dissimilar sentences to the standard question statement manually entered from the knowledge base is referred to as the first type of training sample. It can be understood that the constructed training sample set usually includes the first type of training sample.

[0052] Data augmentation is a common method for solving the problem of training with few samples. Since the original corpus data in the knowledge base is very scarce, this application employs various data augmentation methods to increase the effective data volume. However, different data augmentation methods have different characteristics. The following data augmentation methods can be used in this embodiment:

[0053] Data augmentation method one: Adding punctuation marks to the original corpus data, referred to as punctuation augmentation.

[0054] Punctuation marks (,,.,:;!) are added to the original corpus data. The method of addition is to add a set number of punctuation marks at random positions in the sentence to be processed, resulting in a punctuated expanded sentence. The set number can usually be arbitrarily selected within the range of 1 to 3 / text length. If the text length of the sentence to be processed is less than or equal to 3, no punctuation is added. Question marks are usually not considered during punctuation expansion because they have some additional semantic meaning. For example, assuming the sentence to be processed is "I want to go to your company," the expanded sentence might be "I want to, go to your company." Punctuation expansion is performed on similar sentences and unrelated sentences of the standard question sentence. Typically, the standard question sentence, i.e., the anchor sentence, is not expanded. That is, the sentence to be processed is a similar or dissimilar sentence to the standard question sentence.

[0055] By expanding the punctuation, the training sample set can include a second type of training samples. The second type of training samples consists of standard question statements (reference samples), similar sentences with added punctuation (same type samples) obtained by adding punctuation to similar sentences of the standard question statements, and dissimilar sentences with added punctuation (different type samples) obtained by adding punctuation to dissimilar sentences of the standard question statements. It can be understood that the second type of training samples is obtained by adding punctuation to similar and dissimilar sentences of the standard question statements in the first type of training samples.

[0056] The second method of data augmentation is to add noise to the original corpus data based on EDA (Easy Data Augmentation for Text Classification Tasks), which is referred to as noise augmentation.

[0057] Noise is added to the original corpus data. The addition method is one of the following methods or any combination thereof, to obtain the noisy augmented sentence:

[0058] Method 1: Synonym Replacement (SR)

[0059] At least one non-stop word is randomly selected from the statement to be processed, and its corresponding synonym or homophone is used to replace it in the statement. Specifically, without considering stop words, n words are randomly selected from the statement to be processed, and then synonyms or homophones are randomly selected from a thesaurus or homophone dictionary for replacement.

[0060] Method 2: Randomly Insert (RI)

[0061] Perform at least one random insertion in the statement to be processed: randomly select a non-stop word, and insert the synonym corresponding to the non-stop word into a random position in the statement to be processed. Specifically, disregarding stop words, randomly select a word, and then randomly select a synonym from the set of synonyms of that word to insert into a random position in the statement to be processed. This process can be repeated n times.

[0062] Method 3: Randomly Swap (RS)

[0063] Perform at least one random swap within the statement to be processed: randomly select two words and swap their positions. Specifically, randomly select two words within the statement to be processed and swap their positions; this process can be repeated n times.

[0064] Method 4: Randomly Delete (RD)

[0065] Each word in the statement to be processed is randomly deleted based on a set probability (e.g., a probability of p).

[0066] For example, suppose the statement to be processed is "I want to go to your company". The statement after noise-added expansion (assuming random deletion is used in method 4) might be "I want to go to your company". Similar to punctuation expansion, noise-added expansion is also a data expansion for similar and dissimilar sentences of the standard question statement. Usually, the standard question statement, i.e., the anchor sentence, is not expanded. That is to say, the statement to be processed is a similar or dissimilar sentence of the standard question statement.

[0067] By adding noise and augmenting, the training sample set can include a third type of training samples. The third type of training samples consists of standard question statements (reference samples), noisy augmented similar sentences (same type samples) obtained by adding noise based on EDA to similar sentences of the standard question statements, and noisy augmented dissimilar sentences (different type samples) obtained by adding noise based on EDA to dissimilar sentences of the standard question statements. It can be understood that the third type of training samples are obtained by adding noise based on EDA to similar and dissimilar sentences of the standard question statements in the first type of training samples.

[0068] Noise-adding augmentation is simple and effective. Given the frequent occurrence of word omissions, synonyms, or homophones, this application uses methods 1 and 4 as the main methods for noise-adding augmentation. Noise-adding augmentation significantly alters the text of the statement to be processed. While fully augmenting the data, it also introduces some erroneous data, thus creating difficult samples. These samples effectively simulate the informal and non-standard expressions in the business scenario of robot voice dialogue, as well as inaccuracies such as typos and misspellings in the speech recognition results.

[0069] In practice, other data augmentation methods can be used, such as back-translation, to increase the variety of training samples in the training sample set. Back-translation involves translating the data to be processed into text in a specified language, and then translating the translated text back into the original language. For example, the Chinese text to be processed can be translated into English text, and then the English text can be translated back into Chinese text to obtain the back-translated augmented text. For instance, if the original text is "I want to go to your company," the back-translated augmented text might be "I want to go to your company."

[0070] Therefore, the training sample set can include: a first type of training sample (original corpus data), a second type of training sample (data after punctuation expansion), a third type of training sample (data after noisy expansion), and so on. For each type of training sample, the training samples will include: the standard question statement, similar samples of the standard question statement, and dissimilar samples of the standard question statement. It can be understood that the standard question statement is a manually entered statement serving as a standard question; the similar samples indicate statements similar to the standard question statement; and the dissimilar samples indicate statements dissimilar to the standard question statement.

[0071] S102. During the iterative training of the initial text matching model in the diversity stage using the training sample set, each training sample is processed by hybrid encoding based on linear interpolation and then input into the initial text matching model. The output is the first distance between the standard question statement and the sample of the same type, and the second distance between the standard question statement and the sample of the opposite type.

[0072] Since user problems are diverse, how to maximize the diversity of model input data on a limited training dataset to improve the accuracy of text matching models is a technical problem that needs to be solved in the diversity stage. In this embodiment of the invention, hybrid encoding processing based on linear interpolation is used for data augmentation in the diversity stage to enhance the diversity of model input data.

[0073] S103. Adjust the model parameters of the text matching model according to the output first distance, second distance and loss function of the text matching model until the loss function of the text matching model meets the set conditions, and obtain the text matching model trained in the diversity stage.

[0074] In one alternative implementation, the initial text matching model can be structured as a Triplet Network, which is suitable for few-shot learning scenarios. The structure of a Triplet Network is as follows: Figure 2As shown, this includes three identical feedforward neural networks (Nets) that share model parameters. A feedforward neural network typically consists of an input layer, one or more hidden layers, and an output layer. Data is passed through the network layer by layer until the output layer, without any feedback loops in between.

[0075] Each training sample used for iterative training of the Triplet Network consists of three samples, referred to as a sample triple: a reference sample, a sample from the same class, and a sample from a different class. Training is performed through comparisons between these samples. During each training iteration, a training sample (sample triple) is input: a reference sample, a sample from the same class, and a sample from a different class. The Triplet Network outputs two values: the distance between the feature vectors of the reference sample and the sample from the same class, and the distance between the reference sample and the sample from the different class, at the Net (embedding) layer. These distances characterize the similarity between the samples. The two values ​​output by the Triplet Network are the first distance between the feature vector of the reference sample and the feature vector of the sample from the same class, and the second distance between the feature vector of the reference sample and the feature vector of the sample from the different class. The first and second distances can be the L2 distance (Euclidean distance) between the feature vectors, or cosine distance, etc., without limitation.

[0076] Let the reference sample be denoted as x, and the out-of-class sample be denoted as x. - Samples of the same type are represented as x. + The essence of a Triplet Network is x - and x + The distance relative to x is encoded, and the distance can be Euclidean distance, as shown in formula [1]:

[0077]

[0078] Please refer to Figure 2 The Triplet Network also includes a comparator to process the vector formed by the two distances mentioned above. For each sample in a training sample (sample triplet), a parameter-shared feedforward neural network (Net) is trained to obtain the feature vectors of the three samples. Through iterative model training, x... + The first distance d between the eigenvectors corresponding to x and x + As small as possible, and x - The second distance d between the eigenvectors corresponding to x and x - Make it as large as possible, and make x... + The first distance d between the eigenvectors corresponding to x and x + and x- The second distance d between the eigenvectors corresponding to x and x - Maintain a minimum interval threshold, for example, set the minimum interval threshold to 1.

[0079] In this embodiment of the application, the loss function of the text matching model is shown in formula [2]. The loss function is used to characterize the difference between the reference sample, i.e., the standard question statement, and the samples of the same type and the samples of different types:

[0080]

[0081] Where const represents a constant, through d + and d - The initial variable can be normalized to the range (0, 1), d + d - As shown in formulas [3] and [4] respectively:

[0082]

[0083]

[0084] It should be noted that the trained text matching model is usually used to obtain the feature vectors of the two input sentences respectively, and to calculate the matching degree between the two sentences through similarity calculation. Therefore, the trained text matching model can include two trained feedforward neural networks.

[0085] In one alternative implementation, a curriculum-based learning approach can be used to train the text matching model. Curriculum learning is a learning strategy that mimics human learning methods, allowing the model to start learning from easy samples and gradually progress to learning more complex samples and knowledge.

[0086] In this embodiment, an initial text matching model can be iteratively trained using a training sample set for at least one stage to obtain a trained text matching model. Each of the at least one stage involves iterative training until the loss function of the text matching model satisfies a predetermined condition. In the case of a course-based learning approach, iterative training in adjacent stages is performed consecutively. The text matching model trained in the previous stage is used as the initial text matching model for the next stage, and the text matching model trained in the last stage is used as the final trained text matching model. The at least one stage includes the diversity stage.

[0087] Based on the structure of the initial text matching model described above, the process of iteratively training the initial text matching model using a training sample set for at least one stage is explained in detail. The at least one stage includes a diversity stage; in an optional implementation, the at least one stage may further include a baseline stage, which precedes the diversity stage; in an optional implementation, the at least one stage may further include a progressive stage, which follows the diversity stage. In specific implementations, the baseline stage, diversity stage, and progressive stage form a course-based learning training method, progressing from easy to difficult.

[0088] The iterative training process of the above three stages will be described in turn below.

[0089] Please refer to Figure 3a During the baseline phase, the initial text matching model can be iteratively trained using either the first type of training samples or the second type of training samples, or both types of training samples and the second type of training samples from the training sample set.

[0090] During the iterative training process in the baseline phase, the specific steps for each iteration include:

[0091] Step a1: Perform one-hot encoding on the standard question statement, the same type of sample, and the different type of sample in the training sample to obtain their respective one-hot encoding matrices. Then, encode their respective one-hot encoding matrices based on the pre-trained language representation model to obtain their respective language representation model encoding matrices.

[0092] It should be noted that if the training samples are of the first type, the sentences that need to be one-hot encoded and encoded based on the pre-trained language representation model include the standard question sentence, similar sentences of the standard question sentence, and dissimilar sentences of the standard question sentence; if the training samples are of the second type, the sentences that need to be one-hot encoded and encoded based on the pre-trained language representation model include the standard question sentence, punctuation-expanded similar sentences of the standard question sentence, and punctuation-expanded dissimilar sentences of the standard question sentence.

[0093] One-hot encoding, also known as "one-hot coding", is a commonly used method for text feature extraction. Essentially, it uses an N-bit state register to encode N states, with each state having an independent register bit and each register bit containing only one state.

[0094] The pre-trained language representation model can be any of BERT, RoBERTa, or ALBERT. BERT (Bidirectional Encoder Representations from Transformers) uses a bidirectional encoder of the Transformer. BERT consists of a pre-training part for training the language model and a model fine-tuning part. The model fine-tuning part uses the pre-trained language model for fine-tuning training and is widely used in tasks such as text classification and text matching. RoBERTa and ALBERT are both improved versions of BERT.

[0095] Step a2: Input the language representation model encoding matrices corresponding to the standard question statement, the similar samples, and the dissimilar samples into the corresponding feedforward neural networks in the initial text matching model, and output the first distance between the standard question statement and the similar samples, and the second distance between the standard question statement and the dissimilar samples.

[0096] Specifically, the language representation model encoding matrix corresponding to the standard question statement is input into... Figure 2 The feedforward neural network corresponding to x, and the language representation model encoding matrix corresponding to the same type of samples are input. Figure 2 Chinese x + The corresponding feedforward neural network, the language representation model encoding matrix corresponding to the outlier sample is input. Figure 2 Chinese x - The corresponding feedforward neural network.

[0097] Specifically, the first distance between the standard question statement and the similar sample can be the Euclidean distance between the feature vector corresponding to the standard question statement and the feature vector corresponding to the similar sample, and the second distance between the standard question statement and the dissimilar sample can be the Euclidean distance between the feature vector corresponding to the standard question statement and the feature vector corresponding to the dissimilar sample.

[0098] Step a3: Adjust the model parameters of the text matching model based on the output first distance, second distance, and loss function of the text matching model. The loss function is used to characterize the differences between the standard question statement and the samples of the same type and the samples of different types.

[0099] Specifically, the loss value of the loss function is calculated by the comparator based on the first distance and the second distance, and the model parameters of the text matching model are adjusted with the goal of reducing the loss of the text matching model.

[0100] It is worth noting that the above process is only one iterative training process in the baseline stage. In practical applications, it may be necessary to repeat the training process multiple times to ensure that the loss function of the text matching model meets the set conditions in this stage. Specifically, the loss function meeting the set conditions can mean that the degree of loss reduction is less than a set threshold, thus requiring multiple iterations of the above training process.

[0101] In the baseline stage of course-based learning, a baseline model can be obtained by directly training with high-quality, highly discriminative raw corpus data annotated by customer service personnel; alternatively, a baseline model can be obtained by training with data augmented with punctuation marks, which can improve the text matching model's ability to recognize common basic questions; or a baseline model can be obtained by training with raw corpus data plus data augmented with punctuation marks, which can better learn the feature vector representation of the model's input data.

[0102] Please refer to Figure 3b In the diversity stage, the initial text matching model can be iteratively trained using either the first type of training samples, the second type of training samples, or both types of training samples from the training sample set. In this diversity stage, when iteratively training the initial text matching model using the first and second type of training samples, the accuracy of the text matching model can be improved by adding punctuation-enlarged data to the original corpus.

[0103] During the iterative training process in the aforementioned diversity phase, the specific steps for each iteration include:

[0104] Step b1: Perform hybrid encoding processing based on linear interpolation on the standard question statement, the same type of sample, and the different type of sample in the training sample to obtain their respective hybrid encoding matrices.

[0105] In one alternative implementation, step b1 may include the following steps:

[0106] Step b1-1: Perform one-hot encoding on the standard question statement, the same type of sample, and the different type of sample in the training sample to obtain their respective one-hot encoding matrices.

[0107] It should be noted that if the training samples are of the first type, the statements requiring hybrid encoding based on linear interpolation include the standard question statement, similar sentences of the standard question statement, and dissimilar sentences of the standard question statement. If the training samples are of the second type, the statements requiring hybrid encoding based on linear interpolation include the standard question statement, punctuation-expanded similar sentences of the standard question statement, and punctuation-expanded dissimilar sentences of the standard question statement. The process of hybrid encoding based on linear interpolation for each statement is the same. For any statement, one-hot encoding is performed to form a one-hot encoding matrix with a dimension of shape = (text length, total number of words). This one-hot encoding matrix is ​​the one-hot encoding result, denoted as T. The vocabulary includes the encoding range of all characters.

[0108] Step b1-2: Encode the one-hot encoding matrices corresponding to the standard question statement, the similar samples, and the dissimilar samples respectively based on the pre-trained language representation model to obtain their respective language representation model encoding matrices. Then, multiply the respective language representation model encoding matrices with the preset word list vector matrix and normalize them to obtain their respective sentence encoding prediction result matrices.

[0109] Specifically, the pre-trained language representation model can be any of BERT, RoBERTa, ALBERT, etc. For example, BERT can be used to encode the one-hot encoding matrix T, and the resulting BERT encoding matrix is ​​denoted as BERT(T). The dimension of BERT(T) is shape = (text length, BERT encoding vector dimension), where the BERT encoding vector dimension is fixed, typically 768.

[0110] Multiply BERT(T) by the pre-encoded word vector matrix W, where the dimension shape of W is (text length, total number of words), and then normalize it using softmax to obtain the sentence encoding prediction result matrix M, where the dimension shape of M is (total number of words, dimension of BERT encoding vectors). The sentence encoding prediction result matrix M is calculated as shown in formula [5]:

[0111] M = softmax(BERT(T) * W) T )

[0112] Steps b1-3: Perform linear interpolation on the sentence encoding prediction result matrix and one-hot encoding matrix corresponding to the standard question statement, the similar sample, and the dissimilar sample to obtain their respective hybrid encoding matrices.

[0113] Specifically, the sentence coding prediction result matrix M and the one-hot coding matrix T are subjected to linear interpolation, i.e., mixup, to obtain the expanded mixed coding matrix X. A hyperparameter λ for linear interpolation can be set, which represents the proportion of the one-hot coding matrix in the linear interpolation process. The sum of the proportions of the one-hot coding matrix and the sentence coding prediction result matrix in the linear interpolation process is 1. The calculation method of the mixed coding matrix X is shown in formula [6]:

[0114] X=λT+(1-λ)M

[0115] During the model iterative training process in the diversity phase, the parameter value of hyperparameter λ can be set to a fixed value, such as λ = 0.2. To further enhance diversity, during the model iterative training process in the diversity phase, the parameter value of hyperparameter λ can be initialized to a minimum preset parameter value (e.g., 0.05). When iteratively training the model based on the current parameter value of the hyperparameter, if the loss of the loss function decreases less than a set threshold, the current parameter value of hyperparameter λ is increased by a set adjustment step size (e.g., 0.05) until the maximum preset parameter value of the hyperparameter (e.g., 0.2) is reached.

[0116] For an example illustrating the hybrid encoding process based on linear interpolation for any statement, please refer to [reference needed]. Figure 4 Assuming the current statement is "I want to check the weather", the statement is first one-hot encoded to obtain a one-hot encoding matrix T. The one-hot encoding matrix T is then encoded using BERT to obtain a BERT encoding matrix BERT(T). The BERT encoding matrix BERT(T) is then multiplied by a preset word vector matrix W and normalized using softmax to obtain a sentence encoding prediction result matrix M. Based on the hyperparameter λ, the sentence encoding prediction result matrix M and the one-hot encoding matrix T are linearly interpolated to obtain an expanded hybrid encoding matrix X.

[0117] The data augmentation method based on hybrid encoding processing using linear interpolation obtains the sentence encoding prediction matrix by multiplying the language representation model encoding matrix and the vocabulary vector matrix. Since the vocabulary includes the encoding range of all characters, it effectively utilizes other similar semantic information in the vocabulary and integrates the encoding information in the vocabulary into the language representation model encoding matrix, effectively improving the diversity of the model input data. By using linear interpolation to mix the sentence encoding prediction matrix with the one-hot encoding matrix, the advantages of both are combined, making the increase in encoding information within a controllable range and ensuring the trainability of the model.

[0118] Step b2: The hybrid encoding matrices corresponding to the standard question statement, the similar samples, and the dissimilar samples are respectively input into the corresponding feedforward neural networks in the initial text matching model, and the first distance between the standard question statement and the similar samples, and the second distance between the standard question statement and the dissimilar samples are output.

[0119] Specifically, input the hybrid encoding matrix corresponding to the standard question statement. Figure 2 The feedforward neural network corresponding to x, and the hybrid encoding matrix corresponding to the same type of samples as input. Figure 2 Chinese x + The corresponding feedforward neural network, the hybrid encoding matrix corresponding to the heterogeneous samples is input. Figure 2 Chinese x - The corresponding feedforward neural network.

[0120] Step b3: Adjust the model parameters of the text matching model based on the output first distance, second distance, and loss function of the text matching model. The loss function is used to characterize the differences between the standard question statement and the samples of the same type and the samples of different types.

[0121] Specifically, the loss value of the loss function is calculated by the comparator based on the first distance and the second distance, and the model parameters of the text matching model are adjusted with the goal of reducing the loss of the text matching model.

[0122] It is worth noting that the above process is only one iterative training process in the diversity stage. In practical applications, it may be necessary to repeat the iterative training multiple times to ensure that the loss function of the text matching model meets the set conditions in this stage. Specifically, meeting the set conditions means that when the maximum preset parameter value of the hyperparameter (e.g., 0.2) is reached, the degree of loss reduction of the loss function is less than a set threshold. Therefore, the above iterative training process can be executed multiple times.

[0123] In the diversity phase of course-based learning, training can be performed using high-quality, highly discriminative raw corpus data annotated by customer service personnel, or using data augmented with punctuation marks, or a combination of raw corpus data and augmented punctuation marks. Furthermore, hybrid coding based on linear interpolation is employed for data augmentation to enhance the diversity of the model's input data. Additionally, the hyperparameter λ of the linear interpolation can be flexibly adjusted; a larger value results in greater diversity of the model's input data. During iterative training, the hyperparameter is initialized to a minimum preset value, e.g., λ = 0.05. Each time the decrease in the loss function on the validation set falls below a set threshold, the value of λ is increased by a set adjustment step size, e.g., 0.05, until it reaches a maximum preset value, e.g., λ = 0.2. If the decrease in the loss function is less than the set threshold when the hyperparameter reaches its maximum preset value, the iterative training of the diversity phase is complete. By continuously increasing the diversity of the model's input data, the accuracy of the text matching model is improved.

[0124] Please refer to Figure 3c In the progressive stage, the initial text matching model can be iteratively trained using the first and / or second type of training samples in the training sample set, as well as the third type of training samples selected based on the progressive factor τ. The progressive factor τ represents the proportion of the selected third type of training samples to the total number of third type training samples (e.g., τ = 0.1). When iteratively training the model based on the current sample size, if the loss of the loss function decreases less than a set threshold, the sample size of the third type of training samples corresponding to the progressive factor τ is increased based on the current sample size until all third type training samples are added.

[0125] In the iterative training process of the progressive phase, the specific steps of each iteration include:

[0126] Step c1: Perform one-hot encoding on the standard question statement, the same type of sample, and the different type of sample in the training sample to obtain their respective one-hot encoding matrices. Then, encode their respective one-hot encoding matrices based on the pre-trained language representation model to obtain their respective language representation model encoding matrices.

[0127] It should be noted that if the training samples are of the first type, the sentences requiring one-hot encoding and encoding based on the pre-trained language representation model include the standard question sentence, similar sentences of the standard question sentence, and dissimilar sentences of the standard question sentence. If the training samples are of the second type, the sentences requiring one-hot encoding and encoding based on the pre-trained language representation model include the standard question sentence, punctuation-expanded similar sentences of the standard question sentence, and punctuation-expanded dissimilar sentences of the standard question sentence. If the training samples are of the third type, the sentences requiring one-hot encoding and encoding based on the pre-trained language representation model include the standard question sentence, noise-expanded similar sentences of the standard question sentence, and noise-expanded dissimilar sentences of the standard question sentence. The pre-trained language representation model can be any of BERT, RoBERTa, ALBERT, etc.

[0128] Step c2: Input the language representation model encoding matrices corresponding to the standard question statement, the similar samples, and the dissimilar samples into the corresponding feedforward neural networks in the initial text matching model, and output the first distance between the standard question statement and the similar samples, and the second distance between the standard question statement and the dissimilar samples.

[0129] Specifically, the language representation model encoding matrix corresponding to the standard question statement is input into... Figure 2 The feedforward neural network corresponding to x, and the language representation model encoding matrix corresponding to the same type of samples are input. Figure 2 Chinese x + The corresponding feedforward neural network, the language representation model encoding matrix corresponding to the outlier sample is input. Figure 2 Chinese x - The corresponding feedforward neural network.

[0130] Step c3: Adjust the model parameters of the text matching model based on the output first distance, second distance, and loss function of the text matching model. The loss function is used to characterize the differences between the standard question statement and the samples of the same type and the samples of different types.

[0131] Specifically, the loss value of the loss function is calculated by the comparator based on the first distance and the second distance, and the model parameters of the text matching model are adjusted with the goal of reducing the loss of the text matching model.

[0132] It is worth noting that the above process is only one iterative training process in the incremental stage. In practical applications, it may be necessary to repeat the iterative training multiple times to ensure that the loss function of the text matching model meets the set conditions in this stage. Specifically, the loss function meets the set conditions when all third-class training samples are added, and the degree of loss reduction is less than a set threshold. Therefore, the above iterative training process can be executed multiple times.

[0133] In the progressive stage of course-based learning, training can be conducted using the original corpus data, data augmented with punctuation, and data augmented with noise. A progressive learning strategy is adopted, introducing a progressive factor τ, starting from 0.1. Each time the decrease in the loss function of the validation set is less than a set threshold, the amount of data in the third type of training samples is increased by 0.1 times, until all third-type training samples are added. If the decrease in the loss function is less than the set threshold after all third-type training samples are added, the iterative training of the progressive stage is completed. This allows the model to gradually learn difficult samples, progressing from easy to difficult. Since difficult samples can simulate the non-standard expressions such as colloquialisms in the business scenarios of robot voice dialogue, as well as inaccuracies such as typos and misspellings in the speech recognition results, the accuracy of the text matching model can be improved, reducing the adverse effects of these issues on intent recognition.

[0134] The following provides a unified explanation of how to adjust the model parameters of the text matching model in each iteration of training at each stage. The model parameters of the text matching model refer to the parameters used to characterize the structure of the text matching model. Specifically, the model parameters of the text matching model can include model parameters shared by the three feedforward neural networks in the text matching model. These shared model parameters can include the network parameters of each layer in the feedforward neural networks. For each network layer, the network parameters can include, for example, but are not limited to, the number of neurons in each network layer, the connection relationships between neurons in other network layers, and the connection weights.

[0135] In one alternative implementation, backpropagation can be used to adjust the model parameters of the text matching model. Specifically, based on the first distance, the second distance, and the loss function of the text matching model, the loss value of the text matching model is determined. Then, with the goal of reducing the loss of the text matching model, the network parameters of each network layer in the text matching model are adjusted layer by layer through backpropagation, starting from the last network layer of the text matching model.

[0136] For example, starting from the last network layer of the text detection model, based on the structure of each network layer in the text matching model and the connection relationships and connection weights between different network layers, the partial derivative of the loss value of the text matching model can be taken forward to obtain the loss value of each network layer. The loss value of each network layer is used to characterize the matching bias caused by each network layer. Then, with the goal of reducing the loss of the text matching model, the network parameters of each network layer are adjusted sequentially based on the loss value of each network layer.

[0137] The text matching model training method provided in this application addresses the problem of limited training data, where model training can only be performed based on similar question data in a knowledge base. It augments each training sample in the training sample set by adding dissimilar samples of the standard question statement in addition to standard question statements and their similar samples. During iterative training in the diversity phase using the training sample set, the standard question statements, similar samples, and dissimilar samples in the training samples are augmented using a hybrid encoding process based on linear interpolation before being input into the initial text matching model. The model parameters are adjusted by comparing samples. By augmenting each training sample in the training sample set, and considering the diverse range of questions raised by clients, the hybrid encoding process based on linear interpolation enhances the diversity of the model's input data on a limited training basis. This, in turn, effectively improves the accuracy of the text matching model through iterative training in the diversity phase.

[0138] Furthermore, considering that each training sample includes standard question statements, similar samples, and dissimilar samples, the initial text matching model adopts a triplet network structure, which includes three feedforward neural networks with identical structures and shared model parameters. Triplet networks are characterized by fast inference speed. In each iteration of training, standard question statements, similar samples, and dissimilar samples are respectively input into the corresponding feedforward neural networks, and the model parameters of the text matching model are adjusted by comparing the samples.

[0139] Furthermore, for scenarios with limited training data, various data augmentation methods were employed, including punctuation augmentation, noise augmentation, and back-translation augmentation. In addition, considering the different characteristics of the augmented data obtained by different data augmentation methods, a course-based learning training method was designed, which prioritizes easier tasks and gradually increases the difficulty. This training method allows the model to learn the main contradictions of the problem first, focusing on the primary issues before addressing the secondary ones. This approach is more advantageous than the traditional, all-in-one training method, making the trained text matching model closer to the real-world scenario. It is particularly suitable for intent recognition in dialogue question-answering systems.

[0140] Corresponding to the training method of the aforementioned text matching model, embodiments of this application also provide an intent recognition method, capable of based on Figure 1The text matching model trained by the method shown accurately matches the input speech recognition results with standard question statements that conform to the customer's intent, thereby completing the intent recognition task and improving the accuracy of intent recognition.

[0141] Please refer to Figure 5 The following is a flowchart illustrating an intent recognition method according to an embodiment of this application. The method may include the following steps:

[0142] S501. Obtain the speech recognition statement corresponding to the speech to be recognized.

[0143] The speech to be recognized usually refers to the question raised by the customer in the dialogue question and answer system. Generally, the speech recognition is obtained by speech recognition (ASR) to obtain the corresponding text format speech recognition statement. After the speech recognition statement is preprocessed, it enters the question recognition process. The speech recognition statement corresponding to the speech to be recognized obtained in S501 usually refers to the preprocessed speech recognition statement.

[0144] S502. Input the speech recognition statement and the standard question statement in the knowledge base into a pre-trained text matching model, and output the matching degree between the speech recognition statement and the standard question statement; wherein, the text matching model is based on the above... Figure 1 The text matching model shown was trained using the training method described. It can be understood that the matching degree characterizes the similarity between the speech recognition statement and the standard question statement.

[0145] S503. Based on the matching degree between the speech recognition result and each standard question statement, determine the standard question statement that matches the speech to be recognized as the intent recognition result.

[0146] In one alternative implementation, the standard question statement with the highest matching degree to the speech recognition statement is usually taken as the intent recognition result. Based on the accurate recognition of the customer's intent, subsequent processes such as generating responses are then generated according to the intent recognition result.

[0147] The intent recognition method provided in this application uses a text matching model trained by the above-mentioned text matching model training method to perform text matching between the speech recognition statement corresponding to the speech to be recognized and the standard question statement in the knowledge base. Since the training method of the text matching model addresses the problem of insufficient training data, it expands the training sample set by adding heterogeneous samples of labeled question statements in addition to standard question statements and similar samples. Furthermore, it uses hybrid encoding processing based on linear interpolation for data augmentation during the diversity stage to improve the diversity of model input data. Therefore, it can improve the accuracy of the text matching model, making the trained text matching model applicable to intent recognition scenarios where the speech to be recognized has diversity. Based on the accurate text matching of the text matching model for a variety of speech to be recognized, it can accurately identify the real customer intent expressed by the speech to be recognized, thereby improving the accuracy of intent recognition.

[0148] The following uses a business scenario of voice robot dialogue as an example to illustrate the training method of the text matching model and the intent recognition method provided in this application embodiment. The processing flow involved in voice robot dialogue is as follows: Figure 6As shown, the question posed by the customer to the voice robot is the speech to be recognized. The speech to be recognized undergoes Automatic Speech Recognition (ASR) to obtain the corresponding speech recognition result. After preprocessing, the speech recognition result enters the question recognition process. The preprocessing may include sensitive word processing, stop word processing, and traditional Chinese character replacement, etc. It can be understood that the speech recognition result is a text-formatted speech recognition statement. In the question recognition process, if the question is not empty, entity recognition and intent recognition are performed. Then, the response process begins, generating the voice robot's response based on the intent recognition result, and continuing with subsequent processes, ultimately completing the voice robot's dialogue flow. The text matching model training method and intent recognition method provided in this application embodiment can be applied to intent recognition in the question recognition process. In the knowledge base of a voice robot, standard question statements and similar sentences in text format are usually pre-entered manually. To address the issue of limited training data in the knowledge base, dissimilar sentences of standard question statements are added to the knowledge base, resulting in a first type of training sample consisting of standard question statements, similar sentences, and dissimilar sentences. The first type of training sample is then augmented with punctuation and noise to obtain second and third types of training samples. The augmented training sample set is used to iteratively train the initial text matching model for at least one stage, resulting in a trained text matching model for voice robot dialogue. This at least one stage includes a diversity stage for data augmentation using hybrid coding based on linear interpolation. Furthermore, using the trained text matching model, the pre-processed speech recognition statements (text format) in the voice robot dialogue process are matched with standard question statements (text format) in the voice robot's knowledge base. The standard question statement matching the speech to be recognized (the question posed by the customer to the voice robot) is determined as the intent recognition result corresponding to the customer's question to the voice robot. The voice robot's response is then generated based on the intent recognition result. As can be seen, in the business scenario of voice robot dialogue, this solution effectively solves the problem of insufficient training data for voice robots. Data augmentation improves the accuracy of the text matching model, making the trained model applicable to intent recognition scenarios where the speech to be recognized has diverse meanings. Based on the accurate text matching of various speech samples, the text matching model can accurately identify the true customer intent expressed by the speech, thereby improving the accuracy of intent recognition. The above-described business scenario of voice robot dialogue is merely an illustrative example of the application scenario of the method provided in this application embodiment. The text matching model training method and intent recognition method provided in this application embodiment can also be applied to other dialogue and question-and-answer business scenarios.

[0149] In addition, with the above Figure 1Corresponding to the training method of the text matching model shown, this application embodiment also provides a training device for a text matching model. Figure 7 This is a schematic diagram of the structure of a text matching model training device 700 provided in an embodiment of this application, including:

[0150] The construction module 701 is used to construct a training sample set based on standard question statements in the knowledge base. Each training sample in the training sample set includes: the standard question statement, similar samples of the standard question statement, and dissimilar samples of the standard question statement. The standard question statement is a statement manually entered as a standard question. The similar samples are used to indicate statements similar to the standard question statement, and the dissimilar samples are used to indicate statements dissimilar to the standard question statement.

[0151] The diversity training module 702 is used to perform hybrid encoding processing based on linear interpolation on each training sample and input it into the initial text matching model during the iterative training of the initial text matching model in the diversity stage using the training sample set, and output the first distance between the standard question statement and the sample of the same type, and the second distance between the standard question statement and the sample of the opposite type.

[0152] The first adjustment module 703 is used to adjust the model parameters of the text matching model according to the output first distance, second distance and loss function of the text matching model until the loss function of the text matching model meets the set conditions, so as to obtain the text matching model trained in the diversity stage.

[0153] In one optional implementation, the initial text matching model comprises three identical feedforward neural networks that share model parameters; during the iterative training process of the diversity phase:

[0154] The diversity training module 702 is specifically used to perform hybrid encoding processing based on linear interpolation on the standard question statement, the similar samples, and the dissimilar samples in the training samples during each iteration of training to obtain their respective hybrid encoding matrices; input the hybrid encoding matrices corresponding to the standard question statement, the similar samples, and the dissimilar samples into the corresponding feedforward neural networks in the initial text matching model, and output the first distance between the standard question statement and the similar samples, and the second distance between the standard question statement and the dissimilar samples;

[0155] The first adjustment module 703 is specifically used to adjust the model parameters of the text matching model based on the first distance, the second distance, and the loss function of the text matching model output by the diversity training module 702. The loss function is used to characterize the difference between the standard question statement and the samples of the same type and the samples of different types.

[0156] In one alternative implementation, the diversity training module 702 uses the first type of training samples and / or the second type of training samples in the training sample set to perform iterative training of the initial text matching model in a diversity phase.

[0157] In one optional implementation, the diversity training module 702 is specifically used to perform one-hot encoding on the standard question statement, the similar samples, and the dissimilar samples in the training samples during each iteration of training, to obtain their respective one-hot encoding matrices; to encode the one-hot encoding matrices of the standard question statement, the similar samples, and the dissimilar samples based on a pre-trained language representation model, to obtain their respective language representation model encoding matrices; to multiply the respective language representation model encoding matrices with a preset vocabulary vector matrix and normalize them, to obtain their respective sentence encoding prediction result matrices; and to perform linear interpolation on the sentence encoding prediction result matrices and one-hot encoding matrices of the standard question statement, the similar samples, and the dissimilar samples, to obtain their respective hybrid encoding matrices.

[0158] In one alternative implementation, the training device 700 further includes:

[0159] The training control module 704 is used to perform at least one stage of iterative training on the initial text matching model using the training sample set, and to take the text matching model trained in the last stage as the final trained text matching model; the model iterative training in each stage of the at least one stage continues until the loss function of the text matching model satisfies the set conditions; the at least one stage includes a diversity stage.

[0160] In an optional implementation, the at least one stage further includes a baseline stage, which, prior to the diversity stage, involves an iterative training process implemented by a baseline training module 705 and a second adjustment module 706. The baseline training module 705 uses first-class training samples and / or second-class training samples from the training sample set to iteratively train the initial text matching model for the baseline stage.

[0161] The baseline training module 705 is used to perform one-hot encoding on the standard question statement, the similar samples, and the dissimilar samples in the training samples during each iteration of training to obtain their respective one-hot encoding matrices. Then, it encodes each of these one-hot encoding matrices based on a pre-trained language representation model to obtain their respective language representation model encoding matrices. The language representation model encoding matrices corresponding to the standard question statement, the similar samples, and the dissimilar samples are input into the corresponding feedforward neural networks in the initial text matching model to output a first distance between the standard question statement and the similar samples, and a second distance between the standard question statement and the dissimilar samples.

[0162] The second adjustment module 706 is used to adjust the model parameters of the text matching model based on the first distance, the second distance, and the loss function of the text matching model output by the baseline training module 705.

[0163] In an optional implementation, the at least one stage further includes a progressive stage, which, following the diversity stage, involves an iterative training process implemented by a progressive training module 707 and a third adjustment module 708. The progressive training module 707 uses first-class and / or second-class training samples from the training sample set, as well as third-class training samples selected based on a progressive factor, to perform iterative training on the initial text matching model during the progressive stage.

[0164] The progressive training module 707 is used to perform one-hot encoding on the standard question statement, the similar samples, and the dissimilar samples in the training samples during each iteration of training to obtain their respective one-hot encoding matrices. Then, it encodes each of these one-hot encoding matrices based on a pre-trained language representation model to obtain their respective language representation model encoding matrices. The language representation model encoding matrices corresponding to the standard question statement, the similar samples, and the dissimilar samples are input into a specified feedforward neural network in the initial text matching model to output a first distance between the standard question statement and the similar samples, and a second distance between the standard question statement and the dissimilar samples.

[0165] The third adjustment module 708 is used to adjust the model parameters of the text matching model based on the first distance, the second distance, and the loss function of the text matching model output by the progressive training module 707.

[0166] Obviously, the training device for the text matching model in this application embodiment can be used as described above. Figure 1The main body executing the training method of the text matching model shown is thus able to realize the training method of the text matching model in... Figure 1 The functions implemented are the same, so they will not be described in detail here.

[0167] In addition, with the above Figure 5 Corresponding to the intent recognition method shown, this application also provides an intent recognition device. Figure 8 This is a schematic diagram of the structure of an intent recognition device 800 provided in an embodiment of this application, including:

[0168] The acquisition module 801 is used to acquire the speech recognition statement corresponding to the speech to be recognized.

[0169] Text matching module 802 is used to input the speech recognition statement and preset question statements in the knowledge base into a pre-trained text matching model, and output the matching degree between the speech recognition statement and the question statements; wherein, the text matching model is based on the above... Figure 1 The text matching model shown was trained using the training method described above.

[0170] The intent recognition module 803 is used to determine a standard question statement that matches the speech to be recognized as the intent recognition result based on the matching degree between the speech recognition statement and each question statement.

[0171] Obviously, the intent recognition device in the embodiments of this application can be used as described above. Figure 5 The execution entity of the intent recognition method shown can therefore realize the intent recognition method in Figure 5 The functions implemented are the same, so they will not be described in detail here.

[0172] Figure 9 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Please refer to it. Figure 9 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for other business operations.

[0173] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 9 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0174] Memory is used to store programs. Specifically, programs may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0175] The processor reads the corresponding computer program from non-volatile memory into memory and then runs it, forming a text matching model training device or intent recognition device at the logical level. The processor executes the program stored in memory and specifically performs various processes to implement the above-described text matching model training method or intent recognition method embodiments, achieving the same technical effect. To avoid repetition, these details will not be repeated here.

[0176] The methods executed by the text matching model training device or the intent recognition device disclosed in the embodiments of this application can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above methods can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0177] Of course, in addition to software implementation, the electronic device of this application does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0178] This application also proposes a computer-readable storage medium that stores one or more programs, the programs including instructions, which, when executed by a processor, implement the various processes of the above-described text matching model training method or intent recognition method embodiment, and can achieve the same technical effect. To avoid repetition, these will not be described again here.

[0179] The processor is the processor in the electronic device described in the above embodiments. The computer-readable storage medium is, for example, a computer read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk.

[0180] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described text matching model training method or intent recognition method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0181] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0182] The foregoing has described specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0183] In summary, the above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

[0184] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0185] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0186] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

Claims

1. A training method for a text matching model, characterized in that, include: A training sample set is constructed based on standard question statements in the knowledge base. Each training sample in the training sample set includes: the standard question statement, similar samples of the standard question statement, and dissimilar samples of the standard question statement. The standard question statement is a statement manually entered as a standard question. The similar samples are used to indicate statements that are similar to the standard question statement, and the dissimilar samples are used to indicate statements that are not similar to the standard question statement. During the iterative training of the initial text matching model using the training sample set in the diversity phase, each training sample is processed by hybrid encoding based on linear interpolation and then input into the initial text matching model. The output is a first distance between the standard question statement and the samples of the same type, and a second distance between the standard question statement and the samples of different types. The hybrid encoding process based on linear interpolation includes: performing one-hot encoding on the standard question statement, the samples of the same type, and the samples of different types in the training samples to obtain their respective one-hot encoding matrices; encoding each corresponding one-hot encoding matrix based on a pre-trained language representation model to obtain their respective language representation model encoding matrices; multiplying each corresponding language representation model encoding matrix by a preset vocabulary vector matrix and normalizing the result to obtain their respective sentence encoding prediction matrix; and performing linear interpolation on the corresponding sentence encoding prediction matrix and the one-hot encoding matrix to obtain their respective hybrid encoding matrix. The model parameters of the text matching model are adjusted based on the output first distance, second distance, and loss function of the text matching model until the loss function of the text matching model meets the set conditions, thus obtaining the text matching model trained in the diversity stage.

2. The method of claim 1, wherein, The initial text matching model includes three feedforward neural networks with identical structures and shared model parameters; During the iterative training process in the aforementioned diversity phase, the specific steps for each iteration include: The standard question statements, similar samples, and dissimilar samples in the training samples are respectively subjected to hybrid encoding processing based on linear interpolation to obtain their respective hybrid encoding matrices; The hybrid encoding matrices corresponding to the standard question statement, the similar samples, and the dissimilar samples are respectively input into the corresponding feedforward neural networks in the initial text matching model, and the first distance between the standard question statement and the similar samples, and the second distance between the standard question statement and the dissimilar samples are output. Based on the first distance, the second distance, and the loss function of the text matching model, the model parameters of the text matching model are adjusted. The loss function is used to characterize the differences between the standard question statement and the samples of the same type and the samples of different types.

3. The method of claim 2, wherein, The training sample set includes a first type of training samples and / or a second type of training samples; The iterative training of the initial text matching model using the training sample set in the diversity phase includes: The initial text matching model is iteratively trained in a diversity phase using the first type of training samples and / or the second type of training samples in the training sample set. In the first type of training samples, the similar samples of the standard question statement are similar sentences of the standard question statement, and the dissimilar samples are dissimilar sentences of the standard question statement. The similar sentences and dissimilar sentences of the standard question statement are pre-entered into the knowledge base. In the second type of training samples, the similar samples of the standard question statements are punctuation-expanded similar sentences obtained by adding punctuation marks to similar sentences of the standard question statements, and the dissimilar samples are punctuation-expanded dissimilar sentences obtained by adding punctuation marks to dissimilar sentences of the standard question statements.

4. The method of claim 1, wherein, The sentence encoding prediction result matrix and the one-hot encoding matrix are linearly interpolated based on the hyperparameters of linear interpolation. The hyperparameters represent the proportion of the one-hot encoding matrix in the linear interpolation process, and the sum of the proportions of the one-hot encoding matrix and the sentence encoding prediction result matrix in the linear interpolation process is 1. During the iterative training process in the diversity stage, the parameter values ​​of the hyperparameters are initialized to the minimum preset parameter values. When iterative training is performed based on the current parameter values ​​of the hyperparameters, if the loss of the loss function decreases by less than a set threshold, the current parameter values ​​of the hyperparameters are increased according to a set adjustment step size until the maximum preset parameter value of the hyperparameters is reached.

5. The method of claim 3, wherein, The initial text matching model is iteratively trained using the training sample set for at least one stage, and the text matching model trained in the last stage is taken as the final trained text matching model; the iterative training in each stage of the at least one stage continues until the loss function of the text matching model satisfies the set conditions; the at least one stage includes the diversity stage.

6. The method of claim 5, wherein, The at least one stage further includes a baseline stage, and the iterative training of the initial text matching model using the training sample set for at least one stage further includes: Prior to the iterative training in the diversity phase, the initial text matching model to be trained is subjected to iterative training in the baseline phase using the first type of training samples and / or the second type of training samples in the training sample set. During the iterative training process in the baseline phase, the specific steps for each iteration include: One-hot encoding is performed on the standard question statements, similar samples, and dissimilar samples in the training samples to obtain their respective one-hot encoding matrices. Then, the respective one-hot encoding matrices are encoded based on the pre-trained language representation model to obtain their respective language representation model encoding matrices. The language representation model encoding matrices corresponding to the standard question statement, the similar samples, and the dissimilar samples are respectively input into the corresponding feedforward neural networks in the initial text matching model, and the first distance between the standard question statement and the similar samples, and the second distance between the standard question statement and the dissimilar samples are output. The model parameters of the text matching model are adjusted based on the output of the first distance, the second distance, and the loss function of the text matching model.

7. The method of claim 5, wherein, The training sample set also includes a third type of training samples; wherein, the similar samples of the standard question statements in the third type of training samples are noisy expanded similar sentences obtained by performing EDA-based noise addition processing on similar sentences of the standard question statements, and the dissimilar samples are noisy expanded dissimilar sentences obtained by performing EDA-based noise addition processing on dissimilar sentences of the standard question statements. The at least one stage further includes a progressive stage, and the iterative training of the initial text matching model using the training sample set for at least one stage further includes: After the iterative training in the diversity phase, the initial text matching model is iteratively trained using the first and / or second type of training samples in the training sample set, and the third type of training samples selected based on the progressive factor. The progressive factor represents the proportion of the selected third type of training samples to the total number of third type training samples. If the loss function decreases less than a set threshold during iterative training based on the current sample size, the sample size of the third type of training samples corresponding to the progressive factor is increased based on the current sample size until all third type training samples are added. In the iterative training process of the progressive phase, the specific steps of each iteration include: One-hot encoding is performed on the standard question statements, similar samples, and dissimilar samples in the training samples to obtain their respective one-hot encoding matrices. Then, the respective one-hot encoding matrices are encoded based on the pre-trained language representation model to obtain their respective language representation model encoding matrices. The language representation model encoding matrices corresponding to the standard question statement, the similar samples, and the dissimilar samples are respectively input into the corresponding feedforward neural networks in the initial text matching model, and the first distance between the standard question statement and the similar samples, and the second distance between the standard question statement and the dissimilar samples are output. The model parameters of the text matching model are adjusted based on the output of the first distance, the second distance, and the loss function of the text matching model.

8. The method of claim 7, wherein, The EDA-based noise addition processing of similar or dissimilar sentences of the standard question statement includes one or any combination of the following methods: At least one non-stop word is randomly selected from the statement to be processed, and the corresponding synonyms or homophones of each non-stop word are used to replace it in the statement to be processed. Perform at least one random insertion in the statement to be processed: randomly select a non-stop word and insert the synonym corresponding to the non-stop word into a random position in the statement to be processed; Perform at least one random swap in the statement to be processed: randomly select two words and swap their positions; Each word in the statement to be processed is randomly deleted based on a set probability; The statement to be processed is a similar or dissimilar statement to the standard question statement.

9. An intention recognition method characterized by, include: Obtain the speech recognition statement corresponding to the speech to be recognized; The speech recognition statement and the standard question statement in the knowledge base are input into a pre-trained text matching model, and the matching degree between the speech recognition statement and the standard question statement is output; wherein, the text matching model is trained according to the training method of the text matching model as described in any one of claims 1 to 8; Based on the matching degree between the speech recognition result and each standard question statement in the knowledge base, the standard question statement that matches the speech to be recognized is determined as the intent recognition result.

10. A training device for a text matching model, characterized in that, include: The construction module is used to construct a training sample set based on standard question statements in the knowledge base. Each training sample in the training sample set includes: the standard question statement, similar samples of the standard question statement, and dissimilar samples of the standard question statement. The standard question statement is a statement manually entered as a standard question. The similar samples are used to indicate statements similar to the standard question statement, and the dissimilar samples are used to indicate statements dissimilar to the standard question statement. A diversity training module is used to perform a hybrid encoding process based on linear interpolation on each training sample during the iterative training of the initial text matching model using the training sample set. The input of each training sample into the initial text matching model is then processed, and the output is a first distance between the standard question statement and the samples of the same class, and a second distance between the standard question statement and the samples of different classes. The hybrid encoding process based on linear interpolation includes: performing one-hot encoding on the standard question statement, the samples of the same class, and the samples of different classes in the training samples to obtain their respective one-hot encoding matrices; encoding each corresponding one-hot encoding matrix based on a pre-trained language representation model to obtain their respective language representation model encoding matrices; multiplying each corresponding language representation model encoding matrix with a preset vocabulary vector matrix and normalizing the result to obtain their respective sentence encoding prediction matrix; and performing linear interpolation on the corresponding sentence encoding prediction matrix and the one-hot encoding matrix to obtain their respective hybrid encoding matrix. The first adjustment module is used to adjust the model parameters of the text matching model according to the output first distance, second distance and loss function of the text matching model until the loss function of the text matching model meets the set conditions, so as to obtain the text matching model trained in the diversity stage.

11. An intent recognition device, characterized in that, include: The acquisition module is used to acquire the speech recognition statement corresponding to the speech to be recognized; A text matching module is used to input the speech recognition statement and a standard question statement in a knowledge base into a pre-trained text matching model, and output the matching degree between the speech recognition result and the standard question statement; wherein, the text matching model is trained according to the training method of the text matching model as described in any one of claims 1 to 8; The intent recognition module is used to determine the standard question statement that matches the speech to be recognized as the intent recognition result based on the matching degree between the speech recognition result and each standard question statement in the knowledge base.

12. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the training method of the text matching model as described in any one of claims 1 to 8, or to implement the intent recognition method as described in claim 9.

13. A computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the training method of the text matching model as described in any one of claims 1 to 8, or to implement the intent recognition method as described in claim 9.

Images