Method, system, device and medium for training intent recognition model

Abstract

The present application relates to a method, system, device, and medium for training an intent recognition model, wherein the method comprises: fine-tuning a pre-trained model using original data, scoring and sorting the original training set data using the fine-tuned pre-trained model, wherein the original training set data includes original data and synthesized data; filtering the sorted original training set data and dividing the filtered data into different training subsets; and finally, training the intent recognition model using the different training subsets to obtain a final trained intent recognition model. This application solves the problems of low training data quality and poor performance of the trained model when training the intent recognition model, thereby improving model performance and recognition accuracy.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to a method, system, device and medium for training an intent recognition model. Background Art

[0002] Intent recognition is a subtask of natural language understanding and has a wide range of applications. For example, in conversational systems, it can help robots understand user intent. In conversational scenarios, intent recognition models can help robots understand user intent. For example, when a user asks, "What's the weather like today?" an intent recognition model can help the robot understand that the question corresponds to the intent of "asking about the weather." When the types of intents are limited, this task can be considered a classification task.

[0003] However, in most real-world scenarios, it's difficult to collect large amounts of training data to train intent recognition models. Therefore, existing techniques often augment this small amount of data, generating large amounts of synthetic data. This synthetic data, combined with the original data, forms the training dataset, significantly expanding the size of the training set. However, due to the varying quality of synthetic data, while expanding the training set, it also leads to varying degrees of degradation in its quality, which in turn affects the final performance of the intent recognition model. Poorly augmented data not only fails to improve model performance, but may even cause the model to crash.

[0004] Currently, no effective solution has been proposed for the problems of low training data quality and poor performance of trained models when training intent recognition models in related technologies. Summary of the Invention

[0005] The embodiments of the present application provide a method, system, device and medium for training an intent recognition model to at least solve the problems of low training data quality and poor performance of the trained model when training the intent recognition model in the related art.

[0006] In a first aspect, an embodiment of the present application provides a method for training an intent recognition model, the method comprising:

[0007] Fine-tuning the pre-trained model using the original data, and scoring and sorting the original training set data using the fine-tuned pre-trained model, wherein the original training set data includes the original data and the synthesized data;

[0008] Filter the sorted original training set data and divide the filtered data into different training subsets;

[0009] The intent recognition model is trained using the different training subsets to obtain a final trained intent recognition model.

[0010] In some embodiments, scoring and ranking the original training set data using the fine-tuned pre-training model includes:

[0011] Calculate the single sentence generation probability of each data in the original training set data and convert the probability into perplexity;

[0012] Arrange each sentence in the original training set data from small to large according to the size of the perplexity.

[0013] In some embodiments, screening the sorted original training set data includes:

[0014] Discard the preset screening ratio of data from the original training set data arranged at the end.

[0015] In some embodiments, training the intent recognition model using the different training subsets includes:

[0016] Starting from the training subset with the smallest perplexity, other training subsets are merged into the current training set one by one, and the model is trained multiple times until the model converges, and finally a trained intent recognition model is obtained.

[0017] In a second aspect, an embodiment of the present application provides a system for training an intent recognition model, the system comprising:

[0018] A data sorting module is used to fine-tune the pre-training model using the original data, and score and sort the original training set data using the fine-tuned pre-training model, wherein the original training set data includes original data and synthetic data;

[0019] The screening module is used to screen the sorted original training set data and divide the screened data into different training subsets;

[0020] The training module is used to train the intent recognition model using the different training subsets to obtain a final trained intent recognition model.

[0021] In some embodiments, the data sorting module is further configured to calculate the probability of generating a single sentence for each piece of data in the original training set data, and convert the probability into perplexity.

[0022] Arrange each sentence in the original training set data from small to large according to the size of the perplexity.

[0023] In some embodiments, the screening module is further configured to discard a preset screening ratio of data from the original training set data arranged at the tail.

[0024] In some embodiments, the training module is also used to start from a training subset with the smallest perplexity, merge other training subsets into the current training set one by one, train the model multiple times until the model converges, and finally obtain a trained intent recognition model.

[0025] In a third aspect, an embodiment of the present application provides an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the computer program, the method for training an intent recognition model as described in the first aspect above is implemented.

[0026] In a fourth aspect, an embodiment of the present application provides a storage medium on which a computer program is stored, which, when executed by a processor, implements the method for training an intent recognition model as described in the first aspect above.

[0027] Compared with the related art, the method for training the intent recognition model provided in the embodiment of the present application fine-tunes the pre-trained model through the original data, and scores and sorts the original training set data through the fine-tuned pre-trained model, wherein the original training set data includes original data and synthetic data; the sorted original training set data is filtered, and the filtered data is divided into different training subsets; finally, the intent recognition model is trained through different training subsets to obtain the final trained intent recognition model.

[0028] Compared to the prior art scheme of using only original data or synthetic data with enhanced data to train the model, the present application uses original data to fine-tune the pre-trained language model. The fine-tuned pre-trained model not only has rich semantic information, but also masters the relevant semantic distribution of the original data. Therefore, it has a strong ability to identify the data quality in the training data set. Then, the enhanced data is scored and sorted based on the fine-tuned pre-trained model, and the sorted data set is screened and divided into different training subsets, which produces two effects: 1) The training data can be arranged from easy to difficult, which is more suitable for model learning, thereby improving model performance; 2) Some overly difficult data is eliminated, which can be regarded as screening the training data set and removing the worst quality data, thereby ensuring the stability of training. Finally, the divided training data subsets are put into model training one by one to obtain a final intent recognition model with better performance, which solves the problems of low training data quality and poor performance of the trained model when training the intent recognition model, and improves model performance and recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0030] Figure 1 2 is a schematic diagram of an application environment of the method for training an intent recognition model according to an embodiment of the present application;

[0031] Figure 2 is a flowchart of a method for training an intent recognition model according to an embodiment of the present application;

[0032] Figure 3 is a structural block diagram of a system for training an intent recognition model according to an embodiment of the present application;

[0033] Figure 4 Schematic diagram of the internal structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application. In addition, it can also be understood that although the efforts made in this development process may be complex and lengthy, for ordinary technicians in the field related to the contents disclosed in the present application, some changes such as design, manufacturing or production based on the technical contents disclosed in the present application are only conventional technical means and should not be understood as the contents disclosed in the present application being insufficient.

[0035] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it refer to independent or alternative embodiments that are mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments unless there is a conflict.

[0036] Unless otherwise defined, the technical or scientific terms used in this application should have the ordinary meaning understood by a person of ordinary skill in the technical field to which this application belongs. The words "one", "a", "the" and the like used in this application do not indicate a limit on quantity and may indicate the singular or plural. The terms "include", "comprise", "have" and any variations thereof used in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units that are inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The word "multiple" used in this application means greater than or equal to two. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The terms "first", "second", "third" and the like involved in this application are merely used to distinguish similar objects and do not represent a specific ordering of the objects.

[0037] The method for training the intent recognition model provided in this application can be applied to Figure 1 In the application environment shown, Figure 1 Schematic diagram of the application environment of the method for training the intention recognition model according to the embodiment of the present application. Figure 1 As shown. The terminal 11 communicates with the server 10 through a network. The server 10 fine-tunes the pre-trained model using the original data, and scores and sorts the original training set data using the fine-tuned pre-trained model, wherein the original training set data includes original data and synthetic data; the sorted original training set data is filtered, and the filtered data is divided into different training subsets; finally, the intent recognition model is trained using different training subsets to obtain a final trained intent recognition model. The terminal 11 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices, and the server 10 can be implemented using an independent server or a server cluster consisting of multiple servers.

[0038] This embodiment provides a method for training an intent recognition model. Figure 2 is a flow chart of a method for training an intent recognition model according to an embodiment of the present application. Figure 2 As shown, the process includes the following steps:

[0039] Step S201: fine-tune the pre-trained model using the original data, and score and sort the original training set data using the fine-tuned pre-trained model, wherein the original training set data includes the original data and the synthesized data;

[0040] Preferably, in this embodiment, the original data is text data, such as natural language data; the pre-training model is preferably a Bert model, but other natural language models, such as RNN, can also be used in other scenarios;

[0041] First, obtain the original data, fine-tune the BERT pre-trained model with the original data, and then use the fine-tuned BERT pre-trained model to score and sort the original data and synthetic data in the original training set. The specific scoring and sorting steps include:

[0042] S1: Calculate the single sentence generation probability of each data in the original training set data and convert the probability into perplexity ppl. The calculation formula of the single sentence generation probability is shown in the following formula (1):

[0043]

[0044] Among them, S represents a sentence, k represents the number of words in a sentence, and ω represents a word.

[0045] S2: Arrange each sentence in the original training set according to its perplexity, from smallest to largest. Generally speaking, the better the sentence, the greater its probability of generation and the lower its perplexity, meaning the model is less confused by the sentence. Therefore, the arranged data can be considered to be ranked from high to low quality.

[0046] It should be noted that the synthetic data in this embodiment is generated through existing data enhancement technology. There are two main categories of specific data enhancement technologies: 1) Rule-based generation. The data enhancement process is completed in a rule-based manner. The main rule strategy is to randomly delete part of the text information, randomly replace keywords based on a synonym dictionary, etc. to generate synthetic data. 2) A generative model based on deep learning technology uses a certain similarity assumption to generate text and produce synthetic data. For example, using machine translation technology, the original text is translated from Chinese into non-Chinese languages such as English, and then the translation is translated back into Chinese to obtain synthetic data.

[0047] In this embodiment, the pre-trained language model is fine-tuned using the original data. The fine-tuned pre-trained model not only has rich semantic information, but also grasps the relevant semantic distribution of the original data, and has a strong ability to identify the data quality in the training dataset. In addition, in this embodiment, the training data is arranged from easy to difficult, making the data more suitable for model learning, which helps improve model performance.

[0048] Step S202, screening the sorted original training set data, and dividing the screened data into different training subsets;

[0049] Preferably, the original training set data after the above sorting is screened, and a preset screening ratio of data is discarded from the original training set data arranged at the end. For example, if the screening ratio is set to 20%, the data at the end of the original training set data needs to be discarded. This part of the data has the highest confusion and can be considered as too difficult data. Since the Bert pre-training model has been fine-tuned using the original data, it is not unfamiliar with the sentences in the original data. Therefore, synthetic data that is similar to the sentences in the original data in the semantic space will also be given a higher generation probability, while the data with the worst quality has the lowest generation probability and the highest confusion. Therefore, discarding this part of the data can be regarded as discarding the part of the data with the worst quality, thereby ensuring the quality stability of the training data set.

[0050] Furthermore, after completing the data screening, the remaining data after screening, i.e., the first 80% of the data, is divided into different training subsets. It should be noted that before determining the number of training subsets, the data is divided into different subsets ranging from 2 to 10. Finally, according to actual testing, it is found that the model performance obtained when the data is divided into 4 training subsets is the best. Therefore, in this embodiment, it is preferred to divide the data into 4 training subsets;

[0051] Step S203: Train the intent recognition model using different training subsets to obtain a final trained intent recognition model.

[0052] Preferably, the training subsets obtained in step S202 are divided starting from the one with the smallest perplexity to train the intent recognition model. After one cycle of training, the loss of the intent recognition model on the validation set is verified. When the loss on the validation set does not improve compared with the loss of the previous cycle, the training subset with the second smallest perplexity is merged into the current training data set to start a new round of training. This process is repeated, and other training subsets are merged into the current training set one by one until the model converges, and finally a trained intent recognition model is obtained. This model training method can better improve the model performance and the recognition accuracy of the model.

[0053] It should be noted that the intent recognition model used in this application can be selected according to actual conditions and is not specifically limited.

[0054] Through the above steps S201 to S203, this embodiment uses the original data to fine-tune the pre-trained language model. The fine-tuned pre-trained model not only has rich semantic information, but also masters the relevant semantic distribution of the original data. Therefore, it has a strong ability to identify the data quality in the training data set. Next, the enhanced data is scored and sorted based on the fine-tuned pre-trained model, and the sorted data set is screened and divided into different training subsets, which produces two effects: 1) The training data can be arranged from easy to difficult, which is more suitable for model learning, thereby improving model performance; 2) Some overly difficult data is eliminated, which can be regarded as screening the training data set and removing the worst quality data, thereby ensuring the stability of training. Finally, the divided training data subsets are put into model training one by one to obtain a final intent recognition model with better performance, which solves the problems of low training data quality and poor performance of the trained model when training the intent recognition model, and improves model performance and recognition accuracy.

[0055] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0056] This embodiment also provides a system for training an intent recognition model, which is used to implement the above-mentioned embodiments and preferred implementation methods, and will not be repeated here. As used below, the terms "module", "unit", "sub-unit", etc. can implement a combination of software and / or hardware for a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0057] Figure 3 This is a structural block diagram of a system for training an intent recognition model according to an embodiment of the present application. Figure 4 As shown, the system includes a data sorting module 31, a screening module 32 and a training module 33:

[0058] The data sorting module 31 is used to fine-tune the pre-training model using the original data, and score and sort the original training set data using the fine-tuned pre-training model, wherein the original training set data includes original data and synthetic data; the screening module 32 is used to screen the sorted original training set data, and divide the screened data into different training subsets; the training module 33 is used to train the intent recognition model using different training subsets to obtain the final trained intent recognition model.

[0059] Through the above system, in this embodiment, the data sorting module 31 scores and sorts the enhanced data based on the fine-tuned pre-training model, the screening module 32 screens the sorted data set and divides it into different training subsets, and finally, the training module 33 puts the divided training data subsets into model training one by one to obtain a final intent recognition model with better performance, which solves the problems of low training data quality and poor performance of the trained model when training the intent recognition model, and improves the model performance and recognition accuracy.

[0060] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be repeated here.

[0061] Furthermore, it should be noted that each of the aforementioned modules can be either a functional module or a program module, and can be implemented via software or hardware. For modules implemented via hardware, each of the aforementioned modules can be located in the same processor; or each of the aforementioned modules can be located in different processors in any combination.

[0062] This embodiment further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0063] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0064] In addition, in conjunction with the intent recognition model training method in the above embodiments, embodiments of the present application may provide a storage medium for implementation. The storage medium stores a computer program; when the computer program is executed by a processor, it implements any of the intent recognition model training methods in the above embodiments.

[0065] In one embodiment, a computer device is provided, which may be a terminal. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a method for training an intent recognition model is implemented. The display screen of the computer device may be a liquid crystal display or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or a key, trackball, or touchpad provided on the computer device housing, or an external keyboard, touchpad, or mouse.

[0066] In one embodiment, Figure 4 is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application, such as Figure 4 As shown, an electronic device is provided, which may be a server, and its internal structure diagram may be as shown in FIG. Figure 4 As shown. The electronic device includes a processor, a network interface, an internal memory, and a non-volatile memory connected via an internal bus, wherein the non-volatile memory stores an operating system, a computer program, and a database. The processor is used to provide computing and control capabilities, the network interface is used to communicate with external terminals via a network connection, the internal memory is used to provide an environment for the operation of the operating system and the computer program, and when the computer program is executed by the processor, it implements a method for training an intent recognition model. The database is used to store data.

[0067] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the electronic device to which the solution of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0068] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, which can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0069] Those skilled in the art should understand that the various technical features of the above-described embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the various technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0070] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A method for training an intent recognition model, characterized in that: The method comprises: Fine-tuning the pre-trained model using the original data, and scoring and ranking the original training set data using the fine-tuned pre-trained model, wherein the original training set data includes original data and synthetic data, and the synthetic data is generated using existing data augmentation technology, wherein scoring and ranking the original training set data using the fine-tuned pre-trained model includes: Calculate the single sentence generation probability of each data in the original training set data and convert the probability into perplexity; arrange each sentence in the original training set data in ascending order according to the perplexity; Filter the sorted original training set data and divide the filtered data into different training subsets; The intent recognition model is trained using the different training subsets to obtain a final trained intent recognition model, wherein the training of the intent recognition model using the different training subsets includes: Starting from the training subset with the smallest perplexity, merge the other training subsets into the current training set one by one, train the model multiple times until the model converges, and finally obtain a trained intent recognition model. After one cycle of training, the loss of the intent recognition model on the validation set is verified. When the loss on the validation set does not improve compared with the loss of the previous cycle, merge the training subset with the second smallest perplexity into the current training data set, and start a new round of training. Repeat this process, merge the other training subsets into the current training set one by one, until the model converges, and finally obtain a trained intent recognition model.

2. The method according to claim 1, characterized in that The screening of the sorted original training set data includes: Discard the preset screening ratio of data from the original training set data arranged at the end.

3. A system for training an intent recognition model, characterized in that: The system comprises: A data sorting module is configured to fine-tune the pre-trained model using the original data, and score and sort the original training set data using the fine-tuned pre-trained model, wherein the original training set data includes original data and synthetic data, and the synthetic data is generated using existing data augmentation technology. The scoring and sorting of the original training set data using the fine-tuned pre-trained model includes: Calculate the single sentence generation probability of each data in the original training set data and convert the probability into perplexity; arrange each sentence in the original training set data in ascending order according to the perplexity; The screening module is used to screen the sorted original training set data and divide the screened data into different training subsets; A training module is configured to train the intent recognition model using the different training subsets to obtain a final trained intent recognition model, wherein training the intent recognition model using the different training subsets includes: Starting from the training subset with the smallest perplexity, merge the other training subsets into the current training set one by one, train the model multiple times until the model converges, and finally obtain a trained intent recognition model. After one cycle of training, the loss of the intent recognition model on the validation set is verified. When the loss on the validation set does not improve compared with the loss of the previous cycle, merge the training subset with the second smallest perplexity into the current training data set, and start a new round of training. Repeat this process, merge the other training subsets into the current training set one by one, until the model converges, and finally obtain a trained intent recognition model.

4. The system according to claim 3, characterized in that The screening module is further configured to discard data of a preset screening ratio from the original training set data arranged at the tail.

5. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the method for training an intent recognition model according to any one of claims 1 to 2.

6. A storage medium, characterized in that The storage medium stores a computer program, wherein the computer program is configured to execute the method for training an intent recognition model according to any one of claims 1 to 2 when running.

Images