Intent classification method, system, terminal and medium based on neural network model

Abstract

The present invention discloses an intent classification method based on a neural network model, including: obtaining intent classification task data; writing regular expression rules and corresponding logic selection operations for intent classification data sets; converting regular expressions into finite state automata; converting finite state automata into three-dimensional tensors; decomposing three-dimensional tensors using tensor decomposition methods to obtain three matrices to initialize TFRNN parameters; initializing the parameters of the logic selection layer, and forming an intent classification model with the logic selection layer and TFRNN; inputting intent classification task data into the intent classification model for training, and outputting intent classification results. The method converts regular expressions into their corresponding recurrent neural network forms, and correctly initializes the neural network, so that running the neural network before training is equivalent to directly running the regular expression. After training, the recurrent neural network has very strong interpretability.

Description

Technical Field

[0001] The present invention relates to the field of software technology, and in particular to an intention classification method, system, terminal and medium based on a neural network model. Background Art

[0002] Regular expressions are a mature rule system with important applications in the field of natural language processing. Due to the controllable, interpretable and high-precision characteristics of the rules, they are the only choice for some natural language processing applications that have high requirements for model accuracy. However, rule-based systems also have certain defects, such as the need for a large amount of labor costs to write rules, and the inability to make good use of large amounts of data. In the era of big data, neural networks, as a model based on data learning, can make good use of large amounts of data and learn the knowledge in them, but correspondingly, neural networks also have obvious defects, such as poor interpretability, uncontrollability, the need for a large amount of labor costs to annotate data, and difficulty in achieving high accuracy. The existing methods that combine regular expressions and neural networks have some defects. The mainstream methods usually use multi-task learning or knowledge distillation, such as "LEARNING FROM RULES GENERALIZING LABELED EXEMPLARS". Although they can achieve good results on downstream tasks, they do not completely integrate the advantages of neural networks and regular expressions. Rules are only used as a way to assist training. Neural networks are still uninterpretable and the entire system is still uncontrollable. At the same time, in some systems, even after the model is trained, it is still impossible to get rid of the use of rule results as model input. Summary of the invention

[0003] In response to the defects in the prior art, the embodiments of the present invention provide an intent classification method, system, terminal and medium based on a neural network model, which have cold start capability and can achieve an accuracy similar to that of regular expressions without any training. The trained model has extremely strong interpretability.

[0004] In a first aspect, an embodiment of the present invention provides an intention classification method based on a neural network model, comprising:

[0005] Get intent classification task data;

[0006] Write regular expression rules and corresponding logical selection operations for intent classification datasets;

[0007] Convert regular expressions to finite state automata;

[0008] Convert finite state automata to three-dimensional tensors;

[0009] The three-dimensional tensor is decomposed using the tensor decomposition method to obtain three matrices to initialize the parameters of TFRNN;

[0010] Initialize the parameters of the logic selection layer, and combine the logic selection layer and TFRNN to form an intent classification model;

[0011] The intent classification task data is input into the intent classification model for training, and the intent classification results are output.

[0012] Furthermore, the three-dimensional tensor contains all the information of the regular expression, the first dimension is the vocabulary size of the words involved in the finite state automaton, and the second dimension and the third dimension are the number of states of the finite state automaton.

[0013] Furthermore, the logic selection layer includes three logic operations, including AND operation, OR operation and priority selection.

[0014] Furthermore, the priority selection method is: low priority score = low priority result - high priority result.

[0015] In a second aspect, an embodiment of the present invention provides an intention classification system based on a neural network model, comprising: a data acquisition module, a rule writing module, a first conversion module, a second conversion module, a tensor decomposition module, an intention classification model generation module and a training module.

[0016] The data acquisition module is used to acquire intent classification task data;

[0017] The rule writing module is used to write regular expression rules and corresponding logical selection operations for the intent classification data set;

[0018] The first conversion module is used to convert the regular expression into a finite state automaton;

[0019] The second conversion module is used to convert the finite state automaton into a three-dimensional tensor;

[0020] The tensor decomposition module is used to decompose the three-dimensional tensor using a tensor decomposition method to obtain three matrices to initialize the parameters of the TFRNN;

[0021] The intent classification model generation module is used to initialize the parameters of the logic selection layer, and form an intent classification model with the logic selection layer and TFRNN;

[0022] The training module is used to input the intent classification task data into the intent classification model for training and output the intent classification result.

[0023] Furthermore, the three-dimensional tensor contains all the information of the regular expression, the first dimension is the vocabulary size of the words involved in the finite state automaton, and the second dimension and the third dimension are the number of states of the finite state automaton.

[0024] Furthermore, the logic selection layer includes three logic operations, including AND operation, OR operation and priority selection.

[0025] Furthermore, the priority selection method is: low priority score = low priority result - high priority result.

[0026] In a third aspect, an embodiment of the present invention provides an intelligent terminal, comprising a processor, an input device, an output device and a memory, wherein the processor, input device, output device and memory are interconnected, the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions to execute the method described in the above embodiment.

[0027] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the method described in the above embodiment.

[0028] Beneficial effects of the present invention:

[0029] The embodiment of the present invention provides an intention classification method, system, terminal and medium based on a neural network model, which converts a regular expression into its corresponding recurrent neural network form, and correctly initializes the neural network so that running the neural network before training is equivalent to directly running the regular expression. After training, the recurrent neural network has very strong interpretability. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for the specific embodiments or the description of the prior art. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn according to the actual scale.

[0031] Figure 1 A flowchart of an intention classification method based on a neural network model provided by the first embodiment of the present invention is shown;

[0032] Figure 2 A structural block diagram of an intention classification system based on a neural network model provided by another embodiment of the present invention is shown;

[0033] Figure 3A structural block diagram of an intelligent terminal provided by another embodiment of the present invention is shown. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0036] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0037] It should be further understood that the term "and / or" used in the present specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0038] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0039] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by those skilled in the art to which the present invention belongs.

[0040] like Figure 1 FIG. 1 is a flowchart of a method for classifying intentions based on a neural network model provided by a first embodiment of the present invention, the method comprising the following steps:

[0041] S101: Obtain intent classification task data.

[0042] Specifically, the intent classification task is to predict the intent category of a given text, which is a text classification task.

[0043] S102: Write regular expression rules and corresponding logical selection operations for the intent classification dataset.

[0044] Specifically, there are several regular expressions for each category of text. If the text is matched by a regular expression of a certain category, it is considered that the intent category of the text is likely to belong to this category. If regular expressions of multiple categories match the text, a logical selection mechanism is required. Therefore, regular expression rules and corresponding logical selection operations are written for the intent classification data set to facilitate subsequent data processing.

[0045] S103: Convert the regular expression into a finite state automaton.

[0046] S104: Convert the finite state automaton into a three-dimensional tensor.

[0047] The three-dimensional tensor contains all the information of the regular expression. The first dimension is the vocabulary size of the words involved in the finite state automaton, and the second and third dimensions are the number of states of the finite state automaton.

[0048] S105: Decompose the three-dimensional tensor using the tensor decomposition method to obtain three matrices to initialize the parameters of the TFRNN.

[0049] S106: Initialize the parameters of the logic selection layer, and use the logic selection layer and TFRNN to form an intent classification model.

[0050] Specifically, each category of regular expressions is converted into its corresponding finite state automaton. For example, we have a simple regular expression for the intent of "query flight time", as shown in the second row of Table 1, where "$" means accepting any input, and "*" means that the preceding character can be repeated any number of times. This regular expression is converted into the finite state automaton in the third row of Table 1, where s0 of the finite state automaton represents the starting state, and s2 represented by two circles is the ending state. If the finite state automaton reaches the ending state from the starting state after reading the text, the regular expression is determined to match.

[0051]

[0052] Table 1 Regular expression converted into finite state automaton

[0053] Next, the finite state automaton is converted into a three-dimensional tensor, which contains all the information of the regular expression. The first dimension is the vocabulary size of the words involved in the finite state automaton, and the second and third dimensions are the number of states of the finite state automaton. This tensor can be seen as a stack of transfer matrices corresponding to each word. Taking the automaton in Table 1 as an example, for the word flight, its transfer matrix T is 3x3 in size, where T[0][2]=T[0][0]=T[1][1]=T[2][2]=1, and the rest are 0. After conversion to tensor representation, we can use the classic forward algorithm (Forward Algorithm) and Viterbi Algorithm in dynamic programming to run the finite state automaton. Taking the Viterbi algorithm as an example, at each moment t, we can get a vector with a dimension equal to the number of states, which is called the state vector h t The i-th element represents the path with the highest score from the starting state to state i after receiving and reading t words. The state vector at the next time t+1 can be calculated from the state vector at time t and the state transfer matrix corresponding to the t+1th word to get h t+1 =f(h t ,T t+1 ). This process is similar to the calculation process of a recurrent neural network (RNN), and the state vector can be regarded as a hidden vector in the recurrent neural network. Through the cyclic calculation, the value corresponding to the end state in the state vector at the last moment determines whether the regular expression matches after reading the entire sentence. Specifically, for the forward algorithm, if there is an end state with a final value greater than or equal to 1, the regular expression matches. For the Viterbi algorithm, if there is an end state with a value equal to 1, the regular expression matches.

[0054] Since the three-dimensional tensor may be too large, the tensor rank decomposition method is used to decompose the three-dimensional tensor. A three-dimensional tensor of A×B×C is decomposed into three matrices of size A×R, B×R, and C×R, where R is the size of the rank. The three matrices are used to approximate the tensor and the three decomposed matrices are used for loop calculation. This can greatly reduce the parameters of the intent classification model, reduce the computing resources consumed by the intent classification model training, and improve the training speed of the intent classification model. The introduction of word vectors improves the model effect and generalization ability.

[0055] The logic selection layer is used to sort the operation results of the finite state automaton into scores for each category. The logic selection layer contains three logic operations, including AND operation, OR operation and priority selection. The priority selection method is: low priority score = low priority result - high priority result.

[0056] Through steps S103 to S106, the regular expression is converted into a corresponding trainable recurrent neural network, and the recurrent neural network is initialized so that running the neural network during training is equivalent to directly running the regular expression, achieving an accuracy rate close to that of the regular expression without any training. In addition, the logic selection layer and TFRNN constitute an intent classification model, and the entire model can be trained.

[0057] S107: Input the intent classification task data into the intent classification model for training, and output the intent classification result.

[0058] Experiments were conducted on the SMS text classification dataset using three different settings: zero data, a small amount of data, and all data. The experimental data results are shown in Table 2:

[0059]

[0060] Table 2 Experimental results of SMS dataset

[0061] Through the above experiments, the intent classification model in this embodiment has better effects than the traditional recurrent neural network, which is mainly reflected in:

[0062] 1. Better performance than the baseline model in the case of small data training.

[0063] 2. Even when the full dataset is fully trained, the baseline model is still fully trained.

[0064] 3. The trained model can be restored to a finite state automaton and has strong interpretability.

[0065] This embodiment provides an intent classification method based on a neural network model, which converts a regular expression into its corresponding recurrent neural network form and correctly initializes the neural network so that running the neural network before training is equivalent to directly running the regular expression. After training, the recurrent neural network has very strong interpretability.

[0066] The first embodiment of the present invention provides an intention classification method based on a neural network model. Correspondingly, another embodiment of the present invention also provides an intention classification system based on a neural network model. Figure 2 , which is a schematic diagram of the structure of an intention classification system based on a neural network model provided by another embodiment of the present invention. Since the device embodiment is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described below is only illustrative.

[0067] like Figure 2As shown, an intention classification system based on a neural network model provided by another embodiment of the present invention is shown, including: a data acquisition module 201, a rule writing module 202, a first conversion module 203, a second conversion module 204, a tensor decomposition module 205, an intention classification model generation module 206 and a training module 207, wherein the data acquisition module 201 is used to acquire input intention classification data; the rule writing module 202 is used to write regular expression rules and corresponding logical selection operations for the intention classification data set; the first conversion module 203 is used to convert the regular expression into a finite state automaton; the second conversion module 204 is used to convert the finite state automaton into a three-dimensional tensor; the tensor decomposition module 205 is used to decompose the three-dimensional tensor using a tensor decomposition method to obtain three matrices to initialize the parameters of the TFRNN; the three-dimensional tensor contains all the information of the regular expression, the first dimension is the vocabulary size of the words designed for the finite state automaton, and the second and third dimensions are the number of states of the finite state automaton. The intent classification model generation module 206 is used to initialize the parameters of the logic selection layer, and the logic selection layer and TFRNN constitute the intent classification model; the training module 207 is used to input the input intent classification data into the intent classification model for training, and output the intent classification result. The logic selection layer contains three logical operations, including AND operation, OR operation and priority selection. The priority selection method is: low priority score = low priority result - high priority result.

[0068] This embodiment provides an intention classification system based on a neural network model, which converts regular expressions into their corresponding recurrent neural network forms and correctly initializes the neural network so that running the neural network before training is equivalent to directly running the regular expression. After training, the recurrent neural network has very strong interpretability.

[0069] Another embodiment of the present invention also provides an intelligent terminal, such as Figure 3 As shown, a structural block diagram of an intelligent terminal is shown, which includes a processor 301, an input device 302, an output device 303 and a memory 304. The processor 301, the input device 302, the output device 303 and the memory 304 are interconnected. The memory 304 is used to store computer programs. The computer programs include program instructions. The processor 301 is configured to call the program instructions to execute the method described in the above embodiment.

[0070] It should be understood that in the embodiment of the present invention, the processor 301 may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0071] The input device 302 may include a touch pad, a microphone, etc., and the output device 303 may include a display (LCD, etc.), a speaker, etc.

[0072] The memory 304 may include a read-only memory and a random access memory, and provide instructions and data to the processor 301. A portion of the memory 304 may also include a non-volatile random access memory. For example, the memory 304 may also store information about the device type.

[0073] In a specific implementation, the processor 301, input device 302, and output device 303 described in the embodiment of the present invention can execute the implementation described in the method embodiment provided in the embodiment of the present invention, and can also execute the implementation described in the system embodiment of the present invention, which will not be repeated here.

[0074] The present invention further provides an embodiment of a computer-readable storage medium, wherein the computer storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the method described in the above embodiment.

[0075] The computer-readable storage medium may be an internal storage unit of the terminal described in the foregoing embodiments, such as a hard disk or memory of the terminal. The computer-readable storage medium may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Further, the computer-readable storage medium may also include both an internal storage unit of the terminal and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the terminal. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0076] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0077] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the terminals and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0078] In the several embodiments provided in the present application, it should be understood that the disclosed terminal and method can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or it can be an electrical, mechanical or other form of connection.

[0079] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

Claims

1. An intention classification method based on a neural network model, characterized in that: include: S101: Acquire intent classification task data; S102: Write regular expression rules and corresponding logic selection operations for the intent classification data set; S103: Convert the regular expression into a finite state automaton; S104: converting the finite state automaton into a three-dimensional tensor; S105: Decompose the three-dimensional tensor using a tensor decomposition method to obtain three matrices to initialize the parameters of the TFRNN; S106: Initialize the parameters of the logic selection layer, and use the logic selection layer and TFRNN to form an intent classification model. The TFRNN converts regular expressions into corresponding trainable recurrent neural networks; the tensor is regarded as a stack of transfer matrices corresponding to each word; the forward algorithm and Viterbi algorithm are used to run the finite state automaton, and a vector with a dimension of the number of states can be obtained at each time t, which is called the state vector h t , where the i-th element represents the path with the highest score from the starting state to state i after receiving and reading t words. The state vector at the next time t + 1 can be calculated from the state vector at time t and the state transfer matrix corresponding to the t+1th word ,Through cyclic calculation, the value corresponding to the end state in the state vector at the last moment determines whether the regular expression matches after reading the entire sentence. For the forward algorithm, if there is an end state with a final value greater than or equal to 1, the regular expression matches. For the Viterbi algorithm, if there is an end state with a value equal to 1, the regular expression matches; S107: Input the intent classification task data into the intent classification model for training, and output the intent classification result; The three-dimensional tensor contains all the information of the regular expression, the first dimension is the vocabulary size of the words involved in the finite state automaton, and the second and third dimensions are the number of states of the finite state automaton; The logic selection layer is used to sort the operation results of the finite state automaton into scores for each category. The logic selection layer includes three logic operations, including AND operation, OR operation and priority selection.

2. An intention classification system based on a neural network model, characterized in that: include: Data acquisition module, rule writing module, first conversion module, second conversion module, tensor decomposition module, intent classification model generation module and training module, The data acquisition module is used to acquire intent classification task data; The rule writing module is used to write regular expression rules and corresponding logical selection operations for the intent classification data set; The first conversion module is used to convert the regular expression into a finite state automaton; The second conversion module is used to convert the finite state automaton into a three-dimensional tensor; The tensor decomposition module is used to decompose the three-dimensional tensor using a tensor decomposition method to obtain three matrices to initialize the parameters of the TFRNN; The intent classification model generation module is used to initialize the parameters of the logic selection layer, and the logic selection layer and TFRNN constitute the intent classification model. The TFRNN converts regular expressions into corresponding trainable recurrent neural networks; the tensor is regarded as a stack of transfer matrices corresponding to each word; the forward algorithm and Viterbi algorithm are used to run the finite state automaton, and at each time t, a vector with a dimension of the number of states can be obtained, which is called the state vector h t , where the i-th element represents the path with the highest score from the starting state to state i after receiving and reading t words. The state vector at the next time t + 1 can be calculated from the state vector at time t and the state transfer matrix corresponding to the t+1th word ,Through cyclic calculation, the value corresponding to the end state in the state vector at the last moment determines whether the regular expression matches after reading the entire sentence. For the forward algorithm, if there is an end state with a final value greater than or equal to 1, the regular expression matches. For the Viterbi algorithm, if there is an end state with a value equal to 1, the regular expression matches; The training module is used to input the intent classification task data into the intent classification model for training and output the intent classification result; The three-dimensional tensor contains all the information of the regular expression, the first dimension is the vocabulary size of the words involved in the finite state automaton, and the second and third dimensions are the number of states of the finite state automaton; The logic selection layer is used to sort the operation results of the finite state automaton into scores for each category. The logic selection layer includes three logic operations, including AND operation, OR operation and priority selection.

3. An intelligent terminal, comprising a processor, an input device, an output device and a memory, wherein the processor, the input device, the output device and the memory are interconnected, the memory is used to store a computer program, the computer program comprises program instructions, and is characterized in that: The processor is configured to call the program instructions to execute the method according to claim 1.

4. A computer-readable storage medium, characterized in that: The computer storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor, the processor is caused to perform the method according to claim 1.

Images