A question and answer method based on knowledge distillation, a terminal device, and a storage medium

Abstract

The application relates to a question and answer method based on knowledge distillation, a terminal device and a storage medium, and the method comprises the following steps: S1, collecting question and answer data in a demand field to form a training set; S2, performing feature extraction on each training data in the training set; S3, constructing and training a precision sorting model with high network complexity through the training set to generate a teacher network model; S4, constructing a precision sorting model with low network complexity, training the precision sorting model with low network complexity through the training set based on knowledge distillation and the teacher network model to generate a precision sorting student network model; S5, constructing a rough sorting model with low network complexity, training the rough sorting model with low network complexity through the training set based on knowledge distillation and the teacher network model to generate a rough sorting student network model; and S6, sequentially searching for corresponding answers from a question and answer database through the rough sorting student network model and the precision sorting student network model after performing feature extraction on a question to be answered. The application can improve the efficiency of a question and answer system.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of automatic question answering, and in particular to a question answering method based on knowledge distillation, a terminal device and a storage medium. BACKGROUND

[0002] With the rapid development of information technology, the increasing network information makes it difficult for users to quickly find the required content from the large amount of information returned by the search engine. People's demand for fast and accurate information is increasingly strong, and the automatic question answering system has emerged as the times require. The automatic question answering system provides a natural language question exchange mode for people, directly provides the required answers instead of related web pages, and has the characteristics of convenience, speed and efficiency. It is very popular with users.

[0003] The two necessary links for constructing a FAQ question answering system based on a vertical field are key information analysis and similarity calculation of the question, that is, element identification, wherein the key information analysis includes keyword extraction and deep semantic analysis. A large-scale vertical field FAQ question answering system often has a large enough question and answer library, that is, a corpus. A small model with low complexity and simple structure has low accuracy, and the automatic answering effect is poor; using a large model with high complexity can meet the effect requirement, but the large-scale parameter quantity is large, and a long reasoning calculation time is required, the prediction speed is slow, the efficiency of the question answering system is affected, and the production deployment requirement is not met. SUMMARY

[0004] To solve the above problems, the present application provides a question answering method based on knowledge distillation, a terminal device and a storage medium.

[0005] The specific scheme is as follows:

[0006] A question answering method based on knowledge distillation, comprising the following steps:

[0007] S1: collecting question and answer data in the required field to form a training set;

[0008] S2: performing feature extraction on each training data in the training set, wherein the extracted features include keyword features and semantic information features;

[0009] S3: constructing and training a precision ranking model with high network complexity by using the training set, and taking the trained model as a teacher network model;

[0010] The precision ranking model is used to extract the similarity between the semantic information features of two question sentences;

[0011] S4: constructing a precision ranking model with low network complexity, training the precision ranking model with low complexity based on knowledge distillation and the teacher network model by using the training set, and taking the trained model as a precision ranking student network model;

[0012] S5: Construct a coarse ranking model with low network complexity, train the coarse ranking model with the training set based on knowledge distillation and a teacher network model, and take the trained model as a coarse ranking student network model;

[0013] The coarse ranking model is used to extract the similarity between the keyword features of two question sentences.

[0014] S6: After feature extraction of the question to be answered, the corresponding answer is found from the question and answer library through the coarse ranking student network model and the fine ranking student network model in turn.

[0015] Further, the training data in the training set is composed of a standard question and a similar question similar to the standard question.

[0016] Further, the step S1 further includes pre-processing of the collected question and answer data, and the specific pre-processing process includes standard question extraction, answer acquisition, abnormal data cleaning and punctuation symbol cleaning.

[0017] Further, the semantic information feature is extracted by using a Bert based on a Transformer bidirectional encoder representation model.

[0018] Further, in the process of training the student network model based on knowledge distillation and the teacher network model, the loss function L Student is:

[0019] L Student = H (y, f (x)) + l * ||z t -z s || 2

[0020] Wherein, H represents a cross-entropy loss function, y represents a true answer, f(x) represents a prediction output of the student network model, l represents a distillation loss influence degree adjustment parameter, z t represents Logits output by the teacher network model, and z s represents Logits output by the student network model.

[0021] A question and answer terminal device based on knowledge distillation includes a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the method of the above-mentioned embodiment of the application when executing the computer program.

[0022] A computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the method of the above-mentioned embodiment of the application.

[0023] The application adopts the above technical solution, uses knowledge distillation, lets a student network with low complexity learn the ranking ability and semantic understanding ability of a teacher network, the teacher network guides the student network training, reduces the problem complexity of the student network, and finally obtains a good, small and fast student network model. It has important application value for promoting the deployment of large-scale FAQ question and answer systems on edge devices and the efficiency improvement of question and answer systems brought by attribute recognition inference speed improvement. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 A flowchart of an embodiment of the application is shown. DETAILED DESCRIPTION

[0025] To further illustrate the embodiments, the application provides accompanying drawings. These drawings are part of the disclosure of the application and mainly serve to illustrate the embodiments and can be used to explain the operating principle of the embodiments in conjunction with the related description of the specification. Those of ordinary skill in the art should be able to understand other possible implementations and advantages of the application by referring to these contents.

[0026] The application will be further described in conjunction with the accompanying drawings and specific embodiments.

[0027] Embodiment one:

[0028] The embodiment of the application provides a question and answer method based on knowledge distillation, as shown in the figure, the method comprises the following steps: Figure 1

[0029] S1: Collect question and answer data in the demand field to form a training set.

[0030] In the collection of question and answer data, real business scenario question and answer corpus can be crawled, and then question and answer data pairs can be generated manually or semi-manually. In this embodiment, 150W question and answer data pairs, i.e. 150W standard questions and corresponding standard question answers, are collected.

[0031] Since 150W question and answer data is not enough to support intelligent question and answer in a certain field in the field of intelligent question and answer, in order to automatically supplement the data quantity, in this embodiment, 750W similar questions are generated using a similar data generation module according to 150W standard questions, which together with the 150W standard questions form a training set. The similar data generation module can use existing technologies, which will not be described here.

[0032] The embodiment also includes preprocessing the collected question and answer data, and the specific preprocessing process includes: standard question extraction, answer acquisition, abnormal data cleaning, punctuation cleaning, etc. After preprocessing, 900W training data obtains 850W training data.

[0033] ​S2: feature extraction is performed on each training data in the training set, and the extracted features include keyword features and semantic information features.

[0034] The extraction of keyword features and semantic information features can use existing models respectively. In this embodiment, the extraction of semantic information features preferably uses Bert (Bidirectional Encoder Representations from Transformers) based on a Transformer bidirectional encoder representation model, which performs model compression on information acquisition of natural language.

[0035] It should be noted that keyword features can also be used as keyword features in other embodiments, which are not limited here.

[0036] S3: a fine ranking model with high network complexity is constructed and trained by using the training set, and the trained model is used as a teacher network model.

[0037] The fine ranking model is used to extract the similarity between the semantic information features of two questions, and the input is the semantic information features of two questions, and the output is the similarity between the semantic information features. The training data used to train the fine ranking model should include the semantic information features of two questions and the semantic similarity of the two questions. In this embodiment, when the input two questions are a standard question and its similar question, the semantic similarity is 1, and when the input two questions are two different standard questions or a standard question and a similar question of another standard question, the semantic similarity is 0.

[0038] The fine ranking model with high network complexity can select a deep DNN ranking model.

[0039] S4: a fine ranking model with low network complexity is constructed, based on knowledge distillation and a teacher network model, the training set is used to train the fine ranking model with low complexity, and the trained model is used as a fine ranking student network model.

[0040] The fine ranking model in step S4 has the same model function as the fine ranking model in step S3, but due to its low network complexity, it can use a shallow DNN ranking model, a non-deep model such as LR / FM.

[0041] S5: a coarse ranking model with low network complexity is constructed, based on knowledge distillation and a teacher network model, the training set is used to train the coarse ranking model with low complexity, and the trained model is used as a coarse ranking student network model.

[0042] The coarse ranking model is used to extract the similarity between the keyword features of two question sentences, and the input is the keyword features of the two question sentences, and the output is the similarity between the keyword features. The training data used to train the coarse ranking model should include the keyword features of two question sentences and the keyword feature similarity corresponding to the two question sentences. In this embodiment, when the input two question sentences are a standard question and its similar question, the keyword feature similarity is 1, and when the input two question sentences are two different standard questions or a standard question and another standard question similar question, the keyword feature similarity is 0.

[0043] As a pre-stage of the fine ranking model, the coarse ranking model needs to find a balance between speed and accuracy. Its positioning is not to pursue accurate similar questions, and the effect can be worse than that of the fine ranking model, but it can be compensated by the return quantity, so the small size and fast speed of the model are important goals of the coarse ranking model.

[0044] The knowledge distillation used by the coarse ranking model is not limited to using a large coarse ranking model as a teacher model, but uses a fine ranking model as a teacher model, and a coarse ranking model such as FM or a double-tower DNN model as a student model. The coarse ranking model simulates the ranking result of the fine ranking model link to guide the optimization process of the coarse ranking model.

[0045] S6: After feature extraction of the question to be answered, the corresponding answer is found from the question and answer library by the coarse ranking student network model and the fine ranking student network model in turn.

[0046] After feature extraction of the question to be answered, first, the similar question sentences with high keyword feature similarity are found from the question and answer library by the coarse ranking student network model and its keyword features, and then the similar question sentences with high semantic similarity are found from all the similar question sentences with high keyword feature similarity by the fine ranking student model, and then the corresponding answer is obtained according to the similar question sentences with high semantic similarity.

[0047] In the above steps S4 and S5, the process of training the student network model based on knowledge distillation and teacher network model includes:

[0048] S401: Obtain the true answer in the training data of the input model; Specifically, it is obtained by manual annotation and keyword in the question sentence;

[0049] S402: Calculate the cross-entropy loss H(y, f(x)) of the true label and the student network model according to the training data;

[0050] S403: According to the training data, the Logits output by the teacher network model and the Logits output by the student network model are calculated, the Logits output by the student network model is fitted to the Logits output by the teacher network model, so that the teacher network model enhances the generalization ability of the student network model ||z t -z s || 2 ;

[0051] S404: Calculate the total loss function L Student :

[0052] L Student =H(y,f(x))+λ*||z t -z s || 2

[0053] Wherein, H represents the cross-entropy loss function, y represents the true answer, f(x) represents the prediction output of the student network model, and lambda represents the influence degree adjustment parameter of distillation loss, z t represents the Logits output by the teacher network model (the last layer output of the neural network), z s represents the Logits output by the student network model.

[0054] The embodiment of the application proposes a question and answer method based on knowledge distillation, which performs "slimming" on the whole process of the question and answer system. Directly using the conventional Ground Truth to train the small model is not ideal in effect, so different modules of the question and answer system can be learned by a small model with lower complexity to obtain a small and fast model. When training the small model, in addition to using the conventional Ground Truth to train the small model, the large model is used to assist the training of the small model, and some knowledge learned by the large model is transferred to the small model.

[0055] Embodiment two:

[0056] The application also provides a question and answer terminal device based on knowledge distillation, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps in the above method embodiments of the embodiment one of the application when executing the computer program.

[0057] Further, as an executable solution, the knowledge distillation-based question answering terminal device can be a desktop computer, a notebook computer, a palm computer, a cloud server, and the like. The knowledge distillation-based question answering terminal device can include, but is not limited to, a processor, a memory. Those skilled in the art can understand that the above-mentioned composition structure of the knowledge distillation-based question answering terminal device is only an example of the knowledge distillation-based question answering terminal device, and does not constitute a limitation on the knowledge distillation-based question answering terminal device, and can include more or fewer components than the above, or combine certain components, or different components, for example, the knowledge distillation-based question answering terminal device can also include an input / output device, a network access device, a bus, and the like, and the embodiments of the present application do not limit this.

[0058] Further, as an executable solution, the processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, and the like. The processor is the control center of the knowledge distillation-based question answering terminal device, and connects various parts of the knowledge distillation-based question answering terminal device through various interfaces and lines.

[0059] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the knowledge distillation-based question answering terminal device by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required by a function; the data storage area can store data created according to the use of the mobile phone, and the like. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, for example, a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory devices.

[0060] The application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the method of the above-mentioned embodiments of the application.

[0061] The modules / units of the knowledge distillation-based question and answer terminal device integration, if realized in the form of software function units and sold or used as independent products, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. When the computer program is executed by a processor, the steps of each method embodiment can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms, etc. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM) and software distribution medium, etc.

[0062] Although the application is specifically shown and described in connection with the preferred embodiments, those skilled in the art should understand that various changes in form and details can be made to the application without departing from the spirit and scope of the application as defined in the appended claims.

Claims

1. A knowledge distillation based question answering method, characterized in that, The method comprises the following steps: S1: collecting question and answer data in a demand field to form a training set; S2: extracting features from each training data in the training set, the extracted features including keyword features and semantic information features; S3: constructing and training a fine ranking model with high network complexity by using the training set, and taking the trained model as a teacher network model; the fine ranking model with high network complexity includes a DNN ranking model with deep layers; The fine ranking model is used to extract the similarity between the semantic information features of two question sentences; S4: constructing a fine ranking model with low network complexity, training the fine ranking model with low network complexity by using the training set based on knowledge distillation and the teacher network model, and taking the trained model as a fine ranking student network model; the fine ranking model with low network complexity includes a DNN ranking model with shallow layers and a LR / FM non-depth model; S5: constructing a coarse ranking model with low network complexity, training the coarse ranking model with low network complexity by using the training set based on knowledge distillation and the teacher network model, and taking the trained model as a coarse ranking student network model; the coarse ranking model with low network complexity includes an FM model or a double-tower DNN model; The coarse ranking model is used to extract the similarity between the keyword features of two question sentences. S6: after extracting features from a question to be answered, sequentially passing the question through the coarse ranking student network model and the fine ranking student network model to find a corresponding answer from a question and answer database.

2. The knowledge distillation based question answering method of claim 1, wherein: The training data in the training set is composed of a standard question and a similar question similar to the standard question.

3. The knowledge distillation based question answering method of claim 1, wherein: The step S1 further comprises preprocessing the collected question and answer data, and the specific preprocessing process includes standard question extraction, answer acquisition, abnormal data cleaning, and punctuation symbol cleaning.

4. The knowledge distillation based question answering method of claim 1, wherein: The extraction of semantic information features adopts a Bert based on a Transformer bidirectional encoder representation model.

5. The knowledge distillation based question answering method of claim 1, wherein: Loss function in a process of training a student network model based on knowledge distillation and a teacher network model is: wherein, represents a cross-entropy loss function, y represents a true answer, represents a predicted output of the student network model, and λ represents a distillation loss influence degree adjustment parameter, represents Logits output by the teacher network model, represents Logits output by the student network model.

6. A question answering terminal device based on knowledge distillation, characterized by: The computer program is executed by the processor to realize the steps of the method according to any one of claims 1-5.

7. A computer readable storage medium storing a computer program, characterized in that: The computer program is executed by the processor to realize the steps of the method according to any one of claims 1-5.

Images