Method, device, computing device and storage medium for generating intelligent question-answering model

Abstract

The present invention provides a method for generating an intelligent question-answering model, comprising: expanding an original sample data set so that there are positive samples corresponding to questions in the original sample data set, generating a new sample data set; inputting the generated new sample data set into an existing question-answering model, wherein the existing question-answering model obtains feature codes related to the question from a knowledge base and documents in the new sample data set, and calculating the loss L of the existing question-answering model. qa ; Add the obtained feature encoding to the improved contrastive learning model, calculate the similarity between the true sample and the difficult positive and negative samples in the new sample data set, and obtain the contrast loss L cl ; The loss L of the existing question-answering model qa and contrast loss L cl The loss L of the final question-answering model is obtained by combining them, and the final question-answering model is trained. The present invention adds a contrastive learning model to the existing question-answering model, and the model has stronger generalization ability. At the same time, the effect of knowledge base question answering is better than that of traditional methods.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and in particular to a method, device, computing equipment and storable medium for generating an intelligent question-answering model. Background Art

[0002] In a natural language-based question-answering system, the knowledge base is a necessary resource for answering factual questions. The knowledge base can also be called a knowledge graph, which is widely used in major browsers and recommendation algorithms. For example, Google Chrome has Google Knowledge Graph, Microsoft's Bing search has Bing Satori, Baidu's Baidu Knowledge Graph, and Facebook Social Graph in the recommendation system Facebook and Alicoco in Taobao, etc.

[0003] The knowledge base often contains a lot of triple information, each of which describes a fact. Generally, triples are expressed in two forms: (head entity, relationship, tail entity) and (entity, attribute, value). Taking the first form as an example, the natural language "the capital of China is Beijing" can be expressed in the form of a triple as (China, capital, Beijing).

[0004] With the rapid development of Internet technology, the information in the knowledge base has also shown an explosive growth trend. At present, in the knowledge base question-answering model in the Internet field, users generally ask questions in natural language and input them into the knowledge base question-answering model. The question-answering model performs semantic analysis and output on the questions raised by users, and then uses the information in the knowledge base to query and match, and then gives answers.

[0005] However, the current knowledge base question-answering model does not have a deep understanding of the knowledge base information, that is, the knowledge base may lack critical knowledge that can be used to correctly answer questions, resulting in a low accuracy rate of the answer of the question-answering model. The question-answering model needs to be further improved. Summary of the invention

[0006] The purpose of the embodiments of the present invention is to provide a method, apparatus, terminal device and storable medium for generating an intelligent question-answering model to solve the accuracy problem in knowledge base question-answering.

[0007] In order to achieve the aforementioned invention objectives, the technical solutions provided by the embodiments of the present invention are as follows:

[0008] In a first aspect, an embodiment of the present invention provides a method for generating an intelligent question-answering model, the method comprising:

[0009] Expanding the original sample data set so that there are positive samples corresponding to the questions in the original sample data set, generating a new sample data set, wherein the positive samples are defined as questions with the same answers as the original sample data set, and questions with different answers from the original sample data set are defined as negative samples;

[0010] The generated new sample dataset is input into the existing question answering model, which obtains the feature encoding related to the question from the knowledge base and documents in the new sample dataset, and calculates the loss L of the existing question answering model. qa ;

[0011] The obtained feature encoding is added to the improved contrastive learning model, and the similarity between the true sample and the difficult positive and negative samples in the new sample dataset is calculated respectively to obtain the contrast loss L cl ;

[0012] Linear fusion is used to transform the loss L of the existing question-answering model qa and contrast loss L cl The loss L of the final question-answering model is obtained by combining them, and the final question-answering model is obtained by training.

[0013] Specifically, the expanding the original sample data set so that there are positive samples corresponding to the problems in the original sample data set, and generating a new sample data set specifically includes:

[0014] According to the questions in the original sample data set and the triple information of the knowledge base, positive samples corresponding to the questions in the original sample data set are generated, where questions with the same answers as in the original sample data set are defined as positive samples, and questions with different answers are defined as negative samples;

[0015] For each data sample, three positive samples are generated to obtain a new sample data set, where the original sample data set is the public question-answering data set WebQuestionSP.

[0016] Specifically, the existing question-answering model obtains feature encoding related to the question from the knowledge base and documents in the new sample data set, specifically including:

[0017] The existing question-answering model encodes the question, knowledge base triple information and document information respectively, and obtains their semantic expressions q, KB i and doc j , among which, KB i Represents the semantic expression of the i-th subgraph in the knowledge base, i = 1, 2, .., l kb , l kb Represents the total number of subgraphs in the knowledge base; doc j Represents the semantic expression of the jth subgraph in the document, j = 1, 2, .., l doc , ldoc Indicates the total number of documents;

[0018] Use LSTM attention encoder to encode the question and obtain the question feature vector

[0019] Obtaining entity feature vectors of the knowledge base and entity feature vectors of the document;

[0020] Obtain the entity feature vector ent related to the question based on the entity feature vector of the knowledge base and the entity feature vector of the document;

[0021] The problem feature vector The entity feature vector ent is used as the feature encoding.

[0022] Specifically, encoding the obtained positive samples using a momentum update encoder specifically includes:

[0023] Label the original sample data set as true sample x query , calculate the true sample x query The feature vector query of

[0024] The three expanded positive sample data are marked as Calculate the feature vector (k1, k2, k3) of the positive sample, where

[0025]

[0026]

[0027] θ k ←mθ k +(1-m)θ q , m∈[0,1)

[0028] where f q =encoder,f k =moment encoder is the existing question-answering model and momentum update encoder, the former model parameter θ q Back propagation is used to update the model parameters θ k The momentum update equation is adopted, and m is the momentum update parameter.

[0029] Specifically, the obtained feature encoding is added to the improved contrastive learning model, and the similarity between the true sample and the difficult positive and negative samples in the new sample data set is calculated respectively to obtain the contrast loss L cl , and then linear fusion is used to transform the loss L of the existing question-answering model qa and contrast loss L clCombined, the loss L of the final question-answering model includes:

[0030] Define a negative sample queue Z neg ={z1, z2, ..., z K}, where z i represents the i-th negative sample in the sample data set, and K represents the number of negative sample queues.

[0031] Specifically, the difficult positive sample vector k hard is the sample that is least similar to the true sample among all positive samples. Here, the embodiment of the present invention proposes a forward inference method to obtain difficult positive samples. This method calculates the cosine similarity between the true sample and the positive sample, and finally extracts the positive sample with the smallest similarity as k hard :

[0032]

[0033] For negative sample vectors, the embodiment of the present invention proposes a new method (negative fusion method), that is, linearly fusing positive samples and true samples to form a new difficult negative sample vector Then update the negative sample queue Z neg The specific formula is as follows:

[0034]

[0035] Among them, λ represents the allocation weight.

[0036] In contrast to extracting difficult positive samples, the difficult negative sample queue is the queue of all negative samples Z neg The part of samples that is most similar to the true sample. First, neg Sort the negative samples and true samples from large to small according to their similarity, i.e.: ||query||||z i ||>||query||||z j ||, (i<j), and then only the top-K negative samples with the largest similarity are extracted to form satisfy:

[0037]

[0038] where λ neg is a hyperparameter set artificially. Through experiments, it is found that neg =0.95 Can reach the same level as the full Z neg The same or even better experimental results are achieved, while the number of difficult negative sample queues in the embodiment of the present invention is only 2000, which is much smaller than K.

[0039] According to the difficult negative sample queue vector obtained above and difficult positive sample vector k hard , the embodiment of the present invention further proposes the following improved contrast loss function L cl ,

[0040]

[0041] The final model loss function L,

[0042] L=λ qa L qa +λ cl L cl

[0043] Among them, τ is a constant, λ qa and λ cl Represent the loss weights of the question-answering model and the comparison model respectively.

[0044] In a second aspect, an embodiment of the present invention provides a device for generating an intelligent question-answering model, the device comprising:

[0045] A sample set generation module is used to expand the original sample data set so that there are positive samples corresponding to the questions in the original sample data set, and generate a new sample data set, wherein the positive samples are defined as questions with the same answers as the original sample data set, and questions with different answers from the original sample data set are defined as negative samples;

[0046] The encoding module is used to input the generated new sample data set into the existing question-answering model, wherein the existing question-answering model obtains feature encodings related to the question from the knowledge base and documents in the new sample data set, and calculates the loss L of the existing question-answering model. qa ;

[0047] The contrast module is used to add the obtained feature encoding to the improved contrast learning model, calculate the similarity between the true sample and the difficult positive and negative samples in the new sample data set, and obtain the contrast loss L cl ;

[0048] The training module is used to use linear fusion to transform the loss L of the existing question-answering model qa and contrast loss L cl The loss L of the final question-answering model is obtained by combining them, and the final question-answering model is obtained by training.

[0049] Preferably, the encoding module specifically includes:

[0050] The first encoding unit is used in the existing question-answering model to encode the question, knowledge base triple information and document information respectively, and obtain their semantic expressions q, KB respectively. i and docj , among which, KB i Represents the semantic expression of the i-th subgraph in the knowledge base, i = 1, 2, .., l kb , here l kb Represents the total number of subgraphs in the knowledge base; doc j Represents the semantic expression of the jth subgraph in the document, j = 1, 2, .., l doc , l doc Indicates the total number of documents;

[0051] The second encoding unit is used to encode the question using the LSTM attention encoder to obtain the question feature vector A first vector acquisition unit, used to obtain an entity feature vector of a knowledge base and an entity feature vector of a document;

[0052] The second vector acquisition unit is used to obtain an entity feature vector ent related to the question according to the entity feature vector of the knowledge base and the entity feature vector of the document, and convert the question feature vector The entity feature vector ent is used as the feature encoding.

[0053] In a third aspect, an embodiment of the present invention provides a computing device, the computing device comprising a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement the method for generating an intelligent question-answering model as described above.

[0054] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the method for generating an intelligent question and answer model as described above.

[0055] The embodiment of the present invention adds a contrastive learning model to the existing question-answering model, so that the new question-answering model has stronger model generalization ability. In addition, traditional contrastive learning simply replaces the existing negative samples with positive samples when updating the negative sample queue, and cannot effectively retain difficult positive and negative samples, thereby resulting in reduced optimization efficiency and performance. In the embodiment of the present invention, a positive inference method and a negative fusion method are also proposed for the samples. This method can effectively extract difficult positive and negative samples and discard tens of thousands of negative sample queues (the present invention only needs about 3% of the traditional negative queues), which not only greatly improves the time and space efficiency, but also improves the performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The following will explain the preferred implementation mode in a clear and easy-to-understand manner with reference to the accompanying drawings, and further explain the above-mentioned characteristics, technical features, advantages and implementation methods of the intelligent question and answer model generation method, device, terminal device and storable medium of the embodiments of the present invention.

[0057] Figure 1 A schematic diagram of the overall model framework of the method for generating an intelligent question-answering model according to an embodiment of the present invention;

[0058] Figure 2 A schematic diagram of data expansion of a method for generating an intelligent question-answering model according to an embodiment of the present invention;

[0059] Figure 3 A schematic diagram of a question-answering model of a method for generating an intelligent question-answering model according to an embodiment of the present invention;

[0060] Figure 4 A schematic diagram of a comparative learning model of a method for generating an intelligent question-answering model according to an embodiment of the present invention;

[0061] Figure 5 A schematic diagram of a computing device for a method for generating an intelligent question-answering model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0062] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the specific implementation methods of the present invention will be described below with reference to the accompanying drawings. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings and other implementation methods can be obtained based on these drawings without creative work.

[0063] In order to simplify the drawings, only the parts related to the present invention are schematically shown in each figure, and they do not represent the actual structure of the product. In addition, in order to simplify the drawings and facilitate understanding, in some figures, only one of the parts with the same structure or function is schematically drawn or marked. In this article, "one" not only means "only one", but also means "more than one".

[0064] In order to achieve the aforementioned invention objectives, the technical solutions provided by the embodiments of the present invention are as follows:

[0065] like Figure 1 As shown, an embodiment of the present invention provides a method for generating an intelligent question-answering model, the method comprising:

[0066] S1. Expand the original sample data set so that there are positive samples corresponding to the questions in the original sample data set, and generate a new sample data set, wherein the positive samples are defined as questions with the same answers as the original sample data set, and questions with different answers to the original sample data set are defined as negative samples;

[0067] S2. Input the generated new sample dataset into the existing question-answering model, which obtains the feature encoding related to the question from the knowledge base and documents in the new sample dataset, and calculates the loss L of the existing question-answering model qa ;

[0068] S3. Add the obtained feature encoding to the contrastive learning model, calculate the similarity between the true sample and the positive and negative samples in the new sample data set, and obtain the contrast loss L cl ;

[0069] S4. Use linear fusion to transform the loss L of the existing question-answering model qa and contrast loss L cl The loss L of the final question-answering model is obtained by combining them, and the final question-answering model is obtained by training.

[0070] like Figure 2 As shown, the data set is first expanded and then passed in Figure 3 The question-answering model outputs the feature encoding of the question and related entities, and then passes the feature encoding into Figure 4 In the comparison model, the loss function is calculated and used Figure 2 The question-answering model predicts the final answer.

[0071] First, the original sample dataset is expanded so that there are positive samples corresponding to the original sample dataset. Specifically, the questions in the original sample dataset and the knowledge base triple information are used to generate positive samples corresponding to the questions in the original sample dataset, where questions with the same answers in the original sample dataset are defined as positive samples, and questions with different answers are defined as negative samples. In addition, three positive samples are generated for each data sample to obtain a new sample dataset, where the original sample dataset is the public question-answering dataset WebQuestionSP.

[0072] The generated new sample data set is added to the existing question-answering model, which obtains the feature encoding related to the question from the knowledge base and documents in the new sample data set, and calculates the loss of the existing question-answering model, which is specifically:

[0073] Input the new sample data set into the existing question-answering model, wherein the existing question-answering model mainly includes: SGReader(·) for question understanding based on knowledge base and KAReader(·) for question understanding based on document;

[0074] The glove model is used to encode the question, knowledge base triple information and document information respectively, and their semantic expressions q, KB are obtained respectively. i and doc j , among which, KBi Represents the semantic expression of the i-th subgraph in the knowledge base, i = 1, 2, .., l kb , here l kb Represents the total number of subgraphs in the knowledge base; doc j Represents the semantic expression of the jth subgraph in the document, j = 1, 2, .., l doc , l doc Indicates the total number of documents;

[0075] Use LSTM attention encoder to encode the question and obtain the question feature vector According to the following formula, we can obtain the entity feature vector ent related to the question and the probability s of each entity information being the answer. e and the loss L of the question-answering model qa ,

[0076]

[0077] ent=[e kb ;e d ]

[0078]

[0079] L qa = bce(s e , answer)

[0080] Among them, e kb Represents the entity feature vector of the knowledge base, e d Represents the entity feature vector of the document. The two are concatenated to form the entity feature vector ent, σ s (·) represents the activation function, W s represents a training weight matrix and bce(·) represents the binary cross entropy function.

[0081] The problem feature vector obtained The entity feature vector ent is added to the improved contrastive learning model, and the similarity between the true sample and the difficult positive and negative samples in the new sample data set is calculated respectively to obtain the contrast loss. Then, the loss L of the question-answering model is fused linearly. qa and contrast loss L cl Combined, the loss L of the final overall model is obtained and the model is trained. Finally, the question-answering model is used to predict and generate answers. The specific steps are:

[0082] Separate the new sample data set, where the original sample data set is marked as true sample x query , the three expanded positive sample data are marked as According to the following formula, the feature vectors query and (k1, k2, k3) of the true sample and the positive sample are calculated respectively.

[0083]

[0084]

[0085]

[0086]

[0087] θ k ←mθ k +(1-m)θ q , m∈[0,1)

[0088] where f q =encoder,f k =moment encoder is the existing question-answering model and momentum encoder, the former model parameter θ q Back propagation is used to update the model parameters θ k The momentum update equation is used, where m is the momentum update parameter; the true sample feature vector query is composed of the true sample x query Using the problem feature vector And the entity feature vector ent is concatenated. Similarly, the positive sample feature vector is (k1, k2, k3).

[0089] Define a negative sample queue Z neg ={z1, z2, ..., z K}, where z i represents the i-th negative sample in the sample data set, and K represents the number of negative sample queues. In the traditional contrastive learning method, K is an artificially set hyperparameter, generally 60,000, but the lengthy queue increases the computational and storage costs. Compared with traditional contrastive learning, the embodiment of the present invention innovatively proposes the concept of difficult positive and negative samples, and only uses difficult positive and negative samples for contrastive learning. This solution greatly simplifies the original learning process and improves optimization efficiency and model performance.

[0090] Specifically, the difficult positive sample vector k hard is the sample that is least similar to the true sample among all positive samples. Here, the embodiment of the present invention proposes a forward inference method to obtain difficult positive samples. This method calculates the cosine similarity between the true sample and the positive sample, and finally extracts the positive sample with the smallest similarity as k hard :

[0091]

[0092] For negative sample vectors, the embodiment of the present invention proposes a new method (negative fusion method), that is, linearly fusing positive samples and true samples to form a new difficult negative sample vector Then update the negative sample queue Z neg The specific formula is as follows:

[0093]

[0094] Among them, λ represents the allocation weight.

[0095] In contrast to extracting difficult positive samples, the difficult negative sample queue is the queue of all negative samples Z neg The part of samples that is most similar to the true sample. First, neg Sort the negative samples and true samples from large to small according to their similarity, i.e.: ||query||||z i ||>||query||||z j ||, (i<j), and then only the top-K negative samples with the largest similarity are extracted to form satisfy:

[0096]

[0097] where λ neg is a hyperparameter set artificially. Through experiments, it is found that neg =0.95 Can reach the same level as the full Z neg The same or even better experimental results are achieved, while the number of difficult negative sample queues in the embodiment of the present invention is only 2000, which is much smaller than K.

[0098] According to the difficult negative sample queue vector obtained above and difficult positive sample vector k hard , the embodiment of the present invention further proposes the following improved contrast loss function L cl ,

[0099]

[0100] The final model loss function L,

[0101] L=λ qa L qa +λ cl L cl

[0102] Among them, τ is a constant, λ qa and λ cl Represent the loss weights of the question-answering model and the comparison model respectively.

[0103] The above method is applied to the following embodiments to demonstrate the technical effects of the present invention, and the specific steps in the embodiments will not be repeated.

[0104] The embodiment of the present invention is experimented on the public dataset WebQuestionSP. In order to objectively evaluate the performance of the algorithm of the present invention, the embodiment of the present invention uses the common evaluation criteria Hit@1 and F1 in the question-answering system. Hit@1 represents the accuracy of answer prediction, and F1 represents the accuracy of the model. The experimental results obtained are shown in Table 1, which compares the experimental results of multiple knowledge base question-answering models of different degrees, among which KDReader is the experimental effect of the question-answering model in the embodiment of the present invention, and Ours is the experimental result of the embodiment of the present invention.

[0105] The test results in Table 1 reflect that the embodiment of the present invention can effectively learn the information of the knowledge base triples and has a better effect in predicting answers than the traditional question-answering model method.

[0106] Table 1 Comparison of experimental results of various models based on different levels of knowledge base information

[0107]

[0108] On the other hand, an embodiment of the present invention provides a device for generating an intelligent question-answering model, the device comprising:

[0109] A sample set generation module is used to expand the original sample data set so that there are positive samples corresponding to the questions in the original sample data set, and generate a new sample data set, wherein the positive samples are defined as questions with the same answers as the original sample data set, and questions with different answers from the original sample data set are defined as negative samples;

[0110] The encoding module is used to input the generated new sample data set into the existing question-answering model, wherein the existing question-answering model obtains feature encodings related to the question from the knowledge base and documents in the new sample data set, and calculates the loss L of the existing question-answering model. qa ;

[0111] The contrast module is used to add the obtained feature encoding to the improved contrast learning model, calculate the similarity between the true sample and the difficult positive and negative samples in the new sample data set, and obtain the contrast loss L cl ;

[0112] The training module is used to use linear fusion to transform the loss L of the existing question-answering model qa and contrast loss L cl The loss L of the final question-answering model is obtained by combining them, and the final question-answering model is obtained by training.

[0113] Preferably, the encoding module specifically includes:

[0114] The first encoding unit is used in the existing question-answering model to encode the question, knowledge base triple information and document information respectively, and obtain their semantic expressions q, KB respectively. i and doc j , among which, KB i Represents the semantic expression of the i-th subgraph in the knowledge base, i = 1, 2, .., l kb , here l kb Represents the total number of subgraphs in the knowledge base; doc j Represents the semantic expression of the jth subgraph in the document, j = 1, 2, .., l doc , l doc Indicates the total number of documents;

[0115] The second encoding unit is used to encode the question using the LSTM attention encoder to obtain the question feature vector A first vector acquisition unit, used to obtain an entity feature vector of a knowledge base and an entity feature vector of a document;

[0116] The second vector acquisition unit is used to obtain an entity feature vector ent related to the question according to the entity feature vector of the knowledge base and the entity feature vector of the document, and convert the question feature vector The entity feature vector ent is used as the feature encoding.

[0117] First, the original sample dataset is expanded so that there are positive samples corresponding to the original sample dataset. Specifically, the questions in the original sample dataset and the knowledge base triple information are used to generate positive samples corresponding to the questions in the original sample dataset, where questions with the same answers in the original sample dataset are defined as positive samples, and questions with different answers are defined as negative samples. In addition, three positive samples are generated for each data sample to obtain a new sample dataset, where the original sample dataset is the public question-answering dataset WebQuestionSP.

[0118] The generated new sample data set is added to the existing question-answering model, which obtains the feature encoding related to the question from the knowledge base and documents in the new sample data set, and calculates the loss of the existing question-answering model, which is specifically:

[0119] Input the new sample data set into the existing question-answering model, wherein the existing question-answering model mainly includes: SGReader(·) for question understanding based on knowledge base and KAReader(·) for question understanding based on document;

[0120] The glove model is used to encode the question, knowledge base triple information and document information respectively, and their semantic expressions q, KB are obtained respectively. iand doc j , among which, KB i Represents the semantic expression of the i-th subgraph in the knowledge base, i = 1, 2, .., l kb , here l kb Represents the total number of subgraphs in the knowledge base; doc j Represents the semantic expression of the jth subgraph in the document, j = 1, 2, .., l doc , l doc Indicates the total number of documents;

[0121] Use LSTM attention encoder to encode the question and obtain the question feature vector According to the following formula, we can obtain the entity feature vector ent related to the question and the probability s of each entity information being the answer. e and the loss L of the question-answering model qa ,

[0122]

[0123] ent=[e kb ;e d ]

[0124]

[0125] L qa = bce(s e , answer)

[0126] Among them, e kb Represents the entity feature vector of the knowledge base, e d Represents the entity feature vector of the document. The two are concatenated to form the entity feature vector ent, σ s (·) represents the activation function, W s represents a training weight matrix and bce(·) represents the binary cross entropy function.

[0127] The problem feature vector obtained The entity feature vector ent is added to the contrastive learning model, and the similarity between the true sample and the positive and negative samples in the new sample data set is calculated respectively, so as to obtain the contrast loss. Then, the loss L of the question-answering model is fused linearly. qa and contrast loss L cl Combined, the loss L of the final overall model is obtained and the model is trained. Finally, the question-answering model is used to predict and generate answers. The specific steps are:

[0128] Separate the new sample data set, where the original sample data set is marked as true sample x query , the three expanded positive sample data are marked as According to the following formula, the feature vectors query and (k1, k2, k3) of the true sample and the positive sample are calculated respectively.

[0129]

[0130]

[0131]

[0132]

[0133] θ k ←mθ k +(1-m)θ q , m∈[0,1)

[0134] where f q =encoder,f k =moment encoder is the existing question-answering model and momentum encoder, the former model parameter θ q Back propagation is used to update the model parameters θ k The momentum update equation is used, where m is the momentum update parameter; the true sample feature vector query is composed of the true sample x query Using the problem feature vector And the entity feature vector ent is concatenated. Similarly, the positive sample feature vector is (k1, k2, k3).

[0135] Define a negative sample queue Z neg ={z1, z2, ..., z K}, where z i represents the i-th negative sample in the sample data set, and K represents the number of negative sample queues. In the traditional contrastive learning method, K is an artificially set hyperparameter, generally 60,000, but the lengthy queue increases the computational and storage costs. Compared with traditional contrastive learning, the embodiment of the present invention innovatively proposes the concept of difficult positive and negative samples, and only uses difficult positive and negative samples for contrastive learning. This solution greatly simplifies the original learning process and improves optimization efficiency and model performance.

[0136] Specifically, the difficult positive sample vector k hard is the sample that is least similar to the true sample among all positive samples. Here, the embodiment of the present invention proposes a forward inference method to obtain difficult positive samples. This method calculates the cosine similarity between the true sample and the positive sample, and finally extracts the positive sample with the smallest similarity as k hard :

[0137]

[0138] For negative sample vectors, the embodiment of the present invention proposes a new method (negative fusion method), that is, linearly fusing positive samples and true samples to form a new difficult negative sample vector Then update the negative sample queue Z neg The specific formula is as follows:

[0139]

[0140] Among them, λ represents the allocation weight.

[0141] In contrast to extracting difficult positive samples, the difficult negative sample queue is the queue of all negative samples Z neg The part of samples that is most dissimilar to the true samples. First, neg Sort the negative samples and true samples from large to small according to their similarity, i.e.: ||query||||z i ||>||query||||z j ||, (i<j), and then only the top-K negative samples with the largest similarity are extracted to form satisfy:

[0142]

[0143] where λ neg is a hyperparameter set artificially. Through experiments, it is found that neg =0.95 Can reach the same level as the full Z neg The same or even better experimental results are achieved, while the number of difficult negative sample queues in the embodiment of the present invention is only 2000, which is much smaller than K.

[0144] According to the difficult negative sample queue vector obtained above and difficult positive sample vector k hard , the embodiment of the present invention further proposes the following improved contrast loss function L cl ,

[0145]

[0146] The final model loss function L,

[0147] L=λ qa L qa +λ ci L cl

[0148] Among them, τ is a constant, λ qa and λ cl Represent the loss weights of the question-answering model and the comparison model respectively.

[0149] According to the above embodiment, in contrast to extracting difficult positive samples, the difficult negative sample queue is the queue of all negative samples Z neg The part of the sample that is most similar to the true sample, through experiments, it is found that λ neg =0.95 Can reach the same level as the full Z neg The same or even better experimental results are achieved, while the number of difficult negative sample queues in the embodiment of the present invention is only 2000, which is much smaller than K.

[0150] In the third aspect, an embodiment of the present invention also provides a computing device, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by the processor to implement the method for generating an intelligent question and answer model as described above.

[0151] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the method for generating an intelligent question and answer model as described above.

[0152] Please refer to Figure 5 , which shows a schematic diagram of the structure of a computing device 1500 provided in one embodiment of the present application. The computing device 1500 can be used to implement the method for generating the intelligent question-answering model as described above in the above embodiment.

[0153] Specifically:

[0154] The computing device 1500 includes a central processing unit (CPU) 1501, a system memory 1504 including a random access memory (RAM) 1502 and a read-only memory (ROM) 1503, and a system bus 1505 connecting the system memory 1504 and the central processing unit 1501. The computing device 1500 also includes a basic input / output system (I / O system) 1506 that helps transfer information between various components in the computer, and a mass storage device 1507 for storing an operating system 1513, application programs 1514, and other program modules 1515.

[0155] The basic input / output system 1506 includes a display 1508 for displaying information and an input device 1509 such as a mouse and a keyboard for user inputting information. The display 1508 and the input device 1509 are connected to the central processing unit 1501 through an input / output controller 1510 connected to the system bus 1505. The basic input / output system 1506 may also include an input / output controller 1510 for receiving and processing inputs from a plurality of other devices such as a keyboard, a mouse, or an electronic stylus. Similarly, the input / output controller 1510 also provides output to a display screen, a printer, or other types of output devices.

[0156] The mass storage device 1507 is connected to the central processing unit 1501 through a mass storage controller (not shown) connected to the system bus 1505. The mass storage device 1507 and its associated computer-readable media provide non-volatile storage for the computing device 1500. That is, the mass storage device 1507 may include a computer-readable medium (not shown) such as a hard disk or CD-ROM drive. Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, EPROM, EEPROM, flash memory or other solid-state storage technology, CD-ROM, DVD or other optical storage, cassettes, magnetic tape, disk storage or other magnetic storage devices.

[0157] Of course, those skilled in the art will appreciate that the computer storage media are not limited to the above mentioned ones. The above mentioned system memory 1504 and large capacity storage device 1507 can be collectively referred to as memory.

[0158] According to various embodiments of the present application, the computing device 1500 may also be connected to a remote computer on a network through a network such as the Internet. That is, the computing device 1500 may be connected to a network 1512 through a network interface unit 1511 connected to the system bus 1505, or the network interface unit 1511 may be used to connect to other types of networks or remote computer systems (not shown).

[0159] The memory also includes one or more programs, which are stored in the memory and configured to be executed by one or more processors. The one or more programs include a method for implementing the generation of the intelligent question-answering model as described above.

[0160] In an exemplary embodiment, a computer-readable storage medium is also provided, wherein at least one instruction, at least one program, code set or instruction set is stored in the storage medium, and the at least one instruction, at least one program, code set or instruction set implements the method for generating the intelligent question-answering model as described above when executed by the processor of the terminal. Optionally, the above-mentioned computer-readable storage medium can be ROM (Read-Only Memory), RAM (Random Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk and optical data storage device, etc.

[0161] In an exemplary embodiment, a computer program product is also provided. When the computer program product is executed, it is used to implement the method for generating the intelligent question and answer model as described above.

[0162] It should be understood that the "plurality" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0163] In addition, the step numbers described in this document only illustrate a possible execution order between the steps. In some other embodiments, the above steps may not be executed in the order of the numbers, such as two steps with different numbers are executed at the same time, or two steps with different numbers are executed in the opposite order to that shown in the figure. The embodiments of the present application are not limited to this.

[0164] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the device embodiment drawings provided in the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0165] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a terminal device (which can be a personal computer, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0166] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, which can be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integration. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)), etc.

[0167] It should be noted that the above embodiments can be freely combined as needed. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered as the protection scope of the present invention.

Claims

1. A method for generating an intelligent question-answering model, characterized in that: The method comprises: Expanding the original sample data set so that there are positive samples corresponding to the questions in the original sample data set, generating a new sample data set, wherein the positive samples are defined as questions with the same answers as the original sample data set, and questions with different answers from the original sample data set are defined as negative samples; The generated new sample dataset is input into the existing question answering model, which obtains the feature encoding related to the question from the knowledge base and documents in the new sample dataset, and calculates the loss L of the existing question answering model. qa ; The obtained feature encoding is added to the improved contrastive learning model, and the similarity between the true sample and the difficult positive and negative samples in the new sample dataset is calculated respectively to obtain the contrast loss L cl ; Linear fusion is used to transform the loss L of the existing question-answering model qa and contrast loss L cl The final question-answering model is trained and the feature encoding is added to the improved contrastive learning model to calculate the similarity between the true sample and the difficult positive sample and negative sample in the new sample data set, and the contrast loss L is obtained. cl , linear fusion is used to transform the loss L of the existing question-answering model qa and contrast loss L cl Combined, the loss L of the final question-answering model includes: Define a negative sample queue Z neg ={z1, z2, ..., z K }, where z i represents the i-th negative sample in the sample data set, and K represents the number of negative sample queues; Calculate the cosine similarity between the true sample and the positive sample, and finally extract the positive sample with the smallest similarity as the difficult positive sample vector k hard : Linearly fuse the positive sample and the true sample to form a new difficult negative sample vector Then update the negative sample queue Z neg , the specific formula is as follows: Among them, λ represents the allocation weight; Z neg According to the similarity between negative samples and true samples, we sort them from large to small and get ||query||z i ||>||queny||||zj||, (i<j), extract the top-K negative samples with the largest similarity to form satisfy: where λ neg is a pre-set hyperparameter; according to the obtained difficult negative sample queue vector and difficult positive sample vector k hard , and obtain the improved contrast loss function L cl , Among them, τ is a constant; Thus, the final model loss function L is obtained. L=λ qa L qa +λ cl L cl λ qa and λ cl Represent the loss weights of the existing question-answering model and the comparison model respectively.

2. The method for generating an intelligent question-answering model according to claim 1, characterized in that: The expanding the original sample data set so that there are positive samples corresponding to the problems in the original sample data set and generating a new sample data set specifically includes: According to the questions in the original sample data set and the knowledge base triple information, positive samples corresponding to the questions in the original sample data set are generated, where questions with the same answers as those in the original sample data set are defined as positive samples, and questions with different answers are defined as negative samples; For each data sample, three positive samples are generated to obtain a new sample data set, where the original sample data set is the public question-answering data set WebQuestionSP.

3. The method for generating an intelligent question-answering model according to claim 2, characterized in that: The existing question-answering model obtains feature encoding related to the question from the knowledge base and documents in the new sample dataset, specifically including: The existing question-answering model encodes the question, knowledge base triple information and document information respectively, and obtains their semantic expressions q, KB i and doc j , among which, KB i Represents the semantic expression of the i-th subgraph in the knowledge base, i = 1, 2, .., l kb , here l kb Represents the total number of subgraphs in the knowledge base; doc j Represents the semantic expression of the jth subgraph in the document, j = 1, 2, .., l doc , l doc Indicates the total number of documents; Use LSTM attention encoder to encode the question and obtain the question feature vector Obtain entity feature vectors of the knowledge base and entity feature vectors of the document; Obtain the entity feature vector ent related to the question according to the entity feature vector of the knowledge base and the entity feature vector of the document; The problem feature vector The entity feature vector ent is used as the feature encoding.

4. The method for generating an intelligent question-answering model according to claim 3, characterized in that: The method further comprises: Label the original sample data set as true sample x query , calculate the true sample x query The feature vector query of The three expanded positive sample data are marked as Calculate the feature vector (k1, k2, k3) of the positive sample, where i k ←mθ k +(1-m)θ q ,m∈[0,1) where f q =encoder,f k =moment encoder are both existing question-answering models and momentum update encoders.

5. A device for generating an intelligent question-answering model based on the method for generating an intelligent question-answering model according to claim 1, characterized in that: The device comprises: A sample set generation module is used to expand the original sample data set so that there are positive samples corresponding to the questions in the original sample data set, and generate a new sample data set, wherein the positive samples are defined as questions with the same answers as the original sample data set, and questions with different answers from the original sample data set are defined as negative samples; The encoding module is used to input the generated new sample data set into the existing question-answering model, and the existing question-answering model obtains the feature encoding related to the question from the knowledge base and documents in the new sample data set, and calculates the loss L of the existing question-answering model. qa ; The comparison module is used to add the obtained feature encoding to the contrastive learning model, calculate the similarity between the true sample and the positive and negative samples in the new sample data set, and obtain the contrast loss L cl ; The training module is used to use linear fusion to transform the loss L of the existing question-answering model qa and contrast loss L cl The loss L of the final question-answering model is obtained by combining them, and the final question-answering model is obtained by training.

6. The device for generating an intelligent question-answering model according to claim 5, characterized in that: The encoding module specifically includes: The first encoding unit is used in the existing question-answering model to encode the question, knowledge base triple information and document information respectively, and obtain their semantic expressions q, KB respectively. i and doc j , among which, KB i Represents the semantic expression of the i-th subgraph in the knowledge base, i = 1, 2, .., l kb , here l kb Represents the total number of subgraphs in the knowledge base; doc j Represents the semantic expression of the jth subgraph in the document, j = 1, 2, .., l doc , l doc Indicates the total number of documents; The second encoding unit is used to encode the question using the LSTM attention encoder to obtain the question feature vector A first vector acquisition unit, used to obtain an entity feature vector of a knowledge base and an entity feature vector of a document; The second vector acquisition unit is used to obtain an entity feature vector ent related to the question according to the entity feature vector of the knowledge base and the entity feature vector of the document, and convert the question feature vector The entity feature vector ent is used as the feature encoding.

7. A computing device, characterized in that The computing device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method for generating an intelligent question and answer model as described in any one of claims 1 to 4.

8. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method for generating an intelligent question and answer model as described in any one of claims 1 to 4.

Images