Question and answer matching model training method and device, electronic equipment, and storage medium

Abstract

Embodiments of the present disclosure relate to a training method and device of a question and answer matching model, an electronic device and a storage medium. In the embodiments of the present disclosure, training data is dynamically constructed based on a knowledge base, and a historical question and answer matching model is combined to realize continuous iteration and update of the model. For example, similar questions in the knowledge base are used as problems to retrieve a knowledge database, a plurality of candidate similar questions are co-recalled, and the problem and the plurality of candidate similar questions are constructed as a plurality of similar question pairs. Then, the historical question and answer matching model is used to finely sort the plurality of similar question pairs, and part of the similar question pairs are selected for labeling to obtain incremental training samples, so that the historical question and answer matching model is trained by using the incremental training samples. This training process does not depend on manual labeling, and automatic labeling can be realized by combining the fine sorting result and the knowledge base. On the one hand, the artificial cost can be reduced; on the other hand, the knowledge base can be fully covered to avoid partial bias effect, and the model update efficiency can be effectively improved by focusing on confusing knowledge points.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the technical field of computer, in particular to a training method and device of a question and answer matching model, an electronic device and a non-transitory computer-readable storage medium. BACKGROUND

[0002] With the development of artificial intelligence, intelligent dialogue systems are applied more and more widely. The core of the intelligent dialogue system is a question and answer matching model. The intelligent dialogue system can match user questions and knowledge points in a knowledge base by using the question and answer matching model, and then return answers corresponding to the knowledge points most similar to the user questions to the user. With the continuous change of the knowledge base, how to dynamically update the question and answer matching model so that the model can continuously learn to adapt to changing application scenarios is the key to improving user experience.

[0003] The update of the question and answer matching model needs a large amount of training data support. If only relying on manual annotation of training data, on the one hand, the rhythm is slow and the labor cost is high, on the other hand, the manual annotation of training data does not have high coverage of the knowledge base, which may produce local bias effect and cannot effectively focus on confusing knowledge points, resulting in low model update efficiency.

[0004] Therefore, it is urgent to provide a training scheme of a question and answer matching model to realize dynamic update of the question and answer matching model. The above description of the problem discovery process is only used to assist understanding of the technical solution of the present disclosure and does not mean that the above content is prior art. SUMMARY

[0005] In order to solve at least one problem existing in the prior art, at least one embodiment of the present disclosure provides a training method, device, electronic device and non-transitory computer-readable storage medium of a question and answer matching model.

[0006] In a first aspect, an embodiment of the present disclosure provides a training method of a question and answer matching model, the method comprising:

[0007] obtaining a plurality of second similar questions corresponding to a first similar question;

[0008] based on the first similar question and the plurality of second similar questions, constructing a plurality of similar question pairs; each similar question pair includes the first similar question and one second similar question;

[0009] selecting one or more similar question pairs from the plurality of similar question pairs and labeling to obtain a first training sample;

[0010] training a first question and answer matching model based on the first training sample.

[0011] In a second aspect, an embodiment of the present disclosure also provides a training device of a question and answer matching model, the device comprising:

[0012] a retrieval unit, configured to obtain a plurality of second similar questions corresponding to the first similar question;

[0013] a training set determination unit, configured to construct a plurality of similar question pairs based on the first similar question and the plurality of second similar questions; each similar question pair comprises the first similar question and one second similar question; select one or more similar question pairs from the plurality of similar question pairs and label to obtain a first training sample;

[0014] a model training unit, configured to train the first question and answer matching model based on the first training sample.

[0015] In a third aspect, the embodiments of the present disclosure further provide an electronic device, comprising: a processor and a memory; the processor is configured to execute the steps of the training method of the question and answer matching model according to the program or instruction stored in the memory.

[0016] In a fourth aspect, the embodiments of the present disclosure further provide a non-transitory computer readable storage medium for storing programs or instructions, which make the computer execute the steps of the training method of the question and answer matching model according to the first aspect.

[0017] It can be seen that, in at least one embodiment of the present disclosure, the training data is dynamically constructed based on the knowledge base, and the historical question and answer matching model is combined to realize the continuous iteration and update of the question and answer matching model. In some embodiments, the similar questions in the knowledge base are retrieved as questions to retrieve the knowledge database, a plurality of candidate similar questions can be recalled, and the question and the plurality of candidate similar questions are constructed into a plurality of similar question pairs; then the historical question and answer matching model can be used to sort the plurality of similar question pairs, and part of the similar question pairs are selected for labeling to obtain an incremental training sample, so that the historical question and answer matching model is trained by using the incremental training sample. This training process does not depend on manual labeling, and automatic labeling can be realized by combining the sorting result and the knowledge base. Compared with manual labeling, on the one hand, the rhythm of model updating can be improved and the labor cost can be reduced; on the other hand, the knowledge base can be fully covered to avoid partial bias effect, and the model updating efficiency can be effectively improved by focusing on the knowledge points that are easy to confuse. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings needed in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0019] Figure 1 is an exemplary application scenario provided by the embodiments of the present disclosure;

[0020] Figure 2 is an exemplary block diagram of a question and answer device provided by an embodiment of the present disclosure;

[0021] Figure 3 is an exemplary block diagram of a model offline training module provided by an embodiment of the present disclosure;

[0022] Figure 4 is an exemplary block diagram of a training set determination unit provided by an embodiment of the present disclosure;

[0023] Figure 5 is an exemplary block diagram of an electronic device provided by an embodiment of the present disclosure;

[0024] Figure 6 is an exemplary flow chart of a training method of a question and answer matching model provided by an embodiment of the present disclosure;

[0025] Figure 7 is an exemplary flow chart of another training method of a question and answer matching model provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] In order to more clearly understand the above objectives, features and advantages of the present disclosure, further specific details of the present disclosure will be given below with reference to the drawings and embodiments. It can be understood that the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. The specific embodiments described herein are only used to explain the present disclosure, and not to limit the present disclosure. All other embodiments obtained by those of ordinary skill in the art based on the described embodiments of the present disclosure are within the scope of protection of the present disclosure.

[0027] It should be noted that, in this document, relational terms such as "first" and "second", and the like, are used solely to distinguish one entity or action from another entity or action, without necessarily requiring or implying any such actual relationship or order between such entities or actions.

[0028] In order to facilitate the understanding of the scheme of the embodiments of the present disclosure, the terms related to the embodiments of the present disclosure are explained as follows:

[0029] Question (query): a question requested by a user to an intelligent dialogue system.

[0030] Knowledge point: a standard question of a question.

[0031] Similar question: a similar question corresponding to the standard question of a question.

[0032] Knowledge base: a data set storing knowledge points, similar questions and answers.

[0033] Question and answer matching model: used to identify the matching degree of questions and similar questions.

[0034] Training set: a set of training data used to train the question and answer matching model.

[0035] Topn: top n candidate elements in the ranking, n being a preset constant.

[0036] In some embodiments, the training scheme of the question and answer matching model is "manual annotation + data mixing". Specifically, the data to be annotated is constructed based on logs or knowledge bases, and the training set is obtained by manual annotation. The newly annotated training data and the historically annotated training data are mixed together, and a new model is trained. This scheme requires a large amount of manual annotation cost, resulting in low efficiency of model iteration and update.

[0037] In some embodiments, the training scheme of the question and answer matching model is "knowledge base random sampling + full training". Specifically, similar questions under the same knowledge point in the knowledge base are sampled as positive samples, and similar questions under different knowledge points in the knowledge base are sampled as negative samples, and then all the data sampled are used for model training. The premise of this scheme is that the similar questions under the same knowledge point have the same intent, and the similar questions under different knowledge points have different intents, i.e. based on the knowledge base distribution assumption. However, in actual application scenarios, the intents of similar questions under the same knowledge point may not be consistent, so the model training is biased based on the knowledge base distribution assumption, and cannot be effectively combined with the historical question and answer matching model.

[0038] In some embodiments, the training scheme of the question and answer matching model is "search topn random sampling + full training". Based on the topn random sampling of search recall, positive and negative samples are constructed, and the full amount of data sampled is used for model training. This scheme can certainly alleviate the model learning bias problem based on random sampling of knowledge points to a certain extent, but cannot completely solve it.

[0039] The method and apparatus for training a question and answer matching model, electronic device and non-transitory computer readable storage medium provided by the embodiments of the present disclosure can dynamically construct training data based on a knowledge base and combine a historical question and answer matching model to realize continuous iteration and update of the question and answer matching model. In some embodiments, similar questions in the knowledge base are retrieved as problems to retrieve a knowledge database, a plurality of candidate similar questions are co-recalled, and the problem and the plurality of candidate similar questions are constructed as a plurality of similar question pairs. Then, the plurality of similar question pairs are precisely sorted by the historical question and answer matching model, and part of the similar question pairs are selected for labeling to obtain incremental training samples, so that the historical question and answer matching model is trained by using the incremental training samples. This training process does not depend on manual labeling, and automatic labeling can be realized by combining the precise sorting result and the knowledge base. Compared with manual labeling, the embodiments of the present disclosure can improve the rhythm of model updating and reduce the labor cost on the one hand, and can comprehensively cover the knowledge base to avoid partial bias effect and effectively focus on the knowledge points that are easy to confuse to improve the model updating efficiency on the other hand.

[0040] The embodiments of the present disclosure can be applied to any electronic device with an intelligent dialogue system, such as various dialogue robots, for example, mobile devices such as smartphones, tablet computers, and smart sports equipment, and for example, fixed devices such as desktop computers, smart home appliances, and servers. In some embodiments, the embodiments of the present disclosure can be applied to any application scenario that requires dialogue, for example, some applications in life, such as smart speakers of various brands, and some applications in work, such as various efficient customer service robots. It should be understood that the application scenarios of the embodiments of the present disclosure are only some examples or embodiments of the present disclosure, and the present disclosure can also be applied to other similar scenarios without creative labor.

[0041] Figure 1 An example application scenario diagram is provided for the embodiments of the present disclosure. As shown in Figure 1 The application scenario includes a user device 10, a user device 11, a user 12, a user 13, a question and answer device 14, and a knowledge database 15. In some embodiments, some users can establish a connection with the question and answer device 14 through the user device (for example, the user device 10 and the user device 11), and then ask the question and answer device 14 to obtain the answer fed back by the question and answer device 14. Some users (for example, the user 12 and the user 13) can directly operate the question and answer device 14 to obtain the answer to the question. In some embodiments, the user's query can be a consultation type question, such as “Do you know it will rain today?”, or other types of questions, such as casual conversation: “Hello!”. In the present embodiment, the question is a question with an answer.

[0042] The question-answering device 14 has an intelligent dialogue system, which includes a question-answering matching model. This model allows the intelligent dialogue system to match the user's question with the closest relevant knowledge points or similar questions, and then retrieve the corresponding answer from the knowledge base and provide it to the user. The intelligent dialogue system can be a software system, a hardware system, or a combination of both. The question-answering device 14 can provide the hardware environment required for the intelligent dialogue system to run. For example, the intelligent dialogue system is a software system running on an operating system, and the hardware system of the question-answering device 14 is a hardware environment that supports the operating system. For ease of description, the question-answering device 14 is used as the execution entity instead of the intelligent dialogue system.

[0043] The knowledge database 15 is used to store knowledge bases and other data. In some embodiments, the knowledge database 15 can provide an external retrieval interface to receive question retrieval requests and provide feedback on similar questions. In some embodiments, the knowledge database 15 can be implemented as a server with storage and retrieval functions, such as a server with an ElasticSearch component. In some embodiments, the server can be a local server or a cloud server. The server can be a single server or a server cluster. The server cluster can be centralized or distributed.

[0044] In some embodiments, after obtaining a question (query), the question-answering device 14 sends a question retrieval request to the knowledge database 15. Upon receiving the question retrieval request, the knowledge database 15 searches its knowledge base and returns similar questions found. The question-answering device 14 selects the most similar question and then sends an answer search request to the knowledge database 15, the request carrying the most similar question. Upon receiving the answer search request, the knowledge database 15 searches its knowledge base and returns the answer to the most similar question to the question-answering device 14. The question-answering device 14 then returns the received answer to the user.

[0045] Figure 2 This is an exemplary block diagram of a question-answering device 20 provided in an embodiment of the present disclosure. In some embodiments, the question-answering device 20 can be implemented as... Figure 1 The question-and-answer device 14 or a part of the question-and-answer device 14 is used to provide a question-and-answer matching service.

[0046] like Figure 2 As shown, the question-answering device 20 can be divided into multiple modules, such as: online model service module 21, offline model training module 22, and other modules that provide question-answering matching services.

[0047] The model online service module 21 is configured to deploy the question and answer matching model trained by the model offline training module 22 online, so that the question and answer matching model provides online question and answer matching service. The model online service module 21 can use the question and answer matching model to match the most similar knowledge points or similar questions of the user's question, and then retrieve the knowledge base to obtain the corresponding answers and feed back to the user.

[0048] The model offline training module 22 is configured to dynamically construct training data based on the knowledge base and combine the historical question and answer matching model to realize continuous iteration and update of the question and answer matching model. In some embodiments, the historical question and answer matching model is the question and answer matching model deployed online by the model online service module 21. In some embodiments, the historical question and answer matching model is a FAQ (Frequently Asked Questions) model. In some embodiments, the model offline training module 22 determines that the knowledge base has changed, and then trains the historical question and answer matching model based on the dynamically constructed training data.

[0049] In some embodiments, the function of the model offline training module 22 can be integrated into the model online service module 21, or can be configured as a module independent of the question and answer device 20. The model offline training module 22 can be a software module, a hardware module, or a combination of software and hardware.

[0050] Figure 3 An exemplary block diagram of a model offline training module 30 is provided for the embodiments of the present disclosure. In some embodiments, the model offline training module 30 can be implemented as the model offline training module 22 or a part of the model offline training module 22 in the question and answer device 20. Figure 2

[0051] As shown in Figure 3 , the model offline training module 30 can include but is not limited to the following units: a retrieval unit 31, a training set determination unit 32, and a model training unit 33. In some embodiments, the model offline training module 30 determines the corresponding relationship between the knowledge points and the similar questions in the knowledge base in advance, wherein the knowledge base can include a plurality of knowledge points, and each knowledge point can correspond to a plurality of similar questions.

[0052] ​The retrieval unit 31 is configured to obtain a plurality of second similar questions corresponding to the first similar question. In some embodiments, the retrieval unit 31 retrieves the knowledge database based on the first similar question in the knowledge base to obtain the plurality of second similar questions. In this embodiment, the retrieval unit 31 retrieves the knowledge database based on the similar question (i.e., the first similar question) in the knowledge base as a question, and coarsely recalls a plurality of candidate similar questions (i.e., the second similar questions). In some embodiments, the retrieval unit 31 traverses the first similar question in the knowledge base as a question, and retrieves the knowledge database to obtain a plurality of second similar questions corresponding to each first similar question fed back by the knowledge database. In this embodiment, the retrieval unit 31 traverses the similar question in the knowledge base, retrieves the knowledge database based on each similar question as a question, and coarsely recalls a plurality of candidate similar questions corresponding to each similar question, so as to achieve comprehensive coverage of the knowledge base and avoid local bias effect.

[0053] The training set determination unit 32 is configured to determine an incremental training set based on the retrieval result of the retrieval unit 31. Since the incremental training set is a training set constructed based on the retrieval result of all similar questions in the knowledge base, the incremental training set can make the trained question and answer matching model pay attention to the knowledge points prone to confusion, and solve the problem of knowledge points prone to confusion.

[0054] In some embodiments, the training set determination unit 32 is further configured to sample the historical training set to obtain a sampled training set. The historical training set can be understood as a training set used in the training phase of the deployed online question and answer matching model. The sampled training set can prevent the trained question and answer matching model from forgetting knowledge.

[0055] In some embodiments, the training set determination unit 32 combines the incremental training set and the sampled training set into a complete training set. In some embodiments, the training set determination unit 32 replaces the historical training set with the complete training set for use in the next model training. Since the complete training set not only includes part of the training samples in the historical training set, but also includes the incremental training set constructed based on the retrieval result of all similar questions in the knowledge base, the data amount of the complete training set is much larger than the amount of the existing artificially annotated training data, so that the training overfitting problem can be avoided. In addition, the complete training set can make the trained question and answer matching model pay attention to the head error, and improve the model effect (e.g., improve the matching accuracy), where the head error can be understood as the knowledge points or similar questions that are often matched incorrectly by the historical question and answer matching model.

[0056] The model training unit 33 is configured to train the historical question and answer matching model based on the incremental training set. In some embodiments, the model training unit 33 is further configured to train the historical question and answer matching model based on the sampling training set. After the model training unit 33 trains the historical question and answer matching model (i.e., the first question and answer matching model) using the complete training set, a new question and answer matching model (i.e., the second question and answer matching model) that can be deployed online is obtained. The model training unit 33 can replace the first question and answer matching model with the second question and answer matching model, so that the second question and answer matching model provides the question and answer matching service. In this embodiment, the knowledge base and the historical question and answer matching model are combined, without making assumptions about the distribution of the knowledge base, avoiding model training bias caused by assumptions about the distribution of the knowledge base, and effectively focusing on the head errors of the historical question and answer matching model to continuously boost and update the model.

[0057] In some embodiments, the division of the units in the model offline training module 30 is only a logical functional division, and actual implementation can have another division manner. For example, at least two of the retrieval unit 31, the training set determination unit 32, and the model training unit 33 can be implemented as one unit; the retrieval unit 31, the training set determination unit 32, or the model training unit 33 can also be divided into multiple sub-units. It can be understood that each unit or sub-unit can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether the functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application.

[0058] Figure 4 An exemplary block diagram of a training set determination unit 40 provided in an embodiment of the present disclosure is shown. In some embodiments, the training set determination unit 40 can be implemented as the training set determination unit 32 or part of the training set determination unit 32 in the model offline training module 30. Figure 3

[0059] As shown in FIG. 4, the training set determination unit 40 can include, but is not limited to, the following units: an incremental training set labeling sub-unit 41, a historical training set sampling sub-unit 42, and a complete training set determination sub-unit 43. Figure 4 The incremental training set labeling sub-unit 41 can construct a plurality of similar question pairs. For example, based on a first similar question in the knowledge base and a plurality of second similar questions (i.e., candidate similar questions) corresponding to the first similar question, a plurality of similar question pairs are constructed; each similar question pair includes the first similar question and one second similar question.

[0060]

[0061] ​​In some embodiments, the incremental training set labeling subunit 41 can select one or more similar question pairs from the plurality of similar question pairs. For example, a similar question pair is selected from the plurality of similar question pairs based on a first question and answer matching model (i.e., a historical question and answer matching model). In some embodiments, the incremental training set labeling subunit 41 obtains scores corresponding to the plurality of similar question pairs based on the first question and answer matching model; and then sorts the plurality of similar question pairs in descending order of the scores; and selects similar question pairs based on the sorting, for example, selects topn similar question pairs.

[0062] In some embodiments, the incremental training set labeling subunit 41 can label the selected similar question pairs to obtain first training samples (i.e., incremental training samples). For example, the selected similar question pairs are labeled based on a correspondence between knowledge points in a knowledge base and similar questions. In some embodiments, the incremental training set labeling subunit 41 determines a second knowledge point corresponding to a second similar question pair in the selected similar question pairs based on the correspondence; and a first knowledge point corresponds to a first similar question pair in the selected similar question pairs. The incremental training set labeling subunit 41 compares the first knowledge point and the second knowledge point, if the second knowledge point and the first knowledge point are consistent, the selected similar question pairs are labeled as positive labels to obtain first training samples with positive labels; if the second knowledge point and the first knowledge point are inconsistent, the selected similar question pairs are labeled as negative labels to obtain first training samples with negative labels. The first training samples with positive labels and the first training samples with negative labels constitute an incremental training set.

[0063] The historical training set sampling subunit 42 is configured to sample a historical training sample set (referred to as a historical training set) to obtain second training samples, and a plurality of second training samples constitute a sampling training set. In some embodiments, the historical training set sampling subunit 42 can randomly sample the historical training set, or can use other sampling methods.

[0064] The complete training set determination subunit 43 combines the incremental training set and the sampling training set as a complete training set.

[0065] In some embodiments, the division of units in the training set determination unit 40 is only a logical functional division. In actual implementation, there may be other division methods. For example, at least two units from the incremental training set labeling subunit 41, the historical training set sampling subunit 42, and the complete training set determination subunit 43 can be implemented as one unit; the incremental training set labeling subunit 41, the historical training set sampling subunit 42, or the complete training set determination subunit 43 can also be divided into multiple subunits. It is understood that each unit or subunit can be implemented using electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application.

[0066] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. In some embodiments, the electronic device can be implemented as follows: Figure 1 The question-and-answer device 14 or a part of the question-and-answer device 14.

[0067] like Figure 5 As shown, the electronic device includes at least one processor 51, at least one memory 52, and at least one communication interface 53. The various components of the electronic device are coupled together via a bus system 54. The communication interface 53 is used for information transmission with external devices. It can be understood that the bus system 54 is used to implement communication between these components. In addition to a data bus, the bus system 54 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 5 The general labeled all buses as Bus System 54.

[0068] It is understood that the memory 52 in this embodiment may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory.

[0069] In some implementations, memory 52 stores elements such as executable units or data structures, or subsets thereof, or extended sets thereof: operating systems and applications.

[0070] The operating system includes various system programs, such as the framework layer, core library layer, and driver layer, used to implement various basic tasks and handle hardware-based tasks. The application programs include various applications, such as media players and browsers, used to implement various application tasks. The program implementing the training method of the question-answering matching model provided in this disclosure can be included in the application programs.

[0071] In this embodiment of the disclosure, the processor 51 executes the steps of various embodiments of the question-answering matching model training method provided in this embodiment by calling the program or instructions stored in the memory 52, specifically, the program or instructions stored in the application program.

[0072] The training method for the question-answering matching model provided in this embodiment can be applied to or implemented by the processor 51. The processor 51 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the hardware of the processor 51 or by instructions in software form. The processor 51 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor.

[0073] The steps of the training method for the question-answering matching model provided in this embodiment can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software units in the decoding processor. The software units can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 52, and processor 51 reads information from memory 52 and combines it with hardware to complete the steps of the method.

[0074] Figure 6 This is an exemplary flowchart illustrating a training method for a question-answering matching model provided in this disclosure. The method is executed by an electronic device. For ease of description, the following embodiments use an electronic device as the execution entity to illustrate the flow of the question-answering matching model training method. In some embodiments, the electronic device pre-determines the correspondence between knowledge points and similar questions in a knowledge base, which is stored in a knowledge database.

[0075] like Figure 6 As shown, in step 601, the electronic device retrieves multiple second similar questions from the knowledge base based on the first similar questions in the knowledge base. In some embodiments, the electronic device traverses the first similar questions in the knowledge base as questions and retrieves the knowledge database to obtain multiple second similar questions corresponding to each first similar question returned by the knowledge database.

[0076] In step 602, the electronic device constructs a plurality of question pairs based on the first similar question and the plurality of second similar questions (i.e., candidate similar questions); each question pair includes the first similar question and one second similar question.

[0077] In step 603, the electronic device selects one or more question pairs from the plurality of question pairs based on a first question-answer matching model (i.e., a historical question-answer matching model). In some embodiments, the electronic device obtains scores corresponding to the plurality of question pairs respectively based on the first question-answer matching model; then sorts the plurality of question pairs in descending order of the scores; and selects question pairs based on the sorting, for example, selects top n question pairs, where n can be 10.

[0078] In step 604, the electronic device labels the selected question pairs based on the corresponding relationship to obtain first training samples (i.e., incremental training samples). In some embodiments, the electronic device determines a second knowledge point corresponding to a second question pair in the selected question pairs based on the corresponding relationship between knowledge points in the knowledge base and similar questions; wherein the first similar question in the selected question pair corresponds to a first knowledge point. The electronic device compares the second knowledge point and the first knowledge point, if the second knowledge point and the first knowledge point are consistent, the selected question pair is labeled as a positive label to obtain a first training sample with a positive label; if the second knowledge point and the first knowledge point are inconsistent, the selected question pair is labeled as a negative label to obtain a first training sample with a negative label.

[0079] In step 605, the electronic device trains the first question-answer matching model based on the first training samples. In some embodiments, the electronic device obtains second training samples based on sampling the historical training sample set, and trains the first question-answer matching model based on the second training samples. In some embodiments, after the electronic device trains the first question-answer matching model based on the first training samples and the second training samples, a second question-answer matching model is obtained; and the second question-answer matching model is used to replace the first question-answer matching model, so that the second question-answer matching model provides question-answer matching services. In some embodiments, the electronic device replaces the historical training sample set with a training sample set (i.e., a complete training set) composed of the first training samples and the second training samples for use in the next training.

[0080] Figure 7 is an exemplary flowchart of a training method of a question-answer matching model provided by the embodiments of the present disclosure, the execution subject of the training method is an electronic device, and the execution subject is omitted for ease of description. The training method includes the following 13 steps:

[0081] 1. Retrieving the knowledge database by taking the similar questions in the knowledge base as the questions (query).

[0082] 2. Coarse recall of multiple candidate similar questions corresponding to the question based on the knowledge database.

[0083] 3. Construction of multiple similar question pairs (Pairs) based on the question and the multiple candidate similar questions, each Pair including the question and a candidate similar question, i.e., the number of Pairs is the same as the number of candidate similar questions.

[0084] 4. Input of the multiple Pairs into a historical question and answer matching model.

[0085] 5. Obtaining of scores of each Pair output by the historical question and answer matching model.

[0086] 6. Fine sorting of the multiple Pairs in descending order of the scores, selection of top-n similar question pairs, and n can be 10.

[0087] 7. Determination of a second knowledge point corresponding to a candidate similar question in each similar question pair in the top-n similar question pairs with reference to a correspondence between knowledge points and similar questions, wherein the question in the similar question pair corresponds to a first knowledge point.

[0088] 8. Comparison of the second knowledge point and the first knowledge point, if the second knowledge point and the first knowledge point are consistent, labeling the similar question pair with a positive label to obtain a positive label sample, and if the second knowledge point and the first knowledge point are inconsistent, labeling the similar question pair with a negative label to obtain a negative label sample. Wherein the positive label sample and the negative label sample constitute an incremental training set.

[0089] 9. Full sampling of the incremental training set, i.e., the incremental training set is used for model training in its entirety.

[0090] 10. Partial sampling of the historical training set to obtain a sampled training set.

[0091] 11. Combination of the incremental training set and the sampled training set into a complete training set, and replacement of the historical training set.

[0092] 12. Training of the historical question and answer matching model using the complete training set to obtain a new question and answer matching model.

[0093] 13. Replacement of the historical question and answer matching model with the new question and answer matching model to provide a question and answer matching service.

[0094] It should be noted that, for each of the foregoing method embodiments, in order to simply describe, each is expressed as a combination of a series of actions, but those skilled in the art can understand that the disclosed embodiments are not limited by the described action sequence, because according to the disclosed embodiments, certain steps can be performed in other sequences or simultaneously. In addition, those skilled in the art can understand that the embodiments described in the specification are all optional embodiments.

[0095] The embodiments of the present disclosure further provide a non-transitory computer readable storage medium storing programs or instructions, which cause a computer to perform the steps of the method for training a question and answer matching model. To avoid repetition, the steps will not be described here.

[0096] It should be noted that, in this document, the terms "comprising", "including", or any other variant thereof are intended to cover a non-exclusive inclusion, such that processes, methods, articles, or apparatuses that comprise a list of elements not only include those elements, but also include other elements not expressly listed, or also include elements inherent in such processes, methods, articles, or apparatuses. Without more limitations, the elements defined by the statement "comprise" do not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0097] Those skilled in the art can understand that although some embodiments described herein include certain features included in other embodiments but not others, the combination of features of different embodiments means within the scope of the present disclosure and forms different embodiments.

[0098] Those skilled in the art can understand that the description of each embodiment is focused on, and the parts not described in detail in a certain embodiment can refer to the relevant description of other embodiments.

[0099] Although the embodiments of the present disclosure are described in conjunction with the drawings, various modifications and changes can be made by those skilled in the art without departing from the spirit and scope of the present disclosure, and such modifications and changes fall within the scope defined by the appended claims.

Claims

1. A training method for a question-answering matching model, characterized in that, The method includes: Obtain multiple second similar questions corresponding to the first similar question; Based on the first similar question and the plurality of second similar questions, a plurality of similar question pairs are constructed; each similar question pair includes the first similar question and a second similar question. One or more similar question pairs are selected from the plurality of similar question pairs and labeled to obtain a first training sample. The first training sample is obtained by identifying the knowledge points corresponding to the similar questions in the similar question pairs through a knowledge base and labeling the similar question pairs. The knowledge base includes a plurality of knowledge points and the knowledge points correspond to a plurality of similar questions. A first question-answering matching model is trained based on the first training sample and the second training sample, wherein the second training sample is obtained by randomly sampling historical training samples.

2. The method according to claim 1, characterized in that, The annotation process, which yields the first training sample, includes: Based on the correspondence between knowledge points and similar questions in the knowledge base, the second knowledge point corresponding to the second similar question in the selected similar question pair is determined; wherein, the first similar question in the selected similar question pair corresponds to the first knowledge point; If the second knowledge point is consistent with the first knowledge point, then the selected similar question pair is labeled with a positive label to obtain the first training sample with a positive label; If the second knowledge point is inconsistent with the first knowledge point, the selected similar question pair is labeled with a negative label to obtain the first training sample with a negative label.

3. The method according to claim 1, characterized in that, The method further includes: The first question-answering matching model is trained based on the second training sample.

4. The method according to claim 3, characterized in that, The method further includes: After training the first question-answering matching model based on the first training sample and the second training sample, the second question-answering matching model is obtained; Replace the first question-and-answer matching model with the second question-and-answer matching model so that the second question-and-answer matching model provides question-and-answer matching services.

5. The method according to claim 4, characterized in that, The method further includes: Replace the historical training sample set with the training sample set consisting of the first training sample and the second training sample.

6. The method according to claim 1, characterized in that, The step of selecting one or more similar question pairs from the plurality of similar question pairs includes: Based on the first question-answering matching model, the scores corresponding to the multiple similar question pairs are obtained respectively; Sort the multiple similar question pairs according to their scores from highest to lowest; Based on the ranking, select one or more similar question pairs.

7. The method according to claim 1, characterized in that, The first similar question and the plurality of second similar questions all belong to a knowledge base, which stores similar questions and their corresponding knowledge points.

8. The method according to claim 7, characterized in that, The process of obtaining multiple second similar questions corresponding to the first similar question includes: Each first similar question in the knowledge base is taken as a question and retrieved from the knowledge database to obtain multiple second similar questions corresponding to each first similar question returned by the knowledge database; wherein, the knowledge base is stored in the knowledge database.

9. A training device for a question-answering matching model, characterized in that, The device includes: A retrieval unit is used to obtain multiple second similar questions corresponding to the first similar question; The training set determination unit is used to construct multiple similar question pairs based on the first similar question and the multiple second similar questions; each similar question pair includes the first similar question and a second similar question; select one or more similar question pairs from the multiple similar question pairs and label them to obtain a first training sample, wherein the first training sample is obtained by determining the knowledge points corresponding to the similar questions in the similar question pairs through a knowledge base and labeling the similar question pairs; the knowledge base includes multiple knowledge points and the knowledge points correspond to multiple similar questions. The model training unit is used to train a first question-answering matching model based on the first training sample and the second training sample, wherein the second training sample is obtained by randomly sampling historical training samples.

10. An electronic device, characterized in that, include: Processor and memory; The processor executes the steps of the method as described in any one of claims 1 to 8 by invoking programs or instructions stored in the memory.

11. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores a program or instructions that cause a computer to perform the steps of the method as described in any one of claims 1 to 8.

Images