Question and answer pair evaluation data generation method and device, computer device, and storage medium

Abstract

The application relates to the fields of artificial intelligence and medical health technology, and discloses a kind of question and answer pair evaluation data generation method, device, computer equipment and storage medium, the method is based on the framework of adversarial learning, two initial question and answer generation models and initial question and answer discrimination models are used to game with each other, so as to improve the quality of question and answer pair data generation, by introducing meta-learning mechanism, using a small amount of initial labeled initial evaluation question and answer pair data as metadata, the initial question and answer generation model and the initial question and answer discrimination model are quickly adapted, the question and answer pair data generation efficiency is improved, large-scale artificial labeling and compilation are avoided, the generation mode of the provided GPT type large model evaluation data set is lower in cost, higher in efficiency, can cover multiple fields and multiple tasks, is more objective, accurate, comprehensive, has large scale, diversity and high difficulty, high quality effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical fields of artificial intelligence, natural language processing and medical health, and particularly relates to a method and device for generating evaluation data for question and answer pairs, a computer device and a storage medium. BACKGROUND

[0002] A GPT (Generative Pre-trained Transformer, GPT) large model for vertical domain document understanding refers to a model that uses the technical framework of a generative pre-trained transformer to pre-train and fine-tune on document data of a specific domain, such as medical health domain documents, so as to achieve in-depth understanding and diversified application of document content. The GPT large model has strong natural language generation capability and can generate different forms and purposes of output, such as summaries, questions and answers, reasoning, classification, etc., according to the input document content. Evaluating the performance and effect of the GPT large model is an important research topic, and appropriate evaluation indicators and data sets need to be designed to objectively and comprehensively test and compare the model. Generally, the evaluation of the GPT large model for vertical domain document understanding needs to consider domain knowledge and context information, so a special evaluation data set for a specific domain needs to be constructed, including document content, questions and answers, etc. At present, the construction of the evaluation data set for the GPT large model for vertical domain document understanding mainly relies on manual annotation and editing.

[0003] The inventors realize that the above-mentioned manual annotation and editing method for obtaining the GPT large model evaluation data set has the problems of high cost, low efficiency, difficulty in covering multiple domains and multiple tasks, inconsistent quality, possible subjective bias, errors or omissions, difficulty in ensuring the size, diversity and difficulty of the data set, and possible problems of too small, too simple or overfitting data set. SUMMARY

[0004] The present application provides a method and device for generating evaluation data for question and answer pairs, a computer device and a storage medium to solve the technical problems of high cost and low efficiency in the manual annotation and editing method for obtaining the GPT large model evaluation data set.

[0005] In a first aspect, a method for generating evaluation data of question and answer pairs is provided, including: obtaining vertical field documents, a generative adversarial network model, and initial evaluation question and answer pair data of a plurality of tasks, the initial evaluation question and answer pair data being obtained by initially labeling the vertical field documents, and the generative adversarial network model including an initial question and answer generation model and an initial question and answer discrimination model; performing meta-training on a preset meta-learning model through the initial evaluation question and answer pair data of each task to obtain meta-parameters, and updating the generative adversarial network model through the meta-parameters; inputting the vertical field documents into the updated initial question and answer generation model, so that the updated initial question and answer generation model outputs intermediate evaluation question and answer pair data based on a target task and a task weight of the target task; evaluating the intermediate evaluation question and answer pair data based on the vertical field documents through the updated initial question and answer discrimination model to obtain an evaluation probability value, the evaluation probability value representing a reasonable degree of the intermediate evaluation question and answer pair data; calculating a model loss function between the updated initial question and answer discrimination model and the updated initial question and answer generation model; iteratively training the updated generative adversarial network model through the model loss function until a preset condition is reached; and inputting the vertical field documents into the iteratively trained initial question and answer generation model to obtain final evaluation question and answer pair data as evaluation data of question and answer pairs of the vertical field documents.

[0006] In a second aspect, a question and answer pair evaluation data generation apparatus is provided, comprising: an acquisition module configured to acquire a vertical field document, a generative adversarial network model, and initial evaluation question and answer pair data of a plurality of tasks, the initial evaluation question and answer pair data being obtained by initial labeling of the vertical field document, and the generative adversarial network model comprising an initial question and answer generation model and an initial question and answer discrimination model; a meta-training module configured to perform meta-training on a preset meta-learning model by using the initial evaluation question and answer pair data of each task to obtain meta parameters, and update the generative adversarial network model by using the meta parameters; an intermediate evaluation question and answer pair data output module configured to input the vertical field document into the updated initial question and answer generation model, so that the updated initial question and answer generation model outputs intermediate evaluation question and answer pair data based on a target task and a task weight of the target task; an evaluation module configured to evaluate the intermediate evaluation question and answer pair data based on the vertical field document by using the updated initial question and answer discrimination model to obtain an evaluation probability value, the evaluation probability value representing a rationality of the intermediate evaluation question and answer pair data; a model loss function determination module configured to calculate a model loss function between the updated initial question and answer discrimination model and the updated initial question and answer generation model; a model training module configured to perform iterative training on the updated generative adversarial network model by using the model loss function until a preset condition is reached; and a question and answer pair evaluation data generation module configured to input the vertical field document into the initial question and answer generation model after the iterative training to obtain final evaluation question and answer pair data as question and answer pair evaluation data of the vertical field document.

[0007] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the question and answer pair evaluation data generation method when executing the computer program.

[0008] In a fourth aspect, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program, and the computer program implements the steps of the question and answer pair evaluation data generation method when executed by a processor.

[0009] In the aforementioned scheme implemented by the question-and-answer pair evaluation data generation method, apparatus, computer equipment, and storage medium, the method obtains initial evaluation question-and-answer pairs for multiple tasks after preliminary annotation of vertical category documents. Based on these initial evaluation question-and-answer pairs for multiple tasks, meta-training is performed on a preset meta-learning model using metadata to update the generative adversarial network model. Vertical domain documents are then input into the updated initial question-and-answer generation model to obtain intermediate evaluation question-and-answer pairs based on the weights of each task. The intermediate evaluation question-and-answer pairs are evaluated using an updated initial question-and-answer discrimination model. The model loss functions of the two updated models are then calculated, and the two models are iteratively trained until preset conditions are met. Finally, vertical domain documents are input into the iterative training model. The initial question-answering generation model, after training, obtains question-answering pair evaluation data for documents in this vertical domain. This method uses an adversarial learning framework to improve the quality of question-answering pair data generation by having two opposing initial question-answering generation models and an initial question-answering discrimination model compete with each other. By introducing a meta-learning mechanism, the initial evaluation question-answering pair data obtained from a small amount of initial annotation is used as metadata to quickly adapt the initial question-answering generation model and the initial question-answering discrimination model, thereby improving the efficiency of question-answering pair data generation. This avoids large-scale manual annotation and compilation, and provides a lower-cost, more efficient, multi-domain, multi-task, more objective, accurate, comprehensive, large-scale, diverse, and high-difficulty, high-quality GPT-like large model evaluation dataset generation method. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram of an application environment for a question-and-answer pair evaluation data generation method according to an embodiment of the present invention;

[0012] Figure 2 This is a flowchart illustrating a question-and-answer pair evaluation data generation method in one embodiment of the present invention;

[0013] Figure 3 This is a flowchart illustrating the method for determining the original parameters in one embodiment of the present invention;

[0014] Figure 4 This is a schematic diagram of a question-and-answer pair evaluation data generation device according to an embodiment of the present invention;

[0015] Figure 5 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0016] Figure 6 FIG. 6 is another structural schematic diagram of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0017] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0018] The embodiments of the present application provide a method for generating question and answer pair evaluation data, which can be applied to a terminal, a server, or software running in the terminal or the server. In some embodiments, the terminal can be an electronic device such as a smartphone, a tablet computer, a notebook computer, or a desktop computer; the server can be configured as a standalone physical server, a server cluster composed of multiple physical servers, or a distributed system, and can also be configured as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, and big data and artificial intelligence platforms; and the software can be an application program for generating question and answer pair evaluation data, but is not limited to the above forms.

[0019] For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to Figure 1 For example, referring to

[0020] The application will be described in detail below through specific embodiments. Please refer to Figure 2 as shown, Figure 2 A flowchart of the method for generating evaluation data of question and answer pairs according to an embodiment of the application is shown in FIG. 1. The method includes the following steps:

[0021] In step S210, initial evaluation question and answer pair data is obtained from vertical field documents and a generative adversarial network model.

[0022] The initial evaluation question and answer pair data is obtained by initial labeling of the vertical field documents. The initial labeling can be performed manually or by other methods known to those skilled in the art. The initial evaluation question and answer pair data can be a relatively small amount of data, which is much less than the amount of data labeled manually in related technologies. The initial evaluation question and answer pair data includes an initial question and an initial answer matching the initial question. The initial question can be directly labeled or obtained after manual editing.

[0023] The generative adversarial network model includes an initial question and answer generation model and an initial question and answer discrimination model. In this embodiment, the initial question and answer generation model and the initial question and answer discrimination model can be pre-built based on the GPT technical framework and pre-trained generative models. The initial question and answer generation model is used as a question and answer generator (Question Answer Generator) to generate a question Q and an answer A according to the vertical field documents. The initial question and answer discrimination model is used as a question and answer discriminator (Question Answer Discriminator) to determine whether the question Q and the answer A are reasonable according to the vertical field documents.

[0024] In an embodiment, the generative adversarial network model comprises two sub-network models, one of which is used as an initial question and answer generation model, and the other of which is used as an initial question and answer discrimination model; the sub-network model comprises two conversion modules, one of which is used as an encoder, and the other of which is used as a decoder, the encoder is used to convert the vertical field document, the input question or the input answer into a hidden state vector, and the decoder is used to generate an output question or an output answer according to the input hidden state vector; the conversion module comprises a plurality of conversion sub-modules, the conversion sub-module comprises a self-attention layer, a difference connection layer, a normalization layer and a feedforward neural network, and the self-attention layer comprises a scaling dot product attention and a multi-head attention. In this embodiment, a plurality of layer normalization operation modules are further included in the sub-network model, the layer normalization operation module is used for normalization processing on each position of the input sequence, part of the layer normalization operation modules are integrated in the feedforward neural network with residual connection and layer normalization, and through the normalization processing, the model convergence can be accelerated and the model stability can be improved. The self-attention layer, the difference connection layer and the feedforward neural network in the conversion sub-module are connected in sequence, and the normalization layer is used for one layer normalization operation on each layer.

[0025] For example, the initial question-answer generation model and the initial question-answer discrimination model are built based on the GPT technical framework, which is a Transformer-based autoregressive language model consisting of an encoder and a decoder: the encoder is responsible for converting the input vertical field document, question Q or answer A into a hidden state vector H, and the decoder is responsible for generating the output question Q' or answer A' according to the hidden state vector H. The encoder and decoder in this embodiment are both stacked with multiple self-attention layers and feed-forward neural network layers, followed by a residual connection and a layer normalization operation. In this embodiment, the self-attention layer is a mechanism that can capture the dependency between different positions in the input sequence, consisting of a scaled dot-product attention and a multi-head attention. The scaled dot-product attention is a method that calculates the correlation score between each position and other positions in the input sequence, and then performs weighted average on the input sequence according to the score, consisting of a query matrix Q, a key matrix K and a value matrix V; the multi-head attention is a method that divides the input sequence into multiple subspaces and performs scaled dot-product attention on each subspace respectively, and then splices the results, consisting of a linear transformation matrix W. In this embodiment, the feed-forward neural network is a method that maps the input sequence to another dimensional space and increases the model expression ability through a nonlinear activation function, consisting of two linear transformation matrices W1 and W2 and a ReLU function. The residual connection is a method that adds the input sequence to the output of the self-attention layer or the feed-forward neural network layer, which can avoid the problem of gradient vanishing or explosion. The layer normalization is a method that normalizes the input sequence at each position, which can accelerate the model convergence and improve the model stability.

[0026] In this embodiment, the generative adversarial network model is constructed based on an adversarial learning framework, which is an adversarial learning method implemented based on the principle of a generative adversarial network (GAN). The GAN is a method of using two neural networks as opponents to play against each other, thereby improving the quality of generated data; one neural network acts as a generator (Generator), i.e., the initial question and answer generation model described above, responsible for generating data similar to the real data distribution; the other neural network acts as a discriminator (Discriminator), i.e., the initial question and answer discrimination model described above, responsible for determining whether the input data is real data or generated data. The GAN minimizes the mutual information distance between the generator and the discriminator, so that the generator can generate data that is increasingly close to the real data distribution, while the discriminator can increasingly distinguish between real data and generated data. By alternately updating the parameters of the generator and the discriminator and maintaining a dynamic balance between the two, the two reach a Nash equilibrium, i.e., when the other party is fixed, either party cannot improve its own benefits by changing its own strategy.

[0027] Here, the GAN used consists of a generator and a discriminator. The task of the generator is to generate samples similar to real data from random noise, and the task of the discriminator is to distinguish between real data and generated data. The two compete with each other and eventually reach a Nash equilibrium, i.e., the data generated by the generator cannot be distinguished by the discriminator.

[0028] The loss function of an example GAN can be represented as:

[0029]

[0030] where x is real data, z is random noise, p data is the distribution of real data, p z is the distribution of noise, D(x) is the output probability of the discriminator for x, G(z) is the output sample of the generator for z, is the expectation function symbol, and log is the logarithmic function symbol.

[0031] The method provided in this embodiment uses a framework based on adversarial learning, and uses two generative models as opponents to play against each other, thereby improving the quality of generated question and answer pair data.

[0032] In the embodiment, the plurality of tasks can be a plurality of different types or difficulty question and answer tasks, including but not limited to fill-in-the-blank, multiple-choice, reasoning, etc.

[0033] In step S220, the preset meta-learning model is meta-trained by the initial evaluation question and answer pair data of each task to obtain meta-parameters, so as to update the generative adversarial network model through the meta-parameters.

[0034] Wherein, is to find an initial parameter that can perform well on multiple tasks and can reach the optimal solution through a small amount of gradient update. For example, Figure 3 As shown in FIG. 2, in step S220, that is, the preset meta-learning model is meta-trained by the initial evaluation question and answer pair data of each task to obtain meta-parameters, including the following steps:

[0035] In step S221, the initial evaluation question and answer pair data of each task is divided into a support set and a query set of each task to obtain the support set and the query set of each task. The initial evaluation question and answer pair data of each task includes a plurality of initial question and answer pair data, and each initial question and answer pair data includes an initial question and an initial answer matched with the initial question. The number of initial question and answer pair data in the support set is greater than the number of initial question and answer pair data in the query set. It should be noted that the total number of initial question and answer pair data in the support set and the query set can be less than or equal to the total number of initial question and answer pair data in the initial evaluation question and answer pair data of the task, that is, the full amount of initial question and answer pair data can be divided, or part of the initial question and answer pair data can be divided.

[0036] In step S222, the initial question and answer generation model and the initial question and answer discrimination model are respectively updated by the first gradient of the support set of each task, and the adapted parameters are calculated.

[0037] In step S223, the query set of each task is used to calculate a plurality of task sub-losses of the adapted parameters in each task, and a total task loss is determined.

[0038] In step S224, the initial parameters of the preset meta-learning model are updated by the second gradient of the total task loss until the initial parameters converge to obtain the meta-parameters.

[0039] For steps S221-S224, the initial evaluation question and answer pair data of all tasks are taken as metadata, a number of tasks Ti are randomly sampled from the metadata, and a support set Si and a query set Qi are respectively sampled for each task Ti, wherein the support set Si is used for fast adaptation of the generative model, and the query set Qi is used for evaluating the performance of the adapted model; then, the support set Si is used to perform K-step gradient update (i.e., the aforementioned first gradient update) on the initial question and answer generation model and the initial question and answer discrimination model (hereinafter referred to as the generative model), and to calculate the adapted parameter P'; wherein K is a hyperparameter, representing the number of steps of fast adaptation; then, the query set Qi is used to calculate the loss function Li of the adapted parameter P' on each task Ti, and the initial parameter P is updated by gradient according to the average loss function L of all tasks Ti, i.e., the second gradient update; it should be noted that the hyperparameters selected for the first gradient update and the second gradient update can be the same or different; finally, the above steps are repeated until the initial parameter P converges, and the meta-parameter P is obtained.

[0040] For example, taking a preset meta-learning model as a meta-learner L, the meta-learner L is a meta-learning method implemented based on a model-agnostic meta-learning (MAML) algorithm. Meta-learning is a method of letting machines "learn to learn", i.e., letting machines be able to quickly adapt and optimize according to different tasks. The goal of meta-learning is to find a general model or algorithm that can achieve good performance on new tasks with a small amount of training data and iteration times. MAML optimizes model parameters on multiple tasks alternately, so that the model can reach the optimal solution with a small amount of gradient updates on any task. MAML is applicable to any gradient descent-based model without modifying the model structure or loss function.

[0041] An optimization process of an example MAML can be represented as:

[0042]

[0043] wherein θ is a model parameter, is the i-th task, is a task distribution, is a loss function of the i-th task, f θ is a model function, α and β are learning rates, θ i ' is the parameter of the i-th task after one gradient update, θ' is the parameter of all tasks after one gradient update, is a gradient operator, and argmin is a variable value corresponding to the minimum value of a function.

[0044] In one embodiment, updating the generative adversarial network (GAN) model using meta-parameters includes: rapidly adapting the GAN model using meta-parameters to obtain adapted parameters (FastAdaptation); and updating the GAN model based on the adapted parameters. The initial question-answering generation model M1 and the initial question-answering discrimination model M2 are updated using the adapted parameters P'. In this case, the method further includes setting task weights W according to multiple question-answering tasks T of different types or difficulties.

[0045] The method provided in the above embodiments improves the efficiency of question-answer pair data generation by introducing a meta-learning-based mechanism and using a small amount of manually labeled or edited question-answer pair data as metadata to quickly adapt the generative model.

[0046] In some embodiments, the determination of the total task loss includes: determining the average task loss and the variance of the task sub-loss based on the multiple task sub-losses of each task; determining the difficulty index of each task based on the average task loss and the variance of the task sub-loss, thus obtaining the difficulty index of each task; determining the proportion of the difficulty index of each task to the sum of the difficulty indices of all tasks, as the loss weight of each task, thus obtaining the loss weight of all tasks; determining the task loss of each task based on the loss weight of each task and the average task loss, and obtaining the total task loss based on the task loss of all tasks.

[0047] The method for determining the total task loss provided in the above embodiments is that the question-answering pair evaluation data generation method introduces a multi-task learning strategy. This multi-task learning strategy is a multi-task learning method based on dynamic weight allocation. Multi-task learning methods utilize the correlation and complementarity between different tasks, simultaneously training a single model to complete the learning of multiple tasks, which can improve the model's generalization ability and robustness. The multi-task mechanism dynamically adjusts the weight of each task in the loss function according to the difficulty and importance of different tasks, balancing the influence and contribution of different tasks. An example step is as follows: First, based on each task T... i The average loss of the task sub-task L i And the variance of the task sub-loss S i Calculate each task T i Difficulty Index D i D i =L i / S i Then, based on each task T i Difficulty Index D i Calculate each task T i The loss weight W in the loss function i Wi = D i / sum(D j ), wherein D i is a difficulty index of the task T i , and sum is a summation function; finally, a total loss L of tasks is calculated according to the weight W i and the loss L i of each task T i , and the model parameters are updated according to the gradient descent method; wherein L = sum(W i *L i ), wherein L is the total loss of tasks, L i is the loss of the task T i , W i is the weight of the task T i , and sum is a summation function.

[0048] Through the above multi-task learning strategy, the generative model is simultaneously trained by using multiple different types or difficulty of question and answer tasks, and the generation difficulty is dynamically adjusted according to the task weight, thereby improving the generation diversity of question and answer pair data.

[0049] In step S230, the vertical field document is input into the updated initial question and answer generation model, so that the updated initial question and answer generation model outputs intermediate evaluation question and answer pair data based on the target task and the task weight of the target task.

[0050] In an embodiment, before the vertical field document is input into the updated initial question and answer generation model, the method further comprises: setting the task weight of each task; or, obtaining an adjustment weight of a task to be adjusted, and determining the adjustment weight as the task weight of the task to be adjusted, the task to be adjusted being one or more of the multiple tasks. That is, the generation difficulty can be dynamically adjusted by adjusting the task weight, so that the generation diversity of question and answer pair data can be improved. That is, the task weight is used to adjust the data difficulty of the intermediate evaluation question and answer pair data output by the updated initial question and answer generation model.

[0051] After the generative adversarial network model is updated by the meta-parameters respectively, so that the initial question and answer generation model and the initial question and answer discrimination model in the generative adversarial network model are both updated, the vertical field document is input into the updated initial question and answer generation model, and the updated initial question and answer generation model randomly selects a task type as a target task according to the vertical field document, the multiple tasks and the task weight corresponding to each task, and generates a question and an answer according to the corresponding difficulty, to obtain one intermediate evaluation question and answer pair data of the task. At this time, since the generative adversarial network model has not been trained, the intermediate evaluation question and answer pair data may have problems in accuracy, etc., and step S240 of evaluating the above intermediate evaluation question and answer pair data can be further performed.

[0052] Step S240, the updated initial question and answer discrimination model is used to evaluate the intermediate evaluation question and answer pair data based on the vertical field document, and an evaluation probability value is obtained, which represents the rationality of the intermediate evaluation question and answer pair data.

[0053] After the updated initial question and answer generation model outputs the intermediate evaluation question and answer pair data, it needs to be evaluated. At this time, the updated initial question and answer discrimination model can be triggered to output a probability value as an evaluation probability value according to the vertical field document and the intermediate evaluation question and answer pair data, so as to represent the rationality of the intermediate evaluation question and answer pair data.

[0054] Step S250, a model loss function between the updated initial question and answer discrimination model and the updated initial question and answer generation model is calculated.

[0055] The determination method of the model loss function can be realized by the method known by those skilled in the art, such as cross-discriminative loss, etc.

[0056] Step S260, the updated generative adversarial network model is iteratively trained by the model loss function until a preset condition is reached.

[0057] For example, the network model parameters of the updated generative adversarial network model can be updated by iteration based on the model loss function until a preset condition is reached, which includes but is not limited to at least one of the following: a preset number of times, a preset time, loss function convergence, etc. When the preset condition is reached, it indicates that the training of the updated generative adversarial network model is completed. If the preset condition is not reached, steps S230-S260 are repeatedly executed.

[0058] In an embodiment, the network model parameters of the updated generative adversarial network model can be updated by gradient descent method.

[0059] Step S270, the vertical field document is input into the iteratively trained initial question and answer generation model to obtain final evaluation question and answer pair data as the question and answer pair evaluation data of the vertical field document.

[0060] The trained initial question and answer generation model has better accuracy, better generation speed, and can also generate more diverse question and answer pair data. Therefore, the vertical field document can be input into the iteratively trained initial question and answer generation model, so that the obtained question and answer pair evaluation data (i.e. final evaluation question and answer pair data) can better meet the needs of the evaluation of the GPT type large model facing the vertical field document understanding.

[0061] In an embodiment, after the vertical field document input is iteratively trained on the initial question and answer generation model to obtain the final evaluation question and answer pair data, the method further comprises: generating an evaluation data set of the vertical field document according to the final evaluation question and answer pair data, the vertical field document, and the association relationship between the final evaluation question and answer pair data and the vertical field document.

[0062] In this way, when the performance and effect of a certain GPT type large model need to be evaluated in the future, the evaluation data set can be directly selected to evaluate the GPT type large model, and the evaluation requirement of the GPT type large model can be met.

[0063] In an embodiment, the question and answer pair evaluation data generation method can support question and answer pair evaluation data generation for multiple vertical field documents. The question and answer pair evaluation data generation for multiple vertical field documents is a repetition of the above steps S210-S270, which will not be repeated here. It should be noted that the multiple vertical field documents can be in the same field or in different fields.

[0064] Below, through a specific embodiment, the above-mentioned embodiment provides the question and answer pair evaluation data generation method for exemplary description. The GPT type large model evaluation question and answer pair data generation method for vertical field document understanding includes the following main steps: input a vertical field document D; initialize two generative models M1 and M2 constructed and pre-trained based on the GPT technology framework; wherein M1 is used as a question and answer generator (Question Answer Generator), responsible for generating a question Q and an answer A according to the document D; M2 is used as a question and answer discriminator (Question Answer Discriminator), responsible for judging whether it is reasonable according to the document D, question Q and answer A; a small amount of artificially labeled or edited question and answer pair data is used as metadata S, and meta-training (Meta Training) is performed on the meta-learner (Meta Learner) L to obtain meta-parameters P; the meta-parameters P are used for fast adaptation (Fast Adaptation) of M1 and M2 to obtain adapted parameters P'; M1 and M2 are updated respectively using the adapted parameters P', and task weights W are set according to a plurality of different types or difficulty of question and answer tasks T; the following sub-steps are repeatedly executed: M1 randomly selects a task type according to the document D, task T and weight W and generates a question Q' and an answer A' according to the corresponding difficulty; M2 outputs a probability value S' representing the degree of reasonableness according to the document D, question Q' and answer A'; the loss function L between M1 and M2 is calculated, and M1 and M2 parameters are updated according to the gradient descent method; it is judged whether the preset condition C (such as the number of iterations, time limit, etc.) is reached, if yes, the loop is exited; otherwise, the sub-steps are continued to be executed; the final generated results Q' and A' are output as evaluation question and answer pair data, and are stored in the evaluation data set together with the document D; it is judged whether there is a new vertical field document input, if yes, the initial step is returned; otherwise, the method is ended.

[0065] Next, the question and answer pair evaluation data generation method provided by the above embodiment is exemplarily described through another specific embodiment. The question and answer pair evaluation data generation method provided by the present embodiment includes the following main steps: first, input a vertical field document D, such as a popular science document in the medical health field, then initialize two generative models M1 and M2 constructed based on the GPT technology framework and pre-trained, use a small amount of artificially annotated or edited question and answer pair data as metadata S, the question and answer pair data is, for example, the question is how long does a common cold generally recover, and the answer is generally 10 days. Meta-training is performed on the meta-learner L to obtain the meta-parameter P, which is used to quickly adapt M1 and M2 to obtain the post-adaptation parameter P', which is used to update M1 and M2, and set the task weight W according to a plurality of different types or difficulty of question and answer tasks T, judge whether the preset condition C is reached, if the preset condition is reached, output the final generation result Q' and A' as the evaluation question and answer pair data, and store it together with the document D in the evaluation data set, if the preset condition is not reached, M1 randomly selects a task type according to the document D, the task T and the weight W and generates a question Q' and an answer A' according to the corresponding difficulty, M2 outputs a probability value S' representing the reasonable degree according to the document D, the question Q' and the answer A', calculates the loss function L between M1 and M2, and updates the parameters of M1 and M2 according to the gradient descent method, continues to judge whether the preset condition is reached, if not, continue the above steps until the preset condition is reached, to output the final generation result Q' and A' as the evaluation question and answer pair data. The final generation result exists a plurality of task types of questions and answers.

[0066] It can be seen that, in the above scheme, the initial evaluation question and answer pair data of multiple tasks is obtained by preliminarily labeling the vertical document, the initial evaluation question and answer pair data of the multiple tasks is used as metadata for meta-training to update the generative adversarial network model, the intermediate evaluation question and answer pair data based on the weight of each task is obtained by inputting the vertical field document into the updated initial question and answer generation model, the intermediate evaluation question and answer pair data is evaluated by the updated initial question and answer discrimination model, the model loss function of the two updated models is calculated, the two models are iteratively trained until a preset condition is reached, and finally the question and answer pair evaluation data of the vertical field document is obtained by inputting the vertical field document into the iteratively trained initial question and answer generation model. The method uses the framework based on adversarial learning, and the two initial question and answer generation models and the initial question and answer discrimination model that are opponents of each other are used to compete with each other, so as to improve the generation quality of the question and answer pair data. By introducing the meta-learning mechanism, the initial evaluation question and answer pair data obtained by a small amount of initial labeling is used as metadata to quickly adapt the initial question and answer generation model and the initial question and answer discrimination model, the generation efficiency of the question and answer pair data is improved, large-scale manual labeling and compilation are avoided, and a generation method of a GPT type large model evaluation data set with lower cost, higher efficiency, covering multiple fields and multiple tasks, being more objective, accurate, comprehensive, large-scale, diverse, difficult, and high-quality is provided.

[0067] The embodiment of the present application adopts a framework based on adversarial learning (Adversarial Learning), and two generative models (one as a question and answer generator and one as a question and answer discriminator) that are opponents of each other are used to compete with each other, so as to improve the generation quality of the question and answer pair data. A mechanism based on meta-learning (Meta Learning) is also introduced, and a small amount of manually labeled or edited question and answer pair data is used as metadata (MetaData) to quickly adapt (Fast Adaptation) the generative model, thereby improving the generation efficiency of the question and answer pair data. In addition, the embodiment of the present application also designs a strategy based on multi-task learning (Multi-task Learning), which uses multiple question and answer tasks of different types or difficulties (such as fill-in-the-blank, multiple-choice, reasoning, etc.) to simultaneously train the generative model, and dynamically adjusts the generation difficulty according to the task weight, thereby improving the generation diversity of the question and answer pair data.

[0068] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0069] In an embodiment, a question and answer pair evaluation data generation device is provided, which corresponds to the question and answer pair evaluation data generation method in the above embodiment. As shown inFigure 4 As shown, the question and answer pair evaluation data generation apparatus includes an acquisition module 601, a meta-training module 602, an intermediate evaluation question and answer pair data output module 603, an evaluation module 604, a model loss function determination module 605, a model training module 606, and a question and answer pair evaluation data generation module 607. The functions of each module are described in detail as follows: the acquisition module 601 is configured to acquire vertical field documents, a generative adversarial network model, and initial evaluation question and answer pair data of a plurality of tasks. The initial evaluation question and answer pair data is obtained by initially labeling the vertical field documents. The generative adversarial network model includes an initial question and answer generation model and an initial question and answer discrimination model. The meta-training module 602 is configured to perform meta-training on a preset meta-learning model by using the initial evaluation question and answer pair data of each task to obtain meta-parameters, and update the generative adversarial network model by using the meta-parameters. The intermediate evaluation question and answer pair data output module 603 is configured to input the vertical field documents into the updated initial question and answer generation model, so that the updated initial question and answer generation model outputs intermediate evaluation question and answer pair data based on a target task and a task weight of the target task. The evaluation module 604 is configured to evaluate the intermediate evaluation question and answer pair data based on the vertical field documents by using the updated initial question and answer discrimination model to obtain an evaluation probability value, which represents the rationality of the intermediate evaluation question and answer pair data. The model loss function determination module 605 is configured to calculate a model loss function between the updated initial question and answer discrimination model and the updated initial question and answer generation model. The model training module 606 is configured to iteratively train the updated generative adversarial network model by using the model loss function until a preset condition is reached. The question and answer pair evaluation data generation module 607 is configured to input the vertical field documents into the iteratively trained initial question and answer generation model to obtain final evaluation question and answer pair data, which is used as the question and answer pair evaluation data of the vertical field documents.

[0070] In an embodiment, the meta-training module 602 is specifically configured to: divide the initial evaluation question and answer pair data of each task into a support set and a query set of each task, to obtain the support set and the query set of each task, wherein the initial evaluation question and answer pair data of each task includes a plurality of initial question and answer pair data, and each initial question and answer pair data includes an initial question and an initial answer matched with the initial question; perform first gradient updates on the initial question and answer generation model and the initial question and answer discrimination model by using the support set of each task, respectively, and calculate to obtain adapted parameters; calculate a plurality of task sub-losses of the adapted parameters in each task by using the query set of each task, and determine a total task loss; perform second gradient updates on initial parameters of the preset meta-learning model by using the total task loss until the initial parameters converge, to obtain the meta-parameters.

[0071] In an embodiment, the meta-training module 602 is further configured to: determine a task sub-average loss and a task sub-loss variance according to the plurality of task sub-losses of each task; determine a difficulty index of each task based on the task sub-average loss and the task sub-loss variance, to obtain the difficulty index of each task; determine a proportion of the difficulty index of each task in a total sum of difficulty indexes of all tasks as a loss weight of each task, to obtain the loss weights of all tasks; determine a task loss of each task according to the loss weight of each task and the task sub-average loss, and obtain a total task loss based on the task losses of all tasks.

[0072] In an embodiment, the meta-training module 602 is further configured to: quickly adapt the generative adversarial network model using the meta-parameters, to obtain adapted parameters; and update the generative adversarial network model based on the adapted parameters.

[0073] In an embodiment, the generative adversarial network model includes two sub-network models, one of which is used as an initial question-answer generation model and the other of which is used as an initial question-answer discrimination model; each sub-network model includes two conversion modules, one of which is used as an encoder and the other of which is used as a decoder, the encoder being configured to convert the vertical domain document, the input question, or the input answer into a hidden state vector, and the decoder being configured to generate an output question or an output answer based on the input hidden state vector; each conversion module includes a plurality of conversion sub-modules, each conversion sub-module including a self-attention layer, a residual connection layer, a normalization layer, and a feedforward neural network, and the self-attention layer including a scaled dot-product attention and a multi-head attention.

[0074] In an embodiment, the apparatus further includes a dataset generation module configured to: input the vertical domain document into the initial question-answer generation model after iterative training, to obtain final evaluation question-answer pair data, and generate an evaluation dataset of the vertical domain document based on the final evaluation question-answer pair data, the vertical domain document, and a correlation between the final evaluation question-answer pair data and the vertical domain document.

[0075] In an embodiment, the apparatus further includes a task weight generation module configured to: set a task weight of each task before inputting the vertical domain document into the updated initial question-answer generation model; or, obtain an adjustment weight of a task to be adjusted, and determine the adjustment weight as the task weight of the task to be adjusted, the task to be adjusted being one or more of the plurality of tasks.

[0076] The embodiment of the present application provides a kind of question and answer pair evaluation data generation device, by the initial evaluation question and answer pair data of multiple tasks obtained after being preliminarily marked to vertical document, initial evaluation question and answer pair data of the multiple tasks as metadata is carried out meta-training, to update generative adversarial network model, the intermediate evaluation question and answer pair data based on each task weight is obtained by inputting vertical field document into updated initial question and answer generation model, the intermediate evaluation question and answer pair data is evaluated by updated initial question and answer discrimination model, the model loss function of two models after updating is calculated, the above two models are iteratively trained until reaching preset condition, finally the question and answer pair evaluation data of the vertical field document is obtained by inputting vertical field document into the initial question and answer generation model after iteration training, the method is based on the framework of adversarial learning, utilizes two initial question and answer generation models and initial question and answer discrimination model as each other's opponent and is mutually played, to improve the quality of question and answer pair data generation, by introducing meta-learning mechanism, using the initial evaluation question and answer pair data obtained by a small amount of initial labeling as metadata, the initial question and answer generation model and initial question and answer discrimination model are quickly adapted, the question and answer pair data generation efficiency is improved, large-scale manual labeling and compilation are avoided, and a generation mode of GPT type large model evaluation data set with lower cost, higher efficiency, capable of covering multiple fields and multiple tasks, more objective, accurate, comprehensive, large-scale, diverse and high-quality is provided.

[0077] The specific limitations of the question and answer pair evaluation data generation device can be referred to the limitations of the question and answer pair evaluation data generation method in the above, which will not be repeated here. Each module in the above question and answer pair evaluation data generation device can be realized by software, hardware and their combination in whole or in part. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0078] In one embodiment, a computer device is provided, which can be a server, and its internal structure diagram can be as shown in Figure 5 The computer device includes a processor, a memory, a network interface and a database connected by a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes non-volatile and / or volatile storage medium, internal memory. The non-volatile storage medium stores operating system, computer program and database. The internal memory provides an environment for the operation of operating system and computer program in non-volatile storage medium. The network interface of the computer device is used to communicate with external client through network connection. The computer program is executed by the processor to realize a function or step of a server side based on question and answer pair evaluation data generation method.

[0079] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements client-side functions or steps of a question-and-answer pair evaluation data generation method.

[0080] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following steps: acquiring vertical domain documents, a generative adversarial network (GAN) model, and initial evaluation question-answer pairs for multiple tasks; the initial evaluation question-answer pairs are obtained by initially labeling the vertical domain documents; the GAN model includes an initial question-answer generation model and an initial question-answer discrimination model; performing meta-training on a preset meta-learning model using the initial evaluation question-answer pairs for each task to obtain meta-parameters, which are then used to update the GAN model; and inputting the vertical domain documents into the updated initial question-answer generation model. The updated initial question-answer generation model outputs intermediate evaluation question-answer pairs based on the target task and its weights. The updated initial question-answer discrimination model evaluates these intermediate pairs based on vertical domain documents, obtaining evaluation probability values ​​that characterize the reasonableness of the intermediate evaluation question-answer pairs. A model loss function is calculated between the updated initial question-answer discrimination model and the updated initial question-answer generation model. The updated generative adversarial network model is iteratively trained using this model loss function until preset conditions are met. Vertical domain documents are input into the iteratively trained initial question-answer generation model to obtain the final evaluation question-answer pairs, which serve as the evaluation data for the question-answer pairs in the vertical domain documents.

[0081] The computer device provided in the above embodiment obtains initial evaluation question and answer pair data of multiple tasks by performing preliminary labeling on the vertical documents, performs meta-training on a preset meta-learning model based on the initial evaluation question and answer pair data of the multiple tasks as metadata, updates the generative adversarial network model based on the meta-parameters, inputs the vertical domain document into the updated initial question and answer generation model to obtain intermediate evaluation question and answer pair data based on a weight of each task, evaluates the intermediate evaluation question and answer pair data based on the vertical domain document by using the updated initial question and answer evaluation model to obtain an evaluation probability value, and reflects the rationality of the intermediate evaluation question and answer pair data by using the evaluation probability value. The model loss function between the updated initial question and answer evaluation model and the updated initial question and answer generation model is calculated, the updated generative adversarial network model is iteratively trained until a preset condition is reached, and finally, the vertical domain document is input into the iteratively trained initial question and answer generation model to obtain the question and answer pair evaluation data of the vertical domain document. The method uses the framework of adversarial learning, and the two initial question and answer generation models and the initial question and answer evaluation model that are opponents of each other are used to compete with each other, so that the quality of the question and answer pair data is improved. By introducing the meta-learning mechanism, the initial evaluation question and answer pair data obtained by using a small amount of initial labeling is used as the metadata, the initial question and answer generation model and the initial question and answer evaluation model are quickly adapted, the efficiency of generating the question and answer pair data is improved, large-scale manual labeling and compilation are avoided, and a generation method of a GPT type large model evaluation data set that is lower in cost, higher in efficiency, can cover multiple fields and multiple tasks, is more objective, accurate, comprehensive, has large scale, diversity, high difficulty, and high quality is provided.

[0082] In one embodiment, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the following steps: obtaining a vertical domain document, a generative adversarial network model, and initial evaluation question and answer pair data of multiple tasks. The initial evaluation question and answer pair data is obtained by performing initial labeling on the vertical domain document. The generative adversarial network model includes an initial question and answer generation model and an initial question and answer evaluation model. The initial evaluation question and answer pair data of each task is used to perform meta-training on a preset meta-learning model to obtain meta-parameters, and the generative adversarial network model is updated based on the meta-parameters. The vertical domain document is input into the updated initial question and answer generation model, so that the updated initial question and answer generation model outputs intermediate evaluation question and answer pair data based on a target task and a task weight of the target task. The intermediate evaluation question and answer pair data is evaluated based on the vertical domain document by using the updated initial question and answer evaluation model to obtain an evaluation probability value. The evaluation probability value represents the rationality of the intermediate evaluation question and answer pair data. A model loss function between the updated initial question and answer evaluation model and the updated initial question and answer generation model is calculated. The updated generative adversarial network model is iteratively trained based on the model loss function until a preset condition is reached. The vertical domain document is input into the iteratively trained initial question and answer generation model to obtain final evaluation question and answer pair data as question and answer pair evaluation data of the vertical domain document.

[0083] The computer readable storage medium provided by the above embodiment, by using two initial question and answer generation models and initial question and answer discrimination models as opponents to compete with each other based on a framework of adversarial learning, the quality of generated question and answer pair data is improved, by introducing a meta-learning mechanism, using a small amount of initial labeled initial evaluation question and answer pair data as metadata, the initial question and answer generation model and the initial question and answer discrimination model are quickly adapted, the question and answer pair data generation efficiency is improved, large-scale manual labeling and compilation are avoided, and a generation mode of a GPT type large model evaluation data set with lower cost, higher efficiency, covering multiple fields and multiple tasks, being more objective, accurate, comprehensive, large-scale, diverse, difficult, and high-quality is provided.

[0084] It should be noted that the functions or steps that the computer readable storage medium or the computer device can achieve correspond to the related descriptions of the server side and the client side in the foregoing method embodiments, and will not be described again here to avoid repetition.

[0085] Those skilled in the art can understand that all or part of the processes in the above embodiment methods can be completed by a computer program instructing related hardware, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-described embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0086] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is exemplified, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0087] The above-described embodiments are only used to illustrate the technical solutions of the present application, but not limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those ordinarily skilled in the art should understand: the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for generating question-answer pair evaluation data, characterized in that, include: The process involves acquiring initial evaluation question-answer pairs for vertical domain documents, generative adversarial network models, and multiple tasks. These initial evaluation question-answer pairs are obtained by initially labeling the vertical domain documents. The generative adversarial network models include an initial question-answer generation model and an initial question-answer discrimination model. The preset meta-learning model is meta-trained using the initial evaluation question-and-answer data for each task to obtain meta-parameters, which are then used to update the generative adversarial network model. The vertical domain document is input into the updated initial question-answer generation model so that the updated initial question-answer generation model outputs intermediate evaluation question-answer pair data based on the target task and the task weight of the target task; The updated initial question-answering identification model evaluates the intermediate evaluation question-answer pairs based on the vertical domain documents to obtain evaluation probability values, which characterize the reasonableness of the intermediate evaluation question-answer pairs. Calculate the model loss function between the updated initial question answering discrimination model and the updated initial question answering generation model; The updated generative adversarial network model is iteratively trained using the model loss function until the preset conditions are met. The vertical domain documents are input into the initial question-answer generation model after iterative training to obtain the final evaluation question-answer pair data, which is used as the question-answer pair evaluation data for the vertical domain documents.

2. The question-and-answer pair evaluation data generation method as described in claim 1, characterized in that, The pre-defined meta-learning model is meta-trained using the initial evaluation question-and-answer data for each task to obtain meta-parameters, including: The initial evaluation question-answer pair data for each task is divided into a support set and a query set for each task, resulting in a support set and a query set for each task. The initial evaluation question-answer pair data for each task includes multiple initial question-answer pair data, and each initial question-answer pair data includes an initial question and an initial answer that matches the initial question. The initial question-answering generation model and the initial question-answering discrimination model are updated using the support set for each task, and the adapted parameters are calculated. The adapted parameters are calculated using the query set of each task, and the total task loss is determined. The initial parameters of the preset meta-learning model are updated using the total task loss as a second gradient until the initial parameters converge, thus obtaining the meta-parameters.

3. The question-and-answer pair evaluation data generation method as described in claim 2, characterized in that, The methods for determining the total task loss include: The average sub-loss and variance of the sub-loss are determined based on the multiple sub-losses of each task. The difficulty index of each task is determined based on the average loss of the task sub-task and the variance of the loss of the task sub-task, thus obtaining the difficulty index of each task. The proportion of the difficulty index of each task to the sum of the difficulty indices of all tasks is determined and used as the loss weight of each task. The task loss for each task is determined based on the loss weight of each task and the average loss of each task sub-task, and the total task loss is obtained based on the task losses of all tasks.

4. The question-answer pair evaluation data generation method as described in any one of claims 1-3, characterized in that, The generative adversarial network model is updated using the meta-parameters, including: The generative adversarial network model is rapidly adapted using the meta-parameters to obtain the adapted parameters. The generative adversarial network model is updated based on the adapted parameters.

5. The question-answer pair evaluation data generation method as described in any one of claims 1-3, characterized in that, The generative adversarial network model includes two sub-network models, one of which serves as the initial question-answer generation model and the other as the initial question-answer discrimination model. The sub-network model includes two transformation modules, one of which acts as an encoder and the other as a decoder. The encoder is used to convert the vertical domain document, input question, or input answer into a hidden state vector, and the decoder is used to generate an output question or output answer based on the input hidden state vector. The transformation module includes multiple transformation sub-modules, each including a self-attention layer, a residual connection layer, a normalization layer, and a feedforward neural network. The self-attention layer includes scaled dot product attention and multi-head attention.

6. The question-answer pair evaluation data generation method as described in any one of claims 1-3, characterized in that, After inputting the vertical domain documents into the iteratively trained initial question-answer generation model to obtain the final evaluation question-answer pair data, the method further includes: The evaluation dataset for the vertical domain documents is generated based on the final evaluation question-and-answer pairs, the vertical domain documents, and the association between the final evaluation question-and-answer pairs and the vertical domain documents.

7. The question-answer pair evaluation data generation method as described in any one of claims 1-3, characterized in that, Before inputting the vertical domain document into the updated initial question-answering generation model, the method further includes: Set the task weight for each task; or, Obtain the adjustment weight of the task to be adjusted, and determine the adjustment weight as the task weight of the task to be adjusted, wherein the task to be adjusted is one or more of a plurality of tasks.

8. A question-and-answer pair evaluation data generation device, characterized in that, include: The acquisition module is used to acquire vertical domain documents, generative adversarial network models, and initial evaluation question-answer pair data for multiple tasks. The initial evaluation question-answer pair data is obtained by initially annotating the vertical domain documents. The generative adversarial network model includes an initial question-answer generation model and an initial question-answer discrimination model. The meta-training module is used to perform meta-training on the preset meta-learning model using the initial evaluation question-and-answer pair data for each task, and obtain meta-parameters, so as to update the generative adversarial network model using the meta-parameters. The intermediate evaluation question-answer pair data output module is used to input the vertical domain document into the updated initial question-answer generation model, so that the updated initial question-answer generation model outputs intermediate evaluation question-answer pair data based on the target task and the task weight of the target task; The evaluation module is used to evaluate the intermediate evaluation question-and-answer pairs based on the vertical domain document using the updated initial question-and-answer identification model, and obtain an evaluation probability value, which represents the reasonableness of the intermediate evaluation question-and-answer pairs. The model loss function determination module is used to calculate the model loss function between the updated initial question answering discrimination model and the updated initial question answering generation model; The model training module is used to iteratively train the updated generative adversarial network model using the model loss function until a preset condition is met. The question-answer pair evaluation data generation module is used to input the vertical domain document into the initial question-answer generation model after iterative training to obtain the final evaluation question-answer pair data, which is used as the question-answer pair evaluation data for the vertical domain document.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images