Government affairs question generation method and device

Abstract

The present invention discloses a method and device for generating government affairs questions, which can be used in the field of artificial intelligence technology. The method comprises: obtaining government affairs data, open source question and answer data and self-built government affairs question and answer text data; inputting the government affairs data into a pre-established UNILM network model for training to obtain a pre-trained model, wherein the UNILM network model is pre-established according to Bert Chinese model parameters; inputting the open source question and answer data and self-built government affairs question and answer text data into the pre-trained model for fine-tuning to obtain a question generation model; and using the question generation model to generate government affairs questions. The present invention can improve the effect of generating Chinese data questions.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and device for generating government affairs questions. Background Art

[0002] Question Generation (QG) refers to generating relevant questions based on articles and answers, and the answers can be obtained from the articles. The main application scenarios are: in the dialogue system, the chatbot actively asks questions to increase the continuity of interaction; in the construction of question-answering and machine reading comprehension datasets, automatically generating questions can reduce the work of manual analysis; in the process of building question-answer corpus, questions can be automatically generated, which is conducive to the rapid construction of original question-answering data of the question-answering system. At present, there is a lack of unified question-answering corpus in the government affairs field. In order to reduce the cost of manual analysis, question generation technology is used to automatically generate questions for government affairs data to help build datasets.

[0003] The current government affairs question generation method has poor effect on Chinese data question generation. Therefore, there is an urgent need for a government affairs question generation solution that can overcome the above problems. Summary of the invention

[0004] An embodiment of the present invention provides a method for generating government affairs questions, which is used to improve the effect of generating Chinese data questions. The method includes:

[0005] Obtain government data, open source question and answer data, and self-built government question and answer text data;

[0006] Inputting the government data into a pre-established UNILM network model for training to obtain a pre-trained model, wherein the UNILM network model is pre-established according to Bert Chinese model parameters;

[0007] Input the open source question-and-answer data and the self-built government question-and-answer text data into the pre-training model for fine-tuning to obtain a question generation model;

[0008] Use the problem generation model to generate government affairs problems.

[0009] The embodiment of the present invention provides a government affairs question generation device, which is used to improve the effect of generating Chinese data questions. The device includes:

[0010] Data acquisition module, used to obtain government data, open source question and answer data, and self-built government question and answer text data;

[0011] A model training module, used for inputting the government data into a pre-established UNILM network model for training to obtain a pre-trained model, wherein the UNILM network model is pre-established according to Bert Chinese model parameters;

[0012] A model fine-tuning module, used to input the open source question and answer data and the self-built government question and answer text data into the pre-trained model for fine-tuning to obtain a question generation model;

[0013] The question generation module is used to generate government affairs questions using the question generation model.

[0014] An embodiment of the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the government affairs issue generating method.

[0015] An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program for executing the above-mentioned government affairs question generating method.

[0016] The embodiment of the present invention obtains government data, open source question and answer data, and self-built government question and answer text data; inputs the government data into a pre-established UNILM network model for training to obtain a pre-trained model, wherein the UNILM network model is pre-established according to Bert Chinese model parameters; inputs the open source question and answer data and self-built government question and answer text data into the pre-trained model for fine-tuning to obtain a question generation model; and uses the question generation model to generate government questions. For a UNILM network model that is not open source in Chinese, the embodiment of the present invention uses Bert Chinese model parameters for initialization when establishing it, and then inputs the government data into the established UNILM network model for training to obtain a pre-trained model, and fine-tunes the pre-trained model for downstream tasks by fusing open source question and answer data and self-built government question and answer text data, thereby effectively improving the model effect and the Chinese data question generation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0018] Figure 1 A schematic diagram of a method for generating government affairs questions in an embodiment of the present invention;

[0019] Figure 2 It is a flow chart of a method for generating government affairs questions in a specific embodiment of the present invention;

[0020] Figure 3 This is a structural diagram of a device for generating government affairs questions in an embodiment of the present invention;

[0021] Figure 4It is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0023] First, the technical terms in the embodiments of the present invention are introduced:

[0024] Question Generation (QG): refers to generating relevant questions based on the article, whose answers can be obtained from the article, and can be preset or calculated.

[0025] NQG (Neural Question Generation): Neural network question generation.

[0026] Word Embedding: A general term for language models and representation techniques in natural language processing, which embeds a high-dimensional sparse space with all dimensions into a low-dimensional dense vector space.

[0027] Pre-trained model: The pre-trained model in NLP is learned on massive corpus data sets to predict word tokens based on the text representation information of the context. It can be used to solve specific downstream tasks. Deep learning has high requirements for data, especially the amount of labeled data. The pre-trained model can apply powerful representation capabilities to various tasks, solving the problem of lack of a large amount of labeled data in some tasks.

[0028] As mentioned above, the current government question generation method has poor results for Chinese data question generation. Question generation is mainly divided into rule-based and neural approach methods. The rule-based method mainly extracts the relevant entities of the target sentence, fills them into the manually written template (according to the rules and grammar), and then selects one or several most suitable templates according to the sorting method to generate questions. The advantage is that it is very smooth, but the disadvantage is that it is very dependent on manual templates and the quality is not high. The neural approach uses a seq2seq model to encode paragraphs and answers. Due to the lack of relevant data sets in Chinese, there are fewer related research and application results, and the effect is poor. The pre-trained model has achieved absolute advantages in various tasks of natural language understanding.

[0029] In order to improve the effect of generating Chinese data questions, the embodiment of the present invention provides a method for generating government affairs questions, such as Figure 1 As shown, the method may include:

[0030] Step 101: Obtain government data, open source question and answer data, and self-built government question and answer text data;

[0031] Step 102: input the government data into a pre-established UNILM network model for training to obtain a pre-trained model, wherein the UNILM network model is pre-established according to Bert Chinese model parameters;

[0032] Step 103: Input the open-source question-and-answer data and the self-built government question-and-answer text data into the pre-training model for fine-tuning to obtain a question generation model;

[0033] Step 104: Generate government affairs questions using the question generation model.

[0034] Depend on Figure 1 As shown, the embodiment of the present invention obtains government data, open source question and answer data and self-built government question and answer text data; inputs the government data into a pre-established UNILM network model for training to obtain a pre-trained model, wherein the UNILM network model is pre-established according to Bert Chinese model parameters; inputs the open source question and answer data and self-built government question and answer text data into the pre-trained model for fine-tuning to obtain a question generation model; and uses the question generation model to generate government questions. For the UNILM network model that is not open source in Chinese, the embodiment of the present invention uses the Bert Chinese model parameters for initialization when establishing it, and then inputs the government data into the established UNILM network model for training to obtain a pre-trained model, and fine-tunes the pre-trained model for downstream tasks by fusing the open source question and answer data and the self-built government question and answer text data, thereby effectively improving the model effect and the Chinese data question generation effect.

[0035] In an embodiment, government data, open source question and answer data, and self-built government question and answer text data are obtained; the government data is input into a pre-established UNILM network model for training to obtain a pre-trained model, wherein the UNILM network model is pre-established according to Bert Chinese model parameters.

[0036] In this embodiment, the government data is input into a pre-established UNILM network model for training to obtain a pre-trained model, including:

[0037] Performing word segmentation processing on the government data to obtain an input sequence, wherein the input sequence includes a plurality of words;

[0038] Determine the representation of the input sequence by combining word vectors, position vectors and text segment information;

[0039] The representation of the input sequence is combined with word vectors, position vectors and text segment information and input into a pre-established UNILM network model to output a text vector representation, wherein the UNILM network model includes a multi-layer Transformer network, and in the multi-layer Transformer network, each layer of the Transformer network includes: a unidirectional language network model, a bidirectional language network model and an end-to-end language network model.

[0040] In this embodiment, the government data is input into a pre-established UNILM network model for training to obtain a pre-trained model, which also includes:

[0041] After obtaining the input sequence, masking the input sequence using a self-attention mask matrix;

[0042] Determining the representation joint word vector, position vector and text segment information of the input sequence, including: determining the representation joint word vector, position vector and text segment information of the input sequence after masking.

[0043] The following introduces the UNILM network model structure. First, define the problem: Q = argM Q AX(Prob{Q|P,A}), Q represents the generated question, P represents a paragraph, and A represents the known answer. UNILM is a deep Transformer network. The UNILM network model contains multiple layers of Transformer networks. Its pre-training process uses three unsupervised language model objectives: bidirectional language network model, unidirectional language network model, and end-to-end language network model (sequence-to-sequence LM). The model uses a Transformer network with shared parameters and also uses specific self-attention masks to control the context information used in prediction. When fine-tuning downstream tasks, the UNILM network model can be regarded as a unidirectional encoding, bidirectional encoding, and end-to-end model to adapt to different downstream tasks (natural language understanding and generation tasks). In the question generation task, an end-to-end structure is used to input the interval of the text and the question in the text, and the answer is output. The UNILM network model is comprehensively compared with the BERT model on the GLUE, SQuAD 2.0, and CoQA datasets. It set new records in three natural language generation tasks, including CNN / DailyMail summary generation (ROUGE-L is 40.63, an improvement of 2.16), CoQA question generation (F1 value is 82.5, an improvement of 37.1), and SQuAD question generation (BLEU-4 is 22.88, an improvement of 6.5).

[0044] In the specific implementation, during the training phase of the UNILM network model, the input sequence is masked using the self-attention mask matrix, and some words are randomly masked. The target task is to restore these words. For example, the input X is a sequence, a text fragment, and the sequence representation is the same as Bert, including word vector, position encoding, and sequence encoding. The sequence encoding can be used as a unidirectional language model, a bidirectional language model, and an end-to-end training method. The backbone network consists of 24 layers of Transformers, and the output of each layer is the input of the next layer. Each layer controls the attention range of each word through the masking moment, thereby ensuring the joint training of multiple training objectives. Given an input sequence sequence x=x1…xn, (x1 is a word), a vector representation with context information is obtained for each word through a multi-layer Transformer network. The representation of the input word sequence combines word vectors, position vectors, and text segment information. The input vector is then input into the multi-layer Transformer network, and the self-attention mechanism is used to calculate the text representation in conjunction with the entire input sequence. When masking the input, the masking operation is to replace it with a predefined mask matrix. The input is then fed into the Transformer network to calculate the output vector, which is then fed into the softmax classifier to predict the masked words. The parameters of the UNILM model are learned by minimizing the cross entropy loss between the predicted words and the original true words. In the end-to-end mode, the structure is "[SOS]S1[EOS]S2[EOS]", where S1 (i.e., source segment) can notice any word within the entire segment, and the words in S2 (target segment) can only notice the previous words in the target part, themselves, and S1. The self-attention mask matrix of the end-to-end language model is set to all 0 for the source sequence, that is, all words can notice each other. For the target sequence, the mask matrix is ​​set to infinity, so that the source sequence cannot see the target sequence; the upper right corner value is set to infinity, and the other values ​​are 0 to achieve that the target segment can only notice the previous context and ignore the context.

[0045] In the embodiment, the open source question and answer data and the self-built government affairs question and answer text data are input into a pre-trained model for fine-tuning to obtain a question generation model; and the question generation model is used to generate government affairs questions.

[0046] In this embodiment, the open source question and answer data and the self-built government question and answer text data are input into the pre-training model for fine-tuning, including:

[0047] Determine a source sequence and a target sequence according to the open source question and answer data and the self-built government question and answer text data;

[0048] Masking the target sequence using a self-attention mask matrix;

[0049] The source sequence and the masked target sequence are fed into the pre-trained model for fine-tuning.

[0050] In specific implementation, in the fine-tuning of the pre-trained model, the UNILM model can be used in the downstream tasks of natural language understanding and natural language generation. Fine-tuning can make the model better applied to downstream specific tasks. Fine-tuning of the natural language generation task is similar to pre-training using self-attention masks. Let S1 and S2 represent the source sequence and the target sequence respectively. The fine-tuning of the model is to randomly mask a certain proportion of words in the target sequence S2, so that the model learns to recover the masked words. The training goal is to maximize the likelihood of the masked words based on the context. The question generation task is a type of NLG and can be expressed as an end-to-end problem. The first part is the input article and answer, and the second part is the generated question. By randomly masking the words in the question, the model learns how to recover the masked words to train the model parameters. We fine-tuned UNILM for 10 epochs on the training set. We set the batch size to 32, the masking probability to 0.7, and the learning rate to 2e-5. The label smoothing rate is 0.1, and other hyperparameters are the same as before training.

[0051] A specific embodiment is given below to illustrate the government affairs problem generation scheme in the embodiment of the present invention. Figure 2 As shown in the figure, the model training phase: 1. Initialize the model parameters, and select the Bert Chinese pre-trained model parameters open sourced from the official Google website as the initialization parameters; it has strong representation capabilities and consistent model structure; the cost of training from scratch is huge. 2. Parameter update: Use the crawled 3G government data (government-related text data, not affected by the format) to pre-train the model and update the parameters; this enhances the model's representation capabilities for government vocabulary. 3. The tasks of pre-training and model training are the same, with the goals of unidirectional, bidirectional, and end-to-end tasks, respectively.

[0052] Model fine-tuning stage: fine-tuning is performed using open source question-and-answer data (based on Baidu Dureader dataset and Wikipedia's open source question-and-answer corpus) and self-built government question-and-answer data (5,000 pairs of text data manually annotated and constructed, with the data content consisting of questions (Q) and answers (A)). The reason for mixing is that there is a large amount of open source data but lacks domain knowledge, while the amount of self-built data is small. Experiments have shown that mixing can improve model performance.

[0053] Question generation phase: The process of question generation is to use the model to decode the decoder, input the paragraph and question interval, perform encoder encoding, and decode to generate questions. The target data is the field data we need to decompose, which is government data here. Data preprocessing: Preprocess the article and the manually annotated answers to obtain the interval positions of the paragraph and the answer in the paragraph. Model calculation: Encode the paragraph and the answer with the UNILM model, and decode to generate questions after model calculation. Four cloze tasks are designed for the training objectives of different language models. In a cloze task, some words are randomly masked with a mask matrix, and then the corresponding output vector is calculated through the Transformer network, and then the output vector is fed into the softmax classifier to predict the masked words. The goal of UNILM parameter optimization is to minimize the cross entropy between the predicted value and the true value of the masked words.

[0054] The specific steps for generating government affairs questions are as follows:

[0055] (1) Load the vocabulary and build a word segmenter;

[0056] (2) Load the BERT Chinese model, configure network parameters, and set it to the UNILM network structure;

[0057] (3) Parameter update: Load government data and perform model pre-training. Use the pre-training method of UNILM to train on new data and update model parameters.

[0058] (4) Task fine-tuning: The task is an end-to-end model structure, and the objective function is maximization. Open source Q&A data (based on Baidu Dureader dataset and Wikipedia's open source corpus of questions and answers) and self-built government Q&A data (5000 pairs of text data manually annotated and constructed, the data content is questions (Q) and answers (A)) are fine-tuned; the reason for mixing is that there are more open source data but lack of domain knowledge, and the amount of self-built data is small. Experiments have shown that mixing can improve the model effect. Fine-tuning process: Load qa data (open source Q&A data (based on Baidu Dureader dataset and Wikipedia's open source corpus of questions and answers) and self-built government Q&A data (5000 pairs of text data manually annotated and constructed, the data content is questions (Q) and answers (A)) for fine-tuning), question generation is an NLG task, and an end-to-end structure is selected. The fine-tuning process is similar to pre-training using self-attention masks. Let S1 and S2 represent the source sequence and target sequence respectively, and construct the input [SOS]S1[EOS]S2[EOS]. The fine-tuning of the model is to randomly mask a certain proportion of words in the target sequence, allowing the model to learn to recover the masked words. The training goal is to maximize the likelihood of the masked words based on the context. This is slightly different from pre-training. During pre-training, the words in the source sequence and the target sequence are randomly masked, that is, both ends are involved in the training, while during fine-tuning, only the target sequence is involved because fine-tuning focuses more on the target end. It is worth noting that during fine-tuning, the end marker [EOS] of the target end can also be masked, allowing the model to learn to predict, so that the model can learn to automatically end the NLG task.

[0059] The government affairs question generation solution provided by the embodiment of the present invention can effectively automatically generate questions from documents, help solve the problem of constructing government affairs corpus in the question-answering system, reduce the cost of manually disassembling corpus, and save time and cost. For UNILM models that do not have Chinese open source, the open source Chinese Bert model is selected for initialization, and the model's representation of domain knowledge is enhanced by training and fine-tuning in self-built government affairs data. The model effect is improved by fusing general question generation data and self-built government affairs question generation data to fine-tune downstream tasks. Applying an end-to-end structural model, questions can be automatically generated in batches.

[0060] Based on the same inventive concept, the embodiment of the present invention also provides a device for generating government affairs problems, as described in the following embodiments. Since the principles of solving these problems are similar to the method for generating government affairs problems, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0061] Figure 3 FIG. 1 is a structural diagram of a device for generating government affairs questions in an embodiment of the present invention. Figure 3 As shown, the device comprises:

[0062] The data acquisition module 301 is used to obtain government data, open source question and answer data, and self-built government question and answer text data;

[0063] A model training module 302 is used to input the government data into a pre-established UNILM network model for training to obtain a pre-trained model, wherein the UNILM network model is pre-established according to Bert Chinese model parameters;

[0064] A model fine-tuning module 303 is used to input the open source question and answer data and the self-built government question and answer text data into a pre-trained model for fine-tuning to obtain a question generation model;

[0065] The question generation module 304 is used to generate government affairs questions using the question generation model.

[0066] In one embodiment, the model training module 302 is further used to:

[0067] Performing word segmentation processing on the government data to obtain an input sequence, wherein the input sequence includes a plurality of words;

[0068] Determine the representation of the input sequence by combining word vectors, position vectors and text segment information;

[0069] The representation of the input sequence is combined with word vectors, position vectors and text segment information and input into a pre-established UNILM network model to output a text vector representation, wherein the UNILM network model includes a multi-layer Transformer network, and in the multi-layer Transformer network, each layer of the Transformer network includes: a unidirectional language network model, a bidirectional language network model and an end-to-end language network model.

[0070] In one embodiment, the model training module 302 is further used to:

[0071] After obtaining the input sequence, masking the input sequence using a self-attention mask matrix;

[0072] Determining the representation joint word vector, position vector and text segment information of the input sequence, including: determining the representation joint word vector, position vector and text segment information of the input sequence after masking.

[0073] In one embodiment, the model fine-tuning module 303 is further used to:

[0074] Determine a source sequence and a target sequence according to the open source question and answer data and the self-built government question and answer text data;

[0075] Masking the target sequence using a self-attention mask matrix;

[0076] The source sequence and the masked target sequence are fed into the pre-trained model for fine-tuning.

[0077] In summary, the embodiment of the present invention obtains government data, open source question and answer data and self-built government question and answer text data; inputs the government data into a pre-established UNILM network model for training to obtain a pre-trained model, wherein the UNILM network model is pre-established according to Bert Chinese model parameters; inputs the open source question and answer data and self-built government question and answer text data into the pre-trained model for fine-tuning to obtain a question generation model; and uses the question generation model to generate government questions. For the UNILM network model that is not open source in Chinese, the embodiment of the present invention uses the Bert Chinese model parameters for initialization when establishing it, and then inputs the government data into the established UNILM network model for training to obtain a pre-trained model, and fine-tunes the pre-trained model for downstream tasks by fusing the open source question and answer data and the self-built government question and answer text data, thereby effectively improving the model effect and the Chinese data question generation effect.

[0078] Based on the above invention concept, Figure 4 As shown, the present invention also proposes a computer device 400, including a memory 410, a processor 420, and a computer program 430 stored in the memory 410 and executable on the processor 420, wherein the processor 420 implements the aforementioned government affairs issue generation method when executing the computer program 430.

[0079] Based on the aforementioned inventive concept, the present invention proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the aforementioned government affairs issue generating method is implemented.

[0080] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0081] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0082] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0083] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0084] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for generating government affairs problems, characterized in that: include: Obtain government data, open source question and answer data, and self-built government question and answer text data; Inputting the government data into a pre-established UNILM network model for training to obtain a pre-trained model, wherein the UNILM network model is pre-established according to Bert Chinese model parameters; Input the open source question-and-answer data and the self-built government question-and-answer text data into the pre-training model for fine-tuning to obtain a question generation model; Use the problem generation model to generate government affairs problems; The government data is input into a pre-established UNILM network model for training to obtain a pre-trained model, including: performing word segmentation processing on the government data to obtain an input sequence, wherein the input sequence includes multiple words; determining a representation joint word vector, a position vector and text segment information of the input sequence; inputting the representation joint word vector, the position vector and the text segment information of the input sequence into a pre-established UNILM network model, and outputting a text vector representation, wherein the UNILM network model includes a multi-layer Transformer network, and in the multi-layer Transformer network, each layer of the Transformer network includes: a unidirectional language network model, a bidirectional language network model and an end-to-end language network model; The government data is input into a pre-established UNILM network model for training to obtain a pre-trained model, which also includes: after obtaining the input sequence, using a self-attention mask matrix to mask the input sequence; determining the representation joint word vector, position vector and text segment information of the input sequence, including: determining the representation joint word vector, position vector and text segment information of the masked input sequence.

2. The method for generating government affairs questions according to claim 1, characterized in that: The open source Q&A data and self-built government Q&A text data are input into the pre-trained model for fine-tuning, including: Determine a source sequence and a target sequence according to the open source question and answer data and the self-built government question and answer text data; Masking the target sequence using a self-attention mask matrix; The source sequence and the masked target sequence are fed into the pre-trained model for fine-tuning.

3. A device for generating government affairs questions, characterized in that: include: Data acquisition module, used to obtain government data, open source question and answer data, and self-built government question and answer text data; A model training module, used for inputting the government data into a pre-established UNILM network model for training to obtain a pre-trained model, wherein the UNILM network model is pre-established according to Bert Chinese model parameters; A model fine-tuning module, used to input the open source question and answer data and the self-built government question and answer text data into the pre-trained model for fine-tuning to obtain a question generation model; A question generation module is used to generate government affairs questions using a question generation model; The model training module is further used to: perform word segmentation processing on the government data to obtain an input sequence, wherein the input sequence includes multiple words; determine the representation joint word vector, position vector and text segment information of the input sequence; input the representation joint word vector, position vector and text segment information of the input sequence into a pre-established UNILM network model, and output a text vector representation, wherein the UNILM network model includes a multi-layer Transformer network, and in the multi-layer Transformer network, each layer of the Transformer network includes: a unidirectional language network model, a bidirectional language network model and an end-to-end language network model; The model training module is further used to: after obtaining the input sequence, mask the input sequence using a self-attention mask matrix; determine the representation joint word vector, position vector and text segment information of the input sequence, including: determine the representation joint word vector, position vector and text segment information of the input sequence after masking.

4. The government affairs question generating device according to claim 3, characterized in that: The model fine-tuning module is further used to: Determine a source sequence and a target sequence according to the open source question and answer data and the self-built government question and answer text data; Masking the target sequence using a self-attention mask matrix; The source sequence and the masked target sequence are fed into the pre-trained model for fine-tuning.

5. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 2 is implemented.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the method according to any one of claims 1 to 2.

Images