Text generation method, question and answer system and related apparatus

Abstract

The application discloses a text generation method, a question and answer system and related devices, and relates to the technical field of computers. The method comprises the following steps: obtaining text input data, context data, domain-specific prompts, context-specific prompts and task-specific prompts; using a prompt generator in a question and answer system to generate a first comprehensive vector according to the context data and the domain-specific prompts and the context-specific prompts; and using a generative question and answer model to analyze and predict the text input data and the context data in combination with the first comprehensive vector and the task-specific prompts to obtain corresponding generated text. Different fine-grained multi-level prompts provided by the domain-specific prompts, the context-specific prompts and the task-specific prompts are generated corresponding to different text input data, the semantic understanding of the input text by the question and answer system is improved from different angles, and the accuracy of the generated text is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a text generation method, a question and answer system and related devices. BACKGROUND

[0002] With the rapid development of computer technology, the field of artificial intelligence has begun to attract widespread attention, and people's demand for machines to understand and answer natural language questions has gradually increased. The proposal of the machine reading comprehension task has promoted the research of the question and answer model, which requires the machine to generate answer text from the input question text.

[0003] Prompt learning fixes the parameters of the pre-trained model, trains a prompt vector, and applies the pre-trained model to different downstream tasks, which has the advantages of flexibility and efficiency. However, the prompt vector of the prompt learning in the related technology is irrelevant to the input, and a uniform prompt vector is used for all input texts, which is not optimized for the question and answer task, resulting in a lack of semantic understanding of the input text by the question and answer system, and further resulting in the generated text being difficult to meet the accuracy requirements. SUMMARY

[0004] The present application aims to at least solve one of the technical problems existing in the prior art. To this end, the embodiments of the present application provide a text generation method, a question and answer system and related devices, which contain multiple levels of prompts of different fine-grained, improve the semantic understanding of the input text by the question and answer system from different angles, and thus improve the accuracy of the generated text.

[0005] In a first aspect, the embodiments of the present application provide a text generation method applied to a question and answer system, the question and answer system comprising a prompt generator and a generative question and answer model, the prompt generator comprising a first encoder, a first decoder and a first multi-layer perception, and the generative question and answer model comprising a second encoder, a second decoder, a second multi-layer perception and a third multi-layer perception, the method comprising:

[0006] obtaining text input data, context data of the text input data, domain-specific prompts, context-specific prompts and task-specific prompts;

[0007] inputting the context data into the first encoder to obtain first encoding output data, and inputting the first encoding output data, the domain-specific prompts and the context-specific prompts into the first decoder to generate a first comprehensive prompt;

[0008] perceiving the first comprehensive prompt using the first multi-layer perception to obtain a first comprehensive vector, and perceiving the task-specific prompt using the second multi-layer perception to obtain a first task vector;

[0009] concatenate the first comprehensive vector and the first task vector to obtain a second comprehensive vector, and input the second comprehensive vector, the text input data and the context data into the second encoder to obtain second encoding output data;

[0010] perceive the task-specific vector by using the third multi-layer perception machine to obtain a second task vector, and input the second task vector and the second encoding output data into the second decoder to obtain generated text of the text input data.

[0011] In some embodiments of the present application, before the first encoding output data, the field-specific prompt and the context-specific prompt are input into the first decoder, the method further comprises:

[0012] obtaining a question and answer data set; the question and answer data set comprises context information of multiple fields;

[0013] clustering the context information based on field keywords to obtain at least one clustering result;

[0014] obtaining a field according to the clustering result, and generating the field-specific prompt of each field.

[0015] In some embodiments of the present application, after the field-specific prompt of each field is generated, the method further comprises:

[0016] selecting any two fields to form a field pair; the field pair comprises a first field and a second field;

[0017] calculating a field parameter of the field pair, a first field parameter of the first field and a second field parameter of the second field based on the length of the field-specific prompt;

[0018] calculating an independence parameter of the field pair according to the field parameter, the first field parameter and the second field parameter;

[0019] if the independence parameter does not satisfy a preset threshold, re-clustering the context information based on field keywords.

[0020] In some embodiments of the present application, the first encoding output data, the field-specific prompt and the context-specific prompt are input into the first decoder to generate a first comprehensive prompt, comprising:

[0021] concatenating the field-specific prompt and the context-specific prompt to obtain a concatenated prompt;

[0022] inputting the first encoding output data and the concatenated prompt into the first decoder for decoding to generate the first comprehensive prompt.

[0023] In some embodiments of the present application, the perception of the first comprehensive prompt by the first multi-layer perception machine comprises:

[0024] Obtaining the number of network layers of the second encoder;

[0025] Inputting the first comprehensive vector into the first multi-layer perception machine to sequentially generate first comprehensive key information corresponding to the number of network layers;

[0026] Obtaining first comprehensive value information of the first comprehensive vector according to the first comprehensive key information;

[0027] Obtaining the first comprehensive vector according to the first comprehensive key information and the first comprehensive value information, wherein the first comprehensive vector is represented in the form of key-value pairs.

[0028] In some embodiments of the present application, the first task vector comprises first task key information and first task value information; and the splicing of the first comprehensive vector and the first task vector to obtain a second comprehensive vector comprises:

[0029] Obtaining a first attention vector of a first attention layer of the second encoder, wherein the first attention vector comprises first key information and first value information;

[0030] Splicing the first task key information, the first comprehensive key information and the first key information to obtain second comprehensive key information of a second comprehensive vector;

[0031] Splicing the first task value information, the first comprehensive value information and the first value information to obtain second comprehensive value information of the second comprehensive vector;

[0032] Obtaining the second comprehensive vector according to the second comprehensive key information and the second comprehensive value information.

[0033] In some embodiments of the present application, the second task vector comprises second task key information and second task value information; and the inputting of the second task vector and the second encoding output data into the second decoder to obtain the generated text of the text input data comprises:

[0034] Obtaining a second attention vector of a second attention layer of the second decoder, wherein the second attention vector comprises second key information and second value information;

[0035] Splicing the second task key information and the second key information to obtain third comprehensive key information;

[0036] Splicing the second task value information and the second value information to obtain third comprehensive value information;

[0037] generating text of the text input data according to the third comprehensive key information, the third comprehensive value information and the second encoding output data.

[0038] In some embodiments of the present application, the question and answer system further comprises a training process, and the training process comprises:

[0039] selecting a preset number of training field pairs, and calculating an independence parameter of the training field pairs;

[0040] calculating an independence constraint loss value according to the independence parameter;

[0041] obtaining a text loss value of a predicted generated text and a text label;

[0042] obtaining a preset loss value according to the independence constraint loss value and the text loss value;

[0043] updating the first multi-layer perception, the second multi-layer perception, the third multi-layer perception, the task-specific prompt, the field-specific prompt and the context-specific prompt according to the preset loss value.

[0044] In a second aspect, the embodiments of the present application also provide a text generation device, which applies the text generation method as described in the first aspect of the present application, and comprises:

[0045] an obtaining module, configured to obtain text input data, context data of the text input data, a field-specific prompt, a context-specific prompt and a task-specific prompt;

[0046] a first input module, configured to input the context data to the first encoder to obtain first encoding output data, and input the first encoding output data, the field-specific prompt and the context-specific prompt to the first decoder to generate a first comprehensive prompt;

[0047] a perception module, configured to perceive the first comprehensive prompt by using the first multi-layer perception to obtain a first comprehensive vector, and perceive the task-specific prompt by using the second multi-layer perception to obtain a first task vector;

[0048] a second input module, configured to splice the first comprehensive vector and the first task vector to obtain a second comprehensive vector, and input the second comprehensive vector, the text input data and the context data to the second encoder to obtain second encoding output data;

[0049] The generating module is configured to perceive the task-specific vector by using the third multi-layer perception machine to obtain a second task vector, and input the second task vector and the second encoding output data into the second decoder to obtain the generated text of the text input data.

[0050] In a third aspect, the embodiments of the present application further provide a question and answer system, which applies the text generation method as described in the first aspect of the present application, and applies the text generation method as described in the first aspect of the present application, comprising:

[0051] The prompt generator is configured to generate the first comprehensive prompt according to the context data, the domain-specific prompt and the context-specific prompt; wherein the prompt generator comprises a first encoder, a first decoder and a first multi-layer perception machine, the first encoder is connected with the first decoder, and the first decoder is connected with the first multi-layer perception machine.

[0052] The generative question and answer model is configured to obtain the generated text according to the text input data, the context data, the first comprehensive prompt and the task-specific prompt; wherein the generative question and answer model comprises a second encoder, a second decoder, a second multi-layer perception machine and a third multi-layer perception machine; the second encoder is connected with the first multi-layer perception machine and the second multi-layer perception machine, and the second decoder is connected with the second encoder and the third multi-layer perception machine.

[0053] In a fourth aspect, the embodiments of the present application further provide an electronic device, comprising a memory and a processor, the memory stores a computer program, and the processor implements the text generation method as described in the first aspect of the present application when executing the computer program.

[0054] In a fifth aspect, the embodiments of the present application further provide a computer readable storage medium, which stores a program, and the program is executed by a processor to implement the text generation method as described in the first aspect of the present application.

[0055] The embodiments of the present application at least have the following beneficial effects:

[0056] The embodiment of the application provides a text generation method, a question and answer system and related devices, wherein the question and answer system comprises a prompt generator and a generative question and answer model, the prompt generator comprises a first encoder, a first decoder and a first multi-layer perception, and the generative question and answer model comprises a second encoder, a second decoder, a second multi-layer perception and a third multi-layer perception. The text generation method comprises the following steps: obtaining text input data, context data of the text input data, a domain-specific prompt, a context-specific prompt and a task-specific prompt; inputting the context data into the first encoder to obtain first encoding output data; inputting the first encoding output data, the domain-specific prompt and the context-specific prompt into the first decoder to generate a first comprehensive prompt; perceiving the first comprehensive prompt by using the first multi-layer perception to obtain a first comprehensive vector, and perceiving the task-specific prompt by using the second multi-layer perception to obtain a first task vector; splicing the first comprehensive vector and the first task vector to obtain a second comprehensive vector; inputting the second comprehensive vector, the text input data and the context data into the second encoder to obtain second encoding output data; and perceiving the task-specific vector by using the third multi-layer perception to obtain a second task vector, and inputting the second task vector and the second encoding output data into the second decoder to obtain generated text of the text input data. Therefore, different fine-grained multi-level prompts provided by the domain-specific prompt, the context-specific prompt and the task-specific prompt are generated corresponding to different text input data, the semantic understanding of the input text by the question and answer system is improved from different angles, and the accuracy of the generated text is improved.

[0057] Additional aspects and advantages of the application will be made apparent by the following description and the appended claims. BRIEF DESCRIPTION OF DRAWINGS

[0058] The above and / or additional aspects and advantages of the application will become apparent and be readily appreciated from the following description, including the appended claims, taken in conjunction with the accompanying drawings.

[0059] Figure 1 is a question and answer system architecture schematic diagram provided by an embodiment of the application;

[0060] Figure 2 is a flowchart schematic diagram of a text generation method provided by an embodiment of the application;

[0061] Figure 3 is Figure 2 is a flowchart schematic diagram before step S102 in the method;

[0062] Figure 4 is Figure 3 is a flowchart schematic diagram after step S203 in the method;

[0063] Figure 5Yes Figure 2 a flowchart of the process in step S102;

[0064] Figure 6 Yes Figure 2 a flowchart of the process in step S103;

[0065] Figure 7 Yes Figure 2 a flowchart of the process in step S104;

[0066] Figure 8 Yes Figure 2 a flowchart of the process in step S105;

[0067] Figure 9 is a flowchart of the text generation method provided by another embodiment of the present application;

[0068] Figure 10 is a schematic diagram of a text generation device module provided by an embodiment of the present application;

[0069] Figure 11 is a structural schematic diagram of an electronic device provided by an embodiment of the present application.

[0070] Reference signs: acquisition module 100, first input module 200, perception module 300, second input module 400, generation module 500, question and answer system 1000, prompt generator 1100, first encoder 1101, first decoder 1102, first multi-layer perception machine 1103, generative question and answer model 1200, second encoder 1201, second decoder 1202, second multi-layer perception 1203, third multi-layer perception machine 1204, first electronic device 2000, processor 2001, memory 2002. DETAILED DESCRIPTION

[0071] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and cannot be used to limit the present application.

[0072] The embodiments of the present application will be described in detail below, and examples of the embodiments are shown in the drawings, wherein the same or similar reference signs represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0073] In the description of the present application, it should be understood that the orientation description, such as the orientation or position relationship indicated by the upper, lower, front, rear, left, right and the like, is based on the orientation or position relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the present application.

[0074] In the description of the present application, the meaning of several is one or more, the meaning of multiple is more than two, greater than, less than, more than, etc. are understood as not including the number, above, below, etc. are understood as including the number. If the first, second is described, it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features or the order of indicated technical features.

[0075] In the description of the present application, unless otherwise explicitly limited, the words such as setting, installing, connecting and the like should be broadly understood, and the person skilled in the art can reasonably determine the specific meaning of the above words in the present application in combination with the specific content of the technical scheme.

[0076] With the rapid development of computer technology, the field of artificial intelligence has begun to attract widespread attention, and people's demand for machines to understand and answer natural language questions has gradually increased. The proposal of the machine reading comprehension task has promoted the research of the question and answer model, which requires the machine to generate answer text from the input question text.

[0077] Prompt learning fixes the parameters of the pre-trained model, trains a prompt vector, and applies the pre-trained model to different downstream tasks, which has the advantages of flexibility and efficiency. However, the prompt learning in the related art is mainly applied in the field of recommendation system, and the prompt method specially designed for the question and answer system is relatively few. Because in the recommendation task, the structure of input and output is relatively simple, and the model is easier to understand and process. However, in the question and answer task, the relationship between the question and the answer is more complex, and requires more in-depth semantic understanding and reasoning ability.

[0078] The prompt vector in the related technology is mostly irrelevant to the input, that is, the same prompt vector is used for all input texts of the same data set, and the prompt is not optimized separately for the question and answer task, which limits the expression ability of the prompt, and thus the question and answer system lacks semantic understanding of the input text, and further leads to difficulty in meeting the rate requirements of the generated text. In addition, the related technology mainly realizes question and answer by fine-tuning the pre-trained language model as a question and answer system on the question and answer task. With the continuous increase of the parameter scale of the pre-trained language model, a large amount of computing resources (such as GPUs) are required for fine-tuning, and the fine-tuning method needs to store a complete backup for each data set, which puts high requirements on the storage of the hard disk, resulting in an increase in storage costs.

[0079] Based on this, the embodiment of the application provides a text generation method, a question and answer system and related devices, which can obtain text input data, context data of the text input data, a domain-specific prompt, a context-specific prompt and a task-specific prompt, then input the context data to a first encoder to obtain first encoding output data, and input the first encoding output data, the domain-specific prompt and the context-specific prompt to a first decoder to generate a first comprehensive prompt. A first multi-layer perceptron is used to perceive the first comprehensive prompt to obtain a first comprehensive vector, and a second multi-layer perceptron is used to perceive the task-specific prompt to obtain a first task vector. Then, the first comprehensive vector and the first task vector are spliced to obtain a second comprehensive vector, and the second comprehensive vector, the text input data and the context data are input to a second encoder to obtain second encoding output data. Finally, a third multi-layer perceptron is used to perceive the task-specific vector to obtain a second task vector, and the second task vector and the second encoding output data are input to a second decoder to obtain the generated text of the text input data. Thus, different fine-grained multi-level prompts provided by the domain-specific prompt, the context-specific prompt and the task-specific prompt correspond to different prompts with different text input data, which improves the semantic understanding of the input text of the question and answer system from different angles and improves the accuracy of the generated text.

[0080] The embodiment of the application provides a text generation method, a question and answer system and related devices, which are specifically described by the following embodiments. First, the question and answer system in the embodiment of the application is described.

[0081] Reference Figure 1As shown in the question and answer system architecture diagram, the question and answer system 1000 includes a prompt generator 1100 and a generative question and answer model 1200, wherein the prompt generator 1100 is configured to generate a first comprehensive prompt according to context data, a domain-specific prompt and a context-specific prompt, and the generative question and answer model 1200 is configured to obtain generated text according to text input data, context data, the first comprehensive prompt and a task-specific prompt. Specifically, the text input data is a question text input by a user into the question and answer system, the context data is background information, a previous relationship and other related data of the text input data, and the domain-specific prompt, the context-specific prompt and the task-specific prompt are initial prompt data corresponding to the text input data and / or the context data. Thus, the semantic understanding of the question and answer system for the text input data and the context data is improved from different angles according to the multi-level prompts.

[0082] In some embodiments of the present application, the prompt generator 1100 in the question and answer system 1000 further includes a first encoder 1101, a first decoder 1102 and a first multi-layer perception 1103. It can be understood that the first encoder 1101 and the first decoder 1102 correspond to a pre-trained small language model, such as a UnifiedQA-small model or other encoder-decoder models. The generative question and answer model 1200 in the question and answer system 1000 further includes a second encoder 1201, a second decoder 1202, a second multi-layer perception 1203 and a third multi-layer perception 1204. The second encoder 1201 and the second decoder 1202 correspond to a pre-trained large language model. The large language model is a language model with a large number of parameters, which is pre-trained on multiple language tasks, such as ChatGPT, Bloom, etc., and the present application does not limit this.

[0083] Specifically, the first encoder 1101 is connected with the first decoder 1102, and the first decoder 1102 is connected with the first multi-layer perception machine 1103. The second encoder 1201 is connected with the first multi-layer perception machine 1103 and the second multi-layer perception machine 1203, and the second decoder 1202 is connected with the second encoder 1201 and the third multi-layer perception machine 1204. In the prompt generator 1100, the context data is first input to the first encoder 1101 for encoding, and then decoded by the first decoder 1102, and the context-specific prompt and the domain-specific prompt are input to the first decoder 1102, and finally the first comprehensive prompt is generated and perceived by the first multi-layer perception machine 1103 to obtain the corresponding first comprehensive vector. In the generative question and answer model 1200, the first comprehensive vector obtained by the first multi-layer perception machine 1103 in the prompt generator 1100 is first received, the first task vector obtained by perceiving the task-specific prompt by the second multi-layer perception machine 1203 is spliced, the second comprehensive vector obtained by splicing, the text input data and the context data are input to the second encoder 1201 for encoding, and the third multi-layer perception machine 1204 is used to perceive the task-specific prompt to obtain the second task vector, and finally the second decoder 1202 obtains the generated text of the text input data.

[0084] It should be noted that, Figure 1 The architecture of the question and answer system described in the embodiments of the present application is only for more clearly illustrating the technical solutions of the embodiments of the present application, and does not constitute a limitation on the technical solutions provided by the embodiments of the present application, which can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. Those skilled in the art can know that, as the architecture evolves and new application scenarios appear, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems. Figure 1 Figure 1 The components shown in the embodiments of the present application can be realized by hardware and / or software. Figure 1

[0085] It should be noted that the architecture of the question and answer system described in the embodiments of the present application is for more clearly illustrating the technical solutions of the embodiments of the present application, and does not constitute a limitation on the technical solutions provided by the embodiments of the present application, which can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. Those skilled in the art can know that, as the architecture evolves and new application scenarios appear, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0086] ​​The text generation method provided by the embodiments of the present application is described below, which relates to the technical field of computers, and in particular to the technical field of artificial intelligence and natural language processing. The text generation method provided by the embodiments of the present application can be applied in a terminal, can also be applied in a server, and can also be a computer program running in the terminal or the server. For example, the computer program can be a native program or a software module in an operating system; can be a local application program, that is, a program that needs to be installed in an operating system to run, such as a client supporting text generation, that is, a program that can run only by being downloaded into a browser environment. In summary, the above computer program can be any form of application program, module or plug-in. The terminal communicates with the server through a network. The text generation method can be executed by the terminal or the server, or cooperatively executed by the terminal and the server.

[0087] In some embodiments, the terminal can be a smartphone, a tablet computer, a notebook computer, a desktop computer, a smart watch or the like. The server can be a standalone server, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and basic cloud computing services such as big data and artificial intelligence platforms; or a service node in a blockchain system, the service nodes in the blockchain system form a peer-to-peer (P2P) network, and the P2P protocol is an application layer protocol running on the transmission control protocol (TCP) protocol. The server can install a server of a text generation system, which can interact with the terminal through the server, for example, the server installs corresponding software, which can be an application program for implementing the text generation method, but is not limited to the above forms. The terminal and the server can be connected through communication connection modes such as Bluetooth, universal serial bus (USB) or network, and the embodiments are not limited herein.

[0088] The application is operable in a variety of general purpose or special purpose computer systems environments or configurations. Examples of well known computing systems, environments, and / or configurations that can be suitable for use with the application include personal computers, server computers, handheld or laptop devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. The application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in local and remote computer storage media including memory storage devices.

[0089] A text generation method in an embodiment of the application is described below.

[0090] Figure 2 is an optional flowchart of a text generation method provided by an embodiment of the disclosure, Figure 2 The method in can include but is not limited to including steps S101 to S105. It can be understood that the embodiment does not make specific limitation on the order of steps S101 to S105 in Figure 2 The order of steps S101 to S105 in can be adjusted or some steps can be reduced, added or increased according to actual needs.

[0091] The text generation method of the embodiment is applied to Figure 1 The question-answering system 1000 shown in includes a prompt generator 1100 and a generative question-answering model 1200, the prompt generator 1100 includes a first encoder 1101, a first decoder 1102 and a first multi-layer perception 1103, and the generative question-answering model 1200 includes a second encoder 1201, a second decoder 1202, a second multi-layer perception 1203 and a third multi-layer perception 1204.

[0092] In step S101, text input data, context data of the text input data, domain-specific prompts, context-specific prompts and task-specific prompts are obtained.

[0093] It can be understood that the shareable prompt parameters learned from a specific task can effectively enhance the performance of the pre-trained language model in the downstream task. In some embodiments, the task-specific prompt contains knowledge information shared by all samples in this task, so as to well stimulate the performance of the large language model.

[0094] In a question-answering scenario, especially in machine reading comprehension, the context contains the answer or evidence supporting the answer. The context in the question-answering dataset can usually be divided into several fields. For example, in the NewsQA dataset, the context can be divided into political, economic, social, and other fields. In some embodiments, in order to improve the semantic understanding of the context, different field-specific prompts are used for contexts from different fields.

[0095] It should be understood that the field-specific prompt includes specific shared information within the field, and has more fine-grained knowledge compared with the task-specific prompt. However, even in the same field, there will still be differences between contexts, so it is important to utilize these different information to accurately answer the question. In some embodiments, the understanding of each context is enhanced by constructing context-specific prompts, which have more fine-grained knowledge compared with the field-specific prompt.

[0096] In some embodiments, text input data, context data of the text input data, field-specific prompts, context-specific prompts, and task-specific prompts are obtained. The text input data is the question text input by the user to the question-answering system, and the context data is the background information, the relationship before and after, and other related data of the text input data. The field-specific prompt, the context-specific prompt, and the task-specific prompt are initial prompt data corresponding to the text input data and / or the context data. Thus, the semantic understanding of the text input data and the context data by the question-answering system is improved from different angles according to the multi-level prompts of different fine-grained.

[0097] In step S102, the context data is input to the first encoder to obtain first encoding output data, and the first encoding output data, the field-specific prompt, and the context-specific prompt are input to the first decoder to generate a first comprehensive prompt.

[0098] In some embodiments, after obtaining the context data, the context data is input to the first encoder 1101 in the prompt generator 1100 for encoding to obtain first encoding output data. Then, the obtained field-specific prompt and context-specific prompt are input to the first decoder 1102 together with the first encoding output data for decoding to obtain a first comprehensive prompt.

[0099] In some embodiments, the domain-specific prompt and the context-specific prompt are initial prompt data, which can be initialized according to the context data and a pre-trained vocabulary. It can be understood that the pre-trained vocabulary is a table containing a large number of words and their vector representations, which can be trained on a large corpus and capture the semantic and grammatical relationships between words. For example, according to the domain to which the context data belongs, the pre-trained vocabulary is searched to find the relevant words and vector representations of the domain-specific prompt. Therefore, the domain-specific prompt and the context-specific prompt are both input into the prompt generator 1100 in the form of vector representations. Specifically, the domain-specific prompt and the context-specific prompt are decoded by the first decoder 1102 in combination with the first encoding output data to learn the knowledge and connections therein, and finally generate the first comprehensive prompt corresponding to the context data.

[0100] In step S103, the first comprehensive vector is obtained by using the first multi-layer perceptron to perceive the first comprehensive prompt, and the first task vector is obtained by using the second multi-layer perceptron to perceive the task-specific prompt.

[0101] The multi-layer perceptron (MLP) is a kind of feedforward neural network model. It can be understood that through the nonlinear transformation of multiple hidden layers, the multi-layer perceptron can learn the complex features of the input data and map them to the output space.

[0102] In some embodiments, the first comprehensive vector is obtained by using the first multi-layer perceptron 1103 to perceive the first comprehensive prompt, i.e., the low-dimensional vector of the first comprehensive prompt is input into the first multi-layer perceptron 1103 for mapping to obtain the preset high-dimensional vector of the corresponding first comprehensive vector. At the same time, the first task vector is obtained by using the second multi-layer perceptron 1203 to perceive the task-specific prompt, i.e., the low-dimensional vector of the task-specific prompt is input into the second multi-layer perceptron 1203 for mapping to obtain the preset high-dimensional vector of the corresponding first task vector. In this way, the mapping of the features is completed, which meets the form of inputting into the second encoder 1201 and effectively reduces the amount of parameters to be optimized.

[0103] In step S104, the first comprehensive vector and the first task vector are spliced to obtain a second comprehensive vector, and the second comprehensive vector, the text input data and the context data are input into the second encoder to obtain second encoding output data.

[0104] In some embodiments, the question and answer system 1000 analyzes and predicts the generated text from the user input question text. First, the obtained text input data and its corresponding context data are input into the second encoder 1201 of the generative question and answer model 1200, while the first comprehensive vector and the first task vector are spliced to obtain the second comprehensive vector and input into the second encoder 1201. The second encoder 1201 encodes the input data to obtain the second encoding output data.

[0105] In step S105, the third multi-layer perception machine is used to perceive the task-specific vector to obtain a second task vector, and the second task vector and the second encoding output data are input into the second decoder to obtain the generated text of the text input data.

[0106] In some embodiments, the third multi-layer perception machine 1204 is used to perceive the task-specific vector to obtain a second task vector, that is, the low-dimensional vector of the task-specific prompt is input into the third multi-layer perception machine 1204 for mapping to obtain a preset high-dimensional vector of the corresponding second task vector. Thus, the mapping of the features is completed, and the second task vector in the form of the high-dimensional vector meets the form of inputting into the second decoder 1202.

[0107] In some embodiments, the second task vector and the second encoding output data are input into the second decoder 1202, and the generated text of the text input data is obtained after decoding, that is, the answer text corresponding to the question text is generated. Thus, according to the different fine-grained multi-level prompts provided by the domain-specific prompt, the context-specific prompt and the task-specific prompt, different prompts are generated corresponding to different text input data, which improves the semantic understanding of the input text from different angles, thereby improving the accuracy of the generated text.

[0108] Referring to Figure 3 In some embodiments of the present application, the above step S102 can further include, but is not limited to, the following steps S201 to S203.

[0109] In step S201, a question and answer data set is obtained; the question and answer data set includes context information of multiple domains.

[0110] In some embodiments, the question and answer data set is obtained, and the question and answer data set includes context information of multiple domains. It can be understood that different domains such as politics, economy and society can be correspondingly divided according to different context information. In order to improve the semantic understanding of the context, different domain-specific prompts can be used for the context from different domains.

[0111] In step S202, the context information is clustered based on domain keywords to obtain at least one clustering result.

[0112] Since most of the question and answer datasets do not have explicit information about the domain, that is, the context information in the question and answer dataset does not show the domain to which each context information belongs. In some embodiments, an unsupervised clustering algorithm can be used to cluster the context information based on domain keywords, thereby obtaining at least one clustering result. Suppose we have a set of context information, which is related to the movie review domain. We want to cluster these context information according to domain keywords.

[0113] For example, when the domain keyword is related to the movie review domain, it includes but is not limited to "movie", "plot", "actor", "director", "rating", etc. For each context information, use text processing techniques (such as word segmentation, word vector representation, etc.) to convert it into a vector representation. A pre-trained word vector model such as Word2Vec or GloVe can be used to obtain the vector representation of each keyword. Then, for the vector representation of each context information, use an unsupervised clustering algorithm such as K-means, etc. In the clustering process, the keyword vector is used as a reference, so that the clustering result is more domain-related. Thus, clustering based on domain keywords obtains at least one clustering result, avoiding additional labeling costs, and having better generalization ability and adaptability.

[0114] Step S203, obtaining the domain according to the clustering result, and generating domain-specific prompts for each domain.

[0115] In some embodiments, different clustering clusters can be obtained according to the results of the clustering algorithm, and each clustering cluster represents a group of context information with similar characteristics, so that each clustering cluster is named as the corresponding domain, and the information in the domain is encapsulated to generate the corresponding domain-specific prompt. It can be understood that the results of the clustering algorithm can be affected by the selection of keywords, vector representation and clustering algorithm. Those skilled in the art can try different parameters and methods according to actual needs to obtain better clustering results, and this embodiment does not limit this.

[0116] Referring to Figure 4 In some embodiments of the present application, the above step S203 can further include but is not limited to the following steps S301 to S304.

[0117] Step S301, selecting any two domains to form a domain pair; the domain pair includes a first domain and a second domain.

[0118] In some embodiments, different domains D = {D1,..., Dn} can be obtained by a clustering algorithm. n n represents the number of domains. Each domain has shared prompts within the domain, so the domain-specific prompt corresponds to D i represents the domain Di a shared prompt within, ρ represents the length of the domain-specific prompt, d p represents the embedding length of the pre-trained small language model in the prompt generator 1100, i.e., the embedding length of the first encoder 1101 and the first decoder 1102.

[0119] It can be understood that focusing on the knowledge of each domain can effectively enhance the semantic understanding of the language model to the context. In some embodiments, in order to guide each domain-specific prompt focus on the shared information unique to the domain D i , the learning process of the independence constraint prompt is used. Specifically, any two domains are selected to form a domain pair, which includes a first domain and a second domain.

[0120] In step S302, the domain parameters of the domain pair, the first domain parameters of the first domain and the second domain parameters of the second domain are calculated based on the length of the domain-specific prompt.

[0121] In some embodiments, the domain parameters of the domain pair, the first domain parameters of the first domain and the second domain parameters of the second domain are calculated based on the length of the domain-specific prompt. For example, the first domain selected is D a , the second domain is D b , the domain pair is (D a , D b ), and the domain parameters of the domain pair are calculated based on the corresponding domain-specific prompts and and their lengths:

[0122]

[0123] wherein, ρ represents the length of the domain-specific prompt, H represents the center matrix, 1 is the unit matrix, φ and ψ represent the kernel function, and tr represents the trace operation, which represents the sum of the elements along the main diagonal of a square matrix.

[0124] Similarly, the first domain parameters of the first domain are calculated:

[0125]

[0126] wherein,

[0127] Similarly, the second domain parameters of the second domain are calculated:

[0128]

[0129] wherein,

[0130] Step S303, calculating the independence parameter of the field pair according to the field parameter, the first field parameter and the second field parameter.

[0131] In some embodiments, by calculating the independence between different field pairs, the guidance field-specific prompt can be constrained to pay more attention to the shared information specific to the field rather than the information between fields. Specifically, the independence parameter of the field pair is calculated according to the field parameter, the first field parameter and the second field parameter:

[0132]

[0133] Wherein, when CKA=0 represents the first field D a and the second field D b are independent.

[0134] It can be understood that the value range of CKA is 0 to 1, where 0 represents that the two feature sets are independent, and 1 represents that the two feature sets are completely the same. When the value of CKA is 0, it means that the two feature sets are linearly independent, that is, there is no correlation between them, nor any shared information or important statistical relationship.

[0135] Step S304, if the independence parameter does not satisfy the preset threshold, re-clustering the context information based on the field keywords.

[0136] It can be understood that when there are n fields, it is necessary to calculate the independence parameter of the field pair composed of any two fields times of iteration, which requires a large amount of time cost when n is large. In some embodiments, by randomly sampling m field pairs as Θ, the independence constraint loss value of the independence parameter in each iteration is calculated:

[0137]

[0138] In some embodiments, when the independence parameter does not satisfy the preset threshold, for example, the independence constraint loss value calculated according to the independence parameter is greater than the preset threshold, it means that the independence between different fields does not meet the requirements. Re-clustering the context information based on the field keywords, those skilled in the art can try different parameters and methods according to actual needs to obtain better clustering results, and the present embodiment does not limit this.

[0139] Referring to FIG. 4, Figure 5 In some embodiments of the present application, the above step S102 can include but is not limited to the following steps S401 to S402.

[0140] Step S401, splicing the field-specific prompt and the context-specific prompt to obtain a spliced prompt.

[0141] In some embodiments, a concatenated hint is obtained by concatenating domain-specific hints and context-specific hints. Specifically, for context data c i Assuming it belongs to domain D j The first encoder 1101 of the prompt generator 1100 will use context data c i As input, and corresponding to the context data c i Domain-specific hints received and context-specific hints Then, concatenation is performed to obtain the concatenated vector. Right now in, ρ represents the length of the domain-specific hint, k represents the length of the context-specific hint, and d p This indicates the embedding length of the small language model pre-trained in the prompt generator 1100.

[0142] Step S402: Input the first encoded output data and the concatenation prompt into the first decoder for decoding to generate the first comprehensive prompt.

[0143] In some embodiments, the first encoded output data and the concatenation prompt are input to the first decoder 1102 for decoding to generate the first comprehensive prompt. It is understood that this embodiment changes the original input format of the first decoder 1102 by adding the concatenation prompt as input. Therefore, the original decoder embedding layer of the pre-trained small language model can be deleted, and this embodiment does not impose any restrictions on this.

[0144] Reference Figure 6 As shown, in some embodiments of this application, step S103 may include, but is not limited to, steps S501 to S504.

[0145] Step S501: Obtain the number of network layers of the second encoder.

[0146] In some embodiments, the generative question-answering model 1200 includes a multi-layer network, such as 12 or 24 layers, and the number of network layers in the second encoder 1201 is the same as the number of network layers in the second decoder 1202. The number of network layers in the second encoder 1201 is obtained, which is equivalent to obtaining the number of network layers in the generative question-answering model 1200, and is denoted as L.

[0147] Step S502: Input the first synthesis vector into the first multilayer perceptron to generate the first synthesis key information corresponding to the network layer number in sequence.

[0148] In some embodiments, the first comprehensive vector is input into the first multi-layer perceptron 1103, and first comprehensive key information corresponding to the number of network layers is sequentially generated, starting from the first layer network, the first multi-layer perceptron generates the first comprehensive key information of the first layer network, and so on until the first comprehensive key information of the Lth layer is generated. Specifically, the first comprehensive key information of the lth layer network can be represented as ρ represents the length of the domain-specific prompt, k represents the length of the context-specific prompt, and d represents the embedding length of the pre-trained large language model in the generative question and answer model 1200, which can be understood as the embedding length of the second encoder 1201.

[0149] In step S503, the first comprehensive value information of the first comprehensive vector is obtained according to the first comprehensive key information.

[0150] In some embodiments, the first comprehensive value information of the first comprehensive vector can be obtained according to the first comprehensive key information. Specifically, the first comprehensive value information of the lth layer network can be represented as

[0151] In step S504, the first comprehensive vector is obtained according to the first comprehensive key information and the first comprehensive value information, and the first comprehensive vector is represented in the form of key-value pairs.

[0152] In some embodiments, according to the first comprehensive key information of the lth layer network and the first comprehensive value information the first comprehensive vector of the lth layer network can be obtained, and the first comprehensive vector is represented in the form of key-value pairs. Thus, through the mapping of the first multi-layer perceptron 1103, the first comprehensive prompt of the low-dimensional vector can be mapped to the first comprehensive vector of the pre-set high-dimensional vector that meets the input form of each layer of the second encoder 1201.

[0153] It can be understood that the knowledge related to the context data is obtained from the first encoder 1101 in the prompt generator 1100 and then integrated into the prompt generation process in the first decoder 1102. In this way, the semantics between the prompt and the context can be better learned, because the domain-specific prompt and the context-specific prompt are closely related to the context data.

[0154] In some embodiments, the first task vector obtained by perceiving the task-specific prompt using the second multi-layer perceptron 1203 is also represented in the form of key-value pairs of the first task key information and the first task value information. Thus, the mapping of the features is completed, which meets the input form of the second encoder 1201, effectively reducing the amount of optimized parameters.

[0155] Referring to Figure 7As shown, in some embodiments of the present application, the step S104 can include but is not limited to the following steps S601 to S604.

[0156] Step S601, obtaining a first attention vector of a first attention layer of the second encoder.

[0157] It can be understood that there are usually different types of attention in the encoder-decoder architecture model, including self-attention in the encoder. Specifically, the self-attention in the encoder refers to obtaining the context information of each word by calculating the relevance between different positions of the words in the input sentence when encoding the sentence. Each self-attention layer has an original K vector and a V vector. In each self-attention layer, the input sequence will be linearly transformed to obtain corresponding query Q, key K and value V vectors. These vectors are calculated independently in each self-attention layer in order to perform attention calculation and context weighted summation operation.

[0158] In some embodiments, the first attention vector of the first attention layer of the second encoder 1201 is obtained, the first attention layer being a self-attention layer, and the corresponding first attention vector is obtained, wherein the first attention vector includes first key information and corresponding first value information. For example, for the first layer self-attention of the second encoder 1201, the first key information of the first attention vector is K l , and the first value information is V l ,

[0159] Step S602, concatenating the first task key information, the first comprehensive key information and the first key information to obtain the second comprehensive key information of the second comprehensive vector.

[0160] In some embodiments, the task-specific prompt is obtained by adding a trainable vector in front of different attention layers of the generative question and answer model 1200. Specifically, a prefix P∈R t×d is added in front of different attention layers, where t represents the length of the task-specific prompt, and d represents the embedding length of the generative question and answer model 1200. The task vector T is obtained by perceptual learning through a multi-layer perceptron, T = MLP(P), T = {T1, T2,..., TL}, where L is the number of network layers of the question and answer model, and T L = (T l , T l，K ), l，V T l，K represents the task key information, and T l，V represents the task value information.

[0161] ​In some embodiments, the task vector T is composed of a first task vector and a second task vector, the first task vector acting on the second encoder 1201, and the second task vector acting on the second decoder 1202. Specifically, a trainable vector P e R t×d The first task vector is trained and represented by the key-value pair form of the second task key information and the second task value information.

[0162] In some embodiments, the first attention layer of the l-th layer of the second encoder 1201 corresponds to the first task vector The first task key information and the corresponding first task value information Specifically, the first task key information The first comprehensive key information and the first key information K l are spliced to obtain the second comprehensive key information of the second comprehensive vector:

[0163]

[0164] Wherein, K l ' e R (t+ρ+k+M)×d , p represents the length of the domain-specific prompt, k represents the length of the context-specific prompt, t represents the length of the task-specific prompt, d represents the embedding length of the generative question and answer model, and M represents the length of the input sequence, i.e. the length of the text input data and the context data input into the second encoder 1201.

[0165] In step S603, the first task value information, the first comprehensive value information and the first value information are spliced to obtain the second comprehensive value information of the second comprehensive vector.

[0166] In some embodiments, corresponding to the second comprehensive key information, the first task value information The first comprehensive value information and the first value information V l are spliced to obtain the second comprehensive value information of the second comprehensive vector:

[0167]

[0168] Wherein, V l ' e R (t+ρ+k+M)×d , p represents the length of the domain-specific prompt, k represents the length of the context-specific prompt, t represents the length of the task-specific prompt, d represents the embedding length of the generative question and answer model, and M represents the length of the input sequence, i.e. the length of the text input data and the context data input into the second encoder 1201.

[0169] Step S604, obtaining a second comprehensive vector according to the second comprehensive key information and the second comprehensive value information.

[0170] In some embodiments, the second comprehensive vector is obtained according to the second comprehensive key information and the second comprehensive value information, so as to combine the text input data and the corresponding context data to be input into the second encoder 1201 for encoding. It can be understood that, The information containing the field-specific prompt and the context-specific prompt, and the knowledge from the pre-trained language model related to the context data, while T E Contain shared information within the task. In this way, different levels of prompts are integrated into the second encoder 1201 of the generative question and answer model 1200, effectively enhancing semantic understanding and improving question and answer performance and accuracy.

[0171] Referring to Figure 8 It is shown that in some embodiments of the present application, the above step S105 can include but is not limited to the following steps S701 to S704.

[0172] Step S701, obtaining a second attention vector of a second attention layer of the second decoder.

[0173] It can be understood that there are usually different types of attention in the encoder-decoder architecture model, which also includes self-attention and cross-attention in the decoder. Specifically, the self-attention in the decoder refers to the attention of the decoder to the part it has generated during the generation of the target sentence, and the next word to be generated is determined according to the generated part. The cross-attention in the decoder refers to the attention of the decoder to the input sentence in the encoder when generating the target sentence. The cross-attention mechanism enables the decoder to utilize the context information in the encoder, so as to better understand the input sentence and generate a target sentence related to the input sentence.

[0174] In some embodiments, the second attention layer of the second decoder 1202 is obtained, which specifically includes a self-attention layer and a cross-attention layer, and the corresponding second attention vector is obtained, wherein the second attention vector includes second key information and corresponding second value information. For example, for the first layer self-attention of the second decoder 1202, the second key information of the second attention vector is K l , and the second value information is V l , For example, for the first layer cross-attention of the second decoder 1202, the second key information of the second attention vector is K l , and the second value information is V l ,

[0175] Step S702, splicing the second task key information and the second key information to obtain third comprehensive key information.

[0176] In some embodiments, the task-specific prompt is trained by adding a trainable vector before the different attention of the generative question answering model 1200. Specifically, a trainable vector P ∈ R t×d The second task vector is trained, and the second task vector is also represented by the key-value pair form of the second task key information and the second task value information.

[0177] In some embodiments, the lth layer self-attention layer of the second decoder 1202 corresponds to the second task vector including the second task key information and the corresponding second task value information Specifically, the second task key information and the second key information K l are spliced to obtain third comprehensive key information:

[0178]

[0179] wherein K l ' ∈ R (t+M)×d , t represents the length of the task-specific prompt, d represents the embedding length of the generative question answering model, and M represents the length of the input sequence.

[0180] In some embodiments, the lth layer cross-attention layer of the second decoder 1202 corresponds to the second task vector including the second task key information and the corresponding second task value information Specifically, the second task key information and the second key information K l are spliced to obtain third comprehensive key information:

[0181]

[0182] wherein K l ' ∈ R (t+M)×d , t represents the length of the task-specific prompt, d represents the embedding length of the generative question answering model, and M represents the length of the input sequence.

[0183] Step S703, splicing the second task value information and the second value information to obtain third comprehensive value information.

[0184] In some embodiments, corresponding to the third comprehensive key information, the second task value information and the second value information V l are spliced to obtain third comprehensive value information:

[0185]

[0186] wherein, V l '∈R (t+M)×d , t represents the length of the task-specific prompt, d represents the embedding length of the generative question answering model, and M represents the length of the input sequence.

[0187] In some embodiments, the second task value information V and the second value information V l are spliced to obtain third comprehensive value information V

[0188]

[0189] wherein, V l '∈R (t+M)×d , t represents the length of the task-specific prompt, d represents the embedding length of the generative question answering model, and M represents the length of the input sequence.

[0190] Step S704, obtaining the generated text of the text input data according to the third comprehensive key information, the third comprehensive value information, and the second encoding output data.

[0191] In some embodiments, the second decoder 1202 is used to obtain the generated text of the text input data according to the third comprehensive key information, the third comprehensive value information, and the second encoding output data. Thus, different fine-grained multi-level prompts provided by the domain-specific prompt, the context-specific prompt, and the task-specific prompt are added in each layer of the generative question answering model 1200 corresponding to the prompts, which improves the semantic understanding of the input text by the question answering system 1000 from different angles, thereby improving the accuracy of the generated text.

[0192] Referring to Figure 9 In some embodiments of the present application, the question answering system 1000 further includes a training process, which can include but is not limited to the following steps S801 to S805.

[0193] Step S801, selecting a preset number of training domain pairs and calculating the independence parameter of the training domain pairs.

[0194] In some embodiments, after clustering the context information in the question answering data set using an unsupervised clustering algorithm, different training domains are obtained, and a preset number of training domain pairs are selected by random sampling, and the independence parameter of the training domain pairs is calculated. It can be understood that the calculation process of the independence parameter is the same as the above-mentioned embodiments, and this embodiment will not be repeated here.

[0195] Step S802, calculating the independence constraint loss value according to the independence parameter.

[0196] In some embodiments, the independence constraint loss value is used to measure the independence between different domains obtained by clustering, and the independence constraint loss value is calculated according to the independence parameter. Specifically, the independence parameter is calculated for the training domain pair randomly sampled, and the independence constraint loss value is obtained by accumulating each independence parameter

[0197] Step S803, obtaining the text loss value of the predicted generated text and the text label.

[0198] In some embodiments, the text loss value is used to measure the difference between the predicted generated text and the target sequence, and the text loss value is calculated by obtaining the predicted generated text and the text label. It can be understood that the predicted generated text is the generated text obtained by the question and answer system 1000 in the training process, and the text label is the target sequence in the training process, and the text loss value is calculated by using the cross entropy loss function

[0199]

[0200] wherein y t represents the t-th element of the target sequence, and x represents the predicted generated text.

[0201] Step S804, obtaining the preset loss value according to the independence constraint loss value and the text loss value.

[0202] In some embodiments, the preset loss value is calculated according to the independence constraint loss value and the text loss value

[0203]

[0204] wherein λ represents a hyperparameter, used to control the weight of the independence constraint loss value , which can be set by a person skilled in the art according to actual needs, and the present example does not limit this.

[0205] Step S805, updating the first multi-layer perceptron, the second multi-layer perceptron, the third multi-layer perceptron, the task-specific prompt, the domain-specific prompt and the context-specific prompt according to the preset loss value.

[0206] ​In some embodiments, the weights of the first multi-layer perception 1103, the second multi-layer perception 1203 and the third multi-layer perception 1204 are updated according to the preset loss value, and the task-specific prompt, the domain-specific prompt and the context-specific prompt are updated. It can be understood that the embodiments of the present application adopt the mode of prompt learning, that is, the overall parameters of the fixed pre-training language model are not updated, for example, the parameters of the first encoder 1101 and the first decoder 1102 in the prompt generator 1100 are frozen, and the parameters of the second encoding 1201 and the second decoder 1202 in the generative question and answer model 1200 are frozen. Only a part of small parameters such as the three prompt vectors and the multi-layer perception are updated, thereby reducing the demand for computing resources (such as GPU), and for different question and answer data sets, only the parameters of the prompt vector part need to be stored, thereby effectively reducing the computing demand and storage cost.

[0207] It can be seen that the embodiments of the present application propose multi-level prompts: task-specific prompts, domain-specific prompts and context-specific prompts. Since the domain-specific prompts and the context-specific prompts are highly related to the text input data and the context data thereof, different prompts are generated with different inputs, which can well stimulate the semantic understanding of the pre-training language model for the input text. At the same time, the task-specific prompts focus on the knowledge shared by the entire task, the domain-specific prompts introduce specific shared knowledge within each domain without additional labeling cost, and the context-specific prompts generate different prompts for different inputs and introduce the knowledge contained in different contexts. The three different levels of prompts provide different fine-grained input semantic information to solve the defect of using a unified prompt for all inputs in the related art, and improve the semantic understanding of the question and answer system 1000 from three different angles. In addition, the prompt generator 1100 can effectively enhance the understanding of the context data by introducing the knowledge related to the context data from other small pre-training language models into the prompt generation process, thereby improving the accuracy of the generated text.

[0208] The embodiments of the present application also provide a text generation device, which can implement the above-mentioned text generation method, and refer to Figure 10 In some embodiments of the present application, the text generation device comprises:

[0209] The acquisition module 100 is configured to acquire the text input data, the context data of the text input data, the domain-specific prompt, the context-specific prompt and the task-specific prompt.

[0210] The first input module 200 is configured to input the context data to the first encoder to obtain first encoding output data, and input the first encoding output data, the domain-specific prompt and the context-specific prompt to the first decoder to generate a first comprehensive prompt.

[0211] The perception module 300 is configured to perceive the first comprehensive prompt by using a first multi-layer perception machine to obtain a first comprehensive vector, and perceive the task-specific prompt by using a second multi-layer perception machine to obtain a first task vector.

[0212] The second input module 400 is configured to splice the first comprehensive vector and the first task vector to obtain a second comprehensive vector, and input the second comprehensive vector, the text input data and the context data into a second encoder to obtain second encoding output data.

[0213] The generation module 500 is configured to perceive the task-specific vector by using a third multi-layer perception machine to obtain a second task vector, and input the second task vector and the second encoding output data into a second decoder to obtain the generated text of the text input data.

[0214] The specific implementation of the text generation apparatus in the embodiment is basically the same as the specific implementation of the text generation method, and will not be repeated here.

[0215] Figure 11 An electronic device 2000 provided by the embodiment of the present application is shown. The electronic device 2000 includes a processor 2001, a memory 2002, and a computer program stored in the memory 2002 and executable on the processor 2001, and the computer program is configured to execute the text generation method described above when executed.

[0216] The processor 2001 and the memory 2002 can be connected by a bus or other means.

[0217] The memory 2002, as a non-transitory computer readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs, such as the text generation method described in the embodiments of the present application. The processor 2001 executes the non-transitory software programs and instructions stored in the memory 2002, thereby implementing the text generation method described above.

[0218] The memory 2002 can include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required by a function; the data storage area can store the text generation method described above. In addition, the memory 2002 can include a high-speed random access memory 2002, and can also include a non-transitory memory 2002, such as at least one storage device memory device, a flash memory device or other non-transitory solid-state memory device. In some embodiments, the memory 2002 can optionally include a memory 2002 remotely arranged with respect to the processor 2001, and these remote memories 2002 can be connected to the electronic device 2000 through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0219] The non-transitory software programs and instructions required to implement the text generation method described above are stored in the memory 2002, and when executed by one or more processors 2001, perform the text generation method described above, for example, perform the method steps S101 to S105 in Figure 2 the method steps S201 to S203 in Figure 3 the method steps S301 to S304 in Figure 4 the method steps S401 to S402 in Figure 5 the method steps S501 to S504 in Figure 6 the method steps S601 to S604 in Figure 7 the method steps S701 to S704 in Figure 8 the method steps S801 to S805 in Figure 9

[0220] The embodiments of the present application also provide a storage medium, which is a computer readable storage medium, and the storage medium stores a computer program. The computer program is executed by a processor to implement the text generation method described above. The memory is a non-transitory computer readable storage medium, which can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged with respect to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0221] ​The text generation method, the question and answer system and the related device provided by the embodiments of the present application can obtain text input data, context data of the text input data, domain-specific prompts, context-specific prompts and task-specific prompts, then input the context data to a first encoder to obtain first encoding output data, and input the first encoding output data, the domain-specific prompts and the context-specific prompts to a first decoder to generate first comprehensive prompts. A first multi-layer perception machine is used to perceive the first comprehensive prompts to obtain a first comprehensive vector, and a second multi-layer perception machine is used to perceive the task-specific prompts to obtain a first task vector. Then the first comprehensive vector and the first task vector are spliced to obtain a second comprehensive vector, and the second comprehensive vector, the text input data and the context data are input to a second encoder to obtain second encoding output data. Finally, a third multi-layer perception machine is used to perceive the task-specific vector to obtain a second task vector, and the second task vector and the second encoding output data are input to a second decoder to obtain generated text of the text input data. Thus, different fine-grained multi-level prompts provided by the domain-specific prompts, the context-specific prompts and the task-specific prompts correspond to different prompts with different text input data, improve the semantic understanding of the input text of the question and answer system from different angles, and improve the accuracy of the generated text. Moreover, a prompt learning mode is adopted, that is, the overall parameters of the fixed pre-trained language model are not updated, only a part of small parameters such as the three prompt vectors and the multi-layer perception machine are updated, thereby reducing the demand for computing resources (such as GPU), and for different question and answer data sets, only the parameters of the prompt vector part need to be stored, thereby effectively reducing the computing demand and storage cost.

[0222] The above-described embodiments are merely illustrative, and units described as separate components can or can not be physically separate, that is, can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments.

[0223] Those of ordinary skill in the art will appreciate that all or certain steps, systems of the methods disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. Certain physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software can be distributed on computer readable media, which can include computer storage media (or non-transitory media) and communication media (or transitory media). As is well known to those of ordinary skill in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, storage devices storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer. Further, it should be appreciated by those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.

[0224] It should also be appreciated that various embodiments provided by the present application can be combined in any manner to achieve different technical effects. The above is a specific description of the preferred embodiments of the present application, but the present application is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application.

Claims

1. A text generation method, characterized in that, An application is made in a question-answering system, the question-answering system comprising: a prompt generator and a generative question-answering model, the prompt generator comprising: a first encoder, a first decoder, and a first multilayer perceptron, the generative question-answering model comprising: a second encoder, a second decoder, a second multilayer perceptron, and a third multilayer perceptron, the method comprising: Acquire text input data, context data of the text input data, domain-specific prompts, context-specific prompts, and task-specific prompts; The context data is input to the first encoder to obtain the first encoded output data, and the first encoded output data, domain-specific prompts and context-specific prompts are input to the first decoder to generate the first comprehensive prompt; The first comprehensive vector is obtained by perceiving the first comprehensive prompt using the first multilayer perceptron, and the first task vector is obtained by perceiving the task-specific prompt using the second multilayer perceptron. The first comprehensive vector and the first task vector are concatenated to obtain the second comprehensive vector, and the second comprehensive vector, the text input data, and the context data are input to the second encoder to obtain the second encoded output data; The third multilayer perceptron is used to perceive the task-specific vector to obtain the second task vector, and the second task vector and the second encoded output data are input to the second decoder to obtain the generated text of the text input data.

2. The text generation method according to claim 1, characterized in that, Before inputting the first encoded output data, domain-specific hints, and context-specific hints into the first decoder, the method further includes: Obtain the question-and-answer dataset; the question-and-answer dataset includes contextual information from multiple domains; The context information is clustered based on domain keywords to obtain at least one clustering result; Based on the clustering results, domains are obtained, and domain-specific hints are generated for each domain.

3. The text generation method according to claim 2, characterized in that, After generating the domain-specific hints for each of the domains, the method further includes: Select any two of the domains to form a domain pair; the domain pair includes a first domain and a second domain. Calculate the domain parameters of the domain pair, the first domain parameter of the first domain, and the second domain parameter of the second domain based on the length of the domain-specific prompt; The independence parameters of the domain pair are calculated based on the domain parameters, the first domain parameters, and the second domain parameters; If the independence parameter does not meet the preset threshold, the context information is re-clustered based on domain keywords.

4. The text generation method according to claim 1, characterized in that, The step of inputting the first encoded output data, domain-specific hints, and context-specific hints into the first decoder to generate a first comprehensive hint includes: The concatenated hint is obtained by combining the domain-specific hint and the context-specific hint; The first encoded output data and the concatenated prompt are input into the first decoder for decoding to generate the first comprehensive prompt.

5. The text generation method according to claim 4, characterized in that, The step of using the first multilayer perceptron to perceive the first integrated prompt to obtain the first integrated vector includes: Obtain the number of network layers of the second encoder; The first synthesis vector is input into the first multilayer perceptron to sequentially generate the first synthesis key information corresponding to the network layer number; The first comprehensive value information of the first comprehensive vector is obtained based on the first comprehensive key information; The first comprehensive vector is obtained based on the first comprehensive key information and the first comprehensive value information, and the first comprehensive vector is represented in the form of key-value pairs.

6. The text generation method according to claim 5, characterized in that, The first task vector includes first task key information and first task value information; the step of concatenating the first composite vector and the first task vector to obtain the second composite vector includes: Obtain the first attention vector of the first attention layer of the second encoder; the first attention vector includes first key information and first value information; The first task key information, the first comprehensive key information, and the first key information are concatenated to obtain the second comprehensive key information of the second comprehensive vector; The first task value information, the first comprehensive value information, and the first value information are concatenated to obtain the second comprehensive value information of the second comprehensive vector; The second comprehensive vector is obtained based on the second comprehensive key information and the second comprehensive value information.

7. The text generation method according to claim 5, characterized in that, The second task vector includes second task key information and second task value information; the step of inputting the second task vector and the second encoded output data into the second decoder to obtain the generated text of the text input data includes: Obtain the second attention vector of the second attention layer of the second decoder; the second attention vector includes second key information and second value information; The second task key information and the second key information are concatenated to obtain the third comprehensive key information; The second task value information and the second value information are concatenated to obtain the third comprehensive value information; The generated text of the text input data is obtained based on the third comprehensive key information, the third comprehensive value information, and the second encoded output data.

8. The text generation method according to any one of claims 1 to 7, characterized in that, The question-answering system also includes a training process, which includes: Select a preset number of training neighborhood pairs and calculate the independence parameter of the training neighborhood pairs; The independence constraint loss value is calculated based on the independence parameters. Obtain the text loss values ​​of the predicted generated text and text labels; Based on the independence constraint loss value and the text loss value, a preset loss value is obtained; The first multilayer perceptron, the second multilayer perceptron, the third multilayer perceptron, the task-specific prompt, the domain-specific prompt, and the context-specific prompt are updated based on the preset loss value.

9. A text generation device, characterized in that, The text generation method according to any one of claims 1 to 8 includes: The acquisition module is used to acquire text input data, context data of the text input data, domain-specific prompts, context-specific prompts, and task-specific prompts; The first input module is used to input the context data into the first encoder to obtain the first encoded output data, and input the first encoded output data, domain-specific prompts and context-specific prompts into the first decoder to generate a first comprehensive prompt; The perception module is used to perceive the first comprehensive prompt using the first multilayer perceptron to obtain a first comprehensive vector, and to perceive the task-specific prompt using the second multilayer perceptron to obtain a first task vector. The second input module is used to concatenate the first comprehensive vector and the first task vector to obtain a second comprehensive vector, and input the second comprehensive vector, the text input data and the context data to the second encoder to obtain second encoded output data; The generation module is used to use the third multilayer perceptron to perceive the task-specific vector to obtain a second task vector, and input the second task vector and the second encoded output data into the second decoder to obtain the generated text of the text input data.

10. A question-and-answer system, characterized in that, The text generation method according to any one of claims 1 to 8 includes: A prompt generator is configured to generate the first comprehensive prompt based on the context data, the domain-specific prompt, and the context-specific prompt; wherein the prompt generator includes: a first encoder, a first decoder, and a first multilayer perceptron, the first encoder being connected to the first decoder, and the first decoder being connected to the first multilayer perceptron; A generative question-answering model is used to obtain the generated text based on the text input data, the context data, the first comprehensive prompt, and the task-specific prompt; wherein, the generative question-answering model includes: a second encoder, a second decoder, a second multilayer perceptron, and a third multilayer perceptron; the second encoder is connected to the first multilayer perceptron and the second multilayer perceptron, and the second decoder is connected to the second encoder and the third multilayer perceptron.

11. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the text generation method as described in any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that, The storage medium stores a program that is executed by a processor to implement the text generation method as described in any one of claims 1 to 8.

Images