Intelligent question-answering method, server, storage medium and program product

Abstract

This application provides an intelligent question-answering method, server, storage medium, and program product. The method relates to the field of artificial intelligence. It generates task instructions containing partial context information and additional context not included in the task instructions by acquiring input query information and context information matching the query information. The additional context is then input together with the task instructions into an extended task model. The extended task model generates a response result for the query information based on the encoded additional context and the task instructions. The extended task model is capable of processing longer input text. In scenarios requiring long input context information, the extended task model can receive additional context that cannot be included in the task instructions, thereby learning complete context information and improving model performance and generation quality.

Description

Technical Field

[0001] The present application relates to computer technology, and in particular to an intelligent question-answering method, a server, a storage medium, and a program product. Background Art

[0002] With the development of language models, language models based on the Transformer architecture, which are pre-trained on a large corpus, are very popular in the field of Natural Language Processing (NLP). In recent years, more and more natural language processing tasks need to process long-length contexts, such as Open-Domain Question-Answering (ODQA) based on Retrieval Augmented Generation (RAG). However, such models are still limited by the input length in the fine-tuning and reasoning stages of downstream tasks. For example, the input length of some common language models is limited to 2048 tokens.

[0003] In scenarios where RAG technology is used, retrieval enhancement often results in longer context information, and the input can easily exceed the maximum length that the model can handle, making it difficult or even impossible to fine-tune and reason about the model on overly long contexts. In addition, under the setting of in-context learning (ICL), the input context also includes some examples, and the input length becomes longer. However, due to the limitation of the maximum input length of the model, only part of the context can be intercepted as the model input, resulting in the model only being able to learn part of the input context, while the part of the context that is not input cannot be learned, affecting the performance and generation quality of the model. Therefore, how to enable the model to have the ability to process longer input texts becomes particularly important to improve the generation quality of the model. Summary of the invention

[0004] The present application provides an intelligent question-answering method, server, storage medium and program product to solve the problem that in a scenario where the context is too long, due to the limitation of the model input length, only part of the context can be intercepted for input into the model, and the model cannot learn the complete context information, which affects the model performance and generation quality.

[0005] In a first aspect, the present application provides an intelligent question-answering method, comprising:

[0006] Acquire input query information and context information matched by the query information;

[0007] Generate a task instruction and an additional context according to the query information and the context information, wherein the task instruction includes part of the context information, and the additional context includes part of the information in the context information that is not included in the task instruction;

[0008] The task instruction and the additional context are input into an extended task model, and a response result of the query information is generated by the extended task model according to the additional context and the task instruction.

[0009] In a second aspect, the present application provides an intelligent question-answering method, comprising:

[0010] Receive input query information;

[0011] Performing retrieval enhancement according to the query information to obtain context information matching the query information;

[0012] Generate a task instruction and an additional context according to the query information and the context information, wherein the task instruction includes at least part of the context information, and the additional context includes part of the context information that is not included in the task instruction;

[0013] Inputting the task instruction and the additional context into an extended task model, and generating a response result of the query information according to the additional context and the task instruction through the extended task model;

[0014] The response result is outputted.

[0015] In a second aspect, the present application provides an intelligent question-answering method, comprising:

[0016] In response to a call request for an extended task model, a task instruction and an additional context are obtained, wherein the task instruction includes query information and partial information of context information matched by the query information, and the additional context includes partial information of the context information that is not included in the task instruction;

[0017] Inputting the task instruction and the additional context into an extended task model, and generating a response result of the query information according to the additional context and the task instruction through the extended task model;

[0018] The response result is output.

[0019] In a fourth aspect, the present application provides a server, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the server executes a method provided in any of the preceding aspects.

[0020] In a fifth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the method provided in any of the foregoing aspects is implemented.

[0021] In a sixth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the method provided in any of the aforementioned aspects.

[0022] The intelligent question-answering method, server, storage medium and program product provided by the present application generate a task instruction containing partial context information and additional context not included in the task instruction by obtaining input query information and context information matching the query information; the additional context is input into an extended task model together with the task instruction, and a response result of the query information is generated by the extended task model according to the additional context encoding and the task instruction; the extended task model used has the ability to process longer input texts. In scenarios where longer context information needs to be input, the extended task model can receive additional context that cannot be included in the task instruction in the form of additional context, so that the complete context information can be learned, thereby improving the performance and generation quality of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0024] Figure 1 A schematic diagram of an exemplary system architecture applicable to the present application;

[0025] Figure 2 A flowchart of an intelligent question-answering method provided for an exemplary embodiment of the present application;

[0026] Figure 3 A schematic diagram of the architecture of an extended task model provided for an exemplary embodiment of the present application;

[0027] Figure 4 A reasoning flow chart for generating a response result by extending a task model provided by an exemplary embodiment of the present application;

[0028] Figure 5 A schematic diagram of the structure of an extended converter layer provided for an exemplary embodiment of the present application;

[0029] Figure 6 A flow chart of the extended task model training process provided in an example embodiment of the present application;

[0030] Figure 7A flowchart of an intelligent question-answering method provided for another exemplary embodiment of the present application;

[0031] Figure 8 A flowchart of an intelligent question-answering method provided for another exemplary embodiment of the present application;

[0032] Fig. 9 A schematic diagram of the structure of a server provided in an embodiment of the present application.

[0033] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0034] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0035] It should be noted that the user information (including but not limited to user device information, user attribute information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0036] First, the terms involved in this application are explained:

[0037] Token: Also called discrete tag or discrete token, it refers to the discrete representation of speech or text. In order to distinguish the discrete representation of text and speech, the discrete representation of speech is called speech token, and the discrete representation of text is called text token. The vocabulary used by the model contains speech tokens and text tokens that may appear in the prediction results. Each speech token in the vocabulary corresponds to a speech unit, such as a phoneme, syllable, speech segment, etc. Each text token in the vocabulary corresponds to a text unit, such as a character, word, etc.

[0038] Visual Question Answering (VQA) task: Based on the input image and question, determine the answer to the question from the visual information of the input image.

[0039] Image Caption (IC) task: Generate description text for the input image.

[0040] Visual entailment task: predict the semantic relevance of an input image and text, i.e., entailment, neutral, or contradiction.

[0041] Referential expression and comprehension task: locate the image area in the input image corresponding to the input text based on the input text.

[0042] Image generation task: Generate an image based on the input description text.

[0043] Text-based sentiment classification task: predict the sentiment classification information of the input text.

[0044] Text summarization task: Generate summary information of the input text.

[0045] Multimodal tasks: refers to downstream tasks whose input and output data involve multiple modal data such as images and text, such as visual question answering tasks, image description tasks, visual implication tasks, referential expression and understanding tasks, image generation tasks, etc.

[0046] Multimodal pre-trained model: refers to a pre-trained model whose input and output data involve multiple modal data such as images and text. After fine-tuning and training, it can be applied to multimodal task processing.

[0047] Pre-trained language model: A pre-trained model obtained by pre-training a large-scale language model (LLM).

[0048] A large model refers to a deep learning model with large-scale model parameters, usually containing hundreds of millions, tens of billions, or even hundreds of billions of model parameters. A large model can also be called a foundation model / foundation model (FM). It is pre-trained with large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks and has good generalization capabilities, such as large-scale language models and multi-modal pre-training models.

[0049] When the big model is used in practice, only a small number of samples are needed to fine-tune the pre-trained model and it can be applied to different tasks. The big model can be widely used in natural language processing, computer vision and other fields. Specifically, it can be applied to computer vision tasks such as visual question answering, image description, image generation, as well as natural language processing tasks such as text classification, information extraction, sentiment analysis, machine translation, question and answer systems, and dialogue systems. The main application scenarios of the big model include intelligent search, digital assistants, intelligent robots, online education, office software, e-commerce, intelligent design, etc.

[0050] At present, most language models are based on the Transformer architecture, and such language models are widely used in the field of natural language processing (NLP). Usually, such models are limited by the input length. For example, the input length of models such as Bloomz and Llama-1 is limited to 2048 tokens. In scenarios using RAG technology, the contextual information obtained by retrieval enhancement is often overlapping, and the input can easily exceed the maximum input length that the model can receive, making it difficult or even impossible to fine-tune and reason about the model in scenarios with too long context.

[0051] The present application provides an extended task model, which is obtained by extending and training the pre-trained task model. The extended task model can not only receive the task instructions that the original pre-trained decoder task model can receive, but also receive additional context information (called additional context), by encoding the additional context into additional context features, mapping the task instructions to query features, and generating the response results of the query information according to the additional context features and query features. The extended task model has the ability to process longer input texts, and the additional context can be encoded as features without limiting the length of the additional context. Therefore, in the model training and reasoning process, even if the context information that needs to be input is long, the extended task model can receive and learn the complete context information in the form of additional context, thereby improving the performance of the model and the generation quality of the model.

[0052] Among them, the pre-trained decoder task model can be various pre-trained models based on the Transformer architecture, such as a pre-trained language model, a large model, etc., which is not specifically limited in this embodiment.

[0053] Based on the above-mentioned extended task model, the present application provides an intelligent question-answering method, which obtains input query information; generates task instructions according to the query information, and obtains additional context corresponding to the query information; inputs the task instructions and the additional context into the extended task model, and generates a response result of the query information according to the additional context and the task instructions through the extended task model. The extended task model has the ability to process longer input texts. During the model reasoning process, the model can not only receive task instructions that the original pre-trained decoder task model can receive, but also receive additional context. By learning the information of the additional context, the model performance can be improved and the quality of the generated response results can be improved.

[0054] The solution of the present application can be applied to application scenarios such as intelligent search, digital assistants, intelligent robots, online education, office software, e-commerce, and intelligent design. Specifically, it can be applied to natural language processing tasks such as text classification, information extraction, sentiment analysis, machine translation, question-and-answer systems, and dialogue systems. It can also be applied to computer vision tasks such as visual question-and-answer, image description, and image generation. This embodiment is not specifically limited here.

[0055] Figure 1 FIG. 1 is a schematic diagram of an exemplary system architecture applicable to the present application. Figure 1 As shown, the system architecture includes a server and a terminal device. There is a communication link between the server and the terminal device, which can realize the communication connection between the server and the terminal device.

[0056] The server is a device with computing capabilities deployed in the cloud or locally, such as a cloud cluster, etc. The server stores an extended task model, which is obtained by extending and training the pre-trained task model.

[0057] The end-side device can be an electronic device running the downstream task system, and specifically can be a hardware device with network communication function, computing function and information display function, including but not limited to smart phones, tablet computers, desktop computers, local servers, cloud servers, etc. In the process of the end-side device running the downstream task system, it is necessary to generate a response result based on the query information input by the user with the help of the extended task model. For example, the downstream task system run by the end-side device can realize functions such as intelligent question and answer, intelligent dialogue, information extraction, and text classification. When realizing at least one function of the downstream task system, it is necessary to use the extended task model to perform the processing of the corresponding task and obtain the task processing result (i.e., the response result).

[0058] Specifically, during the operation of the downstream task system, the terminal device obtains the query information (ie, query) input by the user and the context information matched by the query information, and sends the input query information and the context information matched by the query information to the server.

[0059] The server generates a task instruction and additional context based on the input query information and the context information matched by the query information. In particular, due to the limitation of the input length of the pre-trained decoder task model, the task instruction includes part of the context information matched by the query information, and the additional context includes another part of the context information not included in the task instruction. The server inputs the task instruction and the additional context into the extended task model, and generates a response result of the query information based on the additional context and the task instruction through the extended task model. The server outputs the response result to the end-side device.

[0060] The end-side device receives the response result returned by the server, and continues to execute the processing logic of the downstream task system according to the response result to realize the function of the downstream task system.

[0061] In addition, the server can also be responsible for building an extended task model based on the pre-trained decoder task model, as well as training the extended task model.

[0062] In an example scenario, the server may also provide an application programming interface (API) of the extended task model to the outside. The end-side device calls the extended task model through the API of the extended task model, so that the extended task model implements the corresponding task processing and obtains the task processing result.

[0063] Exemplarily, the end-side device receives the query information input by the user, obtains the context information matching the query information according to the query information, and generates the task instruction and the additional context according to the query information and the context information. The end-side device sends a call request to the server according to the API of the extended task model, and the call request includes the task instruction and the additional context. The server receives the call request, obtains the task instruction and the additional context, inputs the task instruction and the additional context into the extended task model, and generates the response result of the query information according to the task instruction and the additional context through the extended task model. The server returns the response result to the end-side device.

[0064] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0065] Figure 2 This is a flow chart of an intelligent question-answering method provided by an exemplary embodiment of the present application. The execution subject of this embodiment is the server in the aforementioned system architecture. Figure 2 As shown, the specific steps of this method are as follows:

[0066] Step S201: Acquire input query information and context information matched by the query information.

[0067] The method of this embodiment can be applied to various natural language processing tasks, such as RAG-based question-answering tasks.

[0068] In an example scenario, the end-side device can receive query information (i.e., Query) input by the user, and based on the RAG technology, perform retrieval enhancement on the query information to obtain context information matching the query information. The context information matching the query information can be a retrieval enhancement result based on the query information, that is, related knowledge information searched or retrieved based on the query information.

[0069] The end-side device sends a task request to the server, and the task request includes query information and context information matching the query information. The server receives the task request sent by the end-side device, and extracts the query information and context information matching the query information from the task request, thereby obtaining the input query information and context information matching the query information.

[0070] In another example scenario, the end-side device can receive query information (i.e., Query) input by the user. The end-side device sends a task request to the server, which includes the query information. The server receives the task request and extracts the query information contained in the task request. Based on the RAG technology, the server performs retrieval enhancement on the query information to obtain context information that matches the query information.

[0071] In this embodiment, q is used to represent the input query information, and k={k 1 ,k 2 ,…,k C} represents the context information matched by the query information, including C independent context texts, k 1 ,k 2 ,…,k C They represent the token sequence of C context texts respectively. C is an integer, which represents the number of context texts matched by the query information. Usually, the context information matched by the query information contains the knowledge required to correctly respond to the query information q, but there may also be noise.

[0072] In actual applications, the server obtains the query information input by the user and the implementation method of the context information matched by the query information can be designed and adjusted according to the actual application requirements, and is not specifically limited here. i , can be retrieved from a given corpus by a retriever based on query information q.

[0073] Step S202: Generate a task instruction and additional context based on the query information and the context information. The task instruction includes at least part of the context information, and the additional context includes the part of the context information that is not included in the task instruction.

[0074] After obtaining the input query information and the context information matching the query information, the server generates a task instruction based on the query information and the context information. Among them, the task instruction includes at least part of the context information, and the task instruction meets the input length limit of the pre-trained decoder task model. Further, the part of the context information that is not included in the task instruction is used as additional context.

[0075] Exemplarily, in a scenario where the obtained context information is relatively long, when implementing the generation of a task instruction based on the query information and the context information, the server inserts part of the query information and the context information into the corresponding positions in the prompt (i.e., Prompt) according to the input length limit of the original pre-trained decoder task model and the prompt word, and then the corresponding task instruction can be generated. The length of this task instruction meets the input length limit of the pre-trained decoder task model.

[0076] The generated task instruction can be expressed as: x(q, k max , P). Where P represents the prompt word, and k max represents the context information included in the task instruction. k max includes m context texts in the context information and can be expressed as k max =(k 1 , k 2 , …, k m ) ∈ k, and m < C. m refers to the number of contexts that achieve the model throughput, and m is a positive integer. Inserting the first m context texts k 1 , k 2 , …, k m and the query information into the prompt word, the obtained task instruction meets the model input length limit. However, inserting the first m + 1 context texts k 1 , k 2 , …, k m+1 and the query information into the prompt word, the obtained task instruction will exceed the model input length limit.

[0077] Taking the remaining context information that is not included in the task instruction as additional context, it can be expressed as: k add ={k m+1 , k m+2 , …, k C [[ID=46}}. In some scenarios where the context information is relatively long, the additional context k addThe length of the generated task instruction is often several times or even more than the context contained in it.

[0078] Among them, when applied to different task scenarios, the prompt words (i.e., Prompt) used can be configured and adjusted according to actual application requirements, and are not specifically limited here. Task instructions are generated based on query information and context information, and any existing input sequence implementation method that meets the model input length limit can be used, and this embodiment will not be repeated here.

[0079] Step S203: input the task instruction and the additional context into the extended task model, and generate a response result of the query information according to the task instruction and the additional context through the extended task model.

[0080] In this embodiment, the extended task model used is obtained by expanding and training the pre-trained task model. Among them, the pre-trained decoder task model can be various pre-trained models based on the Transformer architecture, such as a pre-trained language model, a large model, etc., which is not specifically limited in this embodiment.

[0081] The extended task model can not only receive the task instructions that the original pre-trained decoder task model can receive, but also receive additional context that is not included in the task instructions. Furthermore, the extended task model encodes the additional context into additional context features, maps the task instructions into query features, and generates a response result of the query information based on the additional context features and the query features.

[0082] The extended task model has the ability to process longer input texts and can encode additional context as features without limiting the length of the additional context. During model training and task reasoning, even if the context information matched by the query information is longer, the extended task model can receive and learn the complete context information in the form of additional context, which improves the performance of the model and the quality of model generation.

[0083] In this embodiment, by obtaining input query information and context information matching the query information, a task instruction containing partial context information and additional context not included in the task instruction is generated; the additional context is input into the extended task model together with the task instruction, and the extended task model is used to generate a response result of the query information based on the task instruction and the additional context. The model can learn to utilize richer additional context, thereby improving the performance of the model and thus improving the quality of the generated response result.

[0084] In practical applications, the pre-trained decoder task model includes an embedding layer, at least one transformer layer (i.e., a transformer block or a deformation structure of a transformer block), and an output layer. The embedding layer is used to map the task instructions to query features. At least one transformer layer is used to transform the query features to obtain transformed features. The output layer is used to generate a response result based on the transformed features.

[0085] Figure 3 A schematic diagram of the architecture of an extended task model provided for an example embodiment of the present application. In an optional embodiment, the extended task model is obtained by adding an additional encoding module to the pre-trained task model and expanding at least one transformer layer in the pre-trained decoder task model into a corresponding extended transformer layer. Figure 3 As shown, the extended task model includes an additional encoding module, an embedding layer, at least one extended transformer layer and an output layer.

[0086] The additional encoding module is used to encode the additional context to obtain additional context features. The additional encoding module outputs the additional context features to each extended transformer layer. The additional encoding module can use any pre-trained text encoding model, such as any pre-trained BERT (Bidirectional Encoder Representations from Transformers, a deep bidirectional language representation model based on the Transformer architecture), etc., which is not specifically limited here.

[0087] The embedding layer of the extended task model retains the original embedding layer of the pre-trained decoder task model. The embedding layer is used to map the task instructions to query features and output the query features to the first extended transformer. The embedding layer can be implemented using any text embedding model, which is not specifically limited here.

[0088] At least one extended transformer layer is used to: fuse and transform the query features and additional context features mapped to the task instructions, and output the query features used to generate the response results. At least one extended transformer layer is stacked in sequence, and the query features output by the previous extended transformer layer are used as the query features input by the next extended transformer layer. The extended transformer layer is used to: fuse and transform the query features and additional context features input by this layer, obtain the query features output by this layer, and input the query features output by this layer to the next extended transformer layer or the output layer. The query features output by the last extended transformer layer are input to the output layer as the query features used to generate the response results.

[0089] The output layer of the extended task model retains the original output layer of the pre-trained decoder task model. The output layer is used to generate a response result based on the query features output by the last extended transformer layer.

[0090] Figure 4 The inference flow chart of generating response results by extending the task model provided in this embodiment. Figure 4 As shown, based on Figure 3 In the extended task model shown in the above step S203, the task instruction and the additional context are input into the extended task model, the additional context is encoded into the additional context feature through the extended task model, the task instruction is mapped into the query feature, and the response result of the query information is generated according to the additional context feature and the query feature, which can be implemented by the following steps:

[0091] Step S41: input the additional context into the additional encoding module of the extended task model, and input the task instruction into the embedding layer of the extended task model.

[0092] In this embodiment, when the additional context is input into the extended task model together with the task instruction, the task instruction is input into the embedding layer of the extended task model, and the additional context is input into the additional encoding module.

[0093] Step S42: encode the additional context through an additional encoding module to obtain additional context features, and input the additional context features into at least one extended transformer layer.

[0094] In this step, the additional encoding module encodes each context text contained in the additional context to obtain the corresponding context features, and the context features of each context text are spliced ​​(or connected) to obtain the additional context features. Exemplarily, the additional context can be expressed as: add ={k m+1 ,k m+2 ,…,k C}, the additional context features can be expressed as: in, Enc(·) represents the encoding process of the additional encoding module.

[0095] In an optional embodiment, the context texts included in the additional context may be concatenated, and the concatenated result of the additional context may be encoded by an additional encoding module to obtain additional context features.

[0096] In this step, the additional context features output by the additional encoding module are input to each extended transformer layer as the additional context features input to each extended transformer layer.

[0097] Step S43: Map the task instruction into a query feature through the embedding layer, and input the query feature into the first extended transformer layer.

[0098] In this step, the task instruction is mapped into a query feature through the embedding layer, and this process can be expressed as: h q = Emb(x(q, k max , P)). Where x(q, k max , P) represents the task instruction generated according to the query information q, the context information, and the prompt word P, and this task instruction contains partial context information k max = (k 1 , k 2 , …, k m ), m < C. Emb(·) represents the processing process of the embedding layer.

[0099] In this step, the query feature output by the embedding layer is input into the first extended transformer layer as the query feature input to the first extended transformer layer.

[0100] Step S44: Through the extended transformer layer, fuse and transform the query feature input to this layer and the additional context feature to obtain the query feature output by this layer, input the query feature output by this layer into the next extended transformer layer, and input the query feature output by the last extended transformer layer into the output layer.

[0101] In this embodiment, any extended transformer layer j fuses and transforms the query feature input to this layer (represented by ) and the additional context feature (h add ) to obtain the query feature output by this layer (represented by ); input the query feature output by this layer into the next extended transformer layer or the output layer. The query feature output by the last extended transformer layer (represented by ) is used as the query feature for generating the response result and is input into the output layer.

[0102] Exemplarily, the first extended transformer layer fuses and transforms the query feature h q input to this layer and the additional context feature h add to obtain the query feature output by this layer (represented by ). The query feature output by the first extended transformer layer is used as the query feature input to the second extended transformer layer and is input into the second extended transformer layer, that is,

[0103] through the second extended transformer layer, the query feature input to this layer and the additional context feature hadd Perform fusion and transformation processing to obtain the query features output by this layer (using The query features output by the second extended transformer layer are Query features as input to the third expansion transformer layer Input to the third extended transformer layer, i.e.

[0104] Similarly, after being processed by N expansion transformer layers, the query feature output by the last (i.e., Nth) expansion transformer layer is As the query feature h′ used to generate the response result q , input to the output layer.

[0105] Step S45: Generate a response result of the query information through the output layer according to the query features output by the last extended transformer layer.

[0106] In this step, the query feature h′ is outputted by the last extended transformer layer through the output layer. q , query feature h′ q Convert it into text tags and obtain the output text tag sequence, that is, the response result.

[0107] In an optional embodiment, any transformer layer in the pre-trained decoder task model is expanded into a corresponding extended transformer layer, which can be achieved in the following manner: before the original transformer layer (i.e., the transformer block), an extension module based on the cross-attention mechanism is added. The query features and additional context features input to this layer are input to the extension module, and the query features and additional context features input to this layer are cross-attention calculated through the extension module to fuse the query features and additional context features to obtain fused features. The fused features are input to the original transformer layer (i.e., the transformer block).

[0108] Figure 5 This is a schematic diagram of the structure of the extended converter layer provided in the embodiment of the present application. Figure 5 As shown, the extended transformer layer includes: a transformer block and an extension module based on a cross-attention mechanism. Figure 5 ×N indicates that N extended transformer layers are stacked. In the above step S44, the query features input by the layer and the additional context features are fused and transformed through the extended transformer layer to obtain the query features output by the layer, which specifically includes:

[0109] Through the expansion module of the extended transformer layer, the query features and additional context features of this layer are cross-attentioned to obtain the fused features. Through the transformer block, the fused features are transformed to obtain the query features of this layer.

[0110] Among them, the transformer block is usually a neural network block based on the self-attention mechanism, which usually includes a self-attention module and a feedforward neural network. Figure 5 In the figure, the feedforward neural network in the transformer block is a multilayer perceptron (MLP) as an example for exemplary explanation. The feedforward neural network in the transformer block can also be implemented by other structures with similar functions. In addition, the transformer block can also include a normalization layer, a standardization layer, etc. The transformer block is the original transformer layer of the pre-trained decoder task model (that is, the Transformer Block or a deformed structure of the Transformer Block). The implementation principle of the transformer block will not be repeated here. Figure 5 The “×N” in indicates that the extended task model contains N stacked extended transformer layers.

[0111] like Figure 5 As shown, the extension module based on the cross-attention mechanism may include a cross-attention module and a multi-layer perceptron. The cross-attention module is used to perform cross-attention calculations on the query features and additional context features input into this layer, so as to fuse the query features and additional context features to obtain fused features. The multi-layer perceptron is located between the cross-attention module and the self-attention module of the transformer block. As a connection layer between the two attention modules, the fused features output by the cross-attention module can be further extracted and transformed, and then input into the transformer block, which can improve the performance of the model. In addition, between the cross-attention module and the transformer block, a normalization layer, a standardization layer, etc. may also be included, which may be specifically designed and adjusted according to actual application requirements, and are not specifically limited here.

[0112] In an alternative embodiment, if Figure 5 As shown, the extended transformer layer may also include: a mapping module. The mapping module is used to align the dimension of the additional context features with the query features input by this layer to obtain the aligned additional context features. Among them, the mapping module can be implemented by a mapping layer (Projector) having a dimension of aligning high-dimensional hidden features. For example, the mapping module can be implemented by MLP and a fully connected layer. The specific design and adjustment can be made according to the actual application requirements, and no specific limitation is made here.

[0113] Furthermore, through the extension module, cross-attention calculation is performed on the query features input into this layer and the aligned additional context features to obtain fused features.

[0114] For example, taking the j-th extended transformer layer as an example, the extended module of the extended transformer layer includes a mapping module, a cross attention module and a multi-layer perceptron MLP. The query feature input of this layer is transformed into and additional context features hadd Perform fusion and transformation processing to obtain the query features output by this layer (using The process of expressing) is as follows:

[0115] Through the mapping module, the additional context feature h add The query features of this layer input The dimensions of the aligning function are aligned to obtain additional context features after alignment, which is expressed as h kv =Proj(h add ), Proj(·) represents the processing process of the mapping module.

[0116] Furthermore, the aligned additional context feature h kv As key features (i.e. key) and value features (i.e. value), and the query features input to this layer (i.e. query) to perform cross attention calculation to obtain fusion features, which can be expressed as: Where Cross-attn(·) represents the process of cross attention calculation. c After multi-layer perceptron MLP processing, the fusion feature h is obtained m ,h m =MLP(h c ). The fusion feature h after MLP processing m Output to the Transformer block.

[0117] Furthermore, the transformer block performs the fusion feature h output by the expansion module. m Perform transformation processing based on the self-attention mechanism to obtain the query features output by this layer

[0118] The extended transformer layer provided in this embodiment can fuse the additional context features with the query features input by each layer, and transform the fused features, so as to better learn the complete context information including the additional context, and improve the representation ability of the query features finally obtained for generating the response results, thereby improving the quality of the generated response results and improving the model performance.

[0119] Figure 6 This is a flowchart of the extended task model training process provided by an example embodiment of the present application. Figure 6 As shown in the figure, the specific steps of extended task model training are as follows:

[0120] Step S601: Expand the pre-trained decoder task model to construct an expanded task model to be trained.

[0121] In this embodiment, the extended task model is obtained by adding an additional encoding module to the pre-trained decoder task model and expanding at least one transformer layer in the pre-trained decoder task model into a corresponding extended transformer layer. Figure 3 As shown, the extended task model includes an additional encoding module, an embedding layer, at least one extended transformer layer and an output layer. Among them, the additional encoding module can be initialized using the parameters of a pre-trained text encoder (such as pre-trained BERT, etc.) to minimize the amount of randomly initialized parameters.

[0122] In addition, if Figure 5 As shown, the extended transformer layer includes a transformer block and an extension module based on a cross-attention mechanism, and the extension module based on the cross-attention mechanism may include a cross-attention module and a multi-layer perceptron. Among them, the transformer block is the transformer layer in the retained pre-trained decoder task model. The cross-attention module and the multi-layer perceptron in the extension module can be initialized using pre-trained parameters (such as the parameters of the corresponding modules in the pre-trained Bloomz-1b7 model), or they can be initialized randomly.

[0123] In an alternative embodiment, if Figure 5 As shown, the extended transformer layer may further include a mapping module. The mapping module is used to align the dimension of the additional context features with the query features input by this layer to obtain the aligned additional context features. The parameters of the mapping module may be initialized at any time.

[0124] In this embodiment, the architecture and implementation principle of the constructed extended task model to be trained refer to the relevant content of the aforementioned embodiment, which will not be repeated here in this embodiment.

[0125] Step S602: Acquire training data, where the training data includes: query samples, reference responses to the query samples, and context information matched by the query samples.

[0126] In this embodiment, the training data used to train the extended task module includes not only the query sample and the context information matched by the query sample, but also the reference response of the query sample, that is, the correct response or standard response of the query sample.

[0127] The training data can be constructed using a fine-tuned training set of similar task scenarios. For example, a fine-tuned dataset for a question-answering task can be obtained, which usually includes query information (Q) and reference responses (A). Through a retriever / search engine, retrieval / search is performed based on the query information Q in the fine-tuned dataset, and the retrieval / search results are used as context information for query information matching.

[0128] Optionally, the results of the retrieval / search can be sorted from the highest to the lowest according to the degree of matching with the query information, and the sorted results are used as context information. In this way, the generated task instructions can include the first m context information with higher matching degrees in the context information, which can improve the performance of the model.

[0129] Furthermore, training data including query samples, reference responses of query samples, and context information matching the query samples is constructed.

[0130] Step S603: Generate instruction samples and additional context samples according to the query samples, reference responses of query samples, and context information matching the query samples. The instruction samples include at least partial information of the context information matching the query samples; the additional context samples include the part of the context information matching the query samples that is not included in the instruction samples.

[0131] In this embodiment, task instructions are generated according to the query samples and the context information matching the query samples in the training data. To distinguish from the task instructions in the inference process, the task instructions generated based on the training data during the training process are called instruction samples in this embodiment. Among them, the instruction samples include at least partial information of the context information matching the query samples, and the instruction samples meet the input length limit of the pre-trained decoder task model. Furthermore, the part of the context information matching the query samples that is not included in the instruction samples is used as the additional context samples.

[0132] The generated instruction samples can be expressed as: x(q′, a, k′ max , P′), which can be expressed as a sequence of text tokens {x 1 , x 2 , …}. Among them, q′ represents the query sample, a represents the reference response of the query sample, P′ represents the prompt word, and k′ max represents the context information included in the instruction samples. k′ max includes m′ context texts in the context information, which can be expressed as k′ max =(k 1 , k 2 , …, k m′ ) ∈ k, m′ < C′. m′ refers to the number of contexts that reach the model throughput, and m′ is a positive integer. Insert the first m′ context texts k 1 , k 2 , …, k m′ , the query sample q′ and the reference response a into the prompt word P′ to obtain the instruction samples, and the instruction samples meet the model input length limit.

[0133] The remaining context information not included in the instruction samples is used as the additional context samples, which can be expressed as: k′ add={k m′+1 ,k m′+2 ,…,k C′}.

[0134] Step S604: take the instruction sample as the input sequence, and input the additional context sample and the input sequence into the extended task model to be trained, and generate a prediction result of the input sequence according to the input sequence and the additional context sample through the extended task model to be trained.

[0135] In this embodiment, the instruction sample (which can be represented as a text tag sequence {x 1 ,x 2 ,…}), as the input sequence. 1 ,x 2 ,…} and additional context samples k′ add , input the extended task model to be trained, and use the extended task model to be trained to convert the additional context sample k′ add Encoded as additional context feature h′ add , the input sequence {x 1 ,x 2 ,…} is mapped to the query feature h′ q′ , and based on the additional context features h′ of the additional context samples add and the query feature h′ of the input sequence q′ , generate prediction results for the input sequence.

[0136] Similar to the fine-tuning training process of the pre-trained decoder task model, in this embodiment, the extended task model is fine-tuned with the autoregressive language modeling task as the training target. Specifically, in the process of generating the prediction result of the input sequence, multiple time steps are iterated, and the first time step predicts the first text tag in the input sequence based on the given start tag. In the subsequent time steps, the next text tag is predicted based on the first several known text tags in the input sequence. Finally, the prediction result of the input sequence can be obtained.

[0137] Among them, the prediction results of the input sequence include Q ω (x i |x <i ,k′ add ), that is, based on the first i-1 text tokens in the input sequence (denoted as x <i ) and additional context samples k′ add , predict the i-th text tag (i.e. x i ) probability.

[0138] Step S605: According to the prediction result of the input sequence, the parameters of the extended task model are adjusted to obtain a trained extended task model.

[0139] In this embodiment, the language modeling loss is calculated according to the prediction result of the input sequence, and the parameters of the extended task model are adjusted according to the language modeling loss to obtain a trained extended task model.

[0140] Specifically, the following formula (1) can be used to calculate the language modeling loss based on the prediction results of the input sequence:

[0141]

[0142] Among them, J ω (D) represents the language modeling loss, ω represents the parameters of the extended task model. D represents the set of training data. x(q′,a,k′ max ,P′),k′ add ∈D, represents the input sequence and additional context samples determined based on any training data. ω (x i |x <i ,k′ add ) represents the first i-1 text tokens in the input sequence (denoted as x <i ) and additional context samples k′ add , predict the i-th text tag (i.e. x i ) probability.

[0143] In addition, the calculation method of language modeling loss can also be modified based on formula (1), for example, the logarithmic operation is changed to an exponential operation, etc. The specific design and adjustment can be made according to the actual application requirements, and no specific limitation is made here.

[0144] The training goal of this embodiment is to minimize the language modeling loss on the set of training data. After the language modeling loss is calculated, the parameters of the extended task model are adjusted by back propagation according to the language modeling loss to obtain a trained extended task model.

[0145] In this embodiment, based on the constructed training data, the extended task model to be trained is fine-tuned to obtain a trained extended task model.

[0146] The parameters of the additional encoding module (denoted by φ) are initialized with the parameters of a well-pretrained encoding model on a large-scale corpus. The pretrained parameters have good performance in many downstream tasks based on text encoding. Since the parameters of the mapping module are randomly initialized, during the model training process, the random gradients of the random parameters of the mapping module in the extended task model are likely to affect the pretrained parameters in the additional encoding module, weakening the ability of the additional encoding module to encode information, thereby affecting the performance of the extended task model.

[0147] In an optional embodiment, during the training process of the extended task model, the parameters of the additional encoding module are fixed, and the parameters of the extended task model other than the parameters of the additional encoding module (including the embedding layer, at least one extended transformer layer and the output layer) are adjusted. This can better prevent the random gradients of the random parameters of the newly introduced mapping module from destroying the pre-trained parameters of the additional encoding module, thereby improving the performance of the trained extended task model.

[0148] In another optional embodiment, during the training process of the extended task model, in at least the first training iteration, the parameters of the additional coding module are fixed, and the parameters of the extended task model except the parameters of the additional coding module (including the embedding layer, at least one extended transformer layer and the output layer) are adjusted; in subsequent training iterations, all parameters of the extended task model (including the additional coding module, the embedding layer, at least one extended transformer layer and the output layer) are adjusted.

[0149] The method of this embodiment is to construct an extended task model to be trained by extending the pre-trained decoder task model; obtain training data, the training data includes: query samples, reference responses of query samples, and context information matching the query samples; generate instruction samples according to the query samples, reference responses of query samples, and context information matching the query samples, the instruction samples at least contain part of the context information matching the query samples; use part of the context information matching the query samples that is not included in the instruction samples as additional context samples; use the instruction samples as input sequences, input the additional context samples and the input sequences into the extended task model to be trained, and generate the prediction results of the input sequence according to the additional context samples and the input sequences through the extended task model to be trained; adjust the parameters of the extended task model according to the prediction results of the input sequences, and obtain the extended task model after training. The extended task model obtained by training can not only receive the task instructions that the original pre-trained decoder task model can receive, but also receive additional context information (called additional context), and the extended task model has the ability to process longer input texts, and the additional context can be encoded as features without limiting the length of the additional context. Therefore, during model training and reasoning, even if the context information required for input is long, the extended task model can receive and learn the complete context information in the form of additional context, which improves the performance of the model and the quality of model generation.

[0150] Figure 7 This is a flow chart of an intelligent question-answering method provided by another exemplary embodiment of the present application. The execution subject of this embodiment may be a device for implementing the intelligent question-answering function, or may be a server on which the intelligent question-answering system runs. Figure 7 As shown, the specific steps of this method are as follows:

[0151] Step S701: Receive input query information.

[0152] Exemplarily, the end-side device may receive query information (ie, Query) input by a user. The end-side device sends a task request to the server, where the task request includes the query information. The server receives the task request and extracts the query information included in the task request.

[0153] Step S702: Perform retrieval enhancement according to the query information to obtain context information matching the query information.

[0154] In this embodiment, the intelligent question-answering system also has a search enhancement function, which can be implemented by a server running the intelligent question-answering system. The server performs search enhancement on the query information based on the RAG technology to obtain context information matching the query information.

[0155] The server obtains context information that matches the query information, which can be implemented by using any retrieval enhancement method for a question-and-answer scenario, and is not specifically limited here.

[0156] Exemplarily, the server may call a retriever to retrieve context information matching the query information from a given corpus based on the query information q. And / or, the server may call a search engine to search for context information matching the query information based on the query information q.

[0157] Step S703: Generate a task instruction and an additional context according to the query information and the context information, wherein the task instruction includes at least part of the context information, and the additional context includes part of the context information that is not included in the task instruction.

[0158] The specific implementation principle and technical effect of this step can be found in the relevant content of the aforementioned step S202, which will not be repeated here.

[0159] Step S704: input the task instruction and the additional context into the extended task model, and generate a response result of the query information according to the task instruction and the additional context through the extended task model.

[0160] The specific implementation principle and technical effect of this step can be found in the relevant content of the aforementioned step S203, which will not be repeated here.

[0161] Step S705: output the response result.

[0162] After obtaining the response result, the server outputs the response result.

[0163] For example, the server can output the response result through a visual interface; or the server can return the response result to the terminal device that sent the query information. The implementation method of the server outputting the response result can be designed and adjusted according to actual application requirements and is not specifically limited here.

[0164] In this embodiment, the extended task model used is obtained by expanding and training the pre-trained task model. Among them, the pre-trained decoder task model can be various pre-trained models based on the Transformer architecture, such as a pre-trained language model, a large model, etc., which is not specifically limited in this embodiment.

[0165] The extended task model can not only receive the task instructions that the original pre-trained decoder task model can receive, but also receive additional context that is not included in the task instructions. Furthermore, the extended task model encodes the additional context into additional context features, maps the task instructions into query features, and generates a response result of the query information based on the additional context features and the query features.

[0166] The extended task model has the ability to process longer input texts, and can encode additional context as features without limiting the length of the additional context. During model training and task reasoning, even if the context information matched by the query information is longer, the extended task model can receive and learn the complete context information in the form of additional context, which improves the performance of the model and the quality of model generation.

[0167] For the architecture and training process of the extended task model used in this embodiment, please refer to the relevant content of the aforementioned embodiment and will not be repeated here.

[0168] In this embodiment, input query information is received; retrieval enhancement is performed based on the query information to obtain context information matching the query information; task instructions and additional context are generated based on the query information and the context information, wherein the task instructions at least include part of the context information, and the additional context includes part of the context information that is not included in the task instructions; the additional context is input into an extended task model together with the task instructions, the additional context is encoded into additional context features through the extended task model, the task instructions are mapped into query features, and a response result to the query information is generated based on the additional context features and the query features. The model can learn to utilize richer additional context, thereby improving the performance of the model and thus improving the quality of generated response results.

[0169] Figure 8 This is a flowchart of an intelligent question-answering method provided by another exemplary embodiment of the present application. The execution subject of this embodiment may be a server running a trained extended task model. Figure 8As shown, the specific steps of this method are as follows:

[0170] Step S801, in response to a call request for an extended task model, obtain a task instruction and additional context, wherein the task instruction includes query information and part of the context information matched by the query information, and the additional context includes part of the context information not included in the task instruction.

[0171] In this embodiment, the server provides an application programming interface (API) of the extended task model to the outside. The terminal device calls the extended task model through the API of the extended task model, so that the extended task model implements the corresponding task processing and obtains the task processing result.

[0172] Exemplarily, the terminal device receives the query information input by the user, obtains the context information matching the query information according to the query information, and generates the task instruction and the additional context according to the query information and the context information. For the specific implementation principle of this process, please refer to the relevant content of the above embodiment, which will not be repeated here.

[0173] The end-side device sends a call request to the server according to the API of the extended task model, and the call request includes a task instruction and an additional context. The server receives the call request, extracts the task instruction and the additional context from the call request, and thus obtains the task instruction and the additional context.

[0174] Step S802: Input the task instruction and the additional context into the extended task model, and generate a response result of the query information according to the task instruction and the additional context through the extended task model.

[0175] After obtaining the task instruction and additional context, the server inputs the task instruction and additional context into the extended task model, encodes the additional context into additional context features through the extended task model, maps the task instruction into query features, and generates a response result of the query information based on the additional context features and the query features.

[0176] The specific implementation principle and technical effect of this step can be found in the relevant content of the aforementioned step S203, which will not be repeated here.

[0177] Step S803: output the response result.

[0178] After obtaining the response result, the server returns the response result to the end-side device.

[0179] In this embodiment, the extended task model used is obtained by expanding and training the pre-trained task model. Among them, the pre-trained decoder task model can be various pre-trained models based on the Transformer architecture, such as a pre-trained language model, a large model, etc., which is not specifically limited in this embodiment.

[0180] The extended task model can not only receive the task instructions that the pre-trained decoder task model can receive, but also receive additional context that is not included in the task instructions. Furthermore, the extended task model encodes the additional context into additional context features, maps the task instructions into query features, and generates a response result of the query information based on the additional context features and the query features.

[0181] The extended task model has the ability to process longer input texts, and can encode additional context as features without limiting the length of the additional context. During model training and task reasoning, even if the context information matched by the query information is longer, the extended task model can receive and learn the complete context information in the form of additional context, which improves the performance of the model and the quality of model generation.

[0182] For the architecture and training process of the extended task model used in this embodiment, please refer to the relevant content of the aforementioned embodiment and will not be repeated here.

[0183] In this embodiment, by obtaining the task instructions and additional context corresponding to the query information, the additional context is input into the extended task model together with the task instructions, and the extended task model generates a response result for the query information according to the task instructions and the additional context. The model can learn to utilize richer additional context, thereby improving the performance of the model and thus improving the quality of the generated response results.

[0184] Fig. 9 A schematic diagram of the structure of a server provided in an embodiment of the present application. Fig. 9 As shown, the server includes: a memory 901 and a processor 902. The memory 901 is used to store computer-executable instructions and can be configured to store various other data to support operations on the server. The processor 902 is connected to the memory 901 in communication and is used to execute the computer-executable instructions stored in the memory 901 to implement the technical solution provided by any of the above method embodiments. The specific functions and technical effects that can be achieved are similar and will not be repeated here.

[0185] Optional, such as Fig. 9 As shown, the server also includes: a firewall 903, a load balancer 904, a communication component 905, a power supply component 906 and other components. Fig. 9 Only some components are shown schematically, which does not mean that the server only includes Fig. 9 Components shown. Fig. 9 In the description, the server is only taken as a cloud server deployed in the cloud as an example for exemplary description. The server can also be deployed locally, and this embodiment is not specifically limited here.

[0186] An embodiment of the present application also provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the method of any of the aforementioned embodiments is implemented. The specific functions and technical effects that can be achieved are not repeated here.

[0187] The present application also provides a computer program product, including a computer program, which implements the method of any of the above embodiments when executed by a processor. The computer program is stored in a readable storage medium, and at least one processor of the server can read the computer program from the readable storage medium. At least one processor executes the computer program so that the server executes the technical solution provided by any of the above method embodiments. The specific functions and technical effects that can be achieved are not repeated here.

[0188] The embodiment of the present application provides a chip, including: a processing module and a communication interface, the processing module can execute the technical solution of the server in the aforementioned method embodiment. Optionally, the chip also includes a storage module (such as a memory), the storage module is used to store instructions, the processing module is used to execute the instructions stored in the storage module, and the execution of the instructions stored in the storage module enables the processing module to execute the technical solution provided by any of the aforementioned method embodiments.

[0189] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the methods of various embodiments of the present application.

[0190] It should be understood that the above processor can be a central processing unit (CPU), a graphics processing unit (GPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in at least one processor.

[0191] The memory may include high-speed random access memory (RAM), and may also include non-volatile storage, such as at least one disk storage, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a disk or an optical disk, etc.

[0192] The above storage may be an object storage service (OSS).

[0193] The above-mentioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0194] The above-mentioned communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as a mobile hotspot (WiFi), a second-generation mobile communication system (2G), a third-generation mobile communication system (3G), a fourth-generation mobile communication system (4G) / Long Term Evolution (LongTerm Evolution, referred to as LTE), a fifth-generation mobile communication system (5G) and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (Near Field Communication, referred to as NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (Radio Frequency Identification, referred to as RFID) technology, infrared technology, ultra-wideband (Ultra Wide Band, referred to as UWB) technology, Bluetooth technology and other technologies.

[0195] The power supply assembly provides power to various components of the device where the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply assembly is located.

[0196] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0197] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in a dedicated integrated circuit. Of course, the processor and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0198] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0199] The order of the above-mentioned embodiments of the present application is only for description and does not represent the advantages and disadvantages of the embodiments. In addition, in some of the processes described in the above-mentioned embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed or executed in parallel in the order in which they appear in this article, and are only used to distinguish between different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or less operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit "first" and "second" to different types. The meaning of "multiple" is more than two, unless otherwise clearly and specifically defined.

[0200] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0201] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary technical means in the art that are not disclosed in the present application.

[0202] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An intelligent question-answering method, characterized in that: include: Acquire input query information and context information matched by the query information; Generate a task instruction and an additional context according to the query information and the context information, wherein the task instruction includes at least part of the context information, and the additional context includes part of the context information that is not included in the task instruction; The task instruction and the additional context are input into an extended task model, and a response result of the query information is generated by the extended task model according to the task instruction and the additional context.

2. The method according to claim 1, characterized in that The acquiring of input query information and context information matching the query information includes: The query information sent by the receiving terminal device is retrieved and enhanced according to the query information to obtain context information matching the query information; or, The query information sent by the receiving end-side device and the context information matching the query information.

3. The method according to claim 1, characterized in that The extended task model is obtained by extending and training a pre-trained decoder task model, and the task instructions and additional context are generated according to the query information and the context information, including: Generate a task instruction according to the query information and the context information, wherein the task instruction includes at least part of the context information, and the task instruction satisfies an input length restriction of the decoder task model; Part of the context information that is not included in the task instruction is used as the additional context.

4. The method according to claim 1, characterized in that: The extended task model includes an additional encoding module, an embedding layer, at least one extended transformer layer and an output layer, The extended task model is obtained by adding an additional encoding module to a pre-trained decoder task model and expanding at least one transformer layer in the decoder task model into a corresponding extended transformer layer. The decoder task model includes the embedding layer, the at least one transformer layer and the output layer.

5. The method according to claim 4, characterized in that The step of inputting the task instruction and the additional context into an extended task model, and generating a response result of the query information according to the task instruction and the additional context through the extended task model, includes: Input the additional context into the additional encoding module for encoding to obtain the additional context feature, and input the additional context feature into the at least one extended transformer layer; Inputting the task instruction into the embedding layer, mapping the task instruction into a query feature through the embedding layer, and inputting the query feature into the first extended transformer layer; Through the extended transformer layer, the query features input to this layer and the additional context features are fused and transformed to obtain the query features output by this layer, and the query features output by this layer are input to the next extended transformer layer or the output layer; The output layer generates a response result of the query information according to the query features output by the last extended transformer layer.

6. The method according to claim 5, characterized in that The extended transformer layer includes: a transformer block, an extension module based on a cross-attention mechanism, The query features input by the layer and the additional context features are fused and transformed through the extended transformer layer to obtain the query features output by the layer, including: Through the extension module, cross attention calculation is performed on the query features inputted in the current layer and the additional context features to obtain fusion features; The fused features are transformed through the transformer block to obtain the query features output by this layer.

7. The method according to claim 6, characterized in that The extended transformer layer further comprises: a mapping module, and the method further comprises: By means of the mapping module, the additional context feature is aligned with the query feature dimension of the current layer input to obtain the aligned additional context feature; The step of performing cross attention calculation on the query features inputted in the current layer and the additional context features through the extension module to obtain fusion features includes: Through the expansion module, cross-attention calculation is performed on the query features input into the current layer and the aligned additional context features to obtain fused features.

8. The method according to any one of claims 1 to 7, characterized in that The training process of the extended task model includes: Expand the pre-trained decoder task model and construct the extended task model to be trained; Acquire training data, the training data including: a query sample, a reference response to the query sample, and context information matched by the query sample; Generate an instruction sample and an additional context sample according to the query sample, the reference response of the query sample, and the context information matched by the query sample, wherein the instruction sample includes at least part of the context information matched by the query sample, and the additional context sample includes part of the context information matched by the query sample that is not included in the instruction sample; Taking the instruction sample as an input sequence, inputting the additional context sample and the input sequence into the extended task model to be trained, and generating a prediction result of the input sequence according to the input sequence and the additional context sample by the extended task model to be trained; According to the prediction result of the input sequence, the parameters of the extended task model are adjusted to obtain a trained extended task model.

9. The method according to claim 8, characterized in that The extended task model to be trained comprises: an additional encoding module, an embedding layer, at least one extended transformer layer and an output layer, The additional encoding module is initialized using parameters of a pre-trained text encoder.

10. The method according to claim 9, characterized in that Also includes: During the training process of the extended task model, the parameters of the additional encoding module are fixed, and the parameters of the extended task model other than the parameters of the additional encoding module are adjusted.

11. The method according to claim 9, characterized in that Also includes: During the training process of the extended task model, in at least one previous training iteration, the parameters of the additional coding module are fixed, and the parameters of the extended task model except the parameters of the additional coding module are adjusted; in subsequent training iterations, all parameters of the extended task model are adjusted.

12. An intelligent question-answering method, characterized in that: include: Receive input query information; Performing retrieval enhancement according to the query information to obtain context information matching the query information; Generate a task instruction and an additional context according to the query information and the context information, wherein the task instruction includes at least part of the context information, and the additional context includes part of the context information that is not included in the task instruction; Inputting the task instruction and the additional context into an extended task model, and generating a response result of the query information according to the additional context and the task instruction through the extended task model; The response result is output.

13. An intelligent question-answering method, characterized in that: include: In response to a call request for an extended task model, a task instruction and an additional context are obtained, wherein the task instruction includes query information and partial information of context information matched by the query information, and the additional context includes partial information of the context information that is not included in the task instruction; Inputting the task instruction and the additional context into an extended task model, and generating a response result of the query information according to the additional context and the task instruction through the extended task model; The response result is output.

14. A server, characterized in that: include: at least one processor; as well as a memory communicatively coupled to the at least one processor; The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the server executes the method described in any one of claims 1-13.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the method according to any one of claims 1 to 13 is implemented.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

Images