Replay generation method, device and equipment of large language model based on retrieval enhancement

Abstract

The invention provides a reply generation method, device and equipment of a large language model based on retrieval enhancement, relates to the technical field of artificial intelligence, and aims to generate a high-quality reply statement. The method comprises the following steps: acquiring an initial knowledge feature vector sequence corresponding to input information; the input information at least comprises a query statement and a knowledge document retrieved according to the query statement; performing multi-stage attention interaction according to the initial knowledge feature vector sequence, and removing document knowledge feature vectors with low correlation with the query statement in different stages of attention interaction by adopting different discarding strategies according to the importance degree of the document knowledge feature vectors in the attention interaction process; obtaining a final knowledge feature vector sequence; each stage of attention interaction comprises at least one layer of attention interaction; and generating a reply statement of the query statement according to the final knowledge feature vector sequence.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device and apparatus for generating responses based on a retrieval-enhanced large language model. Background Art

[0002] The large language model, with its superior text understanding and generation capabilities, has greatly changed the way people obtain information on the Internet. Users have gradually shifted from relying on traditional search engines to directly communicating with large language models to meet their information needs. In terms of information provision, the current large language model lacks timeliness because its world knowledge is fixed in the model parameters. At the same time, because the information is generated by calculating the model parameters rather than citing reliable external knowledge sources, the quality and reliability of the responses are also not guaranteed.

[0003] In related technologies, the generation technology of retrieval enhancement introduces a retrieval system for the large language model, regards user input as a query, retrieves relevant information from the knowledge base, and splices this information with the query statement as external knowledge and provides it to the large language model as a new input to improve the quality and timeliness of its responses. Considering the time and space consumption brought by a large amount of external knowledge to the large language model, some retrieval enhancement technologies propose to introduce additional compression modules to pre-compress the excessively long external knowledge and provide the refined external knowledge to the large language model. This improves the information density of external knowledge, while reducing the time complexity of the retrieval generation model in processing external knowledge and improving model efficiency.

[0004] However, although the retrieval-enhanced large language model technology that introduces a compression module has achieved certain results in improving the quality of large language model responses, the introduction of an additional compression module requires training to adapt to the downstream large language model, which affects the overall consistency and generalization of the retrieval generation model framework; moreover, information compression based on an additional generation model will cause external knowledge to be edited twice, thereby introducing the potential risk of information distortion, which in turn affects the reliability of responses generated by the large language model. Summary of the invention

[0005] In view of the above problems, embodiments of the present application provide a method, apparatus and device for generating responses based on a retrieval-enhanced large language model, so as to overcome the above problems or at least partially solve the above problems.

[0006] In a first aspect of an embodiment of the present application, a method for generating a reply based on a retrieval-enhanced large language model is disclosed, the method comprising:

[0007] Acquire an initial knowledge feature vector sequence corresponding to input information; the input information at least includes: a query statement, and a knowledge document retrieved according to the query statement; the initial knowledge feature vector sequence at least includes: a query knowledge feature vector corresponding to the query statement, and a document knowledge feature vector corresponding to the knowledge document;

[0008] Performing multi-stage attention interaction according to the initial knowledge feature vector sequence, and according to the importance of the document knowledge feature vector in the attention interaction process, using different discarding strategies at different stages of attention interaction to remove the document knowledge feature vector with low relevance to the query statement, so as to obtain a final knowledge feature vector sequence; each stage of attention interaction includes at least one layer of attention interaction;

[0009] A reply statement to the query statement is generated according to the final knowledge feature vector sequence.

[0010] Optionally, the large language model performs multi-layer attention interaction on the input knowledge feature vector sequence based on a multi-head attention mechanism at each stage of attention interaction, and the importance of the document knowledge feature vector at each layer of attention interaction is calculated in the following manner, including:

[0011] Calculate the attention value of the query knowledge feature vector for the document knowledge feature vector on each attention head;

[0012] The attention values ​​of all query knowledge feature vectors for the document knowledge feature vector on all attention heads are accumulated to obtain the importance of the document knowledge feature vector.

[0013] Optionally, calculating the attention value of the query knowledge feature vector for the document knowledge feature vector on each attention head includes:

[0014] According to the query knowledge feature vector and the query mapping matrix of the attention head, a query representation of the query knowledge feature vector on the attention head is obtained, and according to the key mapping matrix of the query knowledge feature vector and the attention head, a key representation of the query knowledge feature vector on the attention head is obtained;

[0015] According to the query representation and the key representation, an attention value of the query knowledge feature vector with respect to the document knowledge feature vector on each attention head is obtained.

[0016] Optionally, a multi-stage attention interaction is performed according to the initial knowledge feature vector sequence, and according to the importance of the document knowledge feature vector in the attention interaction process, different discard strategies are used in the attention interaction at different stages to remove the document knowledge feature vector with low relevance to the query statement, so as to obtain a final knowledge feature vector sequence, including:

[0017] Performing a first-stage attention interaction according to the initial knowledge feature vector sequence to obtain a first knowledge feature vector sequence;

[0018] Determine the importance of the document knowledge feature vector according to the attention value of the query knowledge feature vector to the document knowledge feature vector during the attention interaction process;

[0019] According to the importance of the document knowledge feature vector, the document knowledge feature vectors with low relevance to the query statement in the first knowledge feature vector sequence are removed to obtain a second knowledge feature vector sequence;

[0020] A second-stage attention interaction and a third-stage attention interaction are performed according to the second knowledge feature vector sequence to obtain a final knowledge feature vector sequence.

[0021] Optionally, performing a second-stage attention interaction and a third-stage attention interaction according to the second knowledge feature vector sequence to obtain a final knowledge feature vector sequence includes:

[0022] Performing a second-stage attention interaction according to the second knowledge feature vector sequence to obtain a third knowledge feature vector sequence;

[0023] Remove all document knowledge feature vectors from the third knowledge feature vector sequence to obtain a fourth knowledge feature vector sequence;

[0024] A third stage of attention interaction is performed according to the fourth knowledge feature vector sequence to obtain a final knowledge feature vector sequence.

[0025] Optionally, performing a first-stage attention interaction according to the initial knowledge feature vector sequence to obtain a first knowledge feature vector sequence includes:

[0026] Perform multi-layer attention interaction in sequence according to the initial knowledge feature vector sequence to obtain a knowledge feature vector sequence of each layer;

[0027] According to the similarity of the knowledge feature vector sequences of two adjacent layers, the stability value of the knowledge feature vector is determined;

[0028] When the stability value of the knowledge feature vector is greater than the stability threshold, the knowledge feature vector sequence corresponding to the attention interaction of this layer is determined as the first knowledge feature vector sequence.

[0029] Optionally, performing a second-stage attention interaction according to the second knowledge feature vector sequence to obtain a third knowledge feature vector sequence includes:

[0030] Performing multi-layer attention interaction in sequence according to the second knowledge feature vector sequence, obtaining a knowledge feature vector sequence of each layer;

[0031] Calculate the sum of the attention values ​​of the query knowledge feature vector to the document knowledge feature vector in the knowledge feature vector sequence of each layer;

[0032] When the sum of the attention values ​​is less than the attention threshold, the knowledge feature vector sequence corresponding to the attention interaction of this layer is determined as the third knowledge feature vector sequence.

[0033] Optionally, according to the importance of the document knowledge feature vector, the document knowledge feature vectors with low relevance to the query statement in the first knowledge feature vector sequence are removed to obtain a second knowledge feature vector sequence, including:

[0034] Sorting the document knowledge feature vectors in the first knowledge feature vector sequence in descending order of importance of the document knowledge feature vectors to obtain a sorting result;

[0035] According to the ranking result, a portion of the document knowledge feature vectors whose cumulative importance results account for a previous ratio threshold are retained, and the remaining document knowledge feature vectors are removed from the first knowledge feature vector sequence to obtain a second knowledge feature vector sequence.

[0036] In a second aspect of an embodiment of the present application, a reply generation device based on a retrieval-enhanced large language model is disclosed, the device comprising:

[0037] An acquisition module is used to acquire an initial knowledge feature vector sequence corresponding to input information; the input information at least includes: a query statement, and a knowledge document retrieved according to the query statement; the initial knowledge feature vector sequence at least includes: a query knowledge feature vector corresponding to the query statement, and a document knowledge feature vector corresponding to the knowledge document;

[0038] An interaction module is used to perform multi-stage attention interaction according to the initial knowledge feature vector sequence, and according to the importance of the document knowledge feature vector in the attention interaction process, different discard strategies are used in different stages of attention interaction to remove the document knowledge feature vector with low relevance to the query statement, so as to obtain a final knowledge feature vector sequence; each stage of attention interaction includes at least one layer of attention interaction;

[0039] A generation module is used to generate a reply statement of the query statement according to the final knowledge feature vector sequence.

[0040] According to a third aspect of an embodiment of the present application, an electronic device is disclosed, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method for generating a reply based on a retrieval-enhanced large language model described in the first aspect of the embodiment of the present application are implemented.

[0041] The embodiments of the present application include the following advantages:

[0042] In an embodiment of the present application, during the multi-stage attention interaction of the large language model on the initial knowledge feature vector sequence, different discard strategies are used in the attention interaction at different stages to remove the document knowledge feature vectors with low relevance to the query statement according to the importance of the document knowledge feature vector, thereby achieving redundant information filtering without consuming additional computing resources. By removing the document knowledge feature vectors with low relevance to the query statement, the computational complexity and video memory resource consumption of the large language model are improved. In addition, considering the inter-layer distribution differences of the attention interaction at different layers of the large language model, different discard strategies are used in the attention interaction at different stages to remove the document knowledge feature vectors with low relevance to the query statement, thereby better balancing the response quality and model processing efficiency of the large language model based on retrieval enhancement to generate high-quality response statements. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0044] Figure 1 It is a flowchart of the steps of a method for generating a reply based on a retrieval-enhanced large language model provided in an embodiment of the present application;

[0045] Figure 2 It is an overall architecture diagram of a reply generation method based on a retrieval-enhanced large language model provided in an embodiment of the present application;

[0046] Figure 3 It is a structural schematic diagram of a reply generation device based on a retrieval-enhanced large language model provided in an embodiment of the present application;

[0047] Figure 4 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and understandable, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0049] The retrieval-enhanced large language model technology aims to provide the large language model with timely, reliable and relevant external knowledge through the retrieval system, helping the large language model to generate high-quality response information. The widely used retrieval-enhanced model framework is to treat the query content raised by the user to the large language model as a query statement and find relevant documents through the retrieval model. Then, the document content and the query statement, as well as the instruction statement of the large language model, are spliced ​​together as the overall input of the large language model, so as to obtain the generated response of the large language model.

[0050] Excessively long external knowledge will lead to high consumption of time and space resources to process redundant information. Related technologies improve the quality of large language model responses by introducing retrieval-enhanced large language model technology with compression modules. However, the introduction of additional compression modules requires training to adapt to the downstream large language model, which affects the overall consistency and generalization of the retrieval generation model framework. Moreover, information compression based on additional generation models will cause external knowledge to be edited twice, thereby introducing potential risks of information distortion, which in turn affects the reliability of responses generated by the large language model.

[0051] In order to overcome the limitations of the related art, this application proposes the following technical ideas:

[0052] (1) The information value of the introduced external document knowledge is evaluated through the calculation data inside the large language model, so as to identify the document knowledge feature vectors with high information density (i.e., document knowledge feature vectors with high importance) and filter out redundant information without the need for additional computing resources; the computational complexity and memory resource consumption of the large language model are improved by discarding the document knowledge feature vectors with low information density.

[0053] (2) Taking advantage of the important position of the query knowledge feature vector corresponding to the query word in the dialogue process, the importance of the document knowledge feature vector is evaluated based on the attention distribution of the query knowledge feature vector, thereby discarding the low-quality document knowledge feature vectors in the knowledge feature vector sequence.

[0054] (3) Taking into account the differences in the inter-layer distribution of attention interactions at different layers of the large language model, different discard strategies are used in the attention interactions at different stages to remove document knowledge feature vectors with low relevance to the query statement, thereby better balancing the response quality and model processing efficiency of the retrieval-enhanced large language model to generate high-quality response statements.

[0055] In conjunction with the accompanying drawings, the reply generation method based on the retrieval-enhanced large language model provided in the embodiment of the present application is described in detail in Sections 1.1, 1.2, 1.3 and 1.4 respectively.

[0056] 1.1 Response generation method based on retrieval-enhanced large language model:

[0057] Reference Figure 1 As shown, Figure 1 1 is a flowchart of a method for generating a reply based on a large language model enhanced by retrieval provided in an embodiment of the present application. Figure 1 As shown, the reply generation method based on the retrieval-enhanced large language model may include steps S110 to S130:

[0058] Step S110: Obtain an initial knowledge feature vector sequence corresponding to input information; the input information includes at least: a query statement, and a knowledge document retrieved according to the query statement; the initial knowledge feature vector sequence includes at least: a query knowledge feature vector corresponding to the query statement, and a document knowledge feature vector corresponding to the knowledge document.

[0059] In the embodiment of the present application, input information refers to information used to input a large language model; wherein a query statement refers to a query statement raised by a user (i.e., description information of a question), and a knowledge document is an external knowledge document introduced according to the query statement. For a query statement, a relevant document set is retrieved in the knowledge base through a retrieval model, and the document set is spliced ​​together to form a knowledge document.

[0060] Each knowledge feature vector in the initial knowledge feature vector sequence is a knowledge token, the knowledge token corresponding to the query statement is called a query knowledge feature vector, and the knowledge token corresponding to the knowledge document is called a document knowledge feature vector. Specifically, obtaining the initial knowledge feature vector sequence corresponding to the input information includes: converting the input information into a string of knowledge tokens through the tokenization mechanism of the large language model, and mapping it into an implicit representation sequence, i.e., the initial knowledge feature vector sequence, through the embedding layer.

[0061] In some embodiments, the input information also includes a large language model instruction, and the input information of the large language model is formed by splicing the large language model instruction, the query statement and the knowledge document. Among them, the role of the large language model instruction is to help the large language model understand the query statement, so as to provide an accurate answer or perform a specific task. Correspondingly, the initial knowledge feature vector sequence includes: the query knowledge feature vector corresponding to the query statement, the document knowledge feature vector corresponding to the knowledge document, and the instruction knowledge feature vector corresponding to the large language model instruction.

[0062] Step S120: performing multi-stage attention interaction according to the initial knowledge feature vector sequence, and according to the importance of the document knowledge feature vector in the attention interaction process, adopting different discarding strategies in different stages of attention interaction to remove the document knowledge feature vector with low relevance to the query statement, so as to obtain the final knowledge feature vector sequence; each stage of attention interaction includes at least one layer of attention interaction.

[0063] In the embodiment of the present application, the large language model includes multiple layers of attention interaction. Considering the inter-layer distribution difference of attention interaction in different layers of the large language model, that is, in different attention interaction layers, the attention distribution of the query knowledge feature vector on the document knowledge feature vector is different. Therefore, the multi-layer attention interaction of the large language model is divided into different stages, and each stage of attention interaction includes at least one layer of attention interaction.

[0064] Different discard strategies are used in different stages of attention interaction to remove document knowledge feature vectors with low relevance to the query statement; among them, the importance of the document knowledge feature vector is determined based on the attention distribution of the query knowledge feature vector generated during the attention interaction of the large language model on the document knowledge feature vector. The importance of the document knowledge feature vector represents the information value of the document knowledge feature vector. The higher the importance of the document knowledge feature vector, the higher the information value of the document knowledge feature vector, and the lower the importance of the document knowledge feature vector, the lower the information value of the document knowledge feature vector. Therefore, the document knowledge feature vector with high information value can be identified based on the importance of the document knowledge feature vector.

[0065] In order to balance the response quality and model processing efficiency of the large language model based on retrieval enhancement, different abandonment strategies are adopted in the attention interaction at different stages to remove the document knowledge feature vectors with low relevance to the query statement, that is, the document knowledge feature vectors with low information value (low importance) are removed at different stages, so that attention interaction is performed based on the document knowledge feature vectors with high importance at each stage, and there is no need to process the document knowledge feature vectors with low importance, thereby increasing the computational complexity and memory resource consumption of the large language model.

[0066] Step S130: generating a reply statement to the query statement according to the final knowledge feature vector sequence.

[0067] In the embodiment of the present application, the final knowledge feature vector sequence refers to a knowledge feature vector sequence that has fully learned the content of the knowledge document and understood the user's intention, and a reply statement that meets the query statement intention can be generated through the final knowledge feature vector sequence. Specifically, based on the final knowledge feature vector sequence, a reply statement for the query statement is generated, including: mapping the final knowledge feature vector sequence to the word table space through the large language model head, obtaining the predicted probability distribution of the next knowledge feature vector, thereby iteratively generating a reply of the large language model, that is, generating a reply statement for the query statement.

[0068] Through the above implementation process, in the process of multi-stage attention interaction of the large language model on the initial knowledge feature vector sequence, different abandonment strategies are used at different stages of attention interaction to remove document knowledge feature vectors with low relevance to the query statement according to the importance of the document knowledge feature vector, so as to achieve redundant information filtering without consuming additional computing resources. By removing document knowledge feature vectors with low relevance to the query statement, the computational complexity and video memory resource consumption of the large language model are improved. In addition, considering the inter-layer distribution differences of attention interaction at different layers of the large language model, different abandonment strategies are used at different stages of attention interaction to remove document knowledge feature vectors with low relevance to the query statement, so as to better balance the response quality and model processing efficiency of the large language model based on retrieval enhancement, so as to generate high-quality response statements.

[0069] The following takes a user-large language model interaction as an example to illustrate the process of the reply generation method based on the retrieval-enhanced large language model.

[0070] Specifically, for a query statement q proposed by a user, the retrieval model R is used to first retrieve the relevant document set D in the knowledge base C, and the document set D is spliced ​​together to form a knowledge document T. Then, the knowledge document T, the query statement q and the large language model instruction Inst are spliced ​​together to form the input information of the large language model LLM, that is, the input information I = [Inst, T, q]. Finally, the large language model generates a reply statement r to the query statement according to the method of steps S110 to S130 above.

[0071] For example, the reply statement r of the query statement is expressed as:

[0072] r=LLM([Inst,T,q]),

[0073] When the large language model processes the input information, it first converts the input information into a string of knowledge tokens through the tokenization mechanism of the large language model, and maps it to the initial knowledge feature vector sequence through the embedding layer. After that, through the multi-head attention mechanism and forward feedback processing in the Transformer layer (i.e., the attention interaction layer) based on the causal mask, a new layer of knowledge feature vector sequence that integrates the contextual semantics of the input text is obtained; and different discard strategies are used in the attention interaction at different stages to remove document knowledge feature vectors with low relevance to the query statement. Finally, the final knowledge feature vector sequence is mapped to the vocabulary space through the language model head to obtain the probability distribution of the next predicted knowledge feature vector, thereby iteratively generating the final response of the large language model, that is, the response sentence r of the query statement.

[0074] 1.2 Importance of document knowledge feature vector:

[0075] In the embodiment of the present application, the key to balancing the generation effect and computational efficiency of the large language model is how to evaluate the value of information in the knowledge document. Since the large language model is based on the multi-head attention mechanism and forward feedback processing in the Transformer layer of the causal mask, and the design principle of the attention mechanism in the Transformer structure is to integrate the contextual information important for the understanding and semantic modeling of the current word, the attention mechanism is of great value for evaluating the importance of the document knowledge feature vector. Considering that the query statement contains the user's true intention, the degree of attention of the query knowledge feature vector corresponding to the query statement to the document knowledge feature vector can intuitively reflect the importance of the document knowledge feature vector. Therefore, the embodiment of the present application evaluates the importance of knowledge document information based on the cumulative attention value of the document knowledge feature vector.

[0076] In a specific implementation, the large language model performs multi-layer attention interaction on the input knowledge feature vector sequence based on a multi-head attention mechanism at each stage of attention interaction, and the importance of the document knowledge feature vector at each layer of attention interaction is calculated in the following manner, including step A1 and step A2:

[0077] Step A1: Calculate the attention value of the query knowledge feature vector for the document knowledge feature vector on each attention head.

[0078] Specifically, based on the query mapping matrix of the query knowledge feature vector and the attention head, the query representation of the query knowledge feature vector on the attention head is obtained, and, based on the key mapping matrix of the query knowledge feature vector and the attention head, the key representation of the query knowledge feature vector on the attention head is obtained; based on the query representation and the key representation, the attention value of the query knowledge feature vector for the document knowledge feature vector on each attention head is obtained.

[0079] For example, the query representation of the query knowledge feature vector on the attention head and the key representation of the query knowledge feature vector on the attention head are respectively expressed as:

[0080]

[0081] in, represents the query representation (i.e., Q representation) of the i-th query knowledge feature vector on the m attention heads of the l-th layer Transformer of the large language model; represents the key representation (i.e., K representation) of the i-th query knowledge feature vector on the m attention heads of the l-th layer Transformer of the large language model; represents the implicit representation of the i-th query knowledge feature vector in the l-th layer Transformer of the large language model; Represents the query mapping matrix of the m attention heads of the l-th layer Transformer in the large language model; Represents the key mapping matrix of the m attention heads in the l-th layer Transformer of the large language model.

[0082] For example, the attention value of the query knowledge feature vector to the document knowledge feature vector on each attention head is expressed as:

[0083]

[0084] in, It represents the attention value of the i-th query knowledge feature vector to the j-th document knowledge feature vector in the m-th attention head of the l-th layer Transformer of the large language model.

[0085] Step A2: Accumulate the attention values ​​of all query knowledge feature vectors to the document knowledge feature vector on all attention heads to obtain the importance of the document knowledge feature vector.

[0086] For example, the importance of the document knowledge feature vector can be expressed as:

[0087]

[0088] Among them, m l (d j ) represents the importance of the document knowledge feature vector at the jth position in the knowledge feature vector sequence in the lth layer Transformer of the large language model; h represents the number of attentions in the multi-head attention mechanism of the large language model; |q| represents the length of the query knowledge feature vector corresponding to the query statement; q sIndicates the starting position of the query knowledge feature vector in the knowledge feature vector sequence, q e Indicates the end position of the query knowledge feature vector in the knowledge feature vector sequence.

[0089] In this way, the important position of the query knowledge feature vector in the dialogue process is utilized, and the importance of the document knowledge feature vector is evaluated based on the attention distribution of the query knowledge feature vector. Therefore, the information value of the introduced external document knowledge is evaluated through the calculation data inside the large language model, and the low-quality document knowledge feature vectors in the knowledge feature vector sequence are removed to achieve the filtering of redundant information without the need for additional computing resources. The computational complexity and memory resource consumption of the large language model are improved by discarding the document knowledge feature vectors with low information density.

[0090] 1.3 Abandonment strategy removes document knowledge feature vectors that have low relevance to the query statement:

[0091] Each layer of attention interaction (Transformer) in the large language model has different functions, and the resulting attention distribution is also very different. Specifically, the distribution of query attention on document knowledge feature vectors (document words) of each layer of the large language model was analyzed through prior experiments. The prior experiments counted the query word attention obtained by all document knowledge feature vectors in each layer of Transformer on multiple knowledge-intensive dialogue datasets. It was found that the query attention of document knowledge feature vectors on low-level Transformers was distributed more evenly, and discarding the knowledge feature vectors of this layer would introduce significant model performance loss. This means that the modeling of query knowledge feature vectors is still in the stage of integrating global context information at this time, so the importance of document knowledge feature vectors is relatively average, and document knowledge feature vectors cannot be discarded. On the high-level Transformer, the sum of query attention received by all document knowledge feature vectors decreases sharply, and is mainly concentrated on a small number of important document knowledge feature vectors. Discarding most of the low-attention document knowledge feature vectors has no obvious effect on model performance. Therefore, the embodiment of the present application believes that the internal processing mechanism of the large language model is a pyramid structure, that is, the global context information is first coarse-grainedly integrated at the low level, and the middle and high levels are more focused on deeply understanding and analyzing user intentions based on instruction content and query statements and making answers.

[0092] Therefore, in order to adapt to the distribution differences of different Transformer layers of a large language model, an embodiment of the present application proposes an inefficient word abandonment method that relies on the Transformer layer, that is, according to the importance of the document knowledge feature vector in the attention interaction process, different abandonment strategies are used in different stages of attention interaction to remove the document knowledge feature vectors with low relevance to the query statement, thereby significantly reducing the computational complexity and memory resource overhead of the large language model at the middle and high levels.

[0093] In a specific implementation, the above step S120 "performing multi-stage attention interaction according to the initial knowledge feature vector sequence, and according to the importance of the document knowledge feature vector in the attention interaction process, adopting different discarding strategies at different stages of attention interaction to remove the document knowledge feature vector with low relevance to the query statement, to obtain the final knowledge feature vector sequence" specifically includes steps B1 to B4:

[0094] Step B1: The initial knowledge feature vector sequence undergoes a first-stage attention interaction to obtain a first knowledge feature vector sequence.

[0095] Step B2: Determine the importance of the document knowledge feature vector according to the attention value of the query knowledge feature vector to the document knowledge feature vector during the attention interaction process.

[0096] Step B3: According to the importance of the document knowledge feature vector, the document knowledge feature vectors with low relevance to the query statement in the first knowledge feature vector sequence are removed to obtain a second knowledge feature vector sequence.

[0097] Step B4: Perform the second-stage attention interaction and the third-stage attention interaction according to the second knowledge feature vector sequence to obtain the final knowledge feature vector sequence.

[0098] In the embodiment of the present application, the first stage of attention interaction includes multi-layer attention interaction, and the bottom L1 layer of the large language model is divided into the first stage of attention interaction. The first stage of attention interaction is a coarse-grained global context fusion stage, and the importance of the document knowledge feature vector is evenly distributed. All document knowledge feature vectors are retained in the first stage of attention interaction, so that global information can be fully interactively modeled.

[0099] The first knowledge feature vector sequence refers to the knowledge feature vector sequence output after the first stage of attention interaction. For example, if the attention interaction from the first layer to the fifth layer of the large language model is the first stage of attention interaction, then the first knowledge feature vector sequence is the knowledge feature vector sequence output from the fifth layer of attention interaction.

[0100] After obtaining the first knowledge document, execute step B2. In each layer of attention interaction above the first stage of attention interaction, the sum of the query attention received by all document knowledge feature vectors is sharply reduced, and is mainly concentrated on a small number of important document knowledge feature vectors. Removing most of the low-attention document knowledge feature vectors has no obvious effect on the performance of the large language model. Therefore, according to the attention value of the query knowledge feature vector to the document knowledge feature vector during the attention interaction process, the importance of the document knowledge feature vector is determined, and then according to the importance of the document knowledge feature vector, the document knowledge feature vector with low relevance to the query statement in the first knowledge feature vector sequence is removed.

[0101] Among them, the second knowledge feature vector sequence does not contain document knowledge feature vectors with low importance. In the subsequent second-stage attention interaction and third-stage attention interaction process, attention interaction is performed based on the second knowledge feature vector sequence to deeply understand and analyze the user's intention and make answers based on the instruction content and query statement. In this way, the computational complexity and video memory resource overhead of the large language model at the middle and high levels (second-stage attention interaction and third-stage attention interaction) are significantly reduced.

[0102] In some optional embodiments, the above step B1 of "performing a first-stage attention interaction according to the initial knowledge feature vector sequence to obtain a first knowledge feature vector sequence" includes steps B11 to B13:

[0103] Step B11: Perform multi-layer attention interaction in sequence according to the initial knowledge feature vector sequence to obtain a knowledge feature vector sequence for each layer.

[0104] Step B12: Determine the knowledge feature vector stability value according to the similarity between the knowledge feature vector sequences of two adjacent layers.

[0105] Step B13: When the stability value of the knowledge feature vector is greater than the stability threshold, the knowledge feature vector sequence corresponding to the attention interaction of this layer is determined as the first knowledge feature vector sequence.

[0106] In an embodiment of the present application, the end of the first stage of attention interaction is based on the similarity evaluation of the knowledge feature vectors in two adjacent layers, that is, when the overall knowledge feature vector tends to be stable, it is considered that the coarse-grained global context has been fully modeled, and at this time, the document knowledge feature vectors with low importance (i.e., low information value) should be removed.

[0107] For example, the calculation method of the stability value of the knowledge feature vector is expressed as:

[0108]

[0109] in, represents the implicit representation of the i-th knowledge feature vector in the l-th layer Transformer of the large language model, and the i-th knowledge feature vector is one of the query knowledge feature vector, the document knowledge feature vector, and the instruction knowledge feature vector; represents the implicit representation of the i-th knowledge feature vector in the (l-1)th layer Transformer of the large language model; |I| represents the length of the knowledge feature vector sequence.

[0110] The stability threshold is set according to the actual situation. l When it is greater than the stability threshold, the first stage of attention interaction ends, and the corresponding layer output knowledge feature vector sequence is determined as the first knowledge feature vector sequence.

[0111] In this way, based on the similarity evaluation of the knowledge feature vectors in two adjacent layers, it is determined whether the first stage of attention interaction is ended. After the first stage of attention interaction is ended, a first knowledge feature vector sequence is obtained, and according to the importance of the document knowledge feature vectors, the document knowledge feature vectors with low relevance to the query statement in the first knowledge feature vector sequence are removed to obtain a second knowledge feature vector sequence, thereby reducing the computational complexity and video memory resource overhead of the large language model at the middle and high levels (second stage attention interaction and third stage attention interaction).

[0112] In some optional embodiments, in the above step B3, "removing document knowledge feature vectors with low relevance to the query statement in the first knowledge feature vector sequence according to the importance of the document knowledge feature vector to obtain a second knowledge feature vector sequence" specifically includes:

[0113] The document knowledge feature vectors in the first knowledge feature vector sequence are sorted in descending order of their importance to obtain a sorting result; based on the sorting result, a portion of the document knowledge feature vectors whose cumulative results of the importance of the document knowledge feature vectors account for a previous proportion threshold are retained, and the remaining document knowledge feature vectors are removed from the first knowledge feature vector sequence to obtain a second knowledge feature vector sequence.

[0114] For example, based on the distribution of the query knowledge feature vector's attention on the document knowledge feature vector (i.e., the importance of the document knowledge feature vector), the document knowledge feature vectors occupying the first b (such as b=85%) are selected to be retained, and the remaining document knowledge feature vectors are removed to obtain a second knowledge feature vector sequence, so that the large language model can further focus on the modeling and understanding of important information.

[0115] For example, the calculation method for retaining and removing document knowledge feature vectors is:

[0116] D = ID U UD,

[0117]

[0118] Among them, m L1 (d i ) indicates that the document knowledge feature vector ranked at the i-th position in the first knowledge feature vector sequence is at the L-th position. 1 The importance of the layer (i.e., the last layer of the first stage attention interaction); m L1 (d j ) indicates that the document knowledge feature vector at the jth position in the first knowledge feature vector sequence is in the Lth position. 1 The importance of the layer; D represents all document knowledge feature vectors in the first document knowledge feature vector sequence; ID represents the set of retained document knowledge feature vectors; UD represents the set of removed document knowledge feature vectors.

[0119] Specifically, the calculation method for removing the document knowledge feature vectors with low relevance to the query statement in the first knowledge feature vector sequence is expressed as:

[0120] H L1 =Drop(H L1 , UD),

[0121] Among them, H L1 represents the first knowledge feature vector sequence, Drop represents a removal operation, and after the first knowledge feature vector sequence is subjected to the removal operation, a second knowledge feature vector sequence is obtained.

[0122] In some optional embodiments, the above step B4 of "performing a second-stage attention interaction and a third-stage attention interaction according to the second knowledge feature vector sequence to obtain a final knowledge feature vector sequence" includes steps B41 to B43:

[0123] Step B41: Perform a second-stage attention interaction based on the second knowledge feature vector sequence to obtain a third knowledge feature vector sequence.

[0124] Step B42: remove all document knowledge feature vectors in the third knowledge feature vector sequence to obtain a fourth knowledge feature vector sequence.

[0125] Step B43: Perform the third stage of attention interaction according to the fourth knowledge feature vector sequence to obtain a final knowledge feature vector sequence.

[0126] In the embodiment of the present application, the large language model pays more attention to deeply understanding and analyzing the user's intention based on the instruction content and query sentence and making answers at the middle and high levels. The multi-layer attention interaction above the first stage attention interaction of the large language model is divided into the second stage attention interaction and the third stage attention interaction.

[0127] Among them, the second-stage attention interaction includes multi-layer attention interaction. The second-stage attention interaction is the important external information modeling stage. In this stage, the importance of document knowledge feature vectors is concentrated in a part of document knowledge feature vectors. Therefore, multi-layer attention interaction is performed based on the second knowledge feature vector sequence (excluding document knowledge feature vectors with low importance) to obtain the third knowledge feature vector sequence.

[0128] After the second stage of attention interaction, the large language model has fully mastered the knowledge in the knowledge document. At this time, all document knowledge feature vectors in the third knowledge feature vector sequence are removed to obtain the fourth knowledge feature vector sequence, and the third stage of attention interaction is performed based on the fourth knowledge feature vector sequence. The third stage of attention interaction includes multi-layer attention interaction. The third stage of attention is the user intent understanding and analysis stage. The large language model focuses on solving the information processing modeling of the current problem by performing multi-layer attention interaction on the fourth knowledge feature vector sequence.

[0129] In some optional embodiments, a second stage of attention interaction is performed according to the second knowledge feature vector sequence to obtain a third knowledge feature vector sequence, including: performing multiple layers of attention interaction in sequence according to the second knowledge feature vector sequence to obtain a knowledge feature vector sequence of each layer; calculating the sum of attention values ​​of the query knowledge feature vector for the document knowledge feature vector in the knowledge feature vector sequence of each layer; and when the sum of the attention values ​​is less than an attention threshold, determining the knowledge feature vector sequence corresponding to the attention interaction of that layer as the third knowledge feature vector sequence.

[0130] In the embodiment of the present application, the sum of the query attention received by all document knowledge feature vectors in each layer above the first stage of attention interaction of the large language model is sharply reduced. When the sum of the attention values ​​of a certain layer is less than the attention threshold, it means that the large language model has completed the modeling of high-importance document words, and it is believed that the large language model has fully mastered the knowledge in the knowledge document. At this time, the second stage of attention interaction is ended, and the knowledge feature vector sequence output by the attention interaction of this layer is determined as the third knowledge feature vector sequence.

[0131] For example, the second stage of attention interaction starts from the 6th layer of attention interaction. If the sum of the attention values ​​of the query knowledge feature vector for the document knowledge feature vector in the knowledge feature vector sequence of the 10th layer is less than the attention threshold, the knowledge feature vector sequence corresponding to the 10th layer is determined as the third knowledge feature vector sequence.

[0132] For example, the sum of the attention values ​​of the query knowledge feature vector to the document knowledge feature vector in the knowledge feature vector sequence of each layer is calculated as:

[0133]

[0134] Among them, M 1 It represents the sum of the attention values ​​of the query knowledge feature vector in the first layer of attention interaction in the large language model to the document knowledge feature vector in the knowledge feature vector sequence (i.e., the sum of the importance of the document knowledge feature vector in the knowledge feature vector sequence of the first layer of attention interaction in the large language model); γ represents the attention threshold. In some embodiments, the value of γ can be 0.1.

[0135] In some embodiments, whether to end the second-stage attention interaction is determined based on the proportion of the sum of the attention of the document knowledge feature vectors in the total attention, that is, when the proportion of the sum of the attention of the document knowledge feature vectors in the total attention is less than a proportion threshold, the second-stage attention interaction is ended, and the knowledge feature vector sequence output by this layer of attention interaction is determined as the third knowledge feature vector sequence.

[0136] After the second stage of attention interaction, the large language model has fully mastered the knowledge in the knowledge document, and all document knowledge feature vectors in the third knowledge feature vector sequence are removed, so that the large language model can focus on information processing modeling to solve the current problem by performing multi-layer attention interaction on the fourth knowledge feature vector sequence. For example, the calculation of removing all document knowledge feature vectors in the third knowledge feature vector sequence can be expressed as:

[0137]

[0138] in, Represents the third knowledge feature vector sequence (i.e., the knowledge feature vector sequence output by the last layer of the second stage attention interaction), and the fourth knowledge feature vector sequence is obtained by removing all document knowledge feature vectors in the third knowledge feature vector sequence.

[0139] In this way, according to the difference in attention distribution of query knowledge feature vectors to document knowledge feature vectors at different layers of the large language model (i.e., the importance of query knowledge feature vectors to document knowledge feature vectors at different layers), the corresponding number of document knowledge feature vectors are removed at different stages, so that the large language model can focus on modeling document words with high importance in the second stage of attention interaction, and focus on information processing modeling to solve the current problem in the third stage of attention interaction, thereby reducing the computational complexity and video memory resource overhead of the large language model at the middle and high levels, and maximally balancing the response quality and model processing efficiency of the retrieval-enhanced large language model to generate high-quality response sentences.

[0140] For example, refer to Figure 2 As shown, Figure 2This is an overall architecture diagram of a reply generation method based on a retrieval-enhanced large language model provided in an embodiment of the present application. The process of generating a reply sentence by a large language model is as follows: first, the large language model instruction, query sentence and knowledge document are spliced ​​to obtain input information, so that the large language model generates a corresponding reply sentence based on the input information and internal model calculation.

[0141] Specifically, the large language model includes multiple layers of attention interaction (i.e., multiple layers of Transformer layers based on causal masks). The input information is first input into the wordization and word vectorization embedding layers for tokenization processing to obtain the initial knowledge feature vector sequence corresponding to the input information (including: query knowledge feature vector, document knowledge feature vector, instruction knowledge feature vector).

[0142] Next, the first stage of attention interaction (L 1 Layer attention interaction) is performed to achieve coarse-grained global context fusion; when the overall knowledge feature vector tends to be stable, it is considered that the coarse-grained global context has been fully modeled. At this time, the first stage of attention interaction is ended to obtain a first knowledge feature vector sequence, and according to the importance of the document knowledge feature vector, the document knowledge feature vector with low relevance to the query statement in the first knowledge feature vector sequence is removed to obtain a second knowledge feature vector sequence.

[0143] Then, the second stage of attention interaction (L 2 layer) to model the document knowledge feature vectors with high importance; when the sum of the attention values ​​of a certain layer is less than the attention threshold, it is considered that the large language model has fully mastered the knowledge in the knowledge document. At this time, all document knowledge feature vectors in the third knowledge feature vector sequence are removed to obtain the fourth knowledge feature vector sequence.

[0144] Finally, the third stage of attention interaction (total L) is performed according to the fourth knowledge feature vector sequence. 3 Layer) to focus on information processing modeling to solve the current problem, and finally map the final knowledge feature vector sequence to the vocabulary space through the language model head to obtain the probability distribution of the next predicted knowledge feature vector, thereby iteratively generating the final response of the large language model.

[0145] The reply generation method based on the retrieval-enhanced large language model implemented in this application can maximize the balance between the reply quality and model processing efficiency of the retrieval-enhanced large language model and generate high-quality reply sentences.

[0146] 1.4 Time complexity analysis:

[0147] In order to better illustrate the performance of the reply generation method based on the retrieval-enhanced large language model of this application, the time complexity analysis is analyzed in Section 1.4.

[0148] Specifically, assuming that the overall input length of the large language model is N (that is, the length of the input information is N), the length of the knowledge document is Nd, and the proportion of its total length is α α. The ratio of the number of document knowledge feature vectors removed in the first knowledge feature vector sequence to the total number of document knowledge feature vectors is β, and the number of layers of the large language model is L. The time complexity C of the original model forward processing (only considering the generation of the first word) is expressed as:

[0149] C=O(N 2 L),

[0150] The time complexity of the reply generation method based on the retrieval-enhanced large language model provided in the embodiment of the present application is It is expressed as:

[0151]

[0152] Let L 1 =nL,L 2 =mL; m, n<1, we can get:

[0153]

[0154] Through statistical analysis, we can get n≈0.625, m≈0.156, β≈0.7, α≈0.85, then That is, the time complexity of the reply generation method based on the retrieval-enhanced large language model is about 66% of the original one.

[0155] Therefore, the reply generation method based on the retrieval-enhanced large language model provided in the embodiment of the present application can significantly improve the time complexity of the model while ensuring the performance of the retrieval-enhanced large language model. Table 1 shows the comparison of the model effect (F1, EM) and reasoning speed (query delay) of the retrieval-enhanced model (RAG) composed of the retriever BGE-base and the large language model Llama2-7B under the reply generation method based on the retrieval-enhanced large language model of the present application (RAG-DDT). It can be seen from Table 1 that in different datasets (KILT-NQ, KILT-HotpotQA, and KILT-TriviaQA are three datasets), the reply generation method based on the retrieval-enhanced large language model of the present application can significantly improve the query delay time.

[0156] Table 1 Performance comparison results of different methods

[0157]

[0158] The present application also provides a reply generation device based on a retrieval-enhanced large language model, referring to Figure 3 As shown, Figure 3 : is a structural schematic diagram of a reply generation device based on a retrieval-enhanced large language model provided in an embodiment of the present application, the device comprising:

[0159] The acquisition module 310 is used to acquire an initial knowledge feature vector sequence corresponding to input information; the input information at least includes: a query statement, and a knowledge document retrieved according to the query statement; the initial knowledge feature vector sequence at least includes: a query knowledge feature vector corresponding to the query statement, and a document knowledge feature vector corresponding to the knowledge document;

[0160] The interaction module 320 is used to perform multi-stage attention interaction according to the initial knowledge feature vector sequence, and according to the importance of the document knowledge feature vector in the attention interaction process, adopt different discarding strategies in different stages of attention interaction to remove the document knowledge feature vector with low relevance to the query statement, so as to obtain a final knowledge feature vector sequence; each stage of attention interaction includes at least one layer of attention interaction;

[0161] The generation module 330 is used to generate a reply statement of the query statement according to the final knowledge feature vector sequence.

[0162] In an optional embodiment, the large language model performs multi-layer attention interaction on the input knowledge feature vector sequence based on a multi-head attention mechanism at each stage of attention interaction, and the device further includes an importance calculation module for calculating the importance of the document knowledge feature vector at each layer of attention interaction; the importance calculation module includes:

[0163] A first calculation module is used to calculate the attention value of the query knowledge feature vector for the document knowledge feature vector on each attention head;

[0164] The accumulation module is used to accumulate the attention values ​​of all query knowledge feature vectors to the document knowledge feature vector on all attention heads to obtain the importance of the document knowledge feature vector.

[0165] In an optional embodiment, the first calculation module includes:

[0166] A representation calculation module, configured to obtain a query representation of the query knowledge feature vector on the attention head according to the query knowledge feature vector and the query mapping matrix of the attention head, and to obtain a key representation of the query knowledge feature vector on the attention head according to the query knowledge feature vector and the key mapping matrix of the attention head;

[0167] The attention value calculation module is used to obtain the attention value of the query knowledge feature vector to the document knowledge feature vector on each attention head according to the query representation and the key representation.

[0168] In an optional embodiment, the interaction module includes:

[0169] A first attention interaction module, used for performing a first-stage attention interaction according to the initial knowledge feature vector sequence to obtain a first knowledge feature vector sequence;

[0170] A first determination module is used to determine the importance of the document knowledge feature vector according to the attention value of the query knowledge feature vector to the document knowledge feature vector during the attention interaction process;

[0171] A first removal module, configured to remove document knowledge feature vectors having low relevance to the query statement from the first knowledge feature vector sequence according to the importance of the document knowledge feature vector, to obtain a second knowledge feature vector sequence;

[0172] The second attention interaction module is used to perform second-stage attention interaction and third-stage attention interaction according to the second knowledge feature vector sequence to obtain a final knowledge feature vector sequence.

[0173] In an optional embodiment, the second attention interaction module includes:

[0174] A first interaction submodule, configured to perform a second-stage attention interaction according to the second knowledge feature vector sequence to obtain a third knowledge feature vector sequence;

[0175] A second removal module is used to remove all document knowledge feature vectors in the third knowledge feature vector sequence to obtain a fourth knowledge feature vector sequence;

[0176] The second interaction submodule is used to perform a third-stage attention interaction according to the fourth knowledge feature vector sequence to obtain a final knowledge feature vector sequence.

[0177] In an optional embodiment, the first attention interaction module includes:

[0178] A first attention interaction submodule is used to perform multi-layer attention interaction in sequence according to the initial knowledge feature vector sequence to obtain a knowledge feature vector sequence of each layer;

[0179] A second determination module is used to determine the stability value of the knowledge feature vector according to the similarity of the knowledge feature vector sequences of two adjacent layers;

[0180] The third determination module is used to determine the knowledge feature vector sequence corresponding to the attention interaction of this layer as the first knowledge feature vector sequence when the stability value of the knowledge feature vector is greater than the stability threshold.

[0181] In an optional embodiment, the first interaction submodule includes:

[0182] A second attention interaction module, used to perform multi-layer attention interaction in sequence according to the second knowledge feature vector sequence to obtain a knowledge feature vector sequence of each layer;

[0183] The second calculation module is used to calculate the sum of attention values ​​of the query knowledge feature vector to the document knowledge feature vector in the knowledge feature vector sequence of each layer;

[0184] The fourth determination module is used to determine the knowledge feature vector sequence corresponding to the attention interaction of this layer as the third knowledge feature vector sequence when the total of the attention values ​​is less than the attention threshold.

[0185] In an optional embodiment, the first removal module includes:

[0186] A sorting module, used to sort the document knowledge feature vectors in the first knowledge feature vector sequence in descending order of importance of the document knowledge feature vectors to obtain a sorting result;

[0187] The first removal submodule is used to retain a part of the document knowledge feature vectors whose cumulative results of the importance of the document knowledge feature vectors account for a previous proportion threshold according to the sorting result, and remove the remaining document knowledge feature vectors from the first knowledge feature vector sequence to obtain a second knowledge feature vector sequence.

[0188] The present application also provides an electronic device, referring to Figure 4 , Figure 4 Schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 4 As shown, the electronic device 400 includes: a memory 410 and a processor 420. The memory 410 and the processor 420 are connected via a bus communication. A computer program is stored in the memory 4410. The computer program can be run on the processor 420 to implement the steps of the reply generation method based on the retrieval-enhanced large language model described in the embodiment of the present application.

[0189] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for generating a reply based on a retrieval-enhanced large language model described in the embodiment of the present application are implemented.

[0190] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0191] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods and devices according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0192] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0193] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0194] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0195] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0196] The above is a detailed introduction to the reply generation method, device and equipment based on a retrieval-enhanced large language model provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A reply generation method based on a retrieval-enhanced large language model, characterized in that: include: Obtaining an initial knowledge feature vector sequence corresponding to the input information; The input information at least includes: a query statement, and a knowledge document retrieved according to the query statement; the initial knowledge feature vector sequence at least includes: a query knowledge feature vector corresponding to the query statement, and a document knowledge feature vector corresponding to the knowledge document; Performing multi-stage attention interaction according to the initial knowledge feature vector sequence, and according to the importance of the document knowledge feature vector in the attention interaction process, using different discarding strategies at different stages of attention interaction to remove the document knowledge feature vector with low relevance to the query statement, so as to obtain a final knowledge feature vector sequence; each stage of attention interaction includes at least one layer of attention interaction; A reply statement to the query statement is generated according to the final knowledge feature vector sequence.

2. The method according to claim 1, characterized in that The large language model performs multi-layer attention interaction on the input knowledge feature vector sequence at each stage of attention interaction based on the multi-head attention mechanism. The importance of the document knowledge feature vector at each layer of attention interaction is calculated in the following manner, including: Calculate the attention value of the query knowledge feature vector for the document knowledge feature vector on each attention head; The attention values ​​of all query knowledge feature vectors for the document knowledge feature vector on all attention heads are accumulated to obtain the importance of the document knowledge feature vector.

3. The method according to claim 2, characterized in that Calculating the attention value of the query knowledge feature vector for the document knowledge feature vector on each attention head includes: According to the query knowledge feature vector and the query mapping matrix of the attention head, a query representation of the query knowledge feature vector on the attention head is obtained, and according to the key mapping matrix of the query knowledge feature vector and the attention head, a key representation of the query knowledge feature vector on the attention head is obtained; According to the query representation and the key representation, an attention value of the query knowledge feature vector with respect to the document knowledge feature vector on each attention head is obtained.

4. The method according to claim 1, characterized in that: A multi-stage attention interaction is performed according to the initial knowledge feature vector sequence, and according to the importance of the document knowledge feature vector in the attention interaction process, different discard strategies are used in different stages of attention interaction to remove the document knowledge feature vector with low relevance to the query statement, so as to obtain a final knowledge feature vector sequence, including: Performing a first-stage attention interaction according to the initial knowledge feature vector sequence to obtain a first knowledge feature vector sequence; Determine the importance of the document knowledge feature vector according to the attention value of the query knowledge feature vector to the document knowledge feature vector during the attention interaction process; According to the importance of the document knowledge feature vector, the document knowledge feature vectors with low relevance to the query statement in the first knowledge feature vector sequence are removed to obtain a second knowledge feature vector sequence; A second-stage attention interaction and a third-stage attention interaction are performed according to the second knowledge feature vector sequence to obtain a final knowledge feature vector sequence.

5. The method according to claim 4, characterized in that The second stage attention interaction and the third stage attention interaction are performed according to the second knowledge feature vector sequence to obtain a final knowledge feature vector sequence, including: Performing a second-stage attention interaction according to the second knowledge feature vector sequence to obtain a third knowledge feature vector sequence; Remove all document knowledge feature vectors from the third knowledge feature vector sequence to obtain a fourth knowledge feature vector sequence; A third stage of attention interaction is performed according to the fourth knowledge feature vector sequence to obtain a final knowledge feature vector sequence.

6. The method according to claim 4, characterized in that Performing a first-stage attention interaction according to the initial knowledge feature vector sequence to obtain a first knowledge feature vector sequence, including: Perform multi-layer attention interaction in sequence according to the initial knowledge feature vector sequence to obtain a knowledge feature vector sequence of each layer; According to the similarity of the knowledge feature vector sequences of two adjacent layers, the stability value of the knowledge feature vector is determined; When the stability value of the knowledge feature vector is greater than the stability threshold, the knowledge feature vector sequence corresponding to the attention interaction of this layer is determined as the first knowledge feature vector sequence.

7. The method according to claim 5, characterized in that Performing a second-stage attention interaction according to the second knowledge feature vector sequence to obtain a third knowledge feature vector sequence, including: Performing multi-layer attention interaction in sequence according to the second knowledge feature vector sequence, obtaining a knowledge feature vector sequence of each layer; Calculate the sum of the attention values ​​of the query knowledge feature vector to the document knowledge feature vector in the knowledge feature vector sequence of each layer; When the sum of the attention values ​​is less than the attention threshold, the knowledge feature vector sequence corresponding to the attention interaction of this layer is determined as the third knowledge feature vector sequence.

8. The method according to claim 4, characterized in that According to the importance of the document knowledge feature vector, the document knowledge feature vectors with low relevance to the query statement in the first knowledge feature vector sequence are removed to obtain a second knowledge feature vector sequence, including: Sorting the document knowledge feature vectors in the first knowledge feature vector sequence in descending order of importance of the document knowledge feature vectors to obtain a sorting result; According to the ranking result, a portion of the document knowledge feature vectors whose cumulative importance results account for a previous ratio threshold are retained, and the remaining document knowledge feature vectors are removed from the first knowledge feature vector sequence to obtain a second knowledge feature vector sequence.

9. A reply generation device based on a retrieval-enhanced large language model, characterized in that: include: An acquisition module, used to acquire an initial knowledge feature vector sequence corresponding to the input information; The input information at least includes: a query statement, and a knowledge document retrieved according to the query statement; the initial knowledge feature vector sequence at least includes: a query knowledge feature vector corresponding to the query statement, and a document knowledge feature vector corresponding to the knowledge document; An interaction module is used to perform multi-stage attention interaction according to the initial knowledge feature vector sequence, and according to the importance of the document knowledge feature vector in the attention interaction process, different discard strategies are used in different stages of attention interaction to remove the document knowledge feature vector with low relevance to the query statement, so as to obtain a final knowledge feature vector sequence; each stage of attention interaction includes at least one layer of attention interaction; A generation module is used to generate a reply statement of the query statement according to the final knowledge feature vector sequence.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for generating a reply based on a retrieval-enhanced large language model as described in any one of claims 1 to 8 are implemented.

Images