Document processing method, document question and answer optimization method, electronic device and storage medium

Abstract

The embodiments of the present invention provide a document processing method, a document question-answering optimization method, an electronic device, and a storage medium, which relate to the technical field of natural language processing. By inputting the chapter encoding sequence of the preprocessed document chapter data into a large language model, calculating the first key vector sequence and the first value vector sequence of each network layer of the large language model, taking each group of the first key vector sequence and the first value vector sequence as an intermediate attention value of the document chapter data, calculating the weight score of each intermediate attention value, and according to the size of each weight score, eliminating a preset proportion of intermediate attention values, and taking the retained intermediate attention values ​​as a compact attention representation of the document chapter data. Therefore, when a user asks a question, an answer text can be generated according to the compact attention representation, which can improve the computational efficiency of the large language model, and thus significantly improve the speed and efficiency of the question-answering process.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a document processing method, a document question-and-answer optimization method, an electronic device, and a storage medium. Background Art

[0002] Document Question Answering (DocQA) is a natural language processing technology that allows users to ask questions about specific documents or document collections in natural language and automatically retrieve relevant information from these documents to form answers.

[0003] In recent years, with the development of deep learning technology, especially the application of large-scale pre-trained models, the performance of document question answering systems has been significantly improved. These models are pre-trained on large amounts of text data and can learn rich semantic representation capabilities, thereby better understanding the relationship between document content and user questions.

[0004] However, in the related art, when processing long documents, the large language model needs to encode the entire document, which greatly increases the computational complexity, resulting in very high computational cost of document question answering based on the large language model. Summary of the invention

[0005] In view of this, an object of embodiments of the present invention is to provide a document processing method, a document question and answer optimization method, an electronic device and a storage medium to at least partially improve the above-mentioned problems.

[0006] In order to achieve the above purpose, the technical solution adopted by the embodiment of the present invention is as follows:

[0007] In a first aspect, an embodiment of the present invention provides a document processing method, the method comprising:

[0008] Inputting the preprocessed chapter encoding sequence of the document chapter data into the large language model, calculating the first key vector sequence and the first value vector sequence of each network layer of the large language model, and taking each group of the first key vector sequence and the first value vector sequence as an intermediate attention value of the document chapter data;

[0009] A weight score of each intermediate attention value is calculated, and according to the size of each weight score, a preset proportion of the intermediate attention values ​​are eliminated, and the retained intermediate attention values ​​are used as compact attention representations of the document chapter data.

[0010] Optionally, the document chapter data further includes a candidate question set, the candidate question set includes at least one candidate question, and the weight score of each intermediate attention value is calculated, and according to the size of each weight score, a preset proportion of the intermediate attention values ​​are eliminated, and the retained intermediate attention values ​​are used as the compact attention representation of the document chapter data, including:

[0011] Selecting at least one candidate question from the candidate question set as a target question;

[0012] Splicing each of the target questions into a target question text;

[0013] Performing word segmentation encoding on the target question text to obtain a target question text encoding sequence;

[0014] Inputting the target question text encoding sequence and each of the intermediate attention values ​​into the large language model, and calculating the attention weight of each of the intermediate attention values;

[0015] According to each of the attention weights, a weight score of each of the intermediate attention values ​​is calculated, and based on each of the weight scores, a preset proportion of the intermediate attention values ​​are eliminated, and the retained intermediate attention values ​​are used as a compact attention representation of the document chapter data.

[0016] Optionally, inputting the target question text encoding sequence and each of the intermediate attention values ​​into the large language model and calculating the attention weight of each of the intermediate attention values ​​comprises:

[0017] Inputting the target question text encoding sequence into the embedding network of the large language model to calculate an input vector sequence;

[0018] Inputting the input vector sequence into the first network layer of the large language model, and calculating a second query vector sequence, a second key vector sequence, and a second value vector sequence;

[0019] splicing the first key vector sequence and the second key vector sequence of the first network layer, and splicing the first value vector sequence and the second value vector sequence to obtain a spliced ​​key vector sequence and a spliced ​​value vector sequence;

[0020] Calculate the attention weight of the first network layer and the input vector sequence of the next network layer according to the second query vector sequence, the concatenated key vector sequence and the concatenated value vector sequence;

[0021] Repeat the step of inputting the input vector sequence into the first network layer of the large language model, calculating the second query vector sequence, the second key vector sequence, and the second value vector sequence, to the step of calculating the attention weight of the first network layer and the input vector sequence of the next network layer according to the second query vector sequence, the concatenated key vector sequence, and the concatenated value vector sequence, to obtain the attention weight of each network layer.

[0022] Optionally, the step of calculating a weight score of each of the intermediate attention values ​​according to each of the attention weights, and removing a preset proportion of the intermediate attention values ​​based on each of the weight scores, and using the retained intermediate attention values ​​as a compact attention representation of the document chapter data, comprises:

[0023] Calculate the weight score of each intermediate attention value by a weight score formula;

[0024] The weight scores of the intermediate attention values ​​of each network layer are compared respectively, and for each network layer, a preset proportion of the intermediate attention values ​​with high weight scores is retained;

[0025] The retained intermediate attention values ​​are used as compact attention representations of the document chapter data.

[0026] Optionally, before the step of inputting the preprocessed chapter encoding sequence of the document chapter data into the large language model, calculating the first key vector sequence and the first value vector sequence of each network layer of the large language model, and using each group of the first key vector sequence and the first value vector sequence as an intermediate attention value of the document chapter data, the method further includes:

[0027] Performing word segmentation coding on a first preset number of documents respectively, and dividing them into a plurality of document chapter data; the document chapter data includes a chapter coding sequence;

[0028] Obtaining a second preset number of frequently asked questions, and modifying each of the frequently asked questions according to the content of each of the documents, to obtain the first preset number multiplied by the second preset number of candidate questions;

[0029] Each candidate question is associated with the corresponding document chapter data to obtain a candidate question set for each document chapter data.

[0030] Optionally, the method further comprises:

[0031] In the case that the document passage data does not have the associated candidate question, at least one general instruction question is added to the document passage data.

[0032] In a second aspect, an embodiment of the present invention provides a document question-answering optimization method, the method comprising:

[0033] Processing each document by the document processing method described in any one of the above items to obtain a plurality of document chapter data; the document chapter data includes document chapter content and a compact attention representation, and the compact attention representation includes a plurality of first key vector sequences and a plurality of first value vector sequences;

[0034] Calculate the question semantic vector of the user's question and the content semantic vector of the content of each of the document chapter data;

[0035] Calculating the similarity between the question semantic vector and each content semantic vector, and finding the document chapter data with the highest similarity as the target document chapter data;

[0036] The compact attention representation of the target document chapter data and the user question are input into a large language model to generate an answer text.

[0037] Optionally, the step of inputting the compact attention representation of the target document chapter data and the user question into a large language model to generate an answer text comprises:

[0038] Encoding the user question word segmentation into a user question code sequence;

[0039] Input the user question encoding sequence into the embedding network of the large language model to calculate the input vector sequence;

[0040] Inputting the input vector sequence into the first network layer of the large language model, and calculating a second query vector sequence, a second key vector sequence, and a second value vector sequence;

[0041] splicing the first key vector sequence and the second key vector sequence of the first network layer, and splicing the first value vector sequence and the second value vector sequence to obtain a spliced ​​key vector sequence and a spliced ​​value vector sequence;

[0042] Calculate the attention weight of the first network layer and the input vector sequence of the next network layer according to the second query vector sequence, the concatenated key vector sequence and the concatenated value vector sequence;

[0043] Repeat the step of inputting the input vector sequence into the first network layer of the large language model and calculating the second query vector sequence, the second key vector sequence and the second value vector sequence, until the step of inputting the input vector sequence into the first network layer of the large language model and calculating the second query vector sequence, the second key vector sequence and the second value vector sequence, to obtain the hidden state of the last network layer;

[0044] The hidden state is input into the output layer of the large language model to generate a response text.

[0045] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements any of the above-described methods when executing the program.

[0046] In a fourth aspect, an embodiment of the present invention provides a storage medium having a computer program stored thereon, wherein the computer program implements any of the above-described methods when executed by a processor.

[0047] The embodiment of the present invention provides a document processing method, a document question and answer optimization method, an electronic device and a storage medium. After the document chapter data is processed by compact attention representation, the document can be converted into a compact attention representation. In this way, when facing multiple different questions related to the document, the previously generated compact attention representation can be directly reused without the need to encode and decode the document itself again. This method greatly reduces the amount of calculation; it can also effectively identify and retain key information in the document in the preprocessing stage, while removing unnecessary content. This process not only increases the density of the remaining information, but also significantly shortens the sequence length used for calculation. Since the computational complexity of the large model based on the Transformer architecture is proportional to the square of the input sequence length, reducing the sequence length can greatly improve the computational efficiency of the model while reducing the required computing resources. This solves the problem of inefficiency in traditional document question and answer systems, thereby significantly improving the speed and efficiency of the question and answer process.

[0048] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0050] Figure 1 A schematic structural block diagram of an electronic device provided by an embodiment of the present invention;

[0051] Figure 2 A flowchart of a document processing method provided by an embodiment of the present invention;

[0052] Figure 3 A schematic diagram of an intermediate attention value provided by an embodiment of the present invention;

[0053] Figure 4 Another flowchart of a document processing method provided by an embodiment of the present invention;

[0054] Figure 5 A schematic diagram of a process flow of step S224 provided in an embodiment of the present invention;

[0055] Figure 6 A schematic diagram of a concatenated key vector sequence and a concatenated value vector sequence provided by an embodiment of the present invention;

[0056] Figure 7 A flowchart of a document question and answer optimization method provided in an embodiment of the present invention.

[0057] Icon: 100 - electronic device; 101 - memory; 102 - communication interface; 103 - processor; 104 - communication bus. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0059] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. At the same time, in the description of the present invention, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0061] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0062] As described in the background technology, when the related document question-answering system processes long documents, the large language model needs to encode the entire document, which leads to high computational complexity, and thus the document question-answering system based on such a model faces the problem of high computational cost.

[0063] Based on the above situation, an embodiment of the present invention provides a document processing method, a document question and answer optimization method, an electronic device and a storage medium. By preprocessing the document into document chapter data and performing compact attention representation processing on the document chapter data, the document can be converted into a compact attention representation, so that when a user asks a question, an answer text can be generated according to the compact attention representation, which can improve the computational efficiency of the large language model and thus significantly improve the speed and efficiency of the question and answer process.

[0064] To implement the process steps and functions of each example of the present invention, please refer to Figure 1 , Figure 1 A schematic structural block diagram of an electronic device provided by an embodiment of the present invention. The electronic device 100 includes a memory 101 and a processor 103, and the memory 101 and the processor 103 are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses 104 or signal lines. The memory 101 can be used to store software programs and modules, and the processor 103 executes the software programs and modules stored in the memory 101 to perform various functional applications and data processing.

[0065] The electronic device 100 may be, but is not limited to, a personal computer (PC), a server, a distributed computer, etc. It is understandable that the electronic device 100 is not limited to a physical server, but may also be a virtual machine on a physical server, a virtual machine built on a cloud platform, etc., which can provide the same functions as the server or virtual machine. The operating system of the electronic device 100 may be, but is not limited to, a Windows system, a Linux system, etc.

[0066] Among them, the memory 101 can be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc.

[0067] The communication connection between the electronic device 100 and an external device is achieved through at least one communication interface 102 (which can be wired or wireless).

[0068] The processor 103 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the embodiment of the present invention can be completed by the hardware integrated logic circuit in the processor 103 or the instructions in the form of software. The processor 103 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0069] Understandably, Figure 1 The structure shown is for illustration only. The electronic device 100 may also include Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown. Figure 1Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0070] The document processing method provided by the present invention is described below in an exemplary manner. Specifically, Figure 2 A flowchart of a document processing method provided by an embodiment of the present invention is shown in FIG. Figure 2 , the execution subject of this method can be the above Figure 1 The electronic device 100 shown in FIG. 1 includes: Figure 2 The following steps are described:

[0071] S210: Input the preprocessed chapter encoding sequence of the document chapter data into the large language model, calculate the first key vector sequence and the first value vector sequence of each network layer of the large language model, and use each group of the first key vector sequence and the first value vector sequence as an intermediate attention value of the document chapter data.

[0072] The document chapter data can be a fragment of a document, for example, a part of a college admission document can be encoded by word segmentation to obtain a digital sequence, which is the chapter coding sequence of the document chapter data. The word segmentation coding can adopt Qwen word segmenter.

[0073] Input this chapter encoding sequence into a pre-trained large language model, and calculate the first key vector sequence and the first value vector sequence of each network layer in the large language model. The large language model can be obtained by training the Qwen-72B basic model. The large language model includes an attention mechanism module, which generates a key vector sequence (K) and a value vector sequence (V) based on the input chapter encoding sequence. These vectors are used to capture the relationship between different parts of the input sequence. A set of KVs is used as an intermediate attention value for the document chapter data.

[0074] For example, the text content of the document chapter data is "In order to ensure the smooth progress of the school's undergraduate enrollment work and standardize enrollment behavior", and its chapter encoding sequence is "[116947, 90476, 119, 46448, 198, 116252, 198, 100012, 101907, 100344, 100714, 102559, 104635, 104360, 102088, 71817, 3837, 101931, 104635, 101070]", with a length of 20. Each layer of the network includes 20 groups of KVs, that is, 20 intermediate attention values. Figure 3 , if the large language model includes i network layers and the passage coding sequence has j KVs, then i*j intermediate attention values ​​will be obtained in the end. For example, the language model used in this embodiment includes 28 network layers, and the passage coding sequence has 20 KVs, so there are 560 KVs in total.

[0075] S220: Calculate the weight score of each intermediate attention value, remove a preset proportion of intermediate attention values ​​according to the size of each weight score, and use the retained intermediate attention values ​​as the compact attention representation of the document chapter data.

[0076] After calculating all the intermediate attention values ​​of the document chapter data, we calculate the weight score of each intermediate attention value, sort them according to the weight score, and eliminate the intermediate attention values ​​with lower scores according to a preset ratio (for example, 70% with lower scores). This can remove unimportant attention allocations, and the retained high-score attention values ​​constitute a compact attention representation of the document. This compact attention representation can better capture the key information of the document, reduce redundant information, and improve processing efficiency.

[0077] This method calculates the intermediate attention value of the passage encoding sequence and compresses the intermediate attention value to obtain a compact attention representation. When the user asks a question, the compact attention representation can be used to generate the answer text, which can improve the computational efficiency of the large language model and significantly improve the speed and efficiency of the question-answering process.

[0078] The document chapter data is obtained through preprocessing. In a possible implementation, before step S210, the method may further include the following steps:

[0079] S201: Perform word segmentation coding on a first preset number of documents respectively, and divide them into a plurality of document chapter data, wherein the document chapter data includes a chapter coding sequence.

[0080] For example, we collected enrollment documents from 100 universities and used the Qwen word segmenter to segment and encode each enrollment document. At this time, the content of each enrollment document corresponds to a coding sequence. Each coding sequence is segmented using window segmentation. The window size is set to 4800 and the overlap area size is set to 300. After encoding and segmentation, each fragment can be organized into document chapter data, as shown in Table 1.

[0081] Table 1

[0082]

[0083] S202: Obtain a second preset number of high-frequency questions, and modify each high-frequency question according to the content of each document to obtain the first preset number multiplied by the second preset number of candidate questions.

[0084] According to the content of each document, the high-frequency questions are modified, that is, the high-frequency questions are modified into questions related to the content of each document. For example, 100 high-frequency questions are obtained, and one of the questions is such as "Can the computer science and technology major of University B be applied for intra-school transfer?", then corresponding to the admission document of University A, the question is modified to "Can the computer science and technology major of University A be applied for intra-school transfer?" As above, we collected the admission documents of 100 universities, and we will get 100*100 candidate questions.

[0085] S203: Associating each candidate question with each corresponding document chapter data to obtain a candidate question set for each document chapter data.

[0086] Then, each candidate question is associated with the corresponding document chapter data through labeling and other methods. For example, in the enrollment document, the starting position of each question corresponding to the answer is marked, and which document chapter data the starting position belongs to. Finally, the candidate question is placed in the corresponding document chapter data. At this time, the structure of each document chapter data is shown in Table 2.

[0087] Table 2

[0088]

[0089] After the above processing, there will often be some document chapter data without candidate questions, and the candidate question set is empty. In order to supplement these document chapter data, in a possible implementation, the method may further include the steps of:

[0090] S204: When the document chapter data has no associated candidate question, add at least one general instruction question to the document chapter data.

[0091] First, design some general instruction questions related to the document, such as "What does the text specifically talk about?", "What information in the text requires candidates to pay attention to?", and then randomly select 8 from these general instruction questions as candidate questions for the document chapter data.

[0092] The document chapter data processed as above also includes a candidate question set, and the candidate question set includes at least one candidate question. In a possible implementation, see Figure 4 , the above step S220 may include the following steps:

[0093] S221: Select at least one candidate question from the candidate question set as a target question.

[0094] S222: Concatenate each target question into a target question text.

[0095] S223: Perform word segmentation encoding on the target question text to obtain a target question text encoding sequence.

[0096] S224: Input the target question text encoding sequence and each intermediate attention value into the large language model, and calculate the attention weight of each intermediate attention value.

[0097] S225: Calculate the weight score of each intermediate attention value according to each attention weight, and remove a preset proportion of intermediate attention values ​​based on each weight score, and use the retained intermediate attention values ​​as a compact attention representation of the document chapter data.

[0098] For each document chapter data, at least one candidate question is selected from the candidate question set, for example, 3 are selected, the texts of the 3 candidate questions are spliced ​​to obtain the spliced ​​target question text, the target question text is segmented and encoded to obtain the target question text encoding sequence, the segmentation encoding uses the same segmenter as above, and then the target question text encoding sequence and the intermediate attention value calculated in step S210 are used to calculate the attention weights of each intermediate attention value of the large language model, and then the weight score of each intermediate attention value is calculated according to each attention weight, and finally, according to the weight score, a certain proportion of the original intermediate attention values ​​are eliminated, and the retained intermediate attention values ​​are used as the compact attention representation of the document chapter data. At this time, the structure of each document chapter data is shown in Table 3.

[0099] Table 3

[0100]

[0101] There are many ways to calculate the attention weight. In order to integrate more information, the target question text encoding sequence and the intermediate attention value sequence can be combined for calculation. In one possible implementation, see Figure 5 , the above step S224 may include the following steps:

[0102] S2241: Input the target question text encoding sequence into the embedding network of the large language model and calculate the input vector sequence.

[0103] Through the processing of the embedding network of the large language model, the input vector sequence of the target question text encoding sequence can be obtained, which will serve as the basic input of the large language model. The input vector sequence is the vector sequence corresponding to the concatenated target question text.

[0104] S2242: Input the input vector sequence into the first network layer of the large language model, and calculate a second query vector sequence, a second key vector sequence, and a second value vector sequence.

[0105] In large language models, such as models based on the Transformer architecture, after the input vector sequence is fed into the first network layer, a new sequence of query (Q), key (K), and value (V) vectors is calculated through the self-attention mechanism and the feed-forward network (FFN).

[0106] S2243: Concatenate the first key vector sequence and the second key vector sequence of the first network layer, and concatenate the first value vector sequence and the second value vector sequence to obtain a concatenated key vector sequence and a concatenated value vector sequence.

[0107] The first key vector sequence and the first value vector sequence corresponding to the chapter content in the document chapter data are concatenated with the second key vector sequence and the second value vector sequence of the question text. The concatenated key vector sequence and the concatenated value vector sequence help the large language model to better understand the relationship between the question and the document.

[0108] The concatenated key vector sequence and value vector sequence are as follows: Figure 6 shown.

[0109] S2244: Calculate the attention weight of the first network layer and the input vector sequence of the next network layer according to the second query vector sequence, the concatenated key vector sequence and the concatenated value vector sequence.

[0110] The calculation formula of attention weight is:

[0111]

[0112] in, is the second query vector sequence, is the concatenation key vector sequence, is the dimension of the concatenated key vector sequence, is the transpose of the concatenation key vector sequence.

[0113] The calculation formula for the input vector sequence of the next network layer is:

[0114]

[0115] in, is the concatenated value vector sequence, is the attention weight.

[0116] Will As the input vector sequence of the next network layer.

[0117] S2245: Repeat the steps of inputting the input vector sequence into the first network layer of the large language model, calculating the second query vector sequence, the second key vector sequence and the second value vector sequence, until the step of calculating the attention weight of the first network layer and the input vector sequence of the next network layer according to the second query vector sequence, the concatenated key vector sequence and the concatenated value vector sequence, to obtain the attention weight of each network layer.

[0118] Repeatedly calculate the attention weight of each layer and the input vector sequence of the next network layer, and finally obtain the attention weight of each network layer.

[0119] In a possible implementation, the above step S225 may include the following steps:

[0120] S2251: Calculate the weight score of each intermediate attention value through the weight score formula.

[0121] The weighted score formula is:

[0122]

[0123] in, represents the weight score of the j position in the intermediate attention value, N represents the length of the target question text encoding sequence, k represents the k position of the target question text encoding sequence, Represents the attention weight between the intermediate attention value position i and the target question text encoding sequence position j.

[0124] For example, if the length of the passage coding sequence is 10, each network layer has 10 intermediate attention values, that is, 10 first key vector sequences K1 and first value vector sequences V1. The length of the target question text coding sequence is 20, then 20 second query vector sequences Q2, second key vector sequences K2, and second value vector sequences V2 are calculated for the target question text coding sequence. Therefore, for the 10 intermediate attention values ​​of each network layer, the weight score of the intermediate attention value at position 1 is calculated. First, the intermediate attention value at this position and the attention weights of 20 Q2, K2, and V2 are calculated. The attention weight can be calculated using the above formula Calculate and get 20 attention weights. Take the average of these 20 attention weights to get the weight score of the intermediate attention value at position 1. Use the same method to calculate the weight scores of the 10 intermediate attention values.

[0125] S2252: Compare the weight scores of the intermediate attention values ​​of each network layer respectively, and for each network layer, retain a preset proportion of the intermediate attention values ​​with high weight scores.

[0126] Retaining the intermediate attention values ​​with high scores specifically refers to comparing the same network layer. Different network layers will not be compared. When comparing, the scores of different positions are compared, and the intermediate attention values ​​of the positions with high scores are retained.

[0127] As above, the weight scores of the 10 intermediate attention values ​​can retain the 3 corresponding intermediate attention values ​​with higher weight scores.

[0128] S2253: The retained intermediate attention values ​​are used as compact attention representations of the document chapter data.

[0129] The remaining intermediate attention values ​​are used as the compact attention representation of the document chapter data.

[0130] Furthermore, the embodiment of the present invention also provides a document question answering optimization method, see Figure 7 , the document question answering optimization method includes the following steps:

[0131] S310: Process each document by a document processing method to obtain a plurality of document chapter data, wherein the document chapter data includes document chapter content and a compact attention representation, and the compact attention representation includes a plurality of first key vector sequences and a plurality of first value vector sequences.

[0132] S320: Calculate the question semantic vector of the user's question and the content semantic vector of the content of each document chapter data.

[0133] S330: Calculate the similarity between the question semantic vector and each content semantic vector, and find out the document chapter data with the highest similarity as the target document chapter data.

[0134] S340: Input the compact attention representation of the target document chapter data and the user question into a large language model to generate an answer text.

[0135] By processing the enrollment documents of various colleges and universities through the above document processing method, multiple document chapter data can be obtained. Now the question raised by the user is "How much is the tuition fee of University A?" First, the question is analyzed semantically to calculate the question semantic vector of the question, and then the content semantic vector of the content text corresponding to each document chapter data is calculated. The question semantic vector and each content semantic vector are used to find the document chapter data most relevant to the question raised by the user, which can be one or more. The most relevant method can be to calculate the cosine similarity of the question semantic vector and each content semantic vector, and the document chapter data with high cosine similarity is regarded as the related chapter of the question. According to the related chapter found, its already calculated compact attention representation can be found, and it can be input into the large language model together with the user question to generate the answer text.

[0136] This document question-answering optimization method generates answer text by inputting the calculated compact attention representation and user questions into a large language model. By applying the compact attention mechanism, the key information in the document can be effectively identified and retained in the preprocessing stage, while unnecessary content is removed. The final sequence length used for calculation is significantly shortened. Since the computational complexity of the large model based on the Transformer architecture is proportional to the square of the input sequence length, reducing the sequence length can greatly improve the computational efficiency of the model, while reducing the required computing resources, thereby improving the efficiency of document question-answering.

[0137] In a possible implementation, step S340 may include the following steps:

[0138] S341: Encode the user question word segments into user question code sequences.

[0139] S342: Input the user question encoding sequence into the embedding network of the large language model to calculate the input vector sequence.

[0140] S343: Input the input vector sequence into the first network layer of the large language model, and calculate a second query vector sequence, a second key vector sequence, and a second value vector sequence.

[0141] S344: concatenate the first key vector sequence and the second key vector sequence of the first network layer, and concatenate the first value vector sequence and the second value vector sequence to obtain a concatenated key vector sequence and a concatenated value vector sequence.

[0142] S345: Calculate the attention weight of the first network layer and the input vector sequence of the next network layer according to the second query vector sequence, the concatenated key vector sequence and the concatenated value vector sequence.

[0143] S346: Repeat the steps of inputting the input vector sequence into the first network layer of the large language model, calculating the second query vector sequence, the second key vector sequence and the second value vector sequence, until the step of inputting the input vector sequence into the first network layer of the large language model, calculating the second query vector sequence, the second key vector sequence and the second value vector sequence, and obtaining the hidden state of the last network layer.

[0144] S347: Input the hidden state into the output layer of the large language model to generate the answer text.

[0145] First, the user question is segmented and encoded and input into the embedding network of the large language model to calculate the input vector sequence. Then, the processing logic of steps S2242 to S2245 is consistent with the above. After the processing is completed, the hidden state of the last network layer is obtained and input into the output layer of the large language model to generate the answer text.

[0146] In summary, the document processing method, document question-answering optimization method, electronic device and storage medium provided by the embodiment of the present invention can convert the document into a compact attention representation after performing compact attention representation processing on the document chapter data. In this way, when facing multiple different questions related to the document, the previously generated compact attention representation can be directly reused without the need to encode and decode the document itself again. This method greatly reduces the amount of calculation; it can also effectively identify and retain key information in the document in the preprocessing stage, while removing unnecessary content. This process not only increases the density of the remaining information, but also significantly shortens the sequence length used for calculation. Since the computational complexity of the large model based on the Transformer architecture is proportional to the square of the input sequence length, reducing the sequence length can greatly improve the computational efficiency of the model while reducing the required computing resources. This solves the problem of inefficiency in the traditional document question-answering system, thereby significantly improving the speed and efficiency of the question-answering process.

[0147] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and a module, a program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0148] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0149] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a computer-readable storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0150] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0151] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A document processing method, characterized in that: The method comprises: Inputting the chapter encoding sequence of the preprocessed document chapter data into the large language model, calculating the first key vector sequence and the first value vector sequence of each network layer of the large language model, and taking each group of the first key vector sequence and the first value vector sequence as an intermediate attention value of the document chapter data; the document chapter data also includes a candidate question set, and the candidate question set includes at least one candidate question; Selecting at least one candidate question from the candidate question set as a target question; Splicing each of the target questions into a target question text; Performing word segmentation encoding on the target question text to obtain a target question text encoding sequence; Inputting the target question text encoding sequence and each of the intermediate attention values ​​into the large language model, and calculating the attention weight of each of the intermediate attention values; According to each of the attention weights, the weight score of each of the intermediate attention values ​​is calculated by a weight score formula; the weight score formula is: in, represents the weight score of the j position in the intermediate attention value, N represents the length of the target question text encoding sequence, k represents the k position of the target question text encoding sequence, Represents the attention weight of the intermediate attention value position i and the target question text encoding sequence position j; According to the size of each weight score, a preset proportion of the intermediate attention values ​​are eliminated, and the retained intermediate attention values ​​are used as the compact attention representation of the document chapter data.

2. The method according to claim 1, characterized in that The step of inputting the target question text encoding sequence and each of the intermediate attention values ​​into the large language model and calculating the attention weight of each of the intermediate attention values ​​comprises: Inputting the target question text encoding sequence into the embedding network of the large language model to calculate an input vector sequence; Inputting the input vector sequence into the first network layer of the large language model, and calculating a second query vector sequence, a second key vector sequence, and a second value vector sequence; splicing the first key vector sequence and the second key vector sequence of the first network layer, and splicing the first value vector sequence and the second value vector sequence to obtain a spliced ​​key vector sequence and a spliced ​​value vector sequence; Calculate the attention weight of the first network layer and the input vector sequence of the next network layer according to the second query vector sequence, the concatenated key vector sequence and the concatenated value vector sequence; Repeat the step of inputting the input vector sequence into the first network layer of the large language model, calculating the second query vector sequence, the second key vector sequence, and the second value vector sequence, to the step of calculating the attention weight of the first network layer and the input vector sequence of the next network layer according to the second query vector sequence, the concatenated key vector sequence, and the concatenated value vector sequence, to obtain the attention weight of each network layer.

3. The method according to claim 1, characterized in that The step of removing a preset proportion of the intermediate attention values ​​according to the size of each weight score and using the retained intermediate attention values ​​as the compact attention representation of the document chapter data includes: The weight scores of the intermediate attention values ​​of each network layer are compared respectively, and for each network layer, a preset proportion of the intermediate attention values ​​with high weight scores is retained; The retained intermediate attention values ​​are used as compact attention representations of the document chapter data.

4. The method according to claim 1, characterized in that Before the step of inputting the preprocessed chapter encoding sequence of the document chapter data into the large language model, calculating the first key vector sequence and the first value vector sequence of each network layer of the large language model, and using each group of the first key vector sequence and the first value vector sequence as an intermediate attention value of the document chapter data, the method further includes: Performing word segmentation coding on a first preset number of documents respectively, and dividing them into a plurality of document chapter data; the document chapter data includes a chapter coding sequence; Obtaining a second preset number of frequently asked questions, and modifying each of the frequently asked questions according to the content of each of the documents, to obtain the first preset number multiplied by the second preset number of candidate questions; Each candidate question is associated with the corresponding document chapter data to obtain a candidate question set for each document chapter data.

5. The method according to claim 4, characterized in that The method further comprises: In the case that the document passage data does not have the associated candidate question, at least one general instruction question is added to the document passage data.

6. A document question-answering optimization method, characterized in that: The method comprises: Processing each document by the document processing method according to any one of claims 1 to 5 to obtain a plurality of document chapter data; the document chapter data includes document chapter content and a compact attention representation, and the compact attention representation includes a plurality of first key vector sequences and a plurality of first value vector sequences; Calculate the question semantic vector of the user's question and the content semantic vector of the content of each of the document chapter data; Calculating the similarity between the question semantic vector and each content semantic vector, and finding the document chapter data with the highest similarity as the target document chapter data; The compact attention representation of the target document chapter data and the user question are input into a large language model to generate an answer text.

7. The method according to claim 6, characterized in that The step of inputting the compact attention representation of the target document chapter data and the user question into a large language model to generate an answer text comprises: Encoding the user question word segmentation into a user question code sequence; Input the user question encoding sequence into the embedding network of the large language model to calculate the input vector sequence; Inputting the input vector sequence into the first network layer of the large language model, and calculating a second query vector sequence, a second key vector sequence, and a second value vector sequence; splicing the first key vector sequence and the second key vector sequence of the first network layer, and splicing the first value vector sequence and the second value vector sequence to obtain a spliced ​​key vector sequence and a spliced ​​value vector sequence; Calculate the attention weight of the first network layer and the input vector sequence of the next network layer according to the second query vector sequence, the concatenated key vector sequence and the concatenated value vector sequence; Repeat the step of inputting the input vector sequence into the first network layer of the large language model and calculating the second query vector sequence, the second key vector sequence and the second value vector sequence, until the step of inputting the input vector sequence into the first network layer of the large language model and calculating the second query vector sequence, the second key vector sequence and the second value vector sequence, to obtain the hidden state of the last network layer; The hidden state is input into the output layer of the large language model to generate a response text.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the document processing method described in any one of claims 1 to 5 or the document question and answer optimization method described in claim 6 or 7 is implemented.

9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the document processing method described in any one of claims 1 to 5 or the document question and answer optimization method described in claim 6 or 7 is implemented.

Images