Long text processing method and device, electronic equipment and storage medium
Patent Information
- Application Number
- CN202411079393.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2026-09-22
- Estimated Expiration
- 2044-08-07
AI Technical Summary
[0006]传统的LLM基座Transformer架构在处理长文本时存在以下问题：资源开销大：每个token对应一组q、k、v特征，导致计算和存储成本高昂；长上下文难以处理：随着序列长度增加，计算复杂度呈平方级增长，难以处理超长文本
[0061]本申请实施例编码器对当前词元的当前键特征和当前值特征进行降维压缩，节省存储及计算资源，解码器可以有效解析高层语义信息，通过编码器-解码器结构能够有效融合长距离的上下文信息，提升模型对长文本的理解能力，可以显著节约训练资源和推理开销。
Smart Images
Figure CN119128050B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular to a long text processing method, apparatus, electronic device, and storage medium. Background Technology
[0002] With the rise of pre-trained language models (LLMs) based on large corpora, their application scenarios in long text processing are receiving increasing attention, including but not limited to the following aspects:
[0003] 1. Professional Literature Reading and Analysis: In fields such as scientific research, medicine, and law, professionals need to read and understand a large number of professional documents and reports. These documents are often very long, and models are needed to process and understand long texts.
[0004] 2. Long books and documents: For users who need to quickly obtain summaries of long books or documents, long text processing capabilities can help them quickly grasp the core content.
[0005] 3. Content creation and editing: In the field of content creation, such as novel writing and screenwriting, long text processing capabilities can help authors organize their thoughts, generate ideas, or proofread manuscripts.
[0006] Traditional LLM-based Transformer architectures suffer from the following problems when processing long texts: High resource overhead: Each token corresponds to a set of q, k, v features, resulting in high computation and storage costs; Difficulty in handling long contexts: As the sequence length increases, the computational complexity increases quadratically, making it difficult to handle extremely long texts. Summary of the Invention
[0007] In order to solve the above-mentioned technical problems, or at least partially solve the above-mentioned technical problems, this application provides a long text processing method, apparatus, electronic device and storage medium.
[0008] Firstly, this application provides a long text processing method, including:
[0009] Obtain the current key feature, current value feature, and current query feature corresponding to the current word element in the target long text, as well as the first global state feature corresponding to the previous word element;
[0010] The current key feature, the current value feature, and the first global state feature are input into the encoder so that the encoder outputs a second global state feature, which includes: the high-level semantic information of the current word and all words preceding the current word;
[0011] The second global state feature and the current query feature are input into the decoder so that the decoder outputs the lexical representation corresponding to the current lexical.
[0012] Optionally, the first global state feature includes: a first global key feature and a first global value feature. The current key feature, the current value feature, and the first global state feature are input into the encoder so that the encoder outputs a second global state feature, including:
[0013] The encoder determines the high-level semantic information of the current word and all words preceding the current word based on the current key feature, the current value feature, the first global key feature, and the first global value feature, and outputs it as the second global state feature.
[0014] Optionally, the encoder determines high-level semantic information of the current lexical and all lexical preceding the current lexical based on the current key feature, the current value feature, the first global key feature, and the first global value feature, as the second global state feature, including:
[0015] The encoder concatenates the current key feature and the first global key feature to obtain the concatenated key feature;
[0016] The encoder concatenates the current value feature and the first global value feature to obtain the concatenated value feature;
[0017] The encoder inputs the splicing key feature and the splicing value feature into the first multilayer perceptron module to obtain the second global state feature.
[0018] Optionally, the second global state feature includes: a second global key feature and a second global value feature. The second global state feature and the current query feature are input into the decoder so that the decoder outputs the lexical representation corresponding to the current lexical, including:
[0019] The decoder determines the high-level semantic information weights corresponding to the current word element based on the current query features and the second global key features.
[0020] The decoder determines the lexical representation corresponding to the current lexical based on the high-level semantic information weights and the second global value features.
[0021] Optionally, the decoder determines the high-level semantic information weights corresponding to the current lexical unit based on the current query feature and the second global key feature, including:
[0022] The decoder concatenates the current query feature and the second global key feature to obtain the concatenated feature;
[0023] The decoder inputs the spliced features into the second multilayer perceptron module to obtain the high-level semantic information weights corresponding to the current word.
[0024] Optionally, the decoder determines the lexical representation corresponding to the current lexical based on the high-level semantic information weights and the second global value feature, including:
[0025] The decoder multiplies the high-level semantic information weights with the second global value features bitwise to obtain the lexical representation corresponding to the current lexical.
[0026] Optionally, the current key feature, current value feature, and current query feature corresponding to the current term in the target long text are obtained, including:
[0027] Obtain the target long text, and segment the target long text into words to obtain multiple word units;
[0028] Determine the feature vector corresponding to each of the lexical units;
[0029] Perform matrix operations on the feature vector corresponding to each word and the key matrix to obtain the current key feature;
[0030] Perform matrix operations on the feature vector and value matrix corresponding to each word to obtain the current value feature;
[0031] Perform matrix operations on the feature vector corresponding to each term and the query matrix to obtain the current query feature.
[0032] Secondly, this application provides a long text processing apparatus, comprising:
[0033] The acquisition module is used to acquire the current key feature, current value feature, and current query feature corresponding to the current word element in the target long text, as well as the first global state feature corresponding to the previous word element.
[0034] The encoder input module is used to input the current key feature, the current value feature and the first global state feature into the encoder so that the encoder outputs a second global state feature. The second global state feature includes: the high-level semantic information of the current word and all words preceding the current word.
[0035] The decoder input module is used to input the second global state feature and the current query feature into the decoder so that the decoder outputs the lexical representation corresponding to the current lexical.
[0036] Optionally, the first global state feature includes: a first global key feature and a first global value feature, and the encoder input module includes:
[0037] The first determining unit is used by the encoder to determine the high-level semantic information of the current word and all words preceding the current word based on the current key feature, the current value feature, the first global key feature, and the first global value feature, so as to use as the second global state feature, and output it.
[0038] Optionally, the first determining unit is further configured to:
[0039] The encoder concatenates the current key feature and the first global key feature to obtain the concatenated key feature;
[0040] The encoder concatenates the current value feature and the first global value feature to obtain the concatenated value feature;
[0041] The encoder inputs the splicing key feature and the splicing value feature into the first multilayer perceptron module to obtain the second global state feature.
[0042] Optionally, the second global state feature includes: a second global key feature and a second global value feature, and the decoder input module includes:
[0043] The second determining unit is used by the decoder to determine the high-level semantic information weight corresponding to the current word element based on the current query feature and the second global key feature;
[0044] The third determining unit is used by the decoder to determine the lexical representation corresponding to the current lexical based on the high-level semantic information weights and the second global value features.
[0045] Optionally, the second determining unit is further configured to:
[0046] The decoder concatenates the current query feature and the second global key feature to obtain the concatenated feature;
[0047] The decoder inputs the spliced features into the second multilayer perceptron module to obtain the high-level semantic information weights corresponding to the current word.
[0048] Optionally, the third determining unit is further configured to:
[0049] The decoder multiplies the high-level semantic information weights with the second global value features bitwise to obtain the lexical representation corresponding to the current lexical.
[0050] Optionally, the acquisition module includes:
[0051] The acquisition unit is used to acquire the target long text, segment the target long text into words, and obtain multiple word units;
[0052] The fourth determining unit is used to determine the feature vector corresponding to each of the lexical units;
[0053] The first operation unit is used to perform matrix operations on the feature vector corresponding to each word and the key matrix to obtain the current key feature;
[0054] The second operation unit is used to perform matrix operations on the feature vector and value matrix corresponding to each word to obtain the current value feature.
[0055] The third operation unit is used to perform matrix operations on the feature vector corresponding to each word and the query matrix to obtain the current query feature.
[0056] Thirdly, this application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;
[0057] Memory, used to store computer programs;
[0058] A processor, when executing a program stored in memory, implements the long text processing method described in any of the first aspects.
[0059] Fourthly, this application provides a computer-readable storage medium storing a program for a long text processing method, wherein when the program for the long text processing method is executed by a processor, it implements the steps of any of the long text processing methods described in the first aspect.
[0060] The technical solutions provided in this application have the following advantages compared with the prior art:
[0061] The encoder in this application performs dimensionality reduction and compression on the current key features and current value features of the current word, saving storage and computing resources. The decoder can effectively parse high-level semantic information. Through the encoder-decoder structure, long-distance contextual information can be effectively fused, improving the model's ability to understand long texts and significantly saving training resources and inference overhead. Attached Figure Description
[0062] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0063] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0064] Figure 1 A flowchart illustrating a long text processing method provided in this application embodiment;
[0065] Figure 2 A schematic diagram illustrating a long text processing method provided in an embodiment of this application;
[0066] Figure 3 A structural diagram of a long text processing device provided in an embodiment of this application;
[0067] Figure 4 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation
[0068] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0069] Traditional LLM-based Transformer architectures generate corresponding query, key, and value features for each input token in long texts and perform a self-attention mechanism. Self-attention allows each word in a sentence to pay attention to all words in the sentence, thereby capturing long-term dependencies between words. Its principle is as follows: Generating Q, K, and V matrices: For each token vector in the input sentence, it is multiplied by three different matrices Wq, Wk, and Wv to obtain the corresponding Query (Q), Key (K), and Value (V) vectors; Calculating attention scores: The dot product between each Query and all Keys is calculated to obtain the attention score. A higher attention score indicates a stronger correlation between the two words; Scaling and normalization: The attention score is divided by the square root of the Key vector dimension, and then normalized using the Softmax function to obtain the attention weights between each word and all other words; Weighted summation: The Value matrix is multiplied by the attention weight matrix to obtain the context representation of each word. This means that the final representation of each token is obtained by a weighted sum of all related words in the sentence.
[0070] In simple terms, the Self-Attention mechanism uses Q to match K, finds relevant values, and then sums these values with weights to obtain the final word representation.
[0071] However, the above methods for processing long text have the following problems: high resource consumption: each token corresponds to a set of q, k, v features, resulting in high computation and storage costs; difficulty in handling long contexts: as the sequence length increases, the computational complexity increases quadratically, making it difficult to handle extremely long texts. Therefore, this application provides a long text processing method, apparatus, electronic device, and storage medium, which improves the kv feature dimension of the LLM model, enabling the model to adaptively learn feature compression and selection capabilities in long contexts during training; and automatically performs feature compression and selection in long contexts during inference.
[0072] This application provides a long text processing method that can be applied to both the training and inference phases of an LLM model, such as... Figure 1 As shown, this long text processing method includes:
[0073] Step S101: Obtain the current key feature, current value feature, and current query feature corresponding to the current word element in the target long text, as well as the first global state feature corresponding to the previous word element;
[0074] In this embodiment of the application, the target long text can refer to the long text to be processed, such as: novels, books, documents, scripts, etc.
[0075] The previous word refers to the word preceding the current word, that is, the word adjacent to and preceding the current word. The first global state feature is obtained by inputting the key feature, value feature of the previous word, and the global state feature corresponding to the word two words before the previous word into the encoder.
[0076] In one embodiment of this application, step S101 obtains the current key feature, current value feature, and current query feature corresponding to the current word element in the target long text, including:
[0077] Obtain the target long text, segment the target long text into words to obtain multiple tokens; determine the feature vector corresponding to each token; perform matrix operations on the feature vector corresponding to each token and the key matrix to obtain the current key feature K. t Perform matrix operations on the feature vector and value matrix corresponding to each word to obtain the current value feature V. t Perform matrix operations on the feature vector corresponding to each term and the query matrix to obtain the current query feature Q. t .
[0078] In other words, the target long text is first segmented to obtain multiple tokens. A token can be a single character or word. Each token corresponds to a unique ID, and each ID corresponds to a feature vector. The feature vector of each token is transformed into a vector of the current key feature, current value feature, and current query feature through matrix operations of a preset query matrix, a preset key matrix, and a preset value matrix. The role of the current query feature is mainly to guide the model to focus on relevant information, capture semantic relevance, and ultimately help the model better understand the meaning of the language. Each token in each encoder structure corresponds to three feature vectors: Q, K, and V.
[0079] Step S102: Input the current key feature, the current value feature, and the first global state feature into the encoder so that the encoder outputs the second global state feature;
[0080] In this step, such as Figure 2 As shown, the encoder can use a gating mechanism to fuse the current key feature K. t Current value feature V t and the first global state feature (HK) t-1 HV t-1 This yields the second global state feature, which is the new hidden state (HK). t HV t The second global state feature includes: the high-level semantic information of the current word and all words preceding the current word.
[0081] Step S103: Input the second global state feature and the current query feature into the decoder so that the decoder outputs the lexical representation corresponding to the current lexical.
[0082] In this step, such as Figure 2 As shown, the decoder calculates the current query feature Q. t HK in the second global state feature t The relevance is analyzed, and high-level semantic information weights related to the current token are extracted. These extracted high-level semantic information weights are then compared with the HV values in the second global state features. t The fusion process generates the output feature O of the current token. t That is, word representation.
[0083] It should be noted that the number of encoder and decoder structures in this application is the same as the current LLM network structure settings, which are typically 32 to 80.
[0084] After the decoder outputs the lexical representation corresponding to each lexical, the LLM can generate a summary of the target long context based on the lexical representation corresponding to each lexical. This process has not been improved in this paper and will not be described in detail here.
[0085] The encoder in this application performs dimensionality reduction and compression on the current key features and current value features of the current word, saving storage and computing resources. The decoder can effectively parse high-level semantic information. Through the encoder-decoder structure, long-distance contextual information can be effectively fused, improving the model's ability to understand long texts and significantly saving training resources and inference overhead.
[0086] In another embodiment of this application, the first global state feature includes: a first global key feature and a first global value feature. Step S102 inputs the current key feature, the current value feature, and the first global state feature into the encoder so that the encoder outputs a second global state feature, including:
[0087] The encoder determines the high-level semantic information of the current word and all words preceding the current word based on the current key feature, the current value feature, the first global key feature, and the first global value feature, and outputs it as the second global state feature.
[0088] In one embodiment of this application, the encoder determines high-level semantic information of the current lexical and all lexical preceding the current lexical based on the current key feature, the current value feature, the first global key feature, and the first global value feature, as the second global state feature, including:
[0089] The encoder concatenates the current key feature and the first global key feature to obtain the concatenated key feature;
[0090] The encoder concatenates the current value feature and the first global value feature to obtain the concatenated value feature;
[0091] The encoder inputs the splicing key feature and the splicing value feature into the first multilayer perceptron module to obtain the second global state feature.
[0092] Second global state feature (HK) t HV t ) dimensions and first global state features (HK) t-1 HV t-1 The dimensions are consistent.
[0093] This application embodiment uses an encoder to reduce the dimensionality of the current key feature and current value feature of the current word, saving storage and computing resources. In the encoder, a new second global state feature is regenerated based on the current key feature, current value feature and the first global state feature of the previous word, which is used for the calculation of the next word. In this way, the semantic information of the sequence before the current word can be continuously updated. During the training process, the model will adaptively learn the semantic information extraction capability of long sequence features, which can fully realize the extraction and fusion of high-level semantic information of long sequences.
[0094] In another embodiment of this application, the second global state feature includes: a second global key feature and a second global value feature. Step S103 inputs the second global state feature and the current query feature into the decoder, so that the decoder output includes:
[0095] Step S201: The decoder determines the weight of the high-level semantic information corresponding to the current word based on the current query feature and the second global key feature;
[0096] In one embodiment of this application, the decoder determines the high-level semantic information weight corresponding to the current lexical unit based on the current query feature and the second global key feature, including:
[0097] The decoder concatenates the current query feature and the second global key feature to obtain the concatenated feature;
[0098] The decoder inputs the spliced features into the second multilayer perceptron module to obtain the high-level semantic information weights corresponding to the current word.
[0099] In step S202, the decoder determines the lexical representation corresponding to the current lexical based on the high-level semantic information weights and the second global value features.
[0100] In one embodiment of this application, the decoder determines the lexical representation corresponding to the current lexical based on the high-level semantic information weights and the second global value feature, including:
[0101] The decoder multiplies the high-level semantic information weights bitwise with the second global value feature to obtain the lexical representation corresponding to the current lexical. This is equivalent to filtering the second global value feature based on each high-level semantic information weight, retaining desired information and suppressing unwanted information.
[0102] The decoder in this embodiment fully considers the features of the current token while fusing contextual information, ensuring the general performance of the model. After the decoder, high-level semantic information is effectively parsed and parsed according to the output feature format. Similarly, as the training process progresses, LLM can quickly learn how to parse the semantic information of long sequences, thus effectively extracting the high-level semantic information of long sequences.
[0103] In another embodiment of this application, a long text processing apparatus is also provided, such as... Figure 3 As shown, it includes:
[0104] The acquisition module 11 is used to acquire the current key feature, current value feature and current query feature corresponding to the current word element in the target long text, as well as the first global state feature corresponding to the previous word element;
[0105] The encoder input module 12 is used to input the current key feature, the current value feature and the first global state feature into the encoder so that the encoder outputs a second global state feature. The second global state feature includes: the high-level semantic information of the current word and all words preceding the current word.
[0106] The decoder input module 13 is used to input the second global state feature and the current query feature into the decoder so that the decoder outputs the lexical representation corresponding to the current lexical.
[0107] Optionally, the first global state feature includes: a first global key feature and a first global value feature, and the encoder input module includes:
[0108] The first determining unit is used by the encoder to determine the high-level semantic information of the current word and all words preceding the current word based on the current key feature, the current value feature, the first global key feature, and the first global value feature, so as to use as the second global state feature, and output it.
[0109] Optionally, the first determining unit is further configured to:
[0110] The encoder concatenates the current key feature and the first global key feature to obtain the concatenated key feature;
[0111] The encoder concatenates the current value feature and the first global value feature to obtain the concatenated value feature;
[0112] The encoder inputs the splicing key feature and the splicing value feature into the first multilayer perceptron module to obtain the second global state feature.
[0113] Optionally, the second global state feature includes: a second global key feature and a second global value feature, and the decoder input module includes:
[0114] The second determining unit is used by the decoder to determine the high-level semantic information weight corresponding to the current word element based on the current query feature and the second global key feature;
[0115] The third determining unit is used by the decoder to determine the lexical representation corresponding to the current lexical based on the high-level semantic information weights and the second global value features.
[0116] Optionally, the second determining unit is further configured to:
[0117] The decoder concatenates the current query feature and the second global key feature to obtain the concatenated feature;
[0118] The decoder inputs the spliced features into the second multilayer perceptron module to obtain the high-level semantic information weights corresponding to the current word.
[0119] Optionally, the third determining unit is further configured to:
[0120] The decoder multiplies the high-level semantic information weights with the second global value features bitwise to obtain the lexical representation corresponding to the current lexical.
[0121] Optionally, the acquisition module includes:
[0122] The acquisition unit is used to acquire the target long text, segment the target long text into words, and obtain multiple word units;
[0123] The fourth determining unit is used to determine the feature vector corresponding to each of the lexical units;
[0124] The first operation unit is used to perform matrix operations on the feature vector corresponding to each word and the key matrix to obtain the current key feature;
[0125] The second operation unit is used to perform matrix operations on the feature vector and value matrix corresponding to each word to obtain the current value feature.
[0126] The third operation unit is used to perform matrix operations on the feature vector corresponding to each word and the query matrix to obtain the current query feature.
[0127] In another embodiment of this application, an electronic device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus.
[0128] Memory, used to store computer programs;
[0129] The processor, when executing a program stored in memory, implements the long text processing method described in any of the foregoing method embodiments.
[0130] The electronic device provided in this embodiment of the invention allows the processor to execute a program stored in the memory, and the encoder to reduce and compress the current key features and current value features of the current word, thereby saving storage and computing resources. The decoder can effectively parse high-level semantic information, and the encoder-decoder structure can effectively fuse long-distance contextual information, improve the model's ability to understand long texts, and significantly save training resources and inference overhead.
[0131] The communication bus 1140 mentioned in the above-mentioned electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus 1140 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.
[0132] The communication interface 1120 is used for communication between the above-mentioned electronic device and other devices.
[0133] The memory 1130 may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.
[0134] The processor 1110 mentioned above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0135] In another embodiment of this application, a computer-readable storage medium is provided, on which a program for a long text processing method is stored. When the program for the long text processing method is executed by a processor, it implements the steps of the long text processing method described in any of the foregoing method embodiments.
[0136] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0137] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.
Claims
1. A method for processing long text, characterized in that, include: Obtain the current key feature, current value feature, and current query feature corresponding to the current word element in the target long text, as well as the first global state feature corresponding to the previous word element; The current key feature, the current value feature, and the first global state feature are input into the encoder so that the encoder outputs a second global state feature, which includes: the high-level semantic information of the current word and all words preceding the current word; The second global state feature and the current query feature are input into the decoder so that the decoder outputs the lexical representation corresponding to the current lexical. The second global state feature includes a second global key feature and a second global value feature. Inputting the second global state feature and the current query feature into the decoder so that the decoder outputs the lexical representation corresponding to the current lexical includes: the decoder determining the high-level semantic information weight corresponding to the current lexical based on the current query feature and the second global key feature; and the decoder determining the lexical representation corresponding to the current lexical based on the high-level semantic information weight and the second global value feature.
2. The long text processing method according to claim 1, characterized in that, The first global state feature includes: a first global key feature and a first global value feature. The current key feature, the current value feature, and the first global state feature are input into the encoder so that the encoder outputs a second global state feature, including: The encoder determines the high-level semantic information of the current word and all words preceding the current word based on the current key feature, the current value feature, the first global key feature, and the first global value feature, and outputs it as the second global state feature.
3. The long text processing method according to claim 2, characterized in that, The encoder determines high-level semantic information of the current lexical and all lexical preceding the current lexical based on the current key feature, the current value feature, the first global key feature, and the first global value feature, as the second global state feature, including: The encoder concatenates the current key feature and the first global key feature to obtain the concatenated key feature; The encoder concatenates the current value feature and the first global value feature to obtain the concatenated value feature; The encoder inputs the splicing key feature and the splicing value feature into the first multilayer perceptron module to obtain the second global state feature.
4. The long text processing method according to claim 1, characterized in that, The decoder determines the high-level semantic information weights corresponding to the current lexical unit based on the current query features and the second global key features, including: The decoder concatenates the current query feature and the second global key feature to obtain the concatenated feature; The decoder inputs the spliced features into the second multilayer perceptron module to obtain the high-level semantic information weights corresponding to the current word.
5. The long text processing method according to claim 1, characterized in that, The decoder determines the lexical representation corresponding to the current lexical based on the high-level semantic information weights and the second global value feature, including: The decoder multiplies the high-level semantic information weights with the second global value features bitwise to obtain the lexical representation corresponding to the current lexical.
6. The long text processing method according to claim 1, characterized in that, Obtain the current key feature, current value feature, and current query feature corresponding to the current term in the target long text, including: Obtain the target long text, and segment the target long text into words to obtain multiple word units; Determine the feature vector corresponding to each of the lexical units; Perform matrix operations on the feature vector corresponding to each word and the key matrix to obtain the current key feature; Perform matrix operations on the feature vector and value matrix corresponding to each word to obtain the current value feature; Perform matrix operations on the feature vector corresponding to each term and the query matrix to obtain the current query feature.
7. A long text processing device, characterized in that, include: The acquisition module is used to acquire the current key feature, current value feature, and current query feature corresponding to the current word element in the target long text, as well as the first global state feature corresponding to the previous word element. The encoder input module is used to input the current key feature, the current value feature and the first global state feature into the encoder so that the encoder outputs a second global state feature. The second global state feature includes: the high-level semantic information of the current word and all words preceding the current word. A decoder input module is used to input the second global state feature and the current query feature into the decoder so that the decoder outputs the lexical representation corresponding to the current lexical. The second global state feature includes a second global key feature and a second global value feature. Inputting the second global state feature and the current query feature into the decoder so that the decoder outputs the lexical representation corresponding to the current lexical includes: the decoder determining the high-level semantic information weight corresponding to the current lexical based on the current query feature and the second global key feature; and the decoder determining the lexical representation corresponding to the current lexical based on the high-level semantic information weight and the second global value feature.
8. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the long text processing method according to any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program for a long text processing method, which, when executed by a processor, implements the steps of the long text processing method according to any one of claims 1-6.