Long text processing method and device, electronic equipment and storage medium

Abstract

The invention relates to a long text processing method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining a current key feature, a current value feature and a current query feature corresponding to a current lexical element in a target long text, and a first global state feature corresponding to a previous lexical element; the current key feature, the current value feature and the first global state feature are input into an encoder, so that the encoder outputs a second global state feature, and the second global state feature comprises the current lexical element and high-level semantic information of all lexical elements before the current lexical element; and inputting the second global state feature and the current query feature into a decoder, so that the decoder outputs a lexical element representation corresponding to the current lexical element. According to the embodiment of the invention, storage and computing resources can be saved, high-level semantic information can be effectively analyzed, long-distance context information can be effectively fused, and the understanding ability of the model to long texts can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a long text processing method, device, electronic device and storage medium. Background Art

[0002] With the rise of large-scale corpus-based pre-trained language models (LLMs), their application scenarios in long text processing have received increasing attention, including but not limited to the following aspects:

[0003] 1. Professional document reading and analysis: In the fields of scientific research, medicine, law, etc., professionals need to read and understand a large number of professional documents and reports. These documents are often very long, and the model needs to be able to process and understand long texts.

[0004] 2. Long books and documents: For users who need to quickly obtain summaries of long books or documents, long text processing capabilities can help them quickly grasp the core content.

[0005] 3. Content creation and editing: In the field of content creation, such as novel writing and script writing, the ability to process long texts can help authors organize their thoughts, generate creative ideas, or proofread manuscripts.

[0006] The traditional LLM-based Transformer architecture has the following problems when processing long texts: high resource overhead: each token corresponds to a set of q, k, and v features, resulting in high computational and storage costs; long contexts are difficult to process: as the sequence length increases, the computational complexity increases quadratically, making it difficult to process ultra-long texts. Summary of the invention

[0007] In order to solve the above technical problems or at least partially solve the above technical problems, the present application provides a long text processing method, device, electronic device and storage medium.

[0008] In a first aspect, the present application provides a long text processing method, comprising:

[0009] Obtain the current key feature, current value feature, and current query feature corresponding to the current word in the target long text, as well as the first global state feature corresponding to the previous word;

[0010] Inputting the current key feature, the current value feature and the first global state feature into an encoder so that the encoder outputs a second global state feature, wherein the second global state feature includes: high-level semantic information of the current word and all words before the current word;

[0011] The second global state feature and the current query feature are input into a decoder, so that the decoder outputs a word unit representation corresponding to the current word unit.

[0012] Optionally, the first global state feature includes: a first global key feature and a first global value feature, and the current key feature, the current value feature and the first global state feature are input into an encoder so that the encoder outputs a second global state feature, comprising:

[0013] The encoder determines high-level semantic information of the current word-gram and all word-grams before the current word-gram based on the current key feature, the current value feature, the first global key feature and the first global value feature as the second global state feature, and outputs it.

[0014] Optionally, the encoder determines high-level semantic information of the current word-gram and all word-grams before the current word-gram based on the current key feature, the current value feature, the first global key feature, and the first global value feature as the second global state feature, including:

[0015] The encoder performs splicing processing on the current key feature and the first global key feature to obtain a spliced ​​key feature;

[0016] The encoder performs splicing processing on the current value feature and the first global value feature to obtain a spliced ​​value feature;

[0017] The encoder inputs the splicing key feature and the splicing value feature into a first multi-layer perceptron module to obtain the second global state feature.

[0018] Optionally, the second global state feature includes: a second global key feature and a second global value feature, and the second global state feature and the current query feature are input into a decoder so that the decoder outputs a word-unit representation corresponding to the current word-unit, including:

[0019] The decoder determines the high-level semantic information weight corresponding to the current word according to the current query feature and the second global key feature;

[0020] The decoder determines a word-gram representation corresponding to the current word-gram based on the high-level semantic information weight and the second global value feature.

[0021] Optionally, the decoder determines the high-level semantic information weight corresponding to the current word according to the current query feature and the second global key feature, including:

[0022] The decoder performs splicing processing on the current query feature and the second global key feature to obtain a spliced ​​feature;

[0023] The decoder inputs the concatenated features into a second multi-layer perceptron module to obtain a high-level semantic information weight corresponding to the current word.

[0024] Optionally, the decoder determines a word-gram representation corresponding to the current word-gram based on the high-level semantic information weight and the second global value feature, including:

[0025] The decoder multiplies the high-level semantic information weight by the second global value feature bit by bit to obtain a word unit representation corresponding to the current word unit.

[0026] Optionally, the current key feature, current value feature, and current query feature corresponding to the current word in the target long text are obtained, including:

[0027] Obtain a target long text, and segment the target long text to obtain multiple word units;

[0028] Determine a feature vector corresponding to each of the word units;

[0029] Performing matrix operations on the feature vector corresponding to each word unit and the key matrix to obtain the current key feature;

[0030] Performing matrix operations on the feature vector corresponding to each word unit and the value matrix to obtain the current value feature;

[0031] A matrix operation is performed on the feature vector corresponding to each word unit and the query matrix to obtain the current query feature.

[0032] In a second aspect, the present application provides a long text processing device, comprising:

[0033] An acquisition module, used to acquire a current key feature, a current value feature, and a current query feature corresponding to a current word in a target long text, as well as a first global state feature corresponding to a previous word;

[0034] An encoder input module, configured to input the current key feature, the current value feature and the first global state feature into an encoder, so that the encoder outputs a second global state feature, wherein the second global state feature includes: high-level semantic information of the current word element and all word elements before the current word element;

[0035] The decoder input module is used to input the second global state feature and the current query feature into the decoder, so that the decoder outputs the word unit representation corresponding to the current word unit.

[0036] Optionally, the first global state feature includes: a first global key feature and a first global value feature, and the encoder input module includes:

[0037] The first determination unit is used for the encoder to determine the high-level semantic information of the current word and all word-grams before the current word-gram based on the current key feature, the current value feature, the first global key feature and the first global value feature, as the second global state feature, and output it.

[0038] Optionally, the first determining unit is further configured to:

[0039] The encoder performs splicing processing on the current key feature and the first global key feature to obtain a spliced ​​key feature;

[0040] The encoder performs splicing processing on the current value feature and the first global value feature to obtain a spliced ​​value feature;

[0041] The encoder inputs the splicing key feature and the splicing value feature into a first multi-layer perceptron module to obtain the second global state feature.

[0042] Optionally, the second global state feature includes: a second global key feature and a second global value feature, and the decoder input module includes:

[0043] A second determining unit, configured for the decoder to determine a high-level semantic information weight corresponding to a current word element according to the current query feature and the second global key feature;

[0044] The third determining unit is used for the decoder to determine the word unit representation corresponding to the current word unit based on the high-level semantic information weight and the second global value feature.

[0045] Optionally, the second determining unit is further configured to:

[0046] The decoder performs splicing processing on the current query feature and the second global key feature to obtain a spliced ​​feature;

[0047] The decoder inputs the concatenated features into a second multi-layer perceptron module to obtain a high-level semantic information weight corresponding to the current word.

[0048] Optionally, the third determining unit is further used to:

[0049] The decoder multiplies the high-level semantic information weight by the second global value feature bit by bit to obtain a word unit representation corresponding to the current word unit.

[0050] Optionally, the acquisition module includes:

[0051] An acquisition unit, used for acquiring a target long text, segmenting the target long text into words, and obtaining a plurality of word units;

[0052] A fourth determining unit, used to determine a feature vector corresponding to each word unit;

[0053] A first operation unit, configured to perform a matrix operation on the feature vector corresponding to each word unit and the key matrix to obtain the current key feature;

[0054] A second operation unit is used to perform a matrix operation on the feature vector corresponding to each word unit and the value matrix to obtain the current value feature;

[0055] The third operation unit is used to perform a matrix operation on the feature vector corresponding to each word unit and the query matrix to obtain the current query feature.

[0056] In a third aspect, the present application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0057] Memory, used to store computer programs;

[0058] The processor is used to implement any long text processing method described in the first aspect when executing the program stored in the memory.

[0059] In a fourth aspect, the present application provides a computer-readable storage medium, on which a program of a long text processing method is stored. When the program of the long text processing method is executed by a processor, the steps of any of the long text processing methods described in the first aspect are implemented.

[0060] The above technical solution provided by the embodiment of the present application has the following advantages compared with the prior art:

[0061] The encoder of the embodiment of the present application performs dimensionality reduction compression on the current key features and current value features of the current word, saving storage and computing resources. The decoder can effectively parse high-level semantic information. The encoder-decoder structure can effectively integrate long-distance contextual information, improve the model's ability to understand long texts, and can significantly save training resources and reasoning overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0063] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0064] Figure 1 A flowchart of a long text processing method provided in an embodiment of the present application;

[0065] Figure 2 A schematic diagram of a long text processing method provided in an embodiment of the present application;

[0066] Figure 3 A structural diagram of a long text processing device provided in an embodiment of the present application;

[0067] Figure 4 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0069] The traditional LLM-based Transformer architecture generates corresponding query, key, and value features when processing each input word of a long text, and performs a self-attention operation. Self-Attention allows each word in a sentence to pay attention to all words in the sentence, thereby capturing the long-term dependencies between words. The principle is as follows: Generate Q, K, and V matrices: For each token (word element) vector in the input sentence, multiply it by three different matrices Wq, Wk, and Wv to obtain the corresponding Query (Q), Key (K), and Value (V) vectors; Calculate the attention score: Calculate the dot product between each Query and all Keys to obtain the attention score. The higher the attention score, the stronger the correlation between the two words; Scale and normalize: Divide the attention score by the square root of the Key vector dimension, and then use the Softmax function for normalization to obtain the attention weight between each word and all other words; Weighted sum: Multiply the Value matrix by the attention weight matrix to obtain the contextual representation of each word. This means that the final representation of each word is obtained by the weighted sum of all words related to it in the sentence.

[0070] Simply put, the Self-Attention mechanism uses Q to match K, finds the relevant Value, and then weighted sums these Values ​​to get the final word representation.

[0071] However, the above method of processing long texts has the following problems: high resource overhead: each token corresponds to a set of q, k, v features, resulting in high computational and storage costs; long contexts are difficult to process: as the sequence length increases, the computational complexity increases quadratically, making it difficult to process ultra-long texts. To this end, the embodiments of the present application provide a long text processing method, device, electronic device, and storage medium, which improves the kv feature dimension of the LLM model so that the model can adaptively learn the ability to compress and select features on long contexts during training; and automatically perform feature compression and selection on long contexts during reasoning.

[0072] The present application embodiment provides a long text processing method, which can be applied to the training stage and reasoning stage of the LLM model, such as Figure 1 As shown, the long text processing method includes:

[0073] Step S101, obtaining a current key feature, a current value feature, and a current query feature corresponding to a current word in a target long text, as well as a first global state feature corresponding to a previous word;

[0074] In the embodiment of the present application, the target long text may refer to a long text to be processed, such as a novel, a book, a document, a script, etc.

[0075] The previous word refers to a word before the current word, that is, the word adjacent to the current word and located before the current word. The first global state feature is obtained by inputting the key feature, value feature of the previous word and the global state feature corresponding to the previous word into the encoder.

[0076] In one embodiment of the present application, step S101 obtains the current key feature, current value feature and current query feature corresponding to the current word in the target long text, including:

[0077] Obtain the target long text, segment the target long text, and obtain multiple tokens; determine the feature vector corresponding to each token; perform matrix operation on the feature vector corresponding to each token and the key matrix to obtain the current key feature K t ; Perform matrix operations on the feature vector corresponding to each word element and the value matrix to obtain the current value feature V t ; Perform matrix operations on the feature vector corresponding to each word element and the query matrix to obtain the current query feature Q t .

[0078] That is, first segment the target long text to obtain multiple tokens. A token can be a single character or a word, and then each token will correspond to a unique id, and then each id will correspond to a feature vector (embedding). The feature vector of each token will be transformed into the vector of the corresponding current key feature, current value feature and current query feature after matrix operations of the preset query matrix Query, the preset key matrix Key and the preset value matrix Value. The role of the current query feature is mainly to guide the model to pay attention to relevant information, capture semantic relevance, and ultimately help the model better understand the meaning of the language. Each token will correspond to three feature vectors Q, K and V in each encoder structure.

[0079] Step S102, inputting the current key feature, the current value feature and the first global state feature into an encoder, so that the encoder outputs a second global state feature;

[0080] In this step, if Figure 2 As shown, the encoder can use the gating mechanism to fuse the current key feature K t , current value characteristic V t and the first global state feature (HK t-1 , H.V. t-1 ), and obtain the second global state feature, that is, the new hidden state (HK t , H.V. t ), the second global state feature includes: high-level semantic information of the current word and all words before the current word.

[0081] Step S103: input the second global state feature and the current query feature into a decoder, so that the decoder outputs a word-gram representation corresponding to the current word-gram.

[0082] In this step, if Figure 2 As shown, the decoder calculates the current query feature Q t With the second global state feature HK t The high-level semantic information weight related to the current token is extracted, and the extracted high-level semantic information weight is combined with HV in the second global state feature. t Fusion is performed to generate the output feature O of the current token t , that is, word-unit representation.

[0083] It should be noted that the number of encoder and decoder structures in this application is the same as the current LLM network structure setting, and the common number is 32 to 80.

[0084] After the decoder outputs the word-gram representation corresponding to each word-gram, LLM can generate summary information of the target long context based on the word-gram representation corresponding to each word-gram. This process is not improved in this scheme and will not be described in detail in this article.

[0085] The encoder of the embodiment of the present application performs dimensionality reduction compression on the current key features and current value features of the current word, saving storage and computing resources. The decoder can effectively parse high-level semantic information. The encoder-decoder structure can effectively integrate long-distance contextual information, improve the model's ability to understand long texts, and can significantly save training resources and reasoning overhead.

[0086] In another embodiment of the present application, the first global state feature includes: a first global key feature and a first global value feature, and step S102 inputs the current key feature, the current value feature and the first global state feature into an encoder so that the encoder outputs a second global state feature, including:

[0087] The encoder determines high-level semantic information of the current word-gram and all word-grams before the current word-gram based on the current key feature, the current value feature, the first global key feature and the first global value feature as the second global state feature, and outputs it.

[0088] In one embodiment of the present application, the encoder determines the high-level semantic information of the current word-gram and all word-grams before the current word-gram based on the current key feature, the current value feature, the first global key feature, and the first global value feature as the second global state feature, including:

[0089] The encoder performs splicing processing on the current key feature and the first global key feature to obtain a spliced ​​key feature;

[0090] The encoder performs splicing processing on the current value feature and the first global value feature to obtain a spliced ​​value feature;

[0091] The encoder inputs the splicing key feature and the splicing value feature into a first multi-layer perceptron module to obtain the second global state feature.

[0092] The second global state feature (HK t , H.V. t ) dimension and the first global state feature (HK t-1 ,HV t-1 ) are consistent in dimension.

[0093] The embodiment of the present application uses an encoder to perform dimensionality reduction compression on the current key feature and the current value feature of the current word, thereby saving storage and computing resources. In the encoder, a new second global state feature is regenerated based on the current key feature, the current value feature of the current word and the first global state feature of the previous word, and is used for the calculation of the next word. In this way, the semantic information of the sequence before the current word can be continuously updated. During the training process, the model will adaptively learn the semantic information extraction capability of long sequence features, thereby fully realizing the extraction and fusion of long sequence high-level semantic information.

[0094] In another embodiment of the present application, the second global state feature includes: a second global key feature and a second global value feature. Step S103 inputs the second global state feature and the current query feature into a decoder so that the decoder output includes:

[0095] Step S201, the decoder determines the high-level semantic information weight corresponding to the current word according to the current query feature and the second global key feature;

[0096] In one embodiment of the present application, the decoder determines the high-level semantic information weight corresponding to the current word according to the current query feature and the second global key feature, including:

[0097] The decoder performs splicing processing on the current query feature and the second global key feature to obtain a spliced ​​feature;

[0098] The decoder inputs the concatenated features into a second multi-layer perceptron module to obtain a high-level semantic information weight corresponding to the current word.

[0099] Step S202: The decoder determines a word-gram representation corresponding to the current word-gram based on the high-level semantic information weight and the second global value feature.

[0100] In one embodiment of the present application, the decoder determines the word-unit representation corresponding to the current word-unit based on the high-level semantic information weight and the second global value feature, including:

[0101] The decoder multiplies the high-level semantic information weight by the second global value feature bit by bit to obtain the word-unit representation corresponding to the current word-unit, which is equivalent to screening the second global value feature according to each high-level semantic information weight, retaining the desired information and suppressing the unwanted information.

[0102] The decoder of the embodiment of the present application fully considers the characteristics of the current token while integrating the context information, thereby ensuring the general performance of the model; after passing through the decoder, the high-level semantic information can be effectively parsed and parsed according to the output feature format. Similarly, as the training process proceeds, LLM can quickly learn how to parse the semantic information of long sequences, thereby effectively extracting high-level semantic information of long sequences.

[0103] In another embodiment of the present application, a long text processing device is also provided, such as Figure 3 As shown, including:

[0104] An acquisition module 11 is used to acquire a current key feature, a current value feature, and a current query feature corresponding to a current word in a target long text, as well as a first global state feature corresponding to a previous word;

[0105] The encoder input module 12 is used to input the current key feature, the current value feature and the first global state feature into the encoder, so that the encoder outputs a second global state feature, wherein the second global state feature includes: high-level semantic information of the current word and all words before the current word;

[0106] The decoder input module 13 is used to input the second global state feature and the current query feature into the decoder, so that the decoder outputs the word unit representation corresponding to the current word unit.

[0107] Optionally, the first global state feature includes: a first global key feature and a first global value feature, and the encoder input module includes:

[0108] The first determination unit is used for the encoder to determine the high-level semantic information of the current word and all word-grams before the current word-gram based on the current key feature, the current value feature, the first global key feature and the first global value feature, as the second global state feature, and output it.

[0109] Optionally, the first determining unit is further configured to:

[0110] The encoder performs splicing processing on the current key feature and the first global key feature to obtain a spliced ​​key feature;

[0111] The encoder performs splicing processing on the current value feature and the first global value feature to obtain a spliced ​​value feature;

[0112] The encoder inputs the splicing key feature and the splicing value feature into a first multi-layer perceptron module to obtain the second global state feature.

[0113] Optionally, the second global state feature includes: a second global key feature and a second global value feature, and the decoder input module includes:

[0114] A second determining unit, configured for the decoder to determine a high-level semantic information weight corresponding to a current word element according to the current query feature and the second global key feature;

[0115] The third determining unit is used for the decoder to determine the word unit representation corresponding to the current word unit based on the high-level semantic information weight and the second global value feature.

[0116] Optionally, the second determining unit is further configured to:

[0117] The decoder performs splicing processing on the current query feature and the second global key feature to obtain a spliced ​​feature;

[0118] The decoder inputs the concatenated features into a second multi-layer perceptron module to obtain a high-level semantic information weight corresponding to the current word.

[0119] Optionally, the third determining unit is further used to:

[0120] The decoder multiplies the high-level semantic information weight by the second global value feature bit by bit to obtain a word unit representation corresponding to the current word unit.

[0121] Optionally, the acquisition module includes:

[0122] An acquisition unit, used for acquiring a target long text, segmenting the target long text into words, and obtaining a plurality of word units;

[0123] A fourth determining unit, used to determine a feature vector corresponding to each word unit;

[0124] A first operation unit, configured to perform a matrix operation on the feature vector corresponding to each word unit and the key matrix to obtain the current key feature;

[0125] A second operation unit is used to perform a matrix operation on the feature vector corresponding to each word unit and the value matrix to obtain the current value feature;

[0126] The third operation unit is used to perform a matrix operation on the feature vector corresponding to each word unit and the query matrix to obtain the current query feature.

[0127] In another embodiment of the present application, an electronic device is provided, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus;

[0128] Memory, used to store computer programs;

[0129] The processor is used to implement the long text processing method described in any of the above method embodiments when executing the program stored in the memory.

[0130] In the electronic device provided by the embodiment of the present invention, the processor executes the program stored in the memory, and uses the encoder to perform dimensionality reduction compression on the current key features and current value features of the current word, thereby saving storage and computing resources. The decoder can effectively parse high-level semantic information. The encoder-decoder structure can effectively integrate long-distance contextual information, improve the model's ability to understand long texts, and can significantly save training resources and reasoning overhead.

[0131] The communication bus 1140 mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The communication bus 1140 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0132] The communication interface 1120 is used for communication between the above electronic device and other devices.

[0133] The memory 1130 may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0134] The above-mentioned processor 1110 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0135] In another embodiment of the present application, a computer-readable storage medium is provided, on which a program of the long text processing method is stored. When the program of the long text processing method is executed by a processor, the steps of the long text processing method described in any of the aforementioned method embodiments are implemented.

[0136] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0137] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A long text processing method, characterized in that: include: Obtain the current key feature, current value feature, and current query feature corresponding to the current word in the target long text, as well as the first global state feature corresponding to the previous word; Inputting the current key feature, the current value feature and the first global state feature into an encoder so that the encoder outputs a second global state feature, wherein the second global state feature includes: high-level semantic information of the current word and all words before the current word; The second global state feature and the current query feature are input into a decoder, so that the decoder outputs a word unit representation corresponding to the current word unit.

2. The long text processing method according to claim 1, characterized in that: The first global state feature includes: a first global key feature and a first global value feature, and the current key feature, the current value feature and the first global state feature are input into an encoder so that the encoder outputs a second global state feature, including: The encoder determines high-level semantic information of the current word-gram and all word-grams before the current word-gram based on the current key feature, the current value feature, the first global key feature and the first global value feature as the second global state feature, and outputs it.

3. The long text processing method according to claim 2, characterized in that: The encoder determines high-level semantic information of the current word-gram and all word-grams before the current word-gram based on the current key feature, the current value feature, the first global key feature, and the first global value feature as the second global state feature, including: The encoder performs splicing processing on the current key feature and the first global key feature to obtain a spliced ​​key feature; The encoder performs splicing processing on the current value feature and the first global value feature to obtain a spliced ​​value feature; The encoder inputs the splicing key feature and the splicing value feature into a first multi-layer perceptron module to obtain the second global state feature.

4. The long text processing method according to claim 1, characterized in that: The second global state feature includes: a second global key feature and a second global value feature, and the second global state feature and the current query feature are input into a decoder so that the decoder outputs a word unit representation corresponding to the current word unit, including: The decoder determines the high-level semantic information weight corresponding to the current word according to the current query feature and the second global key feature; The decoder determines a word-gram representation corresponding to the current word-gram based on the high-level semantic information weight and the second global value feature.

5. The long text processing method according to claim 4, characterized in that: The decoder determines the high-level semantic information weight corresponding to the current word according to the current query feature and the second global key feature, including: The decoder performs splicing processing on the current query feature and the second global key feature to obtain a spliced ​​feature; The decoder inputs the concatenated features into a second multi-layer perceptron module to obtain a high-level semantic information weight corresponding to the current word.

6. The long text processing method according to claim 4, characterized in that: The decoder determines the word-unit representation corresponding to the current word-unit based on the high-level semantic information weight and the second global value feature, including: The decoder multiplies the high-level semantic information weight by the second global value feature bit by bit to obtain a word unit representation corresponding to the current word unit.

7. The long text processing method according to claim 1, characterized in that: Get the current key feature, current value feature, and current query feature corresponding to the current word in the target long text, including: Obtain a target long text, and segment the target long text to obtain multiple word units; Determine a feature vector corresponding to each of the word units; Performing matrix operations on the feature vector corresponding to each word unit and the key matrix to obtain the current key feature; Performing matrix operations on the feature vector corresponding to each word unit and the value matrix to obtain the current value feature; A matrix operation is performed on the feature vector corresponding to each word unit and the query matrix to obtain the current query feature.

8. A long text processing device, characterized in that: include: An acquisition module, used to acquire a current key feature, a current value feature, and a current query feature corresponding to a current word in a target long text, as well as a first global state feature corresponding to a previous word; An encoder input module, configured to input the current key feature, the current value feature and the first global state feature into an encoder, so that the encoder outputs a second global state feature, wherein the second global state feature includes: high-level semantic information of the current word element and all word elements before the current word element; The decoder input module is used to input the second global state feature and the current query feature into the decoder, so that the decoder outputs the word unit representation corresponding to the current word unit.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; The processor is used to implement the long text processing method described in any one of claims 1 to 7 when executing the program stored in the memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program of the long text processing method, and when the program of the long text processing method is executed by the processor, the steps of the long text processing method described in any one of claims 1 to 7 are implemented.

Images