Long text processing method and device, electronic equipment and storage medium

Abstract

The invention provides a long text processing method and device, electronic equipment and a storage medium, and the method comprises the steps: inputting long text data into a long text processing model, and obtaining a long text processing result; the long text processing model is constructed by adopting a hierarchical mixed attention strategy on the basis of a large language model; according to the hierarchical mixed attention strategy, a first attention mechanism in each network structure of a large language model is replaced by an attention mechanism with a global information retrieval capability, and other attention mechanisms are replaced by attention mechanisms with a local information modeling capability. The hierarchical mixed attention architecture can greatly improve the reasoning efficiency on a long text while ensuring the reasoning effect of the model, reduces the consumption of computing resources, and approaches the modeling effect when only global attention is used.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a long text processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] In recent years, thanks to breakthroughs in deep learning technology and the continuous accumulation of large-scale, high-quality corpora, large language models have demonstrated outstanding performance in many natural language processing tasks. By learning common language patterns and knowledge from massive amounts of text, they have effectively reduced the dependence of downstream tasks on labeled data, thereby promoting the widespread application of natural language processing technology in many practical scenarios.

[0003] However, with the increase in model size and capabilities, their training and application face increasingly severe challenges. The most prominent of these is the constraint of huge computing resource requirements, especially when processing long text sequences. The current Transformer architecture of large-scale language models has obvious defects in computational complexity, namely, the computational complexity increases quadratically with the sequence length. This leads to two serious problems for models that perform well in processing short texts when facing application scenarios that require inputting or generating long texts, such as document summarization, code analysis, long text reading comprehension, and industry report processing. These problems are extremely low inference efficiency, which cannot meet the real-time requirements of practical applications, and huge resource consumption, with a sharp increase in the required memory (GPU memory) and computing power. This limits the application of models in long text processing. Summary of the Invention

[0004] This invention provides a long text processing method, apparatus, electronic device, and storage medium to solve the problems of low inference efficiency and huge resource consumption of large language models in long text processing in the prior art, and to achieve a reduction in computational complexity, an improvement in inference efficiency, a reduction in resource consumption, and a reduction in cost when processing long text.

[0005] This invention provides a long text processing method, comprising: Identify the long text data to be processed; The long text data is input into the long text processing model to obtain the long text processing result output by the long text processing model. The long text processing model is built on the basis of a large language model and adopts a hierarchical hybrid attention strategy. The hierarchical hybrid attention strategy is to replace the first attention mechanism in each network structure of the large language model with a global layer and replace the second attention mechanism in each network structure with a local layer. The first attention mechanism and the second attention mechanism are respectively the first attention mechanism and other attention mechanisms in each of the network structures; The global layer is an attention mechanism with global information retrieval capabilities, and the local layer is an attention mechanism with local information modeling capabilities.

[0006] According to a long text processing method provided by the present invention, each network structure is obtained by equally dividing all network layers in the large language model according to the connection order between network layers. Each network structure contains at least one network layer. The first attention mechanism is the attention mechanism in the first network layer of each network structure, and the second attention mechanism is the attention mechanism in the other network layers of each network structure except the first network layer.

[0007] According to a long text processing method provided by the present invention, the global layer adopts a global attention mechanism, and the local layer adopts a sliding window attention mechanism.

[0008] According to a long text processing method provided by the present invention, the global layer adopts a global attention mechanism, and the local layer adopts a block sparse attention mechanism.

[0009] According to a long text processing method provided by the present invention, the global layer adopts a block sparse attention mechanism, and the local layer adopts a sliding window attention mechanism.

[0010] According to a long text processing method provided by the present invention, the global layer is constructed based on the following steps: An initial global layer is determined, wherein the initial global layer is an attention mechanism with global information retrieval capabilities; Remove the rotation position encoding from the initial global layer to obtain the global layer.

[0011] According to a long text processing method provided by the present invention, the step of removing the rotation position encoding in the initial global layer to obtain the global layer includes: Remove the rotation position encoding from the initial global layer to obtain the target global layer; An attention scaling mechanism is added to the target global layer to obtain the global layer.

[0012] According to a long text processing method provided by the present invention, the attention scaling mechanism is used to scale the query vector at each position in the long text data, and the scaling function of the query vector is determined based on the corresponding position.

[0013] The present invention also provides a long text processing apparatus, comprising: The determination unit is used to determine the long text data to be processed. The processing unit is used to input the long text data into the long text processing model and obtain the long text processing result output by the long text processing model. The long text processing model is built on the basis of a large language model and adopts a hierarchical hybrid attention strategy. The hierarchical hybrid attention strategy is to replace the first attention mechanism in each network structure of the large language model with a global layer and replace the second attention mechanism in each network structure with a local layer. The first attention mechanism and the second attention mechanism are respectively the first attention mechanism and other attention mechanisms in each of the network structures; The global layer is an attention mechanism with global information retrieval capabilities, and the local layer is an attention mechanism with local information modeling capabilities.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the long text processing method as described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the long text processing method as described above.

[0016] The long text processing method, apparatus, electronic device, and storage medium provided by this invention obtain long text processing results by inputting the long text data to be processed into a long text processing model. The long text processing model is constructed on the basis of a large language model using a hierarchical hybrid attention strategy. The hierarchical hybrid attention strategy replaces the first attention mechanism in each network structure of the large language model with an attention mechanism that has global information retrieval capabilities, and replaces other attention mechanisms with attention mechanisms that have local information modeling capabilities. This hierarchical hybrid attention architecture can greatly improve the inference efficiency on long texts while ensuring that the performance of the model on downstream tasks is not reduced, and significantly reduce the consumption of computing resources and memory usage, approaching the modeling effect when only global attention is used, thus achieving a balance between inference efficiency, inference effect, and computing resources. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the long text processing method provided by the present invention; Figure 2 This is an example diagram of global attention weight calculation provided by the present invention; Figure 3 This is an example diagram of the sliding window attention mechanism provided by the present invention; Figure 4 This is an example diagram of the blocky sparse attention mechanism provided by the present invention; Figure 5 This is an example diagram of the network structure after replacement provided by the present invention; Figure 6 This is one of the example diagrams of the hierarchical hybrid attention architecture provided by the present invention; Figure 7 This is the second example diagram of the hierarchical hybrid attention architecture provided by the present invention; Figure 8 This is the third example diagram of the hierarchical hybrid attention architecture provided by the present invention; Figure 9 This is a schematic diagram of the structure of the long text processing device provided by the present invention; Figure 10 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0020] Currently, solutions for expanding the long text processing capabilities of large language models mainly revolve around the Transformer architecture, focusing on improving the computational efficiency of the attention mechanism within the Transformer architecture. The mainstream attention mechanisms can be categorized into two main types: Full Attention and Sparse Attention. The following sections will provide a detailed introduction to these two mainstream approaches and analyze their limitations.

[0021] Global attention is the standard attention mechanism used in the original Transformer model and most current mainstream large-scale language models during the training phase. Its core is that when calculating the attention weight, each token at each position needs to interact with all other tokens in the input sequence. Figure 2 This is an example diagram of global attention weight calculation provided by the present invention, such as... Figure 2As shown (assuming decoding "word 6"), the green area covers the entire input sequence, indicating that the attention window size for each token is equal to the full sequence length (such as the commonly used 32k during training). This mechanism endows the model with powerful global dependency modeling capabilities, allowing any token to directly perceive information about any distance in the sequence. However, this mechanism also has obvious drawbacks, namely, the computational resource consumption increases quadratically with the sequence length, resulting in a computational complexity of O(n²), where n is the sequence length.

[0022] Sparse attention includes sliding window attention, arrowhead attention, and block sparse attention. The core idea of ​​these attention methods is to reduce computational complexity by limiting the scope that each token needs to focus on. It no longer requires each token to focus on all the tokens preceding it, but only on a predefined or dynamically selected sparse subset.

[0023] Taking sliding window attention as an example, Figure 3 This is an example diagram of the sliding window attention mechanism provided by the present invention, such as... Figure 3 As shown, in this attention mechanism, each token only pays attention to the w tokens preceding it, such as... Figure 3 The computational complexity of the three tokens in the sequence is O(w*n), where n is the sequence length. This shows that for a fixed w, the computational complexity of sliding window attention increases linearly with the sequence length, rather than quadratically. Arrow attention is similar to sliding window attention, but it also pays extra attention to the first few tokens, with the same computational complexity. However, neither of these attention mechanisms can accurately capture information outside the window when the sequence length exceeds the window size.

[0024] In addition, block sparse attention is also widely used. Figure 4 This is an example diagram of the blocky sparse attention mechanism provided by the present invention, such as... Figure 4 As shown, this attention mechanism divides the sequence into blocks of fixed size, and each token only focuses on the k most relevant tokens within the blocks. Its computational complexity is O(k*b*n), where b is the size of each block. Figure 4 (where b is 1 and k is 3). This mechanism enables blocky sparse attention to have both global information retrieval and local information modeling capabilities. However, it is weaker than the global attention mechanism in terms of global information retrieval capabilities, and weaker than the sliding window attention mechanism in terms of local information modeling capabilities.

[0025] In summary, both current solutions using global attention mechanisms and those using sparse attention mechanisms have drawbacks. Solutions using global attention mechanisms have high computational complexity, high resource consumption, high cost, and low inference efficiency, while solutions using sparse attention mechanisms risk information loss and cannot completely accelerate the entire inference process or significantly reduce memory usage.

[0026] Specifically, the computational complexity of the global attention mechanism is proportional to the square of the sequence length. This means that when processing long sequences of tens of thousands or even millions, the required computation and memory usage become extremely large, far exceeding the capacity of conventional computing devices. This high computational complexity directly leads to a significant increase in inference time, making it difficult to meet the needs of real-time or near-real-time interactive applications. Furthermore, processing long sequences during the training phase also faces a bottleneck in computational complexity, which significantly increases the time required for single-step training, memory consumption, and overall training cost.

[0027] Correspondingly, forced sparsity may truncate or lose long-distance critical dependencies. For reasoning tasks requiring global context (such as question answering, referencing resolution, and document comprehension), the sparsified attention window cannot capture enough information in long sequences. Moreover, strategies similar to block-based sparse attention require caching all information, failing to significantly reduce memory usage and hindering speech acceleration. Furthermore, optimal sparsity patterns are often highly dependent on specific tasks and data distributions, lacking generality. Predefined fixed patterns (such as sliding window attention) are not applicable to all scenarios and cannot accurately recall information outside the window.

[0028] In response, this invention provides a long text processing method that aims to improve the inference efficiency of large language models on long texts by mixing different types of attention in different network layers. While greatly reducing the consumption of computing resources and memory usage, it approaches the modeling effect of the original large language model using only global attention, breaking through the current technical bottleneck. This unlocks the potential of large language models in key areas of long text processing such as long text analysis, knowledge mining, and automated document processing, and has important theoretical value and urgent practical needs.

[0029] Figure 1 This is a flowchart illustrating the long text processing method provided by the present invention, as shown below. Figure 1 As shown, the method includes: Step 110: Determine the long text data to be processed; Step 120: Input the long text data into the long text processing model to obtain the long text processing result output by the long text processing model; The long text processing model is built on the basis of a large language model and adopts a hierarchical hybrid attention strategy. The hierarchical hybrid attention strategy replaces the first attention mechanism in each network structure of the large language model with a global layer and replaces the second attention mechanism in each network structure with a local layer. The first attention mechanism and the second attention mechanism are the first attention mechanism and other attention mechanisms in each network structure, respectively. The global layer is an attention mechanism with global information retrieval capability, and the local layer is an attention mechanism with local information modeling capability.

[0030] Specifically, considering that the computational complexity of the attention mechanism at the core of the Transformer architecture of current large language models (hereinafter referred to as large models) increases quadratically with the sequence length when processing long text sequences, resulting in models that perform well when processing short texts exhibiting low inference efficiency, a sharp increase in the time required to generate / understand long texts, and huge resource consumption when dealing with long texts, with a dramatic increase in the memory (GPU memory) and computing power required to process long sequences, this invention proposes a hierarchical hybrid attention architecture to replace the current architecture in large language models based on Transformer that uses global attention throughout. This significantly reduces the computational complexity when processing long sequences, improves inference efficiency, reduces GPU memory usage, and at the same time maximizes the retention of the ability of large models to capture long-distance dependencies.

[0031] In detail, before processing long text, it is necessary to first determine the long text to be processed, that is, the long text data to be processed. The long text data here can be long articles, report data, development code, etc., and this embodiment of the invention does not make specific limitations on this.

[0032] Once the long text data to be processed is determined, this embodiment of the invention can process the long text data to obtain the corresponding processing result. That is, a large model can be used to process long text to obtain the long text processing result output by the large model. However, considering the current problems of large models in long text processing, in this embodiment of the invention, in order to improve the inference efficiency and reduce resource consumption during long text processing, the structure of the large model can be improved. That is, the large model can be improved by using a hierarchical hybrid attention strategy, and a long text processing model can be built based on the improved large model to process the long text data.

[0033] Here, the hierarchical hybrid attention strategy refers to replacing the original global attention architecture in a large model with a hierarchical hybrid attention architecture. Specifically, the original large language model based on Transformer uses a global attention mechanism throughout, which is the original global attention architecture. In this embodiment of the invention, to reduce computational complexity and avoid the problem of computational complexity increasing quadratically with sequence length, an alternating hierarchical processing of the attention mechanism in the original large model is proposed. That is, the network layers of the large model are divided into multiple network structures, each containing at least one network layer. Then, the first attention mechanism in each network structure is replaced with a global layer. In other words, the first (global) attention mechanism in each network structure is replaced with an attention mechanism with global information retrieval capabilities, i.e., a global layer. The global layer here can be a block-based sparse attention mechanism, a global attention mechanism, etc. When a global attention mechanism is used, since the result before and after the replacement is the same, this replacement operation can be regarded as the retention of the global attention mechanism. When a non-global attention mechanism is used, such as a block-based sparse attention mechanism, this replacement can speed up the inference speed of the model when processing long texts, thus achieving inference acceleration.

[0034] Accordingly, in addition to replacing the first attention mechanism in each network structure with a global layer, the hierarchical hybrid attention strategy also proposes replacing other (global) attention mechanisms in each network structure besides the first attention mechanism, i.e., the second attention mechanism, with attention mechanisms capable of modeling local information, i.e., local layers. These local layers can employ sliding window attention mechanisms, blocky sparse attention mechanisms, linear attention mechanisms, etc.

[0035] Furthermore, based on a hierarchical hybrid attention strategy, after replacing the attention mechanism of the original large model, in this embodiment of the invention, a model for long text processing, i.e., a long text processing model, can be constructed based on the replaced / improved large model. Here, the replaced / improved large model can be directly used as the long text processing model, or the replaced / improved large model can be used as the initial model and fine-tuned and trained using a small amount of sample long text data to obtain a trained model, which can then be used as the long text processing model.

[0036] As a preferred embodiment of the invention, fine-tuning training is performed, and the fine-tuned model is used as the long text processing model. This setting not only improves the performance of the model on specific long text processing tasks, but also avoids the large amount of time and resources required for "training from scratch". A good effect can be achieved by fine-tuning a small amount of data based on the replaced / improved large model. This can accelerate the model inference efficiency and reduce the memory usage without reducing performance, thus avoiding the high cost of computing resources and time caused by training the model from scratch.

[0037] Once the long text processing model is obtained, in this embodiment of the invention, long text processing can be performed using it. That is, long text data can be input into the long text processing model so that the model can understand, analyze and process it, and output the processed result, i.e., the long text processing result.

[0038] It's important to note that, compared to current large-scale models that use global attention mechanisms, when processing long text, they typically generate query vectors, key vectors, and value vectors for each token in the input long text data, and calculate attention weights among all tokens to achieve information interaction across the entire sequence. The core process is "input embedding (word vectors + positional encoding) - multi-layer self-attention (global computation in each layer) - feedforward network and residual connections - output generation." While this model effectively captures long-range dependencies, its computational complexity (O(n²)) is extremely high. In this embodiment, the layered hybrid attention architecture, by alternately setting global and local layers, can effectively balance inference efficiency and expressive power. Specifically, the local layer restricts each token to interact only with tokens within a fixed window (e.g., a sliding window), thereby reducing computational complexity; the global layer, through global computation (retaining the global computational power of the first key layer in each network structure), can capture long-range dependencies, thus achieving a balance between inference efficiency and inference effect. The core process in this embodiment is "input embedding - alternating global / local attention processing - feedforward network and residual connections - output generation."

[0039] Specifically, in this embodiment of the invention, for the input long text data, in each network structure, the global layer can first capture the global dependencies of all tokens in the sequence. For example, a linear transformation can be used to map the input long text data or the long text sequence output by the previous network structure to queries, keys, and values, and attention weights can be calculated. Next, the local context can be focused to reduce the amount of computation. That is, the output of the global layer can be mapped to new queries, keys, and values, and local attention calculations can be performed. If a sliding window attention mechanism is used, each word token in the long text sequence only calculates attention with w surrounding word tokens, and the interaction range is limited by an attention mask. Finally, the output is obtained through residual connections and layer normalization. After passing through multiple network structures, the final long text processing result can be obtained.

[0040] For example, for a long news report containing 5000 tokens, it is necessary to extract the event timeline and core viewpoints. In each network structure, global dependencies in the long text can be captured in the first layer, such as the relationship between "social practice" mentioned at the beginning and "safety education" and "psychological counseling" in the following text. For example, the grammar and short-distance references within sentences in the long text can be analyzed. Specifically, this can be achieved by generating queries, keys, and values ​​through linear transformations, calculating attention weights, and finally outputting through residual connections and layer normalization. It is worth noting that when processing "social practice," the global layer can simultaneously focus on the 100th token "homework tutoring" and the 4000th token "knowledge transfer." Furthermore, after the initial output, subsequent network layers can be accessed. Local layers focus on local contexts, such as the internal grammar of sentences in long news reports and short-distance referencing. Specifically, the output of the global layer can be mapped to new queries, keys, and values, and local attention calculations can be performed. If a sliding window attention mechanism is used, each token interacts only with its 128 surrounding tokens, and the interaction range is limited by an attention mask. Finally, residual connections and layer normalization are used for output. It is worth noting that when processing the sentence "In summer teaching activities, young students achieved an organic unity of knowledge transfer, ability enhancement, and firm belief" from a long news report, the local layer can focus on the grammatical relationship between "teaching activities" and "knowledge transfer." After processing through multiple network structures, the final long text processing result can be obtained.

[0041] It is worth noting here that, unlike the global attention mechanism of traditional large models, which requires attention to information from all other locations when calculating the attention weight at each location, this embodiment of the invention, through the alternating layering setting, allows the model to continuously "swish" between global information retrieval and local information modeling, rather than simply focusing on global information. This not only reduces computational complexity but also greatly improves inference efficiency, accelerates inference, and reduces GPU memory usage.

[0042] It should be noted that when using a long text processing model to process long texts, the output long text sorting result corresponds to the user's intent. That is, when inputting long text data into the model, the user's intent can also be input, such as understanding long text, text summarization, text summary, etc. For example, prompt text can be constructed based on the user's intent and long text data, and the prompt text can be input into the long text sorting model so that the model outputs the long text processing result corresponding to the user's intent under the prompt text.

[0043] The long text processing method provided by this invention obtains the long text processing result by inputting the long text data to be processed into a long text processing model. The long text processing model is constructed on the basis of a large language model using a hierarchical hybrid attention strategy. The hierarchical hybrid attention strategy replaces the attention mechanism in the first network layer of each network structure of the large language model with an attention mechanism with global information retrieval capability, and replaces the attention mechanism in other network layers of each network structure with an attention mechanism with local information modeling capability. Each network structure is obtained by equally dividing all network layers of the large language model according to the connection order between network layers. This hierarchical hybrid attention architecture can greatly improve the inference efficiency on long texts while ensuring that the performance of the model on downstream tasks is not reduced, and significantly reduce the consumption of computing resources and memory usage, approaching the modeling effect when only global attention is used, thus achieving a balance between inference efficiency, inference effect and computing resources.

[0044] Furthermore, it should be noted that the hierarchical hybrid attention architecture proposed in this embodiment of the invention has excellent scalability. Different attention mechanisms can be used to replace the global layer and the local layer. The number of global layers can also be adjusted by controlling the number of network layers in each network structure. That is, different global / local layer types and the number of global layers can be selected according to different hardware characteristics and the desired acceleration effect.

[0045] Based on the above embodiments, each network structure is obtained by equally dividing all network layers in a large language model according to the connection order between network layers. Each network structure contains at least one network layer. The first attention mechanism is the attention mechanism in the first network layer of each network structure, and the second attention mechanism is the attention mechanism in the other network layers of each network structure except for the first network layer.

[0046] Specifically, in this embodiment of the invention, the process of improving the structure of a large model using a hierarchical hybrid attention strategy can be as follows: First, the network layers of the large model are divided into multiple network structures. That is, all network layers of the large model are equally divided according to the connection order between network layers, so as to divide it into multiple network structures according to the connection order. Each network structure contains at least one network layer, and the number of network layers in each network structure is the same. For example, assuming that the large model has a total of 40 network layers, it can be divided into 40, 20, 10, 8, 5, 4, 2 or 1 network structures, and each network structure can contain 1, 2, 4, 5, 8, 10, 20 or 40 network layers. Next, the attention mechanism in the first network layer of each network structure, i.e., the first attention mechanism, can be replaced with an attention mechanism that has global information retrieval capabilities, i.e., the global layer. Here, it can also be understood as replacing the original global attention mechanism in the first network layer of each network structure with a global layer, and the global layer here can adopt a block sparse attention mechanism, a global attention mechanism, etc. When a global attention mechanism is used, since the result is the same before and after the replacement, this replacement operation can be regarded as the retention of the global attention mechanism. When a non-global attention mechanism is used, such as a block sparse attention mechanism, this replacement can speed up the inference speed of the model when processing long texts, thus achieving inference acceleration.

[0047] Accordingly, in addition to replacing the attention mechanism in the first layer of each network structure with a global layer, the hierarchical hybrid attention strategy also proposes replacing the attention mechanism in the other layers of each network structure (excluding the first layer), i.e., the second attention mechanism, with an attention mechanism capable of modeling local information, i.e., a local layer. This can also be understood as replacing the original global attention mechanism in the other layers of each network structure (excluding the first layer) with a local layer. The local layer can employ mechanisms such as sliding window attention, blocky sparse attention, or linear attention. Figure 5 This is an example diagram of the network structure after replacement provided by the present invention, such as... Figure 5 As shown, in a large model containing 40 network layers, when each network structure contains 4 network layers (a total of 10 network structures), and the global layer adopts a global attention mechanism after replacement, and the length of the input long text data is long enough (meaning the computational cost of local layers is ignored), the computational complexity of the entire model is O(n). , It is the sequence length. This refers to the number of network layers in the network structure. Therefore, improving the model structure based on a hierarchical hybrid attention strategy can significantly reduce its computational complexity, greatly improving inference speed while maintaining inference performance.

[0048] Based on the above embodiments, the global layer adopts a global attention mechanism, and the local layer adopts a sliding window attention mechanism.

[0049] Specifically, in the layered hybrid attention architecture, the global layer is mainly responsible for retrieving global information, while the local layer is responsible for extracting local information.

[0050] Figure 6 This is one of the example diagrams of the hierarchical hybrid attention architecture provided by the present invention, such as... Figure 6 As shown in the embodiments of the present invention, the global layer can adopt a global attention mechanism; the local layer can adopt a sliding window attention mechanism. Compared with the global attention mechanism, the computational complexity of this attention mechanism is significantly reduced (from O(n*n) to O(w*n), where n is the sequence length, i.e. the length of the input long text sequence), and it grows linearly with the sequence length rather than quadratically.

[0051] In this embodiment of the invention, a global attention mechanism is adopted in the global layer and a sliding window attention mechanism is adopted in the local layer. This alternating layering of the global attention mechanism and the sliding window attention mechanism can further accelerate the speech rate when processing long texts, while reducing the memory usage. It has the advantages of simple engineering optimization, acceleration of the entire inference process, and a significant reduction in memory usage.

[0052] Based on the above embodiments, the global layer adopts a global attention mechanism, and the local layer adopts a block sparse attention mechanism.

[0053] Specifically, in addition to using a sliding window attention mechanism as sparse attention, in this embodiment of the invention, block sparse attention can also be used to model local information. Figure 7 This is the second example diagram of the hierarchical hybrid attention architecture provided by the present invention, as shown below. Figure 7 As shown, the local layer uses a block-based sparse attention mechanism, while the global layer still uses a global attention mechanism.

[0054] Compared to the sliding window attention mechanism, the block sparse attention mechanism has a stronger global information retrieval capability, but its local information modeling capability is slightly inferior.

[0055] In this embodiment of the invention, the alternating layered setting of global attention mechanism and block sparse attention mechanism can significantly improve the speed of first character generation during reasoning; and compared with the alternating layered setting of global attention mechanism and sliding window attention mechanism, it has stronger information retrieval capability for long input text sequences and better reasoning effect.

[0056] Based on the above embodiments, the global layer adopts a block sparse attention mechanism, and the local layer adopts a sliding window attention mechanism.

[0057] Specifically, based on the alternating layered setup of the global attention mechanism and the sliding window attention mechanism, in order to achieve better reasoning acceleration, in this embodiment of the invention, a block-based sparse attention mechanism with global information retrieval capability can be used to replace the global attention mechanism. Figure 8 This is the third example diagram of the hierarchical hybrid attention architecture provided by the present invention, as shown below. Figure 8 As shown, the global layer uses a block-based sparse attention mechanism, while the local layer still uses a sliding window attention mechanism.

[0058] Compared to the alternating layered setup of global attention mechanism and sliding window attention mechanism, in this embodiment of the invention, the long context retrieval capability of the model will be slightly reduced, but the computational complexity of the entire model will increase linearly with the sequence length, which can greatly reduce the memory usage and improve the inference speed when processing long text.

[0059] In addition, compared with the pure block sparse attention scheme (the setting of block sparse attention mechanism and block sparse attention mechanism), the alternation strategy of block sparse attention mechanism and sliding window attention mechanism in the embodiments of the present invention can further accelerate the generation speed of the first character during inference and the generation of words during inference, and can save a lot of video memory usage.

[0060] Based on the above embodiments, the global layer is constructed using the following steps: The initial global layer is determined, which is an attention mechanism with global information retrieval capabilities; Remove the rotation position encoding from the initial global layer to obtain the global layer.

[0061] Specifically, while large models that currently use global attention mechanisms perform well within the training length, their performance will significantly decrease when the sequence length far exceeds the maximum length during training, even with techniques such as positional encoding interpolation, making it difficult to effectively handle extremely long text sequences.

[0062] Therefore, in the hierarchical hybrid attention architecture, in order to enable the global layer to process sequences of arbitrary length without being limited by the maximum positional encoding during training, the positional encoding can be removed in this embodiment of the invention, so that the global layer has an inherent potential for length extrapolation; the local layer is responsible for modeling local information, and its positional encoding is not modified.

[0063] In detail, here we can first determine the initial global layer, which is an attention mechanism with global information retrieval capabilities, such as a global attention mechanism or a block sparse attention mechanism; then, we can remove the rotation position encoding in the initial global layer to improve the model's performance when extrapolating long texts, thus obtaining the global layer.

[0064] Based on the above embodiments, the rotation position encoding in the initial global layer is removed to obtain the global layer, which includes: Remove the rotation position encoding from the initial global layer to obtain the target global layer; Add an attention scaling mechanism to the target global layer to obtain the global layer.

[0065] Specifically, the process of removing the rotation position encoding from the initial global layer to obtain the global layer can include: Although positional encoding has been removed from the global layer in the hierarchical hybrid attention architecture to enhance the extrapolation potential of the model, the global layer still faces the problem of overly flattened attention distribution when inferring sequences that far exceed the training length. That is, the attention probability distribution of the softmax output is very uniform, making it difficult for the model to focus on truly relevant tokens.

[0066] Based on this, in order to prevent the probability distribution calculated by softmax in the global layer from becoming excessively flat during long text processing, this embodiment of the invention introduces an additional position-aware attention scaling mechanism during inference to solve the problem of probability flattening in softmax output, thereby enabling the model to extrapolate sequences far exceeding the training length and having stronger extrapolation potential.

[0067] In detail, this could be to obtain the target global layer after removing the rotation position encoding in the initial global layer; then, an attention scaling mechanism can be added to the target global layer to obtain the global layer. Alternatively, it can be understood as introducing a position-related attention scaling mechanism to the attention mechanism with global information retrieval capabilities during the inference phase to obtain the final global layer.

[0068] In this embodiment of the invention, by introducing an attention scaling mechanism, the model can dynamically adjust the weight distribution of the input long text data, enabling the model to focus on key information and pay attention to truly relevant tokens. This improves the reasoning performance on long texts, allowing the model to extrapolate sequences far exceeding the training length, thus solving the current performance bottleneck in long sequence processing.

[0069] Based on the above embodiments, the attention scaling mechanism is used to scale the query vector at each position in long text data, and the scaling function of the query vector is determined based on the corresponding position.

[0070] Specifically, to enable the model to reason about sequences far exceeding the training length, this embodiment of the invention proposes a position-aware attention scaling mechanism during inference. That is, it scales attention based on the position index of the token in the input long text data. Apply the predefined scaling function to the query vector at the corresponding position Scaling is performed to address the issue of excessively flattened probability distributions in softmax calculations, enabling the model to extrapolate sequences far exceeding the training length.

[0071] In detail, the formula for calculating attention in the global layer is as follows, where , , They are three shapes The matrix, For sequence length, It is the dimension of the vector.

[0072] for In the middle position query vector It can be scaled to obtain , .in, It is about Scaling function, , at any position All are greater than or equal to 1. and These are two hyperparameters; the former can be set to 8192, and the latter to 0.1. This scaling function aims to address the problem of excessively flattened probability distributions in the global softmax output when the input sequence length far exceeds the training length. It works by amplifying the probability distributions of later positions in the sequence. This allows for a sharper distribution of attention, which in turn helps the model focus more effectively on information that is more relevant to the current location.

[0073] The long text processing apparatus provided by the present invention is described below. The long text processing apparatus described below can be referred to in correspondence with the long text processing method described above.

[0074] Figure 9 This is a schematic diagram of the structure of the long text processing device provided by the present invention, as shown below. Figure 9 As shown, the device includes: Determining unit 910 is used to determine the long text data to be processed; The processing unit 920 is used to input the long text data into the long text processing model and obtain the long text processing result output by the long text processing model. The long text processing model is built on the basis of a large language model and adopts a hierarchical hybrid attention strategy. The hierarchical hybrid attention strategy is to replace the first attention mechanism in each network structure of the large language model with a global layer and replace the second attention mechanism in each network structure with a local layer. The first attention mechanism and the second attention mechanism are respectively the first attention mechanism and other attention mechanisms in each of the network structures; The global layer is an attention mechanism with global information retrieval capabilities, and the local layer is an attention mechanism with local information modeling capabilities.

[0075] The long text processing device provided by this invention obtains the long text processing result by inputting the long text data to be processed into a long text processing model. The long text processing model is constructed on the basis of a large language model using a hierarchical hybrid attention strategy. The hierarchical hybrid attention strategy replaces the first attention mechanism in each network structure of the large language model with an attention mechanism with global information retrieval capability, and replaces other attention mechanisms with attention mechanisms with local information modeling capability. This hierarchical hybrid attention architecture can greatly improve the inference efficiency on long texts while ensuring that the performance of the model on downstream tasks is not reduced, and significantly reduce the consumption of computing resources and memory usage, approaching the modeling effect when only global attention is used, thus achieving a balance between inference efficiency, inference effect and computing resources.

[0076] Based on the above embodiments, each network structure is obtained by equally dividing all network layers in the large language model according to the connection order between network layers. Each network structure contains at least one network layer. The first attention mechanism is the attention mechanism in the first network layer of each network structure, and the second attention mechanism is the attention mechanism in the other network layers of each network structure except for the first network layer.

[0077] Based on the above embodiments, the global layer adopts a global attention mechanism, and the local layer adopts a sliding window attention mechanism.

[0078] Based on the above embodiments, the global layer adopts a global attention mechanism, and the local layer adopts a block sparse attention mechanism.

[0079] Based on the above embodiments, the global layer adopts a block sparse attention mechanism, and the local layer adopts a sliding window attention mechanism.

[0080] Based on the above embodiments, the device further includes a construction unit for: An initial global layer is determined, wherein the initial global layer is an attention mechanism with global information retrieval capabilities; Remove the rotation position encoding from the initial global layer to obtain the global layer.

[0081] Based on the above embodiments, the construction unit is used for: Remove the rotation position encoding from the initial global layer to obtain the target global layer; An attention scaling mechanism is added to the target global layer to obtain the global layer.

[0082] Based on the above embodiments, the attention scaling mechanism is used to scale the query vector at each position in the long text data, and the scaling function of the query vector is determined based on the corresponding position.

[0083] Figure 10 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 10 As shown, the electronic device may include: a processor 1010, a communications interface 1020, a memory 1030, and a communication bus 1040, wherein the processor 1010, the communications interface 1020, and the memory 1030 communicate with each other through the communication bus 1040. The processor 1010 can call logical instructions in the memory 1030 to execute a long text processing method. This method includes: determining long text data to be processed; inputting the long text data into a long text processing model to obtain the long text processing result output by the long text processing model; the long text processing model is constructed based on a large language model using a hierarchical hybrid attention strategy; the hierarchical hybrid attention strategy replaces the first attention mechanism in each network structure of the large language model with a global layer, and replaces the second attention mechanism in each network structure with a local layer; the first attention mechanism and the second attention mechanism are respectively the first attention mechanism and other attention mechanisms in each network structure; the global layer is an attention mechanism with global information retrieval capability, and the local layer is an attention mechanism with local information modeling capability.

[0084] Furthermore, the logical instructions in the aforementioned memory 1030 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0085] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the long text processing method provided by the above methods, the method comprising: determining long text data to be processed; inputting the long text data into a long text processing model to obtain a long text processing result output by the long text processing model; the long text processing model is constructed on the basis of a large language model using a hierarchical hybrid attention strategy; the hierarchical hybrid attention strategy is to replace the first attention mechanism in each network structure of the large language model with a global layer, and replace the second attention mechanism in each network structure with a local layer; the first attention mechanism and the second attention mechanism are respectively the first attention mechanism and other attention mechanisms in each network structure; the global layer is an attention mechanism with global information retrieval capability, and the local layer is an attention mechanism with local information modeling capability.

[0086] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the long text processing method provided by the above methods. The method includes: determining long text data to be processed; inputting the long text data into a long text processing model to obtain a long text processing result output by the long text processing model; the long text processing model is constructed based on a large language model using a hierarchical hybrid attention strategy; the hierarchical hybrid attention strategy involves replacing the first attention mechanism in each network structure of the large language model with a global layer, and replacing the second attention mechanism in each network structure with a local layer; the first attention mechanism and the second attention mechanism are respectively the first attention mechanism and other attention mechanisms in each network structure; the global layer is an attention mechanism with global information retrieval capability, and the local layer is an attention mechanism with local information modeling capability.

[0087] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0088] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for processing long text, characterized in that, include: Identify the long text data to be processed; The long text data is input into the long text processing model to obtain the long text processing result output by the long text processing model. The long text processing model is built on the basis of a large language model and adopts a hierarchical hybrid attention strategy. The hierarchical hybrid attention strategy is to replace the first attention mechanism in each network structure of the large language model with a global layer and replace the second attention mechanism in each network structure with a local layer. The first attention mechanism and the second attention mechanism are respectively the first attention mechanism and other attention mechanisms in each of the network structures; The global layer is an attention mechanism with global information retrieval capabilities, and the local layer is an attention mechanism with local information modeling capabilities.

2. The long text processing method according to claim 1, characterized in that, Each network structure is obtained by equally dividing all network layers in the large language model according to the connection order between network layers. Each network structure contains at least one network layer. The first attention mechanism is the attention mechanism in the first network layer of each network structure, and the second attention mechanism is the attention mechanism in the other network layers of each network structure except for the first network layer.

3. The long text processing method according to claim 1, characterized in that, The global layer employs a global attention mechanism, while the local layer employs a sliding window attention mechanism.

4. The long text processing method according to claim 1, characterized in that, The global layer employs a global attention mechanism, while the local layer employs a block-based sparse attention mechanism.

5. The long text processing method according to claim 1, characterized in that, The global layer employs a block-based sparse attention mechanism, while the local layer employs a sliding window attention mechanism.

6. The long text processing method according to any one of claims 1 to 5, characterized in that, The global layer is constructed based on the following steps: An initial global layer is determined, wherein the initial global layer is an attention mechanism with global information retrieval capabilities; Remove the rotation position encoding from the initial global layer to obtain the global layer.

7. The long text processing method according to claim 6, characterized in that, The step of removing the rotation position encoding from the initial global layer to obtain the global layer includes: Remove the rotation position encoding from the initial global layer to obtain the target global layer; An attention scaling mechanism is added to the target global layer to obtain the global layer.

8. The long text processing method according to claim 7, characterized in that, The attention scaling mechanism is used to scale the query vector at each position in the long text data, and the scaling function of the query vector is determined based on the corresponding position.

9. A long text processing device, characterized in that, include: The determination unit is used to determine the long text data to be processed. The processing unit is used to input the long text data into the long text processing model and obtain the long text processing result output by the long text processing model. The long text processing model is built on the basis of a large language model and adopts a hierarchical hybrid attention strategy. The hierarchical hybrid attention strategy is to replace the first attention mechanism in each network structure of the large language model with a global layer and replace the second attention mechanism in each network structure with a local layer. The first attention mechanism and the second attention mechanism are respectively the first attention mechanism and other attention mechanisms in each of the network structures; The global layer is an attention mechanism with global information retrieval capabilities, and the local layer is an attention mechanism with local information modeling capabilities.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the long text processing method as described in any one of claims 1 to 8.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the long text processing method as described in any one of claims 1 to 8.

Images