Large language model training method and device based on range position coding, and medium

Abstract

The application discloses a large language model training method and device based on range position coding and a medium, and relates to the technical field of artificial intelligence and natural language processing; the method comprises the following steps: constructing a base model network architecture, removing explicit position coding; configuring a range attention mechanism, assigning an exponentially distributed review range to different attention heads in a multi-head self-attention layer; constructing a sparse attention mask matrix according to the review range, and forcing the model to only pay attention to historical information within the review range; through the structured sparse design, the sequential perception of the full sequence is realized by using the recursive long-short context comparison mechanism, the model is endowed with the original length extrapolation capability and the high-precision long text retrieval capability while the calculation amount is greatly reduced, and the calculation redundancy and numerical instability problems of the traditional explicit position coding in the ultra-long context scene are solved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and natural language processing technology, specifically to a method, device, and medium for training a large language model based on range position encoding. Background Technology

[0002] With the development of deep learning technology, Large Language Models (LLMs) based on the Transformer architecture have achieved remarkable results in many natural language processing tasks. The core of the Transformer architecture lies in the multi-head self-attention mechanism, which allows the model to capture dependencies between different positions in a sequence. However, the standard multi-head self-attention mechanism faces the problem of high computational efficiency and complexity when processing long text sequences, and it does not have position awareness capabilities, requiring the introduction of additional positional encoding.

[0003] Existing positional encoding methods mainly include Absolute Position Encoding (APE), Relative Position Encoding (RPE), and the currently mainstream Rotational Position Encoding (RoPE). Although these explicit positional encoding methods solve the sequence awareness problem to some extent, they still have the following limitations:

[0004] 1. High computational cost and numerical instability: Explicit positional encoding relies on arithmetic operations to inject positional information into the attention matrix. However, research shows that explicit positional encoding introduces additional computational overhead when processing sequences exceeding the pre-training length. More importantly, under low-precision computations such as BFloat16, methods like RoPE may exhibit numerical instability, causing the model's attention mechanism to fail over extremely long distances, leading to long-distance forgetting or "illusion" phenomena in text generation.

[0005] 2. Limited extrapolation capability: Models trained based on a training window (e.g., 4k) often have difficulty generalizing directly to longer sequences (e.g., 32k or longer). To achieve extrapolation of long texts, complex post-processing techniques such as interpolation (PI) or NTK-aware are often required, which can easily lead to a decrease in performance for short texts.

[0006] 3. The trade-off between efficiency and accuracy: To alleviate the computational pressure brought by MHA, various sparse attention schemes have been proposed in existing technologies to reduce computational load. However, existing sparse attention techniques often employ fixed or heuristic sparse patterns, using a uniform window size for all attention heads. This rigid pattern easily undermines the modeling ability for long-distance dependencies, resulting in poor model performance in downstream tasks requiring high-precision context retrieval and complex logical reasoning, making it difficult to balance computational efficiency and retrieval accuracy.

[0007] Furthermore, while existing technologies have proposed various sparse attention mechanisms (such as Longformer and BigBird) to reduce computational complexity, these schemes typically employ fixed-size sliding windows or random sparse patterns, and most still rely on explicit positional encoding (such as RoPE or absolute positional encoding) to maintain sequence order. This reliance leads to insufficient extrapolation capabilities when the model processes sequences exceeding the pre-training length, and the fixed window struggles to flexibly adapt to contextual dependencies of different granularities. Meanwhile, existing research on No-Positional-Encoding (NoPE) studies shows that implicit counting using causal masks alone has weak resolution in shallow networks, failing to match the accuracy of explicit encoding. Therefore, how to achieve efficient long-text extrapolation and high-precision retrieval through structured sparse attention design without relying on explicit positional encoding remains a pressing technical challenge in this field.

[0008] Therefore, there is an urgent need for a location encoding method and model training method that can efficiently handle extremely long contexts, significantly reduce computational load, and maintain high-precision location awareness and extrapolation capabilities. Summary of the Invention

[0009] This invention addresses the shortcomings of existing technologies by providing a training method, device, and medium for large language models based on scoped position encoding. This method abandons traditional explicit arithmetic position encoding and innovatively proposes that sparse context itself can be used for position encoding. This sparse attention mechanism endows the model with implicit global order awareness by constructing a structured sparse multi-head attention mechanism with an exponentially distributed lookback scope. This invention points out that a sparse attention mechanism with this specific contextual acceptance scope is an implicit scoped position encoding (ScoopE), which can significantly improve the model's extrapolation ability and retrieval accuracy in long texts while greatly reducing computational costs.

[0010] To achieve the above objectives, the present invention adopts the following technical solution: Firstly, a method for training large language models based on range-positional encoding is provided, including the following steps: Step S1: Remove the explicit positional encoding module from the Transformer architecture to obtain the model after removing the explicit positional encoding; the explicit positional encoding module includes rotational positional encoding (RoPE), linear offset positional encoding (ALiBi), and learnable positional embedding vectors. Step S2: Assign a lookback range to each attention head h in the multi-head self-attention layer of the model after removing explicit position encoding. This will give you the model with the configured playback range; Step S3: Construct a sparse attention mask matrix based on the lookback range to obtain a model with a configured sparse attention mask matrix; Step S4: Perform forward propagation calculation using the model of the sparse attention mask matrix, where the calculation of attention scores covered by the mask is omitted, and obtain the hidden state of the last layer output. Step S5: Calculate the cross-entropy loss of the autoregressive task using the hidden state output of the last layer, and update the model parameters using the optimizer to finally obtain the trained model.

[0011] Further, in step S2, the playback range The replay range of each attention head (h) is distributed exponentially across different attention heads. For growth factors The exponential distribution with base h and exponent h is the rounded distribution, where h is the index of the attention head, ranging from 1 to H, and H is the number of attention heads in each layer of the model.

[0012] Furthermore, the growth factor The growth factor is determined by the number of attention heads H and the maximum context length T. The range is greater than 1 and less than or equal to 2, which makes the maximum rewind range It equals the maximum context length that the model can handle.

[0013] Further, in step S3, the sparse attention mask matrix Specifically: for each current token on the diagonal, only the past tokens are retained. One token participates in the attention calculation, while the remaining positions are masked and set to negative infinity; wherein, the token includes the current token.

[0014] Further, in step S4, the forward propagation calculation to obtain the hidden state of the last layer output specifically involves: using the model with the configured sparse attention mask matrix output in step S3 to perform forward propagation calculation to obtain the hidden state of the last layer output of the model. The hidden state of the last layer output integrates semantic information and hidden position information inferred through the topological structure.

[0015] Further, step S5 specifically involves: inputting the hidden state output from the last layer into a linear classification layer, and obtaining the predicted probability distribution of the next label in the vocabulary dimension via the Softmax function; constructing a cross-entropy loss function for the autoregressive task based on the predicted probability distribution of the next label; calculating the gradient of each learnable parameter in the model by minimizing the cross-entropy loss function and performing backpropagation through the chain rule; updating the model parameters using an optimizer and explicitly introducing a weight decay term; using a learning rate scheduling strategy during training; and finally obtaining the trained model through closed-loop iterative optimization.

[0016] Furthermore, the method further includes: the trained model comprises stacked Transformer blocks with L network layers; the number of network layers L is greater than or equal to the logarithm with base 2 and T as the argument.

[0017] In a second aspect, the present invention provides an electronic device, including a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-described large language model training method based on range position encoding.

[0018] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the large language model training method based on range position encoding.

[0019] The present invention also provides a computer program product, including a computer program / instruction that, when executed by a processor, implements the large language model training method based on range position encoding.

[0020] The present invention also provides a computer device and a computer-readable storage medium.

[0021] Compared with the prior art, the beneficial effects of the present invention are mainly reflected in the following aspects: 1. Structured position awareness: It abandons the traditional explicit arithmetic position encoding and introduces a sparse attention mask matrix with an exponential distribution in a multi-layer Transformer architecture. It uses the sparse attention mechanism itself as a position encoder, thus avoiding numerical stability problems. 2. High-efficiency computation and memory optimization: By limiting the attention range of different attention heads, the computational load is significantly reduced, theoretically reaching [amount missing]. The computational workload can be reduced by up to 8 times in mainstream configurations such as 32 heads, and the KV cache usage during inference is also reduced, which greatly improves the efficiency and capability of long text processing. 3. Strong native extrapolation capability: The model has excellent native extrapolation capability and can adapt to sequences that are several times longer than the training length (e.g., 4 times) without additional training; this native extrapolation capability ensures that the model can still maintain the consistency and coherence of logical representation when crossing different length thresholds. 4. High-precision long text retrieval: In tasks such as Needle-in-a-Haystack, it can maintain extremely high retrieval accuracy even in ultra-long contexts of 128k, which is significantly better than the traditional RoPE model. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of the training method for a large language model based on range positional encoding. Figure 2 This is a mask visualization diagram of the exponentially distributed look-back range in an embodiment of the present invention, wherein (a) is a look-back range diagram of the first head, (b) is a look-back range diagram of the second head, and (c) is a look-back range diagram of the third head; Figure 3 This is a schematic diagram of the principle of implicit position awareness implemented by the recursive long and short context comparison mechanism. Among them, (a) and (b) are schematic diagrams of basic examples of implicit position awareness implemented by the long and short context comparison mechanism, and (c) is a visual summary of the inductive steps. Figure 4 This is a structural block diagram of a large language model training device; Figure 5 This is a comparison of the model training loss and the loss convergence curve of the Llama3-8B-RoPE baseline model during the pre-training stage in this embodiment of the invention. Figure 6 This is a schematic diagram comparing the perplexity extrapolation performance of embodiments and comparative examples of the present invention under different context lengths; Figure 7 These are schematic diagrams comparing inference latency, throughput, and speedup ratio under different context lengths in embodiments and comparative examples of the present invention; wherein, (a) is a schematic diagram comparing inference latency, and (b) is a schematic diagram comparing throughput and speedup ratio. Figure 8These are heatmaps of retrieval accuracy in the "needle in a haystack" task of the embodiments and comparative examples of the present invention, wherein (a) is the heatmap of retrieval accuracy of RoPE (base), (b) is the heatmap of retrieval accuracy of ScoPE (base), (c) is the heatmap of retrieval accuracy of RoPE (FT-128k), and (d) is the heatmap of retrieval accuracy of ScoPE (FT-128k). Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.

[0026] This invention proposes that sparse context itself can be used for position encoding. This sparse attention mechanism endows the model with implicit global order awareness by constructing a structured sparse multi-head attention mechanism with an exponentially distributed lookback scope. This invention points out that a sparse attention mechanism with this specific context acceptance scope is an implicit scoped position encoding (ScoopE).

[0027] See Figure 1 This invention provides a method for training a large language model based on range position encoding, specifically including the following steps: Step S1, Model Architecture Initialization: The basic model adopts a Transformer-based decoder-only architecture, which contains L stacked Transformer blocks. The number of network layers L is configured to meet preset conditions so that the sequence order resolution capability of the model covers the maximum supported sequence length as the number of layers increases. The decoder-only architecture includes a variant of the LLaMA architecture.

[0028] The core of this step is to completely remove all traditional explicit positional encoding modules from the Decoder-only architecture. Specifically, these traditional explicit positional encoding modules are used to provide positional information, including Rotational Positional Encoding (RoPE), Linear Bias Positional Encoding (ALiBi), and learnable positional embedding vectors. Removing the RoPE from the self-attention mechanism ensures that the calculation of the attention score no longer depends on a preset rotation matrix, and removing the learnable positional embedding vectors from the embedding layer ensures that the input token vector sequence does not carry any absolute or relative positional coordinate information before entering the Transformer block.

[0029] Through the above removal operation, we obtain a model after removing explicit positional encoding. In the initial state, this model has very fragile semantic association modeling ability and sequence awareness ability. Its individual attention module can be regarded as a bag-of-words model without positional information.

[0030] Step S2: Configure the lookback range for the exponential distribution: To reconstruct the model's sequence awareness capability without explicit coordinates, this invention configures an exponentially distributed lookback range for each attention head in each multi-head self-attention layer of the model after removing explicit encoding. This allows us to obtain a model that configures the playback range.

[0031] Assuming each layer has H attention heads, the system provides attention to each head. Assign a fixed playback range h is the index of the attention head. The replay range... It follows a geometric progression among different attention heads:

[0032] Wherein, 𝛾 is the growth factor, which is determined by the number of attention heads H and the maximum context length T. That is, a value close to but not exceeding 2 (e.g., 𝛾=2 or 𝛾=1.5) that maximizes the viewing range. It equals the maximum context length that the model can handle.

[0033] Regarding the specific architecture configuration, this invention stipulates that the L and H of the model are configured according to the model capacity, and the number of layers L is greater than or equal to Log2(T), and guarantees that... The constraint that the model's sequence order discrimination capability can fully cover the preset maximum context length through hierarchical stacking and multi-head combination ensures the global completeness of implicit position awareness; the parameter configurations of current mainstream large language models can meet this requirement.

[0034] Through the above configuration strategy, the playback range This ensures that the model possesses diverse granular perspectives even at a shallow level, with a short-sighted range. The smaller view focuses on capturing local grammar and phrase structure; the longer view focuses on the larger view, capturing paragraph and passage-level dependencies, and the longest view covers the maximum length of the current sequence.

[0035] Step S3: Construct a sparse attention mask matrix: For the replay range of each attention head determined in step S2 Construct a for each head sparse attention mask matrix See also Figure 2 The vertical axis q represents the query, i represents the query number, and the horizontal axis k represents the key, j represents the key number. A standard sparse attention mask matrix is ​​a matrix with a lower triangular focus area (ij>=0). After configuring the range mask (ij<= Each attention head focuses on a different scope. For example, when the context length is 8 and the number of attention heads is 3, the structure is constructed through three mask matrices. , , , The replay ranges are 2, 4, and 8, respectively. The sparse attention mask matrix is ​​implemented through block mask optimization using FlexAttention or FlashAttention, forcing the model to perceive relative positions through topological structures rather than explicit arithmetic values.

[0036] The sparse attention mask matrix Specifically: for each current token on the diagonal, only the past tokens are retained. One token (including the current token) participates in the attention calculation, and the remaining positions are set to negative infinity; let the sparse attention mask matrix... A sparse attention mask matrix with row number i and column number j. The mathematical expression is:

[0037] This step outputs a model configured with a sparse attention mask matrix. This step transforms the model into a hierarchical sequence processor: the lower-level or short-scope head is responsible for capturing local dependencies, while the higher-level or long-scope head captures global dependencies.

[0038] See Figure 3This invention utilizes a recursive long-short context comparison principle to achieve full sequence order awareness, which is the core of this invention's implicit positional encoding. Although the head with the largest viewing range in a single layer has a full-length field of view (able to see all historical tokens), due to the lack of explicit positional encoding, a single layer cannot generate a unique positional code to distinguish all positions (i.e., weak sequence order discrimination capability). ScoPE multi-layer stacking utilizes a recursive long-short context comparison mechanism to decompose the long-distance position discrimination problem into short-distance relative position judgments, thereby expanding the sequence order discrimination capability rather than expanding the field of view.

[0039] The recursive long-short context comparison principle is as follows: Utilizing a multi-layered stacked Transformer structure, the problem of order recognition for long sequences is transformed into a problem where the model can generate different embedding representations for sequences of different orders; for example... Figure 3 (a) Figure 3 As shown in (b), since each token has a different embedding representation, the trained model can distinguish the sequence order of length 1. With a base context length of 2, a length 1 attention head can help a length 2 attention head perceive the input order of length 2, outputting different embedding representations. Longer sequence orders can be achieved through stacking multiple layers of networks. Specifically, as... Figure 3 As shown in (c), if the upper layer network (i.e., the first layer) If a layer can distinguish the sequence order of length c and generate unique embedding features, then the current layer (i.e., the th layer) is capable of doing so. Within a layer, there must exist a short rewind range attention head with a rewind range of c, and a rewind range that is c's long rewind range draws attention to the head, because Since c ≦ 2c, it can be viewed as a problem of distinguishing the order of two different embedded representations, which is transformed into the case of a length of 2. Thus, by combining the information interaction between the short-view scope (focusing on the local) and the long-view scope (focusing on the global), the embedded representation generated in layer l can distinguish lengths of... The sequence order of c; by stacking multiple layers of network, the model's sequence recognition ability increases exponentially with the number of layers, ultimately... This is equal to the maximum context processing length of the configured model. Therefore, the model can recursively decompose the problem of order discrimination of long sequences into the problem of relative order judgment of short sequences, thereby achieving full sequence order perception without explicit position embedding. This sparse attention mechanism with a specific context acceptance range is an implicit scoped position encoding (ScoopE).

[0040] Step S4: Calculate the hidden state of the last layer output using forward propagation. The model with the sparse attention mask matrix output in step S3 is used for forward propagation calculation. By stacking multiple network structures, the model achieves implicit global order perception by using combinations of different lookback ranges, and obtains the hidden state output by the last layer of the model. This state has deeply integrated semantic information and hidden position information inferred from the topology.

[0041] Step S5: Calculate the cross-entropy loss for the autoregressive task and update the model parameters using the optimizer to finally obtain the trained model. The hidden state output from the last layer of the model obtained in step S4 is input into the linear classification layer and processed by the Softmax function to obtain the predicted probability distribution of the next token in the vocabulary dimension. ; Predicted probability distribution based on the true next label Constructing the cross-entropy loss function for autoregressive tasks This loss function measures the difference between the model's predicted sequence and the true context sequence; its mathematical expression is:

[0042] Where N is the number of tags, n is an iterator to N, and V is the vocabulary size. It predicts the probability distribution; By minimizing the cross-entropy loss function, backpropagation is performed using the chain rule to calculate the gradients of each learnable parameter in the model. An adaptive moment estimator (IME) with weight decay is used to update the model parameters. This IEM includes AdamW and explicitly introduces a weight decay term to enhance the model's generalization ability and prevent overfitting. During training, a cosine annealing learning rate scheduling strategy with a warm-up period is used. This strategy dynamically controls the optimizer's base learning rate to ensure the numerical stability and convergence of the large language model under long-sequence training. Finally, the trained model is obtained through closed-loop iterative optimization.

[0043] This invention provides a computer device.

[0044] like Figure 4 As shown, this application provides an electronic device including a memory 101 for storing one or more programs and a processor 102. When the one or more programs are executed by the processor 102, they implement the method as described in any of the first aspects above.

[0045] The system also includes a communication interface 103. The memory 101, processor 102, and communication interface 103 are electrically connected directly or indirectly to each other to enable data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines. The memory 101 can be used to store software programs and modules, and the processor 102 executes various functional applications and data processing by executing the software programs and modules stored in the memory 101. The communication interface 103 can be used for signaling or data communication with other node devices.

[0046] The memory 101 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0047] The processor 102 can be an integrated circuit chip with signal processing capabilities. The processor 102 can be a general-purpose processor 102, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0048] In the embodiments provided in this application, it should be understood that the disclosed methods and systems can also be implemented in other ways. The method and system embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0049] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0050] On the other hand, embodiments of this application provide a computer-readable storage medium storing a computer program thereon. When executed by processor 102, the computer program implements the methods described in any of the first aspects above. If the functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0051] Example 1: Multi-stage progressive training strategy It should be noted that the three-stage progressive training strategy described in this embodiment is only a preferred training implementation method, designed to help the model better adapt to long context distributions, and is not a necessary step in the technical solution of this application. Those skilled in the art will understand that model training based on the aforementioned method can also employ single-stage training, multi-stage mixed training of different lengths, or other conventional training scheduling strategies, as long as the core features of removing explicit positional encoding and configuring exponential distribution lookback range are met, they all fall within the protection scope of this invention.

[0052] To adapt the model to long documents, this embodiment employs a three-stage training method: 1. Pre-training phase: Training is performed on a general corpus, with the maximum context window set to 4k.

[0053] 2. Long Context Fine-tuning Phase 1: Expand the maximum context window to 32k, mix in half of the long texts (document length greater than 16k) into the general corpus, and continue training for 50 to 100 steps to adapt the model to longer context dependencies.

[0054] 3. Long Context Fine-tuning Stage Two: Expand the maximum context window to 128k, mix half of the long texts (document length greater than 32k) into the general corpus, and continue training for 50 to 100 steps to adapt the model to the ultra-long context dependencies.

[0055] Example 2: Computational Complexity Analysis The computational complexity of traditional attention is .

[0056] The computational complexity of the method of this invention is proportional to the sum of the scopes of all heads, when and At that time, the computational workload is approximately:

[0057] Then the reduction ratio It can be approximated as:

[0058] For example, when φ ≈ 2, the model reaches the theoretical longest context, at which point the computational cost is 4 / H of full attention. Typically, H is configured with 32 or 64, achieving an 8-fold or 16-fold reduction in computational cost. In practice, when H = 32 and T = 4096, φ ≈ 1.3, achieving approximately a 3.7-fold reduction in FLOPs, significantly improving computational efficiency and enabling highly efficient long context processing.

[0059] Example 3: To verify the advantages of the large language model training method based on range position encoding proposed in this invention during the model training stage, a comparative experiment was constructed in this embodiment.

[0060] Based on the LLaMA-3-8B model architecture, this invention trains a large language model using standard Rotation Position Encoding (RoPE) with RoPE removed and trained solely on the described ScoPE using the Torchtitan training framework on an NVIDIA A100 computing cluster, and optimizes the operator using FlexAttention. The LLaMA-3-8B-RoPE large language model using standard Rotation Position Encoding (RoPE) serves as a comparison; the LLaMA-3-8B-Scoop model, with RoPE removed and trained solely on the described ScoPE, serves as an example. In Example 1, the pre-trained models are ScoPE(base) and RoPE(base). After fine-tuning in the first stage of long context fine-tuning, the models are ScoPE(FT-32k) and RoPE(FT-32k). After fine-tuning in the second stage of long context fine-tuning, the models are ScoPE(FT-128k) and RoPE(FT-128k).

[0061] 1. Comparison and verification of model training convergence and loss: During the pre-training phase (maximum context window set to 4k), the training loss curves of the two models as the number of training steps increases are recorded as follows: Figure 5 As shown, the training loss of the model in this embodiment (orange curve) is consistently lower than that of the comparative RoPE model (blue curve) throughout the pre-training phase. This indicates that the structured sparsity mechanism proposed in this invention not only reduces the amount of computation but also serves as an effective inductive bias, optimizing the model's loss surface and resulting in faster model convergence and higher training efficiency.

[0062] 2. Verification of long text perplexity and native extrapolation capabilities: This embodiment verifies the modeling ability of the model when directly processing text exceeding the training length without fine-tuning.

[0063] Perplexity level (PPL) is used as the main indicator to evaluate the ability of remote modeling and extrapolation. The lower the PPL, the more accurate the model's prediction of long texts.

[0064] One hundred long documents (ranging from 64k to 128k tokens) from the publicly available Proof-Pile and Gov-Report datasets were selected as the test set to evaluate the model. To measure the performance stability at different lengths, the perplexity of the last 256 tokens under different context lengths was compared. The experimental results are as follows: Figure 6 As shown.

[0065] Native extrapolation: On a model trained with only 4k input length, the comparative RoPE (Base) model experiences a rapid increase in perplexity (PPL) after the input length exceeds 4k, resulting in model collapse. In contrast, the model of this invention, ScoPE (Base), maintains the effective context up to 64k (16 times the training length) without updating any parameters, while keeping the PPL at a low level. This demonstrates that the model trained by this invention has significantly better native extrapolation capabilities than RoPE, adapting to long sequences through simple scope boundary expansion. Furthermore, for fine-tuning at 128k, both models, RoPE (FT-128k) and ScoPE (FT-128k), converge to a considerably low perplexity, indicating that the model of this invention retains the ability to learn from long context data while providing superior initialization properties.

[0066] The invention also provides schematic diagrams comparing inference delays of embodiments and comparative examples under different sequence lengths, such as... Figure 7 As shown in the figure. The comparison chart of pre-filling stage delay times is shown below. Figure 7 As shown in (a) of the diagram, as the length of the input sequence increases, for example from 32k to 256k, the comparative LLaMA-3-8B-RoPE suffers from extremely long user latency due to the need to compute a fully causal attention matrix in its standard self-attention mechanism. In contrast, the LLaMA-3-8B-ScoPE embodiment of the present invention, by employing sparse range attention and limiting computation to the active lookback range, significantly slows down the latency growth. For example, with a context length of 256k, the embodiment of the present invention reduces the absolute latency by approximately 48.5 seconds compared to the comparative example. The system throughput comparison is shown in the figure below. Figure 7 As shown in (b), under the same hardware resource conditions, the embodiment of the present invention significantly outperforms the comparative embodiment in system throughput due to optimization of KV Cache (key-value cache) usage and a substantial reduction in floating-point operations (FLOPs). Especially in scenarios with extremely long sequences, the comparative embodiment is prone to a sharp drop in throughput due to memory bottlenecks, while the present invention can still maintain high and stable concurrent processing capabilities. The speedup is in... Figure 7 (a) and Figure 7 As mentioned in (b), the speedup ratio (performance time ratio) of the embodiments of the present invention compared to the comparative example shows a continuously increasing trend as the sequence length increases. For example, with a context length of 256k, the embodiments of the present invention achieve a speedup ratio of approximately 1.9 times; while with a maximum context length of 256k, the embodiments of the present invention significantly reduce the absolute waiting time by approximately 48.5 seconds compared to the comparative example, further improving the speedup ratio to 2.3, fully demonstrating the computational advantage of this sparse structure under ultra-long text.

[0067] The experimental data above fully demonstrates that this invention not only reduces complexity in theory, but also successfully breaks through the efficiency bottleneck of ultra-long context reasoning in practical engineering applications, providing an effective technical path for low-cost, low-latency deployment of large-scale language models.

[0068] 3. Verification using extremely long contextual searches: This embodiment employs a "needle in a haystack" task to test the model's ability to accurately retrieve information within an extremely long context.

[0069] A specific "needle" (i.e., a random passkey) was inserted into 128k length of distracting text. The model was required to accurately identify the passkey after reading the entire text. A heatmap of retrieval accuracy was plotted with the context length on the X-axis and the insertion depth of the "needle" on the Y-axis. The experimental results are shown below. Figure 8 As shown: On the un-fine-tuned 4K model, such as Figure 8 As shown in (b), the embodiment of the present invention, ScoPE (base), can maintain high retrieval accuracy to a length of approximately 16k; while the comparative example, RoPE (base) Figure 8 In (a) of the model, accuracy begins to decline after 6k characters; after fine-tuning with 128k long text, the comparative RoPE (FT-128k) shows attention decay at the document depth level, with an average accuracy of 86%. Figure 8 In contrast, the model ScoPE(FT-128k) of this embodiment of the invention (c) Figure 8 The entire image (d) remains highlighted, with an average accuracy of 92%, and effectively solves the "dropped needle" problem at the end of long sequences (128k).

[0070] The model trained by this invention can accurately locate and retrieve key information by utilizing its global sequence awareness capability, with a retrieval accuracy exceeding 90%, and the accuracy does not decrease as the key information moves later in the document, proving the effectiveness of this method in understanding ultra-long contexts; through the hierarchical scope mechanism, it maintains a clearer focus of attention in ultra-long sequences, and the retrieval fidelity is significantly better than traditional methods.

[0071] 4. Validation of downstream tasks in general natural language understanding To demonstrate that the sparsity design of this invention does not impair the general intelligence of the model, this embodiment selects mainstream datasets such as MMLU (5-shot), ARC-Challenge (25-shot), HellaSwag (10-shot), and BoolQ for testing on standard NLU benchmarks.

[0072] The experimental results are shown in Table 1. Under the same parameter scale, in commonsense reasoning tasks such as ARC-C, HellaSwag, and PIQA, the scores of the embodiments of the present invention are better than those of the comparative example RoPE. In logical reasoning tasks, the performance of the embodiments of the present invention is on par with that of the comparative example. From the 4k pre-training stage to the 32k fine-tuning stage, the scores of the model of the present invention have improved on all tasks, and no catastrophic forgetting has occurred.

[0073] Table 1. Performance score comparison of the embodiments of the present invention and comparative examples on a general natural language understanding benchmark task.

[0074] Although this invention limits the scope of a single attention head, it achieves global perception through multi-layer stacking, thus not sacrificing general language understanding and generation capabilities, and performs better on some semantic abstraction tasks.

[0075] In summary, this invention constructs a highly efficient and accurate large language model training scheme through an innovative positional encoding paradigm, effectively solving the technical pain points of difficult long document processing, high computing power cost, and weak extrapolation ability.

Claims

1. A method for training a large language model based on range positional encoding, characterized in that, Includes the following steps: Step S1: Remove the explicit positional encoding module from the Transformer architecture to obtain the model after removing the explicit positional encoding; the explicit positional encoding module includes rotational positional encoding (RoPE), linear offset positional encoding (ALiBi), and learnable positional embedding vectors. Step S2: Assign a lookback range to each attention head h in the multi-head self-attention layer of the model after removing explicit position encoding. This will give you the model with the configured playback range; Step S3: Construct a sparse attention mask matrix based on the lookback range to obtain a model with a configured sparse attention mask matrix; Step S4: Perform forward propagation calculation using the model of the sparse attention mask matrix, where the calculation of attention scores covered by the mask is omitted, and obtain the hidden state of the last layer output. Step S5: Calculate the cross-entropy loss of the autoregressive task using the hidden state output of the last layer, and update the model parameters using the optimizer to finally obtain the trained model.

2. The method according to claim 1, characterized in that, In step S2, the playback range The replay range of each attention head (h) is distributed exponentially across different attention heads. For growth factors The exponential distribution with base h and exponent h is the rounded distribution, where h is the index of the attention head, ranging from 1 to H, and H is the number of attention heads in each layer of the model.

3. The method according to claim 2, characterized in that, The growth factor The growth factor is determined by the number of attention heads H and the maximum context length T. The range is greater than 1 and less than or equal to 2, which makes the maximum rewind range It equals the maximum context length that the model can handle.

4. The method according to claim 1, characterized in that, In step S3, the sparse attention mask matrix Specifically: for each current token on the diagonal, only the past tokens are retained. One token participates in the attention calculation, while the remaining positions are masked and set to negative infinity; wherein, the token includes the current token.

5. The method according to claim 1, characterized in that, In step S4, the forward propagation calculation to obtain the hidden state of the last layer output specifically involves: using the model with the configured sparse attention mask matrix output in step S3 to perform forward propagation calculation to obtain the hidden state of the last layer output of the model. The hidden state of the last layer output integrates semantic information and hidden position information inferred through the topological structure.

6. The method according to claim 1, characterized in that, Step S5 specifically involves: inputting the hidden state output from the last layer into a linear classification layer, and obtaining the predicted probability distribution of the next label in the vocabulary dimension via the Softmax function; then constructing a cross-entropy loss function for the autoregressive task based on the predicted probability distribution of the next label; using the minimization of the cross-entropy loss function, performing backpropagation through the chain rule to calculate the gradient of each learnable parameter in the model; updating the model parameters using an optimizer and explicitly introducing a weight decay term; using a learning rate scheduling strategy during training; and finally obtaining the trained model through closed-loop iterative optimization.

7. The method according to claim 1, characterized in that, The method further includes: the trained model comprises stacked Transformer blocks with L network layers; the number of network layers L is greater than or equal to the logarithm with base 2 and T as the argument.

8. An electronic device comprising a memory and a processor, characterized in that, The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the large language model training method based on range position encoding as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the large language model training method based on range position encoding as described in any one of claims 1-7.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the large language model training method based on range position encoding as described in any one of claims 1-7.

Images