Large model long text extrapolation method and system based on mirror image periodic continuation

Abstract

The invention relates to a large-model long text extrapolation method and system based on mirror image periodic continuation, and belongs to the technical field of model processing. The method comprises the following steps: (1) transforming an ROPE position coding strategy; a PEPE-mPSE strategy is used to update an original ROPE position coding strategy; (2) carrying out fine tuning training on the Transform architecture model by using long text data; and (3) performing long text reasoning through the trained Transform architecture model. According to the non-interpolation type large-model long-text extrapolation method, mirror image period continuation is carried out on learned position code distribution of the model in a pre-training text length window, the model is expanded to be beyond the pre-training text length, the long-text extrapolation effect of the model is greatly improved, and compared with a mainstream extrapolation method, the method has the advantages that the method is simple and convenient to implement, and the method is easy to popularize. The method provided by the invention has the possibility of realizing infinite extrapolation on the theoretical level.

Description

Technical Field

[0001] The present invention relates to a large model long text extrapolation method and system based on mirror period extension, belonging to the technical field of model processing. Background Art

[0002] With the rapid development of large model technology and the increasing abundance of large-scale computing resources and data resources, higher requirements are imposed on the ability of these models to process large-scale and complex data. Especially when dealing with long text inputs, large models not only need to have efficient understanding capabilities but also be able to generate high-quality long text outputs. However, in practical applications, when dealing with extremely long texts, large models face a series of challenges, such as huge consumption of computing resources, slow inference speed, limited depth of understanding, etc.

[0003] To address these issues, large model long text extrapolation technology has emerged. Currently, the technologies for improving the performance of large models in long text extrapolation mainly focus on the following two key areas: One is to optimize the attention mechanism window of the large model. By adjusting the attention window of the model, the problem of excessive computational burden caused by the increase in text length can be effectively alleviated. Specific strategies include, but are not limited to, adopting a sliding window mechanism or a block processing method, enabling the model to reduce the demand for computing resources while maintaining good performance.

[0004] The other is to improve the position encoding strategy in the Transformer architecture of the large model. Traditional position encoding methods are difficult to meet the processing requirements of extremely long texts. Therefore, researchers have proposed a series of new encoding schemes, such as relative position encoding, learnable position encoding, etc., to enhance the model's ability to capture long-distance dependency relationships, thereby improving the accuracy and efficiency of its understanding and generating long texts. Among them, ROPE (Rotary Position Embedding) position encoding, as the current mainstream position encoding method for large models, embeds relative position information through a rotation matrix. The design of ROPE does not depend on a fixed dimension size. Therefore, it has better scalability, making it show unique advantages in long text extrapolation. Although ROPE has certain direct extrapolation characteristics and can provide an effective way to embed position information for the Transformer architecture through relative position encoding, it still has deficiencies when dealing with text lengths exceeding the maximum length of the pre-training data.

[0005] Some current long text extrapolation works attempt to address the limitations of ROPE from different perspectives, including methods such as PI (Position Interpolation), NTK (Neural Tangent Kernel), and YaRN. The PI method enhances the flexibility and adaptability of positional encoding through interpolation techniques, enabling the model to better handle sequences of unknown lengths. NTK adjusts the method based on the Neural Tangent Kernel (NTK) to improve the model's learning ability for long-range dependencies by adjusting the rotation base. YaRN proposes a novel mechanism aimed at optimizing the attention mechanism to improve the efficiency and accuracy of processing long texts. However, despite the progress made by the above methods, there are still some deficiencies in current long text extrapolation work. The current mainstream long text extrapolation methods based on the transformation of ROPE (positional interpolation, NTK-series, YaRN, etc.) essentially belong to an interpolation-based long text extrapolation method. One drawback of the interpolation-based method is that it compresses the relative positions between adjacent tokens. In particular, the PI method will directly cause a relatively serious problem of attention resolution distortion. To solve the token distance compression problem introduced by the interpolation-based method, this invention is proposed. Summary of the Invention

[0006] Aiming at the deficiencies of the prior art, the present invention provides a method and system for long text extrapolation of large models based on mirror periodic extension, adopting a non-interpolation-based long text extrapolation method. The present invention is a further improvement on the method based on periodic extension. The main differences between the two are as follows: 1. The ways of transforming the ROPE strategy are different between the two: In the way based on mirror periodic extension, first, perform mirror symmetry processing on the positional encoding distribution, and then use the original positional encoding distribution and the mirror distribution together as a complete period for periodic extension, which results in the extension period becoming twice that of the periodic extension method.

[0007] 2. Mirror periodic extension effectively avoids the problem of discontinuous jumps in the attention distribution at the periodic boundary points in the periodic extension method, making the change of the attention distribution smoother and more continuous, which also causes less perturbation to the model's understanding ability at the jump point positions.

[0008] 3. The extrapolation performance of mirror periodic extension is better than that of the periodic extension method. Although the lengthening of the extension period makes the model require more training steps to learn during the fine-tuning stage, experiments have shown that the method based on mirror periodic extension can have a longer extrapolation distance.

[0009] Term Explanation: 1. Transformer Architecture: Transformer is a deep learning model architecture widely used in natural language processing (NLP) and other sequence-to-sequence tasks. It abandons the traditional recurrent neural network (RNN) structure and instead adopts the self-attention mechanism, enabling the model to process input data in parallel and better capture long-range dependencies. The Transformer architecture model mainly consists of an input representation, an encoder, a decoder, and an output layer. The input representation includes token embedding and positional encoding, which are used to convert text into a vector form that the model can process and retain the word order information. The encoder captures the dependencies between words in the input sequence through the multi-head self-attention mechanism and further processes this information using a feed-forward neural network. Residual connections and layer normalization are used during this process to stabilize the training. The decoder not only contains a similar structure to generate the output sequence but also has an encoder-decoder attention layer to consider the information of the entire input sequence when generating each new word and adopts a masked multi-head self-attention mechanism to prevent the current prediction from being affected by future words. Finally, the output layer converts the output of the decoder into an easily understandable probability distribution to complete tasks such as translation.

[0010] 2. Rotary Position Embedding (ROPE): ROPE is an improved position encoding technique designed to address the problems that traditional position encoding may encounter in long text scenarios. ROPE incorporates position information in a dynamic way by applying a rotation matrix to the query (Q) and key (K) vectors. This method not only retains the advantages of relative position encoding but also can adapt to sequences of different lengths, thereby improving the model's performance in long text tasks.

[0011] 3. Dimension Subspaces: In ROPE, the high-dimensional position encoding is divided into multiple two-dimensional subspaces or planes. These subspaces are called dimension subspaces, which allow the model to independently process position information in different dimensions. In this way, ROPE can effectively expand the expressive power of position encoding without increasing computational complexity, especially when dealing with long sequences.

[0012] 4. Rotation Base: In ROPE, the rotation base is a key parameter used to control the magnitude of the rotation frequency within each dimensional subspace. Each dimensional subspace i has an independent rotation frequency of , and its calculation formula is , where d is the total dimension of the positional encoding, and base is called the rotation base. By default, ROPE sets the base value to 10,000. Adjusting the base value can affect the model's ability to extrapolate long texts. For example, increasing the base value can help the model better adapt to longer sequences.

[0013] 5. PEPE Intervention Start Token Position : The PEPE intervention start token position is a hyperparameter variable defined in the present invention, which represents the starting point at which the mirror period extension method of the present invention starts to intervene on the token position coordinates of the positional encoding. This hyperparameter can be arbitrarily specified within the maximum window of model pre-training, and by default, the maximum length of model training is taken as the starting position. For example: The maximum length of pre-training of the Llama2 model is 4k, then by default . Experiments have shown that when the value of this parameter is less than the maximum window of model pre-training, the extrapolation multiple of the method of the present invention can be significantly improved. For example, for the Llama2 pre-trained model, taking . However, if the value of this parameter is significantly less than the maximum window of model pre-training, it will lead to a decrease in the model's understanding ability within a short-distance window. Therefore, if the ultra-long text fine-tuning dataset is limited, this parameter can be considered for adjustment to obtain a larger extrapolation multiple.

[0014] 6. Critical Dimension : The critical dimension is a hyperparameter variable defined in the present invention, which represents the starting dimension at which the mirror period extension method of the present invention is default to intervene on the dimension coordinates of the positional encoding. As is well known, the distribution of positional encoding fluctuates periodically within each dimensional subspace, and as the dimension increases, the period becomes longer. The critical dimension represents the dimension at which the positional encoding distribution can exactly complete a periodic fluctuation at the maximum length of the model training data. The specific calculation formula: ; where base is the rotation base, and represent the maximum input text lengths in the training and inference stages respectively, and the ratio of the two , when , then the long text extrapolation problem is faced.

[0015] 7. Llama Series Models: Llama is a large language model released by Meta AI. This model is based on the Transformer architecture and incorporates several optimization measures, such as using GPT-3 style pre-normalization, the SwiGLU activation function, and ROPE, etc., to enhance the model's performance. The Llama model family includes different scale versions with parameters ranging from 7B to 65B, as well as different iterative versions such as Llama, Llama2, and Llama3.

[0016] 8. Qwen Series Models: Qwen is a series of open-source large language models developed by the DAMO Academy of Alibaba Group. These models have been extensively trained and possess strong generalization capabilities and diverse functions, enabling them to perform various tasks such as text generation, question answering, dialogue, translation, and text classification. There are multiple versions of the Qwen model, which have been optimized for different computing resource requirements and application scenarios, such as Qwen-7B, Qwen-14B, Qwen-72B, etc., and also include different iterative versions such as Qwen and Qwen2.5.

[0017] The technical solution of the present invention is as follows: A method for extrapolating long texts of large models based on mirror period extension, the steps are as follows: (1) Transformation of the ROPE position encoding strategy; use the PEPE-mPSE strategy to update the original ROPE position encoding strategy; (2) Fine-tune and train the Transformer architecture model using long text data; (3) Perform long text inference through the trained Transformer architecture model.

[0018] Preferably according to the present invention, in step (1), the specific update process is as follows: Initialize hyperparameters, including setting the maximum training window length of ROPE encoding, the current input text length, the position encoding dimension, the position index sequence, and the rotation frequency sequence; Inference text length judgment and processing. If the current input text length exceeds the maximum length of the text window during model training, then perform mirror period extension transformation on the ROPE position encoding strategy; otherwise, directly extrapolate using the original ROPE position encoding strategy;

[0019] Dimension judgment and processing. According to the formula Calculate the critical dimension , taking the Llama2 7B model as an example, the vector space dimension Training window length , rotation base , then its critical dimension , the part exceeding the critical dimension is the high-dimensional space, and vice versa for the low-dimensional space. Next, the critical dimension is determined and the total dimension of the position encoding 、 between 1. When , directly extrapolate; when , perform full-scale mirror periodic extension on all dimension encodings; when , then adopt the strategy of mirror periodic extension for the high-dimensional distribution and direct extrapolation for the low-dimensional distribution. Specifically:

[0020] When the dimension of the position encoding , the position encoding distribution within the subspace of this dimension is incomplete. At this time, through mirror periodic extension, mirror processing is performed on the position encoding distribution with incomplete periodicity within the high-dimensional subspace, and then periodic extrapolation is performed on the position encoding distribution; conversely, the position encoding distribution within the subspace of this dimension is complete, and no transformation processing is required for its position encoding.

[0021] According to the preference of the present invention, in step (2), the fine-tuning training steps are: (21) Prepare a long text dataset; (22) Full-parameter fine-tuning, using the prepared long text dataset to perform full-parameter SFT fine-tuning on the Transformer architecture model.

[0022] According to the preference of the present invention, in step (21), the length of the long text in the long text dataset is 4-8 times the text length during the training of the Transformer architecture model; According to the preference of the present invention, in step (22), the full-parameter SFT fine-tuning process includes: (221) Environment preparation: To support a training set with a length of up to 32K, at least 4 NVIDIA A100 40GB GPUs need to be equipped, adopt an acceleration framework based on Flash Attention 2, and use DeepSpeed for efficient multi-GPU distributed training to improve training efficiency; (222) Selection of the base model: Select an open-source base model, such as a Transformer architecture model that adopts the Rotary Position Embedding (ROPE) mechanism like Llama2, Llama3.1, or Qwen. In the present invention, mainly the Llama2 7B model is used as the basis for improvement; (223) Hyperparameter setting: Set the training batch size, learning rate, and fine-tuning steps; (224) ROPE encoding optimization: Apply the PEPE-mPSE strategy to transform the original ROPE position encoding to enhance the model's ability to understand and process position information in the sequence; (225) Model fine-tuning: Based on the above preparations and optimization measures, perform a full-parameter fine-tuning process to complete model training and make it adapt to the requirements of specific application scenarios.

[0023] Preferably according to the present invention, step (3) specifically includes: ROPE transformation in the inference process; including adjusting the position encoding part in the inference code to make it consistent with the position encoding strategy in the model fine-tuning stage; Long text inference test; Use a long text dataset for inference testing.

[0024] Preferably according to the present invention, mirror period extension refers to: Through the PEPE-mPSE formula, transform and update the position index sequence and rotation frequency sequence of the ROPE encoding to complete the transformation process of the position encoding. The specific process is as follows: Take a high-dimensional subspace as an object. For the position encoding that needs to be mirror period extended, define the mirror extension period as , define the maximum text length in the pre-training stage of the Transformer architecture model as , the length of the current input text is , the rotation frequency of the subspace is , assuming that the position encoding process of ROPE is represented as at the th token, and the position encoding process using the PEPE-mPSE strategy is represented as: ; The PEPE-mPSE strategy is divided into two processes: a. Construct the first complete mirror extension period within the high-dimensional subspace, that is, complete mirror symmetry for the position encoding within the range of the model training text window size on the interval . At this time, and the position encoding in the interval are axisymmetric about . This process is represented as: (1) b. Take the interval as the first complete mirror extension period and complete periodic extension on the interval . At this time, the mirror extension period is: . This process is represented as: (2) Equation (1) is the representation of Equation (2) within the first extension period, and Equation (2) corresponds to the transformation of the above-mentioned ROPE position encoding strategy; the original ROPE position encoding strategy is updated using the PEPE-mPSE strategy, and the process of mirror period extension of the position encoding is completed through Equation (2).

[0025] The above processes a and b are combined as: (3) where mod is the modulo calculation, is to perform floor division on represents that the m position is within the even interval. On the contrary, represents that the m position is within the odd interval, is the first even interval, is the first odd interval, and so on, represents the original ROPE position encoding process, which is represented in the two-dimensional subspace as: (4).

[0026] According to a further preference of the present invention, implementing the transformation of mirror period extension requires meeting two conditions: ① Within the model training text window size L, there exists a critical dimension , and only the high-dimensional position encoding beyond the critical dimension is subjected to period extension, while the low-dimensional positions adopt the method of direct extrapolation; ② When the current input text length exceeds the model training text window size L, the position encoding of the exceeded part adopts the method of period extension, while the position encoding of the unexceeded part is not intervened; Therefore, the implementation process of mirror period extension is comprehensively represented as: (5) where represents the th two-dimensional subspace, is the length of the current input text of the model; Through Equation (5), the transformation of mirror period extension of the position encoding of the high-dimensional subspace can be completed.

[0027] A large model long text extrapolation system based on mirror period extension includes: A transformation module for transforming the ROPE position encoding strategy and updating the original ROPE position encoding strategy using the PEPE-mPSE strategy; The training module is used to fine-tune the Transformer architecture model using long text data; The reasoning module is used to perform long text reasoning through the trained Transformer architecture model.

[0028] The method of the present invention has a high degree of universality and can be widely applied to all Transformer architecture models based on ROPE position encoding. Transformer architecture models include the current mainstream and widely used open source models such as Llama and Qwen series. Therefore, when selecting a model, it has great flexibility and diversity, and the most suitable basic model can be selected according to specific application scenarios and requirements.

[0029] The beneficial effects of the present invention are: 1. The present invention adopts a non-interpolation large-model long text extrapolation method, which greatly improves the long text extrapolation effect of the model by mirroring the position coding distribution learned by the model within the pre-trained text length window and extending it beyond the model pre-trained text length. Compared with the mainstream extrapolation method, the method of the present invention is more capable of achieving infinite extrapolation at the theoretical level.

[0030] 2. The fine-tuning training of the method of the present invention has lower time cost than the current mainstream extrapolation method. Experimental verification shows that the method of the present invention only needs 100-200 steps of fine-tuning to significantly improve the long text extrapolation capability of the model.

[0031] 3. Compared with the mainstream interpolation (linear / non-linear) long text extrapolation method, the present invention can reduce the loss of the model's understanding and reasoning ability within the pre-trained text length window.

[0032] 4. Compared with the large model long text extrapolation method based on cycle extension, the present invention has two main advantages: a. In terms of model attention distribution, the method based on mirror cycle extension can effectively avoid attention distribution at the position of the intervening start token. The jump phenomenon at the position makes the attention distribution smoother and alleviates the inference disturbance of the model near the position caused by the attention jump. b. Under the same fine-tuning text length, the long text extrapolation method based on mirror cycle extension requires slightly more fine-tuning steps (about 1-2 times) than the method based on cycle extension, but the method based on mirror cycle extension has better extrapolation performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a distribution diagram of position coding in different dimensional subspaces of the long text extrapolation method based on mirror cycle extension of the present invention; Among them, the left rotation schematic diagram shows the process of the model applying rotational positional encoding to the 108th dimension of the word vector. In the figure, the blue area represents the range of learned positional encoding within the training window, while the blank area represents the unlearned representation space beyond the training window, that is, the area where encoding representation overflow may occur. The optimization goal of the present invention is to ensure that after applying the positional encoding, the vector can still remain within the blue area; The arrow indicates the actual position of the word vector after applying the rotational positional encoding strategy in the two-dimensional space.

[0034] The dashed line represents the position of the word vector after applying the positional encoding using the ROPE strategy; The solid line shows the position of the word vector after applying the positional encoding using the PEPE-mPSE strategy proposed by the present invention; It can be seen from the figure that at the 5120th token, the ROPE strategy causes vector representation overflow, while the PEPE-mPSE strategy successfully maintains the vector within the blue area; The right waveform diagram shows the encoding representations of two randomly selected positional encoding dimensions (low dimension d = 72 and high dimension d = 108). The abscissa is divided into two parts: Normal: Corresponding to within the length range of the model training window; Extension: Refers to the interval beyond the length of the model training window, that is, the long text extrapolation area; For the low dimension (d = 72), since the model has learned enough periodic distributions in this dimension, continuing to use the ROPE positional encoding will not cause the problem of vector representation overflow. However, in the case of the high dimension (d = 108), since only part of the periodic distribution is learned during the training stage (within the 4k window), vector representation overflow is likely to occur during long text extrapolation. Therefore, the PEPE-mPSE method is selected to complete the positional encoding in the high-dimensional space to avoid the above problems; Figure 2 It is a schematic diagram of the adaptive transformation process of the PEPE-mPSE strategy of the present invention. Specific implementation manners

[0035] The present invention will be further described below through examples in conjunction with the accompanying drawings, but is not limited thereto.

[0036] Example 1: A long text extrapolation method for large models based on mirror periodic extension, the steps are as follows: (1) Transformation of the ROPE positional encoding strategy; Use the PEPE-mPSE strategy to update the original ROPE positional encoding strategy. The specific update process is: Initialize hyperparameters, including setting the maximum training window length of the ROPE encoding, the length of the current input text, the position encoding dimension, the position index sequence, and the rotation frequency sequence; Inference text length judgment and processing. If the length of the current input text exceeds the maximum length of the text window during model training, the ROPE position encoding strategy is transformed by mirror periodic extension; otherwise, the original ROPE position encoding strategy is directly extrapolated; Dimension judgment and processing. According to the formula Calculate the critical dimension , and judge the size relationship between the critical dimension and the total dimension of the position encoding 、 and 1. When , directly extrapolate; when , perform full mirror periodic extension on all dimension encodings; when , then adopt the strategy of mirror periodic extension for high-dimensional distributions and direct extrapolation for low-dimensional distributions. Specifically:

[0037] When the dimension of the position encoding , it is considered that the position encoding distribution within the subspace of this dimension is incomplete. At this time, through mirror periodic extension, the position encoding distribution with incomplete period within the high-dimensional subspace is mirror processed, and then the position encoding distribution is periodically extended outward; otherwise, it is considered that the position encoding distribution within the subspace of this dimension is complete and no transformation processing is required for its position encoding.

[0038] (2) Fine-tune and train the Transformer architecture model using long text data. The steps are as follows: (21) Prepare a long text dataset. The length of the long text in the long text dataset is 4 - 8 times the length of the text during the training of the Transformer architecture model. Through a large number of experiments, it is verified that if the method of this embodiment only uses data with a training text length of 2 times for fine-tuning training, although it can also have good extrapolation performance, especially when shrinking the hyperparameter , the model has a farther extrapolation ability, but this has a certain degree of adverse impact on the model's understanding ability within a short-distance window.

[0039] (22) Full parameter fine-tuning. Use the prepared long text dataset to perform full parameter SFT fine-tuning on the Transformer architecture model, including: (221) Environment preparation: To support a training set with a length of up to 32K, at least 4 NVIDIA A100 40GB GPUs need to be equipped. Adopt an acceleration framework based on Flash Attention 2 and use DeepSpeed for efficient multi-GPU distributed training to improve training efficiency; (222) Base model selection: Select the open-source base model, Llama2 7B model; (223) Hyperparameter settings: Set the training batch size (train_batch_size) to 32, the learning rate to 3e-5, and set the number of fine-tuning steps to 100 to ensure that the model can learn effectively without overfitting; (224) ROPE encoding optimization: Apply the PEPE-mPSE strategy to transform the original ROPE position encoding to enhance the model's ability to understand and process position information in the sequence; (225) Model fine-tuning: Based on the above preparations and optimization measures, perform the full-parameter fine-tuning process to complete model training and make it adapt to the requirements of specific application scenarios.

[0040] The present invention only requires a very small number of fine-tuning steps to achieve good extrapolation ability. Through a large number of experiments, it is verified that when the method of the present invention is used to fine-tune and train the Llama2 model with data of 32k text length, only 100 fine-tuning steps are required to have good extrapolation performance, and the best performance can be achieved if fine-tuned to 200 steps.

[0041] (3) Perform long text inference through the trained Transformer architecture model, specifically including: ROPE transformation in the inference process; including adjusting the position encoding part in the inference code to be consistent with the position encoding strategy in the model fine-tuning stage; Long text inference test; Use the long text dataset for inference test.

[0042] The above mirror period extension refers to: Through the PEPE-mPSE formula, transform and update the position index sequence and rotation frequency sequence of the ROPE encoding to complete the transformation process of the position encoding. The specific process is as follows: Take a high-dimensional subspace as an object. For the position encoding that needs to be mirror period extended, define the mirror extension period as , define the maximum text length in the pre-training stage of the Transformer architecture model as , the length of the current input text is , the rotation frequency of the subspace is . Assume that the position encoding process of ROPE is represented as at the -th token, and the position encoding process using the PEPE-mPSE strategy is represented as: ; The PEPE-mPSE strategy is divided into two processes: a. Construct the first complete mirror extension cycle in the high-dimensional subspace , that is, for the position encoding within the model training text window size range, complete mirror symmetry on the interval . At this time, and the position encoding of the interval is axisymmetric about . This process is expressed as: (1) b. Take the interval as the first complete mirror extension cycle and complete periodic extension on the interval . At this time, the mirror extension cycle is: . This process is expressed as: (2) Equation (1) is the representation of Equation (2) within the first extension cycle. Equation (2) corresponds to the transformation of the above ROPE position encoding strategy; use the PEPE-mPSE strategy to update the original ROPE position encoding strategy and complete the process of mirror periodic extension of the position encoding through Equation (2).

[0043] The above processes a and b are combined as: (3) where mod is the modulo calculation, is to perform floor division on , represents that the m position is within the even interval. On the contrary, represents that the m position is within the odd interval, is the first even interval, is the first odd interval, and so on, represents the original ROPE position encoding process, which is expressed in the two-dimensional subspace as: (4).

[0044] Implementing the transformation of mirror periodic extension requires meeting two conditions: ① Within the model training text window size L, there exists a critical dimension , and only perform periodic extension on the high-dimensional position encoding beyond the critical dimension, and use the direct extrapolation method for the low-dimensional positions; ② When the current input text length exceeds the model training text window size L, use the periodic extension method for the position encoding of the exceeded part, and do not intervene in the position encoding of the unexceeded part; Therefore, the implementation process of mirror periodic extension is comprehensively expressed as: (5) Wherein, represents the th two-dimensional subspace, is the length of the current input text of the model; Through formula (5), the transformation of mirror periodic extension of the position encoding of the high-dimensional subspace can be completed.

[0045] Embodiment 2: This embodiment provides a large model long text extrapolation system based on mirror periodic extension, including: A transformation module for transforming the ROPE position encoding strategy and updating the original ROPE position encoding strategy using the PEPE-mPSE strategy; A training module for fine-tuning and training the Transformer architecture model using long text data; An inference module for performing long text inference through the trained Transformer architecture model.

Claims

1. A method for extrapolating long texts of large models based on mirror period extension, characterized in that The steps are as follows: (1) Transformation of the ROPE position encoding strategy; Update the ROPE position encoding strategy using the PEPE-mPSE strategy; (2) Fine-tune and train the Transformer architecture model using long text data; (3) Perform long text inference through the trained Transformer architecture model.

2. The long text extrapolation method for large models based on mirror period extension according to claim 1, wherein In step (1), the specific update process is: Initialize hyperparameters, including setting the maximum training window length of ROPE encoding, the current input text length, the position encoding dimension, the position index sequence, and the rotation frequency sequence; Inference text length judgment and processing. If the current input text length exceeds the maximum text window length during model training, perform mirror periodic extension transformation on the ROPE position encoding strategy; otherwise, directly extrapolate using the ROPE position encoding strategy. Dimension judgment and processing, according to the formula calculate the critical dimension , and judge the critical dimension and the total dimension of the position encoding 、 the size relationship between 1. When , directly extrapolate; when , perform full mirror cycle extension on all dimension encodings; when , then adopt the strategy of mirror cycle extension with high-dimensional distribution and direct extrapolation with low-dimensional distribution.

3. The long text extrapolation method for large models based on mirror period extension according to claim 2, wherein In step (2), the fine-tuning training steps are: (21) Prepare a long text dataset; (22) Full-parameter fine-tuning. Use the prepared long text dataset to perform full-parameter SFT fine-tuning on the Transformer architecture model.

4. The method for extrapolating long texts of large models based on mirror period extension according to claim 3, wherein In step (21), the length of the long text in the long text dataset is 4 - 8 times the text length during the training of the Transformer architecture model.

5. The method for extrapolating long texts of large models based on mirror period extension according to claim 4, wherein, In step (22), the full-parameter SFT fine-tuning process includes: (221) Environment preparation: Equip 4 NVIDIA A100 40GB GPUs, adopt an acceleration framework based on Flash Attention 2, and use DeepSpeed for multi-GPU distributed training; (222) Base model selection: Select open-source base models Llama2, Llama3.1, or Qwen; (223) Hyperparameter setting: Set the training batch size, learning rate, and fine-tuning steps; (224) ROPE encoding optimization: Apply the PEPE-mPSE strategy to transform the original ROPE position encoding; (225) Model fine-tuning: Based on the above preparations and optimization measures, perform the full-parameter fine-tuning process to complete model training.

6. The method for extrapolating long texts of large models based on mirror period extension according to claim 5, wherein, Step (3) specifically includes: ROPE transformation during the inference process; This includes adjusting the position encoding part in the inference code to be consistent with the position encoding strategy in the model fine-tuning stage; Long text inference test; Use the long text dataset for inference testing.

7. The method for extrapolating long texts of large models based on mirror period extension according to claim 6, wherein Mirror periodic extension means: Through the PEPE-mPSE formula, transform and update the position index sequence and rotation frequency sequence of ROPE encoding to complete the transformation process of position encoding. The specific process is as follows: Taking a high-dimensional subspace as the object, encoding the positions that need to be mirror-periodically extended, defining the mirror extension period as , defining the maximum text length in the pre-training stage of the Transformer architecture model as , the length of the current input text is , the rotation frequency of the subspace is , assuming that the position encoding process of ROPE is represented as at the -th token, and the position encoding process using the PEPE-mPSE strategy is represented as: ; The PEPE-mPSE strategy is divided into two processes: a. Construct the first complete mirror extension period within the high-dimensional subspace , that is, for the position encoding within the size of the model training text window range, complete mirror symmetry on the interval . At this time, and the position encoding of the interval is axisymmetric about . This process is expressed as: (1) b, take the interval as the first complete mirror extension period, and complete the periodic extension on the interval . At this time, the mirror extension period is: . This process is expressed as: (2) The above processes a and b are combined as: (3) where mod is the modulo calculation, for rounding down, represents that the m position is within the even interval. On the contrary, represents that the m position is within the odd interval, is the first even interval, is the first odd interval, and so on, represents the process of the native ROPE position encoding, which is represented in the two-dimensional subspace as: （4）。 8. The method for extrapolating long texts of large models based on mirror period extension according to claim 7, wherein, Two conditions need to be met for implementing the mirror periodic extension transformation: ①Within the model training text window size L, there exists a critical dimension , and only the periodic extension is performed on the high-dimensional position encoding beyond the critical dimension, while the direct extrapolation method is adopted for the low-dimensional positions; ② When the current input text length exceeds the model training text window size L, use the method of periodic extension for the position encoding of the exceeded part, and do not intervene in the position encoding of the unexceeded part; Therefore, the implementation process of mirror periodic extension is comprehensively expressed as: （5） Among them, represents the th two-dimensional subspace, is the length of the current input text of the model; Through formula (5), the modification of mirror periodic extension of the position encoding of the high-dimensional subspace can be completed. can be completed.

9. A large model long text extrapolation system based on mirror period extension, characterized in that It includes: A transformation module for transforming the ROPE position encoding strategy and updating the original ROPE position encoding strategy using the PEPE-mPSE strategy; A training module for fine-tuning and training a Transformer architecture model using long text data; An inference module for performing long text inference through the trained Transformer architecture model.

Images