Method for extending model context window, storage medium, electronic device and computer program product

Abstract

The embodiment of the application provides a model context window expansion method, a storage medium, an electronic device and a computer program product, the method comprises: obtaining the position encoding of RoPE, different RoPE dimensions and the expansion length of the context window of the pre-training model under the first length; according to the different RoPE dimensions, the preset evolutionary search algorithm and the expansion length of the context window, determine the target scaling factor of each RoPE dimension corresponding to the expansion length; wherein, the expansion length is greater than the first length; according to the target scaling factor, the position encoding of RoPE under the first length is subjected to non-uniform interpolation processing, and the context window of the pre-training model with the expansion length is obtained. The model in the related art cannot consider the context information beyond the window range when processing long text, which leads to the problem that the performance is poor in long distance dependence and global consistency, and the breakthrough expansion of the context window of the large language model is realized.

Description

Technical Field

[0001] This application relates to the field of communications, and more specifically, to a method for extending a model context window, a storage medium, an electronic device, and a computer program product. Background Technology

[0002] Large Language Models (LLMs) have achieved remarkable success in the field of Natural Language Processing (NLP). These models, through deep learning techniques, are able to understand and generate natural language text. However, the performance of LLMs is largely limited by the size of their context window, i.e., the number of input tokens the model can process simultaneously.

[0003] Traditional LLMs typically have a fixed context window size, such as 1024 to 4096 tokens. This limitation means that the model cannot consider contextual information beyond the window size when processing long text, resulting in poor performance in long-distance dependencies and global consistency. Summary of the Invention

[0004] This application provides a method for extending a model context window, a storage medium, an electronic device, and a computer program product to at least solve the problem in the related art that the model cannot consider context information beyond the window range when processing long text, resulting in poor performance in terms of long-distance dependencies and global consistency.

[0005] According to one embodiment of this application, a method for extending a model context window is provided, comprising:

[0006] For a context window of a first length for a pre-trained model, obtain the position encoding of the position rotation embedding RoPE of the pre-trained model under the first length, the different RoPE dimensions, and the extended length of the context window;

[0007] Based on the different RoPE dimensions, the preset evolutionary search algorithm, and the expansion length of the context window, the target scaling factor for each RoPE dimension corresponding to the expansion length is determined; wherein, the expansion length is greater than the first length;

[0008] Based on the target scaling factor, the positional encoding of the RoPE at the first length is subjected to non-uniform interpolation to obtain the extended length context window of the pre-trained model.

[0009] According to yet another embodiment of this application, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and the computer program is configured to perform the steps in any of the above method embodiments when it is run.

[0010] According to yet another embodiment of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0011] According to yet another embodiment of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0012] In this embodiment, for a context window of a first length in a pre-trained model, the positional encoding of the position rotation embedding RoPE under the first length, different RoPE dimensions, and the extended length of the context window are obtained. Based on different RoPE dimensions, a preset evolutionary search algorithm, and the extended length of the context window, a target scaling factor is determined for each RoPE dimension corresponding to the extended length; wherein the extended length is greater than the first length. Based on the target scaling factor, non-uniform interpolation is performed on the positional encoding of the RoPE under the first length to obtain the extended length context window of the pre-trained model. This solves the problem in related technologies where models cannot consider contextual information beyond the window range when processing long texts, resulting in poor performance in long-distance dependencies and global consistency, and achieves a breakthrough expansion of the context window for large language models. Attached Figure Description

[0013] Figure 1 This is a hardware structure block diagram of the mobile terminal used in the method embodiments of this application;

[0014] Figure 2 This is a flowchart of a method for expanding a model context window according to an embodiment of this application;

[0015] Figure 3 This is a system architecture block diagram of the model context window extension according to an embodiment of this application;

[0016] Figure 4 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0017] The embodiments of this application will be described in detail below with reference to the accompanying drawings and examples.

[0018] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0019] In related technologies, fine-tuning techniques are typically used to expand the context window, but this usually requires long text that matches the length of the target context, which is relatively scarce in practical applications. Furthermore, the fine-tuning process itself is computationally expensive, requiring significant GPU resources and time.

[0020] Moreover, even with fine-tuning to expand the context window, newly introduced positional indices can still lead to a performance degradation, especially at positions where the model has not been trained. This "catastrophic forgetting" phenomenon makes it difficult for the model to maintain its original performance level in new contexts.

[0021] Traditional methods of expanding the model context window through linear interpolation often rely on manually designed rules, which cannot fully utilize the non-uniform information within the model. When processing new token positions, this can lead to the loss of key information. Furthermore, relying on manually designed rules for position interpolation limits the model's adaptability and generalization ability to new contexts.

[0022] Therefore, the existing technologies have significant limitations in extending the context window of LLM, and a new technical solution is needed to overcome these problems and achieve more efficient long context processing capabilities.

[0023] Based on this, this application proposes a method for expanding the model context window. The technical concept is to utilize the non-uniformity in position embedding, find the optimal rescaling factor for each RoPE dimension and tokens at different positions through a search algorithm, and adjust the RoPE by the rescaling factor. This achieves the expansion of the model context window while maintaining the high performance of the model after the context window is expanded.

[0024] To facilitate understanding of the embodiments of this application, some terms mentioned in the embodiments of this application are explained below:

[0025] The context window is the maximum length of the input sequence that the model can consider at the same time, usually measured in tokens.

[0026] Rotary Position Embedding (RoPE) is a positional encoding technique used in large language models (LLMs). RoPE encodes positional information through rotation operations, encoding the positional information in the sequence into a rotation matrix, which is then applied to the model's input vector to encode the positional information as part of the input vector. This allows RoPE to maintain positional information without being directly limited by the length of the input sequence, theoretically enabling it to support infinitely long sequences.

[0027] In RoPE, positional information is encoded using different frequencies of sine and cosine functions. Each positional embedding dimension corresponds to a specific frequency. With RoPE, the model's input vector is combined with a rotation matrix at each position. These rotation matrices are computed based on position, preserving the relative positional information between words in the sequence, even as the sequence length increases.

[0028] RoPE encodes position information by rotating certain parts of the input vector with sine and cosine values ​​generated based on the position.

[0029] RoPE utilizes sine and cosine values ​​of different frequencies to encode different positional information. Each RoPE dimension has a corresponding frequency, which determines the periodicity of the position embedding and the variation of the rotation angle. In RoPE, the encoding of positional information is directly related to frequency; by adjusting the frequency (or frequency group), the representation of positional information in the sequence can be controlled.

[0030] The scaling factor is used to adjust the rotation angle of RoPE when processing different positional information to accommodate longer sequences. Specifically, the scaling factor can affect the amplitude of the sine and cosine values ​​in the RoPE operation, thereby affecting the representation of positional information within the model.

[0031] The methods and embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of the mobile terminal used in the embodiments of the method of this application. For example... Figure 1 As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0032] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the model context window expansion method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0033] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0034] This embodiment provides an extended method for the model context window running on the aforementioned mobile terminal. Figure 2 This is a flowchart of a method for expanding a model context window according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:

[0035] Step S201: For a context window of a first length of the pre-trained model, obtain the position encoding of the position rotation embedding RoPE of the pre-trained model at the first length, the different RoPE dimensions, and the extended length of the context window.

[0036] In this embodiment of the invention, the first length is the length of the context window to be expanded, and the expansion length is the target length of the context window to be expanded. The pre-trained model can be a large language model that relies on the context window to understand text sequences during training.

[0037] As an example, RoPE encodes the relative position of a token at each position in a sequence with respect to other positions by applying rotation operations in different dimensions. RoPE encoding can adapt to context windows of varying lengths, representing tokens at different positions using different rotation angles.

[0038] As an example, when the model's context window needs to be expanded from a first length to a longer length, the positional encoding of RoPE can be adjusted to ensure that the model can correctly understand the expanded context, thereby improving the model's performance when processing long sequences and avoiding information loss and a decline in context understanding.

[0039] As an example, RoPE dimensions can refer to the various vector dimensions used to represent positional information in RoPE encoding. The positional encoding in each dimension can be calculated using specific frequencies and phases. The frequency differences across different dimensions make the positional encoding non-uniform, enabling the model to distinguish different positions in the sequence.

[0040] As an example, RoPE's positional encoding is generated by applying rotation operations in each dimension, with the rotation angle and frequency parameter in each dimension jointly determining the specific form of the positional encoding. For instance, for each position i in the sequence, its positional encoding value P(i,j) in dimension j can be expressed as follows:

[0041] P(i,j)=A_j*sin(w_j,j)+B_j*cos(w_j,i)

[0042] Where i is the location index, j is the RoPE dimension, A_j and B_j are matrices related to the RoPE dimension j, and w_j is the frequency parameter in this dimension, which determines the rotation angle, i.e., the rate of change of location information in different dimensions.

[0043] In one exemplary embodiment, before obtaining the position encoding of the position rotation embedding RoPE of the pre-trained model at the first length, the different RoPE dimensions, and the extended length of the context window for a first length of the pre-trained model, the method further includes:

[0044] A first text dataset of a preset length is obtained, and the first text dataset is cleaned and segmented to obtain a set of second text datasets with different text lengths; wherein, the maximum text length of the text data in the second text dataset is less than or equal to the preset target length.

[0045] Based on the second text dataset with different text lengths, the preset initial model is fine-tuned to obtain the fine-tuned pre-trained model.

[0046] For example, before expanding the context window, text data for expanding the context window can be obtained and processed. Based on the processed text data of different lengths, progressive local fine-tuning can be performed on the respective models. In this embodiment, the context window can be expanded based on the fine-tuned model.

[0047] As an example, a first text dataset of a preset length can be collected from various data sources (such as the internet, databases, etc.). This preset text dataset can contain text data of different lengths, such as long text data and short text data. The choice of the preset length can be based on the specific application scenario and the size of the target context window.

[0048] As an example, data cleaning refers to removing noise, irrelevant data, and formatting issues from a dataset, such as removing HTML tags, correcting spelling errors, and removing stop words, to ensure the purity and consistency of text data and improve data quality.

[0049] As an example, data segmentation can refer to using a preset segmentation strategy to divide the collected text dataset into smaller segments, ensuring that the length of each segment is less than or equal to the maximum length of the model's fixed context window, i.e., the first length, so as to ensure that the length of the text that the model to be trained can process does not exceed the limit of its context window, which facilitates the initial training of the model to be trained.

[0050] As an example, a segmentation strategy can be used to divide the cleaned first text dataset into a second text dataset. The data segmentation strategy needs to ensure that the length of each segmented text data segment is less than or equal to the preset target length, and can cover a variety of text lengths from short to long texts to meet the training needs of different context windows; the data segmentation strategy also needs to ensure that the segmented contexts are consistent and coherent.

[0051] Methods for maintaining contextual consistency and coherence may include, but are not limited to, context overlap (such as using a sliding window overlap segmentation method to ensure that contextual information is shared between adjacent segments), context completion (such as providing the model with certain contextual information in a segmentation task scenario), and segmented connection training (such as connecting adjacent paragraphs during model training to enable the model to learn to handle cross-paragraph context).

[0052] As an example, data segmentation can be performed using different strategies, such as based on sentence boundaries, paragraph boundaries, or fixed-length windows. In long text processing, a fixed-length strategy is commonly used, such as each segment having a length of 1024 or 2048 tokens. This helps maintain the consistency of the context window and the training efficiency of the model.

[0053] As an example, the initial model can be a model pre-trained on a standard dataset with a short context window (i.e., a context window shorter than the first length). This standard dataset can include short text paragraphs (e.g., 4K or 8K context lengths) to help the model understand basic natural language processing tasks such as machine translation and text classification.

[0054] In one exemplary embodiment, the step of fine-tuning the preset initial model based on the second text dataset with different text lengths to obtain the fine-tuned pre-trained model includes:

[0055] The text data in the second text dataset with different text lengths are sorted in ascending order of text length;

[0056] Based on the sorted second text dataset, the preset initial model is trained in a context window of initial length to obtain the trained pre-trained model; wherein the initial length is less than the first length.

[0057] As an example, the text data in the second text dataset is sorted by its length, for example, starting with shorter text data and gradually transitioning to text data of a preset maximum length. The sorted text data can provide an ordered sequence of training data for expanding the context window of the model training, enabling the model's context window to expand progressively.

[0058] As an example, a pre-defined initial model can be trained based on a sorted second text dataset, with a context window of an initial length. The initial length is less than the first length and represents the shortest context window length the model can handle. This training process can employ a segmented strategy, initially training with shorter text data and gradually introducing longer text data as training progresses until the maximum pre-defined length is reached, thus progressively expanding the model's context window capabilities.

[0059] As an example, the initial model training can follow the standard training process in deep learning frameworks, including but not limited to forward propagation, calculation of loss functions (such as cross-entropy, mean squared error, etc.), and backpropagation to adjust model parameters until the model reaches the expected performance metrics within an initial length of context window.

[0060] Through the above training process, a pre-trained model is finally obtained. The pre-trained model exhibits good performance with an initial context window length and provides a solid foundation for further context window expansion (to the initial length or longer).

[0061] Step S202: Based on the different RoPE dimensions, the preset evolutionary search algorithm, and the expansion length of the context window, determine the target scaling factor for each RoPE dimension corresponding to the expansion length; wherein the expansion length is greater than the first length.

[0062] In this embodiment, an evolutionary search algorithm can be used to search for a set of optimal target scaling factors. Each target scaling factor can correspond to a RoPE dimension, which can adapt to a specific context window expansion length, significantly improving the model's ability to process long texts while maintaining performance under short context windows. This process requires no human intervention and can automatically handle parameter optimization problems, improving the efficiency of model training and tuning.

[0063] In one exemplary embodiment, determining the target scaling factor for each RoPE dimension corresponding to the extended length based on the different RoPE dimensions, a preset evolutionary search algorithm, and the extended length of the context window includes:

[0064] Obtain the scaling factor population for each RoPE dimension with different RoPE dimensions; each individual in the scaling factor population indicates the scaling factor corresponding to the extended length;

[0065] For each scaling factor population of the RoPE dimension, a first scaling factor is selected from the scaling factor population, and the position code of the RoPE is subjected to non-uniform interpolation based on the first scaling factor to obtain the interpolated position code of the RoPE.

[0066] The fitness of the first scaling factor is calculated based on the preset fitness function, the preset fitness threshold, the position code of the RoPE at the first length, and the position code of the RoPE after non-uniform interpolation.

[0067] If the fitness of the first scaling factor is less than or equal to the preset fitness threshold, the scaling factor is reselected until the fitness of the first scaling factor is greater than the preset fitness threshold. The first scaling factor with a fitness greater than the preset fitness threshold is then determined as the target scaling factor for the RoPE dimension.

[0068] The first scaling factor in this application embodiment can be any scaling factor in any population. When the fitness of the first scaling factor is less than or equal to a preset fitness threshold, the first scaling factor can be reselected until the selected first scaling factor is greater than the preset fitness threshold.

[0069] As an example, when running the evolutionary search algorithm for the first time, a scaling factor population can be initialized for each RoPE dimension. Each individual in the population can represent a scaling factor value for that RoPE dimension, which is used in subsequent optimization processes. The scaling factor population can be generated randomly or set based on prior knowledge.

[0070] As an example, an individual can be selected from the scaling factor population, namely the first scaling factor, to evaluate its performance within an extended context window. The positional encoding of RoPE can then be non-uniformly interpolated based on the first scaling factor to fit the length of the target context window. Non-uniform interpolation preserves the relative relationships of each dimension at different locations while avoiding information loss or invalidation of the positional encoding.

[0071] As an example, a fitness function can be predefined to evaluate the performance of the first scaling factor. The fitness function can measure the impact of the scaling factor on model performance based on the model's performance on a specific task, such as loss function, accuracy, BLEU score, etc.

[0072] As an example, a fitness threshold can be preset as the optimization objective. The fitness of the scaling factor should be higher than this threshold to indicate that the scaling factor can effectively improve or maintain the model's performance within the target context length.

[0073] As an example, the fitness of the first scaling factor can be calculated and iterated based on the preset fitness function, the preset fitness threshold, the position code of the RoPE under the first length and the position code of the RoPE after non-uniform interpolation. If the fitness of the first scaling factor is greater than the preset fitness threshold, the iterative operation is stopped and the first scaling factor greater than the preset fitness threshold is determined as the target scaling factor of the RoPE dimension.

[0074] In one exemplary embodiment, obtaining the scaling factor population for each RoPE dimension of different RoPE dimensions includes:

[0075] Based on the preset scaling factor search range and the preset scaling factor population size, a scaling factor population corresponding to each RoPE dimension is generated for each of the different RoPE dimensions.

[0076] In this embodiment of the application, in order to find the optimal RoPE scaling factor to adapt to the expansion of the context window, it is necessary to generate a scaling factor population for each RoPE dimension.

[0077] The preset scaling factor search range can refer to the value range of the scaling factor. For example, the scaling factor search range can be preset to [0.8, 1.2], and the scaling factor can be freely varied within this range to adapt to the expansion of the context window.

[0078] The preset scaling factor is the population size, which determines the number of individuals at the start of the algorithm. For example, the population size can be set to 50-100 individuals, providing sufficient diversity for the evolutionary search algorithm and avoiding local optima.

[0079] As an example, after determining the search range and population size, a scaling factor population can be generated based on the scaling factor of randomly initialized individuals.

[0080] Specifically, when generating the scaling factor population, a separate population can be generated for each RoPE dimension in the model. Since the frequency and location information encoding methods differ for each RoPE dimension, their adaptability to context window expansion also varies, allowing for separate optimization.

[0081] For example, evolutionary search algorithms may include the following settings:

[0082] 1) Initial parameter settings:

[0083] The initial parameters of the search algorithm include population, population size, number of iterations, selection ratio, crossover probability, and mutation probability. The population can be initialized as a set of random parameters or as a rescaling factor trained based on the initial window.

[0084] For example, the population size can be selected as 50-100 individuals, and the specific size can be determined according to the actual computing resources. This application does not impose any restrictions on this embodiment.

[0085] The number of iterations can be set to 100-200 generations, and can be adjusted in practical applications according to the complexity of the problem and the convergence of the model.

[0086] In each generation, 20%-30% of the best individuals can be selected to enter the next round of iterative search. This can be done using tournament selection or roulette selection.

[0087] The crossover probability can be set to 0.6-0.8, indicating that most individuals in the population generate new individuals through crossover.

[0088] The mutation probability can be set to 0.01-0.05. The mutation operation is used to introduce randomness and prevent the algorithm from getting stuck in local optima.

[0089] 2) Search range settings:

[0090] Each individual in the population corresponds to a set of frequency values ​​(w_j) obtained from a short context window to a scaling factor (s_j) after the window is expanded. The search range can be within the scaling factor range of [0.8, 1.2] to avoid scaling too large or too small.

[0091] 3) Convergence condition settings:

[0092] A convergence function can be preset as a convergence condition. The mean squared error (MSE) between the RoPE encoding after non-uniform interpolation and the RoPE encoding of the baseline model before interpolation can be used. The convergence function can be expressed as:

[0093]

[0094] Where i is the location index, j is the RoPE dimension, P(i,j) is the RoPE value of the baseline model, and P^(i,j) is the interpolated RoPE value. The baseline model can be a pre-trained model without an expanded context window.

[0095] The algorithm converges when the change of the preset convergence function over multiple iterations is less than the preset convergence threshold. For example, the preset convergence threshold can be set to ε = 10. -5 .

[0096] 4) Fitness function settings:

[0097] The fitness function is used for fitness evaluation and can be set based on the value of the convergence function. For example, a mean squared error (MSE) fitness evaluation method can be used, which compares the difference between the RoPE code after non-uniform interpolation in each RoPE dimension under an extended context window and the baseline RoPE code before interpolation. The smaller the error, the higher the fitness of the individual. The fitness in the evolutionary search process can be calculated according to the following formula:

[0098]

[0099] Here, 'a' is a weighting coefficient used to balance the impact of loss and accuracy on fitness. The larger the fitness value, the closer the scaling factor corresponding to that individual is to the optimal solution, i.e., the target scaling factor.

[0100] As an example, based on the above settings, RoPE positional encoding with non-uniform interpolation can be used to evaluate the model's performance under the first-length context window and the target context window length, and the fitness value of the first scaling factor can be calculated. By comparing the performance before and after model scaling, the optimization effect of the scaling factor can be determined.

[0101] If the fitness of the first scaling factor is less than or equal to the preset fitness threshold, i.e. the fitness is not up to standard, it means that the performance of the scaling factor under the target context length has not met expectations. Then, an individual can be reselected from the population as the scaling factor (i.e., a new first scaling factor is selected) for iterative calculation. That is, the process of selecting a new first scaling factor, non-uniform interpolation, fitness calculation and evaluation is repeated until the fitness of the reselected first scaling factor is greater than the preset fitness threshold. The iterative calculation stops, and the first scaling factor with a fitness greater than the preset fitness threshold is taken as the target scaling factor for this RoPE dimension.

[0102] The reselected scaling factor can be generated through population evolution (i.e., crossover and mutation) to produce new scaling factor individuals.

[0103] As an example, a single-point or multi-point crossover strategy can be used to select one or more dimensions between two scaling factor individuals to exchange scaling factors and generate new scaling factor individuals.

[0104] As an example, the scaling factor of any dimension of a random individual can be multiplied by a random number within a small range (such as [0.9, 1.1]) to introduce diversity in the scaling factor individuals.

[0105] The target scaling factor obtained in this application embodiment can be used to adjust the RoPE position encoding to adapt to the expanded context window length, ensuring the performance and efficiency of the model when processing longer text.

[0106] Step S203: Based on the target scaling factor, perform non-uniform interpolation processing on the position encoding of the RoPE under the first length to obtain the extended length context window of the pre-trained model.

[0107] The embodiments of this application can adjust the positional encoding of RoPE based on the target scaling factor, i.e., non-uniform interpolation processing, so that the positional encoding capability of the pre-trained model is extended to a new context window length, i.e., an extended-length context window is obtained.

[0108] In one exemplary embodiment, after performing non-uniform interpolation processing on the position encoding of the RoPE, the method further includes:

[0109] Obtain the text dataset of the extended length corresponding to the extended length context window;

[0110] Based on the extended-length text dataset, the pre-trained model after non-uniform interpolation is fine-tuned.

[0111] As an example, after performing non-uniform interpolation on the positional encoding of RoPE, the pre-trained model can be globally fine-tuned based on an extended-length text dataset to ensure the continuity of the context window.

[0112] In one exemplary embodiment, determining the target scaling factor for each RoPE dimension corresponding to the extended length based on the different RoPE dimensions, a preset evolutionary search algorithm, and the extended length of the context window further includes:

[0113] In response to the extended length of the context window being less than the preset target length, the position rotation embedding RoPE position code under the extended length is determined as the new RoPE position code under the first length;

[0114] Obtain the different RoPE dimensions under the new first length and the new extended length of the context window; wherein the new extended length is greater than the new first length;

[0115] Based on the different RoPE dimensions under the new first length, the preset evolutionary search algorithm, and the new extended length of the context window, determine the target scaling factor for each RoPE dimension corresponding to the new extended length.

[0116] As an example, during the expansion of the context window of a pre-trained model, if the length of the expanded context window (e.g., 512K) has not yet reached the preset target length (e.g., 2048K), the current expansion length (e.g., 512K) can be used as the new first length, that is, 512K becomes the new context window length, and different RoPE dimensions under the new first length and the new expansion length (e.g., 1024K) can be obtained.

[0117] The new first-length RoPE dimension and the new extended length can be used in the next stage of context window expansion to specifically optimize RoPE embedding and ensure that the model can smoothly transition to longer context window processing capabilities.

[0118] As an example, the target scaling factor for each RoPE dimension corresponding to the new extended length is determined based on the different RoPE dimensions under the new first length, the preset evolutionary search algorithm, and the new extended length of the context window.

[0119] As an example, the positional encoding of RoPE under the new first length can be non-uniformly interpolated based on the target scaling factor of each RoPE dimension corresponding to the new extended length, to obtain the context window of the new extended length of the pre-trained model.

[0120] As an example, it can be determined whether the length of the expanded context window has reached the target length. If not, the current expanded length can be set as the new first length, and the above process can continue to be executed until the length of the expanded context window reaches the target length, thereby realizing the phased and gradual expansion of the length of the context window.

[0121] In one exemplary embodiment, after obtaining the extended length context window of the pre-trained model, the method further includes:

[0122] In response to a model inference request, a third text dataset with a text length greater than the first length and a validation set corresponding to the third text dataset are obtained.

[0123] The third text dataset is used as the input to the pre-trained model, and the third text dataset is processed through the extended-length context window to obtain the processed fourth text dataset output by the pre-trained model.

[0124] Calculate the performance metrics of the pre-trained model based on the fourth text dataset and the validation set;

[0125] Based on the performance metrics and preset performance metric thresholds, adjust the target scaling factor or fine-tune the pre-trained model.

[0126] As an example, a third text dataset with a length greater than the first dataset and a corresponding validation set can be used to perform model inference on the expanded model. The performance of the pre-trained model can then be monitored based on the inference results. The model's performance metrics include, but are not limited to, loss functions (such as cross-entropy) and accuracy.

[0127] The following example will further illustrate the model inference process and performance monitoring process of a pre-trained model.

[0128] For example, the extended model can be applied to various long text processing tasks, such as document summarization, machine translation, or sentiment analysis of long articles. The following are the specific steps for inference using the extended model:

[0129] Step 1: Input long text. The user inputs text that exceeds the length of the traditional context window (e.g., a document of 500K in length) into the model, and the model processes the input text through an expanded context window (e.g., 2048K).

[0130] Step 2: Segmentation. For extremely long texts (i.e., the third text dataset), the model segments the text according to a defined context window (e.g., 2048K). The positional information of each segment is encoded using RoPE interpolation to ensure that the model can effectively utilize the contextual information in the long text.

[0131] Step 3: Inference computation. The model performs forward propagation on each text segment, generating the processing result for each text segment. For example, in a machine translation task, it generates the target language text corresponding to each text segment.

[0132] Step 4: Result Combination. The model combines the processing results of each text segment to generate the final result for the entire long text. For example, in tasks such as summarizing or translating, it combines global information to generate the final translation result.

[0133] Step 5: Result Validation. To ensure the accuracy of the inference results, the results are compared using a validation set or reference text corresponding to the input long text. The comparison can be based on performance metrics, which may include, but are not limited to, loss functions (such as cross-entropy) and accuracy. For example, a validation set can be used to verify the model's inference performance, and performance metrics such as loss functions (such as cross-entropy) and accuracy can be combined to test the model's accuracy in long text processing. For instance, in machine translation tasks, the results can be compared with those of human translation using metrics such as BLEU and ROUGE.

[0134] As an example, when the model's performance metrics in the expanded context window task drop beyond a preset performance metric threshold, the target scaling factor can be adjusted or the pre-trained model can be fine-tuned.

[0135] The performance indicators can be set based on the following standards:

[0136] Loss function: The loss value (such as cross-entropy or mean squared error) in text processing exceeds a preset loss threshold. For example, if the loss value increases by more than 5%-10%, it triggers an adjustment of the scaling factor or fine-tuning of the pre-trained model.

[0137] Accuracy or other performance metrics: When the accuracy, F1 score, or BLEU score in a text task drops by more than a certain percentage, the scaling factor is adjusted to fine-tune the pre-trained model.

[0138] In this embodiment, for a context window of a first length of the pre-trained model, the positional encoding of the position rotation embedding RoPE under the first length, different RoPE dimensions, and the extended length of the context window are obtained. Based on different RoPE dimensions, a preset evolutionary search algorithm, and the extended length of the context window, a target scaling factor for each RoPE dimension corresponding to the extended length is determined; wherein the extended length is greater than the first length. Based on the target scaling factor, non-uniform interpolation processing is performed on the positional encoding of the RoPE under the first length to obtain the extended length context window of the pre-trained model. This solves the problem in related technologies where models cannot consider contextual information beyond the window range when processing long texts, resulting in poor performance in long-distance dependencies and global consistency. It achieves a breakthrough expansion of the context window for large language models, not only improving the model's performance in long text processing but also saving computational resources and simplifying the implementation process.

[0139] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0140] This embodiment also provides an extended system architecture block diagram of a model context window, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0141] Figure 3 This is a system architecture block diagram of the model context window extension according to the embodiments of this application. The embodiments of this application can run on... Figure 3 In the system architecture shown, such as Figure 3 As shown, the system includes: a data preprocessing module 301, a model training module 302, a RoPE interpolation module 303, an evolutionary search module 304, a model inference module 305, and a performance monitoring module 306. The functions and interaction relationships of each module are as follows:

[0142] The data preprocessing module 301 can be used to collect, clean, and preprocess long text data to provide training and fine-tuning data. The data preprocessing module 301 may include at least one of the following: a data collector, a data cleaner, a data segmenter, etc. Specifically, the data collector can be used to acquire long text data from various data sources; the data cleaner can be used to remove noise and irrelevant data from the acquired long text data; and the data segmenter can be used to divide the long text data into multiple short texts or text segments for model training.

[0143] The model training module 302 can be used to perform initial training of the model and fine-tuning operations on the expanded context window to obtain a pre-trained model. The model training module 302 may include, but is not limited to, an initial trainer and a fine-tuner. The initial trainer can be used to train the initial model on the original short context window to obtain a pre-trained model, and the fine-tuner can be used to fine-tune the pre-trained model on the expanded context window.

[0144] The RoPE interpolation module 303 can be used for non-uniform interpolation of RoPE position embedding, calculating and applying an optimized rotation angle scaling factor. The RoPE interpolation module 303 may include, but is not limited to, an interpolation calculation unit and an interpolation application unit. The interpolation calculation unit can be used to calculate the scaling factor for non-uniform interpolation, and the interpolation application unit can be used to apply the calculated scaling factor to the RoPE position embedding.

[0145] The evolutionary search module 304 can be used to execute an evolutionary search algorithm to optimize the scaling factor of RoPE interpolation and achieve the best performance. The evolutionary search module 304 may include, but is not limited to, a population initialization unit, a fitness evaluation unit, and mutation and crossover units. Specifically, the population initialization unit can be used to generate the scaling factor population corresponding to each RoPE dimension; the fitness evaluation unit can be used to evaluate the fitness of each scaling factor individual; and the mutation and crossover unit can be used to perform mutation and crossover operations to generate a new population.

[0146] The model inference module 305 can be used to apply the extended model for long text processing and analysis during the model inference stage. The model inference module 305 may include, but is not limited to, an inference engine and a request processing unit. The inference engine can be used to process long text using the extended model; the request processing unit can be used to receive and return inference requests from the user.

[0147] The performance monitoring module 306 can be used to monitor model performance and perform fine-tuning of context window restoration or reselection of scaling factors to maintain short text performance. The performance monitoring module 306 may include, but is not limited to, a performance monitor and an adjustment unit. The performance monitor can be used to monitor model performance in real time, and the adjustment unit can be used to adjust model parameters based on the monitoring results.

[0148] Based on the above system, the pre-trained LLM context window can be significantly expanded from the traditional 4096 tokens to 2048k tokens, greatly improving the model's ability to handle long text tasks, such as long document understanding, text generation, and complex question answering. The RoPE position embedding is optimized through non-uniform interpolation and evolutionary search, maintaining the model's high performance after expanding the context window. Furthermore, the expanded model retains the original architecture and only makes minor modifications to the position embedding, making it compatible with most existing optimization methods.

[0149] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0150] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to perform the steps in any of the above method embodiments when it is run.

[0151] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0152] Figure 4 This is a structural block diagram of an electronic device according to an embodiment of this application, such as... Figure 4 As shown, embodiments of this application also provide an electronic device 40, including a memory 401 and a processor 402, wherein the memory 401 stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0153] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0154] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0155] Embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0156] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0157] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A method for extending a model context window, characterized in that, include: For a context window of a first length for a pre-trained model, obtain the position encoding of the position rotation embedding RoPE of the pre-trained model under the first length, the different RoPE dimensions, and the extended length of the context window; Based on the different RoPE dimensions, the preset evolutionary search algorithm, and the expansion length of the context window, the target scaling factor for each RoPE dimension corresponding to the expansion length is determined; wherein, the expansion length is greater than the first length; Based on the target scaling factor, the positional encoding of the RoPE at the first length is subjected to non-uniform interpolation to obtain the extended length context window of the pre-trained model.

2. The method according to claim 1, characterized in that, After performing non-uniform interpolation processing on the position encoding of the RoPE, the method further includes: Obtain the text dataset of the extended length corresponding to the extended length context window; Based on the extended-length text dataset, the pre-trained model after non-uniform interpolation is fine-tuned.

3. The method according to claim 1, characterized in that, The step of determining the target scaling factor for each RoPE dimension corresponding to the expansion length based on the different RoPE dimensions, the preset evolutionary search algorithm, and the expansion length of the context window further includes: In response to the fact that the extended length of the context window is less than the preset target length, the extended length is determined as the new first length; Obtain the different RoPE dimensions under the new first length and the new extended length of the context window; wherein the new extended length is greater than the new first length; Based on the different RoPE dimensions under the new first length, the preset evolutionary search algorithm, and the new extended length of the context window, determine the target scaling factor for each RoPE dimension corresponding to the new extended length.

4. The method according to claim 1, characterized in that, Before obtaining the position encoding of the position rotation embedding RoPE of the pre-trained model at the first length, the different RoPE dimensions, and the extended length of the context window for the first length of the pre-trained model, the method further includes: A first text dataset of a preset length is obtained, and the first text dataset is cleaned and segmented to obtain a set of second text datasets with different text lengths; wherein, the maximum text length of the text data in the second text dataset is less than or equal to the preset target length. Based on the second text dataset with different text lengths, the preset initial model is fine-tuned to obtain the fine-tuned pre-trained model.

5. The method according to claim 4, characterized in that, The pre-trained model is fine-tuned based on the second text dataset with different text lengths to obtain the fine-tuned model, including: The text data in the second text dataset with different text lengths are sorted in ascending order of text length; Based on the sorted second text dataset, the preset initial model is trained in a context window of initial length to obtain the trained pre-trained model; wherein the initial length is less than the first length.

6. The method according to claim 1, characterized in that, The step of determining the target scaling factor for each RoPE dimension corresponding to the expansion length based on the different RoPE dimensions, the preset evolutionary search algorithm, and the expansion length of the context window includes: Obtain the scaling factor population for each RoPE dimension with different RoPE dimensions; each individual in the scaling factor population indicates the scaling factor corresponding to the extended length; For each scaling factor population of the RoPE dimension, a first scaling factor is selected from the scaling factor population, and the position code of the RoPE is subjected to non-uniform interpolation based on the first scaling factor to obtain the interpolated position code of the RoPE. The fitness of the first scaling factor is calculated based on the preset fitness function, the preset fitness threshold, the position code of the RoPE at the first length, and the position code of the RoPE after non-uniform interpolation. If the fitness of the first scaling factor is less than or equal to the preset fitness threshold, the scaling factor is reselected until the fitness of the first scaling factor is greater than the preset fitness threshold. The first scaling factor with a fitness greater than the preset fitness threshold is then determined as the target scaling factor for the RoPE dimension.

7. The method according to claim 6, characterized in that, The process of obtaining the scaling factor population for each RoPE dimension across different RoPE dimensions includes: Based on the preset scaling factor search range and the preset scaling factor population size, a scaling factor population corresponding to each RoPE dimension is generated for each of the different RoPE dimensions.

8. The method according to claim 1, characterized in that, After obtaining the extended length context window of the pre-trained model, the method further includes: In response to a model inference request, a third text dataset with a text length greater than the first length and a validation set corresponding to the third text dataset are obtained. The third text dataset is used as the input to the pre-trained model, and the third text dataset is processed through the extended-length context window to obtain the processed fourth text dataset output by the pre-trained model. Calculate the performance metrics of the pre-trained model based on the fourth text dataset and the validation set; Based on the performance metrics and preset performance metric thresholds, adjust the target scaling factor or fine-tune the pre-trained model.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the method described in any one of claims 1 to 8.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 8.

11. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1 to 8.

Images