Method for expanding size of context window of large language model
The scaling factor of rotation position encoding is optimized through the partition and governance incremental search algorithm, and the problems of performance loss and resource consumption are solved in the expansion of the context window size of the large language model, achieving better performance and resource saving effects.
Patent Information
- Application Number
- CN202411015102.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-12-03
AI Technical Summary
When the prior art expands the context window size of a large language model, there are problems of performance losses and fine-tuning resources consumption, especially when short text fine-tuning generalizes to long text, the performance losses are serious.
The partition and governance incremental search algorithm is used to find the optimal scaling factor for rotational position coding, and the scaling factor sequence is optimized through the partition and governance strategy and incremental search strategy, reducing resource consumption and improving performance.
It achieves better performance and resource savings, solves the problem of performance loss in generalizing short text fine-tuning to long text, and significantly reduces fine-tuning costs.
Smart Images

Figure CN119067167A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of large language models, and in particular to a method for expanding the size of a context window of a large language model. Background Art
[0002] With the rise of large language models based on the Transformer architecture, people have gradually felt that the short text (4k tokens) sequence processing of pre-trained models is insufficient, and then look for ways to continue to obtain a model that can process long text (32k / 64k tokens and above) sequences on this basis. The current mainstream methods mainly focus on the optimization of position encoding, segmented recursive memory, etc. Specifically, by optimizing the relative position encoding, such as the rotation position encoding, the position encoding of the long text sequence is compressed to the original short text length of the model by increasing the rotation frequency, and then through a small amount of fine-tuning, the context window length of the model is expanded at a very low cost. Furthermore, for different dimensions of model embedding, different scaling factors are given to obtain different rotation frequencies, which can more efficiently expand the context window length of the model.
[0003] Although the current optimization for rotation position encoding has basically met the needs of context window expansion, there is still room for further optimization. Compared with the manually specified scaling factor, the scaling factor selected by the model is not only better in performance, but also requires less fine-tuning resources. However, the current method that allows the model to select the scaling factor autonomously requires relatively large resources in the algorithm for finding the optimal scaling factor. Summary of the invention
[0004] In view of the technical problems pointed out in the background technology, the purpose of the present invention is to provide a method for expanding the size of a large language model context window.
[0005] To achieve the purpose of the present invention, the technical solution provided by the present invention is specifically as follows:
[0006] A method for expanding the context window size of a large language model, wherein the method is to find an optimal scaling factor of a rotational position encoding through a divide-and-conquer incremental search algorithm to expand the context window size of the large language model.
[0007] The method adopts a divide-and-conquer strategy for the scaling factor sequence, where the sequence is divided into several segments in each iteration, and only one segment is focused on each time the processing is performed while the other segments are fixed; for the processing of each segment, an incremental search strategy is adopted, that is, the segment of the current optimal scaling factor sequence is increased by a value to obtain a new scaling factor sequence, and the increased value is a number of values uniformly taken from the value range of the segment; then the optimal increment is determined through PPL to update the scaling factor sequence, and the value range is narrowed as the value range of the two sub-segments of the next iteration.
[0008] Compared with the prior art, the present invention has the following beneficial effects:
[0009] Compared with the method of manually specifying the scaling factor (YaRN, etc.), the solution of the present invention not only has better performance, but also reduces the resources required for fine-tuning, allowing the model to be fine-tuned on short texts and generalized to target long texts, greatly saving costs. At the same time, it solves the problem that most current methods (CLEX, etc.) have in generalizing short text fine-tuning to long texts, that is, the target length performance is seriously lost. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] Figure 1 A schematic diagram of a method flow chart provided in an embodiment of the present application;
[0011] Figure 2 Schematic diagram of PPL comparison on the Proof-pile dataset, in which the methods provided in the embodiments of the present application extend Llama2-7B to 64k context window length. DETAILED DESCRIPTION
[0012] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0013] The present invention uses a Divide-and-Conquer Incremental Search (DCIS) algorithm to find the optimal scaling factor of the rotational position encoding to expand the context window size of the large language model.
[0014] First, the rotational position is encoded (RotaryPosition Embedding, RoPE) In short:
[0015]
[0016] Where d is the number of dimensions of the hidden state of each head, m represents the position index, and β i =10000 2i / d ,λ i is the scaling factor for the frequency of dimension i.
[0017] From the above formula, we can see that the scaling factor sequence to be searched is [λ0,λ1,…,λ d / 2-1 ].
[0018] A divide-and-conquer strategy is adopted for the entire scaling factor sequence. Specifically, each iteration divides the sequence into several segments, and only one segment (i.e. Figure 1The red font in the middle) is used to fix the other segments. For the processing of each segment, an incremental search strategy is adopted, that is, the segment of the current optimal scaling factor sequence is increased by one value to obtain a new scaling factor sequence, and the increased value is a number of values uniformly taken from the value range of the segment. Subsequently, the optimal increment is determined by PPL to update the scaling factor sequence, and the value range is narrowed as the value range of the two sub-segments of the next iteration.
[0019] like Figure 1 The case where d=16 and three values are evaluated per segment, with the scaling factor being optimized stepwise, is shown.
[0020] Algorithm 1 gives the corresponding divide-and-conquer incremental search algorithm.
[0021]
[0022] like Figure 1 As shown in Algorithm 1, since the scaling factor is divided into high frequency and low frequency, the initial processing is divided into two sections, each section processes N = d / 2 scaling factors. In each iteration, the current segment to be processed Seg is first obtained through the "Get Processed Segment" function:
[0023] Seg=[(λ i ,λ i+1 ,…,λ i+N-1 ), where i = dN × j, j∈[1,d / N]].
[0024] Then, the "get incremental value" function is used to evenly obtain C incremental values from the value range of the current segment:
[0025] Values(v k )=[R j,l +step×k], where k∈[0,C-1].
[0026] Use the "Calculate PPL" function to calculate PPLs by adding the current scaling factor to each incremental value to get the new scaling factor:
[0027] PPLs(p k )=Calculate PPL(Seg+v k ).
[0028] Finally, PPLs are used to "update" the scaling factor and the range of the next iteration. Specifically, the scaling factor is updated using the smallest increment of PPL, and the smallest C / 3 increments of PPL are used as the range of the next iteration:
[0029] PPLs,Valuse=rank((p k ,v k )),
[0030] F Seg =Seg+v1,
[0031] Where l = minimum value (v1, v2, ..., vC / 3 ),r=maximum(v1,v2,…,v C / 3 ).
[0032] This process continues until the scaling factor of the last segment is divided and returned as the result of this search.
[0033] At the same time, the following four optimizations are added based on experience:
[0034] 1. Since the scaling factor of the manually specified method YaRN is better, it is used as the initial scaling factor to start the search;
[0035] 2. From NTK-aware, we know that low frequency needs more interpolation, so its scaling factor is larger, and the fluctuation will approach the optimal scaling factor faster. Therefore, when processing each layer, we first process the high-dimensional low-frequency ones, and then process the low-dimensional high-frequency ones after updating;
[0036] 3. For PPL greater than 100, this application believes that it has no guiding significance, therefore, this application abandons the incremental value of PPL greater than 100;
[0037] 4. In order to prevent falling into the local optimal solution, when updating the value range of the next layer, in addition to using the optimal increment values of the three PPLs, the upper and lower bounds are expanded outward by one step respectively, where one step is the difference between two adjacent increment values.
[0038] Compared with the method of manually specifying the scaling factor (YaRN, etc.), this algorithm not only has better performance, but also reduces the resources required for fine-tuning, allowing the model to be fine-tuned on short texts and generalized to target long texts, greatly saving costs. At the same time, it solves the problem that most current methods (CLEX, etc.) have in generalizing short text fine-tuning to long texts, that is, the target length performance is seriously lost. Figure 2 As shown in the figure, the smaller the PPL result, the better the model performance. “-16k / -4k” means fine-tuning and generalizing on short text to the target length of 64k.
[0039] At the same time, compared with LongRoPE, a method that also relies on evolutionary search to find the optimal scaling factor, in finding the optimal scaling factor, taking Llama2-7B and default parameters as an example, the time complexity of the algorithm of the present invention ((d-2)×C=(128-2)×10=1260) is reduced to half of that of evolutionary search (T×P=40×64=2560).
[0040] The optional implementation modes of the embodiments of the present invention are described in detail above in conjunction with the accompanying drawings. However, the embodiments of the present invention are not limited to the specific details in the above implementation modes. Within the technical concept of the embodiments of the present invention, various simple modifications can be made to the technical scheme of the embodiments of the present invention, and these simple modifications all belong to the protection scope of the embodiments of the present invention.
Claims
1. A method for expanding the context window size of a large language model, characterized in that: The method is to find the optimal scaling factor of the rotational position encoding through a divide-and-conquer incremental search algorithm to expand the context window size of a large language model.
2. A method for expanding the size of a large language model context window according to claim 1, characterized in that: The method adopts a divide-and-conquer strategy for the scaling factor sequence, where the sequence is divided into several segments in each iteration, and only one segment is focused on each time the processing is performed while the other segments are fixed; for the processing of each segment, an incremental search strategy is adopted, that is, a value is increased by one value for the segment of the current optimal scaling factor sequence to obtain a new scaling factor sequence, and the increased value is a number of values uniformly taken from the value range of the segment; then the optimal increment is determined by PPL to update the scaling factor sequence, and the value range is narrowed as the value range of two sub-segments for the next iteration.
3. A method for expanding the size of a large language model context window according to claim 2, characterized in that: The search starts with the scaling factor of the manually specified method YaRN as the initial scaling factor.
4. The method for expanding the context window size of a large language model according to claim 2, characterized in that: When processing each layer, high-dimensional and low-frequency ones are processed first, and then low-dimensional and high-frequency ones are processed after updating.
5. The method for expanding the context window size of a large language model according to claim 2, characterized in that: Discard incremental values of PPL greater than 100.
6. The method for expanding the context window size of a large language model according to claim 2, characterized in that: When updating the value range of the next layer, in addition to using the optimal PPL increment value, the upper and lower bounds are expanded outward by one step, where one step is the difference between two adjacent increment values.
