Zero-training adaptive expansion relative position coding method based on RoPE

Abstract

The invention relates to the technical field of large language models, in particular to a zero-training self-adaptive expansion relative position coding method based on RoPE. The method comprises the following steps: acquiring the length of a token to be coded; determining a range of relative position coding and an initial length ratio, calculating a product of an upper limit value of the range of relative position coding and the initial length ratio, and determining an initial position and an end position for a block coding region according to the product; and on the basis of the length of the token to be coded, according to the initial position and the end position, determining the multiplexing times and the extended coding position of the relative position coding in the block coding area, and carrying out adaptive extended coding according to the multiplexing times and the extended coding position. Therefore, the problems that in the prior art, time and resources are consumed, due to scarcity of high-quality long text data, the effectiveness of a fine adjustment method is hindered, and the generalization ability of LLMs to wide tasks is reduced are solved, and the long text processing ability and effect of the model are improved.

Description

Technical Field

[0001] The present application relates to the technical field of large language models, and in particular to a RoPE-based zero-training adaptive extended relative position encoding method. Background Art

[0002] The context window length of existing large-scale language models (LLMs) is usually limited because they are trained on fixed-length training sequences. The context window length in the training phase determines the context window length in the inference phase of the model. Once the input text length exceeds the pre-trained context window length, the performance of LLMs will become unpredictable and will suffer severe performance degradation. The model's perplexity will increase significantly when processing long input sequences. Therefore, how to make LLMs effectively process long texts that exceed the pre-trained context window length has become a technical problem that needs to be solved urgently.

[0003] In related technologies, a direct method is to fine-tune the model through a large amount of long texts. By fine-tuning on enough long texts, the model can learn the ability to process long texts, thereby effectively coping with long input texts in the reasoning stage.

[0004] However, this method is time-consuming and resource-consuming. The scarcity of high-quality long text data hinders the effectiveness of fine-tuning methods and reduces the generalization ability of LLMs to a wide range of tasks, which needs to be urgently addressed. Summary of the invention

[0005] The present application provides a zero-training adaptive extended relative position encoding method based on RoPE to solve the problem that the existing technology is both time-consuming and resource-consuming, and the scarcity of high-quality long text data hinders the effectiveness of fine-tuning methods and reduces the generalization ability of LLMs for a wide range of tasks, thereby improving the model's processing capabilities and effects on long texts.

[0006] To achieve the above objectives, the first embodiment of the present application proposes a RoPE-based zero-training adaptive extended relative position coding method, comprising the following steps:

[0007] Get the length of the token to be encoded;

[0008] Determining a range of relative position coding and an initial length ratio, calculating a product of an upper limit value of the range of relative position coding and the initial length ratio, and determining an initial position and an end position for a group coding region according to the product;

[0009] Based on the length of the token to be encoded, according to the initial position and the end position, the number of multiplexing times and the extended coding position of the relative position coding in the group coding area are determined, and adaptive extended coding is performed according to the number of multiplexing times and the extended coding position.

[0010] According to one embodiment of the present application, when determining the number of multiplexing times and the extended coding position of the relative position coding within the group coding area, it also includes:

[0011] The group containing the multiplexing times that are integer powers of 2 is retained.

[0012] According to one embodiment of the present application, when determining the number of multiplexing times and the extended coding position of the relative position coding within the group coding area, it also includes:

[0013] Determine the current common ratio;

[0014] Based on the current common ratio and the product, the number of groups of each integer power of 2 is determined and retained.

[0015] According to one embodiment of the present application, the relative position of the query and the key is negatively correlated with the density of the position encoding.

[0016] According to the zero-training adaptive extended relative position coding method based on RoPE proposed in the embodiment of the present application, by obtaining the length of the token to be encoded, determining the range of the relative position coding and the initial length ratio, and calculating the product of the upper limit value of the range of the relative position coding and the initial length ratio, and determining the initial position and the end position for the group coding area according to the product, the number of reuses and the extended coding position of the relative position coding in the group coding area can be determined based on the length of the token to be encoded, according to the initial position and the end position, and adaptive extended coding is performed according to the number of reuses and the extended coding position. Thus, the problem that the prior art is both time-consuming and resource-consuming, and the scarcity of high-quality long text data hinders the effectiveness of the fine-tuning method, reduces the generalization ability of LLMs for a wide range of tasks, and improves the model's processing ability and effect on long texts.

[0017] To achieve the above-mentioned purpose, the second aspect of the present application proposes a RoPE-based zero-training adaptive extended relative position encoding device, including:

[0018] An acquisition module, used to obtain the length of the token to be encoded;

[0019] A determination module, used to determine the range of relative position coding and the initial length ratio, and calculate the product of the upper limit value of the range of relative position coding and the initial length ratio, and determine the initial position and the end position for the group coding area according to the product;

[0020] The encoding module is used to determine the number of multiplexing and extended coding positions of the relative position coding in the group coding area based on the length of the token to be encoded, according to the initial position and the end position, and perform adaptive extended coding according to the number of multiplexing and the extended coding position.

[0021] According to one embodiment of the present application, when determining the number of multiplexing times and the extended coding position of the relative position coding in the group coding area, the coding module is further used to:

[0022] The group containing the multiplexing times that are integer powers of 2 is retained.

[0023] According to one embodiment of the present application, when determining the number of multiplexing times and the extended coding position of the relative position coding in the group coding area, the coding module is further used to:

[0024] Determine the current common ratio;

[0025] Based on the current common ratio and the product, the number of groups of each integer power of 2 is determined and retained.

[0026] According to one embodiment of the present application, the relative position of the query and the key is negatively correlated with the density of the position encoding.

[0027] According to the RoPE-based zero-training adaptive extended relative position coding device proposed in the embodiment of the present application, by obtaining the length of the token to be encoded, determining the range of the relative position coding and the initial length ratio, and calculating the product of the upper limit value of the range of the relative position coding and the initial length ratio, and determining the initial position and the end position for the group coding area according to the product, the number of reuses and the extended coding position of the relative position coding in the group coding area can be determined based on the length of the token to be encoded, according to the initial position and the end position, and adaptive extended coding is performed according to the number of reuses and the extended coding position. Thus, the problem that the prior art is both time-consuming and resource-consuming, and the scarcity of high-quality long text data hinders the effectiveness of the fine-tuning method, reduces the generalization ability of LLMs for a wide range of tasks, and improves the model's processing ability and effect on long texts.

[0028] To achieve the above-mentioned objectives, the third aspect embodiment of the present application proposes an electronic device, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the RoPE-based zero-training adaptive extended relative position coding method as described in the above-mentioned embodiment.

[0029] To achieve the above objectives, the fourth aspect of the present application proposes a computer-readable storage medium on which a computer program is stored, and the program is executed by a processor to implement the RoPE-based zero-training adaptive extended relative position coding method as described in the above embodiment.

[0030] To achieve the above objectives, the fifth aspect of the present invention proposes a computer program product, including a computer program, which, when executed by a processor, is used to implement the RoPE-based zero-training adaptive extended relative position coding method as described in the above embodiment.

[0031] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0033] Figure 1 A flowchart of a RoPE-based zero-training adaptive extended relative position encoding method provided according to an embodiment of the present application;

[0034] Figure 2 This is a schematic diagram showing an example of the application of a zero-trained adaptive extended relative position coding strategy according to an embodiment of the present application;

[0035] Figure 3 It is a block diagram of a RoPE-based zero-training adaptive extended relative position encoding device provided according to an embodiment of the present application;

[0036] Figure 4 It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0037] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0038] The following describes the RoPE-based zero-training adaptive extended relative position coding proposed according to an embodiment of the present application with reference to the accompanying drawings.

[0039] Figure 1 This is a flowchart of a RoPE-based zero-training adaptive extended relative position encoding method according to an embodiment of the present application.

[0040] Before introducing the RoPE-based zero-training adaptive extended relative position encoding method proposed in the embodiment of the present application, the relevant technical background is first introduced.

[0041] In recent years, the industry has developed a variety of context window expansion methods, which aim to improve the ability of large language models to process long texts, so as to better cope with ultra-long input texts in the reasoning stage. Among them, a direct method is to fine-tune the model through a large amount of long texts. By fine-tuning on enough long texts, the model can learn the ability to process long texts, so as to effectively cope with long input texts in the reasoning stage. In addition, there are some methods that seek to expand the context window length in a more efficient way of fine-tuning. These methods include PI (Parameter Interpolation), CLEX (Context Length Extension), Yarn, LongLora (Extra-long Context) and ABF (Attention-Based Filtering), each of which has its own unique strategy and implementation method, as follows:

[0042] (1) PI method: The PI method combines the features of long text and short text to expand the context window length by interpolating parameters. It attempts to expand the model's parameter space to the range required to process long text without significantly increasing computational complexity. This method interpolates the parameters of long and short texts, allowing the model to take into account the ability to process long and short texts, making it more flexible during reasoning.

[0043] (2) CLEX method: The CLEX method uses contextual information to expand the encoder so that it can process longer texts. It adjusts the encoder structure to adapt to longer input sequences. This method improves the encoder to enhance its ability to process long texts, so that the semantics of long texts can be better captured and understood in the reasoning stage.

[0044] (3) Yarn method: The Yarn method dynamically adjusts the length of the context window so that the model can automatically adjust according to the length of the input text. This method is highly flexible and can adjust the context window in real time according to the specific input situation. The core of the Yarn method lies in its dynamic adaptability, which enables the model to maintain high performance when processing texts of different lengths.

[0045] (4) LongLora method: The LongLora method divides a long text into several parts through hierarchical recursion, processes them separately, and finally summarizes the results. This method reduces the computational complexity through hierarchical processing. The LongLora method divides a long text into smaller sub-texts, processes them separately, and then summarizes the results, thereby reducing the amount of calculation while maintaining the overall understanding of the long text.

[0046] (5) ABF method: The ABF method uses the improvement of the attention mechanism to enable the model to be more efficient in processing long texts. By optimizing the attention mechanism, the ABF method enables the model to capture important information in long texts more accurately. This method improves the efficiency of processing long texts by filtering out key information in the text and reducing unnecessary computational burden.

[0047] However, although the prior art provides several methods for extending the context window, they have the following disadvantages:

[0048] (1) These methods usually require model fine-tuning, which is both time-consuming and resource-consuming, especially considering the computational complexity of the Transformer model.

[0049] (2) The scarcity of high-quality long text data hinders the effectiveness of fine-tuning methods. Most real-world data are short texts, and long texts often lack meaningful long-distance dependencies.

[0050] (3) The fine-tuning process may weaken the model’s strong performance on shorter sequences during the pre-training phase, or overfit the model to the fine-tuning set, thereby reducing the generalization ability of LLMs to a wide range of tasks.

[0051] It is precisely based on the above problems that the embodiment of the present application proposes a zero-training adaptive expansion relative position encoding method based on RoPE, which aims to input the range of the restricted relative position encoding and the length of the input token to be encoded under the condition of zero training, and output a set of relative position encodings suitable for RoPE, satisfying that the length of the relative position encoding is equal to the length of the input token, and the value is within the specified range. In addition, the embodiment of the present application repeatedly uses the relative position encoding to make it meet the conditions that the position encoding is dense when the relative position of the query query and the key key is close, and the position encoding is sparse when it is far away, and meets the assumption that text processing is sensitive to the position of the closer position token of a token, and is insensitive to the position of the farther position token. As a result, the problems that the prior art is both time-consuming and resource-consuming, the scarcity of high-quality long text data hinders the effectiveness of the fine-tuning method, and reduces the generalization ability of LLMs for a wide range of tasks are solved, and the model's processing ability and effect on long texts are improved.

[0052] For example, Figure 1As shown, the RoPE-based zero-training adaptive extended relative position encoding method includes the following steps:

[0053] In step S101, the length of the token to be encoded is obtained.

[0054] That is to say, firstly, the total length of the tokens in the sequence to be processed (such as text, time series, etc.), that is, the length of the token to be encoded, needs to be determined, and this length is used for how to expand or reuse the position coding later.

[0055] In step S102, the range of relative position coding and the initial length ratio are determined, and the product of the upper limit value of the range of relative position coding and the initial length ratio is calculated, and the initial position and the end position for the grouped coding area are determined according to the product.

[0056] Specifically, assuming that the given length of the token to be encoded is seq_len, the range of the relative position encoding can be set to 0 to max_pe, and an initial length ratio is set to r, so as to determine the range of a set of continuous relative position encodings, and the relative position encodings that meet the initial length ratio are continuous relative position encodings. Based on the upper limit value max_pe of the range of the relative position encoding and the initial length ratio r, the product max_pe×r of the upper limit value of the range of the relative position encoding and the initial length ratio is calculated, and the initial position for the grouped encoding area is determined to be 0 and the end position is max_pe×r according to the obtained product result. This is the range for encoding the relative position in the initial stage. For each query query, the relative position of the key closest to it by max_pe×r positions remains unchanged to adapt to the characteristics of the query query and key key that are relatively close to it that are sensitive to the relative position encoding, that is, each query query can perceive the precise position of the key key that is very close to it. For the subsequent relative position encoding, if the relative distance between the actual query query and the key key exceeds max_pe×r, the embodiment of the present application performs group encoding on the relative positions, divides the relative positions into multiple groups, and repeatedly uses the relative position encoding to achieve flexible processing of input sequences of different lengths.

[0057] In step S103, based on the length of the token to be encoded, according to the initial position and the end position, the number of multiplexing times and the extended coding position of the relative position coding in the group coding area are determined, and adaptive extended coding is performed according to the number of multiplexing times and the extended coding position.

[0058] Specifically, when the relative distance between the actual query query and the key key exceeds max_pe×r, the embodiment of the present application divides the relative positions into multiple groups when group encoding the relative positions. The relative position codes in each group will be reused multiple times, and the number of reuses N increases in sequence from small to large according to the relative position (that is, the number of reuses N gradually increases from the position farthest from the query query, that is, the farthest relative position code is reused the most times, and gradually decreases towards the near position). For example, the first group may be reused once, the second group twice, and so on, to ensure that all available relative position codes are effectively utilized, and the relationship between long-distance queries and keys can also be reasonably represented.

[0059] After determining the number of multiplexing times, the embodiments of the present application also need to consider additional tokens that are not included in the max_pe×r range. These additional tokens can be covered with additional position codes, that is, the extended coding position is determined, and adaptive extended coding is performed according to the number of multiplexing times and the extended coding position, that is, the same position code is used for each group whose length does not exceed max_pe×r, and extended codes are generated for additional tokens exceeding max_pe×r.

[0060] It can be understood that the strategy of increasing the number of group encoding and reuse times aims to maximize the use of all available relative position encodings, which helps the model maintain high performance when dealing with long-distance dependencies.

[0061] Optionally, in some embodiments, when determining the number of multiplexing times and the extended coding position of the relative position coding within the group coding area, it also includes: retaining the group where the number of multiplexing times of an integer power of 2 belongs.

[0062] For example, if Figure 2 As shown, the embodiment of the present application can increase the number of multiplexing times N of the relative position coding in each group one by one, and the number of multiplexing times of the relative position coding gradually increases from the farthest relative position to the closer relative position, while expanding the position coding, and finally only retaining the group where the number of multiplexing times of an integer power of 2 is located (i.e. Figure 2 When the length of the token to be encoded seq_len ≥ 48, the group with multiplexing times N = 3 will automatically disappear as the length of the token to be encoded increases, while the groups N = 1, 2 that satisfy the integer power of 2 are still retained).

[0063] Optionally, in some embodiments, when determining the number of multiplexing times and extended coding positions of relative position coding within a group coding area, it also includes: determining a current common ratio; based on the current common ratio and the product, determining and retaining the number of groups of each integer power of 2.

[0064] That is to say, in the process of extended coding in the embodiment of the present application, the number of groups retained that satisfy the multiplexing number N being an integer power of 2 is determined by a geometric progression with an initial value of max_pe×r and a common ratio of 0.5. Figure 2 As shown, as the length of the token to be encoded increases, there are at least max_pe×r=4 groups satisfying N=1, and at least max_pe×r×0.5=2 groups satisfying N=2. This ensures that the relative position coding always maintains the characteristics of from dense to sparse from near to far as the length of the token to be encoded increases.

[0065] It can be seen that the position coding strategy proposed in the embodiment of the present application has the characteristic of adaptively adjusting with the length of the token to be encoded (i.e., the length of the input text), while reusing the existing relative position coding, satisfying the characteristics of less reuse (dense) of position coding for close distances and more reuse (sparse) of position coding for far distances.

[0066] Optionally, in some embodiments, the relative position of the query and the key is negatively correlated with the density of the position encoding.

[0067] That is to say, positions that are closer retain higher encoding accuracy, while positions that are farther away are sparsely encoded, that is, when the relative positions of the query and the key are close, dense position encoding is used to maintain high position sensitivity; when the relative positions are far away, sparse position encoding is used to comply with the assumption in text processing that the positions of distant tokens are insensitive.

[0068] In summary, the RoPE-based zero-training adaptive extended relative position coding method proposed in the embodiment of the present application has the following advantages:

[0069] (1) No need for high-quality annotated data: This application can be used without any training and does not rely on a large amount of high-quality annotated data for fine-tuning, thus reducing the requirements and dependence on data resources.

[0070] (2) No need for a large amount of training resources: By reusing relative position encoding, especially in long text processing, the density of the position encoding is adaptively adjusted according to the distance of the position, which effectively reduces the consumption of computing resources in methods that require fine-tuning and improves the efficiency of expanding the model's long text processing capabilities.

[0071] (3) Enhance the model's ability to process long texts: By adaptively adjusting the position encoding, the model can maintain high performance and stability when processing long texts. In particular, it can retain high encoding accuracy at positions that are close, while sparsely encoding at positions that are far away, thereby improving the model's ability to process long texts and its effect.

[0072] According to the zero-training adaptive extended relative position coding method based on RoPE proposed in the embodiment of the present application, by obtaining the length of the token to be encoded, determining the range of the relative position coding and the initial length ratio, and calculating the product of the upper limit value of the range of the relative position coding and the initial length ratio, and determining the initial position and the end position for the group coding area according to the product, the number of reuses and the extended coding position of the relative position coding in the group coding area can be determined based on the length of the token to be encoded, according to the initial position and the end position, and adaptive extended coding is performed according to the number of reuses and the extended coding position. Thus, the problem that the prior art is both time-consuming and resource-consuming, and the scarcity of high-quality long text data hinders the effectiveness of the fine-tuning method, reduces the generalization ability of LLMs for a wide range of tasks, and improves the model's processing ability and effect on long texts.

[0073] Secondly, the RoPE-based zero-training adaptive extended relative position encoding device proposed according to the embodiment of the present application is described with reference to the accompanying drawings.

[0074] Figure 3 It is a block diagram of a RoPE-based zero-training adaptive extended relative position encoding device according to an embodiment of the present application.

[0075] like Figure 3 As shown, the RoPE-based zero-training adaptive extended relative position encoding device 10 includes: an acquisition module 100, a determination module 200 and an encoding module 300.

[0076] Wherein, the acquisition module 100 is used to obtain the length of the token to be encoded;

[0077] A determination module 200, for determining a range of relative position coding and an initial length ratio, and calculating a product of an upper limit value of the range of relative position coding and the initial length ratio, and determining an initial position and an end position for a group coding region according to the product;

[0078] The coding module 300 is used to determine the number of multiplexing and the extended coding position of the relative position coding in the group coding area based on the length of the token to be encoded, according to the initial position and the end position, and perform adaptive extended coding according to the number of multiplexing and the extended coding position.

[0079] Further, in some embodiments, when determining the number of multiplexing times and the extended coding position of the relative position coding within the group coding region, the coding module 300 is further used to:

[0080] The group containing the multiplexing times that are integer powers of 2 is retained.

[0081] Further, in some embodiments, when determining the number of multiplexing times and the extended coding position of the relative position coding within the group coding region, the coding module 300 is further used to:

[0082] Determine the current common ratio;

[0083] Based on the current common ratio and product, determine and retain the number of groups for each integer power of 2.

[0084] Further, in some embodiments, the relative position of the query and the key is negatively correlated with the density of the position encoding.

[0085] It should be noted that the aforementioned explanation of the RoPE-based zero-training adaptive extended relative position coding method embodiment is also applicable to the RoPE-based zero-training adaptive extended relative position coding device of this embodiment, and will not be repeated here.

[0086] According to the RoPE-based zero-training adaptive extended relative position coding device proposed in the embodiment of the present application, by obtaining the length of the token to be encoded, determining the range of the relative position coding and the initial length ratio, and calculating the product of the upper limit value of the range of the relative position coding and the initial length ratio, and determining the initial position and the end position for the group coding area according to the product, the number of reuses and the extended coding position of the relative position coding in the group coding area can be determined based on the length of the token to be encoded, according to the initial position and the end position, and adaptive extended coding is performed according to the number of reuses and the extended coding position. Thus, the problem that the prior art is both time-consuming and resource-consuming, and the scarcity of high-quality long text data hinders the effectiveness of the fine-tuning method, reduces the generalization ability of LLMs for a wide range of tasks, and improves the model's processing ability and effect on long texts.

[0087] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0088] Memory 401 , processor 402 , and a computer program stored in the memory 401 and executable on the processor 402 .

[0089] When the processor 402 executes the program, the RoPE-based zero-training adaptive extended relative position encoding method provided in the above embodiment is implemented.

[0090] Furthermore, the electronic device further comprises:

[0091] The communication interface 403 is used for communication between the memory 401 and the processor 402 .

[0092] The memory 401 is used to store computer programs that can be executed on the processor 402 .

[0093] The memory 401 may include a high-speed RAM (Random Access Memory) memory, and may also include a non-volatile memory, such as at least one disk memory.

[0094] If the memory 401, the processor 402 and the communication interface 403 are implemented independently, the communication interface 403, the memory 401 and the processor 402 can be connected to each other through a bus and communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0095] Optionally, in a specific implementation, if the memory 401, the processor 402 and the communication interface 403 are integrated on a chip, the memory 401, the processor 402 and the communication interface 403 can communicate with each other through an internal interface.

[0096] The processor 402 may be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application.

[0097] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned RoPE-based zero-training adaptive extended relative position encoding method is implemented.

[0098] An embodiment of the present application also proposes a computer program product, on which a computer program is stored, and when the program is executed by a processor, the above-mentioned RoPE-based zero-training adaptive extended relative position encoding method is implemented.

[0099] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0100] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0101] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A zero-training adaptive extended relative position encoding method based on RoPE, characterized in that: The following steps are involved: Get the length of the token to be encoded; Determining a range of relative position coding and an initial length ratio, calculating a product of an upper limit value of the range of relative position coding and the initial length ratio, and determining an initial position and an end position for a group coding region according to the product; Based on the length of the token to be encoded, according to the initial position and the end position, the number of multiplexing times and the extended coding position of the relative position coding in the group coding area are determined, and adaptive extended coding is performed according to the number of multiplexing times and the extended coding position.

2. The method according to claim 1, characterized in that When determining the number of multiplexing times and the extended coding position of the relative position coding within the group coding area, it also includes: The group containing the multiplexing times that are integer powers of 2 is retained.

3. The method according to claim 1, characterized in that When determining the number of multiplexing times and the extended coding position of the relative position coding within the group coding area, it also includes: Determine the current common ratio; Based on the current common ratio and the product, the number of groups of each integer power of 2 is determined and retained.

4. The method according to claim 1, characterized in that: The relative positions of the query and key are negatively correlated with the density of the positional encoding.

5. A zero-training adaptive extended relative position encoding device based on RoPE, characterized in that: include: An acquisition module, used to obtain the length of the token to be encoded; A determination module, used to determine the range of relative position coding and the initial length ratio, and calculate the product of the upper limit value of the range of relative position coding and the initial length ratio, and determine the initial position and the end position for the group coding area according to the product; The encoding module is used to determine the number of multiplexing and extended coding positions of the relative position coding in the group coding area based on the length of the token to be encoded, according to the initial position and the end position, and perform adaptive extended coding according to the number of multiplexing and the extended coding position.

6. The device according to claim 5, characterized in that When determining the number of multiplexing times and the extended coding position of the relative position coding in the group coding area, the coding module is further used to: The group containing the multiplexing times that are integer powers of 2 is retained.

7. The device according to claim 5, characterized in that When determining the number of multiplexing times and the extended coding position of the relative position coding in the group coding area, the coding module is further used to: Determine the current common ratio; Based on the current common ratio and the product, the number of groups of each integer power of 2 is determined and retained.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the RoPE-based zero-training adaptive extended relative position coding method as described in any one of claims 1 to 4.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the RoPE-based zero-training adaptive extended relative position encoding method as described in any one of claims 1-4.

10. A computer program product, characterized in that It includes a computer program, which, when executed by a processor, is used to implement the RoPE-based zero-training adaptive extended relative position encoding method as described in any one of claims 1 to 4.

Images