A multi-level relative position coding method and system

Abstract

The present invention discloses a parameter value estimation method of xPos relative position coding, a multi-level relative position coding method and system, which relates to the field of deep learning technology, solves the technical problem that the Transformer model has weak robustness and low accuracy when processing long sequences, and the key points of the technical solution are to introduce an exponential decay factor in the xPos relative position coding, use exponential decay to adjust the encoding method of the query and key vector, suppress the oscillation of high-frequency components, enhance the model's ability to capture long-distance dependencies, and further improve the model's length extrapolation ability in unseen long sequence tasks. At the same time, the xPos relative position coding is used for intra-segment coding and inter-segment coding of text sequences to achieve double-layer coding. By combining intra-segment coding and inter-segment coding, the hierarchical structure information in the text sequence is captured, and long sequences that have not been seen during training can be processed more effectively, improving the model's length extrapolation ability.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a multi-level relative position encoding method and system. Background Art

[0002] With the rapid development of deep learning technology, Transformer-based models, especially large language models (LLMs), have achieved remarkable results in the field of natural language processing (NLP). With their powerful self-attention mechanism and parallel training capabilities, these models have performed well in tasks such as language modeling, text summarization, and machine translation. However, Transformer models face a key challenge when processing sequence data, namely the preset sequence length limit. Typically, these models are limited to shorter sequence lengths (such as 512 or 1024 tokens) during the training phase, which often leads to insufficient understanding and processing capabilities for longer sequences in practical applications, limiting their application in processing long documents, multi-round conversations, and tasks that require long-term memory.

[0003] To solve this problem, researchers have proposed a variety of positional encoding (PE) methods to enhance the Transformer model's length extrapolation capability. Among them, relative positional encoding (RPE) is widely considered to be superior in processing unseen long sequences due to its relativity and translation invariance of position information. Among the many RPE methods, Rotary Position Embedding (RoPE) has become one of the current mainstream positional encoding methods because it can effectively capture relative position information while combining complexity and expressiveness.

[0004] Based on RoPE, researchers further proposed xPos as an improved version of RoPE. By introducing a new mathematical mechanism, xPos attempts to further improve the length extrapolation performance of the model while maintaining the excellent characteristics of RoPE. However, there is still a key problem in the practical application of xPos, that is, the value range of the ζ (zeta) parameter is under-estimated. The ζ parameter plays a role in adjusting the rotation amplitude and the degree of position information fusion in xPos. A reasonable estimate of its value range is crucial to ensure the stability and accuracy of the model when processing sequences of different lengths.

[0005] Existing RoPE and xPos methods usually rely on experience or limited theoretical guidance to set the range of the ζ parameter, which may lead to overfitting or information loss in the model when facing extreme length extrapolation tasks. In addition, as the scale of LLMs continues to expand, traditional ζ range estimation methods also face challenges in computational efficiency and adaptability, and cannot meet the needs of more efficient and accurate position information encoding in practical applications.

[0006] In addition, the hierarchical structure and modularity in linguistics provide new inspiration for the position information encoding method. Sentences and paragraphs in natural language have an inherent hierarchical relationship. The words in a paragraph have a close semantic connection, while there is a loose contextual connection between paragraphs. How to capture the local semantic information within a paragraph and effectively encode the relative position information between paragraphs needs to be further solved. Summary of the invention

[0007] The present application provides a multi-level relative position encoding method and system, the technical purpose of which is to scientifically and reasonably determine the parameter value range in xPos relative position encoding, and to improve the robustness and accuracy of the model when processing long sequences.

[0008] The above technical objectives of this application are achieved through the following technical solutions:

[0009] A multi-level relative position coding method, which is implemented by using an xPos relative position coding obtained by a parameter value estimation method of an xPos relative position coding, comprising:

[0010] The text sequence S is divided into a series of non-overlapping modular segments, namely ;in, represents the nth segment, N represents the total number of segments, Represents a serial operation, , and Respectively represent the starting index and ending index of the segment; , Indicates word, Indicates the total length of the text sequence;

[0011] Each segment is encoded by xPos relative position Encode the position within the segment;

[0012] Through xPos relative position coding, each segment is inter-segment coded to complete multi-level relative position coding;

[0013] Among them, the parameter value estimation method of xPos relative position coding includes:

[0014] Step S1: When the index distance between two words in the xPos relative position encoding is n, and the attention expected value sequence of the compatibility score between the two words is s[n], the attention resolution R(s) of the xPos relative position encoding is defined as:

[0015] ;

[0016] Where j represents the index of the compatibility score between two words with distance j; represents the exponential transformation of the compatibility score between two words with a distance j; the attention resolution R(s) is used to evaluate the attention's ability to recognize positions;

[0017] Step S2: Define the query encoding function and key encoding function of the xPos relative position encoding, which are respectively expressed as:

[0018] ;

[0019] ;

[0020] in, represents the query encoding function; represents the key encoding function; q represents the query vector; k represents the key vector; is a real attenuation factor; is the frequency parameter; i represents the imaginary unit; d represents the dimension of the model;

[0021] Then the query encoding function and key encoding function satisfy: ;

[0022] in, Indicates the query code at position (n+r); Indicates the key code at position 0; Indicates the query code at position r; represents the pseudo inner product, n and r represent the position and relative offset respectively;

[0023] Step S3: According to the real attenuation factor Characteristic function for relative position Defined as:

[0024] ;

[0025] in, , represents the attenuation factor of the i'th dimension; Indicates the frequency parameter of the i'th dimension

[0026] Step S4: Define it and combine the attention resolution R(s), query encoding function and key encoding function to The value of is estimated to complete the parameter value estimation of xPos relative position encoding; It is expressed as:

[0027] ;

[0028] in, Represents a hyperparameter.

[0029] Furthermore, the xPos relative position encoding is used to encode each segment. The position within the segment is encoded, including:

[0030] Paragraph The position i'' in the segment is encoded; where ;

[0031] For the word at position i'' in the segment Assign a real-valued embedding vector , and embed the real value into the vector Add to word embedding;

[0032] For words at the same position i'' in different segments, they share the same real-valued embedding vector .

[0033] A multi-level relative position coding system, comprising:

[0034] The partitioning module divides the text sequence S into a series of non-overlapping modular segments, namely ;in, represents the nth segment, N represents the total number of segments, Represents a serial operation, , and Respectively represent the starting index and ending index of the segment; , Indicates word, Indicates the total length of the text sequence;

[0035] The intra-segment encoding module encodes each segment through xPos relative position encoding Encode the position within the segment;

[0036] The inter-segment coding module performs inter-segment coding on each segment through xPos relative position coding to complete multi-level relative position coding;

[0037] The xPos relative position code is obtained by a parameter value estimation method of the xPos relative position code, and the parameter value estimation method of the xPos relative position code includes:

[0038] Step S1: When the index distance between two words in the xPos relative position encoding is n, and the attention expected value sequence of the compatibility score between the two words is s[n], the attention resolution R(s) of the xPos relative position encoding is defined as:

[0039] ;

[0040] Where j represents the index of the compatibility score between two words with distance j; represents the exponential transformation of the compatibility score between two words with a distance j; the attention resolution R(s) is used to evaluate the attention's ability to recognize positions;

[0041] Step S2: Define the query encoding function and key encoding function of the xPos relative position encoding, which are respectively expressed as:

[0042] ;

[0043] ;

[0044] in, represents the query encoding function; represents the key encoding function; q represents the query vector; k represents the key vector; is a real attenuation factor; is the frequency parameter; i represents the imaginary unit; d represents the dimension of the model;

[0045] Then the query encoding function and key encoding function satisfy: ;

[0046] in, Indicates the query code at position (n+r); Indicates the key code at position 0; Indicates the query code at position r; represents the pseudo inner product, n and r represent the position and relative offset respectively;

[0047] Step S3: According to the real attenuation factor Characteristic function for relative position Defined as:

[0048] ;

[0049] in, , represents the attenuation factor of the i'th dimension; Indicates the frequency parameter of the i'th dimension

[0050] Step S4: Define it and combine the attention resolution R(s), query encoding function and key encoding function to The value of is estimated to complete the parameter value estimation of xPos relative position encoding; It is expressed as:

[0051] ;

[0052] in, Represents a hyperparameter.

[0053] Furthermore, the intra-segment encoding module is used for:

[0054] Paragraph The position i'' in the segment is encoded; where ;

[0055] For the word at position i'' in the segment Assign a real-valued embedding vector , and embed the real value into the vector Add to word embedding;

[0056] For words at the same position i'' in different segments, they share the same real-valued embedding vector .

[0057] The beneficial effects of this application are:

[0058] (1) Enhanced length extrapolation capability: The parameter value estimation method of the xPos relative position encoding described in this application introduces an exponential decay factor in the xPos relative position encoding, adjusts the transformation method of the query vector and the key vector, suppresses the oscillation of high-frequency components, enhances the model's ability to capture long-distance dependencies, and further enhances the length extrapolation capability of the Transformer model.

[0059] The xPos relative position encoding, obtained based on the parameter value estimation method of the xPos relative position encoding, is used for intra-segment encoding and inter-segment encoding of text sequences to achieve double-layer encoding. By combining intra-segment and inter-segment encoding, the hierarchical information in the text sequence is captured. Within a segment, the absolute position encoding ensures the position perception of the word within a sentence or paragraph, which helps to understand local semantics. Between segments, the relative position encoding based on the segment index captures the contextual relationship between segments, which helps the model understand the global structure. By combining the absolute position encoding within a segment and the relative position encoding between segments, the model can more accurately capture the hierarchical structure and global contextual relationship in long sequences, thereby significantly improving the accuracy of tasks such as long document understanding and multi-round dialogue generation.

[0060] (2) Improve recognition and generation accuracy: Through the optimized xPos position encoding method, the model can maintain a more stable attention distribution when processing long sequences, reducing information loss or overfitting problems caused by high-frequency oscillations. This enables the model to show higher accuracy and better generation quality in applications such as language modeling, text summarization, and machine translation.

[0061] (3) Optimizing parameter range estimation: The improved ζ parameter range estimation method proposed in this application uses spectrum characteristic analysis and adaptive adjustment mechanism to overcome the inaccuracy problem caused by the existing method relying on empirical settings. This not only improves the robustness and stability of the model when processing sequences of different lengths, but also reduces the complexity of parameter tuning and shortens the development and deployment cycle.

[0062] (4) Improved computational efficiency and numerical stability: By introducing an exponential decay factor and optimizing frequency parameters, this application significantly improves the computational efficiency of position encoding without increasing computational complexity. At the same time, the improved encoding method maintains numerical stability when processing extremely long sequences, avoiding computational overflow or underflow problems caused by high-frequency oscillations, and ensuring the reliability and efficiency of the model in various application scenarios.

[0063] (5) Save resources and reduce costs: By optimizing the calculation process and parameter settings of position coding, this application improves model performance while reducing the consumption of computing resources. This is particularly important for large-scale industrial applications, which can significantly reduce the operating costs of servers and computing equipment and improve the overall economic benefits of the system.

[0064] (6) Good scalability: This application can be used in combination with other technologies that improve length extrapolation capabilities (such as position interpolation, random position information encoding, etc.) to further enhance model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1It is a schematic diagram of the embedding method of absolute position coding and relative position coding in the model in the prior art;

[0066] Figure 2 Schematic diagram of the embedding method of double-layer position information encoding in the model in the embodiment of the present application. DETAILED DESCRIPTION

[0067] The technical solution of the present application will be described in detail below with reference to the accompanying drawings.

[0068] Positional encoding is mainly divided into two categories: Absolute Positional Encoding (APE) and Relative Positional Encoding (RPE). In the Transformer model, positional encoding is usually introduced between the embedding layer and the first Transformer layer. For APE, the positional encoding is added to the word embedding, that is, Input=E+PE. For RPE, the positional information is usually directly integrated into the calculation of the self-attention mechanism, modifying the calculation method of the compatibility score. Figure 1 Schematic diagram of how absolute position encoding and relative position encoding are embedded in the model.

[0069] The xPos relative position encoding described in this application is an improved method for rotational position encoding (RoPE) to further enhance the length extrapolation capability of the Transformer model. xPos introduces an exponential decay factor to adjust the transformation method of the query and key vectors, suppress the oscillation of high-frequency components, and enhance the model's ability to capture long-distance dependencies.

[0070] Step S1: In order to quantify the ability of the attention mechanism in length extrapolation, the attention resolution R(s) is defined to evaluate the attention's ability to recognize positions. When the index distance between two words is n, the expected sequence of their compatibility scores is s[n], then the attention resolution is defined as:

[0071] ;

[0072] Where j represents the index of the compatibility score between two words with distance j; represents the exponential transformation of the compatibility score between two tokens with distance j.

[0073] The larger R(s) is, the stronger the attention mechanism is at distinguishing different positions, which helps to extrapolate the length of the model.

[0074] Specifically, when the attention expectation sequence s[n] is given, the quadratic variation on the interval with index distance n as the starting position and window size W is defined as:

[0075] .

[0076] When observing the attention expectation sequence, the quadratic variation is measured in each cycle, and the window size is dynamically selected according to each cycle. In each cycle, the score shows a sawtooth oscillating downward trend. When the score reaches a low point, it suddenly jumps up, marking the beginning of a new round of periodic oscillation.

[0077] Step S2: Define the query encoding function and key encoding function of the xPos relative position encoding so that they satisfy the following relationship:

[0078] ;

[0079] in, Indicates the query code at position (n+r); Indicates the key code at position 0; Indicates the query code at position r; represents the pseudo inner product, n and r represent the position and relative offset respectively.

[0080] In order to satisfy the above relationship, the query encoding function and key encoding function of xPos relative position encoding are defined, which are expressed as follows:

[0081] ;

[0082] ;

[0083] in, represents the query encoding function; represents the key encoding function; q represents the query vector; k represents the key vector; is a real attenuation factor; is the frequency parameter; i represents the imaginary unit; d represents the dimension of the model.

[0084] when When , the above form degenerates into the form of RoPE, that is, only rotation transformation. The purpose of introduction is to add exponential attenuation on the basis of rotation to suppress the oscillation of high-frequency components.

[0085] Step S3: To further stabilize the compatibility score curve, Characteristic function for relative position Defined as:

[0086] ;

[0087] in, , represents the attenuation factor of the i'th dimension; Represents the frequency parameter of the i'th dimension.

[0088] Specifically, according to the characteristic function of relative position Far-end volatility Define it as:

[0089] ;

[0090] Far-end volatility The upper bound of is estimated, including:

[0091] The indicator set Divided into two subsets, namely: and ;

[0092] When n is large, It can be approximately expressed as only depending on satisfying The upper bound of is:

[0093] .

[0094] Step S4: To suppress the oscillation of high-frequency components, manually set Make it take a smaller value at high frequency. Specifically, we can define for:

[0095] ;

[0096] in, represents a hyperparameter that controls the shape of the attenuation factor curve. When i' increases (corresponding to high-frequency components), Reduced, thereby suppressing the impact of high-frequency components on the compatibility score.

[0097] Specifically, the parameters for xPos relative position encoding Constraints are set to ensure that the far volatility satisfy:

[0098] ;

[0099] in, Indicates the preset fluctuation tolerance.

[0100] Parameters for encoding xPos relative position The constraints are set, including:

[0101] Characteristic function for relative position The increment Define, increment is the change between adjacent positions, which is used to measure the parameter The impact on the stability of position encoding is It is expressed as:

[0102] ;

[0103] According to the increment The square upper bound of the increment The overall volatility of is limited to control the sum of the far-end volatility, which is expressed as:

[0104] ;

[0105] To ensure the far volatility Meeting fluctuation tolerance Required, xPos relative position encoding parameters The constraints that need to be satisfied are expressed as:

[0106] .

[0107] Through the above constraints, accurate control of relative position encoding parameters can be achieved to reduce unstable fluctuations in remote position encoding.

[0108] Combining the attention resolution R(s), the query encoding function and the key encoding function The value of is estimated to realize the parameter value estimation of xPos relative position encoding.

[0109] Finally, according to The function to get the xPos relative position encoding is expressed as:

[0110] ;

[0111] .

[0112] Many research works are based on the assumption that words in natural language are generated in a hierarchical manner. This assumption has led to the development of a variety of hierarchical models, such as hierarchical hidden Markov models, hierarchical recurrent models, and hierarchical topic models. These models attempt to capture the hierarchical structure in text sequences to better understand and generate natural language. The multi-level relative position encoding method described in this application is implemented by the xPos relative position encoding obtained by the parameter value estimation method of the xPos relative position encoding, which represents the input text sequence S as ,in, Indicates word, Represents the total length of the text sequence. The text sequence is divided into a series of non-overlapping modular segments, namely:

[0113] ;

[0114] in, represents the nth segment, N represents the total number of segments, Represents a serial operation, , and The segment division strategy can be simply based on symbol detection (such as newline and period).

[0115] In many practical situations, each segment The number of tokens in follows a similar distribution, and the sequence length L only has a major impact on the number of segments N. Therefore, it is more effective to extrapolate the length of N than the actual length L. This application will integrate intra-segment encoding and inter-segment encoding, as follows:

[0116] Intra-segment encoding aims to identify the position of each word in the segment to which it belongs, so as to capture the semantic information within the segment. , we encode the position i'' within the segment, where Since the number of tokens in each segment is usually bounded (extremely long sentences are rare), the original absolute position encoding can be used. Specifically, for Each word in , we assign a real-valued embedding vector , and add it to the word embedding. For words at the same position i'' in different segments, they share the same real-valued embedding vector .

[0117] Although the intra-segment encoding provides the position of the word unit within the segment, the position relationship across segments may be confused, and the contextual relationship between segments is not captured. For this reason, the inter-segment encoding is used to specify the segment to which each word unit belongs to capture the relationship between segments. Considering that the inter-segment encoding also needs to handle longer sequences that have not been seen during training, we use the relative position information encoding xPos. Unlike the previous RPE defined based on the gap between word unit indices, the inter-segment encoding is defined based on the distance between segment indices.

[0118] This two-layer encoding technology captures the hierarchical structure information in the text sequence by combining intra-segment and inter-segment encoding. Within a segment, the absolute position encoding ensures the position awareness of the word in the sentence or paragraph, which helps to understand the local semantics. Between segments, the relative position encoding based on the segment index captures the contextual relationship between segments, which helps the model understand the global structure.

[0119] This double-layer encoding method can more effectively process long sequences that have not been seen during training, improving the model's length extrapolation capability. Figure 2 Schematic diagram of how the dual-layer position information is encoded and embedded in the model.

[0120] The multi-level relative position coding system described in the present application includes a division module, an intra-segment coding module and an inter-segment coding module.

[0121] The partitioning module is used to divide the text sequence S into a series of non-overlapping modular segments, namely ;in, represents the nth segment, N represents the total number of segments, Represents a serial operation, , and Respectively represent the starting index and ending index of the segment; , Indicates word, Indicates the total length of the text sequence.

[0122] The intra-segment encoding module is used to encode each segment by xPos relative position encoding. The position within the segment is encoded.

[0123] The inter-segment coding module is used to perform inter-segment coding on each segment through xPos relative position coding to complete multi-level relative position coding.

[0124] The intra-segment encoding module is specifically used for:

[0125] Paragraph The position i'' in the segment is encoded; where ;

[0126] For the word at position i'' in the segment Assign a real-valued embedding vector , and embed the real value into the vector Add to word embedding;

[0127] For words at the same position i'' in different segments, they share the same real-valued embedding vector .

[0128] The above are exemplary embodiments of the present application, and the protection scope of the present application is defined by the claims and their equivalents.

Claims

1. A multi-level relative position coding method, characterized in that: The method is implemented by obtaining the xPos relative position coding through the parameter value estimation method of the xPos relative position coding, and includes: The text sequence S is divided into a series of non-overlapping modular segments, namely ;in, represents the nth segment, N represents the total number of segments, Represents a serial operation, , and Respectively represent the starting index and ending index of the segment; , Indicates word, Indicates the total length of the text sequence; Each segment is encoded by xPos relative position The position within the segment is encoded; Through xPos relative position coding, each segment is inter-segment coded to complete multi-level relative position coding; Among them, the parameter value estimation method of xPos relative position coding includes: Step S1: When the index distance between two words in the xPos relative position encoding is n, and the attention expected value sequence of the compatibility score between the two words is s[n], the attention resolution R(s) of the xPos relative position encoding is defined as: ； Where j represents the index of the compatibility score between two words with distance j; represents the exponential transformation of the compatibility score between two words with a distance j; the attention resolution R(s) is used to evaluate the attention's ability to recognize positions; Step S2: Define the query encoding function and key encoding function of the xPos relative position encoding, which are respectively expressed as: ； ； in, represents the query encoding function; represents the key encoding function; q represents the query vector; k represents the key vector; is the real attenuation factor; frequency parameter; i represents the imaginary unit; d represents the dimension of the model; Then the query encoding function and key encoding function satisfy: ； in, Indicates the query code at position (n+r); Indicates the key code at position 0; Indicates the query code at position r; represents the pseudo inner product, n and r represent the position and relative offset respectively; Step S3: According to the real attenuation factor Characteristic function for relative position Defined as: ； in, , represents the attenuation factor of the i'th dimension; represents the frequency parameter of the i'th dimension; Step S4: Define it and combine the attention resolution R(s), query encoding function and key encoding function to The value of is estimated to complete the parameter value estimation of xPos relative position encoding; It is expressed as: ； in, Represents a hyperparameter.

2. The multi-level relative position coding method as claimed in claim 1, characterized in that: The xPos relative position encoding is used for each segment The position within the segment is encoded, including: Paragraph The position i'' in the segment is encoded; where ; For the word at position i'' in the segment Assign a real-valued embedding vector , and embed the real value into the vector Add to word embedding; For words at the same position i'' in different segments, they share the same real-valued embedding vector .

3. A multi-level relative position coding system, characterized in that: include: The partitioning module divides the text sequence S into a series of non-overlapping modular segments, namely ;in, represents the nth segment, N represents the total number of segments, Represents a serial operation, , and Respectively represent the starting index and ending index of the segment; , Indicates word, Indicates the total length of the text sequence; The intra-segment encoding module encodes each segment through xPos relative position encoding The position within the segment is encoded; The inter-segment coding module performs inter-segment coding on each segment through xPos relative position coding to complete multi-level relative position coding; The xPos relative position code is obtained by a parameter value estimation method of the xPos relative position code, and the parameter value estimation method of the xPos relative position code includes: Step S1: When the index distance between two words in the xPos relative position encoding is n, and the attention expected value sequence of the compatibility score between the two words is s[n], the attention resolution R(s) of the xPos relative position encoding is defined as: ； Where j represents the index of the compatibility score between two words with distance j; represents the exponential transformation of the compatibility score between two words with a distance j; the attention resolution R(s) is used to evaluate the attention's ability to recognize positions; Step S2: Define the query encoding function and key encoding function of the xPos relative position encoding, which are respectively expressed as: ； ； in, represents the query encoding function; represents the key encoding function; q represents the query vector; k represents the key vector; is the real attenuation factor; frequency parameter; i represents the imaginary unit; d represents the dimension of the model; Then the query encoding function and key encoding function satisfy: ； in, Indicates the query code at position (n+r); Indicates the key code at position 0; Indicates the query code at position r; represents the pseudo inner product, n and r represent the position and relative offset respectively; Step S3: According to the real attenuation factor Characteristic function for relative position Defined as: ； in, , represents the attenuation factor of the i'th dimension; represents the frequency parameter of the i'th dimension; Step S4: Define it and combine the attention resolution R(s), query encoding function and key encoding function to The value of is estimated to complete the parameter value estimation of xPos relative position encoding; It is expressed as: ； in, Represents a hyperparameter.

4. The multi-level relative position coding system as claimed in claim 3, characterized in that: The intra-segment encoding module is used for: Paragraph The position i'' in the segment is encoded; where ; For the word at position i'' in the segment Assign a real-valued embedding vector , and embed the real value into the vector Add to word embedding; For words at the same position i'' in different segments, they share the same real-valued embedding vector .

Images