Position coding method and device, equipment and storage medium

Abstract

The invention discloses a position coding method and device, equipment and a storage medium, and belongs to the technical field of computers. The method comprises the following steps: acquiring a position coding matrix and to-be-processed data; the position coding matrix comprises elements whose characteristic values are preset values; filtering elements of which the characteristic values are preset values in the position coding matrix to obtain a target matrix; the target matrix comprises elements except the elements of which the characteristic values are preset values in the position coding matrix; and then performing convolution processing on the target matrix and the to-be-processed data to obtain a processing result. Generally, the large language model carries out position coding on to-be-processed data according to the position coding matrix to obtain a processing result. As the most efficient calculation unit of the neural network chip is good at processing convolution operation, in the embodiment of the application, the process of performing position coding by using the position coding matrix is converted into convolution processing, so that the processing efficiency is improved, and efficient operation of the large language model is realized.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a location encoding method, apparatus, device and storage medium. Background Technology

[0002] With the rapid development of computer technology, Large Language Models (LLMs) are being applied more and more widely. For example, they can be used in scenarios such as prediction, recognition, detection, classification, machine translation, question answering, dialogue, generation, and recommendation. Large language models can typically run on neural network chips in the cloud or on the edge, utilizing deep learning techniques to achieve natural language processing tasks.

[0003] Positional encoding is a commonly used algorithm in large language models, helping the model understand the relative positions and order of different elements in a sequence, thereby better capturing the semantic information in the sequence. However, because the computational logic of positional encoding generally cannot be directly calculated using the most efficient computing units of neural network chips, its processing efficiency is relatively low. Summary of the Invention

[0004] This application provides a location encoding method, apparatus, device, and storage medium, which improves processing efficiency. The technical solution is as follows:

[0005] In a first aspect, a positional encoding method is provided, the method comprising: acquiring a positional encoding matrix and data to be processed; the positional encoding matrix including elements with feature values ​​preset values; filtering out the elements with feature values ​​preset values ​​in the positional encoding matrix to obtain a target matrix; the target matrix including elements in the positional encoding matrix excluding the elements with feature values ​​preset values; performing convolution processing on the target matrix and the data to be processed to obtain a processing result; the processing result being the result of positionally encoding the data to be processed according to the positional encoding matrix.

[0006] Secondly, a position encoding device is provided, the device comprising: an acquisition module for acquiring a position encoding matrix and data to be processed; the position encoding matrix including elements with feature values ​​preset values; a filtering module for filtering out the elements with feature values ​​preset values ​​in the position encoding matrix to obtain a target matrix; the target matrix including elements in the position encoding matrix excluding the elements with feature values ​​preset values; and a convolution module for performing convolution processing on the target matrix and the data to be processed to obtain a processing result; the processing result being the result of position encoding the data to be processed according to the position encoding matrix.

[0007] Thirdly, a computer device is provided, the computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program implementing the method described in the first aspect when executed by the processor.

[0008] Fourthly, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0009] Fifthly, a computer program product containing instructions is provided that, when run on a computer, causes the computer to perform the method described in the first aspect.

[0010] This application provides a location encoding method, apparatus, device, and storage medium. According to the scheme provided, a location encoding matrix and data to be processed are obtained. The location encoding matrix includes elements with preset eigenvalues; the preset eigenvalues ​​can be 0. If the location encoding matrix is ​​directly multiplied by the data to be processed to achieve location encoding, there will be significant resource waste, requiring substantial storage space and additional chip computation for acceleration. Therefore, elements with preset eigenvalues ​​in the location encoding matrix are filtered out to obtain a target matrix. The target matrix includes all elements in the location encoding matrix except for those with preset eigenvalues, thus effectively reducing redundant computation and data processing volume. Then, convolution processing is performed on the target matrix and the data to be processed to obtain the processing result. Typically, large language models use a location encoding matrix to perform location encoding on the data to be processed to obtain the processing result. Since the most efficient computing units of neural network chips are good at handling convolution operations, in this embodiment of the application, the process of position encoding using the position encoding matrix is ​​converted into convolution processing. Moreover, it does not require structural modifications or retraining of the large language model, thus improving processing efficiency and enabling efficient operation of the large language model containing the position encoding matrix. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart of a position encoding method provided in an embodiment of this application;

[0013] Figure 2This is a schematic diagram of a position encoding matrix provided in an embodiment of this application;

[0014] Figure 3 This is a flowchart of another position encoding method provided in the embodiments of this application;

[0015] Figure 4 This is a flowchart of another position encoding method provided in the embodiments of this application;

[0016] Figure 5 This is a schematic diagram of the structure of a position encoding device provided in an embodiment of this application;

[0017] Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0019] It should be understood that "multiple" as mentioned in this application refers to two or more. In the description of this application, unless otherwise stated, " / " indicates "or," for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist, for example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, to facilitate a clear description of the technical solutions of this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and that "first," "second," etc., do not necessarily imply differences.

[0020] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0021] Before providing a detailed explanation of the embodiments of this application, the application scenarios of these embodiments will be described first.

[0022] The positional encoding method provided in this application can be used in large language models. Application scenarios for large language models include, but are not limited to: object detection, object recognition, intelligent customer service, language generation, summarization generation, future prediction, cluster analysis, machine translation, question answering, dialogue, and product recommendation.

[0023] The positional encoding method provided in this application can be applied to different types of large language models, as long as the model is an algorithmic model that includes the RoPE operator, such as Transformer models, deep learning models, Large Language Model (LlaMA), Mistral 7B models, Baichuan, Chat General Language Model (ChatGLM), Vision-Language Model (VLM), etc. This application does not impose any limitations on this.

[0024] In many natural language processing tasks, the order of words is crucial, necessitating a method for large language models to capture this information. Positional encoding is introduced to help models understand the relative positions and order of different elements in a sequence, thereby better capturing the semantic information within the sequence. Positional encoding includes sinusoidal, Rotary Position Embedding (RoPE), and Adaptive Learnable Interpretable (ALiBi). The ALiBi operator represents the relative relationships between different positions by adding a learnable bias term to the self-attention mechanism; it can dynamically adapt to different tasks and data. The RoPE operator represents the absolute value of each position by encoding positional information as complex numbers and then rotating them with the input vector (corresponding to the data to be processed); it can directly represent positional information while preserving the original information of the input vector.

[0025] The RoPE operator, a positional encoding matrix, is frequently used in large language models. Since most elements in the RoPE operator are zero, directly multiplying the RoPE operator by the input vector to achieve positional encoding results in significant resource waste, requiring substantial storage space and additional chip computation for acceleration. Furthermore, some computational acceleration chips do not perform as well on matrix multiplication as they do on convolution. The computational logic of the RoPE operator cannot be directly computed using the most efficient computing units of neural network chips, leading to low processing efficiency and reducing the overall efficiency of large language models.

[0026] Based on the aforementioned technical problems, embodiments of this application provide a position encoding method that improves the computational efficiency of algorithm models containing RoPE operators by efficiently converting RoPE operators into operator types that neural network chips are adept at processing. Figure 1 As shown, Figure 1 This is a flowchart of a position encoding method provided in an embodiment of this application. The position encoding method includes:

[0027] S101. Obtain the location encoding matrix and the data to be processed; the location encoding matrix includes elements whose feature values ​​are preset values.

[0028] For example, such as Figure 2 As shown, Figure 2 This is a schematic diagram of a position encoding matrix provided in an embodiment of this application. Figure 2 Chinese R m R is a constant matrix representing the RoPE operator in the positional coding matrix. m It is a two-dimensional matrix, including d 2 There are elements, each corresponding to a feature value, which are... Figure 2 It can be seen from R m Most of the elements in the array are 0 (i.e., the characteristic value is a preset value).

[0029] The data to be processed is a vector that needs to be positionally encoded. Its format needs to match the positional encoding matrix. The format of the data to be processed should be a one-dimensional matrix of length d, so as to match R. m Perform multiplication. Figure 2 From q0 to q d-1 Let Q represent the data to be processed. If the format of the data to be processed is not a one-dimensional matrix of length d, then the data needs to be processed by converting it into a one-dimensional matrix of length d through methods such as neural network models or feature extraction algorithms.

[0030] S102. Filter out the elements in the position coding matrix whose feature values ​​are preset values ​​to obtain the target matrix; the target matrix includes the elements in the position coding matrix excluding the elements whose feature values ​​are preset values.

[0031] like Figure 2 As shown, in large language models, when using the RoPE operator to perform positional encoding on the data to be processed, related technologies involve using R... m Multiply by the data to be processed to obtain the result. (This is because R...) m Including multiple elements with eigenvalues ​​of 0, therefore, directly using R... mPerforming matrix multiplication to implement position encoding results in significant resource waste, leading to substantial storage bandwidth requirements and the need for additional computing chips for acceleration. The storage bandwidth can be the bandwidth of Double Data Rate (DDR) synchronous dynamic random access memory.

[0032] Based on this, R is filtered out in the embodiments of this application. m Elements with eigenvalues ​​of 0 are retained in R. m The target matrix is ​​obtained by finding the non-zero eigenvalues ​​of R. m Elements excluding those with a characteristic value of 0.

[0033] In this embodiment of the application, due to R m The position and eigenvalue of each element in R are determined; therefore, in one implementation, it can be based on R. m Based on the arrangement pattern of elements, elements with non-zero eigenvalues ​​are extracted, or elements with zero eigenvalues ​​are filtered out. For example, based on the above... Figure 2 R m Given a d-row, d-column two-dimensional matrix, extract the elements from the first and second columns of both rows: cos mθ0, -sin mθ0, sin mθ0, cos mθ0; extract the elements from the third and fourth columns of both rows: cos mθ1, -sin mθ1, sin mθ1, cos mθ1; and so on, extracting the elements from the (d-1)th and dth columns of both rows: cos mθ0, -sin mθ1, sin mθ1, cos mθ1; and so on, extracting the elements from the (d-1)th and dth columns of both rows: cos mθ0, -sin mθ1, sin mθ1, cos mθ1, and so on. d / 2-1 -sin mθ d / 2-1 sin mθ d / 2-1 cos mθ d / 2-1 Thus, the target matrix is ​​obtained. In another implementation, for R... m The elements in the matrix are transformed, and the elements of the transformed matrix are truncated according to the truncation step size to obtain the target matrix. This application embodiment does not impose specific restrictions on the implementation method for retaining elements with non-zero eigenvalues, as long as elements with eigenvalues ​​of 0 can be filtered out. This application embodiment does not impose any restrictions on this aspect.

[0034] S103. Perform convolution on the target matrix and the data to be processed to obtain the processing result; the processing result is the result of position encoding the data to be processed according to the position encoding matrix.

[0035] The target matrix and the data to be processed are convolutionally processed using convolutional layers or operators of a large language model. For example, the target matrix and the data to be processed can be pre-processed into features of the same dimension, and then convolution is performed after feature fusion to obtain the processing result. This application does not impose requirements on the convolutional layers and operators; for example, group convolution can be performed after feature fusion, as long as the processing result is consistent with the result after positionally encoding the data to be processed according to the positional encoding matrix.

[0036] A large language model is essentially a neural network processor within a neural network chip, adept at handling convolution operations. Accelerating the processing power of neural network chips often involves speeding up convolution operations. Based on this, in this embodiment, elements with a feature value of 0 in the RoPE operator are filtered out, effectively reducing redundant computation. This transforms the RoPE operator into a convolution operation logic that a neural network processor excels at handling. This maximizes the utilization of the neural network processor's computational power, and the convolution operation logic is compatible with the acceleration capabilities of the neural network chip. Furthermore, it eliminates the need for structural modifications or retraining of the large language model, thus improving the acceleration effect of the RoPE operator.

[0037] It should be noted that the embodiments of this application propose an optimized implementation of the RoPE operator on a neural network processor, which improves the acceleration efficiency of the neural network processor. Any scheme that filters out elements with a feature value of 0 in the RoPE operator and converts the process of position encoding using the RoPE operator into convolution to achieve position encoding is within the protection scope of this application.

[0038] According to the scheme provided in this application, a positional encoding matrix and data to be processed are obtained. The positional encoding matrix includes elements with preset eigenvalues. The preset eigenvalues ​​can be 0. The positional encoding matrix includes multiple elements with eigenvalues ​​of 0. If the positional encoding matrix is ​​directly multiplied by the data to be processed to achieve positional encoding, there will be a significant waste of resources, requiring a large amount of storage space and occupying additional chip computing acceleration. Based on this, the elements with preset eigenvalues ​​in the positional encoding matrix are filtered out to obtain a target matrix. The target matrix includes all elements in the positional encoding matrix except for those with preset eigenvalues. This effectively reduces redundant calculations and lowers the amount of data processed. Then, convolution processing is performed on the target matrix and the data to be processed to obtain the processing result. Typically, large language models use positional encoding matrices to encode the data to be processed to obtain the processing result. Since the most efficient computing unit of a neural network chip is good at convolution operations, in this embodiment, the process of positional encoding using the positional encoding matrix is ​​converted into convolution processing. Moreover, it does not require structural modifications or retraining of the large language model, improving processing efficiency and enabling efficient operation of the large language model containing the positional encoding matrix.

[0039] In some embodiments, based on Figure 1 ,like Figure 3 As shown, Figure 3 This is a flowchart of another position encoding method provided in the embodiments of this application.

[0040] S201. Obtain the location encoding matrix and the data to be processed; the location encoding matrix includes elements whose feature values ​​are preset values.

[0041] S201 and the above Figure 1 The implementation method and the technical effects that can be achieved are consistent with S101, and will not be repeated here.

[0042] S202. Perform dimensional transformation on the position encoding matrix to obtain the first feature.

[0043] In this embodiment, a dimension transformation function is used to perform dimension transformation on the RoPE operator to obtain the first feature. The dimension transformation function can readjust the number of rows, columns, and dimensions of a matrix to transform a specified matrix into a matrix of a specific dimension (i.e., a preset dimension matrix) while keeping the number of elements in the specified matrix unchanged.

[0044] Dimension transformation functions are commonly used data reshaping tools that can flexibly adjust and reshape data. They are used to adjust and reshape data structures such as arrays (or matrices) without changing the number or data type of elements in the original array; they only change the shape of the array and return a new, reshaped array. An array can be transformed into an array of any shape, for example, a two-dimensional array into a three-dimensional array, or vice versa. When using dimension transformation functions, it is essential to ensure that the product of elements in the reshaped array is equal to the product of elements in the original array to prevent element loss after rearrangement.

[0045] The dimension transformation function can be a ReShape function, a view function, a permute function, a stack function, etc., as long as it can change the shape of the array or matrix while keeping the total number of elements unchanged. This application does not limit this.

[0046] For example, taking the ReShape function as the dimension transformation function, the position encoding matrix R... m Perform a dimensionality transformation on the two-dimensional matrix R. m This is converted to a three-dimensional matrix, meaning the first feature R1 is a three-dimensional matrix. The first feature includes d. 2 Each element, and the position encoding matrix R m The number of elements included is consistent. First feature R1 = ReShape(R m , (d^2 / 4, 2, 2)), where d^2 / 4, 2, and 2 represent the lengths of each dimension.

[0047] S203. Determine the truncation step size based on the number of encoding parameters in the position encoding matrix.

[0048] In this embodiment, the encoding parameter refers to the element in the position encoding matrix whose feature value is not a preset value. As described above. Figure 2 The encoding parameter is R m Elements that are not 0, for example, cos mθ0, -sin mθ0, sin mθ0, cos mθ0, cos mθ1, -sin mθ1, sin mθ1, cos mθ1...cos mθ d / 2-1 -sin mθ d / 2-1 sin mθ d / 2-1 cos mθ d / 2-1 The number of encoding parameters is d, where d is an even number. Based on the number of encoding parameters, the truncation step size is determined to extract elements whose feature values ​​are not preset values.

[0049] In some embodiments, the truncation step size is the number of encoded parameters divided by 2 plus 1, that is, the truncation step size is d / 2+1. Using this truncation step size can extract all encoded parameters in the first feature, so as to retain elements whose feature values ​​are not preset values, thereby effectively reducing redundant calculations and reducing the amount of data processing in subsequent convolution processing.

[0050] S204. According to the truncation step size, start truncation from the starting position of the first feature until the last position of the first feature to obtain the target matrix.

[0051] The target matrix includes at least the first element at the starting position of the first feature, the second element at a distance of the starting position equal to the truncation step size, the third element at a distance of the second element equal to the truncation step size, and the last element at the last position of the first feature.

[0052] In this embodiment, according to the truncation step size, truncation begins from the starting position of the first feature. First, the first element at the starting position of the first feature is truncated. Then, the second element, which is truncation step size away from the starting position, is truncated (i.e., the second element is 1 truncation step size away from the starting position). Then, the third element, which is truncation step size away from the second element, is truncated (i.e., the third element is 2 truncation step sizes away from the starting position). Then, the fourth element, which is truncation step size away from the third element, is truncated (i.e., the fourth element is 3 truncation step sizes away from the starting position). This process continues until the element at the last position of the first feature is truncated. The elements truncated above are used as the target matrix.

[0053] For example, taking the ReShape function as the dimension transformation function, the first feature R1 is truncated starting with a truncation step of d / 2+1 to obtain the target matrix R2, which includes 2d elements. The target matrix R2 = R1[S:T:d / 2+1], where S represents the starting position of the first feature R1 and T represents the ending position of the first feature R1.

[0054] In some embodiments, the number of elements in the target matrix is ​​the product of the number of encoding parameters and 2. Here, encoding parameters refer to the elements in the positional encoding matrix whose eigenvalues ​​are not preset values, the number of encoding parameters is d, and the number of elements in the target matrix is ​​2d. By retaining elements whose eigenvalues ​​are not preset values ​​in the target matrix, redundant computation can be effectively reduced and the amount of data processed in subsequent convolutional processing.

[0055] S205. Perform convolution on the target matrix and the data to be processed to obtain the processing result; the processing result is the result of position encoding the data to be processed according to the position encoding matrix.

[0056] S205 and the above Figure 1The implementation method and the technical effects that can be achieved are the same as those in S103, and will not be repeated here.

[0057] In this embodiment, when extracting elements whose feature values ​​are not preset values ​​from the positional encoding matrix, the positional encoding matrix is ​​first transformed to obtain a first feature for subsequent element extraction. The extraction step size is determined based on the number of elements in the positional encoding matrix whose feature values ​​are not preset values, thus extracting these elements. Then, according to the extraction step size, extraction (also known as slicing) is performed from the first feature to obtain the target matrix. By implementing the extraction process through dimensionality transformation and defining the extraction step size, redundant computations and data processing volume can be effectively reduced in subsequent convolutional processing.

[0058] The RoPE operator is essentially a function of the input vector Q (i.e., the data to be processed, q0 to q1). d-1 In this embodiment, dimensional transformation, element truncation, and convolution operations are used to replace the computation process of the RoPE operator. Dimension transformation only changes the data shape, and truncation (also known as slicing) does not involve additional computation. Only convolution itself involves computation, but by filtering out elements with preset feature values, there is no redundant computation in the convolution operation, thus improving processing efficiency.

[0059] In general, most accelerator chips are more efficient at convolution calculations. Therefore, the embodiments of this application are applicable to the computing mode of artificial intelligence (AI) accelerator chips.

[0060] In some embodiments, based on Figure 3 ,like Figure 4 As shown, Figure 4 This is a flowchart of another position encoding method provided in the embodiments of this application.

[0061] S301. Obtain the location encoding matrix and the data to be processed; the location encoding matrix includes elements whose feature values ​​are preset values.

[0062] S302. Perform dimensional transformation on the position encoding matrix to obtain the first feature.

[0063] S303. Determine the truncation step size based on the number of encoding parameters in the position encoding matrix.

[0064] S304. According to the truncation step size, start truncation from the starting position of the first feature until the last position of the first feature to obtain the target matrix.

[0065] S301-S304 and the above Figure 3The implementation methods and achievable technical effects are consistent with those in S201-S204, and will not be repeated here.

[0066] S305. Perform dimensional transformation on the data to be processed to obtain the second feature.

[0067] In this embodiment of the application, a dimension transformation function is used to transform the dimensions of the data to be processed to obtain the second feature.

[0068] For example, the dimension transformation function is the ReShape function, and the data to be processed, Q, is... Figure 2 q0 to q d-1 For example, we perform a dimensionality transformation on the data Q to be processed, converting the one-dimensional matrix Q into a four-dimensional matrix, that is, the second feature Q1 is a four-dimensional matrix. The second feature includes d elements, which is the same as the number of elements included in the data Q to be processed. The second feature Q1 = ReShape(Q, (1, d / 2, 1, 2)), where 1, d / 2, 1, and 2 represent the length of each dimension.

[0069] It should be noted that, Figure 4 This explanation only uses the example of executing S302-S304 first, and then S305. After S301, S305 can be executed first, and then S302-S304 can be executed; or S305 and S302-S304 can be executed simultaneously. This application embodiment does not restrict the execution order of S302-S304 and S305.

[0070] In some embodiments, the number of encoding parameters of the position encoding matrix is ​​d, where d is an even number, and the position encoding matrix is ​​a d×d two-dimensional feature; the above S305 can also be implemented in the following way: Extract features from the data to be processed to obtain the features to be processed; the features to be processed are d×1 one-dimensional features; perform a dimension transformation on the features to be processed according to a dimension transformation function to obtain a second feature; the dimension transformation function is used to transform a specified matrix into a matrix of a preset dimension, while keeping the number of elements in the specified matrix unchanged.

[0071] The data to be processed can be raw data such as images, text, sound, color, and files. It needs to be converted into features that match the positional encoding matrix. For example, if the positional encoding matrix is ​​a d×d two-dimensional matrix, the data to be processed needs to be converted into a d×1 one-dimensional matrix. Based on this, features are extracted from the data to be processed using neural network models or feature extraction algorithms to obtain d×1 one-dimensional features. Then, a dimension transformation function is used to transform the dimensions of the features to be processed to obtain the second feature.

[0072] In this embodiment, for data whose format does not match the positional encoding matrix, the data can be processed to make the resulting features compatible with the positional encoding matrix. This embodiment does not require a specific format for the data to be processed; any data format is acceptable, thus expanding the applicability of positional encoding.

[0073] S306. Perform dimensional transformation on the target matrix to obtain the third feature; the third feature has the same dimension as the second feature.

[0074] Since the target matrix is ​​obtained by truncating the first feature, a dimension transformation function is needed to transform the target matrix to obtain the third feature.

[0075] For example, taking the ReShape function as the dimension transformation function and the target matrix as R2 = R1[S:T:d / 2+1], we perform a dimension transformation on the target matrix R2 to convert it into a four-dimensional matrix. That is, the third feature R3 is a four-dimensional matrix. The third feature includes 2d elements, which is the same number of elements included in the target matrix R2. The third feature R3 = ReShape(R2, (d, 1, 1, 2)), where d, 1, 1, 2 represent the length of each dimension.

[0076] S307. Perform convolution processing on the second and third features to obtain the processing result.

[0077] The second and third features can be convolved using the convolutional layers or convolutional operators of the large language model. For example, the second and third features can be fused and then convolved to obtain the processing result, which is consistent with the result of position encoding the data to be processed according to the position encoding matrix.

[0078] In this embodiment, after the target matrix is ​​captured, the dimensions of the data to be processed and the target matrix are transformed to obtain the second feature and the third feature respectively. The dimension transformation makes the dimension of the third feature consistent with that of the second feature, so that the second feature and the third feature can be convolved to obtain the processing result, thereby improving the processing efficiency and realizing the efficient operation of a large language model containing the position encoding matrix.

[0079] In some embodiments, S307 may further include the following steps: fusing the second feature and the third feature to obtain a fused feature, wherein the number of channels of the fused feature is the number of encoding parameters of the position coding matrix divided by 2; grouping the fused feature to obtain multiple grouped features; wherein the number of multiple grouped features is the number of channels of the fused feature; performing convolution processing on each grouped feature to obtain the processed data corresponding to each grouped feature; wherein the processed data corresponding to each grouped feature includes data from two channels; taking the processed data corresponding to each of the multiple grouped features as the processing result; wherein the processing result includes data from multiple channels, wherein the number of multiple channels is the number of encoding parameters of the position coding matrix.

[0080] In this embodiment, taking the second feature Q1 = ReShape(Q, (1, d / 2, 1, 2)) and the third feature R3 = ReShape(R2, (d, 1, 1, 2)) as an example, both the second and third features are four-dimensional matrices. The second and third features are fused, for example, by concatenating them, to obtain a fused feature. The number of encoding parameters is d, and the number of channels in the fused feature is d / 2, meaning the fused feature includes d / 2 channels. The fused feature is grouped into d / 2 groups, resulting in d / 2 grouped features. For each group, the grouped features are convolved to obtain the corresponding processed data, and so on, to obtain the processed data for each group. The processed data of d / 2 groups is used as the processing result, with each group's processed data including two channels of data; thus, the processing result includes data with d channels.

[0081] The convolution calculation performed on the fused features in this embodiment can be group convolution, a special type of convolution operation. The fused features are grouped along the channel dimension. The fused features include d / 2 channels, and the input d / 2 channels are divided into d / 2 groups, with one channel corresponding to one group. Weights are not shared between groups. After convolution processing of the grouped features using convolution kernels, each group outputs two channels of data, resulting in d channels of data. By performing convolution operations independently on each group, the computational load and number of parameters can be effectively reduced while maintaining the model's expressive power, thus improving computational efficiency.

[0082] In some embodiments, the step of performing convolution processing on each group feature to obtain the processed data corresponding to each group feature can be implemented in the following way: Perform convolution processing on each group feature according to a preset convolution kernel to obtain the processed data corresponding to each group feature; the size of the preset convolution kernel is 1×2.

[0083] In this embodiment, the calculation process of the RoPE operator is equivalent to a convolution operation. The processing result is Result = Conv2D(Q1, R3, group = d / 2, channels = d, kernel_size = (1, 2)), where Conv2D represents the convolution function, Q1 represents the second feature, R3 represents the third feature, group represents the group, each group includes d / 2 channels, channels = d means that d channels are output, and kernel_size = (1, 2) means that the size of the convolution kernel is 1×2.

[0084] For each group's grouped features, a 1×2 convolution kernel is used to perform convolution operations on the grouped features, enabling each group to output two channels of data. This results in a combined output of d channels of data, consistent with the processing result after positional encoding of the data to be processed using the RoPE operator. Thus, this embodiment of the application utilizes efficient convolution operations to achieve positional encoding, improving processing efficiency and enabling efficient computation of large language models containing this positional encoding matrix.

[0085] In some embodiments, the second feature includes d elements, and the first feature includes d elements. 2 The first feature has 2d elements, the second feature has 2d elements, the third feature has 2d elements, the second feature and the third feature are both four-dimensional features, and the first feature is a three-dimensional feature.

[0086] In this embodiment of the application, the first feature is obtained by performing a dimensional transformation on the position transformation matrix, and the first feature includes d. 2 The first feature is a 3D feature, which allows for the extraction of elements with non-preset feature values ​​from the 3D feature to obtain the target matrix. This effectively reduces redundant computation and data processing volume in subsequent convolutional processing. The second feature is obtained after dimensionality transformation of the data to be processed, and it includes d elements. The third feature is obtained after dimensionality transformation of the target matrix, and it has 2d elements. Both the second and third features are 4D features, which allow for fusion of the second and third features before convolution. Utilizing convolutional operations improves processing efficiency, enabling efficient computation of large language models containing the location encoding matrix.

[0087] Verification has confirmed that the computational sequence of the data to be processed using the positional encoding method provided in this application is equivalent to the positional encoding sequence of the data to be processed using the RoPE operator. Thus, this application's embodiments utilize efficient convolution operations to achieve positional encoding without requiring structural modifications or retraining of the large language model, thereby improving processing efficiency and enabling efficient computation of a large language model containing the positional encoding matrix.

[0088] Based on the location encoding method provided in the above embodiments Figure 5 This is a schematic diagram of a location encoding device provided in an embodiment of this application. This device can be implemented as part or all of a computer device by software, hardware, or a combination of both. See also... Figure 5 The position encoding device 50 includes: an acquisition module 501 for acquiring a position encoding matrix and data to be processed; the position encoding matrix includes elements with preset eigenvalues; a filtering module 502 for filtering out the elements with preset eigenvalues ​​in the position encoding matrix to obtain a target matrix; the target matrix includes elements in the position encoding matrix excluding the elements with preset eigenvalues; and a convolution module 503 for performing convolution processing on the target matrix and the data to be processed to obtain a processing result; the processing result is the result of position encoding the data to be processed according to the position encoding matrix.

[0089] Optionally, the position encoding device 50 further includes a dimension transformation module 504;

[0090] The dimension transformation module 504 is used to perform dimension transformation on the position encoding matrix to obtain the first feature;

[0091] The filtering module 502 is further configured to determine the truncation step size based on the number of encoding parameters in the position encoding matrix; and to truncate from the starting position in the first feature to the last position in the first feature according to the truncation step size to obtain the target matrix; wherein the target matrix includes at least the first element at the starting position in the first feature, the second element at a distance of the starting position equal to the truncation step size, the third element at a distance of the second element equal to the truncation step size, and the last element at the last position in the first feature.

[0092] Optionally, the truncation step size is the number of encoding parameters divided by 2 plus 1, and the number of elements in the target matrix is ​​the product of the number of encoding parameters and 2.

[0093] Optionally, the dimension transformation module 504 is also used to perform dimension transformation on the data to be processed to obtain a second feature; perform dimension transformation on the target matrix to obtain a third feature; and the third feature has the same dimension as the second feature.

[0094] The convolution module 503 is also used to perform convolution processing on the second and third features to obtain the processing result.

[0095] Optionally, the convolution module 503 is further configured to fuse the second feature and the third feature to obtain a fused feature, wherein the number of channels of the fused feature is the number of encoding parameters of the positional encoding matrix divided by 2; to group the fused feature to obtain multiple grouped features; wherein the number of multiple grouped features is the number of channels of the fused feature; to perform convolution processing on each grouped feature to obtain the processed data corresponding to each grouped feature; wherein the processed data corresponding to each grouped feature includes data with two channels; and to take the processed data corresponding to each of the multiple grouped features as the processing result; wherein the processing result includes data with multiple channels, wherein the number of multiple channels is the number of encoding parameters of the positional encoding matrix.

[0096] Optionally, the convolution module 503 is also used to perform convolution processing on each group feature according to a preset convolution kernel to obtain the processed data corresponding to each group feature; the size of the preset convolution kernel is 1×2.

[0097] Optionally, the number of coding parameters of the position coding matrix is ​​d, where d is an even number, and the position coding matrix is ​​a two-dimensional feature of d×d.

[0098] The dimension transformation module 504 is also used to extract features from the data to be processed to obtain the features to be processed; the features to be processed are d×1 one-dimensional features; the dimension transformation function is used to transform the features to be processed to obtain the second features; the dimension transformation function is used to transform a specified matrix into a matrix of a preset dimension, while keeping the number of elements in the specified matrix unchanged.

[0099] Optionally, the second feature includes d elements, and the first feature includes d elements. 2 The first feature has 2d elements, the second feature has 2d elements, the third feature has 2d elements, the second feature and the third feature are both four-dimensional features, and the first feature is a three-dimensional feature.

[0100] It should be noted that the location encoding device provided in the above embodiments is only illustrated by the division of the above functional modules when performing location encoding on features. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0101] The functional units and modules in the above embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of the embodiments of this application.

[0102] The location encoding device and location encoding method provided in the above embodiments belong to the same concept. The specific working process and technical effects of the units and modules in the above embodiments can be found in the method embodiments section, and will not be repeated here.

[0103] Based on the location encoding method provided in the above embodiments Figure 6 This application provides a schematic diagram of the structure of a computer device, as shown in the embodiment of the present application. Figure 6 As shown, the computer device 60 includes: a processor 601, a memory 602, and a computer program 603 stored in the memory 602 and executable on the processor 601. When the processor 601 executes the computer program 603, it implements the steps in the position encoding method in the above embodiments.

[0104] Computer device 60 can be a general-purpose computer device or a special-purpose computer device. In specific implementations, computer device 60 can be a desktop computer, portable computer, network server, handheld computer, mobile phone, tablet computer, wireless terminal device, communication equipment, or embedded device. This application embodiment does not limit the type of computer device 60. Those skilled in the art will understand that... Figure 6 The computer device 60 is merely an example and does not constitute a limitation on the computer device 60. It may include more or fewer components than shown, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0105] Processor 601 can be a Central Processing Unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0106] In some embodiments, memory 602 may be an internal storage unit of computer device 60, such as a hard disk or RAM of computer device 60. In other embodiments, memory 602 may be an external storage device of computer device 60, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on computer device 60. Furthermore, memory 602 may include both internal and external storage units of computer device 60. Memory 602 is used to store operating systems, applications, boot loaders, data, and other programs. Memory 602 may also be used to temporarily store data that has been output or will be output.

[0107] This application also provides a computer device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the steps in any of the above method embodiments.

[0108] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps in the various method embodiments described above.

[0109] This application provides a computer program product that, when run on a computer, causes the computer to perform the steps described in the various method embodiments above.

[0110] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above method embodiments of this application can be implemented by a computer program instructing related hardware. This computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate form. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographing device / terminal device, a recording medium, a computer memory, ROM (Read-Only Memory), RAM (Random Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk, and optical data storage devices. The computer-readable storage medium mentioned in this application can be a non-volatile storage medium; in other words, it can be a non-transient storage medium.

[0111] It should be understood that all or part of the steps of the above embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented in whole or in part as a computer program product. The computer program product includes one or more computer instructions. The computer instructions can be stored in the above-described computer-readable storage medium.

[0112] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0113] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0114] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A position encoding method, characterized in that, The method includes: Obtain the location encoding matrix and the data to be processed; the location encoding matrix includes elements whose feature values ​​are preset values. Elements with feature values ​​of preset values ​​in the location encoding matrix are filtered out to obtain a target matrix; the target matrix includes all elements in the location encoding matrix except for those with feature values ​​of preset values. The target matrix and the data to be processed are convolved to obtain the processing result; the processing result is the result of position encoding the data to be processed according to the position encoding matrix.

2. The method as described in claim 1, characterized in that, The method further includes: The position encoding matrix is ​​dimensionally transformed to obtain the first feature; The step of filtering out elements with preset feature values ​​from the position encoding matrix to obtain the target matrix includes: The truncation step size is determined based on the number of encoding parameters in the position encoding matrix. According to the stated truncation step size, starting from the beginning position of the first feature, truncation continues until the end position of the first feature to obtain the target matrix; The target matrix includes at least a first element at the starting position of the first feature, a second element at a distance of the starting position equal to the truncation step length, a third element at a distance of the second element equal to the truncation step length, and the last element at the last position of the first feature.

3. The method as described in claim 2, characterized in that, The truncation step size is the number of encoding parameters divided by 2 and then added by 1, and the number of elements included in the target matrix is ​​the product of the number of encoding parameters and 2.

4. The method according to any one of claims 1-3, characterized in that, The method further includes: The data to be processed is subjected to dimensional transformation to obtain the second feature; The target matrix is ​​subjected to dimensional transformation to obtain a third feature; the third feature has the same dimension as the second feature. The convolution process performed on the target matrix and the data to be processed to obtain the processing result includes: The second feature and the third feature are convolved to obtain the processing result.

5. The method as described in claim 4, characterized in that, The convolution process performed on the second feature and the third feature to obtain the processing result includes: The second feature and the third feature are fused to obtain a fused feature, wherein the number of channels of the fused feature is the number of encoding parameters of the position encoding matrix divided by 2; The fused features are grouped to obtain multiple grouped features; the number of multiple grouped features is equal to the number of channels of the fused features. Each group feature is convolved to obtain the processed data corresponding to each group feature; the processed data corresponding to each group feature includes data from two channels. The processing data corresponding to each of the multiple grouping features is taken as the processing result; the processing result includes data from multiple channels, the number of which is the number of encoding parameters of the position encoding matrix.

6. The method as described in claim 5, characterized in that, The step of performing convolution processing on each group feature to obtain the processed data corresponding to each group feature includes: The pre-defined convolution kernel is used to perform convolution processing on each group of features to obtain the processed data corresponding to each group of features; the size of the pre-defined convolution kernel is 1×2.

7. The method as described in claim 4, characterized in that, The number of encoding parameters of the position encoding matrix is ​​d, where d is an even number, and the position encoding matrix is ​​a two-dimensional feature of d×d. The step of performing dimensional transformation on the data to be processed to obtain the second feature includes: Feature extraction is performed on the data to be processed to obtain the features to be processed; the features to be processed are one-dimensional features of d×1. The second feature is obtained by performing a dimension transformation on the feature to be processed according to the dimension transformation function; the dimension transformation function is used to transform a specified matrix into a matrix of a preset dimension, while the number of elements in the specified matrix remains unchanged.

8. A position encoding device, characterized in that, The device includes: The acquisition module is used to acquire the location encoding matrix and the data to be processed; the location encoding matrix includes elements whose feature values ​​are preset values; A filtering module is used to filter out elements in the position encoding matrix whose feature values ​​are preset values ​​to obtain a target matrix; the target matrix includes all elements in the position encoding matrix except for those whose feature values ​​are preset values. The convolution module is used to perform convolution processing on the target matrix and the data to be processed to obtain a processing result; the processing result is the result of position encoding the data to be processed according to the position encoding matrix.

9. A computer device, characterized in that, The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-7.

Images