Data processing method, device, equipment and storage medium

Abstract

The embodiments of the present disclosure provide a data processing method, apparatus, device and computer-readable storage medium. The method provided by the embodiments of the present disclosure is for an input sequence including multiple word-grams, and only uses a part of the word-grams to create a position coding list, and implements the position coding of the entire input sequence based on the position coding list, wherein the word-grams in the input sequence that are not used to create the position coding list can determine their corresponding position coding from the position coding list by looking up a table. The method of the embodiments of the present disclosure can utilize the association between the position codings of word-grams to derive the position coding related to the remaining word-grams from the position coding determined based on a part of the word-grams, thereby avoiding a large number of repeated calculation operations, reducing the waste of storage space and computing resources, improving the position coding processing, and improving the performance of the model in sequence data processing tasks.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and more specifically, to a data processing method, apparatus, device and storage medium. Background Art

[0002] As a revolutionary neural network architecture, the Transformer model abandons the traditional recurrent neural network (RNN) and convolutional neural network (CNN) and adopts the self-attention mechanism when processing sequence data, which enables the model to globally capture the dependencies between different elements. However, the advantages of this structure also bring a problem, that is, the word order information in the sequence is lost. Since the self-attention mechanism does not consider the position of the element in the sequence, it is invariant under any position permutation of the input sequence, which means that the model cannot distinguish the relative position of the elements in the sequence.

[0003] In fact, for any language, the position and order of words in a sentence are very important information. Therefore, in order for the Transformer model to better learn the word order information in sequence data, positional encoding becomes crucial. Positional encoding is used to introduce word order information into the model's processing of sequence data, thereby helping the model understand the relative position relationship of different words in the input sequence.

[0004] Therefore, an effective data processing method is needed to improve the position encoding used for the Transformer model so as to improve the performance of the model in sequence data processing tasks. Summary of the invention

[0005] Embodiments of the present disclosure provide a data processing method, apparatus, device, and computer-readable storage medium.

[0006] An embodiment of the present disclosure provides a data processing method, which determines, for an input sequence including multiple tokens, a position coding list of the input sequence based on a first token set of the multiple tokens, the position coding list including position codings associated with the tokens in the first token set, and for a second token set, the position codings associated with the tokens in the second token set can be determined using the position coding list, the second token set being the complement of the first token set in the multiple tokens; and uses the position coding list to process the input sequence through an attention model.

[0007] An embodiment of the present disclosure provides a data processing device, including: a list creation module, configured to determine, for an input sequence including multiple word-grams, a position coding list of the input sequence based on a first word-gram set of the multiple word-grams, the position coding list including position codings associated with the word-grams in the first word-gram set, and for a second word-gram set, the position codings associated with the word-grams in the second word-gram set can be determined using the position coding list, the second word-gram set being the complement of the first word-gram set in the multiple word-grams; and a sequence processing module, configured to process the input sequence through an attention model using the position coding list.

[0008] An embodiment of the present disclosure provides a data processing device, including: one or more processors; and one or more memories, wherein a computer executable program is stored in the one or more memories, and when the computer executable program is executed by the processor, the data processing method as described above is performed.

[0009] An embodiment of the present disclosure provides a computer-readable storage medium on which computer-executable instructions are stored. When the instructions are executed by a processor, the instructions are used to implement the data processing method as described above.

[0010] The embodiments of the present disclosure provide a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the data processing method according to the embodiments of the present disclosure.

[0011] The method provided by the embodiment of the present disclosure is for an input sequence including multiple word-grams, and only a part of the word-grams is used to create a position coding list, and the position coding of the entire input sequence is implemented based on the position coding list, wherein the word-grams in the input sequence that are not used to create the position coding list can determine their corresponding position coding from the position coding list by looking up a table. The method of the embodiment of the present disclosure can utilize the association between the position codings of word-grams to derive the position coding related to the remaining word-grams from the position coding determined based on a part of the word-grams, thereby avoiding a large number of repeated calculation operations, reducing the waste of storage space and computing resources, improving the position coding processing, and improving the performance of the model in sequence data processing tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some exemplary embodiments of the present disclosure, and a person of ordinary skill in the art can obtain other drawings based on these drawings without creative work.

[0013] Figure 1 is a schematic diagram showing the use of each word unit by the Transformer model when processing sequence data;

[0014] Figure 2 is a flow chart showing a data processing method according to an embodiment of the present disclosure;

[0015] Figure 3 is a schematic diagram illustrating a data processing method according to an embodiment of the present disclosure;

[0016] Figure 4 is a schematic diagram showing a first implementation of creating a position code list based on an input sequence according to an embodiment of the present disclosure;

[0017] Figure 5 is a schematic diagram showing a second implementation of creating a position code list based on an input sequence according to an embodiment of the present disclosure;

[0018] Fig. 6A is a schematic diagram of a position coding matrix showing relative position coding according to an embodiment of the present disclosure;

[0019] Figure 6B is a schematic diagram showing a position code list for creating relative position codes according to an embodiment of the present disclosure;

[0020] Figure 7 is a schematic diagram showing a data processing device according to an embodiment of the present disclosure;

[0021] Figure 8 A schematic diagram showing a data processing device according to an embodiment of the present disclosure; and

[0022] Fig. 9 A schematic diagram illustrating the architecture of an exemplary computing device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present disclosure more obvious, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.

[0024] In this specification and the accompanying drawings, substantially the same or similar steps and elements are represented by the same or similar reference numerals, and repeated descriptions of these steps and elements will be omitted. At the same time, in the description of the present disclosure, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance or ranking.

[0025] In the embodiments of the present disclosure, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present disclosure pertains. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0027] To facilitate description of the present disclosure, concepts related to the present disclosure are introduced below.

[0028] The data processing method disclosed in the present invention may be based on artificial intelligence (AI). Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence.

[0029] The data processing method of the present disclosure can be based on position encoding technology. Position encoding technology is used to help the attention model better understand the relative position relationship between words or symbols when processing sequence data. In the attention mechanism, position encoding is used to provide a unique position information for each input token so that the model can distinguish tokens at different positions and better capture the dependencies between them. Common position encoding techniques may include absolute position encoding and relative position encoding. Absolute position encoding represents the absolute position of each token by adding position information to the input embedding vector, while relative position encoding captures the position relationship between tokens by calculating the relative distance between them.

[0030] In summary, the solutions provided by the embodiments of the present disclosure involve technologies such as artificial intelligence and position coding. The embodiments of the present disclosure will be further described below in conjunction with the accompanying drawings.

[0031] Figure 1 It is a schematic diagram showing the use of each word unit when the Transformer model processes sequence data.

[0032] In sequence data processing, the sequence data to be processed can be transmitted to a server (for example, through a network) for the server to perform data processing, and an attention model (for example, a Transformer model) can be deployed on the server to process the sequence data using the attention mechanism. Among them, the sequence data to be processed can be text data, which can be obtained by performing speech recognition on voice data or video data, or by performing text recognition on image data or video data, and the present disclosure does not limit this. Optionally, the network can be an Internet of Things based on the Internet and / or telecommunications network, which can be a wired network or a wireless network, for example, it can be an electronic network that can realize information exchange functions such as a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a cellular data communication network, etc. The server can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and basic cloud computing services such as big data and artificial intelligence platforms.

[0033] As mentioned above, in the attention model (for example, the Transformer model), RNN and CNN are not used as the basic model for sequence learning. When processing sequence data, its self-attention mechanism enables the model to globally capture the dependencies between different elements. However, since the self-attention mechanism does not consider the position of the element in the sequence, it is invariant under any permutation of the input sequence, which results in the loss of word order information in the original input sequence. The model cannot distinguish the relative positions of elements in the input sequence, nor can it automatically learn the position information of word units. Figure 1 As shown in , for an attention model containing multiple self-attention modules, when an input sequence enters a self-attention module, since all the tokens in the sequence enter and are processed at the same time, if position information is not provided, the same tokens in the sequence have no grammatical and semantic differences for the self-attention module, and they will produce the same output. Figure 1In the input sequence "I think therefore I am...", the word "I" appears twice, and their initial expression vectors (for example, some kind of embedding vector) are the same. In this case, if no position information is added to it, the attention generated by the self-attention module is the same, and the new expression generated by the model is also the same.

[0034] In any language, the position and order of words in a sentence are very important information. In order to help the model understand the order and position information of word units in the input sequence, position encoding needs to be introduced. For example, common position encoding methods currently include: absolute position encoding (sinusoidal), rotational position encoding (RoPE), and relative position encoding (ALiBi). Among them, absolute position encoding and rotational position encoding can be achieved by adding specific position information to the word vector, such as using sine and cosine functions to generate position encoding to retain the relative position information between words and will not be limited by the length of the sequence. By combining position encoding with word vectors, the model can consider the semantic information and position information of words at the same time during the learning process, so as to better capture the context and order in the sequence data. For relative position encoding, the word unit can be input into the self-attention module with its originally used embedding vector, and the self-attention module can then create an encoding vector for each position of the input sequence and combine it with the attention weight vector. In this way, the generation of position encoding is located inside the self-attention module and integrated with the processing process of the self-attention module.

[0035] However, in existing position encoding methods, a large number of calculation operations are often required based on the word length of the input sequence, and there are also a large number of repeated calculation operations in these calculation operations, which leads to a waste of storage space and computing resources. For example, for absolute position encoding and rotation position encoding in the form of trigonometric functions, since trigonometric functions are periodic, as the position increases, the values ​​of the same dimension have periodic changes, and some position encodings can be derived using the sine-cosine identity transformation formula, so there are a large number of repeated calculation operations and space waste in the process. For another example, for the attention linear bias (ALiBi), in the attention mechanism, there are a large number of identical relative positions between attention heads and within attention heads. Therefore, there are a large number of identical calculations in the process of generating the position encoding matrix based on the relative position, and a large number of identical position encodings are stored in the position encoding matrix, resulting in a waste of computing resources and storage space.

[0036] Based on this, the present disclosure provides a data processing method. The method provided by the embodiment of the present disclosure is for an input sequence including multiple word-grams, and only a part of the word-grams is used to create a position coding list, and the position coding of the entire input sequence is implemented based on the position coding list, wherein the word-grams in the input sequence that are not used to create the position coding list can determine their corresponding position coding from the position coding list by looking up a table. The method of the embodiment of the present disclosure can utilize the association between the position codings of word-grams to derive the position coding related to the remaining word-grams from the position coding determined based on a part of the word-grams, thereby avoiding a large number of repeated calculation operations, reducing the waste of storage space and computing resources, improving the position coding processing, and improving the performance of the model in sequence data processing tasks.

[0037] Figure 2 is a flow chart illustrating a data processing method 200 according to an embodiment of the present disclosure. Figure 3 is a schematic diagram illustrating a data processing method according to an embodiment of the present disclosure.

[0038] like Figure 3 As shown, the data processing method of the present disclosure may include two stages: generating position codes and applying position codes. In the generation stage of position codes, the following may be included: Figure 3 The operations shown in 301 and 302 are used to generate a position encoding matrix, and the generated position encoding matrix can be used together with the input sequence in the application stage of the position encoding for the processing 303 of the attention model, so that the attention model can consider the semantic information and position information of the word unit at the same time during the sequence processing, so as to better capture the context and order in the sequence data.

[0039] Therefore, if Figure 2 As shown, in step S201, for an input sequence including multiple word-grams, a position coding list of the input sequence can be determined based on a first word-gram set of the multiple word-grams, and the position coding list can include position codings associated with the word-grams in the first word-gram set, and for a second word-gram set, the position codings associated with the word-grams in the second word-gram set can be determined using the position coding list, and the second word-gram set can be the complement of the first word-gram set among the multiple word-grams.

[0040] Optionally, in the generation stage of the positional coding, a positional coding list (301) may be first created based on the input sequence, and the positional coding list may be used to generate the positional coding matrix. In an embodiment of the present disclosure, the positional coding list may be created using only the first word-gram set of the plurality of word-grams included in the input sequence (that is, a portion of the word-grams in the input sequence) to achieve deduplication processing of the original positional coding matrix calculation.

[0041] Optionally, for a second word-gram set of multiple word-grams, which includes all word-grams in the multiple word-grams except the word-grams in the first word-gram set used to create the position coding list, the position codings associated with these word-grams can be derived from the position coding list, for example, by looking up the position coding list to determine the associated position codings, and then determining the position codings associated with these word-grams.

[0042] Specifically, considering that different position encoding methods have different calculation processes, the process of creating a position encoding list based on the input sequence may be correspondingly different. In an embodiment of the present disclosure, different position encoding methods can be optimized accordingly based on the operation described in step S201 above.

[0043] Optionally, for various position encoding methods, the position encoding can be used as a bias vector to assign different biases to words at different positions in the input sequence. Based on different position encoding methods, the bias vector can be directly superimposed on the input word embedding vector, or it can be superimposed on the intermediate output of the input sequence processed by the attention model for subsequent processing. In the embodiments of the present disclosure, the above-mentioned absolute position encoding, rotational position encoding, and relative position encoding three position encoding methods can be used as examples to introduce the optimization of position encoding by the data processing method of the present disclosure. For example, as shown in the following reference Figure 4-6B However, it should be understood that the data processing method disclosed in the present invention can also be applied to optimize other position encoding methods.

[0044] In step S202, the position encoding list can be used to process the input sequence through an attention model.

[0045] According to an embodiment of the present disclosure, using the position coding list to process the input sequence through the attention model may include: using the position coding list to determine the position coding matrix of the input sequence based on the multiple word-grams, the position coding matrix may include position coding associated with each word-gram in the multiple word-grams; based on the determined position coding matrix of the input sequence, using the attention model to process the input sequence.

[0046] Optionally, after creating the position coding list, the position coding list can be used to complete the position coding of the entire input sequence, that is, the position coding matrix of the input sequence is determined based on the position coding list, and the position coding matrix is ​​applied to the processing of the input sequence by the attention model.

[0047] The position encoding matrix of the input sequence may include a position encoding vector associated with each word unit in the input sequence, and the position encoding vector may be determined by a table lookup operation on the position encoding list as described above.

[0048] Optionally, the operations of steps S201-S202 described above and the following references Figure 4-6B The described processing may be performed by a processor, wherein the computational data involved (e.g., including input data such as input sequences, intermediate data such as position encoding lists, and output data such as computational results, etc.) may be stored in a memory.

[0049] The data processing method of the present disclosure is introduced below using absolute position encoding and rotational position encoding as examples.

[0050] According to an embodiment of the present disclosure, determining a position coding list of the input sequence based on a first word-gram set of the multiple word-grams may include: selecting the first word-gram set from the multiple word-grams and determining a position coding of each word-gram in the first word-gram set; and creating a position coding list of the input sequence based on the position coding of each word-gram in the first word-gram set.

[0051] Optionally, for absolute position encoding and rotational position encoding, a unique vector representation, i.e., a position encoding vector, can be assigned to the position of each word unit. By combining the position encoding of each word unit with the word vector (e.g., for absolute position encoding, the position encoding of each word unit is superimposed on the word vector, and for rotational position encoding, the word vector is rotated using the position encoding of each word unit), the model can take into account the order information of words during processing, thereby better understanding the structure and semantics of the input sequence.

[0052] As an example, for absolute position encoding, the position encoding of each word unit can be superimposed with the word vector to encode the information of each position in the input sequence directly into the model, so that the model can understand the specific position of each word unit in the sequence. For example, sinusoidal position encoding can be used, by using sine and cosine functions of different frequencies, so that the model captures the complex relationship between positions, and these encodings are related to the absolute value of each position in the sequence.

[0053] For example, for an input sequence including seq tokens and a token at position pos (i.e., the posth token), the position encoding of the token can be represented as a d-dimensional vector PE (pos) , where the dimension d of the position encoding vector is the same as the dimension of the word embedding vector.

[0054] Optionally, the d-dimensional vector PE (pos) It can include multiple groups of quantities, each of which can be represented by a pair of trigonometric functions. For example, for the position encoding vector PE of the word unit (pos) The i-th group weight in can be expressed as follows:

[0055]

[0056] Among them, PE∈R seqxd , PE (pos,2i) and PE (pos,2i+1) They respectively represent the 2i and 2i+1th components of the position encoding vector of the word at position pos. Specifically, 2i can represent the even dimensions in the position encoding vector, i.e. the 0th dimension, the 2nd dimension, etc., and 2i+1 can represent the odd dimensions in the position encoding vector, i.e. the 1st dimension, the 3rd dimension, etc.

[0057] Although such position encoding is based on absolute position, its mathematical structure enables it to capture some relative position information. By using a combination of sine and cosine functions to create a code for each position, the wavelengths are arranged in a geometric series, which means that the code for each position is unique. At the same time, the periodic nature of the sine and cosine functions ensures that the encoding relationship between different positions is continuous and smooth. For example, for adjacent positions, the difference in position encoding is small and proportional to the distance between the two, while for positions that are far apart, the difference in position encoding is large and also proportional to the distance between the two. This continuous and smooth relationship allows the model to learn the relative relationship between positions, not just their absolute positions. In addition, since the range of the sine and cosine functions is between [-1,1], the superposition of position encoding training and word embedding vectors will not cause the results to deviate too far and destroy the semantic information of the original word unit.

[0058] Therefore, for the absolute position code represented in the above example form, according to the characteristics of the sine and cosine functions, the position code PE at the position pos+k (pos+k) Can be made by PE (pos) Linear representation, that is, the positional encoding can express the relative position relationship between the word units at these two positions. Since the sine function sin and the cosine function cos are typical periodic functions, when the dimension i is fixed, for different positions pos, their period is 10000 2i / d *2π. Therefore, as the dimension i increases from 0 to , the period increases from 10000 to 2i / d *2π gradually increases to 10000*2π.

[0059] For example, the position encoding vector for position pos=0,1,2 can be expressed in matrix form as follows:

[0060]

[0061] Among them, each row of the matrix corresponds to a position encoding vector of a position pos.

[0062] As another example, for rotational position encoding, it is actually a method of using absolute position encoding to represent relative position encoding, which is used to integrate the relative position information of the input sequence into the attention model. Specifically, in rotational position encoding, the word embedding vector q corresponding to the mth word in the input sequence is m (as the query vector), and the word embedding vector k corresponding to the nth word n (as key vectors), which respectively integrate the position information R m and R n The vector f q (q m ,m) and f k (k n ,n) can be expressed as follows:

[0063]

[0064]

[0065] Where f(*,t) means adding absolute position information R to * t The function of

[0066] Therefore, through the above representation, the query vector q m and key vector k n Both introduce position information, and their inner product g(q m ,k n ,mn) also contains relative position information R m-n , as shown below:

[0067]

[0068] Therefore, based on the matrix form of the function f(*,t) and the property that the inner product of a vector satisfies linear superposition, any even-dimensional RoPE can be expressed as a concatenation of the above two-dimensional cases, that is, f q (q m ,m) can be expressed as follows:

[0069]

[0070] Among them, the matrix on the left side of formula (6) is R m , which is an orthogonal matrix. The matrix on the right side of formula (6) represents d word embedding vectors.

[0071] Therefore, based on formula (6), due to R m The sparsity of , RoPE can be achieved in the following way, that is, f q (q m,m) can be expressed as follows:

[0072]

[0073] in Represents a bitwise multiplication operation.

[0074] For example, for a word at position m=1, the operation of performing rotation position encoding on it can be expressed as follows:

[0075]

[0076] For the word at position m=2, the operation of rotating the word can be expressed as follows:

[0077]

[0078] For the word at position m=3, the operation of rotating the word can be expressed as follows:

[0079]

[0080] The rotational position encoding of word units at other positions can be deduced in the same way.

[0081] In the absolute position coding (sinusoidal position coding) as described above, the position coding matrix PE∈R seqxd , and for each element in the matrix, trigonometric function calculation operations need to be performed position by position. Therefore, for absolute position encoding, the amount of calculation can be expressed as: f(t,i)*seq*d, and the storage space occupied can be expressed as: seq*d*sizeof(T), where f(t,i) represents trigonometric sine and cosine functions, and sizeof(T) represents the number of bytes of the data type of the element.

[0082] In the rotational position encoding (RoPE) described above, the position encoding matrix PE∈R seqxd , and for each element in the matrix, trigonometric functions need to be calculated position by position, and then superimposed on the word vector. Therefore, for the rotation position encoding, the amount of calculation can be expressed as: f(t,i)*seq*d, and the storage space occupied can be expressed as: seq*2*d*sizeof(T).

[0083] However, for sinusoidal position coding and rotational position coding, since trigonometric functions are periodic, as the position in the position coding matrix increases, the position coding value in the same dimension actually has a periodic change. Therefore, in the embodiments of the present disclosure, the sine-cosine identity transformation formula can be fully utilized to optimize the data processing of sinusoidal position coding and rotational position coding.

[0084] According to an embodiment of the present disclosure, selecting the first word-gram set from the multiple word-grams may include: selecting the first word-gram set from the multiple word-grams according to periodic changes between position codes of different word-grams.

[0085] Optionally, word units that can be used to derive position codes of other word units can be extracted from multiple word units by analyzing the periodic changes of position codes of different word units on the same dimension according to the position changes of the word units. These word units can constitute a first word unit set.

[0086] According to an embodiment of the present disclosure, selecting the first word-gram set from the multiple word-grams may include: when position encoding the input sequence using a trigonometric function, selecting the first word-gram set from the multiple word-grams according to periodic changes in the position encodings of different word-grams on the same dimension according to changes in the positions of the word-grams.

[0087] Optionally, for position coding using trigonometric functions, for example, for absolute position coding as described above, according to the characteristics of the sine and cosine functions, the position coding PE at position pos+k (pos+k) Can be made by PE (pos) Linear representation, and when the dimension i is fixed, for different positions pos, its period is 10000 2i / d *2π, thus, word units at multiple positions can be extracted periodically and used as the first word unit set.

[0088] As an example, for the above absolute position encoding and rotation position encoding, a word unit with an odd position index can be selected from multiple word units to construct a first word unit set. Thus, the position encoding of the word unit with an even position index can be derived from the position encoding of the word unit with an odd position index according to the sine-cosine identity transformation, and then the position encoding matrix of the input sequence is determined.

[0089] Figure 4 is a schematic diagram showing a first implementation of creating a position code list based on an input sequence according to an embodiment of the present disclosure. Figure 4 As shown, for an input sequence including seq tokens, a position encoding list can be constructed based on the tokens with odd position indices and their position encodings.

[0090] For example, for a sinusoidal positional encoding with seq=6 and d=512, the positional encoding matrix can be expressed as follows:

[0091]

[0092] That is to say, when the original sinusoidal position coding is used for position coding, all elements in the position coding matrix need to be calculated, so the calculation amount is f(t,i)*6*512, and the storage space occupied is 6*512*sizeof(T).

[0093] For the above example, by periodically changing the position codes of different word units according to the position changes of the word units on the same dimension, word units with odd position indexes can be selected from multiple word units to construct a first word unit set, and the position code of each word unit in the first word unit set can be calculated.

[0094] The position encoding of each word in the first word set can be expressed as follows:

[0095]

[0096] Therefore, by extracting the first word-unit set and calculating the position code, the calculation of the position code can be deduplicated. In this case, the amount of calculation is f(t,i)*3*512, and the storage space occupied is 3*512*sizeof(T).

[0097] For another example, for rotational position coding with seq=6 and d=512, the position coding matrix can be expressed as follows:

[0098]

[0099] The computational complexity is f(t,i)*6*512, and the storage space occupied is 6*512*2*sizeof(T).

[0100] Similarly, by periodically changing the position codes of different word units according to the position changes of the word units on the same dimension, word units with odd position indexes can be selected from multiple word units to construct a first word unit set, and the position code of each word unit in the first word unit set can be calculated.

[0101] For the first word set, the matrix formed by the corresponding position encoding can be expressed as follows:

[0102]

[0103] Therefore, the amount of calculation for calculating only the position encoding of the first word-unit set is f(t,i)*3*512, and the storage space occupied is 3*512*sizeof(T).

[0104] Note that in the sinusoidal position encoding and rotational position encoding described above, for the word with position index 0, its position encoding corresponds to sin(0) and cos(0), so it is processed separately.

[0105] According to an embodiment of the present disclosure, using the position coding list, determining the position coding matrix of the input sequence based on the multiple word-grams may include: determining the position coding of each word-gram in the second word-gram set using the sine-cosine identity transform according to the position coding list; and creating the position coding matrix of the input sequence based on the position coding of each word-gram in the multiple word-grams.

[0106] Optionally, only the position code of each word in the first word set can be calculated and stored, and for the second word set, the position code of each word therein can be derived from the position code of the word in the first word set using the sine-cosine identity transformation. For example, in the sinusoidal position coding and rotational position coding described above, for word with position indexes of 1 to seq-1, only the position codes of word with odd position indexes can be stored, such as word with position indexes of 1, 3, 5, 7, ... . Using the sine-cosine identity transformation formula, the position code of word with even position index can be calculated, and then the position code of each word in multiple word sets can be determined, thereby constructing the position code matrix of the entire input sequence.

[0107] According to an embodiment of the present disclosure, based on the determined position coding matrix of the input sequence, processing the input sequence using an attention model may include: transforming the input sequence using the position coding matrix, and using the transformed result as an input of the attention model, so as to use the attention model to process the input sequence.

[0108] As described above, for absolute position encoding, transforming the input sequence using the position encoding matrix may include superimposing the position encoding of each word unit with the word vector, and for rotational position encoding, transforming the input sequence using the position encoding matrix may include rotating the word vector using the position encoding of each word unit. The transformed result may be input into the attention model for use by the attention model in the sequence processing process, so as to better understand the structure and semantics of the input sequence.

[0109] The data processing method of the present disclosure is introduced below using relative position coding as an example.

[0110] According to an embodiment of the present disclosure, determining a position coding list of the input sequence based on a first word-gram set of the multiple word-grams may include: determining multiple relative positions between word-grams in the multiple word-grams; for each of the multiple relative positions determined, determining a relative position coding associated with the relative position; and creating a position coding list of the input sequence based on the determined relative position coding.

[0111] Optionally, in addition to using the position encoding of a single word to construct a position encoding list, a position encoding list can also be constructed based on the relative position relationship between word units. For example, for the attention linear bias (ALiBi) position encoding, it is different from the above-mentioned absolute position encoding and rotation position encoding methods. It does not add position information to the word embedding vector, but instead adds the attention score matrix (i.e., Q*K) according to the distance between word units. T )Superposition bias matrix (PE seq,seq ). For example, for the multi-head attention mechanism, for each attention head, a hyperparameter λ can also be set for the bias matrix as follows:

[0112] Q*K T +PE seq,seq *λ=Q*K T +f(p ij )*λ (15)

[0113] Among them, Q represents the query vector, K T represents the transpose of the key vector, f(p ij ) represents the relative position p of i and j ij is a function of the independent variable, where i and j represent position i and position j respectively, p ij The i and j in the table can correspond to PE seq,seq The row and column indices in PE seq,seq According to f(p ij ) is the relative position encoding matrix generated by λ. λ represents a hyperparameter, such as a slope term. Taking λ as an example, slope is a hyperparameter related to the attention head. For example, for the number of attention heads = 8, the values ​​of slope can be: This slope term penalizes the attention score between query-key pairs, the penalty increases as the distance between the key and the query increases, and different attention heads increase their penalties at different rates, which can depend on the slope magnitude.

[0114] As mentioned above, in ALiBi position encoding, for each attention head, the position encoding matrix PE∈R seqxseq , it is necessary to perform calculations position by position and multiply it with the hyperparameter λ. The amount of calculation is: seq*seq+f(p ij)*seq*seq, the storage space occupied is: *seq*seq*sizeof(T).

[0115] However, for ALiBi position encoding, there are a lot of identical relative positions between and within the attention heads in the attention mechanism, which results in the function f(p ij ) There are a lot of identical calculations in the process of generating the position coding matrix, and a large number of identical position codes are stored in the position coding matrix, resulting in a waste of computing resources and storage space.

[0116] Therefore, in the case where there are a large number of identical relative positions in the ALiBi position encoding, a position encoding list can be constructed based on the relative position relationship between word units. For example, for each relative position relationship, any pair of word units with the relative position relationship can be used to construct a position encoding list (for example, according to the function f(p ij )) Determine the corresponding relative position coding, so that each relative position relationship and the corresponding relative position coding are included in the position coding list.

[0117] According to an embodiment of the present disclosure, the first word-gram set may include a plurality of word-gram pairs corresponding to the plurality of relative positions among the plurality of word-grams, and each word-gram pair has a corresponding relative position among the plurality of relative positions.

[0118] As described above, for each relative position relationship in the input sequence, any pair of word units having the relative position relationship can be selected to construct the first word unit set. Therefore, the first word unit set can be composed of multiple word unit pairs, and the position encoding list of the input sequence can be determined based on these word unit pairs.

[0119] Figure 5 is a schematic diagram showing a second implementation of creating a position code list based on an input sequence according to an embodiment of the present disclosure. Figure 5 As shown, for an input sequence including seq tokens, a position encoding list can be constructed based on various relative positions and related token pairs. For example, the position encoding list can include: relative position 1 (relative position relationship between tokens 1 and 2) and relative position encoding (RPE) 1, relative position 2 (relative position relationship between tokens 1 and 3) and RPE 2, and so on. Of course, the same relative position can be determined by other token pairs.

[0120] According to an embodiment of the present disclosure, utilizing the position coding list, determining the position coding matrix of the input sequence based on the multiple word-grams may include: for any two word-grams among the multiple word-grams, based on the relative position between the two word-grams, determining the relative position coding of the relative position between the two word-grams according to the position coding list; and creating the position coding matrix of the input sequence based on the relative position coding of the relative position between any two word-grams among the multiple word-grams.

[0121] Optionally, since the position code list already contains all relative position relationships and their corresponding relative position codes in the input sequence, for any two word units in the input sequence, the corresponding relative position relationship can be found from the position code list by looking up the table and the relative position codes of the relative positions between the two word units can be determined. Therefore, based on the relative position codes of the relative positions between all word unit pairs in the input sequence, the position code matrix of the input sequence can be determined.

[0122] According to an embodiment of the present disclosure, based on the determined position coding matrix of the input sequence, processing the input sequence using an attention model may include: for the input sequence, determining an attention score matrix using the attention model; transforming the attention score matrix using the position coding matrix so that the attention model processes the transformed attention score matrix.

[0123] Optionally, the attention score matrix (i.e., Q*K T )Superposition bias matrix (PE seq,seq ) for the attention model to process the transformed attention score matrix (which incorporates relative position information).

[0124] Fig. 6A is a schematic diagram of a position coding matrix showing relative position coding according to an embodiment of the present disclosure. Fig. 6A The position encoding matrix with causal relationship is shown ( Fig. 6A (a)) and the position encoding matrix without causal relationship ( Fig. 6A Example (b) in (b), where is the attention score matrix Q*K T The operation of superimposing the bias matrix PE multiplied by the hyperparameter λ is the same as described above. Fig. 6A As shown, in the position coding matrix with causal relationship, the element whose column index is greater than the row index does not exist. By using the position coding matrix with causal relationship, the amount of calculation of the elements symmetric about the main diagonal line of the matrix can be compressed.

[0125] Figure 6Bis a schematic diagram illustrating creation of a position code list of relative position codes according to an embodiment of the present disclosure. Figure 6B Taking the position coding matrix with causal relationships as an example, the process of creating a position coding list is shown. Figure 6B For the ALiBi relative position coding of seq=6, the position coding matrix 610 has 21 elements, wherein the relative position relationship existing therein can be determined based on the relative position p of i and j. ij The function f(p ij ) settings to determine.

[0126] According to an embodiment of the present disclosure, the relative position may be a relative distance between word units.

[0127] For example, when the relative position is the relative distance between word units, the relative position p of i and j is ij It can be expressed as the relative distance between i and j |ij|, that is, p ij =|ij|.

[0128] Based on this, through p ij =|ij| (601), the position encoding matrix 610 can be further represented as a position encoding matrix 620, each element of which represents the relative distance between the positions of different word units. Figure 6B Therefore, for this case, word pairs with the same relative distance have the same relative position relationship. Therefore, the position code list 630 can be constructed based on each relative distance that exists.

[0129] Optionally, the attention linear bias relative position encoding can generate a position encoding list of length length (usually, length>=seq), which can be stored in the on-chip memory. When processing the current word, the table can be looked up by coordinate calculation to read the corresponding position encoding from the on-chip memory for calculation.

[0130] Through the above processing, the computation amount of position encoding is reduced, and the multiplication operation with the hyperparameter λ is reduced. For example, for each attention head, the Alibi position encoding matrix PE∈R seqxseq , the amount of calculation is seq+f(p ij )*length, (length≥seq), the storage space occupied is length*sizeof(T). For example, in Figure 6B In the case shown, the original relative position encoding calculation amount is 6*6+f(p ij )*6*6, the storage space occupied is 6*6*sizeof(T), and the calculation amount of the optimized relative position encoding is 6+f(pij )*6, the storage space occupied is 6*sizeof(T), and its computational complexity and storage space are only 1 / 6 of the former.

[0131] As above reference Figure 2-6B The data processing method disclosed herein is directed to an input sequence including multiple word units, and only a part of the word units are used to create a position coding list, and the position coding of the entire input sequence is implemented based on the position coding list, wherein the word units in the input sequence that are not used to create the position coding list can determine their corresponding position coding from the position coding list by looking up a table. This method can utilize the association between the position coding of word units to derive the position coding related to the remaining word units from the position coding determined based on a part of the word units, thereby avoiding a large number of repeated calculation operations, reducing the waste of storage space and computing resources, improving the position coding processing, and improving the performance of the model in sequence data processing tasks.

[0132] Figure 7 is a schematic diagram showing a data processing device 700 according to an embodiment of the present disclosure.

[0133] According to an embodiment of the present disclosure, the data processing device 700 may include a list creation module 701 and a sequence processing module 702 .

[0134] The list creation module 701 may be configured to determine, for an input sequence including a plurality of word-grams, a position code list of the input sequence based on a first word-gram set of the plurality of word-grams, the position code list including position codes associated with the word-grams in the first word-gram set, and for a second word-gram set, position codes associated with the word-grams in the second word-gram set can be determined using the position code list, the second word-gram set being the complement of the first word-gram set in the plurality of word-grams. Optionally, the list creation module 701 may perform the operations described above with reference to step S201.

[0135] For example, in the generation stage of the positional coding, a positional coding list may be first created based on the input sequence, and the positional coding list may be used to generate the positional coding matrix. In an embodiment of the present disclosure, the positional coding list may be created using only the first word-gram set of the plurality of word-grams included in the input sequence (that is, a portion of the word-grams in the input sequence) to achieve deduplication processing of the original positional coding matrix calculation.

[0136] Optionally, for a second word-gram set of multiple word-grams, which includes all word-grams in the multiple word-grams except the word-grams in the first word-gram set used to create the position coding list, the position codings associated with these word-grams can be derived from the position coding list, for example, by looking up the position coding list to determine the associated position codings, and then determining the position codings associated with these word-grams.

[0137] Optionally, for various position encoding methods, the position encoding can be used as a bias vector to assign different biases to words at different positions in the input sequence. Based on different position encoding methods, the bias vector can be directly superimposed on the input word embedding vector, or it can be superimposed on the intermediate output of the input sequence processed by the attention model for subsequent processing.

[0138] The sequence processing module 702 may be configured to process the input sequence using the position encoding list through an attention model. Optionally, the sequence processing module 702 may perform the operations described above with reference to step S202.

[0139] Optionally, after creating the position encoding list, the position encoding list can be used to complete the position encoding of the entire input sequence, that is, the position encoding matrix of the input sequence is determined based on the position encoding list, and the position encoding matrix is ​​applied to the processing of the input sequence by the attention model. For example, the position encoding matrix of the input sequence may include a position encoding vector associated with each word in the input sequence, and the position encoding vector may be determined by a table lookup operation on the position encoding list.

[0140] For example, for absolute position encoding and rotational position encoding, a unique vector representation, namely the position encoding vector, can be assigned to the position of each word unit. By combining the position encoding of each word unit with the word vector (for example, for absolute position encoding, the position encoding of each word unit is superimposed on the word vector, and for rotational position encoding, the word vector is rotated using the position encoding of each word unit), the model can take into account the order information of words during processing, thereby better understanding the structure and semantics of the input sequence.

[0141] As an example, for absolute position encoding, the position encoding of each word unit can be superimposed with the word vector to encode the information of each position in the input sequence directly into the model, so that the model can understand the specific position of each word unit in the sequence. For example, sinusoidal position encoding can be used, by using sine and cosine functions of different frequencies, so that the model captures the complex relationship between positions, and these encodings are related to the absolute value of each position in the sequence.

[0142] For example, for sinusoidal position coding and rotational position coding, since trigonometric functions are periodic, as the position in the position coding matrix increases, the position coding value in the same dimension actually has a periodic change. Therefore, in the embodiments of the present disclosure, the sine-cosine identity transformation formula can be fully utilized to optimize the data processing of sinusoidal position coding and rotational position coding.

[0143] Optionally, word units that can be used to derive position codes of other word units can be extracted from multiple word units by analyzing the periodic changes of position codes of different word units on the same dimension according to the position changes of the word units. These word units can constitute a first word unit set.

[0144] Optionally, only the position code of each word in the first word set may be calculated and stored, and for the second word set, the position code of each word therein may be derived from the position code of the word in the first word set using the sine-cosine identity transformation.

[0145] For absolute position encoding, the position encoding of each word unit can be superimposed on the word vector. For rotational position encoding, the word vector can be rotated using the position encoding of each word unit. The transformed result can be input into the attention model for use in sequence processing, so as to better understand the structure and semantics of the input sequence.

[0146] Optionally, in addition to using the position encoding of a single word unit to construct a position encoding list, a position encoding list can also be constructed based on the relative position relationship between word units. For example, for ALiBi position encoding, unlike the above-mentioned absolute position encoding and rotation position encoding methods, it does not add position information to the word embedding vector, but instead superimposes a bias matrix on the attention score matrix based on the distance between word units.

[0147] Optionally, in the case where there are a large number of identical relative positions in the ALiBi position encoding, a position encoding list can be constructed based on the relative position relationship between word units. For example, for each relative position relationship, the corresponding relative position encoding can be determined based on any pair of word units having the relative position relationship, so that each relative position relationship and the corresponding relative position encoding are included in the position encoding list.

[0148] Optionally, for each relative position relationship in the input sequence, any pair of word units having the relative position relationship can be selected to construct the first word unit set. Therefore, the first word unit set can be composed of multiple word unit pairs, and the position encoding list of the input sequence can be determined based on these word unit pairs.

[0149] Optionally, since the position code list already contains all relative position relationships and their corresponding relative position codes in the input sequence, for any two word units in the input sequence, the corresponding relative position relationship can be found from the position code list by looking up the table and the relative position codes of the relative positions between the two word units can be determined. Therefore, based on the relative position codes of the relative positions between all word unit pairs in the input sequence, the position code matrix of the input sequence can be determined.

[0150] Optionally, a bias matrix can be superimposed on the attention score matrix so that the attention model can process the transformed attention score matrix (incorporating relative position information).

[0151] According to yet another aspect of the present disclosure, a data processing device is provided. Figure 8 A schematic diagram of a data processing device 2000 according to an embodiment of the present disclosure is shown.

[0152] like Figure 8 As shown, the data processing device 2000 may include one or more processors 2010 and one or more memories 2020. The memory 2020 stores a computer readable code, and when the computer readable code is run by the one or more processors 2010, the data processing method described above may be executed.

[0153] The processor in the embodiments of the present disclosure may be an integrated circuit chip having signal processing capabilities. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc., and may be an X86 architecture or an ARM architecture.

[0154] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the embodiments of the present disclosure are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0155] For example, the method or device according to the embodiment of the present disclosure may also be implemented by Fig. 9 The architecture of the computing device 3000 shown in FIG. Fig. 9 As shown, the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, etc. The storage device in the computing device 3000, such as the ROM 3030 or the hard disk 3070, may store various data or files used for processing and / or communication of the data processing method provided in the present disclosure and program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, Fig. 9 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different devices. Fig. 9 One or more components of a computing device are shown.

[0156] According to another aspect of the present disclosure, a computer-readable storage medium is also provided. Computer-readable instructions are stored on the computer storage medium. When the computer-readable instructions are executed by a processor, the data processing method according to the embodiment of the present disclosure described with reference to the above figures can be executed. The computer-readable storage medium in the embodiment of the present disclosure may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronously linked dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM). It should be noted that memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory. It should be noted that memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0157] The embodiments of the present disclosure also provide a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the data processing method according to the embodiments of the present disclosure.

[0158] Embodiments of the present disclosure provide a data processing method, apparatus, device, and computer-readable storage medium.

[0159] The method provided by the embodiment of the present disclosure is for an input sequence including multiple word-grams, and only a part of the word-grams is used to create a position coding list, and the position coding of the entire input sequence is implemented based on the position coding list, wherein the word-grams in the input sequence that are not used to create the position coding list can determine their corresponding position coding from the position coding list by looking up a table. The method of the embodiment of the present disclosure can utilize the association between the position codings of word-grams to derive the position coding related to the remaining word-grams from the position coding determined based on a part of the word-grams, thereby avoiding a large number of repeated calculation operations, reducing the waste of storage space and computing resources, improving the position coding processing, and improving the performance of the model in sequence data processing tasks.

[0160] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of the code contains at least one executable instruction for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0161] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the embodiments of the present disclosure are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0162] The exemplary embodiments of the present disclosure described in detail above are merely illustrative and not restrictive. It should be understood by those skilled in the art that various modifications and combinations may be made to these embodiments or their features without departing from the principles and spirit of the present disclosure, and such modifications should fall within the scope of the present disclosure.

Claims

1. A data processing method, comprising: For an input sequence including a plurality of word-grams, a position code list of the input sequence is determined based on a first word-gram set of the plurality of word-grams; wherein the position code list is composed of position codes associated with the word-grams in the first word-gram set, and for a second word-gram set, the position codes associated with the word-grams in the second word-gram set can be derived from the position code list by utilizing associations between the position codes of the plurality of word-grams, the second word-gram set being the complement of the first word-gram set in the plurality of word-grams, the association comprising one of a periodic change of the position codes of the plurality of word-grams and a relative position relationship between the plurality of word-grams; The input sequence is processed using the positional encoding list through an attention model.

2. The method of claim 1, wherein: Using the position encoding list, processing the input sequence through the attention model includes: Determining a position encoding matrix of the input sequence based on the plurality of word-grams using the position encoding list, the position encoding matrix including a position encoding associated with each word-gram in the plurality of word-grams; Based on the determined position encoding matrix of the input sequence, the input sequence is processed using an attention model.

3. The method of claim 2, wherein: Determining a position encoding list of the input sequence based on a first word-gram set of the plurality of word-grams comprises: Selecting the first word-gram set from the plurality of word-grams, and determining a position code for each word-gram in the first word-gram set; A position encoding list of the input sequence is created based on the position encoding of each word-gram in the first word-gram set.

4. The method of claim 3, wherein: Selecting the first word-gram set from the plurality of word-grams comprises: The first word-gram set is selected from the plurality of word-grams according to periodic changes between position codes of different word-grams.

5. The method of claim 3, wherein: Selecting the first word-gram set from the plurality of word-grams comprises: When position encoding is performed on the input sequence using a trigonometric function, the first word unit set is selected from the plurality of word units according to periodic changes in position encodings of different word units on the same dimension according to changes in positions of the word units.

6. The method of claim 5, wherein: Determining a position encoding matrix of the input sequence based on the plurality of word-grams using the position encoding list comprises: Determine the position code of each word in the second word set by using sine-cosine identity transformation according to the position code list; A position encoding matrix of the input sequence is created based on the position encoding of each word-gram in the plurality of word-grams.

7. The method of claim 3, wherein: Based on the determined position encoding matrix of the input sequence, processing the input sequence using the attention model includes: The input sequence is transformed using the position encoding matrix, and the transformed result is used as the input of the attention model so that the attention model processes the input sequence.

8. The method of claim 2, wherein: Determining a position encoding list of the input sequence based on a first word-gram set of the plurality of word-grams comprises: determining a plurality of relative positions between word-grams in the plurality of word-grams; For each relative position of the determined plurality of relative positions, determining a relative position code associated with the relative position; Based on the determined relative position codes, a position code list of the input sequence is created.

9. The method of claim 8, wherein: The first word-gram set includes a plurality of word-gram pairs corresponding to the plurality of relative positions among the plurality of word-grams, and each word-gram pair has a corresponding relative position among the plurality of relative positions.

10. The method of claim 8, wherein: Determining a position encoding matrix of the input sequence based on the plurality of word-grams using the position encoding list comprises: For any two word-grams of the plurality of word-grams, determining, based on the relative position between the two word-grams, a relative position code of the relative position between the two word-grams according to the position code list; A position encoding matrix of the input sequence is created based on relative position encodings of relative positions between any two word-grams in the plurality of word-grams.

11. The method of claim 8, wherein: Based on the determined position encoding matrix of the input sequence, processing the input sequence using the attention model includes: For the input sequence, determining an attention score matrix using the attention model; The attention score matrix is ​​transformed using the position encoding matrix so that the attention model processes the transformed attention score matrix.

12. The method of claim 8, wherein: The relative position is the relative distance between word units.

13. A data processing device comprising: one or more processors; as well as One or more memories, wherein a computer executable program is stored, and when the computer executable program is executed by the processor, the method of any one of claims 1-12 is performed.

14. A computer program product stored on a computer readable storage medium and comprising computer instructions which, when executed by a processor, cause a computer device to perform the method of any one of claims 1 to 12.

15. A computer-readable storage medium having computer-executable instructions stored thereon, wherein the instructions are used to implement the method according to any one of claims 1 to 12 when executed by a processor.

Images