Sequence processing method, electronic device and storage medium

Abstract

The present disclosure proposes a sequence processing method, an electronic device and a storage medium, wherein the sequence processing method comprises: dividing an input sequence into an ordered first batch of subsequences, and determining a subsequence number corresponding to each subsequence; determining a first position code corresponding to each input word in the subsequence based on the subsequence number; determining a second position code corresponding to each input word based on the position number of the input word in the corresponding subsequence, and determining a corresponding target position code based on the first position code and the second position code; for each input word, adjusting a corresponding first query vector and a first key vector based on the corresponding target position code to obtain a corresponding second query vector and a second key vector; performing attention processing based on each second query vector, each second key vector and each value vector corresponding to each input word to obtain an output sequence corresponding to the input data. The present disclosure can improve the extrapolation of a large language model in the reasoning process.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a sequence processing method, electronic device and medium. Background Art

[0002] When processing input data, large language models often convert the input raw data into multiple input words, then encode each input word into a corresponding feature representation and input it into the attention mechanism for attention processing to generate the corresponding output. However, the attention mechanism itself cannot capture the position information of the input word. When the position of the input word (that is, the order of each input word in the sequence) changes, the output of the attention mechanism will remain unchanged. The output of the attention mechanism is independent of the order of the input word, which does not conform to the characteristics of natural language and will also reduce the output accuracy of the large language model. Therefore, in large language models, it is often necessary to positionally encode the input word.

[0003] In the prior art, when training a large language model, due to limitations in training efficiency and device memory resources, the length of the sample sequence used for training is often short, and the large language model only learns how to handle positional encodings within a certain window length during training. During the inference process, when the length of the actual input sequence of the large language model exceeds the window length during training, the positional encodings used for the part exceeding the window length are not trained. The large language model has not learned how to accurately handle these positional encodings exceeding the window length during training, which makes the quality of the output generated by the large language model at this time lower, the performance of the large language model will be significantly reduced, and the extrapolation of the large language model is poor. Summary of the invention

[0004] The embodiments of the present disclosure provide a sequence processing method, an electronic device, and a storage medium, which can ensure that the position encoding used by a large language model during the reasoning process is trained to ensure that the performance of the large language model does not decrease and improve the extrapolation of the large language model during the reasoning process.

[0005] According to one aspect of the present disclosure, a sequence processing method is provided, the method comprising:

[0006] Converting input data of the large language model into a corresponding input sequence, and determining a first length of the input sequence;

[0007] Determine a first batch of times based on the first length and a first preset sequence length corresponding to the large language model, divide the input sequence into a first batch of times of ordered subsequences, and determine a subsequence number corresponding to each of the subsequences;

[0008] For each subsequence, determining a first position code corresponding to each input word in the subsequence based on the corresponding subsequence serial number, wherein the first position codes corresponding to each input word in a single subsequence are the same;

[0009] For each input word, position coding is performed based on the position number of the input word in the corresponding subsequence to obtain a second position code corresponding to each input word, and a target position code corresponding to the input word is determined based on the first position code and the second position code corresponding to the input word;

[0010] For each input word, generate a first query vector, a first key vector, and a value vector corresponding to the input word, and adjust the first query vector and the first key vector corresponding to the input word based on the target position code corresponding to the input word to obtain a second query vector and a second key vector corresponding to the input word;

[0011] Attention processing is performed based on each of the second query vectors, each of the second key vectors, and each of the value vectors corresponding to each of the input word units, and an output sequence corresponding to the input data is obtained based on a result of the attention processing.

[0012] Optionally, converting the input data of the large language model into a corresponding input sequence and determining a first length of the input sequence includes:

[0013] Performing word segmentation processing on the input data to obtain a plurality of subwords, performing lemma processing on each of the subwords to obtain the input lemmas corresponding to each of the subwords, and determining a first number corresponding to the plurality of input lemmas, wherein the first number is the number of input lemmas obtained after the input data is segmented;

[0014] The input sequence is determined based on the plurality of input word-grams, and the first length is determined based on the first number.

[0015] Optionally, determining a first batch of times based on the first length and a first preset sequence length corresponding to the large language model, and dividing the input sequence into ordered first batch of times of subsequences includes:

[0016] Determine a first quotient of the first length and the first preset sequence length, and round up the first quotient to obtain the first batch of times;

[0017] Divide each input word of a first preset sequence length in the input sequence into a subsequence in sequence, until each input word of the input sequence is divided into a subsequence, to obtain a first batch of subsequences.

[0018] Optionally, the step of dividing each input word of the first preset sequence length in the input sequence into a subsequence in sequence includes:

[0019] Determine the original position sequence number of each input word in the input sequence;

[0020] For each of the input word units, determine a second quotient between the corresponding original position serial number and the first preset sequence length, and round down the second quotient to obtain a first serial number corresponding to the input word unit;

[0021] The plurality of input word units corresponding to the same first sequence number are divided into a subsequence, and the first sequence numbers corresponding to the input word units in the subsequence are used as the subsequence sequence numbers of the subsequence.

[0022] Optionally, performing position encoding based on the position of the input word in the corresponding subsequence to obtain a second position encoding corresponding to each input word includes:

[0023] Determine the original position sequence number corresponding to the input word based on the position of the input word in the input sequence;

[0024] Taking a modulus of the first preset sequence length based on the original position sequence number corresponding to the input word unit to obtain a first remainder;

[0025] The first remainder is subjected to rotation position encoding to obtain a second position code corresponding to the input word unit.

[0026] Optionally, performing attention processing based on the second query vector, the second key vector, and the value vector corresponding to each of the input word units includes:

[0027] For a single second query vector, perform matrix multiplication on the second query vector and each of the second key vectors to obtain an attention score between a first input word unit and each of the second input words unit, wherein the first input word unit is an input word unit corresponding to the second query vector, the second input word unit is an input word unit corresponding to the second key vector, and each of the attention scores constitutes an attention score matrix;

[0028] Determine an attention scaling factor based on a difference between the subsequence number of the subsequence corresponding to the first input word and the subsequence number of the subsequence corresponding to the second input word, and scale the attention score between the first input word and the second input word based on the attention scaling factor to obtain a scaled attention score corresponding to the second input word, wherein each scaled attention score constitutes a scaled attention score matrix;

[0029] The scaled attention score matrix is ​​subjected to exponential normalization to obtain an attention weight matrix, and the attention weight matrix is ​​matrix multiplied with a value matrix to obtain a result of attention processing, wherein the value matrix is ​​composed of the concatenation of each of the value vectors.

[0030] Optionally, the large language model is trained in the following manner:

[0031] Acquire a sample sequence for training the large language model, wherein the sample sequence includes a plurality of sample word units, and a sequence length of the sample sequence is not greater than the first preset sequence length;

[0032] Dividing the sample sequence into a plurality of sample subsequences, determining a sample subsequence number corresponding to each sample word unit, and generating a sample first position code corresponding to each sample word unit based on the sample subsequence number;

[0033] Performing rotation position encoding on each of the sample word units based on the position of the sample word unit in the corresponding sample subsequence to obtain a sample second position code corresponding to each of the sample word units, and superimposing the sample second position code corresponding to each of the sample word units with the sample first position code to obtain a sample target position code corresponding to each of the sample word units;

[0034] Generate a corresponding sample first query vector, a sample first key vector and a sample value vector based on each of the sample word units;

[0035] For each of the sample word-grams, adjusting the corresponding sample first query vector and sample first key vector based on the corresponding sample target position code to obtain a sample second query vector and a sample second key vector corresponding to the sample word-gram;

[0036] Attention processing is performed based on the sample second query vector, the sample second key vector and the sample value vector corresponding to the plurality of sample word units, and a sample output sequence corresponding to the sample sequence is generated based on the result of the attention processing, and back propagation is performed based on the sample output sequence to optimize the parameters of the large language model until the first condition is met, thereby obtaining the trained large language model.

[0037] Optionally, the determining of the sample subsequence number corresponding to each of the sample word-grams includes:

[0038] Determine a second number based on the first preset sequence length and a preset extrapolation length, and determine a third number based on the second number, wherein the third number is a positive integer not greater than the second number;

[0039] Determine a third quotient between the first preset sequence length and the third number, and round up the third quotient to obtain a fourth number, divide every fourth number of the sample word units in the sample sequence into a sample subsequence to obtain ordered sample subsequences of the third number, and determine a sequence number corresponding to each of the sample subsequences as the sample subsequence sequence number corresponding to the sample word unit in the sample subsequence.

[0040] Optionally, performing rotation position encoding on each of the sample word units based on the position of the sample word unit in the corresponding sample subsequence to obtain a sample second position encoding corresponding to each of the sample word units includes:

[0041] Determine the sample original position sequence number of each of the sample word units in the sample sequence;

[0042] For each of the sample word units, the fourth number is modulo based on the corresponding sample original position serial number to obtain a second remainder, and a rotation position code is performed based on the second remainder to obtain a sample second position code corresponding to the sample word unit.

[0043] Optionally, the performing attention processing based on the sample second query vector, the sample second key vector and the sample value vector corresponding to the plurality of sample word-grams includes:

[0044] For a single sample second query vector, matrix multiply the single sample second query vector by each of the sample second key vectors to obtain a sample attention score between a first sample word-element and each of the second sample word-element, wherein the first sample word-element is a sample word-element corresponding to the single sample second query vector, and the second sample word-element is a sample word-element corresponding to the sample second key vector;

[0045] Based on the difference between the sample subsequence number corresponding to the first sample word-unit and the sample subsequence number corresponding to the second sample word-unit, determine a sample attention scaling factor between the first sample word-unit and the second sample word-unit, and scale the corresponding sample attention scores based on the sample attention scaling factor to obtain corresponding scaled sample attention scores, wherein each scaled sample attention score constitutes a scaled sample attention score matrix;

[0046] The scaled sample attention score matrix is ​​subjected to exponential normalization to obtain a sample attention weight matrix, and the sample attention weight matrix and the sample value matrix are subjected to matrix multiplication to obtain a result of attention processing, wherein the sample value matrix is ​​composed of the sample value vectors corresponding to each of the sample word units.

[0047] According to one aspect of the present disclosure, an electronic device is proposed, which includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, wherein the program implements the sequence processing method described above when executed by the processor.

[0048] According to one aspect of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the sequence processing method as described above.

[0049] The present disclosure proposes a sequence processing method, electronic device and medium. The method converts the input data of a large language model into a corresponding input sequence, and then divides the input sequence into multiple subsequences based on the first length of the input sequence and the first preset sequence length, and determines the subsequence number corresponding to each subsequence. The first preset sequence length represents the length of the sample sequence input to the large language model during the training process, that is, the position codes within the first preset sequence length are all trained. Afterwards, the first position code corresponding to the input word is determined based on the subsequence number of the subsequence to which each input word belongs, and the second position code corresponding to the input word is determined according to the position of the input word in the corresponding subsequence. At this time, since the length of each subsequence is not greater than the first preset sequence length, the second position code corresponding to each input word is all trained. At the same time, by determining the first position code corresponding to the input word based on the subsequence number corresponding to the subsequence where the input word is located, and determining the target position code corresponding to the input word based on the first position code and the second position code corresponding to the input word, in the process of training the large language model, it is only necessary to simultaneously train the second position code within the first preset sequence length and the first position code generated based on the subsequence number, so as to ensure that the target position code corresponding to each input word is trained. In this way, in the process of reasoning, even if the length of the input sequence of the large language model exceeds the first preset sequence length, it can be ensured that the position code used by each output word is trained. Afterwards, the first query vector and the first key vector corresponding to the input word are adjusted based on the target position encoding corresponding to the input word to obtain the corresponding second query vector and second key vector, and the position information corresponding to the trained target position encoding is injected into the attention processing process, and attention processing is performed based on the second query vector, the second key vector and the value vector corresponding to each input word to obtain an output sequence corresponding to the input sequence, so that when the processing length of the large language model exceeds the window length learned by the large language model during the training process, the position encoding used is also trained to ensure that the performance of the large language model will not decrease, thereby improving the extrapolation of the large language model during the reasoning process.

[0050] Other features and advantages of the present disclosure will be described in the following description, and partly become apparent from the description, or understood by practicing the present disclosure. The purpose and other advantages of the present disclosure can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings are used to provide further understanding of the technical solution of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solution of the present disclosure and do not constitute a limitation on the technical solution of the present disclosure.

[0052] Figure 1 is a system architecture diagram of a sequence processing method using an embodiment of the present disclosure;

[0053] Figure 2 is a main flow chart of a sequence processing method according to an embodiment of the present disclosure;

[0054] Figure 3 yes Figure 2 A sub-flow chart of step S201;

[0055] Figure 4 A schematic diagram of generating a first position code and a second position code and superimposing them as a target position code based on an embodiment of the present disclosure;

[0056] Figure 5 is a schematic diagram of a process for generating output based on input word-grams according to an embodiment of the present disclosure;

[0057] Figure 6 A schematic diagram of determining a target position encoding of an input word unit and adjusting the attention processing process based on an embodiment of the present disclosure;

[0058] Figure 7 yes Figure 2 A sub-flowchart of step S202;

[0059] Figure 8 yes Figure 7 A sub-flowchart of step S702;

[0060] Fig. 9 It is a schematic diagram of determining a corresponding subsequence number and a position number in the corresponding subsequence based on an original position number of an input word unit according to an embodiment of the present disclosure;

[0061] Fig.10 yes Figure 2 A sub-flowchart of step S204 in FIG. 1 ;

[0062] Fig.11 yes Figure 2A sub-flowchart of step S206 in FIG. 1 ;

[0063] Fig.12 is a flow chart of training a large language model based on an embodiment of the present disclosure;

[0064] Fig.13 yes Fig.12 A sub-flowchart of step S1202;

[0065] Fig.14 yes Fig.12 A sub-flowchart of step S1203;

[0066] Fig.15 yes Fig.12 A sub-flowchart of step S1206;

[0067] Fig.16 It is a schematic diagram of the structure of an electronic device proposed in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the present disclosure more clear, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not used to limit the present disclosure.

[0069] Before further describing the embodiments of the present disclosure in detail, the nouns and terms involved in the embodiments of the present disclosure are described. The nouns and terms involved in the embodiments of the present disclosure are subject to the following interpretations:

[0070] Large language models (LLMs): an artificial intelligence model trained with large-scale corpus data. Through the attention mechanism, large language models can capture the complex structure and semantics of language, thereby achieving text generation. Large language models are mainly used in scenarios such as text summarization, question answering, and machine translation.

[0071] Extrapolation: refers to the performance change of the neural network model when the input sequence length exceeds the sequence length learned by the neural network model during the training process during the inference process. It reflects the generalization ability of the large language model. In the process of training a large language model, if a longer sample sequence is used directly for training, on the one hand, the complexity of the calculation will increase significantly and the training efficiency will be significantly reduced; on the other hand, a longer sample sequence means that a large amount of intermediate data will be generated. Due to the constraints of video memory resources, these intermediate data are difficult to store. Therefore, in the training stage, the size of the sample sequence is generally relatively small. For example, a large language model may need to process an input sequence of 128,000 in the inference stage, but in the training stage, the length of the sample sequence may be only 4,000. At this time, the large language model needs to have a certain degree of extrapolation. When the large language model has poor extrapolation, when the length of the sample sequence exceeds the length of the training stage, the output generated by the large language model will become illogical; on the contrary, when the large language model has good extrapolation, even if the length of the input sequence to be processed is several times the length of the sample sequence in the training stage, the large language model can still generate relatively accurate output.

[0072] Attention mechanism: An important mechanism in large language models. It multiplies the input matrix corresponding to the input sequence with the query weight matrix, key weight matrix and value weight matrix respectively to obtain the query matrix, key matrix and value matrix, and then performs matrix multiplication on the query matrix and the key matrix to obtain the attention score matrix. Each element in this attention score matrix represents the attention score between the two words in the input sequence, that is, the degree of association between the two words. Then, the attention score matrix is ​​processed by softmax to obtain the attention weight matrix, and then the attention weight matrix and the value matrix are matrix multiplied to obtain the corresponding attention operation result matrix. Through this mechanism, the large language model can capture the association between each word in the input sequence, and give higher weights to words with high association to generate output.

[0073] Positional Encoding: Large language models mainly use the attention mechanism to generate the corresponding output sequence according to the input sequence, but the attention mechanism cannot capture the order relationship of each word in the input sequence, which makes the attention mechanism itself produce the same output results for input sequences with the same word but different word order. For example, for the two input texts "I am your student" and "You are my student", the large language model will generate the same output without positional encoding, which does not conform to the characteristics of natural language. Therefore, in the large language model, it is necessary to positionally encode each word in the input sequence. The positional encoding method can be divided into absolute position encoding and relative position encoding. Absolute position encoding is to generate a position embedding matrix, and then add the position embedding matrix and the text embedding matrix obtained based on the input sequence as the input of the attention mechanism. In this way, when the length of the input sequence is long, there will be a large number of untrained position encodings in the position embedding matrix, which leads to poor extrapolation of the large language model. There are also some other position encoding methods in the related art, which can improve the extrapolation of large language models to a certain extent. For example, ReRoPE encoding regards the position encodings of word units that exceed the large language model window as window boundary values. For example, the window length of the large language model is m, and the length of the input sequence is L (L>m). Then for each word unit in the input sequence, the position encoding can be expressed as 1, 2, 3,..., m, m, m. The position encodings used in this method are all trained, but this method has the same position encoding for word units that exceed the window length (i.e., the m+1th word unit to the Lth word unit). After changing the order of these word units that exceed the window length, the large language model will also generate similar outputs, and the position attributes of the word units that exceed the window length become lower.

[0074] In large language models, the input sequence is processed through the attention mechanism. It calculates the attention weights between each word in the input sequence and weights each word to generate an output sequence. In this way, when generating the output sequence, it can pay more attention to the relatively important words in the input sequence and improve the accuracy of the output. However, the attention mechanism itself cannot capture the order information of each word in the input sequence, which makes the attention mechanism generate similar or even identical output sequences for input sequences with the same word but different word order, which does not conform to the characteristics of natural language. Therefore, in large language models, it is often necessary to positionally encode each word in the input sequence before performing attention processing.

[0075] In related technologies, for the sake of training efficiency and limited by video memory resource constraints, the length of the sample sequence used in training the large language model is limited, which makes the large language model only learn positional encodings within a certain window length during the training phase. In the inference phase, in order to improve the contextual understanding ability of the large language model, the length of the input sequence that the large language model needs to process will exceed the window length. At this time, for the word units that exceed the window length, the positional encodings used are all untrained. At this time, the accuracy of the output generated by the large language model will become lower or even illogical, the performance of the large language model will be significantly reduced, and the extrapolation of the large language model will be poor.

[0076] Based on this, the present disclosure proposes a sequence processing method, an electronic device and a storage medium, which can enable the position encoding used by a large language model when processing an input sequence to be trained, so as to ensure that the performance of the large language model does not decrease and improve the extrapolation of the large language model during the reasoning process.

[0077] Description of the system architecture of the present disclosure embodiment

[0078] Figure 1 1 is a system architecture diagram used in the sequence processing method of the embodiment of the present disclosure, which includes: an object terminal 110, the Internet 120, a gateway 130, and a server 140.

[0079] The object terminal 110 is a device used by the object to input input data of the large language model. It includes various forms such as desktop computers, laptops, PDAs (personal digital assistants), mobile phones, vehicle-mounted terminals, home theater terminals, and dedicated terminals. In addition, it can be a single device or a collection of multiple devices. For example, multiple devices are connected through a local area network and share a display device to work together, forming a terminal. The object terminal 110 can also communicate with the Internet 120 in a wired or wireless manner to exchange data.

[0080] The gateway 130 is also called an internetwork connector or a protocol converter. The gateway 130 realizes network interconnection at the transport layer and is a computer system or device that acts as a converter. The gateway 130 is a translator between two systems that use different communication protocols, data formats or languages, or even completely different architectures. At the same time, the gateway 130 can also provide filtering and security functions. The message sent by the object terminal 110 to the server 140 is sent to the corresponding server 140 through the gateway 130. The message sent by the server 140 to the object terminal 110 is also sent to the corresponding object terminal 110 through the gateway 130.

[0081] The server 140 refers to a computer system that can provide sequence processing services to the object terminal 110. Compared with the object terminal 110, the server 140 has higher requirements in terms of stability, security, performance, etc. The server 140 can be a high-performance computer in a network platform, a cluster of multiple high-performance computers, a part of a high-performance computer (such as a virtual machine), a combination of parts of multiple high-performance computers (such as virtual machines), etc. The server 140 can also communicate with the Internet 120 in a wired or wireless manner to exchange data.

[0082] Overall implementation of the sequence processing method of the present disclosure

[0083] The present disclosure provides a sequence processing method, which is applied to a sequence processing device. Figure 2 , sequence processing methods include:

[0084] Step S201, converting input data of the large language model into a corresponding input sequence, and determining a first length of the input sequence;

[0085] Step S202, determining a first batch of times based on the first length and a first preset sequence length corresponding to the large language model, dividing the input sequence into a first batch of times of ordered subsequences, and determining a subsequence number corresponding to each subsequence;

[0086] Step S203, for each subsequence, determining the first position code corresponding to each input word in the subsequence based on the corresponding subsequence serial number, wherein the first position code corresponding to each input word in a single subsequence is the same;

[0087] Step S204, for each input word, position coding is performed based on the position number of the input word in the corresponding subsequence to obtain a second position code corresponding to each input word, and a target position code corresponding to the input word is determined based on the first position code and the second position code corresponding to the input word;

[0088] Step S205, for each input word, generating a first query vector, a first key vector and a value vector corresponding to the input word, adjusting the first query vector and the first key vector corresponding to the input word based on the target position code corresponding to the input word, to obtain a second query vector and a second key vector corresponding to the input word;

[0089] Step S206, performing attention processing based on each second query vector, each second key vector and each value vector corresponding to each input word, and obtaining an output sequence corresponding to the input data based on the result of the attention processing.

[0090] In step S201, the input data is data input into the large language model by the object through the terminal, and it can be data in a text format; the input sequence is a sequence data obtained based on the input data. Exemplarily, when the input data is a text, multiple sub-words are obtained by segmenting the text, and each sub-word obtained by the segmentation is mapped into a corresponding word unit, and the word units corresponding to each sub-word are concatenated to form an input sequence corresponding to the input data; the first length is used to characterize the number of input word units contained in the input sequence, and each input word unit is regarded as a unit length. Exemplarily, the input sequence includes 100 input word units, and the first length corresponding to the input sequence is 100.

[0091] Specifically, refer to Figure 3 In one embodiment, step S201 includes:

[0092] Step S301, performing word segmentation processing on the input data to obtain a plurality of subwords, performing lemma processing on each subword to obtain an input lemma corresponding to each subword, and determining a first number corresponding to the plurality of input lemmas;

[0093] Step S302 , determining an input sequence based on a plurality of input word units, and determining a first length based on a first number.

[0094] In step S301, the input data may be segmented based on subword granularity, thereby dividing the input data into multiple subwords that can respectively represent certain semantics. For example, for the word "unfriendly", based on subword granularity, the word will be split into three parts: "un", "friend", and "ly", each part corresponding to a word element, where "un" represents opposite semantics, "friend" represents friendly semantics, and "ly" indicates that the word is an adverb. In this way, the word is divided into multiple subwords that can represent certain semantic information.

[0095] After the input data is segmented, each subword obtained by the segmentation is lemmatized. Specifically, the large language model will contain a vocabulary, which stores the mapping relationship between the subwords and the corresponding identification IDs. Based on the mapping relationship, each subword obtained by the segmentation of the input data is mapped to the corresponding identification ID, and the input word corresponding to each subword is obtained. It can be understood that the identification ID is an identification number composed of a group of numbers. Exemplarily, for the word "unfriendly", after the segmentation, "un", "friend", and "ly" are obtained, and the corresponding identification IDs are "1561", "5645", and "1655" respectively. After the lemma processing, the input word corresponding to the three subwords "un", "friend", and "ly" are "1561", "5645", and "1655" respectively. In this way, the input data is converted into a form that can be processed by the large language model, and an input sequence corresponding to the input data is subsequently generated. At the same time, the corresponding first number is determined based on the number of input word units obtained based on the input data. Since the input word units and the sub-words obtained by word segmentation are one-to-one corresponding, based on this, the first number is the number of sub-words obtained by word segmentation of the input word units. For example, when the input data is the word "unfriendly", the three sub-words "un", "friend" and "ly" are obtained by word segmentation. At this time, the first number is 3.

[0096] In another embodiment, the input data may be segmented based on character granularity or word granularity. For example, for the word "unfriendly", when segmenting based on word granularity, the entire word is treated as a word and subsequently converted into an input word unit; when segmenting based on word granularity, each English letter in the word corresponds to a word, and each English letter is subsequently converted into an input word unit.

[0097] It is understandable that in some other embodiments, the large language model may also be a multimodal large language model, and the input data may be data in a non-textual mode. For example, the input data may be image data, in which case the image may be segmented into a plurality of fixed-size patches, and each patch may be converted into a corresponding input word through an embedding layer.

[0098] In step S302, the input word grammars corresponding to each subword are spliced ​​according to the order of the subwords in the input sequence to obtain an input sequence corresponding to the input data. For example, the input data is the word "unfriendly". After word segmentation, the three subwords "un", "friend" and "ly" are obtained. After word grammarization, the input word grammars corresponding to these three subwords are "1561", "5645" and "1655" respectively. The input sequence obtained after splicing is "156156451655". At the same time, the first number used to represent the number of subwords obtained by word segmentation of the input data is determined as the first length corresponding to the input sequence. In this way, the first length can represent the number of input word grammars contained in the input sequence.

[0099] In the embodiment disclosed in steps S301 to S302, the input data is processed by performing word segmentation and lemma processing, thereby converting the input data into a corresponding input sequence, thereby converting the input data into a form that can be processed by a large language model; at the same time, a corresponding first number is determined based on the number of sub-words obtained by word segmentation, and a first length of the input sequence is determined based on the first number, so that the first length can represent the number of input lemmas contained in the input sequence, so as to subsequently determine the first batch of times based on the first length and divide the input sequence into the first batch of times of sub-sequences, to ensure that the length of a single sub-sequence is not greater than the first preset sequence length.

[0100] In step S202, the first preset sequence length is used to characterize the maximum sequence length of the corresponding position code learned by the large language model during the training process. It characterizes the window length learned by the large language model during the training process. For example, the length of the sample sequence of the large language model during the training process is 4000, then the window length learned by the large language model during the training process is 4000, and the corresponding first preset sequence length is 4000. The first batch number characterizes the number of subsequences obtained by segmenting the input sequence into multiple subsequences whose lengths are not greater than the first preset sequence length. For example, the first length corresponding to the input sequence is 128000, and the first preset sequence length is 4000. Then, after the input sequence is segmented into multiple subsequences whose lengths are not greater than 4000, 32 subsequences can be obtained. At this time, the first batch number is 32. The subsequence number represents the order of each subsequence in the input sequence. It can be understood that when the input sequence is divided into the first batch of subsequences, multiple consecutive input words in the input sequence are regarded as a subsequence, and the input words themselves are ordered in the input sequence. Then, the first batch of subsequences composed of multiple consecutive input words are also ordered, and each subsequence also has a corresponding subsequence number. Exemplarily, the first length of the input sequence is 128000, that is, it contains 128000 input words, and the position numbers of the input words are 0, 1, 2, ..., 127999 respectively. The first preset sequence length is 4000, then, the subsequence number corresponding to the subsequence composed of the first 4000 input words in the input sequence is 0, the subsequence number corresponding to the subsequence composed of the 4001th to 8000th input words is 2, and so on. The subsequence number corresponding to the subsequence composed of the last 4000 input words is 31.

[0101] In step S203, after the input sequence is divided into multiple subsequences, the first position code corresponding to each input word in the subsequence can be obtained by position encoding the subsequence number. The first position code represents the position information of the subsequence corresponding to each input word. It can be understood that the first position codes corresponding to multiple input words in the same subsequence are the same, while the first position codes of input words belonging to different subsequences are different. Specifically, the subsequence numbers corresponding to each subsequence can be used as a position sequence and input into a trained first position code embedding layer to generate a matrix composed of the first position codes corresponding to each input word, so as to determine the first position code of each input word. The first position code embedding layer can be a neural network layer of a large language model, trained together with other neural network layers of the large language model; or it can be added to the network structure of the trained large language model, and fine-tuned after freezing the parameters of other neural network layers of the large language model. The details of training the first position code embedding layer will be described later and will not be repeated here.

[0102] In step S204, the position serial number of the input word in the corresponding subsequence refers to the position serial number corresponding to each input word obtained after the input sequence is divided into the first number of subsequences, and the serial numbers of each input word are reallocated in each subsequence according to the order of each input word in the subsequence. It can be understood that since the length of the subsequence will not exceed the first preset sequence length, the position serial number of each input word in the corresponding subsequence will not be greater than the first preset sequence length. Exemplarily, the input sequence contains 8000 input words, and the first preset sequence length is 4000. At this time, the input sequence is divided into two subsequences, namely subsequence A and subsequence B, and the corresponding subsequence serial numbers are 0 and 1 respectively. For the 5000th input word b in the input sequence, it is the 1000th input word in subsequence B, then the position of the input word b in the corresponding subsequence B can be represented by the position serial number 1000. Similarly, the position of the 1000th input word in the input sequence in subsequence A can also be represented by the position serial number 1000. Afterwards, each input word is positionally encoded based on its position in the corresponding subsequence to obtain a corresponding second position code. At this time, since the length of a single subsequence will not exceed the first preset sequence length, that is, the position number of each input word in the corresponding subsequence will not be greater than the first preset sequence length, that is, it will not exceed the window length learned by the large language model during training, the sequence number of each input word used to generate the corresponding first position code is trained. Afterwards, the first position code and the second position code corresponding to a single input word can be superimposed as the target position code corresponding to the input word. Exemplarily, refer to Figure 4, when the first pre-sequence length is 4000, the 5000th input word a in the input sequence is the 1000th input word in the second subsequence, and the input word a obtains the corresponding first position code a_1 based on the subsequence number 1 (the subsequence number of the first subsequence is 0), and obtains the corresponding second position code a_2 based on the position number 1000 of the input word a in the second subsequence. By superimposing a_1 and a_2, the target position code corresponding to the input word a is obtained.

[0103] In step S205, the first query vector, the first key vector and the value vector are vectors obtained by performing matrix multiplication of the feature vector corresponding to the input word unit with the query weight matrix, the key weight matrix and the value weight matrix in the attention mechanism, respectively, wherein the query weight matrix, the key weight matrix and the value weight matrix are determined as parameters of the large language model during the training process of the large language model. For example, referring to Figure 5 , the input word is first processed by an embedding layer for word embedding, and each input word is represented as a corresponding word embedding vector. Then, the word embedding vector corresponding to the input word and the corresponding target position code are used as the input of the transformer module. The transformer module includes an attention mechanism. In the attention mechanism, the word embedding vector is multiplied by the query weight matrix, key weight matrix and value weight matrix of the attention mechanism respectively to obtain the corresponding first query vector, second query vector and value vector. For example, the feature vector corresponding to the input word is recorded as , the query weight matrix, key weight matrix and value weight matrix are respectively denoted as , and , then:

[0104]

[0105]

[0106]

[0107] in, , , Input word The corresponding first query vector, first key vector and value vector.

[0108] After obtaining the first query vector, first key vector and value vector corresponding to the input word, referring to the above calculation process, it can be seen that the obtained first query vector, first key vector and value vector themselves do not carry the position information corresponding to the input word. Therefore, at this time, the first query vector and first key vector corresponding to the input word are adjusted based on the target position encoding corresponding to the input word. Specifically, refer to Figure 6 , the target position code corresponding to the input word unit can be multiplied with the first query vector and the first key vector corresponding to the input word unit respectively to obtain the second query vector and the second key vector corresponding to the corresponding input word unit, so as to inject the position information corresponding to each input word unit into the process of attention processing, so that the attention mechanism can capture the position information of the word unit, thereby avoiding similar outputs for input sequences with the same word unit but different word unit orders. It should be noted that since the subsequent attention calculation process is actually based on the query vector and the key vector corresponding to each input word unit to perform matrix multiplication operations, the attention weights between the input word units are obtained, and then the value vector is weighted based on the attention weights to obtain the attention results. To make the large language model generate different outputs for input words with different position information, it is only necessary to change the attention weights of the input words under different position information. Therefore, based on the target position code corresponding to each input word unit, the first query vector and the first key vector corresponding to the input word unit are adjusted to obtain the corresponding second query vector and the second key vector. Exemplarily, the corresponding second query vector and the second key vector can be obtained by multiplying the target position code corresponding to the input word unit with the first query vector and the first key vector corresponding to the input word unit.

[0109] In step S206, after obtaining the second query vector and the second key vector corresponding to each input word, attention processing is performed based on the second query vector, the second key vector and the value vector corresponding to the input word. Specifically, the second key vectors corresponding to multiple words are transposed, and then each second query vector and the transposed second key vector are matrix multiplied to obtain the attention scores between the input words, and the corresponding attention score matrix is ​​generated. The attention score matrix is ​​then softmax processed to obtain the corresponding attention weight matrix, and then the attention weight matrix and the value matrix composed of the value vector are matrix multiplied, so as to perform weighted processing on each value vector based on the attention weight to obtain the result of attention processing. Specifically, the process of attention processing can be expressed as:

[0110]

[0111] in represents the output predicted based on the i-th input word; represents the second query vector corresponding to the i-th input word; represents the second key vector corresponding to the j-th input word after transposition; is a preset scaling factor, illustratively, It can be the dimension of the second key vector. For example, if the second key vector is a 1*128 vector, then is 128; Represents the value vector corresponding to the jth input word. After obtaining the result of the attention processing, the large language model also includes other modules for subsequent processing of the output of the attention mechanism to generate the output sequence of the large language model.

[0112] Exemplarily, in a large language model, a feedforward neural network can be set after the attention mechanism, and the feedforward neural network consists of two linear layers and an activation function. Specifically, the result of the attention processing and the input of the attention mechanism can be added through a residual connection to obtain a superposition result; then the superposition result is mapped to a higher dimension through a linear layer, for example, the linear layer can be composed of an h*4h linear mapping matrix, h is the number of hidden layers of the output of the attention mechanism, and the dimension of the superposition result can be mapped from h to 4h through this linear layer; then nonlinearity is introduced into the mapped superposition result through an activation function, so that the large language model can capture richer feature representations, wherein the activation function can be a ReLU activation function; then, the activated result is reduced to the original dimension through another linear layer to obtain the corresponding output, for example, the first linear layer consists of an h*4h linear mapping matrix, and the activated result can be reduced to the original dimension through a 4h*h linear mapping matrix. After that, the output of the feedforward neural network can be mapped into a corresponding output sequence to obtain an output sequence corresponding to the input data.

[0113] In steps S201 to S206, after converting the input data of the large language model into an input sequence, the input sequence is divided into a plurality of subsequences based on the first preset sequence length, and for each input word, the first position code corresponding to the input word is determined based on the subsequence number corresponding to the subsequence where the input word is located, and the position code is performed based on the position of the input word in the corresponding subsequence, a second position code corresponding to the input word is generated, and the target position code corresponding to the input word is determined based on the first position code and the second position code corresponding to the input word. In this way, during the inference process, even if the length of the input sequence of the large language model exceeds the first preset sequence length, it can be ensured that the position code used for each output word is trained. Afterwards, the first query vector and the first key vector corresponding to the input word are adjusted based on the target position encoding corresponding to the input word to obtain the corresponding second query vector and second key vector, and the position information corresponding to the trained target position encoding is injected into the attention processing process, and attention processing is performed based on the second query vector, the second key vector and the value vector corresponding to each input word to obtain an output sequence corresponding to the input sequence, so that when the processing length of the large language model exceeds the window length learned by the large language model during the training process, the position encoding used is also trained to ensure that the performance of the large language model will not decrease, thereby improving the extrapolation of the large language model during the reasoning process.

[0114] The above is an overall description of steps S201 to S206 in the sequence processing method of the present invention. Since step S201, step S203 and step S205 have been described in detail above, the following focuses on further detailed description of step S202, step S204 and step S206.

[0115] Detailed description of step S202

[0116] In one embodiment, referring to Figure 7 , step S202 includes:

[0117] Step S701, determining a first quotient of a first length and a first preset sequence length, and rounding up the first quotient to obtain a first batch of times;

[0118] Step S702 , dividing each first preset sequence length input word in the input sequence into a subsequence in sequence, until each input word in the input sequence is divided into a subsequence, to obtain a first batch of subsequences.

[0119] In step S701, the quotient of the first length and the first preset sequence length is calculated, and the minimum number of subsequences obtained when the input sequence is divided into multiple subsequences whose lengths are not greater than the first preset sequence length can be determined. It can be understood that since the length of the input sequence is not fixed, when calculating the first quotient between the first length of the input sequence and the first preset sequence length, the first quotient obtained may not be an integer. Based on this, it is necessary to round up the first quotient, so that when the input sequence is divided, the number of subsequences required to make the length of each subsequence not greater than the first preset sequence length can be obtained. Exemplarily, the first length of the input sequence is 10000, the first preset sequence length is 4000, the first quotient between the first length and the first preset sequence length is 2.5, and the first batch number obtained after rounding up is 3. At this time, the input sequence is divided into at least 3 subsequences, so that the length of a single subsequence is not greater than the first preset sequence length.

[0120] In step S702, each input word in the input sequence is ordered. Based on this, each first preset sequence length input word in the input sequence can be divided into a subsequence according to the order of each input word, so as to obtain the first batch of subsequences. Exemplarily, the first length of the input sequence is 10000, and the first preset sequence length is 4000. At this time, the first 4000 input words in the input sequence are divided into the first subsequence, the 4001st to 8000th input words are divided into the second subsequence, and the 8001st to 10000th input words are divided into the third subsequence. In this way, the input sequence is divided into the first batch of subsequences.

[0121] In addition, since in step S702, each input word of the first preset sequence length in the input sequence is divided into a subsequence in turn, except for the last subsequence, the number of input words contained in other subsequences is the first preset sequence length. In this way, when the length of the input sequence changes, that is, when the number of input words contained in the input sequence increases, it is only necessary to generate corresponding position codes for the newly added input words, and there is no need to re-position code other input words in the input sequence. For example, in a large language model with only a decoder structure, the large language model will predict an output word at each time step and add the predicted output word to the input sequence as the input word of the next time step. At this time, it is only necessary to generate the first position code and the second position code corresponding to the added output word and superimpose them as the corresponding target position code, without modifying the target position code of other input words in the input sequence. Exemplarily, the first length of the input sequence at the i-th time step is 8000, the first preset sequence length is 4000, and at the i+1-th time step, the output word at the i-th time step is added to the output word of the i-th time step. After being added to the input sequence, the first length of the input sequence is 8001. At this time, for the newly added input word , since the second subsequence also contains 4000 input tokens, we only need to replace the input tokens As the first input word in the third subsequence, and perform the corresponding position encoding, that is, generate a word based on the subsequence number 2 corresponding to the third subsequence The corresponding first position code, and the word unit generated based on position number 0 The corresponding second position code, and The corresponding first position codes and second position codes are superimposed to obtain the corresponding target position codes, without the need to regenerate position codes for the first 8000 input word units in the previous input sequence.

[0122] In step S701 to step S702, the first batch of times is determined based on the first length of the input sequence and the first preset sequence length; thereafter, each first preset sequence length input word in the input sequence is sequentially divided into a subsequence, thereby dividing the input sequence into the first batch of times subsequences, so that the lengths of the other subsequences except the last subsequence are all the first preset sequence length, and the length of the last subsequence is not greater than the first preset sequence length. Thus, the position number corresponding to each input word is not greater than the first preset sequence length, that is, the position information of each input word will not exceed the window length learned by the large language model during the training process, thereby ensuring that the position encoding used by the large language model during the inference process is trained, so that the large language model can accurately process the input sequence whose length is greater than the sample sequence during inference after being trained using a sample sequence of limited length, thereby improving the extrapolation of the large language model.

[0123] In addition, in another possible implementation, after determining the first batch of times, the input sequence can also be divided into multiple subsequences based on the first batch of times and the first length corresponding to the input sequence. Specifically, the quotient between the first length and the first batch of times is calculated, and rounded up to obtain the maximum number of input words contained in a single subsequence, which is recorded as n. Afterwards, each n consecutive input words in the input sequence are divided into a subsequence in turn, thereby dividing the input sequence into the first batch of times of subsequences. Exemplarily, the first length is 10000, and the first batch of times is 3. At this time, if the input sequence is divided into three subsequences, the maximum number of input words contained in a single subsequence is 3334. Therefore, the first 3334 input words are divided into the first subsequence, the 3335th to 6668th input words are divided into the second subsequence, and the 6669th to 10000th input words are divided into the third subsequence.

[0124] In one embodiment, referring to Figure 8 In step S702, each input word of the first preset sequence length in the input sequence is divided into a subsequence in sequence, including:

[0125] Step S801, determining the original position number of each input word in the input sequence;

[0126] Step S802, for each input word unit, determining a second quotient between the corresponding original position serial number and the first preset sequence length, and rounding down the second quotient to obtain a first serial number corresponding to the input word unit;

[0127] Step S803: divide multiple input word units corresponding to the same first sequence number into a subsequence, and use the first sequence number corresponding to the input word unit in the subsequence as the subsequence sequence number corresponding to the subsequence.

[0128] In step S801, since each input word in the input sequence is in order, the position number corresponding to each input word in the input sequence is the original position number, and the original position number reflects the position order of the subword corresponding to each input word in the input data. For example, the input sequence is a sequence consisting of 8000 ordered input words, and the original position numbers corresponding to the 8000 ordered input words are: 0, 1, 2, ..., 7998, 7999.

[0129] In step S802, for each input word, based on the division of the corresponding original position serial number and the first preset sequence length, the second quotient corresponding to each input word is obtained. The second quotient can reflect that after the input sequence is divided into multiple subsequences not greater than the first preset sequence length, the subsequence corresponding to the input word is the nth subsequence obtained after the input sequence is segmented. Afterwards, the second quotient corresponding to the input word is rounded down, and the result of the rounding down is used as the subsequence serial number corresponding to the input word. Specifically, the first serial number corresponding to each input can be determined by referring to the following formula:

[0130] ;

[0131] in Indicates the first sequence number corresponding to the input word. is the original position number corresponding to the input word, Indicates the first preset sequence length; Represents the floor function.

[0132] For example, the input word The corresponding original position number is 2000, and the first preset sequence length is 4000. At this time, the input word The corresponding second quotient is 0.5, which is rounded down to 0, that is, the input word The corresponding first sequence number is 0. Similarly, the input word units with original position sequence numbers between 0 and 3999 have corresponding first sequence numbers of 0. The input word units with original position sequence numbers between 4000 and 7999 have corresponding first sequence numbers of 1, and so on. In this way, the first sequence numbers corresponding to the input word units can be determined, wherein, except for the largest first sequence number, the remaining first sequence numbers all correspond to the input word units of the first preset sequence length.

[0133] In step S803, after determining the first sequence number corresponding to each input word, since the original position sequence numbers corresponding to each input word are continuous, multiple input words with the same first sequence number are also continuous, and at this time, except for the largest first sequence number, the remaining first sequence numbers all correspond to the first preset sequence length input words. Based on this, the input words corresponding to the same first sequence number are divided into a subsequence. In this way, every first preset sequence length input word in the input sequence can be divided into a subsequence, and a first batch of subsequences is obtained. Exemplarily, referring to Fig. 9 , the input sequence is composed of L input words, the original position numbers corresponding to the L input words are 0 to L-1 respectively, the first preset sequence length is w, after obtaining the first number corresponding to the input word, the first number corresponding to the input word whose original position number is 0 to w-1 is 0, and the first number corresponding to the input word whose original position number is w to L-1 is 1; at this time, the input words with the first number 0 and 1 are divided into a subsequence respectively, and two subsequences are obtained, which are respectively recorded as subsequence 1 and subsequence 2, and the subsequence numbers corresponding to the two subsequences are 0 and 1 respectively.

[0134] In addition, since the first sequence number corresponding to each input word is obtained by rounding down the second quotient, at this time, for the first subsequence, the corresponding subsequence sequence number is 0, that is, the input word whose original position sequence number is less than the first preset sequence length, the corresponding subsequence sequence number is 0, and the first position code generated based on the subsequence sequence number can also be regarded as 0. In this way, after the first position code and the second position code corresponding to the input word are superimposed, the target position code corresponding to the input word in the first subsequence is consistent with the second position code, and the second position code is obtained based on the position of each input word in the corresponding subsequence. For any input word in the first subsequence, the position sequence number of the input word in the first subsequence is the original position sequence number of the input word. That is, at this time, for the input word in the first subsequence, its target position encoding is consistent with the position encoding when the input sequence is not divided into subsequences. In this way, for input words that do not exceed the window length learned by the large language model during the training process, the obtained position encoding is consistent with that when the input sequence is not split. In this way, the sequence processing method proposed in the present disclosure can be compatible with the position encoding method of the large language model in the related art.

[0135] In the embodiment disclosed in steps S801 to S803, the original position serial number of each input word in the input sequence is determined, and then the second quotient corresponding to each input word is determined based on the original position serial number of each input word and the first preset sequence length, and the second quotient is rounded down to obtain the first serial number corresponding to the input word, and then the input words corresponding to the same first serial number are divided into a subsequence, so that every first preset sequence length input words in the input sequence are divided into a subsequence to obtain a first batch of subsequences, and the first serial number corresponding to the input word in the subsequence is used as the subsequence number of the subsequence, so as to determine the subsequence number corresponding to each subsequence, so as to subsequently generate the first position code corresponding to each input word based on the subsequence number.

[0136] Detailed description of step S204

[0137] In one embodiment, referring to Fig.10 In step S204, position encoding is performed based on the position of the input word in the corresponding subsequence to obtain a second position code corresponding to each input word, including:

[0138] Step S1001, determining the original position sequence number corresponding to the input word based on the position of the input word in the input sequence;

[0139] Step S1002, taking a modulus of a first preset sequence length based on the original position sequence number corresponding to the input word unit to obtain a first remainder;

[0140] Step S1003, performing rotation position coding on the first remainder to obtain a second position code corresponding to the input word unit.

[0141] In step S1001, the details of the original position sequence number may refer to the above step S801 and will not be repeated here.

[0142] In step S1002, after determining the original position sequence number corresponding to the input word, a first preset sequence length is modulo processed based on the original position sequence number to obtain a first remainder corresponding to the input word. This first remainder represents the position sequence number of the corresponding input word remaining in the input sequence after the input sequence is segmented into multiple subsequences with a length of the first preset sequence length. The first remainder can represent the position sequence number of the corresponding input word in the subsequence to which it belongs. For example, the input word The corresponding original position number is 5000, and the first preset sequence length is 4000. Then the input word The corresponding first remainder is 1000. At this time, the input word That is, the input word in the second subsequence obtained after the input sequence is segmented, and in the second subsequence, the input word The corresponding position number is 1000. Specifically, step S1002 can be expressed as: ,in is the first remainder corresponding to the input word, When inputting the original position number corresponding to the word unit, is the first preset sequence length. Fig. 9 , the input sequence is composed of L input words, the original position numbers corresponding to the L input words are 0 to L-1, the first preset sequence length is w, the original position numbers of the first w input words in the input sequence are modulo the first preset sequence length, and the obtained numbers are 0 to w-1, and the original position numbers of the wth to L-1th words are modulo w, and the obtained numbers are also 0 to w-1, thereby determining the position numbers of each word in subsequence 1 and subsequence 2 in the corresponding subsequence.

[0143] In step S1003, since the first remainder can reflect the position number of the input word in the corresponding subsequence, the rotation position code is performed on each input word based on the corresponding first remainder to obtain the second position code corresponding to the input word. Specifically, the position number of the input word in the corresponding subsequence is multiplied by the imaginary unit i and the given rotation angle The product of is used as the exponent of the natural base to obtain the corresponding second position encoding. The process of rotation position encoding can be expressed by the following formula:

[0144] ;

[0145] in, Represents the input word; Represents the input word the corresponding second position code; is the position number of the input word in the corresponding subsequence, that is, the input word The corresponding first remainder; is the base of natural logarithms; is an imaginary unit; is the given rotation angle.

[0146] In the embodiment disclosed in steps S1001 to S1003, the original position number of each input word in the input sequence is determined, and a modulo operation is performed on the first preset sequence length based on the original position number to obtain a first remainder that can represent the position number of the input word when it is divided into the corresponding subsequence, thereby mapping the original position number of the input word to the position number of the input word in the corresponding subsequence, and then, a rotation position code is performed based on the first remainder corresponding to the input word to obtain a second position code for representing the position of the input word in the corresponding subsequence.

[0147] The first remainder corresponding to the input word unit is rotated and position-coded to generate the second position code corresponding to the input word unit. Then, the first position code and the second position code corresponding to the first input word unit are superimposed to obtain the target position code corresponding to the input word unit. Specifically, the subsequence The target position encoding of the mth input word in the corresponding subsequence can be expressed as ),in, Represents the input word The corresponding first position code, is the input word The subsequence number corresponding to the subsequence to which it belongs, Represents the functional relationship of mapping the subsequence number to the corresponding first position code.

[0148] Detailed description of step S206

[0149] In one embodiment, after the extrapolation of the large language model is improved by the sequence processing method of the above embodiment, the large language model can process longer input sequences more stably, but this will cause the performance of the large language model within the learned window length to deteriorate. This is because when the large language model processes the input sequence, in the attention mechanism, it is necessary to perform exponential normalization on the attention scores between the first input word and each second input word. A longer input sequence means that when generating the output corresponding to the first input word, more second input words need to be paid attention to, which results in a significant reduction in the attention weight between the first input word and other input words in the same subsequence.

[0150] Based on this, in one embodiment, referring to Fig.11 , step S206 includes:

[0151] Step S1101, for a single sample second query vector, performing matrix multiplication on the second query vector and each second key vector to obtain an attention score between the first input word unit and each second input word unit;

[0152] Step S1102, determining an attention scaling factor based on a difference between a subsequence number of a subsequence corresponding to the first input word and a subsequence number of a subsequence corresponding to the second input word, and scaling the attention score between the first input word and the second input word based on the attention scaling factor to obtain a scaled attention score corresponding to the second input word, wherein each scaled attention score constitutes a scaled attention score matrix;

[0153] Step S1103, perform exponential normalization processing on the scaled attention score matrix to obtain an attention weight matrix, and perform matrix multiplication on the attention weight matrix and the value matrix to obtain the result of attention processing.

[0154] In step S1101, the first input word refers to the input word corresponding to the second query vector, and the second input word refers to the input word corresponding to the second key vector. It can be understood that in the process of attention calculation, it is necessary to calculate the attention scores between each input word. Based on this, for the second query vector corresponding to each input word, the second query vector is matrix multiplied with the second key vector corresponding to each input word to obtain the attention score between the first input word and each second input word. Specifically, each second key vector can be transposed first, and then the second query vector and the transposed second key vector are matrix multiplied. Since the second query vector and the second key vector are adjusted based on the corresponding target position code, that is, the second query vector is obtained by multiplying the first query vector of the first input word with the corresponding target position code, and the second key vector is obtained by multiplying the first key vector of the second input word with the corresponding target position code, the second query vector and the second key vector can be respectively expressed as:

[0155] ;

[0156] ;

[0157]

[0158] in, represents the second query vector; represents the first input word; m represents the position number of the first input word in the corresponding subsequence; represents the subsequence number corresponding to the subsequence to which the first input word belongs, that is, the second query vector is the subsequence number The query vector corresponding to the mth input word in the subsequence of ; is the query weight matrix in the attention mechanism; is the key weight matrix in the attention mechanism; is the base of natural logarithms, is an imaginary unit, is the angle variable; is a function for generating a corresponding first position code according to a subsequence number; represents the second key vector, represents the transposed second key vector; represents the second input word; n represents the position number of the second input word in the corresponding subsequence, is the subsequence number of the subsequence to which the second input word belongs; represents the attention score between the first input token and the second input token, Represents a function that takes the real part.

[0159] Exemplarily, the second query vector and the second key vector are both vectors with a shape of 1*n. After transposing the second key vector, a column vector with a shape of n*1 is obtained. Then, the second query vector and the transposed second key vector are multiplied to obtain a value, which is the attention score between the first input word corresponding to the second query vector and the second input word corresponding to the second key vector.

[0160] In a possible implementation, the second query vectors corresponding to each input word can be concatenated in sequence to form a query matrix, and each matrix row in the query matrix is ​​a second query vector corresponding to an input word; similarly, the second key vectors corresponding to each input word are concatenated in sequence along the dimension of the matrix row to form a key matrix, and then the key matrix is ​​transposed, and then the query matrix and the transposed key matrix are matrix multiplied to obtain an attention score matrix, wherein each element in the attention score matrix represents the attention score of an input word to another input word. Exemplarily, the query matrix is ​​a matrix with a shape of N*K, that is, the query matrix is ​​composed of the second query vectors corresponding to N input words, and the key matrix is ​​a matrix with a shape of M*K, that is, the key matrix is ​​composed of the second key vectors corresponding to M input words, where M and N can be equal. After transposing the key matrix, a matrix with a shape of K*M is obtained. After matrix multiplication of the query matrix and the transposed key matrix, an attention score matrix with a shape of N*M is obtained. The element in the i-th row and j-th column of the attention score matrix represents the attention score of the i-th input word to the j-th input word.

[0161] In step S1102, it can be understood that the first input word and the second input word may belong to different subsequences. When the first input word and the second input word belong to the same subsequence, it means that the first input word and the second input word are in the same attention calculation window, and the relative position difference between the first input word and the second input word is small, that is, the distance between the first input word and the second input word is small. At this time, the association between the second input word and the output corresponding to the first input word is relatively high. On the contrary, when the difference between the subsequence number of the subsequence to which the first input word belongs and the subsequence number of the subsequence to which the second input word belongs is larger, it means that the distance between the subsequence to which the first input word belongs and the subsequence to which the second input word belongs is farther, that is, the distance between the first input word and the second input word is farther, and at this time, the association between the second input word and the output corresponding to the first input word is relatively low. Based on this, according to the difference between the subsequence number corresponding to the first input word and the subsequence number corresponding to the second input word, the attention scaling factor between the first input word and the second input word is determined, and then the corresponding attention score is scaled based on the attention scaling factor; specifically, the attention scaling factor can be expressed by the following formula:

[0162] ;

[0163] in, represents the attention scaling factor between the first input token and the second input token, Represents the difference between the subsequence number corresponding to the first input word and the subsequence number corresponding to the second input word; it is not difficult to see from the above formula that the larger the difference between the subsequence number corresponding to the first input word and the subsequence number corresponding to the second input word, that is, the farther the distance between the subsequence corresponding to the first input word and the subsequence corresponding to the second input word, the smaller the corresponding attention scaling factor. After calculating the corresponding attention scaling factor, the corresponding attention score is scaled based on the attention scaling factor. Specifically, the attention scaling factor is multiplied by the corresponding attention score to obtain the scaled attention score. It can be understood that since the larger the difference between the subsequence number corresponding to the first input word and the subsequence number corresponding to the second input word, the smaller the corresponding attention scaling factor, the greater the reduction in the corresponding attention score. Therefore, the corresponding attention scores are scaled by the attention scaling factor, that is, the attention weight between the first input word and the input words that are relatively close becomes higher, while the attention weight between the input words that are relatively far away is reduced. When predicting the output corresponding to the first input word, it will rely more on other words in the same subsequence as the first input word, and reduce the dependence on input words in other subsequences, so that the attention mechanism in the large language model can maintain a good long-range attenuation characteristic, and improve the performance of the window length learned by the large language model during the training process.

[0164] After scaling the corresponding attention scores based on the attention scaling factor to obtain the scaled attention scores, the scaled attention scores between each first input word and each second input word can form a scaled attention score matrix. Specifically, the scaled attention scores between a single first input word and each second input word can be used as a matrix row, and the scaled attention scores corresponding to different first input words can be used as different matrix rows, thereby obtaining a scaled attention score matrix. Exemplarily, the scaled attention matrix can be a matrix with a shape of m*n, where the first Line The elements of the column are recorded as , Indicates that based on the first The second query vector corresponding to the input word is the same as the first The scaled attention score of the second key vector corresponding to the input word, that is, The input word is the first The scaled attention score of the input tokens, 1≤ ≤m,1≤ ≤n, it can be understood that in the scaled attention score matrix, Each element of the row is the first The scaled attention scores of the input tokens for each input token in the input sequence.

[0165] In step S1103, after obtaining the scaled attention score matrix, the scaled attention score matrix is ​​subjected to exponential normalization. Specifically, each matrix row of the scaled attention score matrix is ​​subjected to exponential normalization, that is, each element in the scaled attention score matrix is ​​used as the exponent of the natural base to obtain an exponential attention score matrix, and then each matrix row in the exponential attention score matrix is ​​subjected to normalization to obtain a corresponding attention weight matrix. For example, the process of exponential normalization of the scaled attention score matrix can refer to the following formula:

[0166] ;

[0167] in, Representation and scaled attention scores The corresponding attention weights are, represents the i-th element of any matrix row in the scaled attention score matrix, e is the natural base, Representation and The jth element in the matrix row, is the number of elements in a single matrix row in the scaled attention matrix, Express The exponential results of all elements in the matrix row are summed. In this way, the attention weights corresponding to each scaled attention score are obtained, thereby converting the scaled attention score matrix into the corresponding attention weight matrix.

[0168] After obtaining the attention weight matrix, the value matrix is ​​weighted based on the attention weight matrix, that is, the attention weight matrix is ​​multiplied by the value matrix to obtain the result of attention processing, where the value matrix is ​​composed of the value vectors corresponding to each input word in the input sequence concatenated in sequence.

[0169] Specifically, the process from step S1101 to step S1103 can be described by the following formula:

[0170] ;

[0171] ;

[0172] in, represents the difference between the subsequence number of the subsequence to which the input word corresponding to the second query vector belongs and the subsequence number of the subsequence to which the input word corresponding to the second key vector belongs; represents an attention scaling factor between the input word corresponding to the second query vector and the input word corresponding to the second key vector; Represents the results of attention processing; represents the query matrix consisting of the second query vectors corresponding to each input word in the input sequence, represents the matrix obtained by transposing the key matrix consisting of the second key vectors corresponding to each input word, represents the value matrix composed of the value vectors corresponding to each input word, Indicates the dimension of the second key vector.

[0173] In the embodiment disclosed in steps S1101 to S1103, by performing matrix multiplication of the second query vector and each second key vector, an attention score between the first input word corresponding to the second query vector and the second input word corresponding to the second key vector is obtained, and based on the difference between the subsequence number of the subsequence to which the first input word belongs and the subsequence number of the subsequence to which the second input word belongs, the corresponding attention scaling factor of the first input word and the second input word is determined, and the corresponding attention scores are scaled accordingly, and the scaled attention scores are then exponentially normalized to obtain an attention weight matrix, so that the attention weight between the first input word and the second input word that is relatively close becomes higher, while the attention weight between the first input word and the second input word that is relatively far away becomes lower. Then, the attention weight matrix and the value matrix are multiplied to obtain the result of attention processing. In this way, when predicting the output corresponding to the first input word based on the result of attention processing, it will rely more on other words in the same subsequence as the first input word, and reduce the dependence on input words in other subsequences, so that the attention mechanism in the large language model can maintain good long-range attenuation characteristics and improve the performance of the large language model within the window length learned during training.

[0174] Detailed description of training a large language model in the disclosed embodiment

[0175] In one embodiment, referring to Fig.12 , the training process of the large language model includes:

[0176] Step S1201, obtaining a sample sequence for training a large language model, wherein the sample sequence includes a plurality of sample word units, and a sequence length of the sample sequence is not greater than a first preset sequence length;

[0177] Step S1202, dividing the sample sequence into a plurality of sample subsequences, determining the sample subsequence number corresponding to each sample word unit, and generating the sample first position code corresponding to each sample word unit based on the sample subsequence number;

[0178] Step S1203, performing rotation position coding on each sample word unit based on the position of the sample word unit in the corresponding sample subsequence to obtain a sample second position code corresponding to each sample word unit, and superimposing the sample second position code corresponding to each sample word unit with the sample first position code respectively to obtain a sample target position code corresponding to each sample word unit;

[0179] Step S1204, generating a corresponding sample first query vector, a sample first key vector and a sample value vector based on each sample word;

[0180] Step S1205, for each sample word, adjusting the corresponding sample first query vector and sample first key vector based on the corresponding sample target position code to obtain a sample second query vector and a sample second key vector corresponding to the sample word;

[0181] Step S1206, performing attention processing based on the sample second query vectors, sample second key vectors and sample value vectors corresponding to the multiple sample word units, and generating a sample output sequence corresponding to the sample sequence based on the result of the attention processing, and performing back propagation based on the sample output sequence to optimize the parameters of the large language model until the first condition is met, thereby obtaining the trained large language model.

[0182] In step S1201, the sample sequence may be a sequence obtained based on the sample text, and the sample sequence is composed of a plurality of sample tokens. It is understandable that the length of the sample sequence is limited, and the window length learned by the large language model during the training process is the maximum length of the sample sequence, that is, the first preset sequence length is the maximum value of the length of the sample sequence, and the length of the sample sequence is not greater than the first preset sequence length.

[0183] In step S1202, after obtaining the sample sequence, the sample sequence is divided into a plurality of sample subsequences, and the sample subsequence number corresponding to each sample subsequence is determined. Specifically, since the length of the sample sequence is not greater than the first preset sequence length, it is not possible to divide the sample sequence by dividing each input word of the first preset sequence length in the sample sequence into a sample subsequence. In this embodiment, the number of sample subsequences to be divided can be determined by random sampling, and then the sample sequence can be equally divided according to the number of samples, so as to divide the sample sequence into a plurality of sample subsequences, and the sample subsequence number corresponding to each sample subsequence is determined, and the sample subsequence number corresponding to the sample subsequence is the sample subsequence number corresponding to the sample word in the sample subsequence. For example, the length of the sample sequence is 4000, and the number of sample subsequences obtained by random sampling that need to be divided is 40. At this time, every 100 consecutive sample word units in the sample sequence are divided into a sample subsequence, so that the sample sequence is evenly divided into 40 sample subsequences, and the length of each sample subsequence is 100. The sample subsequence numbers corresponding to these 100 sample subsequences are 0, 1, 2, ... 99, 100. At this time, the sample subsequence numbers corresponding to the first 100 sample word units are all 0, and the sample subsequence numbers corresponding to the 101st to 200th sample word units are all 1, and so on, to obtain the sample subsequence numbers corresponding to each sample word unit.

[0184] In one embodiment, the large language model includes a first position code embedding layer. In the training stage, the first position code embedding layer is used to generate a sample first position code corresponding to each sample word based on the sample subsequence number. In the inference stage, the first position code embedding layer is used to determine the first position code corresponding to each input word in the input sequence based on the corresponding subsequence number. After determining the sample subsequence number corresponding to each sample word, a sample position sequence is constructed based on the sample subsequence number corresponding to each sample word, and the sample position sequence is input into the first position code embedding layer in the large language model. The first position code embedding layer generates a matrix composed of sample first position codes of each sample word based on the sample position sequence, thereby obtaining the sample first position code corresponding to each sample word.

[0185] In step S1203, after the sample sequence is divided into a plurality of sample subsequences, a rotation position encoding is performed based on the position serial number of each sample word in the corresponding sample subsequence, and a second position code corresponding to the sample word is obtained. Specifically, each sample word in the sample sequence will be divided into a corresponding sample subsequence. Since the sample word is ordered in the sample sequence, after being divided into the corresponding sample subsequence, each sample word in a single sample subsequence is also ordered. Based on this, the position serial number of each sample word in the corresponding sample subsequence can be determined. Exemplarily, the sample sequence contains 4000 sample words, and the number of sample subsequences obtained by sampling is 40. At this time, the sample sequence is evenly divided into 40 sample subsequences, and each sample subsequence should contain 100 sample words. At this time, the first 100 sample words are divided into the first sample subsequence, and the corresponding position numbers are 0, 1, 2, ... 98, 99; the 101st to 200th sample words are divided into the second sample subsequence, and the corresponding position numbers in the second sample subsequence are 0, 1, 2... 98, 99, and so on. The position number of each sample word in the corresponding sample subsequence can be determined. Afterwards, rotation position encoding is performed based on the position number of each sample word in the corresponding sample subsequence, so as to obtain the sample second position code corresponding to each sample word. Specifically, the detailed process of performing rotation position encoding can refer to the detailed process of the above step S703, which will not be repeated here. Afterwards, the sample first position code and the sample second position code corresponding to each sample word unit are superimposed to obtain the sample target position code corresponding to each sample word unit.

[0186] In step S1204, the process of generating the corresponding sample first query vector, sample first key vector and sample value vector based on the sample word can refer to the process of generating the corresponding first query vector, first key vector and value vector based on the input word in the above step S205, which will not be repeated here.

[0187] In step S1205, for each sample word-gram, the target position code corresponding to the sample word-gram is multiplied by the corresponding sample first query vector and sample first key vector respectively to obtain the corresponding sample second query vector and sample second key vector.

[0188] In step S1206, after obtaining the sample second query vector, sample second key vector and sample value vector corresponding to each sample word, the sample second query vector, sample second key vector and sample value vector corresponding to each sample word are spliced ​​in sequence to obtain the sample query matrix, sample key matrix and sample value matrix corresponding to the sample sequence. Afterwards, attention calculation is performed based on the sample query matrix, sample key matrix and sample value matrix, that is, the sample key matrix is ​​transposed, and then the sample query matrix and the transposed sample key matrix are matrix multiplied to obtain the sample attention score matrix, and then the matrix rows in the sample attention score matrix are exponentially normalized in parallel to obtain the sample attention weight matrix, and the sample value matrix is ​​weighted based on the sample attention weight matrix, that is, the sample attention weight matrix and the sample value matrix are matrix multiplied to obtain the attention processing result corresponding to the sample sequence, and the sample output sequence corresponding to the sample sequence is generated based on the result of the attention processing. Afterwards, the corresponding loss value is calculated based on the generated sample output sequence, and back propagation is performed to calculate the gradient of the parameters of each neural network layer of the large language model, and then gradient descent is performed to optimize the parameters of the large language model. Afterwards, it is determined whether the large language model after parameter optimization meets the preset first condition. If the first condition is met, it indicates that the large language model has been trained.

[0189] Specifically, the first condition is a preset condition for judging whether the large language model has been trained. For example, the first condition can be to apply the large language model after optimizing the parameters to a preset test set, perform the corresponding reasoning task based on the data in the test set, and evaluate the results of performing the reasoning task to obtain multiple performance parameters of the large language model, and judge whether the large language model has been trained based on the evaluation results. Thus, the large language model learns how to divide the input sequence into multiple subsequences, and then generates the target position code corresponding to the input word based on the subsequence number of the subsequence to which each word belongs and the position number of the input word in the corresponding subsequence, and performs attention processing based on the target position code. In this way, in the process of reasoning, even if the length of the input sequence of the large language model exceeds the first preset sequence length, the input sequence can be divided into multiple subsequences to ensure that the position code used by each output word is trained, thereby ensuring that the performance of the large language model will not decrease and improving the extrapolation of the large language model in the reasoning process.

[0190] In the embodiment disclosed in steps S1201 to S1206, a sample sequence is divided into a plurality of sample subsequences, and then a sample first position code corresponding to the sample word is generated based on the sequence number of the sample subsequence to which each sample word belongs, and a sample second position code corresponding to the sample word is determined based on the position of the sample word in the corresponding sample subsequence, and the first position code and the second position code corresponding to the sample word are superimposed to obtain a sample target position code corresponding to the sample word, and then the sample first query vector and the sample first key vector corresponding to the sample word are adjusted with the target position code, and attention processing is performed based on the sample second query vector and the sample second key vector obtained after the adjustment to generate a sample output sequence corresponding to the sample sequence and perform back propagation thereon, thereby optimizing the parameters of the large language model, so that the large language model learns how to divide the input sequence into a plurality of subsequences, and then a target position code corresponding to the input word is generated based on the subsequence sequence number of the subsequence to which each word belongs and the position sequence number of the input word in the corresponding subsequence, and attention processing is performed based on the target position code. When the large language model is trained, during the reasoning process, even if the length of the input sequence of the large language model exceeds the first preset sequence length, the input sequence can be divided into multiple subsequences to ensure that the position encoding used for each output word unit is trained, thereby ensuring that the performance of the large language model does not decrease and improving the extrapolation of the large language model during the reasoning process.

[0191] In one embodiment, referring to Fig.13 , step S1202 includes:

[0192] Step S1301, determining a second number based on a first preset sequence length and a preset extrapolation length, and determining a third number based on the second number, wherein the third number is a positive integer not greater than the second number;

[0193] Step S1302, determine the third quotient between the first preset sequence length and the third number, and round up the third quotient to obtain a fourth number, and divide every fourth number of sample words in the sample sequence into a sample subsequence in turn to obtain an ordered third number of sample subsequences, and determine the sequence number corresponding to each sample subsequence as the sample subsequence sequence number corresponding to the sample word in the sample subsequence.

[0194] In step S1301, the preset extrapolation length is the preset maximum length of the input sequence that the large language model needs to process during the inference process. For example, if the user expects the large language model to process an input sequence of 128k in length during the inference process, the preset extrapolation length is 128k. It can be understood that the longer the input sequence that the large language model can process, the more context information the large language model can capture. However, in fact, after the length of the input sequence processed by the large language model exceeds a certain length, processing a longer sequence does not improve the generation quality of the large language model much. The increase in sequence length will cause the amount of calculation of the large language model to increase exponentially during the inference process, resulting in a decrease in the inference efficiency of the large language model. Based on this, in the related art, the length of the sequence processed by the large language model will be limited to a certain length to avoid too low inference efficiency while ensuring the generation quality of the large language model. Based on this, the user will set the maximum length of the sequence processed by the large language model in the inference stage, and the maximum length is the preset extrapolation length. The second number is the maximum number of subsequences when the input sequence is divided into subsequences during the inference stage of the large language model. Specifically, the quotient of the preset extrapolation length and the first preset sequence length can be calculated and rounded up to obtain the second number. For example, the preset extrapolation length is 128k, the first preset sequence length is 4k, and the corresponding second number is 32.

[0195] After determining the second number, it can be understood that during the inference process, the length of the input sequence may be less than the preset extrapolation length. During the training process, it is necessary to enable the large language model to learn how to perform position encoding and attention processing based on the results of position encoding under different numbers of subsequences. Based on this, random sampling is performed from the interval of 1 to the second number to obtain a third number to determine the number of sample subsequences. At the same time, since the third number is sampled from the interval of 1 to the second number, when the third number is sampled as 1, the entire sample sequence is taken as a sample subsequence, and the length of the sample subsequence is the first preset sequence length. In this way, the large language model can accurately train the position encoding within the first preset sequence length during the training process, so that the window length learned by the large language model is the first preset sequence length.

[0196] At the same time, in this embodiment, the third number is determined by randomly sampling from 1 to the second number, which can limit the value range of the third number, avoid the third number value is too dispersed so that the training process of the large language model is difficult to converge, and at the same time, the large language model can be directed to learn within a preset extrapolation length, divide the input sequence into multiple subsequences and generate the target position encoding of each input word unit and perform subsequent attention processing, thereby improving the training efficiency of the large language model.

[0197] In step S1302, after the third number is determined, a third quotient is determined based on the first preset sequence length and the third number, the third quotient representing the number of sample words contained in a single sample subsequence after the sample sequence is divided into the third number of sample subsequences. Since the third quotient may not be an integer, the third quotient is rounded up to obtain a fourth number. Thereafter, every fourth number of sample words in the sample sequence is divided into a sample subsequence in turn, so that the sample sequence can be divided into an ordered third number of sample subsequences, and the sequence number of each sample subsequence is determined to obtain the sample subsequence sequence number corresponding to the sample word in the sample subsequence. Exemplarily, the first preset sequence length is 4000, and the third number is 30. At this time, the third quotient is 133.333, which is rounded up to 134. At this time, the first 134 sample words in the sample sequence are divided into a sample subsequence, and the 135th to 268th sample words are divided into a second subsequence, and so on. The sample sequence can be divided into a third number of sample subsequences, and the sequence number of each sample subsequence is determined at the same time. For example, the sample subsequence consisting of the first 134 sample words has a sample subsequence sequence number of 0, the sample subsequence consisting of the 135th to 268th sample words has a sample subsequence sequence number of 1, and so on.

[0198] In another possible implementation, after determining the fourth number, the sample original position serial number of each sample word unit in the sample sequence may also be determined. For example, the sample sequence contains 4000 sample word units, and the corresponding original position serial numbers are 0, 1, 2, ..., 3998, and 3999. Then, the quotient between the original position serial number of each sample word unit and the fourth number is calculated, and the quotient is rounded down to obtain the second serial number corresponding to the sample word unit. Then, multiple sample word units corresponding to the same second serial number are regarded as a sample sequence, and the second serial number corresponding to each sample word unit in the sample subsequence is regarded as the sample subsequence serial number corresponding to the sample subsequence, so as to divide the sample sequence into a third number of sample subsequences, and determine the sample subsequence serial number corresponding to each sample subsequence.

[0199] In the embodiment disclosed in step S1301 to step S1302, the second number is determined based on the preset extrapolation length and the first preset sequence length, and then a third number is obtained by random sampling from a positive integer not greater than the second number, and then the sample sequence is evenly divided into the third number of sample subsequences, and the sample subsequence number corresponding to each sample word is determined. By limiting the value of the third number to a range not greater than the second number, on the one hand, it is avoided that the large language model is difficult to converge during the training process due to the excessive dispersion of the number of sample subsequences, and at the same time, the large language model can be directed to learn to divide the input sequence within the preset extrapolation length into multiple subsequences, so as to positionally encode each input word, and perform the subsequent attention processing process, thereby improving the training efficiency of the large language model.

[0200] In one embodiment, referring to Fig.14 , step S1203 includes:

[0201] Step S1401, determining the original position sequence number of each sample word in the sample sequence;

[0202] Step S1402 , for each sample word unit, modulo the fourth number based on the corresponding sample original position serial number to obtain a second remainder, and perform rotation position encoding based on the second remainder to obtain a sample second position code corresponding to the sample word unit.

[0203] In step S1401, the details of the original position serial number of the sample can refer to the relevant description in the above step S1002, which will not be repeated here.

[0204] In step S1402, the fourth number is modulo based on the sample original position serial number corresponding to each sample word unit, and the position serial number after each sample word unit is divided into the corresponding sample subsequence can be determined. Exemplarily, the sample sequence includes 4000 sample words, and the corresponding sample original position serial numbers are 0, 1, 2, ... 3999 in sequence, and the fourth number is 100. At this time, for the 150th sample word unit, its corresponding sample original position serial number is 149, and the second remainder obtained by modulo the fourth number is 49, which is the position serial number of the sample word unit in the second sample subsequence, indicating that the sample word unit is the 50th sample word unit in the second sample subsequence.

[0205] After determining the second remainder corresponding to each sample word unit, rotation position encoding is performed based on the second remainder. The details of the rotation position encoding can refer to the detailed process of the above step S703, which will not be repeated here.

[0206] In the embodiment disclosed in steps S1401 to S1402, the sample original position serial number corresponding to each sample word is determined, and then the fourth number is modulo processed based on the sample original position serial number to obtain a second remainder for ensuring the position serial number of the corresponding sample word in the sample subsequence to which it belongs. Then, a rotation position encoding is performed based on the second remainder to obtain the second position code corresponding to each sample word.

[0207] In one embodiment, referring to Fig.15 In step S1206, attention processing is performed based on the sample second query vector, the sample second key vector and the sample value vector, including:

[0208] Step S1501, for a single sample second query vector, perform matrix multiplication on the single sample second query vector and each sample second key vector to obtain a sample attention score between a first sample word-unit and each second sample word-unit, where the first sample word-unit is a sample word-unit corresponding to the single sample second query vector, and the second sample word-unit is a sample word-unit corresponding to the sample second key vector;

[0209] Step S1502, based on the difference between the sample subsequence number corresponding to the first sample word-unit and the sample subsequence number corresponding to the second sample word-unit, determine the sample attention scaling factor between the first sample word-unit and the second sample word-unit, and scale the corresponding sample attention score based on the sample attention scaling factor to obtain the corresponding scaled sample attention score, and each scaled sample attention score constitutes a scaled sample attention score matrix;

[0210] Step S1503, exponentially normalize the scaled sample attention score matrix to obtain a sample attention weight matrix, and perform matrix multiplication on the sample attention weight matrix and the sample value matrix to obtain the result of attention processing, wherein the sample value matrix is ​​composed of the sample value vectors corresponding to each sample word.

[0211] In step S1501, the process of attention processing is the process of calculating the attention scores between sample tokens based on the second query vector and the second key vector corresponding to each sample token. Specifically, for each sample token, the sample token is regarded as the first sample token, each sample token in the sample sequence is sequentially regarded as the second sample token, and the second query vector corresponding to the first sample token is sequentially matrix-multiplied with the second key vector corresponding to each second sample token, so as to obtain the attention scores between the first sample token and the second sample token.

[0212] Specifically, the sample second query vectors corresponding to each sample word unit can be spliced ​​into a sample query matrix along the sequence dimension, and the sample second key vectors corresponding to each sample word unit can be spliced ​​in sequence along the sequence dimension to obtain a sample key matrix. Then, the sample key matrix is ​​transposed, and then the sample query matrix is ​​multiplied with the transposed sample key matrix to obtain a sample attention score matrix. Each element in the sample attention score matrix is ​​the attention score between each pair of sample words in the sample sequence.

[0213] In step S1502, since the first sample word and the second sample word may belong to different sample subsequences, when position encoding is performed after the sequence is divided into subsequences, the position encoding cannot explicitly express the relative distance between the first sample word and the second sample word, which weakens the long-range attenuation characteristic of the sample attention score between the first sample word and the second sample word, which will cause the performance of the large language model within the window length to deteriorate. In order to make the attention mechanism itself maintain a certain long-range attenuation characteristic, the first sample word is relatively close in distance, that is, the attention weight between the sample words in the same sample subsequence is relatively high, and the relative distance is relatively far, that is, the attention weight between the sample words in different sample subsequences is relatively low. Therefore, the corresponding attention scaling factor is determined based on the difference between the sample subsequence number corresponding to the first sample word and the sample subsequence number corresponding to the second sample word. It can be understood that the size of the attention scaling factor is negatively correlated with the difference between the sample subsequence number corresponding to the first sample word and the sample subsequence number corresponding to the second sample word, that is, the larger the difference between the sample subsequence number corresponding to the first sample word and the sample subsequence number corresponding to the second sample word, the smaller the corresponding attention scaling factor, and vice versa. Afterwards, the attention scaling factor is multiplied by the corresponding sample attention score to scale the corresponding sample attention score, so that the attention weight between the first sample word and the second sample word in the same sample subsequence is increased, while the attention weight between the second sample word in a different sample subsequence is reduced, so that the attention mechanism in the large language model can maintain a good long-range attenuation characteristic.

[0214] In step S1503, after obtaining the scaled sample attention score matrix, the scaled sample attention score matrix is ​​subjected to exponential normalization processing, so as to convert the sample attention scores between each sample word into the corresponding sample attention weight. Specifically, in the scaled sample attention score matrix, the elements in each matrix row respectively represent the attention score of a single sample word to each sample word in the sample sequence. For example, the elements in the first matrix row of the scaled sample attention score matrix are the scaled sample attention scores of the first sample word to each sample word in the sample sequence. At this time, each matrix row in the scaled sample attention score matrix is ​​subjected to exponential normalization processing respectively, so as to obtain the sample attention weight matrix. At the same time, the sample value vectors corresponding to each sample word are concatenated in sequence to obtain the sample value matrix corresponding to the sample sequence. The sample attention weight matrix and the sample value matrix are subjected to matrix multiplication operation to obtain the attention processing result corresponding to the sample sequence.

[0215] In the embodiment disclosed in steps S1501 to S1503, a corresponding attention scaling factor is determined based on the difference between the sample subsequence number corresponding to the first sample word and the subsequence number corresponding to the second sample word, and in the process of attention processing, the corresponding sample attention score is scaled based on the attention scaling factor, so that the attention mechanism in the large language model can maintain the characteristics of long-range attenuation. In this way, after the training of the large language model is completed, the large language model can maintain a high output quality while improving the extrapolation performance.

[0216] Description of the apparatus and device of the present disclosure

[0217] The present disclosure also provides an electronic device 1600, including:

[0218] at least one processor, and

[0219] a memory communicatively connected to at least one processor; wherein,

[0220] The memory stores instructions, and the instructions are executed by at least one processor so that when the at least one processor executes the instructions, a method as described in any one of the above embodiments of the present application is implemented.

[0221] Combine the following Fig.16 The hardware structure of the electronic device is described in detail. The electronic device includes: a processor 1610 , a memory 1620 , an input / output interface 1630 , a communication interface 1640 and a bus 1650 .

[0222] The processor 1610 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present disclosure;

[0223] The memory 1620 may be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1620 may store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1620, and the processor 1610 calls and executes the sequence processing method of the embodiments of this disclosure;

[0224] Input / output interface 1630, used to implement information input and output;

[0225] Communication interface 1640, used to realize communication interaction between the device and other devices, which can be realized through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.); and

[0226] Bus 1650 , which transmits information between various components of the device (e.g., processor 1610 , memory 1620 , input / output interface 1630 , and communication interface 1640 );

[0227] The processor 1610 , the memory 1620 , the input / output interface 1630 , and the communication interface 1640 are connected to each other in communication within the device via the bus 1650 .

[0228] An embodiment of the present application also provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the sequence processing method of the above embodiment, which will not be repeated here.

[0229] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "comprises" and "comprising" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0230] It should be understood that in the present disclosure, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0231] It should be understood that in the description of the embodiments of the present disclosure, the meaning of multiple (or multiple items) is more than two, greater than, less than, exceed, etc. are understood as not including the number, and above, below, within, etc. are understood as including the number.

[0232] In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0233] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0234] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0235] It should also be understood that the various implementations provided in the embodiments of the present disclosure can be combined arbitrarily to achieve different technical effects.

[0236] The above is a specific description of the implementation methods of the present disclosure, but the present disclosure is not limited to the above implementation methods. Technical personnel familiar with the art can also make various equivalent modifications or substitutions without violating the spirit of the present disclosure. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present disclosure.

Claims

1. A sequence processing method, characterized in that: The method comprises: Converting input data of the large language model into a corresponding input sequence, and determining a first length of the input sequence; Determine a first batch of times based on the first length and a first preset sequence length corresponding to the large language model, divide the input sequence into a first batch of times of ordered subsequences, and determine a subsequence number corresponding to each of the subsequences; For each subsequence, determining a first position code corresponding to each input word in the subsequence based on the corresponding subsequence serial number, wherein the first position codes corresponding to each input word in a single subsequence are the same; For each input word, position coding is performed based on the position number of the input word in the corresponding subsequence to obtain a second position code corresponding to each input word, and a target position code corresponding to the input word is determined based on the first position code and the second position code corresponding to the input word; For each input word, generate a first query vector, a first key vector, and a value vector corresponding to the input word, and adjust the first query vector and the first key vector corresponding to the input word based on the target position code corresponding to the input word to obtain a second query vector and a second key vector corresponding to the input word; Attention processing is performed based on each of the second query vectors, each of the second key vectors, and each of the value vectors corresponding to each of the input word units, and an output sequence corresponding to the input data is obtained based on a result of the attention processing.

2. The sequence processing method according to claim 1, characterized in that: The converting the input data of the large language model into a corresponding input sequence and determining a first length of the input sequence includes: Performing word segmentation processing on the input data to obtain a plurality of subwords, performing lemma processing on each of the subwords to obtain the input lemmas corresponding to each of the subwords, and determining a first number corresponding to the plurality of input lemmas, wherein the first number is the number of input lemmas obtained after the input data is segmented; The input sequence is determined based on the plurality of input word-grams, and the first length is determined based on the first number.

3. The sequence processing method according to claim 1, characterized in that: The determining of the first batch of times based on the first length and the first preset sequence length corresponding to the large language model, and dividing the input sequence into ordered first batch of times of subsequences, comprises: Determine a first quotient of the first length and the first preset sequence length, and round up the first quotient to obtain the first batch of times; Divide each input word of a first preset sequence length in the input sequence into a subsequence in sequence, until each input word of the input sequence is divided into a subsequence, to obtain a first batch of subsequences.

4. The sequence processing method according to claim 3, characterized in that: The step of sequentially dividing each input word of the first preset sequence length in the input sequence into a subsequence includes: Determine the original position sequence number of each input word in the input sequence; For each of the input word units, determine a second quotient between the corresponding original position serial number and the first preset sequence length, and round down the second quotient to obtain a first serial number corresponding to the input word unit; The plurality of input word units corresponding to the same first sequence number are divided into a subsequence, and the first sequence numbers corresponding to the input word units in the subsequence are used as the subsequence sequence numbers of the subsequence.

5. The sequence processing method according to claim 1, characterized in that: The performing position encoding based on the position of the input word in the corresponding subsequence to obtain a second position encoding corresponding to each input word includes: Determine the original position sequence number corresponding to the input word based on the position of the input word in the input sequence; Taking a modulus of the first preset sequence length based on the original position sequence number corresponding to the input word unit to obtain a first remainder; The first remainder is subjected to rotation position encoding to obtain a second position code corresponding to the input word unit.

6. The sequence processing method according to claim 1, characterized in that: The performing attention processing based on the second query vector, the second key vector, and the value vector corresponding to each of the input word units includes: For a single second query vector, perform matrix multiplication on the second query vector and each of the second key vectors to obtain an attention score between a first input word unit and each of the second input words unit, wherein the first input word unit is an input word unit corresponding to the second query vector, the second input word unit is an input word unit corresponding to the second key vector, and each of the attention scores constitutes an attention score matrix; Determine an attention scaling factor based on a difference between the subsequence number of the subsequence corresponding to the first input word and the subsequence number of the subsequence corresponding to the second input word, and scale the attention score between the first input word and the second input word based on the attention scaling factor to obtain a scaled attention score corresponding to the second input word, wherein each scaled attention score constitutes a scaled attention score matrix; The scaled attention score matrix is ​​subjected to exponential normalization to obtain an attention weight matrix, and the attention weight matrix is ​​matrix multiplied with a value matrix to obtain a result of attention processing, wherein the value matrix is ​​composed of the concatenation of each of the value vectors.

7. The sequence processing method according to claim 1, characterized in that: The large language model is trained in the following way: Acquire a sample sequence for training the large language model, wherein the sample sequence includes a plurality of sample word units, and a sequence length of the sample sequence is not greater than the first preset sequence length; Dividing the sample sequence into a plurality of sample subsequences, determining a sample subsequence number corresponding to each sample word unit, and generating a sample first position code corresponding to each sample word unit based on the sample subsequence number; Performing rotation position encoding on each of the sample word units based on the position of the sample word unit in the corresponding sample subsequence to obtain a sample second position code corresponding to each of the sample word units, and superimposing the sample second position code corresponding to each of the sample word units with the sample first position code to obtain a sample target position code corresponding to each of the sample word units; Generate a corresponding sample first query vector, a sample first key vector and a sample value vector based on each of the sample word-units; For each of the sample word-grams, adjusting the corresponding sample first query vector and sample first key vector based on the corresponding sample target position code to obtain a sample second query vector and a sample second key vector corresponding to the sample word-gram; Attention processing is performed based on the sample second query vector, the sample second key vector and the sample value vector corresponding to the plurality of sample word units, and a sample output sequence corresponding to the sample sequence is generated based on the result of the attention processing, and back propagation is performed based on the sample output sequence to optimize the parameters of the large language model until the first condition is met, thereby obtaining the trained large language model.

8. The sequence processing method according to claim 7, characterized in that: The determining of the sample subsequence number corresponding to each of the sample word units includes: Determine a second number based on the first preset sequence length and a preset extrapolation length, and determine a third number based on the second number, wherein the third number is a positive integer not greater than the second number; Determine a third quotient between the first preset sequence length and the third number, and round up the third quotient to obtain a fourth number, divide every fourth number of the sample word units in the sample sequence into a sample subsequence to obtain ordered sample subsequences of the third number, and determine a sequence number corresponding to each of the sample subsequences as the sample subsequence sequence number corresponding to the sample word unit in the sample subsequence.

9. The sequence processing method according to claim 8, characterized in that: The step of performing rotation position encoding on each of the sample word units based on the position of the sample word unit in the corresponding sample subsequence to obtain a sample second position encoding corresponding to each of the sample word units includes: Determine the sample original position sequence number of each of the sample word units in the sample sequence; For each of the sample word units, the fourth number is modulo based on the corresponding sample original position serial number to obtain a second remainder, and a rotation position code is performed based on the second remainder to obtain a sample second position code corresponding to the sample word unit.

10. The sequence processing method according to claim 8, characterized in that: The performing attention processing based on the sample second query vector, the sample second key vector and the sample value vector corresponding to the plurality of sample word-units includes: For a single sample second query vector, matrix multiply the single sample second query vector by each of the sample second key vectors to obtain a sample attention score between a first sample word-element and each of the second sample word-element, wherein the first sample word-element is a sample word-element corresponding to the single sample second query vector, and the second sample word-element is a sample word-element corresponding to the sample second key vector; Based on the difference between the sample subsequence number corresponding to the first sample word-unit and the sample subsequence number corresponding to the second sample word-unit, determine a sample attention scaling factor between the first sample word-unit and the second sample word-unit, and scale the corresponding sample attention scores based on the sample attention scaling factor to obtain corresponding scaled sample attention scores, wherein each scaled sample attention score constitutes a scaled sample attention score matrix; The scaled sample attention score matrix is ​​subjected to exponential normalization to obtain a sample attention weight matrix, and the sample attention weight matrix and the sample value matrix are subjected to matrix multiplication to obtain a result of attention processing, wherein the sample value matrix is ​​composed of the sample value vectors corresponding to each of the sample word units.

11. An electronic device, characterized in that: The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the sequence processing method as described in any one of claims 1 to 10 is realized.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the sequence processing method according to any one of claims 1 to 10.

Images