Method for processing super-long sequence based on improved transformer model

Abstract

The application provides an ultra-long sequence processing method based on an improved Transform model, and the method is implemented based on a WSformer model which is constructed by improving the Transform model, and specifically comprises the following steps: S1, pre-training of the WSformer model; S1.1, an ultra-long sequence is divided into small sequences by using a sequence segmentation method; S1.2, feature extraction based on a double-layer attention mechanism; word-level feature extraction and sentence-level feature extraction are performed by using the double-layer attention mechanism, and feature encoding of the entire sequence is obtained; S2, improved position vector coding; in the coding process of step S1.2, the position coding of the word is adjusted by using a trigonometric function, and the coding of the position vector is realized. The method can effectively reduce the time cost of long text sequence coding, more accurately depict the position relationship between words, realize effective modeling of long sequence texts, and improve the prediction accuracy and the calculation efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and particularly relates to a super-long sequence processing method based on an improved Transformer model. BACKGROUND

[0002] In the field of natural language processing, recurrent neural networks and convolutional neural networks have been the mainstream methods for sequence modeling. In 2014, Google proposed the seq2seq model (Sutskever I, Vinyals O, Le Q V. Sequence to sequence learning with neural networks[J]. Advances in neural information processing systems, 2014, 27.), which is a simple sequence-to-sequence conversion method. In the field of natural language processing, its applications mainly include translation, summarization, and poetry writing. Since the introduction of seq2seq, there have been many variants of seq2seq, but most of them are based on recurrent neural networks (RNN) and are widely used in various sequence modeling, such as text and speech. In 2017, the paper Attention is all you need first proposed a sequence modeling method based on self-attention mechanism Transformer. However, for modeling of ultra-long sequences, the Transformer method needs to calculate the attention weight of each word in the sequence to other words, which has a very high computational cost. In addition, the Transformer method lacks in capturing long-distance relationships between words. In 2020, Longformer (Beltagy I, Peters M E, Cohan A. Longformer: The long-document transformer[J]. arXiv preprint arXiv:2004.05150, 2020.) used a sliding window mechanism or randomly selected some words to reduce the computational cost of the self-attention mechanism in Transformer, and then captured long-distance information. However, when facing an ultra-long sequence, the sliding window mechanism is likely to destroy the information connection between the words in the near distance, and the way of randomly selecting some words is likely to ignore important word information. In 2022, Non-Position-Transformer (Haviv A, Ram O, Press O, et al. Transformer Language Models without Positional Encodings Still Learn Positional Information[J]. arXiv preprint arXiv:2203.16634, 2022.) proposed that it can capture position information without considering the position encoding of the words.However, when facing a super long sequence, not considering the position encoding of the vocabulary cannot bring efficiency improvement, and the position encoding of the transformer itself also has certain defects.

[0003] In the expression of text language, the meaning expressed by a paragraph is often more accurate than that expressed by a sentence. This is because people will infer the meaning of surrounding sentences based on the meaning expressed by a sentence when reading, so that the meaning expressed by the entire paragraph is more accurate, while ignoring some unnecessary interference information. However, for the processing of super long sequence, the traditional transformer model faces a series of problems such as too large amount of calculation, difficulty in obtaining long distance attention weight between words, and difficulty in distinguishing the sequence of words by position vector coding, and the like. An improved transformer model based super long sequence processing method is proposed. SUMMARY

[0004] The purpose of the present application is to solve the problems in the prior art, and an improved transformer model based super long sequence processing method is proposed. By improving the traditional transformer method, a new super long sequence processing model WSformer is constructed, which can effectively model long sequence text and improve prediction accuracy and computational efficiency.

[0005] To achieve the above purpose, the present application proposes an improved transformer model based super long sequence processing method, which is realized based on the WSformer model constructed by improving the transformer model, and specifically includes the following steps:

[0006] S1. Pre-training of WSformer model:

[0007] S1.1. The super long sequence is divided into small sequences by using a sequence segmentation method;

[0008] S1.2. Feature extraction based on double-layer attention mechanism: word-level feature extraction and sentence-level feature extraction are performed by using a double-layer attention mechanism to obtain feature encoding of the entire sequence;

[0009] S2. Improved position vector coding: the position coding of the vocabulary is adjusted by using a trigonometric function in the coding process of step S1.2 to realize the coding of the position vector.

[0010] As a preferred, in step S1.1, the paragraph is divided into multiple sequences in units of sentences, which is represented by a matrix, the number of rows is the maximum value of the number of words contained in all sentences in the paragraph, and the number of columns is the number of sentences contained in the paragraph.

[0011] As preferred, in step S1.2, in the word-level feature extraction, the self-attention weight between words in each segmented small sequence is calculated, and then summarized; in the sentence-level feature extraction, the attention weight between each small sequence is calculated, and the encoding of the entire text sequence is obtained after summarization.

[0012] As preferred, in step S1, the specific process of pre-training of the WSformer model is as follows:

[0013] Step a. According to certain rules, a super-long sequence P is segmented into multiple relatively independent small sequences:

[0014] ;

[0015] wherein, n denotes the number of segmented small sequences, denotes the k th small sequence;

[0016] Step b. Enter the word-level attention layer, and use the self-attention mechanism to calculate the attention weight of all words in each small sequence:

[0017] ;

[0018] wherein, denotes the word embedding vector, B denotes the batch size of batch training, S denotes the maximum number of small sequences generated by the segmentation of a super-long sequence, wherein the super-long sequence is a paragraph, and the small sequence is a sentence; W denotes the maximum number of words in a small sequence, E W denotes the dimension of the word embedding vector; att() denotes the attention function, denotes the obtained word attention weight matrix;

[0019] Step c. The embedding encoding of the sentence is obtained by using the word-level autoencoder, as shown below:

[0020] ;

[0021] wherein, denotes the output vector of the word-level autoencoder, H W denotes the dimension of the hidden layer of the word-level autoencoder;

[0022] Step d. After the output of the word-level autoencoder is dimensionally converted, it enters the sentence-level attention layer, and the self-attention mechanism is continued to be used to calculate the attention weight of each small sequence for modeling the super-long sequence:

[0023] ;

[0024] wherein, denotes the dimension conversion function, denotes the obtained sentence attention weight matrix, denotes the dimension of the obtained sentence embedding vector after dimension conversion;

[0025] Step e. After the sentence-level autoencoder and dimension conversion, the embedding encoding of the super-long sequence is obtained:

[0026] ;

[0027] wherein denotes the output vector of the sentence-level autoencoder, H S denotes the dimension of the hidden layer in the sentence-level autoencoder;

[0028] Step f. The output vector of the sentence-level autoencoder is dimensionally converted to obtain the paragraph embedding encoding, i.e. the output vector of the WSformer:

[0029] ;

[0030] wherein, denotes the obtained paragraph embedding vector, denotes the dimension of the paragraph vector.

[0031] As a preferred, in step a, when modeling the Chinese sequence, the period is taken as the delimiter by default, and the long text sequence is divided into sentences.

[0032] As a preferred, in step S2, the specific method for encoding the position vector is: when performing the dot product of the two lexical position vectors, first, the odd and even columns of one of the position vectors are exchanged, and then the even columns of the two position vectors are multiplied by -1 to obtain the lexical a and b The encoding of the position vector is as follows:

[0033] ;

[0034] ;

[0035] wherein, d denotes the dimension of the position vector, i.e. the number of words contained in the segmented sentence;

[0036] Let , then based on the improved position encoding, the dot product of the two lexical position vectors is calculated as follows:

[0037] ;

[0038] After simultaneously exchanging the positions of the word pairs, and change as follows:

[0039] ;

[0040] ;

[0041] Let The dot product of the two lexical position vectors is calculated as follows:

[0042] ;

[0043] From the properties of the sin function, .

[0044] The beneficial effects of the present application are:

[0045] 1. The application discloses a long text sequence division method and a two-stage long text sequence coding scheme based on sentences, which effectively reduces the time cost of long text sequence coding.

[0046] 2. The application proposes a position vector coding method based on trigonometric functions, which is more sensitive to the position information before and after the vocabulary and can more accurately depict the position relationship between words.

[0047] 3. The experimental research results on multiple data sets show that compared with the traditional sequence coding method, the improved Transformer model can perform better in long text sequence coding.

[0048] The features and advantages of the present application will be described in detail in conjunction with the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0049] Figure 1 is the workflow diagram of the WSforme model;

[0050] Figure 2 is the flow diagram of feature extraction based on a double-layer attention mechanism during pre-training of the WSformer model;

[0051] Figure 3 is the principle diagram of sequence modeling of the WSformer model;

[0052] Figure 4 is the pre-training flow diagram of the WSformer model;

[0053] Figure 5 is the convergence time diagram of different models on the encyclopedia data set A. DETAILED DESCRIPTION

[0054] The application provides a double-layer Transformer model from the word level to the sentence level, namely a WSformer model. Specifically, a paragraph is divided into multiple sequences in units of sentences, and is represented by a matrix, the number of rows of which is the maximum number of words contained in all sentences in the paragraph, and the number of columns of which is the number of sentences contained in the paragraph. In the word-level encoding layer, the words in a sentence only pay attention to the attention weight calculation of other words in the sentence. In the sentence-level encoding layer, each sentence only pays attention to the attention weight calculation of other sentences in the paragraph, and no longer pays attention to the attention weight calculation of the words in the sentence. In addition, since the position encoding method based on the dot product operation of the word position vector has certain disadvantages, it is not sensitive to the order of the word position, therefore, the application further provides a position vector encoding method based on a trigonometric function, and the implementation method of the application will be described in detail below.

[0055] 1 RELATED WORK

[0056] In the traditional Transformer model, the encoding formula of the word position vector is as follows:

[0057] ;

[0058] ;

[0059] Given the words in a text sequence a and b , according to the above formula, the position vectors are as follows:

[0060] ;

[0061] ;

[0062] The Transformer model uses the vector dot product method to calculate the attention weight between two words, let , then the dot product of the word position vectors is as follows:

[0063] ;

[0064] Suppose that the positions of the words a and b are different, and the above calculation process is used to obtain , because the cosine function is an even function. That is, the traditional Transformer method cannot identify the front and back positions of two words, and the position relationship of the words is very important in sequence modeling, and has an important indicating effect on text understanding.

[0065] 2 WSformer model

[0066] When processing an ultra-long sequence, the meaning of the entire sequence does not depend on a certain word in it, but depends on a small text sequence, and the overall meaning of the sequence depends on the relationship between all the words in the entire text paragraph, and the relationship between the words is measured by self-attention weights.

[0067] In order to solve the shortcomings of the traditional Transformer in processing ultra-long text sequences, the present application proposes a new ultra-long sequence processing model WSformer based on the Transformer, and the working process is as shown in Figure 1 The long text in the Chinese data set is first divided into sentences by a sequence segmentation technique to obtain a plurality of small sequences with relatively short lengths; then a double-layer attention mechanism is used for word-level feature extraction and sentence-level feature extraction to obtain feature encoding of the entire sequence, so that the modeling method of divide and conquer greatly reduces the computational complexity of ultra-long sequence modeling. In the modeling process, a new position vector encoding method is also proposed to further improve the accuracy of sequence modeling. Finally, when the model pre-training is completed, the word vector encoding corresponding to all the words in the sequence is obtained, and various natural language processing downstream tasks are further used to verify the effectiveness of the model.

[0068] 2.1 Model pre-training

[0069] The pre-training process of the WSformer is as shown in Figure 2 , which mainly includes two steps: the first step is text embedding, which maps each word in the long sequence to a numerical embedding vector, and the embedding vector of each word is composed of character encoding, position encoding and segmentation encoding; the second step is feature extraction based on a double-layer attention mechanism, including word-level feature extraction and sentence-level feature extraction. In word-level feature extraction, only the self-attention weight between the words in each segmented small sequence is focused on, and then it is summarized. In sentence-level feature extraction, the attention weight between each small sequence is calculated, and the encoding of the entire text sequence is obtained after summarizing.

[0070] The WSformer performs sequence modeling based on an encoder-decoder, and the working principle is as shown in Figure 3 .

[0071] The pre-training process of the WSformer model is as shown in Figure 4 , through pre-training, the latent features have value attributes, and through the training and recovery of lossy features, the encoder can learn the most important features in the data. The specific process of pre-training is as follows:

[0072] 1) An ultra-long sequence is segmented according to certain rules to obtain a plurality of relatively independent small sequences: P ​

[0073] ;

[0074] wherein, n represents the number of small sequences cut out, represents the k th small sequence. The present application mainly models Chinese sequences, and a period is used as a delimiter by default to separate long text sequences by sentences.

[0075] 2) Enter the word-level attention layer, and use the self-attention mechanism to calculate the attention weight of all words in each small sequence:

[0076] ;

[0077] wherein, represents the word embedding vector, B represents the batch size of batch training, S represents the maximum number of small sequences (sentences) generated by segmenting a super-long sequence (paragraph), W represents the maximum number of words in a small sequence, and E W represents the dimension of the word embedding vector; att() represents an attention function, represents the obtained word attention weight matrix.

[0078] 3) The sentence embedding encoding is obtained by using the word-level autoencoder, as follows:

[0079] ;

[0080] wherein represents the output vector of the word-level autoencoder, H W represents the dimension of the hidden layer in the word-level autoencoder.

[0081] 4) After the output of the word-level autoencoder is dimensionally converted, it enters the sentence-level attention layer, and the self-attention mechanism is used to calculate the attention weight of each small sequence for modeling the super-long sequence:

[0082] ;

[0083] wherein, represents a dimension conversion function, represents the obtained sentence attention weight matrix, represents the dimension of the sentence embedding vector obtained after dimension conversion.

[0084] 5) After the sentence-level autoencoder and dimension conversion, the embedding encoding of the super-long sequence (paragraph) is obtained:

[0085] ;

[0086] wherein denotes the output vector of the sentence-level autoencoder, S denotes the dimension of the hidden layer in the sentence-level autoencoder.

[0087] 6) Dimension conversion is performed on the output vector of the sentence-level autoencoder to obtain a paragraph embedding code, that is, an output vector of the WSformer:

[0088] ;

[0089] wherein, denotes the obtained paragraph embedding vector, denotes the dimension of the paragraph vector.

[0090] Although the long text encoding method based on sequence segmentation will cause a part of information loss, the method is more likely to find representative features, so that the obtained word vector has self-adaptability and can be applied to different downstream tasks.

[0091] In the downstream task, first, the model parameters obtained in the pre-training stage are read, and then the parameters are further fine-tuned on the basis of the pre-training model according to the specific task to obtain the final model result. The present application intends to verify the sequence modeling effect of the WSformer by judging whether there is relevance between the text title and the long text content.

[0092] 2.2 Improved position encoding

[0093] The trigonometric function used for position encoding in the original Transformer is symmetrical, so the order of the words cannot be identified through the dot product of the position vectors, that is, the exchange of the positions of the word pairs has no effect on the attention weight. This is undesirable for text sequence encoding, because the adjustment of the order of the words will affect the overall meaning of the text. In order to solve the above problems, the position encoding of the words is adjusted according to the properties of the trigonometric function. When the dot product of the position vectors of two words is calculated, first, the odd and even columns of one of the position vectors are exchanged, and then the even columns of the two position vectors are multiplied by-1 to obtain the position vectors of the words a and b The encoding of the position vectors is as follows:

[0094] ;

[0095] ;

[0096] wherein, d denotes the dimension of the position vector, that is, the number of words contained in the segmented sentence;

[0097] Let Then, based on the improved position encoding, the dot product of the position vectors of two words is calculated as follows:

[0098] ;

[0099] At the same time, the positions of the vocabulary pairs are exchanged, and The changes are as follows:

[0100] ;

[0101] ;

[0102] Let The dot product of the two vocabulary position vectors is calculated as follows:

[0103] ;

[0104] From the properties of the sin function, That is, after the positions of the two words are exchanged, the attention weight obtained is different. The attention weight between words calculated by the improved position encoding above is not only related to the semantic and distance between words, but also closely related to the relative position between words, which conforms to the conventional idea of natural language understanding.

[0105] 3 Experimental analysis

[0106] In order to verify the effectiveness of the long text sequence modeling method proposed in the present application, the matching of the title and content of the long text is taken as the experimental object, and the experiment is carried out. The experimental process is as follows: first, the long text is segmented into sentences, and the CLS symbol is added between the sentences; then, each sentence is encoded by the Transformer to obtain a sentence-level vector; then, the sentence-level vector is encoded by the Transformer again to obtain a paragraph encoding; finally, the paragraph encoding is passed through a fully connected layer to obtain the encoding label of the long text sequence. After the title and the body of the given paragraph are processed according to the above, the corresponding label vectors are obtained, and the two label vectors are decoded and then passed through a binary classifier to output the result indicating whether the title and the body match.

[0107] 3.1 Dataset and comparison method

[0108] Two data sets are used in the experiment of the present application: Chinese data set A and Chinese data set B published in 2019. Among them, data set A selects the paragraphs with lengths in the interval [4000, 8000] in the two data sets as training and test samples, and the scale of each data set is about 21000, of which 10000 data are used for pre-training of the model, 5000 data are used for model training in the fine-tuning stage, and the remaining 6000 data are used for model testing. In order to ensure the effectiveness of the experimental results, there is no repetition between the data in different stages.

[0109] 3.2 Experimental environment

[0110] The experimental platform used in this paper is the 64-bit open source Linux operating system CentOS7, the processor is the CPU of Core I5-12500H, the memory is 16G, the video memory is 12GB of NVidia GeForce RTX 3060, the experimental environment is python3.7, and the deep learning framework based on TensorFlow is used, the version is 2.4.0.

[0111] 3.3 Parameter configuration

[0112] The initial learning rate of model training is set to 0.0003, and when the number of iterations reaches a certain number, the learning rate begins to decay, and the formula is as follows:

[0113] ;

[0114] where, d has two meanings: if the model has only one level of structure (relative to the three variants of the traditional Transformer and Longformer models proposed in this paper, the WSformer model has only one level (word level) parameters), then , N s represents the number of word-level WSformer model decoders; if the model has two levels of structure, then , H s represents the dimension of the sentence latent vector of the WSformer model, N es represents the number of decoders in the sentence-level WSformer model; step represents the number of steps of the current model training, and warmup represents the linear increase coefficient of the learning rate, which is set to 10000 in the experimental process.

[0115] In order to obtain the best parameters of the WSformer model, this experiment uses different sizes of initial parameters for pre-training. In the ablation experiment, the method of reference (Liu J, Yang Y, Lv S, et al. Attention-based BiGRU-CNN for Chinese question classification[J]. Journal of Ambient Intelligence and Humanized Computing, 2019: 1-12.) is used, and the number of decoders is set to twice the number of encoders in the word-level processing process.

[0116] 3.4 Experimental results and analysis

[0117] 3.4.1 Comparative experiments

[0118] To verify the effectiveness of the long sequence modeling method proposed in the present application, comparative analysis was performed on the traditional Transformer model, the Transformer model without position encoding (Haviv A, Ram O, Press O, et al. Transformer Language Models without Positional Encodings Still Learn Positional Information [J]. arXiv preprint arXiv:2203.16634, 2022.), three variants of the Longformer model, and the WSformer model (improved Transformer model) proposed in the present application. Table 1 and Table 2 respectively give the prediction results of different methods on two data sets, wherein the number in the right bracket of WSformer represents the number of words used for encoding the sentence, and the best experimental result obtained for each sequence length is indicated in bold.

[0119]

[0120]

[0121] As can be seen from the table, no matter on which data set, the original Transformer model has the worst prediction performance, the Longformer model considering position vector encoding has improved prediction accuracy compared with the original Transformer model, and the performance of the three variants has little difference. No matter under which sequence length, the WSformer proposed in the present application has greater performance improvement in prediction accuracy compared with the other five comparative models, indicating that the double-layer encoding structure and position vector encoding method proposed in the present application have certain advantages.

[0122] The long text sequence encoding based on the double-layer Transformer structure can greatly improve the efficiency of encoding, which is another great advantage of the model of the present application. In order to verify its effectiveness, the convergence speed of the above models trained on the encyclopedia data set A is recorded, as shown in Table 3. Figure 5 Figure 5 ​As can be seen from the table, with the increase of the modeling sequence length, the convergence speed of the WSformer model is obviously faster than that of the other several comparative models while ensuring the accuracy. In fact, the complexity of the encoding method adopted by the present application is only related to the complexity of sentence encoding, and is not much related to the length of the entire paragraph, because the present application adopts a two-stage encoding method. In summary, the model proposed by the present application has obvious advantages in both prediction accuracy and computational efficiency.

[0123] 3.4.2 Ablation experiment

[0124] In order to further verify the effectiveness of the two key technologies adopted in the WSformer model of the present application, i.e. the double-layer encoding structure and the improved position vector encoding, corresponding ablation experiments are carried out. The original Transformer model, the Transformer model based on the improved position vector encoding, the WSformer model only using the double-layer encoding structure, and the WSformer model proposed by the present application are used to test the prediction accuracy of the title and the text matching. The experimental results on two test data sets are shown in Tables 3 and 4.

[0125]

[0126]

[0127] As can be seen from the table, the performance of the original Transformer model is the worst, and the increase of any key technology has a certain effect on the prediction performance. Secondly, although the prediction accuracy of the WSformer model has certain fluctuations under different sequence lengths, the difference is not very obvious. Overall, the double-layer encoding structure is more effective than the position vector encoding in improving the performance. Finally, the WSformer model with the two key technologies has the best performance, which further shows that the two key technologies proposed by the present application have complementary effects and are indispensable.

[0128] The title and text matching experiments based on different sources of Chinese data sets show that the method of the present application has great improvement in prediction accuracy and computational efficiency compared with a series of sequence modeling methods such as traditional Transformer, Longformer and Transformer without position vector. Therefore, the WSformer model for ultra-long sequence modeling proposed by the present application greatly improves the encoding accuracy of long text sequences through sequence segmentation, upper attention mechanism and position encoding based on trigonometric function, which can alleviate the two defects of the traditional Transformer model that it is difficult to capture long-distance relationships and the position vector encoding is not sensitive to order.

[0129] The above examples are illustrative of the present application and are not limiting. Other embodiments of the present application will readily occur to those skilled in the art. The application is limited only by the claims.

Claims

1. A method for processing ultra-long sequences based on an improved Transformer model, characterized in that: The implementation of this method is based on the WSformer model, which is an improvement on the Transformer model. Specifically, it includes the following steps: Pre-training of the S1.WSformer model: S1.1 Use sequence segmentation methods to divide the very long sequence into smaller sequences; S1.

2. Feature extraction based on a two-layer attention mechanism: A two-layer attention mechanism is used to extract word-level features and sentence-level features to obtain the feature encoding of the entire sequence; S2. Improved position vector encoding: In the encoding process of step S1.2, trigonometric functions are used to adjust the position encoding of words to achieve the encoding of position vectors; In step S1.1, the paragraph is divided into multiple sequences based on sentences, represented by a matrix. The number of rows is the maximum number of words in all sentences in the paragraph, and the number of columns is the number of sentences in the paragraph. In step S1.2, in word-level feature extraction, the self-attention weights between words in each segmented small sequence are calculated and then summarized; in sentence-level feature extraction, the attention weights between each small sequence are calculated and summarized to obtain the encoding of the entire text sequence. In step S2, the specific method for encoding the position vectors is as follows: when performing the dot product of two word position vectors, first, the odd and even columns of one of the position vectors are interchanged, and then the even columns of both position vectors are multiplied by -1 to obtain the word... a and b The location vector is encoded as follows: ； ； in, d The dimension of the position vector represents the number of words in the segmented sentence. make Based on the improved positional encoding, the dot product of the two word position vectors is calculated as follows: ； After swapping the positions of the word pairs and The changes are as follows: ； ； make The dot product of the two word position vectors is calculated as follows: ； From the properties of the sin function, we know that .

2. The method for processing ultra-long sequences based on an improved Transformer model as described in claim 1, characterized in that: In step S1, the specific process of pre-training the WSformer model is as follows: Step a. Process a very long sequence according to certain rules. P The sequence is split into several relatively independent smaller sequences: ； in, n This indicates the number of smaller sequences that were split into. Indicates the first k A small sequence; Step b. Enter the word-level attention layer and use a self-attention mechanism to calculate the attention weights of all words in each small sequence: ； in, Let B represent the word embedding vector, and let B represent the batch size for batch training. S E represents the maximum number of small sequences generated by splitting a very long sequence, where the very long sequence is a paragraph and the small sequences are sentences; W represents the maximum number of words in the small sequence, and E represents the maximum number of words in the small sequence. W represents the dimension of the word embedding vector; att() represents the attention function. This represents the resulting word attention weight matrix; Step c. Obtain the sentence embedding encoding using a word-level autoencoder, as shown below: ； in, H represents the output vector of a word-level autoencoder. W This represents the dimension of the hidden layer in a word-level autoencoder; Step d. After dimensionality transformation, the output of the word-level autoencoder enters the sentence-level attention layer, where the self-attention mechanism is used to calculate the attention weights for each small sequence in modeling the ultra-long sequence. ； in, This represents the dimension transformation function. This represents the obtained sentence attention weight matrix. This represents the dimension of the sentence embedding vector obtained after dimension transformation; Step e. After sentence-level autoencoder and dimensionality transformation, the embedding encoding of the ultra-long sequence is obtained: ； in, H represents the output vector of the sentence-level autoencoder. S This represents the dimension of the hidden layer in a sentence-level autoencoder; Step f. Perform dimensionality transformation on the output vector of the sentence-level autoencoder to obtain the paragraph embedding code, i.e., the output vector of WSformer: ； in, This represents the obtained paragraph embedding vector. This represents the dimension of the paragraph vector.

3. The method for processing ultra-long sequences based on an improved Transformer model as described in claim 2, characterized in that: In step a, when modeling Chinese sequences, the period is used as the default delimiter, and long text sequences are separated into sentences.

Images