Transformers-based sequence classification prediction method and system

Abstract

The application provides a sequence classification prediction method and system based on a Transformer, which comprises the following steps: constructing an integer id word table, splitting all initial sequences into a training data set; establishing a Transformer neural network model for sequence classification prediction and performing specific initialization, adding a recursive position coding matrix to the model, inputting the training data set into the initialized model for classification prediction training, and completing the training when the cross-entropy loss is minimum; finally, splitting a text sequence data set to be classified and inputting the same into the trained Transformer neural network model for classification prediction to obtain a sequence classification prediction result; the application can capture linear structure information of a sequence while reducing the calculation time of feature capture in a sequence classification task through Summer initialization and Cumsum calculation, thereby improving the convergence ability and prediction accuracy of the model.

Description

Technical Field

[0001] This invention relates to the field of Transformer model prediction optimization technology, and more specifically, to a Transformer-based sequence classification prediction method and system. Background Technology

[0002] The Transformer neural network model is a model based entirely on the self-attention mechanism. Although the self-attention mechanism solves the problem that traditional models cannot consider global information, research on Transformers shows that the self-attention mechanism itself lacks temporal modeling capabilities. Introducing positional encoding can effectively solve the problem of Transformer's lack of temporal modeling capabilities.

[0003] For sequence classification tasks, temporal information is extremely important. For self-attention mechanisms, which lack temporal modeling capabilities, it's difficult to effectively characterize sequence information. To address this, transformers incorporate temporal information into self-attention mechanisms through absolute positional encoding. However, recent research indicates that absolute positional encoding has weak temporal modeling capabilities. Therefore, existing research generally uses two different methods—recursive positional modeling and relative positional modeling—to improve the weakness of absolute positional temporal modeling. Recursive positional encoding utilizes the inherent temporal structure of RNNs for positional modeling, possessing strong temporal capabilities and the ability to capture linear structural information of sequences. However, its lack of parallelization and complex computational methods result in slow model speed and it also suffers from gradient vanishing. Relative positional encoding models temporal information from the perspective of relative word distances, possessing strong temporal modeling capabilities while supporting parallel computation. However, relative positional encoding abandons the recursive structure, thus losing the ability to capture linear structural information of sequences. The key issue is how to effectively retain the advantages of recursive positional encoding, alleviate the gradient vanishing problem of RNNs, and improve its computational speed.

[0004] Positional encoding in Transformers is mainly divided into absolute positional encoding, relative positional encoding, and recursive positional encoding. Absolute positional encoding was initially used in the original Transformer, specifically the sinusoidal positional encoding. Subsequent research has shown that positional encoding during training performs better with large datasets. However, absolute positional encoding has drawbacks, including weak temporal modeling capabilities and the potential for overfitting. Therefore, current Transformer models primarily use relative positional encoding, which can express temporal information based on the relative distances between sequences. While relative positional encoding offers strong temporal modeling and parallel computing capabilities, its structure neglects the linear structural information of the sequences.

[0005] Traditional relative positional encoding sacrifices recursive structure for speed, thus losing the natural ability of recursive structure to capture sequence information. While traditional recursive positional encoding retains this ability, it suffers from high computational complexity and low parallelization, hindering improvements in both training and inference speeds. Therefore, in Transformer models used for sequence classification, a positional encoding method that combines the advantages of both relative and recursive positional encoding, efficiently captures the linear structure information of sequences, and converges easily has yet to emerge.

[0006] Current technologies disclose a short text classification prediction method based on self-attention and joint word and label recognition. This method includes: constructing a vector representation matrix of the short text sequence to obtain the text representation matrix after text self-attention transformation; constructing a vector representation matrix of the label sequence to obtain the interactive text representation matrix after short text and label interaction attention transformation; obtaining the text semantic representation vector z; and finally, comparing the predicted classification result of z with the actual short text label y through a fully connected layer to calculate the prediction error, and obtaining the optimal parameters through continuous iteration. Existing methods consider the mutual influence between words and labels, introducing a self-attention mechanism to utilize the category label information of the dataset itself to map the word vectors of labels and short texts to the same space, thereby achieving better interactive functionality, fully extracting contextual information, increasing interpretability, and improving the accuracy of classification results and the generalization ability of classification tasks. However, while improving the sequence classification accuracy of the model, existing methods cannot maintain low computational complexity, thus failing to balance computational speed and classification quality. Summary of the Invention

[0007] To overcome the shortcomings of existing technologies that cannot improve the accuracy of sequence classification while maintaining low computational complexity, this invention provides a Transformer-based sequence classification prediction method and system that can reduce the computational time complexity of the model, improve efficiency, and enhance the model's ability to capture the linear structure of sequences.

[0008] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0009] A Transformer-based sequence classification prediction method includes the following steps:

[0010] S1: Obtain the initial text sequence dataset and its corresponding categories. Split all sequences in the initial text sequence dataset and remove duplicate strings. Assign each string to a different integer ID and construct a vocabulary. Replace all strings in the split sequences with integer IDs according to the vocabulary and obtain the training dataset.

[0011] S2: Establish a Transformer neural network model for sequence classification prediction;

[0012] S3: Initialize the Transformer neural network model, add a position encoding matrix to the Transformer neural network model, and obtain the initialized Transformer neural network model;

[0013] S4: Input the dataset to be trained into the initialized Transformer neural network model for classification prediction training, and calculate the loss value of classification prediction using the preset loss function. When the loss value of classification prediction is minimized, the training is completed and the trained Transformer neural network model is obtained.

[0014] S5: Obtain the text sequence dataset to be classified, split the text sequence dataset to be classified, and input it into the trained Transformer neural network model for classification prediction to obtain the sequence classification prediction result.

[0015] Preferably, the specific method in step S1 is as follows:

[0016] Obtain the initial text sequence dataset and its corresponding categories, where the i-th sequence S... i and its corresponding i-th category T i Let it be denoted as a data pair (S) i ,T i );

[0017] The initial text sequence dataset includes Chinese sequence datasets, English sequence datasets, and other sequence datasets;

[0018] All sequences in the initial text sequence dataset are split into word-level strings and deduplicated. All strings are sorted in chronological order and an integer id is assigned to each string. Each string corresponds one-to-one with its integer id, and a vocabulary is constructed.

[0019] Each split sequence S i The strings in the dictionary are replaced with the corresponding integer IDs by querying the dictionary to obtain a sequence of integer IDs, and each sequence of integer IDs is filled with the integer ID 0 to the same preset length;

[0020] All filled integer id sequences and their corresponding categories T i Both datasets are saved together as the training dataset.

[0021] Preferably, the Transformer neural network model in step S2 includes a Cumsum layer, an attention layer, a first normalization layer, a feedforward layer, a second normalization layer, a global max pooling layer, a fully connected layer, and a Softmax layer connected in sequence.

[0022] The input of the Cumsum layer also forms a residual connection with the input of the first normalization layer, and the input of the feedforward layer also forms a residual connection with the input of the second normalization layer.

[0023] The Cumsum layer is used for Cumsum calculations.

[0024] Preferably, in step S3, the Transformer neural network model is initialized, and a positional encoding matrix is ​​added to the Transformer neural network model using Cumsum calculation to obtain the initialized Transformer neural network model. The specific method is as follows:

[0025] Initialize the first, second, and third weight matrices W in the Cumsum layer. q W k and W v And calculate and obtain the first, second and third attention matrices Q, K and V;

[0026] The first, second, and third attention matrices Q, K, and V are processed separately, and the first and second position encoding matrices M with the same dimensions as the first and second attention matrices Q and K are generated using the Summer initialization method. q and M k ;

[0027] According to the first and second position coding matrices M q and M k And the first and second attention matrices Q and K are calculated by Hadamard product and Cumsum, respectively, to obtain the initialized first and second attention matrices Q' and K'.

[0028] The initialized first and second attention matrices Q' and K', as well as the third attention matrix V, are used as the attention matrices of the initialized Transformer neural network model to complete the model initialization.

[0029] Preferably, the method for initializing Summer is as follows:

[0030] Set an initializer for a truncated normal distribution with a standard deviation of 0.005 and a mean of 0, and generate positive and negative values ​​near 0. Then, convert all generated negative values ​​to positive values ​​using absolute values.

[0031] The first and second positional encoding matrices M are obtained by Cumsum calculation in the -1 dimension. q and M k .

[0032] Preferably, based on the first and second position coding matrices M q and M k The first and second attention matrices Q and K are obtained by calculating the Hadamard product and Cumsum, respectively, to obtain the initialized first and second attention matrices Q' and K'. The specific method is as follows:

[0033] The initialized first and second attention matrices Q' and K' are obtained according to the following formulas:

[0034]

[0035]

[0036] Here, ⊙ represents the Hadamard product operation, and Cumsum calculation specifically calculates the prefix sum of a preset matrix along the time step dimension.

[0037] Preferably, the attention mechanism in the attention layer is specifically expressed as the following formula:

[0038]

[0039] Among them, time_step k Let K be the time step dimension of the second attention matrix.

[0040] Preferably, the feedforward layer is equipped with a ReLU activation function, and the calculation process of the feedforward layer includes high-dimensional projection matrix multiplication calculation, activation function calculation, and low-dimensional projection matrix multiplication calculation in sequence.

[0041] Preferably, the preset loss function in step S4 is the cross-entropy loss function.

[0042] This invention also provides a Transformer-based sequence classification prediction system, which applies the above-described Transformer-based sequence classification prediction method and includes:

[0043] Sequence acquisition and splitting unit: acquire the initial text sequence dataset and its corresponding categories, split all sequences in the initial text sequence dataset and remove duplicate strings, map each string to a different integer id and build a vocabulary, replace all strings in the split sequences with integer ids according to the vocabulary and acquire the training dataset;

[0044] Model building unit: used to build a Transformer neural network model for sequence classification prediction;

[0045] Initialization unit: Used to initialize the Transformer neural network model, add a position encoding matrix to the Transformer neural network model, and obtain the initialized Transformer neural network model;

[0046] Training and optimization unit: This unit is used to input the dataset to be trained into the initialized Transformer neural network model for classification and prediction training, and to calculate the loss value of classification and prediction using a preset loss function. When the loss value of classification and prediction is minimized, the training is completed and the trained Transformer neural network model is obtained.

[0047] Sequence classification prediction unit: Used to acquire the text sequence dataset to be classified, split the text sequence dataset to be classified, and input it into the trained Transformer neural network model for classification prediction to obtain the sequence classification prediction result.

[0048] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0049] This invention provides a Transformer-based sequence classification prediction method and system. The method involves: acquiring an initial text sequence dataset and its corresponding categories; splitting all sequences in the initial text sequence dataset and removing duplicate strings; mapping each string to a unique integer ID to construct a vocabulary; replacing all strings in the split sequences with integer IDs according to the vocabulary to obtain the training dataset; establishing a Transformer neural network model for sequence classification prediction; initializing the Transformer neural network model by adding a positional encoding matrix; inputting the training dataset into the initialized Transformer neural network model for classification prediction training; calculating the classification prediction loss value using a preset loss function; completing training when the classification prediction loss value is minimized; and finally, acquiring the text sequence dataset to be classified, splitting the text sequence dataset, and inputting it into the trained Transformer neural network model for classification prediction to obtain the sequence classification prediction result.

[0050] This invention simplifies complex matrix operations into low-complexity dot product operations by using Cumsum calculations while preserving recursive information. Compared to other positional encodings, it has a faster computation speed, enabling it to capture the linear structure information of sequences while reducing the computation time for feature capture in sequence classification tasks. In addition, leveraging the speed advantage of relative positional encoding, this invention incorporates trainable and parallelizable linear structure information, which reduces the computational time complexity of the model, improves efficiency, and enhances the model's ability to capture the linear structure of sequences, thereby improving the model's convergence ability and prediction accuracy. Attached Figure Description

[0051] Figure 1 The flowchart is a sequence classification prediction method based on Transformer provided in Example 1.

[0052] Figure 2 This is a schematic diagram of the -1 dimension Cumsum calculation provided in Example 2.

[0053] Figure 3 This is a schematic diagram of the time step dimension Cumsum calculation provided in Example 2.

[0054] Figure 4 This is a schematic diagram of the global max pooling layer calculation provided in Example 2.

[0055] Figure 5 This is a structural diagram of a Transformer-based sequence classification prediction system provided in Example 3. Detailed Implementation

[0056] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent.

[0057] To better illustrate this embodiment, some parts in the accompanying drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions;

[0058] It will be understood by those skilled in the art that certain well-known structures and their descriptions may be omitted in the accompanying drawings.

[0059] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0060] Example 1

[0061] like Figure 1 As shown, this embodiment provides a sequence classification prediction method based on Transformer, including the following steps:

[0062] S1: Obtain the initial text sequence dataset and its corresponding categories. Split all sequences in the initial text sequence dataset and remove duplicate strings. Assign each string to a different integer ID and construct a vocabulary. Replace all strings in the split sequences with integer IDs according to the vocabulary and obtain the training dataset.

[0063] S2: Establish a Transformer neural network model for sequence classification prediction;

[0064] S3: Initialize the Transformer neural network model, add a position encoding matrix to the Transformer neural network model, and obtain the initialized Transformer neural network model;

[0065] S4: Input the dataset to be trained into the initialized Transformer neural network model for classification prediction training, and calculate the loss value of classification prediction using the preset loss function. When the loss value of classification prediction is minimized, the training is completed and the trained Transformer neural network model is obtained.

[0066] S5: Obtain the text sequence dataset to be classified, split the text sequence dataset to be classified, and input it into the trained Transformer neural network model for classification prediction to obtain the sequence classification prediction result.

[0067] In the specific implementation process, firstly, an initial text sequence dataset and its corresponding categories are obtained. All sequences in the initial text sequence dataset are split, and duplicate strings are removed. Each string is mapped one-to-one with a different integer ID to construct a vocabulary. Based on the vocabulary, all strings in the split sequences are replaced using the integer IDs to obtain the training dataset. Next, a Transformer neural network model for sequence classification prediction is established. The Transformer neural network model is initialized by adding a positional encoding matrix, resulting in an initialized model. The training dataset is input into the initialized Transformer neural network model for classification prediction training, and the loss value for classification prediction is calculated using a preset loss function. When the loss value for classification prediction is minimized, training is complete, and the trained Transformer neural network model is obtained. Finally, the text sequence dataset to be classified is obtained, split, and input into the trained Transformer neural network model for classification prediction to obtain the sequence classification prediction results.

[0068] This method simplifies complex matrix operations into low-complexity dot product operations by using Cumsum calculations while preserving recursive information. Compared to other positional encodings, it has a faster computation speed, enabling it to capture the linear structure information of sequences while reducing the computation time for feature capture in sequence classification tasks. In addition, leveraging the speed advantage of relative positional encoding, this method incorporates trainable and parallelizable linear structure information, which reduces the computational time complexity of the model, improves efficiency, and enhances the model's ability to capture the linear structure of sequences, thereby improving the model's convergence ability and prediction accuracy.

[0069] Example 2

[0070] This embodiment provides a sequence classification prediction method based on Transformer, including the following steps:

[0071] S1: Obtain the initial text sequence dataset and its corresponding categories. Split all sequences in the initial text sequence dataset and remove duplicate strings. Assign each string to a different integer ID and construct a vocabulary. Replace all strings in the split sequences with integer IDs according to the vocabulary and obtain the training dataset.

[0072] S2: Establish a Transformer neural network model for sequence classification prediction;

[0073] S3: Initialize the Transformer neural network model, add a position encoding matrix to the Transformer neural network model, and obtain the initialized Transformer neural network model;

[0074] S4: Input the dataset to be trained into the initialized Transformer neural network model for classification prediction training, and use the cross-entropy loss function to calculate the loss value of classification prediction. When the loss value of classification prediction is minimized, the training is completed and the trained Transformer neural network model is obtained.

[0075] S5: Obtain the text sequence dataset to be classified, split the text sequence dataset to be classified, input it into the trained Transformer neural network model for classification prediction, and obtain the sequence classification prediction result.

[0076] The specific method in step S1 is as follows:

[0077] Obtain the initial text sequence dataset and its corresponding categories, where the i-th sequence S... i and its corresponding i-th category T i Let it be denoted as a data pair (S) i ,T i );

[0078] The initial text sequence dataset includes Chinese sequence datasets, English sequence datasets, and other sequence datasets;

[0079] All sequences in the initial text sequence dataset are split into word-level strings and deduplicated. All strings are sorted in chronological order and an integer id is assigned to each string. Each string corresponds one-to-one with its integer id, and a vocabulary is constructed.

[0080] Each split sequence S i The strings in the dictionary are replaced with the corresponding integer IDs by querying the dictionary to obtain a sequence of integer IDs, and each sequence of integer IDs is filled with the integer ID 0 to the same preset length;

[0081] All filled integer id sequences and their corresponding categories T i Save them together as the training dataset;

[0082] The Transformer neural network model in step S2 includes a Cumsum layer, an attention layer, a first normalization layer, a feedforward layer, a second normalization layer, a global max pooling layer, a fully connected layer, and a Softmax layer connected in sequence.

[0083] The input of the Cumsum layer also forms a residual connection with the input of the first normalization layer, and the input of the feedforward layer also forms a residual connection with the input of the second normalization layer.

[0084] The Cumsum layer is used to perform Cumsum calculations;

[0085] In step S3, the Transformer neural network model is initialized, and a positional encoding matrix is ​​added to the Transformer neural network model using Cumsum calculation to obtain the initialized Transformer neural network model. The specific method is as follows:

[0086] Initialize the first, second, and third weight matrices W in the Cumsum layer. q W k and W v And calculate and obtain the first, second and third attention matrices Q, K and V;

[0087] The first, second, and third attention matrices Q, K, and V are processed separately, and the first and second position encoding matrices M with the same dimensions as the first and second attention matrices Q and K are generated using the Summer initialization method. q and M k ;

[0088] According to the first and second position coding matrices M q and M k And the first and second attention matrices Q and K are calculated by Hadamard product and Cumsum, respectively, to obtain the initialized first and second attention matrices Q' and K'.

[0089] The initialized first and second attention matrices Q' and K', and the third attention matrix V are used as the attention matrices of the initialized Transformer neural network model to complete the model initialization;

[0090] The method for initializing Summer is as follows:

[0091] Set an initializer for a truncated normal distribution with a standard deviation of 0.005 and a mean of 0, and generate positive and negative values ​​near 0. Then, convert all generated negative values ​​to positive values ​​using absolute values.

[0092] The first and second positional encoding matrices M are obtained by Cumsum calculation in the -1 dimension. q and M k ;

[0093] According to the first and second position coding matrices M q and M k The first and second attention matrices Q and K are obtained by calculating the Hadamard product and Cumsum, respectively, to obtain the initialized first and second attention matrices Q' and K'. The specific method is as follows:

[0094] The initialized first and second attention matrices Q' and K' are obtained according to the following formulas:

[0095]

[0096]

[0097] Where ⊙ represents the Hadamard product operation, and Cumsum calculation specifically calculates the prefix sum of a preset matrix along the time step dimension;

[0098] The attention mechanism in the attention layer is specifically represented by the following formula:

[0099]

[0100] Among them, time_step k Let K be the time step dimension of the second attention matrix;

[0101] The feedforward layer is equipped with a ReLU activation function, and the calculation process of the feedforward layer includes high-dimensional projection matrix multiplication calculation, activation function calculation, and low-dimensional projection matrix multiplication calculation in sequence.

[0102] In the specific implementation process, the initial text sequence dataset and its corresponding categories are first obtained. Then, all sequences in the initial text sequence dataset are split to obtain the initial string dataset, specifically:

[0103] Obtain the initial text sequence dataset and its corresponding categories, where the i-th sequence S... i and its corresponding i-th category T i Let it be denoted as a data pair (S) i ,T i ), S i = [w1, w2, ..., w i , ..., w n ], T i = [t1, t2, ..., t i , ..., t m ], where n is the number of sequences and m is the number of categories;

[0104] The initial text sequence dataset includes Chinese sequence datasets, English sequence datasets, and other sequence datasets;

[0105] All sequences in the initial text sequence dataset are split into word-level strings and deduplicated. All strings are sorted in chronological order and an integer id is assigned to each string. Each string corresponds one-to-one with its integer id, and a vocabulary is constructed.

[0106] Each split sequence S i The strings in the dictionary are replaced with the corresponding integer IDs by querying the dictionary to obtain a sequence of integer IDs, and each sequence of integer IDs is filled with the integer ID 0 to the same preset length;

[0107] All filled integer id sequences and their corresponding categories T i Save them together as the training dataset;

[0108] Establish a Transformer neural network model for sequence classification prediction;

[0109] Next, the Transformer neural network model is initialized by adding a positional encoding matrix to it, resulting in the initialized Transformer neural network model, specifically:

[0110] Initialize the first, second, and third weight matrices W in the Cumsum layer. q W k and W v And calculate and obtain the first, second and third attention matrices Q, K and V;

[0111] The first, second, and third attention matrices Q, K, and V are processed separately, and the first and second position encoding matrices M with the same dimensions as the first and second attention matrices Q and K are generated using the Summer initialization method. q and M k ;

[0112] The method for initializing Summer is as follows:

[0113] Set an initializer for a truncated normal distribution with a standard deviation of 0.005 and a mean of 0. The weight distribution is w ~ N(0,0.005)∩(0,0.01). Generate positive and negative values ​​near 0, and convert all generated negative values ​​to positive values ​​using absolute values.

[0114] The first and second positional encoding matrices M are obtained by Cumsum calculation in the -1 dimension. q and M k ,like Figure 2 As shown, the -1 dimension is the word embedding dimension, which is the horizontal accumulation;

[0115] According to the first and second position coding matrices M q and M k The first and second attention matrices Q and K are calculated using the Hadamard product and Cumsum, respectively, to obtain the initialized first and second attention matrices Q' and K', as follows:

[0116] The initialized first and second attention matrices Q' and K' are obtained according to the following formulas:

[0117]

[0118]

[0119] Where ⊙ represents the Hadamard product operation, and Cumsum is specifically used to calculate the prefix sum of a preset matrix along the time step dimension, such as... Figure 3 As shown;

[0120] The initialized first and second attention matrices Q' and K', and the third attention matrix V are used as the attention matrices of the initialized Transformer neural network model to complete the model initialization;

[0121] Input the dataset to be trained into the initialized Transformer neural network model for classification prediction training, and use the cross-entropy loss function to calculate the loss value of classification prediction. When the loss value of classification prediction is minimized, the training is completed and the trained Transformer neural network model is obtained.

[0122] The attention mechanism in the attention layer is specifically represented by the following formula:

[0123]

[0124] Among them, time_step k is the time step dimension of the second attention matrix K, and the time step dimension is accumulated vertically;

[0125] A relu activation function is set in the feed-forward layer. The calculation process of the feed-forward layer sequentially includes high-dimensional projection matrix multiplication calculation, activation function calculation, and low-dimensional projection matrix multiplication calculation;

[0126] Finally, obtain the text sequence data set to be classified. After splitting the text sequence data set to be classified, input it into the trained Transformer neural network model for classification prediction to obtain the sequence classification prediction result;

[0127] The following is an illustration of this method with a specific sequence:

[0128] In this embodiment, ['ERNIE Bot is a large language model completely developed by Baidu'] is used as the initial Chinese text sequence data set. First, all characters are de-duplicated and an id dictionary is generated. Then, the sequence is tokenized at the character level to obtain ['文', '心', '一', '言', '完', '全', '是', '百', '度', '自', '研', '的', '大', '语', '言', '模', '型']. After tokenizing all texts, use the id dictionary to query the integer id corresponding to each character. The result obtained by replacing the above sequence with integer ids is [172, 285, 98, 567, 345, 973, 1028, 474, 1002, 3215, 777, 3029, 2056, 2011, 567, 1110, 1129]. Then, all integer ids are filled with '0' to the same length. Here, the length is set to 256, and [172, 285, 98, 567, 345, 973, 1028, 474, 1002, 3215, 777, 3029, 2056, 2011, 567, 1110, 1129, 0, 0, 0, 0, 0,..., 0] is obtained. Since the original sequence length is 17 and 256 - 17 = 239, there are 239 0 ids here;

[0129] After that, the 512-dimensional word vectors corresponding to the string and integer ids are taken out. The word vectors are initial parameters that can be trained from scratch, and the dimension is converted from (1, 256) to (1, 256, 512). Let this sample be x;

[0130] This sample is combined with W with initial dimensions of (512, 256) q 、W k 、W vPerform matrix multiplication to obtain matrices Q1, K1, and V1. Then perform separate operations to obtain matrices Q, K, and V with dimensions (1, 8, 256, 64). Reshape the (256, 256) dimension to (8, 256, 64).

[0131] Next, the Cumsum layer is used to initialize the position encoding matrix M, which also has dimensions (8,256,64), generated by Summer. q M k Perform a Hadamard product operation with matrices Q and K, with dimensions (1, 8, 256, 64). Then perform Cumsum calculation and divide by the Cumsum calculation result of the corresponding position encoding matrix for constraint, as shown in the following formula:

[0132]

[0133]

[0134] Then, the attention level between sequences is calculated using a multiplicative multi-head attention mechanism in the attention layer. The attention mechanism is expressed by the following formula:

[0135]

[0136] Among them, time_step k Let time_step be the time step dimension of the second attention matrix K. In this embodiment, time_step k =256;

[0137] Then, the dimension is restored by reshaping, from (1,8,256,64) to (1,256,512). The layer normalization operation is completed by adding the residuals and the first normalization layer. The formula is: x = x + Attention.

[0138] The feedforward layer then projects the normalized output of the layer into a higher dimension, passes it through the ReLU activation function, and then projects it into a lower dimension, as shown in the following formula:

[0139] FFN(x) = relu(xW1+b1)W2+b2

[0140] Where W1 and b1 are the parameter matrices and bias terms of the first fully connected layer; W2 and b2 are the parameter matrices and bias terms of the second fully connected layer; the ReLU function is relu(x) = max(0,x), the dimension of W1 is (512,2048), the dimension of W2 is (2048,512), and the output dimension is (1,256,512);

[0141] The above output and input values ​​are summed through residuals and then normalized again through the second normalization layer to output the final result.

[0142] Repeat the above steps to train, and calculate the cross-entropy loss according to the following formula:

[0143]

[0144] Among them, y i =1 indicates that the sequence belongs to the i-th category. Predict the probability that a sequence belongs to the i-th class for the classifier;

[0145] When the loss function value is minimized, the trained Transformer neural network model is obtained;

[0146] The text sequence dataset to be classified is split and then input into the trained Transformer neural network model for classification prediction. The matrix output from the second normalization layer is input into the global max pooling layer, such as... Figure 4 As shown, a vector of dimension (1, 512) is obtained and then fed into a fully connected layer. The number of neurons in this fully connected layer is c, which is the number of neurons in the same class. Then, a softmax layer is used for classification prediction, denoted as... In this embodiment, c = 15, and the final sequence classification prediction result is obtained by the following formula:

[0147] The classification prediction result obtained by inputting the split string in this embodiment into the trained Transformer neural network model is: [0.00272879,0.00307608,0.00262474,0.002499,0.00269692,0.00233592,0.00232535,0.00233154,0.9615145,0.00264494,0.00211363,0.00197962,0.00172725,0.00647863,0.00292311];

[0148] Therefore, the category with the highest probability is the 9th category, which is approximately 0.96. In this example, it represents the "technology" category, so the final classification prediction result is "technology".

[0149] To demonstrate the effectiveness of this method, this embodiment selected trainable positional encoding of BERT, relative positional encoding of T5, Deberta, and Rope, and recursive positional encoding of R-transformer for comparative experiments on Chinese and English text sequence datasets Yelp (2 classes), Toutiao (15 classes), and Online (2 classes) to prove the effectiveness and speed advantage of this method. The comparative experimental results are shown in Table 1.

[0150]

[0151] Table 1 Comparison of experimental results

[0152] As shown in Table 1, this method uses Transformer to randomly initialize the text sequence encoding. It uses the optimized initialization method summer to initialize the position encoding matrix and adds recursive position information to the relative position encoding, which enables the model to have a strong ability to capture the linear structure of the sequence. This solves the bottlenecks of recursive position encoding, which cannot be parallelized and is slow, and the problem of relative position encoding, which discards the recursive structure and thus has a weak ability to capture the linear structure of the sequence. This improves the convergence speed and classification ability of the model. Since it pays attention to the information of the linear structure of the sequence, it has a relatively greater improvement as seen in the 15-class classification dataset of toutiao. Therefore, the model performs relatively better in Chinese and more complex multi-class classification tasks.

[0153] This method simplifies complex matrix operations into low-complexity dot product operations by using Cumsum calculations while preserving recursive information. Compared to other positional encodings, it has a faster computation speed, enabling it to capture the linear structure information of sequences while reducing the computation time for feature capture in sequence classification tasks. In addition, leveraging the speed advantage of relative positional encoding, this method incorporates trainable and parallelizable linear structure information, which reduces the computational time complexity of the model, improves efficiency, and enhances the model's ability to capture the linear structure of sequences, thereby improving the model's convergence ability and prediction accuracy.

[0154] Example 3

[0155] like Figure 5 As shown, this embodiment provides a Transformer-based sequence classification prediction system, applying the Transformer-based sequence classification prediction method from Embodiment 1 or 2, including:

[0156] Sequence acquisition and splitting unit 301: used to acquire the initial text sequence dataset and its corresponding categories, split all sequences in the initial text sequence dataset and remove duplicate strings, associate each string with a different integer id and construct a vocabulary, replace all strings in the split sequences with integer ids according to the vocabulary and acquire the training dataset;

[0157] Model building unit 302: Used to build a Transformer neural network model for sequence classification prediction;

[0158] Initialization unit 303: Used to initialize the Transformer neural network model, add a position encoding matrix to the Transformer neural network model, and obtain the initialized Transformer neural network model;

[0159] Training optimization unit 304: is used to input the dataset to be trained into the initialized Transformer neural network model for classification prediction training, and to calculate the loss value of classification prediction using a preset loss function. When the loss value of classification prediction is minimized, the training is completed and the trained Transformer neural network model is obtained.

[0160] Sequence classification prediction unit 305: Used to acquire the text sequence dataset to be classified, split the text sequence dataset to be classified, input it into the trained Transformer neural network model for classification prediction, and obtain the sequence classification prediction result.

[0161] In the specific implementation process, firstly, the sequence acquisition and splitting unit 301 acquires the initial text sequence dataset and its corresponding categories, splits all sequences in the initial text sequence dataset, removes duplicate strings, maps each string to a different integer ID, constructs a vocabulary, and replaces all strings in the split sequences with integer IDs according to the vocabulary to obtain the training dataset; the model building unit 302 builds a Transformer neural network model for sequence classification prediction; the initialization unit 303 initializes the Transformer neural network model, adds a positional encoding matrix to the Transformer neural network model, and obtains the initialized Transformer neural network model; the training and optimization unit 304 inputs the training dataset into the initialized Transformer neural network model for classification prediction training, and calculates the classification prediction loss value using a preset loss function. When the classification prediction loss value is minimized, the training is completed, and the trained Transformer neural network model is obtained; finally, the sequence classification prediction unit 305 acquires the text sequence dataset to be classified, splits the text sequence dataset to be classified, inputs it into the trained Transformer neural network model for classification prediction, and obtains the sequence classification prediction result;

[0162] This system simplifies complex matrix operations into low-complexity dot product operations by using Cumsum calculations while preserving recursive information. Compared to other positional encoding methods, it boasts faster computation speed, enabling it to capture the linear structure information of sequences while reducing feature capture computation time in sequence classification tasks. Furthermore, leveraging the speed advantage of relative positional encoding, this system incorporates trainable and parallelizable linear structure information, reducing the model's computational time complexity, improving efficiency, and enhancing the model's ability to capture the linear structure of sequences. This, in turn, improves the model's convergence ability and prediction accuracy.

[0163] The same or similar labels correspond to the same or similar parts;

[0164] The terms used to describe positional relationships in the accompanying drawings are for illustrative purposes only and should not be construed as limiting this patent.

[0165] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A Transformer-based sequence classification prediction method, characterized in that, The method comprises the following steps: S1: obtaining an initial text sequence dataset and its corresponding category, splitting all sequences in the initial text sequence dataset, and removing duplicates of the same string, corresponding each string to a different integer id and constructing a word table, replacing all strings in the split sequences with integer ids according to the word table to obtain a training dataset; S2: establishing a Transformer neural network model for sequence classification prediction; S3: initializing the Transformer neural network model, adding a position encoding matrix to the Transformer neural network model, and obtaining an initialized Transformer neural network model, specifically: Initialize the preset first, second and third weight matrices W q , W k and W v in the Cumsum layer, and calculate to obtain first, second and third attention matrices Q, K and V; The first, second and third attention matrices Q, K and V are operated separately, and a first and second position encoding matrix M with the same dimensions as the first and second attention matrices Q and K is generated using the Summer initialization method q and M k ; According to the first and second position encoding matrices M q and M k and the first and second attention matrices Q and K, the initialized first and second attention matrices Q' and K' are obtained by Hadamard product and Cumsum calculation, respectively; using the initialized first and second attention matrices Q' and K', and the third attention matrix V as the attention matrix of the initialized Transformer neural network model to complete the initialization of the model; the Summer initialization is specifically: setting an initializer of a truncated normal distribution with a standard deviation of 0.005 and a mean of 0 to generate positive and negative values around 0, and normalizing all generated negative values by absolute value; The first and second position encoding matrices M are calculated by Cumsum in -1 dimension q and M k ; S4: inputting the training dataset into the initialized Transformer neural network model for classification prediction training, and calculating the loss value of the classification prediction using a preset loss function, when the loss value of the classification prediction is the smallest, the training is completed, and a trained Transformer neural network model is obtained; S5: obtaining a text sequence dataset to be classified, splitting the text sequence dataset to be classified and inputting it into the trained Transformer neural network model for classification prediction to obtain a sequence classification prediction result.

2. The Transformer-based sequence classification prediction method of claim 1, wherein, The specific method in step S1 is: An initial text sequence dataset and its corresponding classes are obtained, the ith sequence S i and its corresponding ith class T i is denoted as a data pair (S i , T i ); The initial text sequence dataset includes Chinese sequence dataset, English sequence dataset and other sequence dataset; all sequences in the initial text sequence dataset are split into word-level strings and de-duplicated, all strings are sorted in order and set with integer ids, each string and its integer id are one-to-one corresponding, and a word table is constructed; Each split sequence S i is replaced by the corresponding integer id through the vocabulary query, and the integer id sequence is obtained. Each integer id sequence is filled with the 0 integer id to a preset same length. All filled integer id sequences and their corresponding categories T i are collectively saved as a training dataset.

3. The Transformer-based sequence classification prediction method of claim 1, wherein, The Transformer neural network model in step S2 comprises a Cumsum layer, an attention layer, a first normalization layer, a feedforward layer, a second normalization layer, a global max pooling layer, a fully connected layer and a Softmax layer connected in turn; the input end of the Cumsum layer also forms a residual connection with the input end of the first normalization layer, and the input end of the feedforward layer also forms a residual connection with the input end of the second normalization layer; The Cumsum layer is used for Cumsum calculation.

4. The Transformer-based sequence classification prediction method of claim 1, wherein, According to the first and second position encoding matrices M q and M k and the first and second attention matrices Q and K, the initialized first and second attention matrices Q' and K' are obtained by Hadamard product and Cumsum calculation, and the specific method is: The initialized first and second attention matrices Q' and K' are obtained according to the following formula: wherein, denotes a Hadamard product operation, and Cumsum calculates the prefix sum of the predetermined matrix in the time step dimension.

5. The Transformer-based sequence classification prediction method of claim 4, wherein, The attention mechanism in the attention layer is specifically represented by the following formula: wherein, is the time step dimension of the second attention matrix K.

6. The Transformer-based sequence classification prediction method of claim 5, wherein, The feedforward layer is provided with a relu activation function, and the calculation process of the feedforward layer comprises high-dimensional projection matrix multiplication calculation, activation function calculation and low-dimensional projection matrix multiplication calculation in turn.

7. The Transformer-based sequence classification prediction method of claim 1, wherein, The preset loss function in step S4 is specifically a cross-entropy loss function.

8. A Transformer-based sequence classification prediction system, applying the Transformer-based sequence classification prediction method in any one of claims 1-7, characterized in that, The method comprises the following steps: A sequence acquisition and splitting unit is configured to acquire an initial text sequence dataset and its corresponding category, split all sequences in the initial text sequence dataset, remove duplicates for the same string, correspond each string to a different integer id and build a vocabulary, replace all strings in the split sequences with integer ids according to the vocabulary, and acquire a training dataset; A model building unit is configured to build a Transformer neural network model for sequence classification prediction; An initialization unit is configured to initialize the Transformer neural network model, add a position encoding matrix to the Transformer neural network model, and obtain an initialized Transformer neural network model; A training optimization unit is configured to input the training dataset into the initialized Transformer neural network model for classification prediction training, calculate a loss value of the classification prediction by using a preset loss function, complete the training when the loss value of the classification prediction is the smallest, and obtain a trained Transformer neural network model; A sequence classification prediction unit is configured to acquire a text sequence dataset to be classified, split the text sequence dataset to be classified, input the split text sequence dataset to the trained Transformer neural network model for classification prediction, and obtain a sequence classification prediction result.

Images