Sequence aggregation method, device, equipment and medium based on cls label enhancement

Abstract

This invention provides a sequence aggregation method, apparatus, device, and medium based on CLS tag enhancement. The sequence aggregation method employs a dual-path attention mechanism consisting of a first attention mechanism and a second attention mechanism when encoding the sequence. Different attention mechanisms are used to calculate the attention of CLS tags and other elements. For CLS tags, a second attention mechanism with a content-aware bias is added to the first attention mechanism for attention calculation, thereby enhancing the attention of CLS tags. This allows CLS tags to effectively collect global information from the sequence, improving the aggregation capability of CLS tags and thus enhancing the quality of CLS tag sequence representation, especially the quality of shallow CLS tag representation in the Transformer coding model.

Description

Technical Field

[0001] The embodiments of this invention relate to the field of computer information processing technology, specifically to a sequence aggregation method, apparatus, device, and medium based on CLS marker enhancement. Background Technology

[0002] Sequence-to-representation aggregation is a fundamental problem in deep learning, widely applied in natural language processing, bioinformatics, recommender systems, and other fields. Taking the analysis of traditional Chinese medicine formulas as an example, tasks such as efficacy prediction and similarity calculation for traditional Chinese medicine formulas require aggregating variable-length drug sequences or component sequences into fixed-dimensional vector representations. Transformer coding models typically use special CLS (Classification Substances) to achieve this function. Through a self-attention mechanism, the CLS substances collect information from the entire sequence, and finally, the output at the CLS position is taken as the sequence representation.

[0003] Traditional Transformer encoding models treat CLS-tagged sequences the same as regular elements in the sequence, meaning they use a uniform attention calculation method for all sequence elements (including CLS tags). This approach has the following drawbacks:

[0004] Ambiguous role positioning: The CLS marker is designed as an "aggregation representative" of the sequence, but it uses the same calculation method as ordinary elements, failing to highlight its unique "information collector" role.

[0005] Insufficient information aggregation capability: Standard attention calculation may cause the final representation of CLS tags to be interfered with by irrelevant information or noise, failing to effectively focus on key semantic content.

[0006] Efficiency issues in processing variable-length sequences: Traditional fixed-length padding methods generate a large number of meaningless padding marks when processing sets of sequences with large length differences, resulting in wasted computational resources and decreased model performance.

[0007] Therefore, this traditional encoding method results in poor representation quality for CLS tokens, especially in shallow layers of the Transformer coding model. CLS token representations often struggle to effectively aggregate global information, requiring deeper network layers to achieve better representation quality. There is an urgent need for a method that can endow CLS tokens with stronger information aggregation capabilities and efficiently handle variable-length sequences. Summary of the Invention

[0008] In view of this, the present invention provides a CLS tag-enhanced sequence aggregation method, apparatus, device and computer-readable storage medium to improve the representation quality of CLS tags, especially the representation quality of CLS tags in the shallow layer of the Transformer coding model.

[0009] On one hand, some embodiments of the present invention provide a sequence aggregation method based on CLS marker enhancement, including:

[0010] Obtain the set of variable-length sequences corresponding to the target text;

[0011] Each sequence in the variable-length sequence set is preprocessed to obtain a sequence to be encoded that can be processed by the Transformer encoding model. The preprocessing includes padding the sequences in the variable-length sequence set and adding a CLS marker to the beginning of the sequence.

[0012] The sequence to be encoded is input into the Transformer encoding model for encoding processing to obtain an encoded sequence. The encoding processing includes: at each layer of the Transformer encoding model, using a first attention mechanism to calculate the attention between elements other than the CLS tag in each sequence to be encoded, and using a second attention mechanism to calculate the attention of the CLS tag to the other elements; the second attention mechanism is an attention mechanism that uses the attention between the CLS tag and the other elements calculated by the first attention mechanism as the initial attention, and adjusts the initial attention based on a content-aware bias to obtain the target attention of the CLS tag to the other elements; wherein, the content-aware bias represents the attention of the CLS tag to the semantic content features of the other elements;

[0013] Extract the target hidden state of the CLS tag corresponding to each sequence from the output of each layer of the Transformer coding model, use it as the aggregate representation of each sequence, and output a set containing the aggregate representations of all sequences.

[0014] In some embodiments, the step of filling the sequences in the variable-length sequence set includes:

[0015] The sequences in the variable-length sequence set are divided into multiple groups according to their sequence length;

[0016] Perform a fill operation on each of the groups separately;

[0017] The filling operation includes:

[0018] Calculate the maximum length of the intra-group sequence of the group, and calculate the context aggregation representation of the content of each intra-group sequence;

[0019] Based on the aforementioned context aggregation representation, a gating mechanism is used to generate context-aware filled embeddings corresponding to the sequences within each group;

[0020] The context-aware padding embedding corresponding to each of the group sequences is used to pad the sequence length of each of the group sequences to the corresponding maximum length.

[0021] In some embodiments, when the proportion of the padding portion of the sequence to be encoded is greater than or equal to a preset proportion, the attention matrix corresponding to the sequence to be encoded is a sparse attention matrix.

[0022] In some embodiments, the second attention mechanism is an attention mechanism that adjusts the initial attention based on the content-aware bias and the global position bias to obtain the target attention of the CLS tag to the other elements.

[0023] The global position bias represents the degree of attention the CLS tag pays to the positional features of the other elements.

[0024] In some embodiments, after adding the CLS marker to the beginning of the sequence, the method further includes:

[0025] Assign independent and learnable relative position encoding vectors to the CLS tags;

[0026] Assign a preset absolute position code to the other elements, and perform length-aware scaling on the corresponding absolute position codes according to the initial sequence length of each sequence.

[0027] In some embodiments, the encoding process further includes:

[0028] After each layer of the Transformer encoding model is encoded, the initial hidden state of the CLS tag in the current layer is extracted, and the mean pooling vector, max pooling vector, and attention pooling vector guided by the CLS tag are calculated for the initial hidden states of all other elements in the current layer except the CLS tag.

[0029] The initial hidden state, mean pooling vector, max pooling vector, and attention pooling vector of the CLS-labeled array are concatenated and passed as input to the multilayer perceptron.

[0030] The output of the multilayer perceptron is weighted and summed with the initial hidden state of the CLS to obtain the refined hidden state of the CLS label, and the refined hidden state is used as the target hidden state of the CLS label in the current layer; wherein, the weight of the weighted summation is a refinement strength coefficient that decays exponentially with the number of network layers of the Transformer coding model.

[0031] In some embodiments, the attention calculation of each layer of the Transformer coding model is a multi-head attention calculation, with multiple attention heads having different bias strategies. The different bias strategies include a bias strategy in which the CLS marker is biased towards local neighboring positions, a bias strategy in which the CLS marker is biased towards all positions, and a bias strategy in which the CLS marker selectively focuses on each of the other elements based on the importance of the content of each of the other elements.

[0032] The target hidden state of the CLS-tagged output of each layer of the Transformer encoding model is obtained by concatenating the outputs of multiple attention heads and performing residual connections with the input of each layer, then undergoing layer normalization, and finally passing through a feedforward neural network for residual connections and layer normalization.

[0033] Secondly, some embodiments of the present invention provide a sequence aggregation apparatus based on CLS marker enhancement, comprising:

[0034] The input module is used to obtain a set of variable-length sequences corresponding to the target text;

[0035] A preprocessing module is used to preprocess each sequence in the variable-length sequence set to obtain a sequence to be encoded that can be processed by the Transformer encoding model. The preprocessing includes filling the sequences in the variable-length sequence set and adding a CLS marker to the beginning of the sequence.

[0036] An encoding module is used to input the sequence to be encoded into the Transformer encoding model for encoding processing to obtain an encoded sequence. The encoding processing includes: at each layer of the Transformer encoding model, using a first attention mechanism to calculate the attention between elements other than the CLS tag in each sequence to be encoded, and using a second attention mechanism to calculate the attention of the CLS tag to the other elements; the second attention mechanism is an attention mechanism that uses the attention between the CLS tag and the other elements calculated by the first attention mechanism as the initial attention, and adjusts the initial attention based on a content-aware bias to obtain the target attention of the CLS tag to the other elements; wherein, the content-aware bias represents the attention of the CLS tag to the semantic content features of the other elements;

[0037] The output module is used to extract the target hidden state of the CLS tag corresponding to each sequence from the output of each layer of the Transformer coding model, as the aggregate representation of each sequence, and output a set containing the aggregate representation of all sequences.

[0038] Thirdly, some embodiments of the present invention provide a sequence aggregation device based on CLS marker enhancement, including a processor and a memory, wherein the memory is used to store programs, instructions or code, and the processor is used to execute the programs, instructions or code in the memory to perform the sequence aggregation method as described in any of the foregoing.

[0039] Fourthly, some embodiments of the present invention provide a computer-readable storage medium storing a computer program that is loaded by a processor to execute the sequence aggregation method as described in any of the preceding claims.

[0040] In several embodiments provided by this invention, the sequence aggregation method based on CLS tag enhancement provided by this invention employs a dual-path attention mechanism consisting of a first attention mechanism and a second attention mechanism when encoding the sequence. Different attention mechanisms are used to calculate the attention of CLS tags and other elements. For CLS tags, a second attention mechanism with content-aware bias added to the first attention mechanism is used to calculate the attention of CLS tags, thereby enhancing the attention of CLS tags. This enables CLS tags to effectively collect global information from the sequence, which is beneficial to improving the aggregation capability of CLS tags. As a result, the quality of CLS tag sequence representation can be improved, especially the quality of CLS tag representation in the shallow layer of the Transformer coding model. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0042] Figure 1 This is a schematic diagram of the sequence aggregation method based on CLS marker enhancement provided according to some embodiments of the present invention.

[0043] Figure 2 This is a flowchart illustrating the dynamic grouping and filling steps provided according to some embodiments of the present invention.

[0044] Figure 3This is a flowchart illustrating the steps for refining the CLS mark according to some embodiments of the present invention.

[0045] Figure 4 This is a module architecture diagram of the sequence aggregation method provided in an embodiment of the present invention.

[0046] Figure 5 The diagrams show a comparison of the effects of enhanced CLS attention and standard CLS attention on attention calculation of CLS markers and other elements in a drug sequence, as provided in the embodiments of the present invention. (a) is a diagram showing the effect of standard CLS attention on attention calculation of CLS markers and other elements in a drug sequence, and (b) is a diagram showing the effect of enhanced CLS attention on attention calculation of CLS markers and other elements in a drug sequence, as provided in the embodiments of the present invention.

[0047] Figure 6 This is a schematic diagram showing the experimental results comparing the downstream task performance of various baseline methods with that of the method of this invention.

[0048] Figure 7 This diagram illustrates the experimental comparison of computational efficiency for different filling strategies.

[0049] Figure 8 This diagram illustrates the performance comparison of downstream tasks based on the sequence set method provided in the embodiments of the present invention at different layers of the Transformer coding model CLS representation.

[0050] Figure 9 This is a schematic diagram showing the ablation experiment comparison results of each component of the sequence aggregation method provided in the embodiments of the present invention.

[0051] Figure 10 The following are CLS attention comparison heatmaps for Ephedra Decoction, where (a) is the standard CLS attention heatmap for Ephedra Decoction and (b) is the attention heatmap of Ephedra Decoction according to the present invention.

[0052] Figure 11 This diagram illustrates the comparison between the method of this invention and the standard CLS method in terms of component-level sequence aggregation performance.

[0053] Figure 12 This is a schematic diagram of the structure of a sequence aggregation device based on CLS marker enhancement provided in some embodiments of this application.

[0054] Figure 13 This is a schematic diagram of the structure of a sequence aggregation device based on CLS marker enhancement provided in some embodiments of this application. Detailed Implementation

[0055] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0056] Please see Figure 1 As shown, it is a flowchart of a sequence aggregation method based on CLS marker enhancement provided according to some embodiments of the present invention. In some embodiments, the sequence aggregation method provided by the present invention is applied to a sequence aggregation device, which can be a computer device. The sequence aggregation method provided by the embodiments of the present invention includes S02, S04, S06 and S08, and the description of each step is as follows.

[0057] SO2: Get the set of variable-length sequences corresponding to the target text.

[0058] Target text refers to textual information that needs to be used for predictive analysis or other tasks. A variable-length sequence set is a collection of sequences of varying lengths. Each sequence in the variable-length sequence set can be, but is not limited to, text representing traditional Chinese medicine formulas. Variable-length sequences can be drug sequences or component sequences representing the corresponding ingredients in a traditional Chinese medicine formula.

[0059] The target text can be cleaned and standardized first, such as removing irrelevant characters, standardizing the encoding format, and handling special symbols, to ensure the accuracy of subsequent processing. After cleaning and standardizing the target text, a word segmenter is used to divide the target text into multiple words according to certain rules. Then, each word is organized into a sequence, and sequences of different lengths constitute a variable-length sequence set.

[0060] S04: Preprocess each sequence in the variable-length sequence set to obtain the sequence to be encoded that can be processed by the Transformer encoding model. The preprocessing includes padding the sequences in the variable-length sequence set and adding CLS markers to the beginning of the sequences.

[0061] To improve the efficiency of subsequent processing, the lengths of all sequences can be statistically analyzed, and the sequences can be divided into different groups according to their length ranges to facilitate batch processing or padding operations.

[0062] At the very beginning of each text sequence, a special CLS token (ClassificationToken) is inserted. After being encoded through multiple layers in the Transformer coding model, the final vector representation of the CLS token is used as the aggregate summary or overall representation of the entire sequence. For tasks such as text classification and sentiment analysis, the output of this CLS token is used to make judgments.

[0063] S06: Input the sequence to be encoded into the Transformer encoding model for encoding processing to obtain the encoded sequence. The encoding processing includes: in each layer of the Transformer encoding model, using the first attention mechanism to calculate the attention between other elements in each sequence to be encoded except for the CLS label, and using the second attention mechanism to calculate the attention of the CLS label to other elements.

[0064] The second attention mechanism is an attention mechanism that uses the attention of the CLS tag to other elements calculated by the first attention mechanism as the initial attention, and adjusts the initial attention based on the content-aware bias to obtain the target attention of the CLS tag to other elements; wherein, the content-aware bias represents the attention of the CLS tag to the semantic content features of other elements.

[0065] The first attention mechanism is used to calculate the attention among elements in the sequence to be encoded, excluding the CLS marker. It can be calculated using the standard scaled dot product attention method. The formula for calculating the attention among other elements using the first attention mechanism is shown in formula (1): For (1);

[0066] The attention score represents the score of the sequence excluding the CLS label when calculating attention. The element is the first The degree of attention or relevance of an element. A higher score indicates greater attention or relevance. The more attention is needed when generating its own representation. Information. This represents the position of the "query" element whose attention distribution is currently being calculated. Representing all possible [things] The position of the "key" element being observed. For These are the positional constraints of other elements. In the sequence, position 0 is reserved for the CLS marker. Therefore, this formula... Specifically designed for calculating elements other than the CLS tag (i.e. Attention between elements. Indicates the first The query vector corresponding to the nth position is formed by the nth position. The input of each element represents a learnable linear transformation (query weight matrix). This vector is used to "question" or "probe" information about other elements in the sequence. Indicates the first The key vector corresponding to the i-th position is formed by the i-th key vector. The input of each element represents a linear transformation that can be learned (weight matrix). This vector, obtained from the query, is used to "respond" to the query and represents the information available for attention for that element. Represents the key vector The transpose of. Represents the key vector The dimension. It is a scaling factor; the result of the dot product will change with the dimension. The gradient increases with the increase of , which may cause the gradient to become very small when applying the Softmax function (the vanishing gradient problem). Divided by This is to keep the amplitude of the dot product within a suitable range and stabilize the training process. The square root of the key vector dimension is used as a factor to scale the dot product result, ensuring that the variance of the attention weights remains stable across different dimensions.

[0067] The second attention mechanism is an enhanced version of the first attention mechanism. The attention score of the CLS tag calculated based on the second attention mechanism is at least superimposed on the content-aware bias mentioned above, based on the attention score of the CLS tag calculated based on the first attention mechanism. When the first attention mechanism is used as the scaling dot product attention calculation method, the attention score of the CLS tag on other elements calculated using it is... ) / , This represents the query vector corresponding to the CLS marker, specifically the query vector corresponding to the element at position 0. The second attention mechanism adds at least a content-aware bias to the first attention mechanism. In some embodiments, the attention score of the CLS tag is calculated based on the second attention mechanism to be at least [value missing]. and sum.

[0068] The content-aware bias is dynamically calculated based on the content features of each element in the sequence. Then, an attention score for the CLS tag is calculated using a second attention mechanism, allowing the CLS tag to adjust its focus according to content importance, thereby better capturing the semantic information of the sequence. This design enables the CLS tag to intelligently focus on key content, improving the performance of classification or sequence modeling tasks based on it.

[0069] In some embodiments, the second attention mechanism is an attention mechanism that adjusts the initial attention based on content-aware bias and global position bias to obtain the target attention of the CLS tag to other elements. Here, the global position bias represents the attention of the CLS tag to the positional features of other elements. That is, in some embodiments, the second attention mechanism adds content-aware bias and global position bias to the first attention mechanism. If the global position bias adopts... If we assume that the calculation formula for the attention of the CLS tag is calculated based on the second attention mechanism, then the formula is shown in formula (2).

[0070] (2);

[0071] The attention score for the CLS label represents the CLS label's importance to the first... The degree of attention or relevance of an element. It is a learnable global position bias that enables CLS tags to learn prior preferences for different positions. It is achieved by using a sine function, which has a smooth positional dependence, and its calculation formula is shown in formula (3).

[0072] (3);

[0073] The sequence length of the sequence containing the CLS marker. This represents sinusoidal position encoding, which is the core position signal generator that maps discrete integer positions j to a smooth, continuous, and periodic value. The positional weights are learnable scalar weights that can be learned automatically from training data. This parameter. Its function is to scale the importance of the sinusoidal position signal. The larger the absolute value, the stronger the model's opinion. Location information has a greater impact on attention allocation for CLS tags. The positional bias parameter is a learnable scalar bias that provides a basic "starting point" or "baseline" for the bias term, allowing the model to raise or lower its overall preference for all positions, increasing the model's flexibility. Here, "model" refers to the model used to implement the sequence aggregation method provided in this embodiment of the invention. The sequence prediction method provided in this embodiment adds content-aware bias and global positional bias to the first attention mechanism using a second attention mechanism, thereby enhancing CLS labeling and enabling CLS to identify and prioritize positions where content is more important.

[0074] S08: Extract the target hidden state corresponding to the CLS tag for each sequence from the output of each layer of the Transformer coding model, use it as the aggregate representation of each sequence, and output a set containing the aggregate representations of all sequences.

[0075] In each layer of the Transformer encoding model, the input sequence is encoded. A special marker at position 0, CLS (Category Least Signature), is designed to capture global information about the entire sequence. In the output of each layer, the target hidden state marked with CLS contains that layer's aggregate understanding of the entire sequence. Therefore, by extracting the target hidden state marked with CLS from the output of each layer, a multi-level sequence representation can be obtained. These representations are typically further processed (e.g., concatenated, weighted, or used directly) to ultimately form a single aggregate representation for each input sequence. These aggregate representations form a set, where each element corresponds to the final representation of an input sequence, which can be used for subsequent classification, regression, or other downstream tasks. The multi-layered structure of the Transformer encoding model allows for the progressive construction and optimization of the global semantic representation of the sequence. Here, the target hidden state refers to the hidden state in the final output of each layer.

[0076] The sequence aggregation method based on CLS tag enhancement provided in this invention employs a dual-path attention mechanism consisting of a first attention mechanism and a second attention mechanism when encoding the sequence. Different attention mechanisms are used to calculate the attention of CLS tags and other elements. For CLS tags, a second attention mechanism with content-aware bias is added to the first attention mechanism to enhance the attention of CLS tags. This enables CLS tags to effectively collect global information from the sequence, which is beneficial to improving the aggregation capability of CLS tags. As a result, the quality of CLS tag sequence representation can be improved, especially the quality of CLS tag representation in the shallow layers of the Transformer coding model.

[0077] In some embodiments, sequence padding of the variable-length sequence set in SO4 includes:

[0078] The sequence in the variable-length sequence set is divided into multiple groups according to its length, and a padding operation is performed on each group. For example... Figure 2 As shown, the steps for filling each group include S042, S044 and S046.

[0079] S042: Calculate the maximum length of the sequence within a group and the contextual aggregation representation of the sequence content within each group.

[0080] S044: Based on context aggregation representation, a gating mechanism is used to generate context-aware filled embeddings corresponding to sequences within each group.

[0081] S046: Use context-aware padding embedding corresponding to the sequence within each group to pad the sequence length of each group to the corresponding maximum length.

[0082] This embodiment groups sequences of different lengths according to their length and then performs context-aware padding on the sequences within each group to ensure that the sequence lengths of the sequences within each group are padded to the same value. For each group, the sequence length of each sequence within the group is calculated, and then these sequence lengths are compared to determine the maximum sequence length for each group. A context-aggregated representation of the content of each sequence within the group is then calculated. Here, "sequences within a group" refers to multiple sequences belonging to the same group, while "context-aggregated representation" is a representation that comprehensively reflects the characteristics of the sequence content, considering both the relationships within and between sequences. Based on the context-aggregated representation, the system generates context-aware padding embeddings through a gating mechanism. A gating mechanism typically refers to a mechanism that dynamically adjusts weights based on input information, allowing the padding embeddings to adaptively adjust according to the specific content of the sequence. This context-aware padding embedding differs from traditional fixed padding methods; it better preserves the semantic information of the sequence. By using the generated context-aware padding embedding to pad the sequences within each group, the length of all sequences is uniformly adjusted to the maximum length of the sequences within that group. This avoids the processing difficulties caused by inconsistent sequence lengths. At the same time, the context-aware padding method can more effectively maintain the semantic integrity of the sequences.

[0083] The dynamic grouping sequence filling method provided in this embodiment of the invention can better preserve the semantic information of the sequence compared with zero filling or fixed filling with specific symbols. In particular, it can provide a more reasonable filling strategy when processing sequences with different content features.

[0084] In some embodiments, when the proportion of padding in the sequence to be encoded is greater than or equal to a preset proportion, the attention matrix corresponding to the sequence to be encoded is a sparse attention matrix. In this embodiment, by detecting the proportion of padding (such as sequences padded with special symbols or zero values) in the sequence to be encoded, the attention matrix corresponding to the sequence is set to a sparse attention matrix when this padding proportion is greater than or equal to a preset proportion (a threshold parameter). Sparse attention typically means calculating attention only between partial positions, for example, only focusing on non-padded positions in the sequence and ignoring the fully connected computation of the padding portion of the sequence, thereby saving computational resources.

[0085] In some embodiments, the sequence aggregation method provided by the present invention employs an adaptive padding strategy, dynamically selecting the optimal padding scheme based on the sequence length distribution of each group. First, the sequences in the variable-length sequence set are grouped by length to avoid computational waste caused by mixing long and short sequences. Algorithms or function calls based on sequence length clustering can be used, but are not limited to, to group the sequences in the variable-length sequence set according to grouping rules, wherein the grouping rules can be:

[0086] ;

[0087] The statement corresponding to the above grouping rule can be expressed as: based on the input sequence ( The length feature of ) is used to divide it into a specified number of ( ) group ( This makes the sequence lengths within the same group similar. Represents a list ( ) or array ( ), containing multiple sequences to be processed. Integer ( ), which represents the desired number of groups. Represents a list ( ), the grouping results stored. It is an algorithm or function for sequence length clustering.

[0088] In some embodiments, variable-length sequences are grouped using a K-Means variant based on length distribution according to the following formula (4).

[0089] (4);

[0090] This represents the objective function value, used to characterize the loss value that needs to be minimized, and represents the overall "filling cost" of the grouping scheme. This represents summing over all groups. This represents summing over all sequences within the group. This represents the actual length of sequence s. Indicates grouping Maximum sequence length, For sequence The weights are adjustable parameters. Set to uniform weights or non-uniform weights based on importance.

[0091] Grouping sequences of similar length into the same group ensures that the maximum length within a group is as close as possible to the actual length of each sequence within the group, thereby reducing unnecessary padding.

[0092] For each group, the maximum length within the group is used instead of the global maximum length, and the sequence within each group is adaptively filled according to formula (5).

[0093] (5);

[0094] Indicates the first The padding length of a group is the target length to which all sequences within that group will be uniformly padded. This represents the maximum value operator, indicating the operation from the group. Find the maximum value in the range. ‎ represents a sequence The actual length (valid content length, excluding padding). Represents a sequence Belongs to group .

[0095] In some embodiments, the present invention designs a smart fill value, as shown in formula (6), instead of fixed zero fill. The fill position uses a learnable fill embedding.

[0096] (6);

[0097] The output vector, i.e. the generated smart padding embedding vector, will be used to fill in the blank positions of the sequence. is a learnable parameter matrix. · represents matrix multiplication. express The activation function is used to map the result of the linear transformation to the (0,1) interval, and serves as a gating weight to control the degree to which information is extracted from the embedded table. This is a learnable weight matrix. The input vector, i.e., the contextual aggregation representation of the sequence, is typically derived from the encoding of the sequence's effective content (e.g., obtained through pooling or attention aggregation), where As an aggregated representation of the actual content of the sequence, the gating mechanism enables the padding embedding to be aware of the sequence context, reducing interference with the model.

[0098] In some embodiments, attention sparsity is introduced to reduce the computational overhead of filling positions. For attention matrices where the proportion of filling positions exceeds a threshold, block sparse computation is used, calculating only the attention between actual positions, and the filling positions use pre-computed default attention values. In the context of traditional Chinese medicine formulas, the number of chemical components contained in different drugs varies greatly, ranging from a few to hundreds. If filling is done according to the maximum length, a large amount of computational resources will be wasted on filling positions. Although techniques such as dynamic batch processing exist, they usually require complex data rearrangement and binning, resulting in high implementation costs. However, the dynamic grouping filling method provided in this embodiment of the invention can effectively reduce computational overhead and is suitable for sequence filling applications of drug sequences or component sequences in traditional Chinese medicine formula scenarios.

[0099] The sequence aggregation method provided in this invention effectively reduces the implicit influence of padding positions on the model and improves the robustness of the model through context-aware padding embedding and block sparse attention.

[0100] In some embodiments, after adding a CLS marker to the beginning of the sequence, the preprocessing in S04 further includes: assigning an independent and learnable relative position encoding vector to the CLS marker, assigning a preset absolute position encoding to other elements, and performing length-aware scaling on the corresponding absolute position encoding according to the initial sequence length of each sequence.

[0101] In this embodiment, a special positional encoding strategy is adopted for the CLS marker in sequence processing. Different positional encoding methods are used for the CLS marker and other elements of the sequence, and a length-aware scaling mechanism is introduced. This design solves the asymmetry problem of standard positional encoding, ensuring a symmetrical positional relationship between the CLS marker and other positions in the sequence, avoiding the bias caused by position 0. Simultaneously, length-aware scaling allows the positional encoding to adapt to sequence inputs of different lengths.

[0102] Traditional sequence aggregation methods based on CLS tags assign position 0 as the position code for the CLS tag, resulting in an asymmetric positional relationship between the CLS and the first element of the sequence. In some tasks, this asymmetry can affect the aggregation results. In some embodiments, this invention designs a dedicated positional encoding strategy for the CLS tag to solve the asymmetry problem of standard positional encoding.

[0103] In this embodiment, the CLS tag uses an independent, learnable position embedding instead of fixed position 0 encoding. The position encoding rule for the CLS tag in this embodiment is shown in formula (7).

[0104] (7);

[0105] The position encoding vector representing the CLS token, represents an independent and learnable embedding parameter. The "=" indicates an assignment relationship, meaning that the position encoding of the CLS token is generated by this learnable embedding parameter, which learns the optimal "virtual position" during training, usually between all positions in the sequence.

[0106] In addition, in some embodiments, when calculating the attention between the CLS and other positions, relative position encoding is used instead of absolute position, which makes the relationship between the CLS and all positions symmetric, avoiding the bias caused by position 0. Among them, this relative position encoding is based on formula (8).

[0107] (8);

[0108] represents a multi-layer perceptron, used for non-linear transformation and fusion of input features, represents the absolute position encoding of the represents the input feature vector, which is composed of three parts concatenated together, where represents the dedicated position encoding of the CLS token, represents the difference vector between the position encoding of the CLS token and the position encoding between the positions.

[0109] In some embodiments, the present invention also introduces length-aware position scaling to the position encoding of the CLS token according to formula (9).

[0110] (I9);

[0111] Formula (9) is an adaptive scaling formula for position encoding, designed to enable position encoding to be intelligently adjusted according to the actual length of the sequence, thereby enhancing the model's ability to process sequences of different lengths. Among them, is the position encoding vector of the th position after scaling, represents the original position encoding vector of the th position, is the scaling intensity coefficient, which is an adjustable hyperparameter used to control the scaling amplitude, represents no scaling, maintaining the original encoding, represents obvious scaling effect and large influence of length difference. <00003t2>is usually set to a decimal between 0.1 and 0.3 and can be obtained through learning. is the current sequence length, is the average length of the training set. This enables position encoding to adapt to sequences of different lengths, This represents the relative length ratio, used to measure the degree to which the current sequence length deviates from the average length.

[0112] In some embodiments, S06 further includes refining the CLS; see [link to relevant documentation]. Figure 3 As shown, the process includes S062, S064 and S066.

[0113] S062: After each layer of the Transformer coding model is completed, extract the initial hidden state of the CLS tag in the current layer, and calculate the mean pooling vector, max pooling vector, and attention pooling vector based on the CLS tag for the initial hidden states of all other elements in the current layer except the CLS tag.

[0114] S064: Concatenate the initial hidden state labeled with CLS, the mean pooling vector, the max pooling vector, and the attention pooling vector, and pass them as input to the multilayer perceptron.

[0115] S066: The output of the multilayer perceptron is weighted and summed with the initial hidden state of the CLS to obtain the refined hidden state of the CLS, and the refined hidden state is used as the target hidden state of the CLS in the current layer.

[0116] The weights in the weighted summation are refined strength coefficients that decay exponentially with the number of network layers in the Transformer coding model.

[0117] This embodiment provides a method for refining CLS tags in a Transformer coding model to improve the representation quality of CLS tags through multi-dimensional information fusion. Specifically, a progressive refining mechanism can be used to refine the CLS tags, thereby enabling even shallow CLS tags in the Transformer coding model to achieve good aggregation quality.

[0118] After each encoding layer, the initial hidden state of the CLS tag is extracted, and the mean pooling, max pooling, and attention pooling vectors of the hidden states of other elements are calculated. These four vectors are concatenated and input into a multilayer perceptron for feature integration. Then, the refined hidden state of the CLS tag is obtained through weighted summation, where the weights decay exponentially with the number of layers. This design provided by the embodiments of the present invention enables the CLS tag to integrate global information, local features, and attention-guided semantics, while avoiding premature over-refinement through exponentially decaying refinement intensity control. Through multi-dimensional feature fusion and hierarchical refinement strategies, the semantic representation capability of the CLS tag can be improved, providing richer aggregated information for downstream tasks.

[0119] Specifically, after each layer of the Transformer coding model, a CLS token representation refinement module is introduced. The refinement module refines the CLS token representation according to the formula shown in formula (10).

[0120] (10);

[0121] Formula (10) is a progressive CLS representation refinement formula used to optimize the CLS markers in the Transformer coding model layer by layer to improve its ability to aggregate sequence information. This indicates the hidden state of the refined CLS tag. This represents the original hidden state of the CLS tag before refining. Additionally, + indicates a residual join operation. Indicates the refining strength coefficient. This represents a refined function.

[0122] The refinement function takes into account various interactions between CLS labels and sequence representations, and is expressed as follows:

[0123]

[0124] This is the core function in the Transformer surface model used to refine the representation of CLS tags. By fusing multiple interaction features between CLS tags and sequences, this function generates a richer and more comprehensive refined feature vector, aiming to improve the ability of CLS tags to aggregate sequence information. This represents the hidden state of the CLS tag, which serves as the input basis for the refining process and preserves the original semantic features of the CLS tag. It represents the hidden state of all other elements in the sequence, containing information about each sequence position. It is a learnable weight matrix used to map the concatenated high-dimensional features back to the original dimensional space. This indicates a concatenation operation that connects four feature vectors from different sources along the feature dimension. This is a mean pooling representation. The maximum pooling representation, also known as the attention pooling representation, is calculated using the formula shown in formula (11):

[0125] (11);

[0126] Formula (11) is the CLS tag-guided attention pooling calculation formula in the Transformer coding model, which is used to generate a personalized aggregated representation of CLS tag interaction with the sequence. This is an attention pooling representation, representing the personalized aggregated information that CLS tags filter from the sequence through an attention mechanism. For attention weights, denoted as CLS labels for the th element in the sequence. The level of attention given to each position. Transpose of the hidden state of the CLS tag. This is a learnable attention weight matrix. For the sequence number The hidden state of each location. This indicates that normalization is performed at all positions in the sequence to ensure... .in, .

[0127] In some embodiments, refining strength The calculation formula is shown in formula (12), which is adaptively adjusted according to the number of layers:

[0128] (12);

[0129] Equation (12) is an exponential decay function used to dynamically calculate the Transformer coding model at the 1st epoch. Strength coefficient of the layer for refining CLS markers . The initial refining strength marked as CLS. The decay rate coefficient is an adjustable hyperparameter (usually a positive number) that controls the rate of decay. The higher the value, the faster the decay; the more rapidly the refining strength decreases with increasing layer number. The smaller the value, the slower the decay, and the more gradual the reduction in refining intensity. The refining intensity is higher in shallow layers (to compensate for insufficient polymerization) and gradually decreases in deeper layers (to avoid excessive intervention).

[0130] In some embodiments, the attention computation of each layer of the Transformer encoding model is a multi-head attention computation, with multiple attention heads having different bias strategies. These different bias strategies include a bias strategy where the CLS tag favors local neighboring locations, a bias strategy where the CLS tag favors all locations, and a bias strategy where the CLS tag selectively focuses on other elements based on the importance of their content.

[0131] The target hidden state of the CLS tag output by each layer of the Transformer encoding model is obtained by concatenating the outputs of multiple attention heads and performing residual connections with the input of each layer, followed by layer normalization, and then passing through a feedforward neural network for residual connections and layer normalization. This invention provides an improved CLS tag processing mechanism in the Transformer encoding model, mainly comprising two core parts. First, a multi-head attention bias strategy is adopted: in the attention calculation of each layer, three different bias strategies are set to optimize the attention pattern of the CLS tag. The three different bias strategies are: a local attention bias that makes the CLS tag pay more attention to neighboring positions, a global uniform bias that makes the CLS tag pay even attention to all positions, and a content-selective bias that allows the CLS tag to selectively pay attention to other elements based on their importance. After multi-head attention computation, the final hidden state of the CLS tag, i.e., the target hidden state, is obtained by concatenating the outputs of multiple attention heads, performing residual connections with the input of that layer, undergoing layer normalization, and then performing residual connections and layer normalization through a feedforward neural network. This design enables the CLS tag to aggregate sequence information from multiple perspectives, improving its representation quality. The attention mechanism of the CLS tag is optimized through three different attention head strategies (local, global, and selective), introducing multi-scale CLS aggregation, using different bias strategies on different attention heads, and combining residual connections and layer normalization to ensure stable information flow. Finally, the CLS representation is further processed through a feedforward network, improving the model's ability to capture the overall semantics of the sequence. The CLS tag-enhanced sequence aggregation method provided in this invention, through its learned attention pattern, is less likely to ignore other important information, and in the context of traditional Chinese medicine prescriptions, it is less likely to cause the loss of information about certain key drugs or ingredients.

[0132] The sequence aggregation method based on CLS marker enhancement provided in this embodiment can achieve at least one of the following technical effects:

[0133] First, enhanced CLS aggregation capability. The sequence aggregation method provided in this embodiment of the invention, through a dual-path attention mechanism and content-aware bias, enables CLS tags to more effectively collect global information from sequences. At the same number of layers, the CLS representation quality in this embodiment of the invention is significantly better than methods that use a single attention mechanism to treat CLS tags and other elements, especially with a more pronounced improvement at shallower layers.

[0134] Second, adaptive attention patterns. This invention, through multi-scale attention heads and learnable biases, enables CLS tags to learn a more reasonable attention distribution. In the context of traditional Chinese medicine prescriptions, the model can automatically identify and focus on key drugs and ingredients, avoiding information loss.

[0135] Third, computational efficiency is optimized. The dynamic grouping and filling strategy of this invention significantly reduces unnecessary computation. Compared with fixed maximum length filling, this invention reduces the computational load by an average of 35% while maintaining the same performance, especially in scenarios where the component sequence lengths vary greatly.

[0136] Fourth, padding interference elimination. This invention effectively reduces the implicit influence of padding location on the model and improves the model's robustness through context-aware padding embedding and block sparse attention.

[0137] Fifth, symmetrical positional relationships. The independent position embedding and relative position encoding designed for the CLS marker in this embodiment of the invention solve the problem of asymmetrical positional relationships between CLS and sequence in standard methods, making aggregation more fair and reasonable.

[0138] Figure 4 The diagram illustrates the module architecture of the sequence aggregation method provided in this embodiment of the invention, showing the structure and data flow of the three core components: dual-path CLS attention, progressive refinement, and dynamic padding. The input is a variable-length batch of sequences, which, after dynamic grouping and padding, enters an enhanced Transformer encoding model. Each layer of the Transformer encoding model includes dual-path attention calculation and CLS refinement operations, ultimately outputting an aggregated representation of each sequence.

[0139] To visualize the technical effects of the sequence aggregation method provided in this application embodiment, which employs a dual-path attention mechanism to treat CLS tags and other elements, Figure 5 This diagram illustrates a comparison between the dual-path attention mechanism (enhanced CLS attention) provided in this embodiment of the invention and the conventional single attention mechanism (standard CLS attention) for attention calculation of CLS markers and other elements in a drug sequence. (a) shows the effect of standard CLS attention on attention calculation of CLS markers and other elements in a drug sequence; (b) shows the effect of enhanced CLS attention provided in this embodiment on attention calculation of CLS markers and other elements in a drug sequence. Figure 5 Drug 1 through Drug 4 in the sequence represent the other elements in the sequence, respectively. (Through...) Figure 5 The comparison between Figures (a) and (b) shows that Figure (a) represents standard CLS attention, exhibiting an uneven focusing pattern; while Figure (b) represents the enhanced CLS attention of the present invention, with a more reasonable attention distribution, ensuring appropriate attention to key positions while maintaining global coverage. This demonstrates that the enhanced CLS attention provided by the embodiments of the present invention can enhance CLS aggregation capabilities, and the quality of CLS representation is significantly superior to the sequence aggregation method corresponding to standard CLS attention.

[0140] To demonstrate the technical effectiveness of the sequence aggregation method provided in this application, comparative experiments are conducted below. In these experiments, the target text is a traditional Chinese medicine formula, and the sequences in the variable-length sequence set are various traditional Chinese medicine formulas. The comparative experiments first require experimental configuration, including preparing the dataset, determining the baseline method, and establishing experimental evaluation metrics. In each comparative experiment, the dataset is a traditional Chinese medicine formula efficacy prediction dataset, containing 4,521 formulas, each formula containing 3-20 herbs, and each herb containing 5-200 chemical components. The determined baseline method is based on CLS standard attention, and the method used for comparison with the method provided in this invention is as follows:

[0141] The first baseline method, CLS_standard, encodes CLS-tagged sequences using the standard Transformer model.

[0142] The second baseline method, MeanPool, uses average pooling for sequence aggregation; the third baseline method, MaxPool, uses max pooling for sequence aggregation; the fourth baseline method, Attention_pool, uses attention-weighted pooling for sequence aggregation; and the fifth baseline method, CLS+Auxiliary, uses CLS labeling plus auxiliary loss for sequence aggregation.

[0143] The established experimental evaluation metrics are: aggregate quality characterizing the performance of downstream tasks (such as power prediction), computational efficiency characterizing training and inference time, and memory usage characterizing GPU memory consumption.

[0144] The experimental results comparing the downstream task performance of each baseline method with that of the present invention are as follows: Figure 6 As shown in the first table diagram, it is clear that the present invention achieves optimal results in both accuracy and F1 score, while keeping the training time low.

[0145] Figure 7 The experimental results comparing the computational efficiency of different filling strategies are illustrated. Compared with filling with a fixed maximum length and filling with dynamic batch processing (Bucket) strategies, the dynamic group filling method provided by this invention achieves an effective computation ratio of 82%, which is significantly better than other methods.

[0146] Figure 8 The illustration shows the performance comparison results of downstream tasks based on the sequence aggregation method provided in the embodiments of the present invention in different layers of the Transformer coding model CLS representation. Obviously, the sequence aggregation method provided in the embodiments of the present invention has an improvement in all layers, and the improvement in shallow layers is more significant, indicating that the progressive refinement mechanism is effective.

[0147] Furthermore, to illustrate the technical effects of each component in the sequence aggregation method provided in this embodiment of the invention, an ablation experiment was conducted. The ablation experiment was configured to demonstrate the technical effects of the sequence aggregation method after removing different components. The comparison results of the ablation experiments are as follows: Figure 9 As shown, all components make positive contributions, with the dual-path CLS attention contributing the most.

[0148] The following example uses the prediction of the efficacy of the formula "Ma Huang Tang" to compare the CLS attention distribution provided by this invention with that of the standard CLS attention distribution. The standard CLS attention weights are: Ephedra: 0.45, Cinnamon Twig: 0.32, Apricot Kernel: 0.15, Licorice: 0.08. The CLS attention weights of this invention are: Ephedra: 0.38, Cinnamon Twig: 0.28, Apricot Kernel: 0.21, Licorice: 0.13.

[0149] By comparison, it can be seen that the CLS of this invention has a more balanced attention distribution, with all four drugs receiving reasonable attention, avoiding excessive focus on a single drug. This is consistent with the formulation characteristics of Ephedra Decoction, which combines four drugs and includes the principal, assistant, adjuvant, and guide herbs.

[0150] Figure 10 The following are CLS attention comparison heatmaps for Ma Huang Tang: (a) is the standard CLS attention heatmap for Ma Huang Tang; (b) is the attention heatmap of the present invention for Ma Huang Tang. The color depth represents the attention weight, and it can be observed that the attention distribution of the present invention is smoother and more uniform.

[0151] also, Figure 11 The diagram also illustrates the comparison between the method of this invention and the standard CLS method in terms of component-level sequence aggregation performance.

[0152] This invention also exhibits better polymerization effect at the component level, with a 11.3% increase in the recall rate of key components.

[0153] The above experiments demonstrate that the sequence aggregation method based on CLS marker enhancement proposed in this invention has significant advantages in terms of aggregation quality, computational efficiency, and applicability.

[0154] Please see Figure 12The diagram shows a schematic of a sequence aggregation device 120 based on CLS marker enhancement provided according to some embodiments of this application. The sequence aggregation device 120 includes an input module 121, a preprocessing module 122, an encoding module 123, and an output module 124. The input module 121 is used to obtain a set of variable-length sequences corresponding to the target text. The preprocessing module 122 is used to preprocess each sequence in the variable-length sequence set to obtain a sequence to be encoded that can be processed by the Transformer encoding model. The preprocessing includes padding the sequences in the variable-length sequence set and adding CLS markers to the beginning of the sequences. The encoding module 123 is used to input the sequence to be encoded into the Transformer encoding model for encoding processing to obtain an encoded sequence. The encoding process includes: at each layer of the Transformer encoding model, a first attention mechanism is used to calculate the attention between elements other than the CLS tag in each sequence to be encoded, and a second attention mechanism is used to calculate the attention of the CLS tag to other elements. The second attention mechanism is an attention mechanism that uses the attention of the CLS tag to other elements calculated by the first attention mechanism as the initial attention, and adjusts the initial attention based on a content-aware bias to obtain the target attention of the CLS tag to other elements. The content-aware bias represents the attention of the CLS tag to the semantic content features of other elements. The output module 124 is used to extract the target hidden state of the CLS tag corresponding to each sequence from the output of each layer of the Transformer encoding model, as the aggregated representation of each sequence, and output a set containing all sequence aggregated representations.

[0155] The sequence aggregation apparatus 120 provided in this embodiment of the invention and the sequence aggregation method provided in this embodiment of the invention can achieve essentially the same technical effects, and will not be described in detail here. In addition, the input module 121, the preprocessing module 122, the encoding module 123, and the output module 124 can also be specifically used to implement the corresponding steps in the sequence aggregation method provided in any embodiment of this application.

[0156] Please see Figure 13The diagram shows a schematic of a sequence aggregation device 130 based on CLS tag enhancement provided according to some embodiments of this application. The sequence aggregation device 130 includes a processor 131 and a memory 132. The memory 132 is used to store programs, instructions, or code, and the processor 131 is used to execute the programs, instructions, or code in the memory to complete the sequence aggregation method described in the foregoing embodiments. The memory 132 may be random access memory (RAM), flash memory, read-only memory (ROM), EPROM, non-volatile read-only memory (EEPROM), registers, hard disk, removable disk, etc.

[0157] The sequence aggregation device 130 provided according to the embodiments of the present invention can achieve essentially the same technical effect as the sequence aggregation method provided according to the embodiments of the present invention, and will not be described in detail here.

[0158] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape) or a semiconductor medium (e.g., solid-state disk (SSD)).

[0159] This invention also provides a readable storage medium for storing the sequence aggregation method provided in the above embodiments. Examples include random access memory (RAM), flash memory, read-only memory (ROM), EPROM, non-volatile read-only memory (EEPROM), registers, hard disks, removable disks, or any other form of storage medium in the art.

[0160] It is understood that the term "connection" in the embodiments of the present invention can be interpreted as "electrical connection," "communication connection," etc., if the connected circuits, units, etc. can transmit electrical signals or data to each other.

[0161] It is understood that the specific examples in this document are only intended to help those skilled in the art better understand the embodiments of the present invention, and are not intended to limit the scope of the present invention.

[0162] It is understood that in the various embodiments of the present invention, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0163] It is understood that the various embodiments described in this invention can be implemented individually or in combination, and the embodiments of this invention are not limited in this respect.

[0164] Unless otherwise stated, all technical and scientific terms used in the embodiments of this invention have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this invention is for the purpose of describing particular embodiments only and is not intended to limit the scope of the invention. The term "and / or" as used in this invention includes any and all combinations of one or more of the associated listed items. The singular forms "a," "the," and "the" as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0165] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0166] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A sequence aggregation method based on CLS marker enhancement, characterized in that, include: Obtain the set of variable-length sequences corresponding to the target text; Each sequence in the variable-length sequence set is preprocessed to obtain a sequence to be encoded that can be processed by the Transformer encoding model. The preprocessing includes padding the sequences in the variable-length sequence set and adding a CLS marker to the beginning of the sequence. The sequence to be encoded is input into the Transformer encoding model for encoding processing to obtain an encoded sequence. The encoding processing includes: at each layer of the Transformer encoding model, using a first attention mechanism to calculate the attention between elements other than the CLS tag in each sequence to be encoded, and using a second attention mechanism to calculate the attention of the CLS tag to the other elements; the second attention mechanism is an attention mechanism that uses the attention between the CLS tag and the other elements calculated by the first attention mechanism as the initial attention, and adjusts the initial attention based on a content-aware bias to obtain the target attention of the CLS tag to the other elements; wherein, the content-aware bias represents the attention of the CLS tag to the semantic content features of the other elements; Extract the target hidden state of the CLS tag corresponding to each sequence from the output of each layer of the Transformer coding model, use it as the aggregate representation of each sequence, and output a set containing the aggregate representations of all sequences.

2. The sequence aggregation method according to claim 1, characterized in that, The step of filling the sequences in the variable-length sequence set includes: The sequences in the variable-length sequence set are divided into multiple groups according to their sequence length; Perform a fill operation on each of the groups separately; The filling operation includes: Calculate the maximum length of the intra-group sequence of the group, and calculate the context aggregation representation of the content of each intra-group sequence; Based on the aforementioned context aggregation representation, a gating mechanism is used to generate context-aware filled embeddings corresponding to the sequences within each group; The context-aware padding embedding corresponding to each of the group sequences is used to pad the sequence length of each of the group sequences to the corresponding maximum length.

3. The sequence aggregation method according to claim 1, characterized in that, When the proportion of the padding portion of the sequence to be encoded is greater than or equal to a preset proportion, the attention matrix corresponding to the sequence to be encoded is a sparse attention matrix.

4. The sequence aggregation method according to claim 1, characterized in that, The second attention mechanism is an attention mechanism that adjusts the initial attention based on the content-aware bias and global position bias to obtain the target attention of the CLS tag to the other elements. The global position bias represents the degree of attention the CLS tag pays to the positional features of the other elements.

5. The sequence aggregation method according to claim 1, characterized in that, After adding the CLS marker to the beginning of the sequence, the method also includes: Assign independent and learnable relative position encoding vectors to the CLS tags; Assign a preset absolute position code to the other elements, and perform length-aware scaling on the corresponding absolute position codes according to the initial sequence length of each sequence.

6. The sequence aggregation method according to claim 1, characterized in that, The encoding process also includes: After each layer of the Transformer encoding model is encoded, the initial hidden state of the CLS tag in the current layer is extracted, and the mean pooling vector, max pooling vector, and attention pooling vector guided by the CLS tag are calculated for the initial hidden states of all other elements in the current layer except the CLS tag. The initial hidden state, mean pooling vector, max pooling vector, and attention pooling vector of the CLS-labeled array are concatenated and passed as input to the multilayer perceptron. The output of the multilayer perceptron is weighted and summed with the initial hidden state of the CLS to obtain the refined hidden state of the CLS marker, and this refined hidden state is used as the target hidden state of the CLS marker in the current layer; wherein... The weights in the weighted summation are refined strength coefficients that decay exponentially with the number of network layers in the Transformer coding model.

7. The sequence aggregation method according to any one of claims 1 to 6, characterized in that, The attention calculation of each layer of the Transformer encoding model is a multi-head attention calculation, with multiple attention heads having different bias strategies. The different bias strategies include the bias strategy of the CLS tag favoring local neighbor positions, the bias strategy of the CLS tag favoring all positions, and the bias strategy of the CLS tag selectively focusing on other elements based on the importance of the content of each other element. The target hidden state of the CLS-labeled output of each layer of the Transformer encoding model is obtained by concatenating the outputs of multiple attention heads and performing residual connections with the input of each layer, then undergoing layer normalization, and finally passing through a feedforward neural network for residual connections and layer normalization.

8. A sequence aggregation device based on CLS marker enhancement, characterized in that, include: The input module is used to obtain a set of variable-length sequences corresponding to the target text; A preprocessing module is used to preprocess each sequence in the variable-length sequence set to obtain a sequence to be encoded that can be processed by the Transformer encoding model. The preprocessing includes filling the sequences in the variable-length sequence set and adding a CLS marker to the beginning of the sequence. An encoding module is used to input the sequence to be encoded into the Transformer encoding model for encoding processing to obtain an encoded sequence. The encoding processing includes: at each layer of the Transformer encoding model, using a first attention mechanism to calculate the attention between elements other than the CLS tag in each sequence to be encoded, and using a second attention mechanism to calculate the attention of the CLS tag to the other elements; the second attention mechanism is an attention mechanism that uses the attention between the CLS tag and the other elements calculated by the first attention mechanism as the initial attention, and adjusts the initial attention based on a content-aware bias to obtain the target attention of the CLS tag to the other elements; wherein, the content-aware bias represents the attention of the CLS tag to the semantic content features of the other elements; The output module is used to extract the target hidden state of the CLS tag corresponding to each sequence from the output of each layer of the Transformer coding model, as the aggregate representation of each sequence, and output a set containing the aggregate representation of all sequences.

9. A sequence aggregation device based on CLS marker enhancement, characterized in that, It includes a processor and a memory, the memory being used to store programs, instructions, or code, and the processor being used to execute the programs, instructions, or code in the memory to perform the sequence aggregation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The system contains a computer program that is loaded by a processor to execute the sequence aggregation method as described in any one of claims 1 to 7.

Images