A sequence data processing method and neural network architecture based on global context awareness

Abstract

This invention discloses a global context-aware sequence data processing method and neural network architecture, comprising the following steps: Step 1) acquiring the input sequence; Step 2) using an embedding module to transform the input sequence block into a dense word vector representation; Step 3) using a positional encoding module to inject the absolute or relative position information of each word vector in the input sequence, and generating a processed sequence vector; Step 4) based on the processed sequence vector, an encoder stack generates a high-level sequence representation carrying global context information; Step 5) the decoder stack, based on the output of the encoder stack and the generated partial output sequence, progressively generates the target sequence in an autoregressive manner. This invention, through a global context-aware (self-attention) mechanism, allows all elements in the sequence to perform matrix operations simultaneously when calculating relevance, achieving true data-parallel intra-layer computation.

Description

Technical Field

[0001] This invention relates to the field of sequence data processing, specifically a sequence data processing method and neural network architecture based on global context awareness. Background Technology

[0002] Existing research employs recurrent neural networks and their variants (such as Long Short-Term Memory networks, LSTM) to process sequential data. The basic principle of this type of technology is to simulate the temporal characteristics of sequential data, processing each element of the input sequence sequentially through its internal hidden states, and passing information from previous steps to the current step for computation.

[0003] These technologies suffer from at least the following drawbacks: When a terminal uses a recurrent neural network to process long sequences of data, its recursive structure requires the model to process each element in the sequence sequentially, making parallelization impossible. This sequential processing characteristic severely limits the model's training speed, especially when processing long sequences, requiring extremely long training times. Furthermore, although improved models such as LSTM alleviate the vanishing or exploding gradient problem to some extent by introducing gating mechanisms, as the sequence length increases, the model still struggles to effectively capture the dependencies between distant elements in the sequence, leading to a significant performance drop when processing long documents or complex sentence structures. The root cause of these problems lies in the fact that existing technological solutions are fundamentally limited by the computational paradigm of sequential sequence processing. Summary of the Invention

[0004] The purpose of this invention is to provide a global context-aware sequence data processing method, comprising the following steps:

[0005] Step 1) Obtain the input sequence, wherein the input sequence is one or more of natural language text, speech signal, time-series sensor data, or image sequence;

[0006] Step 2) The embedding module transforms the input sequence block into a continuous vector representation through a lookup table or linear transformation; where the word vector representation has a dimension of 128 to 1024.

[0007] Step 3) Use the positional encoding module to inject the absolute or relative positional information of each word vector in the input sequence, and generate the processed sequence vector;

[0008] Step 4) Based on the processed sequence vector, the encoder stack generates a high-level representation of the sequence carrying global context information;

[0009] Step 5) The decoder stack generates the target sequence step by step in an autoregressive manner based on the output of the encoder stack and the partially generated output sequence.

[0010] Furthermore, in step 3), the step of generating the processed sequence vector includes:

[0011] Step 3.1) Generate a unique positional encoding vector with the same dimension as the word vector for each position in the input sequence, i.e.:

[0012] (1)

[0013] (2)

[0014] In the formula, For location index, For dimensional indexing, The dimension is the word vector. , For position encoding;

[0015] Step 3.2) Add the position encoding vector and the word vector representation element by element to generate the processed sequence vector.

[0016] Furthermore, the encoder stack comprises N stacked encoding layers, all of which have the same structure.

[0017] Furthermore, each coding layer includes a multi-head global context awareness unit, a feedforward neural network unit, and residual connections and layer normalization components surrounding the multi-head global context awareness unit and the feedforward neural network unit.

[0018] Furthermore, in step 4), the step of the encoder stack generating a high-level representation of the sequence carrying global context information includes:

[0019] Step 4.1) Based on a multi-head mechanism, steps 4.1.1) to 4.1.4) are executed in parallel, where each head uses a different projection matrix;

[0020] The projection matrix of the h-th head includes the query projection matrix. Projection matrix Value projection matrix ; It is the word vector dimension; It represents the dimension of the query and key vectors in each header; ; Represents a matrix of real numbers with a rows and b columns;

[0021] Step 4.1.1) Using the learnable weight matrix W Q W K W V Perform a linear projection on the processed sequence vector to generate query, key, and value vectors;

[0022] Step 4.1.2) Scale the attention weights to obtain the scaled attention score;

[0023] Step 4.1.3) Normalize the scaled scores using the Softmax function to obtain attention weights whose sum of all weights is 1;

[0024] Step 4.1.4) Use the attention weights calculated in Step 4.1.3) to perform a weighted summation on all Value vectors to generate a new context-aware representation for the current element.

[0025] Step 4.2) Concatenate the outputs of all the heads and perform a linear transformation to integrate them into the final output.

[0026] Furthermore, the scaling ratio is calculated as follows: for each element in the sequence, calculate the correlation between that element and all elements in the sequence; divide the correlation by the square root of the dimension of the Key vector to obtain the scaling ratio;

[0027] The correlation is shown below:

[0028] (3)

[0029] In the formula, , For query vectors and key vectors.

[0030] Furthermore, the decoder stack comprises N stacked decoding layers, all of which have the same structure.

[0031] Furthermore, the decoding layer includes a mask multi-head global context awareness unit and an encoder-decoder information interaction unit;

[0032] Among them, the masked multi-head global context awareness unit introduces a masking mechanism when calculating self-attention, so that the weights of all positions after the current position in the subsequent Softmax calculation approach zero;

[0033] In the encoding-decoding information interaction unit, the Query vector comes from the output of the layer above the decoder, while the Key and Value vectors come from the final output of the encoder stack.

[0034] Specifically, when generating the first element of the target sequence, the start symbol is used as the query vector.

[0035] Furthermore, in step 5), the decoder stack generates the target sequence step by step in an autoregressive manner, combining the encoder output and the generated partial results. In each step, the decoder predicts the probability distribution of the next word based on the current state and encoder information, and selects the next word through a sampling or search strategy until an end symbol is generated or the maximum length is reached.

[0036] The decoder stack generates the target sequence stepwise in an autoregressive manner, including the following steps:

[0037] Step 5.1) Set the start character <s>As the initial content of the already generated partial result sequence;

[0038] Step 5.2) Generate the target vector repeatedly until an end symbol is generated.< / s> Or the sequence length reaches the preset maximum length L max The steps include:

[0039] Step 5.2.1) Transform the currently generated partial result sequence into the decoder input vector sequence Y through the embedding layer and positional encoding module. embed ;

[0040] Step 5.2.2) Decoder input vector sequence Y embed The process is performed to obtain the output vector H. dec ;

[0041] Step 5.2.3) Apply a linear transformation layer and the Softmax function to the output vector H dec The process involves predicting the probability distribution of the next word, i.e.:

[0042] (4)

[0043] in, and The learnable weight matrix and bias vector for the output layer. This is the encoder's representation of the input sequence. This represents a partially generated sequence of results; It is a probability distribution;

[0044] Step 5.2.4) From the probability distribution obtained in Step 5.2.3), select the next word using a sampling strategy or a search strategy. ;

[0045] Step 5.2.5) Sequence Update: Update the selected words Add it to the already generated partial result sequence.

[0046] Step 5.3) When the selected word It is a terminator, or the length of the generated sequence has reached L. max When the time is reached, the generation process terminates, and the generated sequence is output as the target sequence.

[0047] A neural network architecture applied to the sequence data processing method includes an input acquisition module, an embedding module, a position encoding module, an encoder stack, and a decoder stack.

[0048] The input acquisition module is used to acquire the input sequence;

[0049] The embedding module transforms the input sequence block into a continuous vector representation through a lookup table or linear transformation.

[0050] The positional encoding module injects the absolute or relative positional information of each word vector in the input sequence and generates the processed sequence vector.

[0051] The encoder stack comprises N stacked encoding layers;

[0052] The encoder stack takes the processed sequence vector as input and generates a high-level representation of the sequence carrying global context information.

[0053] The decoder stack comprises N stacked decoding layers;

[0054] The decoder stack generates the target sequence step by step in an autoregressive manner based on the output of the encoder stack and the partially generated output sequence.

[0055] The technical effects of this invention are undeniable, and its beneficial effects are as follows:

[0056] 1) This invention enables true data-parallel in-layer computation by using a global context-aware (self-attention) mechanism, allowing all elements in the sequence to perform matrix operations simultaneously when calculating the correlation.

[0057] At the hardware level, this design enables large-scale matrix multiplication and addition operations during training to be fully parallelized, greatly improving the utilization of computing units.

[0058] 2) In actual benchmark tests (such as the WikiText-103 language modeling task), this invention achieves a training speed improvement of several to tens of times compared to traditional RNN / LSTM models under the same hardware configuration. For example, when processing long sequences (such as sequences of length 2048), this invention avoids recursive computation and can fully utilize the massively parallel cores of the GPU, achieving a training throughput more than 5 times that of the RNN baseline. Furthermore, in the inference phase, the architecture of this invention significantly improves inference speed through optimized self-attention; some optimized variants, such as MoD (Mixture-of-Depths), can achieve inference acceleration of more than 50%.

[0059] 3) This invention allows any two elements in a sequence (regardless of distance) to establish a connection directly through attention weights within a single layer, with a theoretical path length of O(1), thus completely solving the bottleneck of long-term dependency capture.

[0060] This invention demonstrates significant advantages in tasks requiring strong long-range dependency modeling. For example, in long document summarization (such as the PG-19 dataset) and machine translation (such as the WMT dataset), this invention further extends the effective context length to thousands of words by introducing segment-level recursion, improving dependency capture range by 450% compared to traditional Transformers and 80% compared to RNNs.

[0061] 4) This invention introduces residual connections after each sub-layer (self-attention layer, feedforward network), so that the gradient can be directly backpropagated through the identity mapping path during backpropagation, effectively alleviating the gradient vanishing problem.

[0062] This invention incorporates layer normalization to standardize the input of each layer, stabilizing the distribution range of activation values. This makes the training process less sensitive to initialization and learning rate hyperparameters, allowing for the successful training of extremely deep networks (e.g., over 100 layers). Compared to the deep, time-step-unfolding nature of RNNs, gradients in this invention propagate more directly through attention mechanisms and residual connections, resulting in shorter and more direct paths. The model of this invention achieves convergence with fewer training iterations, and the loss curve during training is smoother, exhibiting significantly lower volatility than RNN models.

[0063] 5) This invention can linearly or even superlinearly increase the model capacity (number of parameters) by simply increasing the number of encoding and decoding layers (N) to cope with more complex tasks and larger-scale data. This scalability lays the architectural foundation for the subsequent development of large-scale language models with hundreds of billions or even trillions of parameters (such as the GPT series and BERT series).

[0064] 6) The core architecture of this invention does not depend on a specific input modality. By adjusting the input embedding layer and output head, it can be easily applied to natural language processing (translation, summarization), computer vision (image classification, object detection), speech recognition, and even multimodal tasks. Attached Figure Description

[0065] Figure 1 This is the coding stage of the present invention.

[0066] Figure 2 illustrates the decoding stage of this invention;

[0067] Figure 3 is a schematic diagram of the overall architecture of the present invention. Detailed Implementation

[0068] The present invention will be further described below with reference to embodiments, but it should not be construed that the scope of the present invention is limited to the following embodiments. Various substitutions and modifications made based on ordinary technical knowledge and common practices in the art without departing from the above-described technical concept of the present invention should be included within the scope of protection of the present invention.

[0069] Example 1:

[0070] See Figures 1 to 3 A global context-aware sequence data processing method includes the following steps:

[0071] Step 1) Obtain the input sequence, wherein the input sequence is one or more of natural language text, speech signal, time-series sensor data, or image sequence;

[0072] Step 2) The embedding module transforms the input sequence block into a continuous vector representation through a lookup table or linear transformation; where the word vector representation has a dimension of 128 to 1024.

[0073] Step 3) Use the positional encoding module to inject the absolute or relative positional information of each word vector in the input sequence, and generate the processed sequence vector;

[0074] Step 4) Based on the processed sequence vector, the encoder stack generates a high-level representation of the sequence carrying global context information;

[0075] Step 5) The decoder stack generates the target sequence step by step in an autoregressive manner based on the output of the encoder stack and the partially generated output sequence.

[0076] Example 2:

[0077] A global context-aware sequence data processing method, with the same technical content as Embodiment 1, further comprising, in step 3), generating the processed sequence vector, including:

[0078] Step 3.1) Generate a unique positional encoding vector with the same dimension as the word vector for each position in the input sequence, i.e.:

[0079] (1)

[0080] (2)

[0081] In the formula, For location index, For dimensional indexing, The dimension is the word vector. , For position encoding;

[0082] Step 3.2) Add the position encoding vector and the word vector representation element by element to generate the processed sequence vector.

[0083] Example 3:

[0084] A sequence data processing method based on global context awareness, with the same technical content as any one of embodiments 1-2, further wherein the encoder stack includes N stacked coding layers with the same structure.

[0085] Example 4:

[0086] A sequence data processing method based on global context awareness, with the same technical content as any one of embodiments 1-3, further comprising each coding layer including a multi-head global context awareness unit, a feedforward neural network unit, and residual connections and layer normalization components surrounding the multi-head global context awareness unit and the feedforward neural network unit.

[0087] Example 5:

[0088] A global context-aware sequence data processing method, with the same technical content as any one of embodiments 1-4, further comprising, in step 4), the step of the encoder stack generating a high-level representation of the sequence carrying global context information, including:

[0089] Step 4.1) Based on a multi-head mechanism, steps 4.1.1) to 4.1.4) are executed in parallel, where each head uses a different projection matrix;

[0090] The projection matrix of the h-th head includes the query projection matrix. Projection matrix Value projection matrix ; It is the word vector dimension; It represents the dimension of the query and key vectors in each header; ; Represents a matrix of real numbers with a rows and b columns;

[0091] Step 4.1.1) Using the learnable weight matrix W Q W K W V Perform a linear projection on the processed sequence vector to generate query, key, and value vectors;

[0092] Step 4.1.2) Scale the attention weights to obtain the scaled attention score;

[0093] Step 4.1.3) Normalize the scaled scores using the Softmax function to obtain attention weights whose sum of all weights is 1;

[0094] Step 4.1.4) Use the attention weights calculated in Step 4.1.3) to perform a weighted summation on all Value vectors to generate a new context-aware representation for the current element.

[0095] Step 4.2) Concatenate the outputs of all the heads and perform a linear transformation to integrate them into the final output.

[0096] Example 6:

[0097] A sequence data processing method based on global context awareness, with the same technical content as any one of embodiments 1-5, further wherein the scaling ratio is calculated as follows: for each element in the sequence, calculate the correlation degree between the element and all elements in the sequence; divide the correlation degree by the square root of the dimension of the Key vector to obtain the scaling ratio;

[0098] The correlation is shown below:

[0099] (3)

[0100] In the formula, , For query vectors and key vectors.

[0101] Example 7:

[0102] A sequence data processing method based on global context awareness, with the same technical content as any one of embodiments 1-6, further wherein the decoder stack includes N stacked decoding layers, which have the same structure.

[0103] Example 8:

[0104] A sequence data processing method based on global context awareness, with the same technical content as any one of embodiments 1-7, further wherein the decoding layer includes a masked multi-head global context awareness unit and an encoding-decoding information interaction unit;

[0105] Among them, the masked multi-head global context awareness unit introduces a masking mechanism when calculating self-attention, so that the weights of all positions after the current position in the subsequent Softmax calculation approach zero;

[0106] In the encoding-decoding information interaction unit, the Query vector comes from the output of the layer above the decoder, while the Key and Value vectors come from the final output of the encoder stack.

[0107] Specifically, when generating the first element of the target sequence, the start symbol is used as the query vector.

[0108] Example 9:

[0109] A sequence data processing method based on global context awareness, with the same technical content as any one of embodiments 1-8, further wherein, in step 5), the decoder stack generates the target sequence step by step in an autoregressive manner, combining the encoder output and the generated partial results; wherein, in each step, the decoder predicts the probability distribution of the next word based on the current state and encoder information, and selects the next word through sampling or search strategies until an end symbol is generated or the maximum length is reached.

[0110] The decoder stack generates the target sequence stepwise in an autoregressive manner, including the following steps:

[0111] Step 5.1) Set the start character <s>As the initial content of the already generated partial result sequence;

[0112] Step 5.2) Generate the target vector repeatedly until an end symbol is generated.< / s> Or the sequence length reaches the preset maximum length L max The steps include:

[0113] Step 5.2.1) Transform the currently generated partial result sequence into the decoder input vector sequence Y through the embedding layer and positional encoding module. embed ;

[0114] Step 5.2.2) Decoder input vector sequence Y embed The process is performed to obtain the output vector H. dec ;

[0115] Step 5.2.3) Apply a linear transformation layer and the Softmax function to the output vector H dec The process involves predicting the probability distribution of the next word, i.e.:

[0116] (4)

[0117] (4)

[0118] in, and The learnable weight matrix and bias vector for the output layer. This is the encoder's representation of the input sequence. This represents a partially generated sequence of results; It is a probability distribution;

[0119] Step 5.2.4) From the probability distribution obtained in Step 5.2.3), select the next word using a sampling strategy or a search strategy. ;

[0120] Step 5.2.5) Sequence Update: Update the selected words Add it to the already generated partial result sequence.

[0121] Step 5.3) When the selected word It is a terminator, or the length of the generated sequence has reached L. max When the time is reached, the generation process terminates, and the generated sequence is output as the target sequence.

[0122] Example 10:

[0123] A neural network architecture applied to the sequence data processing method described in any one of embodiments 1-9 includes an input acquisition module, an embedding module, a position encoding module, an encoder stack, and a decoder stack;

[0124] The input acquisition module is used to acquire the input sequence;

[0125] The embedding module transforms the input sequence block into a continuous vector representation through a lookup table or linear transformation.

[0126] The positional encoding module injects the absolute or relative positional information of each word vector in the input sequence and generates the processed sequence vector.

[0127] The encoder stack comprises N stacked encoding layers;

[0128] The encoder stack takes the processed sequence vector as input and generates a high-level representation of the sequence carrying global context information.

[0129] The decoder stack comprises N stacked decoding layers;

[0130] The decoder stack generates the target sequence step by step in an autoregressive manner based on the output of the encoder stack and the partially generated output sequence.

[0131] Example 11:

[0132] A global context-aware sequence data processing method is described below:

[0133] like Figure 3As shown, the architecture of this invention presents a symmetrical encoder-decoder structure, mainly comprising an input sequence embedding module, a sequence position information encoding module, an encoder stack, a decoder stack, and an output sequence generation module. Its core innovation lies in completely abandoning the traditional recursive computation structure and instead adopting a global context-aware model based on a self-attention mechanism, thereby achieving parallel processing of sequence data and fundamentally solving the problem of long-range dependency capture. The data flow is as follows: The input sequence (such as a word sequence) is first transformed into a dense word vector representation through the embedding module. Subsequently, the position encoding module injects the absolute or relative position information of each word vector within the sequence. The processed sequence vectors are fed into the encoder stack, which generates a high-level sequence representation rich in global context information through multi-layer processing. The decoder stack then generates the target sequence step by step in an autoregressive manner based on the encoder output and the generated partial output sequence, with the output module ultimately completing the output of the entire process. Regarding the core components...

[0134] First, to address the inherent permutation invariance (i.e., insensitivity to input order) of self-attention mechanisms and the insensitivity of non-recursive structures to sequence order, the sequence position information encoding module innovatively uses a pre-defined mathematical formula based on sine and cosine functions to generate a unique position encoding vector of the same dimension as the word vector for each position in the sequence. This position encoding vector is then element-wise added to the word sense vector generated by the input sequence embedding module, thus explicitly injecting crucial sequence information into the model. The advantage of this method is that the generated encoding is periodic, making it easier for the model to learn relative position information and possessing a certain extrapolation capability to handle sequences longer than those used during training.

[0135] Secondly, the encoder stack consists of N (e.g., 6) identical coding layers stacked together. Figure 2 As shown, each coding layer includes a multi-head global context-aware unit, a feedforward neural network unit, and residual connections (Add) and layer normalization components surrounding them. Among them, the multi-head global context-aware unit is the core of this invention, and its essence is a multi-head extension of the self-attention mechanism.

[0136] The workflow is as follows: First, linear projection is performed, mapping the input vector linearly through three independent learnable weight matrices (W^Q, W^K, W^V) to generate three sets of vectors: query (Q), key (K), and value (V). This step projects the input vector into a space more suitable for calculating similarity. Next, attention weights are calculated and scaled. For each element in the sequence (represented by its query vector), its correlation with all elements in the sequence (represented by its key vector) is calculated (dot product). Then, the dot product is scaled by dividing it by the square root of the key vector dimension to prevent the dot product from becoming too large and causing the gradient of the Softmax function to vanish. Then, normalization and weighted summation are performed. The scaled scores are normalized using the Softmax function to obtain attention weights whose sum of all weights is 1. Finally, this weight is used to perform a weighted summation of all value vectors to generate a new context-aware representation of the current element. This representation integrates the information of all elements in the sequence, with higher-weight elements contributing more.

[0137] Finally, there's the multi-head mechanism. The above process is executed multiple times in parallel, with each head using a different projection matrix. This allows the model to capture information in parallel across different representation subspaces. Ultimately, the outputs of all heads are concatenated and integrated into the final output through a linear transformation. This design greatly enhances the model's representational power. The feedforward neural network unit is a relatively simple fully connected network, typically consisting of two linear transformations and an intermediate ReLU activation function. It is applied independently and identically to the vector at each location, aiming to further nonlinearly transform and enhance the features extracted by the self-attention layer. The residual connection and layer normalization components introduce residual connections and layer normalization after each sub-layer (self-attention layer and feedforward network). Specifically, this can be expressed as LayerNorm(x + Sublayer(x)), where Sublayer(x) is the output of the sub-layer. Residual connections ensure that information and gradients can effectively propagate across layers, alleviating the gradient vanishing problem in deep networks. Layer normalization stabilizes the input distribution of each layer by normalizing the activation values ​​within the layer, thereby accelerating training convergence and improving the stability of model training. Furthermore, the decoder stack also consists of N identically structured decoding layers stacked together.

[0138] like Figure 3As shown, the decoding layer adds two key designs to the encoder layer to meet the requirements of autoregressive generation tasks: The masked multi-head global context-aware unit (MCB) introduces a masking mechanism when calculating self-attention to ensure that it does not "peek" into future sequence information when generating the current output element (i.e., to guarantee the causality of the generation). Specifically, after calculating the attention score, it replaces the scores of all positions after the current position with a very large negative number (such as -10). 9 In this way, the weights at these positions will approach zero in the subsequent Softmax calculation, thus achieving effective masking. The encoder-decoder information interaction unit is the bridge connecting the encoder and decoder. It is essentially another attention mechanism. In this unit, the Query (Q) vector comes from the output of the layer above the decoder (i.e., the information of the currently generated content), while the Key (K) and Value (V) vectors come from the final output of the encoder stack. This allows the decoder to dynamically and selectively focus on the most relevant parts of the encoder output sequence when generating each element, thereby achieving effective alignment between the source sequence and the target sequence. Each sub-layer in the decoder (masking self-attention layer, encoder-decoder attention layer, feedforward network) is also equipped with residual connections and layer normalization components to ensure the stability of training. In terms of workflow, the input sequence is fed into the encoder stack after embedding and position encoding. The encoder, through its multi-layer structure, transforms the sequence into a high-level representation rich in global context information. The decoder stack then generates the target sequence step by step in an autoregressive manner, combining the encoder output and the generated partial results (the start symbol is usually a special start marker). At each step, the decoder predicts the probability distribution of the next word based on the current state and encoder information, and selects the next word through sampling or search strategies (such as greedy search or cluster search) until the end symbol is generated or the maximum length is reached. Finally, the output generation module completes the sequence output.

Claims

1. A sequence data processing method based on global context awareness, characterized in that, Includes the following steps: Step 1) Obtain the input sequence, wherein the input sequence is one or more of natural language text, speech signal, time-series sensor data, or image sequence; Step 2) The embedding module transforms the input sequence block into a continuous vector representation through a lookup table or linear transformation; where the word vector representation has a dimension of 128 to 1024. Step 3) Use the positional encoding module to inject the absolute or relative positional information of each word vector in the input sequence, and generate the processed sequence vector; Step 4) Based on the processed sequence vector, the encoder stack generates a high-level representation of the sequence carrying global context information; Step 5) The decoder stack generates the target sequence step by step in an autoregressive manner based on the output of the encoder stack and the partially generated output sequence.

2. The sequence data processing method based on global context awareness according to claim 1, characterized in that, Step 3) involves generating the processed sequence vector, which includes: Step 3.1) Generate a unique positional encoding vector with the same dimension as the word vector for each position in the input sequence, i.e.: （1） （2） In the formula, For location index, For dimensional indexing, The dimension is the word vector. , For position encoding; Step 3.2) Add the position encoding vector and the word vector representation element by element to generate the processed sequence vector.

3. The sequence data processing method based on global context awareness according to claim 1, characterized in that, The encoder stack comprises N stacked encoding layers, all of which have the same structure.

4. The sequence data processing method based on global context awareness according to claim 3, characterized in that, Each coding layer includes a multi-head global context awareness unit, a feedforward neural network unit, and residual connections and layer normalization components surrounding the multi-head global context awareness unit and the feedforward neural network unit.

5. The sequence data processing method based on global context awareness according to claim 3, characterized in that, In step 4), the steps of the encoder stack generating a high-level representation of the sequence carrying global context information include: Step 4.1) Based on a multi-head mechanism, steps 4.1.1) to 4.1.4) are executed in parallel, where each head uses a different projection matrix; The projection matrix of the h-th head includes the query projection matrix. Projection matrix Value projection matrix ; It is the word vector dimension; It represents the dimension of the query and key vectors in each header; ; Represents a matrix of real numbers with a rows and b columns; Step 4.1.1) Using the learnable weight matrix W Q W K W V Perform a linear projection on the processed sequence vector to generate query, key, and value vectors; Step 4.1.2) Scale the attention weights to obtain the scaled attention score; Step 4.1.3) Normalize the scaled scores using the Softmax function to obtain attention weights whose sum of all weights is 1; Step 4.1.4) Use the attention weights calculated in Step 4.1.3) to perform a weighted summation on all Value vectors to generate a new context-aware representation for the current element; Step 4.2) Concatenate the outputs of all the heads and perform a linear transformation to integrate them into the final output.

6. The sequence data processing method based on global context awareness according to claim 5, characterized in that, The scaling ratio is calculated as follows: for each element in the sequence, calculate the correlation between that element and all elements in the sequence; divide the correlation by the square root of the dimension of the Key vector to obtain the scaling ratio. The correlation is shown below: （3） In the formula, , For query vectors and key vectors.

7. The sequence data processing method based on global context awareness according to claim 1, characterized in that, The decoder stack consists of N stacked decoding layers, all of which have the same structure.

8. The sequence data processing method based on global context awareness according to claim 7, characterized in that, The decoding layer includes a mask multi-head global context awareness unit and an encoding-decoding information interaction unit; Among them, the masked multi-head global context awareness unit introduces a masking mechanism when calculating self-attention, so that the weights of all positions after the current position in the subsequent Softmax calculation approach zero; In the encoding-decoding information interaction unit, the Query vector comes from the output of the layer above the decoder, while the Key and Value vectors come from the final output of the encoder stack. Specifically, when generating the first element of the target sequence, the start symbol is used as the query vector.

9. A sequence data processing method based on global context awareness according to claim 7, characterized in that, In step 5), the decoder stack generates the target sequence step by step in an autoregressive manner, combining the encoder output and the generated partial results. In each step, the decoder predicts the probability distribution of the next word based on the current state and encoder information, and selects the next word through sampling or search strategies until the end symbol is generated or the maximum length is reached. The decoder stack generates the target sequence stepwise in an autoregressive manner, including the following steps: Step 5.1) Set the start character <s> As the initial content of the already generated partial result sequence;< / s> <s> < / s> <s> Step 5.2) Generate the target vector repeatedly until an end symbol is generated. < / s> Or the sequence length reaches the preset maximum length L max The steps include: Step 5.2.1) Transform the currently generated partial result sequence into the decoder input vector sequence Y through the embedding layer and positional encoding module. embed ; Step 5.2.2) Decoder input vector sequence Y embed The process is performed to obtain the output vector H. dec ; Step 5.2.3) Apply a linear transformation layer and the Softmax function to the output vector H dec The process involves predicting the probability distribution of the next word, i.e.: （4） in, and The learnable weight matrix and bias vector for the output layer. This is the encoder's representation of the input sequence. This represents a partially generated sequence of results; It is a probability distribution; Step 5.2.4) From the probability distribution obtained in Step 5.2.3), select the next word using a sampling strategy or a search strategy. ; Step 5.2.5) Sequence Update: Update the selected words Append to the already generated partial result sequence; Step 5.3) When the selected word It is a terminator, or the length of the generated sequence has reached L. max When the time is reached, the generation process terminates, and the generated sequence is output as the target sequence.

10. A neural network architecture applied to the sequence data processing method according to any one of claims 1-9, characterized in that, It includes an input acquisition module, an embedding module, a position encoding module, an encoder stack, and a decoder stack; The input acquisition module is used to acquire the input sequence; The embedding module transforms the input sequence block into a continuous vector representation through a lookup table or linear transformation. The positional encoding module injects the absolute or relative positional information of each word vector in the input sequence and generates the processed sequence vector. The encoder stack comprises N stacked encoding layers; The encoder stack takes the processed sequence vector as input and generates a high-level representation of the sequence carrying global context information. The decoder stack comprises N stacked decoding layers; The decoder stack generates the target sequence step by step in an autoregressive manner based on the output of the encoder stack and the partially generated output sequence.

Images