Image description generation method based on panoptic segmentation and multi-visual features

Abstract

The present invention provides an image description generation method that cooperates with panoramic segmentation and multiple visual features, comprising: extracting and encoding grid features and mask features based on panoramic segmentation of an input image; and cross-fusing text semantic features with grid features and mask features through dual-stream cross attention.

Description

Technical Field

[0001] The present invention relates to an image description generation method, and is particularly suitable for an image description generation method based on panoramic segmentation and multi-visual feature collaboration used in image description. Background Art

[0002] Existing methods for generating image descriptions usually follow an encoder-decoder architecture, where the features of the image extracted by a CNN are fed into an RNN (usually based on LSTM) to generate the corresponding sentence. Since RNN-based models are limited by their sequential structure, alternative methods have been explored to replace traditional RNNs. In recent years, the Transformer-based encoder-decoder architecture has become the mainstream model for image description. Such models usually use object detection models (such as Faster R-CNN) to extract region-level visual features of the image, which are then fed into an encoder-decoder network to generate the corresponding text description. However, the regional visual features extracted by the object detection model have the following problems: (1) The target region features usually focus on the foreground object in the image and cannot fully represent the fine-grained information of the entire image. This usually causes the description sentence to lack global information of the image and is difficult to summarize the rich visual information in the image; (2) There is feature confusion between multiple target visual features. Due to the overlap of different target detection boxes, confusing features are introduced into the target visual features, which usually causes data bias in training, false correlations in the test process, and reduces the generalization ability of the model. In order to solve these problems, recent work has introduced other visual clues such as semantic concepts, grid features, depth estimation, etc., and combined them with regional features extracted by Faster R-CNN to alleviate the above problems. However, these methods still do not fundamentally solve the limitations of traditional target visual features, thus restricting the further improvement of image description performance. Summary of the invention

[0003] In view of the shortcomings of the above-mentioned technologies, the present invention proposes an image description generation method that coordinates panoramic segmentation and multi-visual features, which can utilize panoramic segmentation features to fundamentally avoid the problems of visual confusion of target features and background missing, and fuse regional features with grid features, thereby significantly improving the performance of the Transformer-based image description model.

[0004] A method for generating image description by coordinating panoptic segmentation and multiple visual features, comprising:

[0005] Extract the grid feature G of the input image and the mask feature M based on panoptic segmentation and encode them;

[0006] The text semantic features are cross-fused with the grid features G and the mask features M through two-stream cross attention.

[0007] Furthermore, the Swin attention mechanism is used to obtain the grid features G of the input image, including:

[0008] Step S111, dividing the input image I into non-overlapping small blocks through the patch partition layer;

[0009] Step S112, mapping each small block into a low-dimensional space with a dimension of C through a linear embedding layer to generate a block embedding vector;

[0010] Step S113, perform four-stage downsampling on the block embedding vector to obtain a shape of H×W×D G The grid feature G

[0011]

[0012] Furthermore, the kMax-DeepLab panoptic segmentation network is used to generate mask features M, including:

[0013] Step S121, obtaining an image feature F of an input image I through a pixel encoder;

[0014] Step S122, using a pixel decoder to upsample the image feature F to extract pixel features;

[0015] Step S123, using a kMax decoder to convert the randomly initialized cluster center into a mask embedding vector;

[0016] Step S124, multiplying the mask embedding vector and the pixel feature to obtain the segmentation feature S;

[0017] Step S125: pooling the image feature F and the segmentation feature S to generate the final mask feature M.

[0018] M(N,D M )=∑ hw S(N,h,w)·F(h,w,D M ) (2)

[0019] Where S=(s1,s2,...,s N ),s l ∈{0,1} H×W represents the mask matrix, l represents the index number of the segmented area, each element represents whether the corresponding pixel belongs to a certain instance object, N represents the number of segmented areas in the image, and the number of each image is not fixed. M It represents the visual feature dimension corresponding to each segmented area, which is consistent with the feature dimension of the feature map F. h and w represent the row and column numbers corresponding to each segmented area. H and W represent the height and width of the input image respectively.

[0020] Furthermore, the text semantics X i The feature acquisition methods include:

[0021] Step S201, by adding the word embedding vector and the position code, the word feature X = (x1, x2, ..., x T ), where T is the sentence length, that is, the number of words contained in each sentence;

[0022] Step S202: Send the word features to the decoder. In each decoding layer, the word features are encoded by the self-attention module to obtain the semantic features X i :

[0023] X i =MultiHead(X i ,X i ,X i ) (2)

[0024] X i =X i +LayerNorm(X i ) (3)

[0025] Among them, i represents the decoding layer index, MultiHead(·) represents the Transformer multi-head attention operation, and LayerNorem(·) represents the normalization operation.

[0026] Furthermore, in step S202, a masked self-attention mechanism operation is performed on the word feature, and then a residual normalization operation is performed to obtain the semantic feature X i .

[0027] Furthermore, the process of cross-attention of two streams to cross-fuse text semantic features with grid features G and mask features M includes:

[0028] In the i-th decoding layer, the grid visual feature G i and the mask visual feature M i As key-value, the text semantic feature X is input into two multi-head attention modules respectively. i As the query vector, we get the mask visual feature M i and text semantic feature X i Fusion features And the grid visual feature G i and text semantic feature X i Fusion

[0029]

[0030]

[0031]

[0032]

[0033] Among them, α M and α G is the dual-stream weight, W M , W G represents the learnable weight parameter, b M , b G represents the learnable bias parameter, which is calculated by the Sigmoid activation function σ(·);

[0034] Get the dual-stream fusion feature R i

[0035]

[0036] Represents the dot product operation.

[0037] Furthermore, the dual-stream fusion feature R i After the residual normalization operation, it is input into the forward feedback layer to form the input of the next decoding layer.

[0038] Furthermore, a position encoding operation is performed on the word features before the masked self-attention mechanism operation to obtain the sequential position information pos of the word

[0039]

[0040] Among them, pos represents the position of the current word in the sequence, j is the index of the embedding dimension, and d is the dimension of the embedding.

[0041] Compared with the prior art, the present invention has the following advantages: (1) The dual-stream cross-attention adopts two multi-head attention modules. In each multi-head attention module, the text feature encoded by self-attention is used as Q, and the encoded grid feature or mask feature is used as K and V for cross-attention. The mask feature and the grid feature are combined, which solves the problem that the grid and mask visual features are encoded separately using Transformer encoders that do not share parameters due to their different visual representation characteristics, and provides rich visual information for the decoding end; (2) The target detection features extracted by Faster R-CNN are replaced by mask features extracted by panoramic segmentation, which avoids visual pseudo-correlation caused by detection frame overlap and data set bias, and at the same time provides background information in the image and fuses it with the grid feature at the decoding end, learns the complementary advantages between the grid feature and the mask feature, and generates more realistic and accurate description text.

[0042] The present invention will be further described below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a framework diagram used by the panoramic segmentation and multi-vision collaborative image description method in the present invention.

[0044] Figure 2 It is a structural schematic diagram of the grid feature extractor used in the method of the present invention.

[0045] Figure 3 It is a structural schematic diagram of the mask feature extractor used in the method of the present invention.

[0046] Figure 4 This is a comparison chart of the descriptions generated by the standard Transformer model and the implementation example. DETAILED DESCRIPTION

[0047] Combination Figure 1 , a method for generating image description by coordinating panoptic segmentation and multi-visual features, comprising the following steps:

[0048] Step S100, extracting and encoding the grid features G and the mask features M based on panoptic segmentation of the input image;

[0049] Step S200, cross-fusion of text semantic features with grid features G and mask features M through dual-stream cross attention.

[0050] Combination Figure 2 In step S100, the Swin attention mechanism (Swin-Transformer) is used to obtain the grid feature G of the input image, including:

[0051] Step S111, the input image I (size is 384×384×3) is divided into 96 non-overlapping small blocks through the patch partition layer, each small block size is 4×4 pixel area, and the feature dimension of each small block is 4×4×3;

[0052] Step S112, mapping each small block into a low-dimensional space with a dimension of C=96 through a linear embedding layer to generate a block embedding vector (96×96×C);

[0053] Step S113, perform four-stage downsampling on the block embedding vector to obtain a shape of H×W×D G The grid feature G

[0054]

[0055] Among them, Gridnet represents the Swin-Transformer grid feature extraction network, R represents the shape description of the matrix, H and W represent the height and width of the feature map (the height and width of the input image) D G Represents the feature dimension, that is, the shape of G is

[0056] H×W×D G .

[0057] In step S113, downsampling is performed using Swin-Transformer Block, and the sampling frequencies of the four stages are ×4, ×8, ×16, and ×32 respectively. The first Swin-Transformer Block is connected to the linear embedding layer (Linear Embedding) in step S112, and a layer downsampling layer (Patch Merging) is set before the last three Swin-Transformer Blocks; the layer downsampling layer (Patch Merging) reduces the number of small blocks and also reduces the feature dimension. The multi-head attention calculation part MSA of each Swin-Transformer Block is a regular window W-MSA and a moving window SW-MSA. The specific process of step S113 is:

[0058] Step S1131, in the first stage of downsampling, two Swin-Transformer Blocks are set to perform self-attention calculation and output a feature vector (96×96×2C);

[0059] Step S1132, in the second stage, the first downsampling layer is used to reduce the feature dimension output (48×48×4C), and the attention is calculated through two Swin-Transformer Blocks, the features are converted to a deeper layer, and the feature vector (48×48×4C) is output;

[0060] Step S1133, in the third stage, the feature dimension output is first reduced by the second downsampling layer (24×24×8C), and attention is calculated by six Swin-Transformer Blocks (two in a group) to output a feature vector (24×24×8C);

[0061] Step S1134, in the fourth stage, the feature dimension output is first reduced through the third downsampling layer (12×12×16C), and the attention is calculated through two Swin-Transformer Blocks to output the feature vector (12×12×16C).

[0062] Combination Figure 3In step S100, a kMax-DeepLab (k-means mask transformer) panoptic segmentation network is used to generate a feature map F and a mask S, including:

[0063] Step S121, obtaining an image feature F of an input image I through a pixel encoder;

[0064] Step S122, using a pixel decoder to upsample the image features F to extract pixel features;

[0065] Step S123, using a kMax decoder to convert randomly initialized cluster centers into mask embedding vectors;

[0066] Step S124, multiplying the mask embedding vector and the pixel feature to obtain segmentation features;

[0067] Step S125: pooling the image feature F and the segmentation feature S to generate the final mask feature M.

[0068] M(N,D M )=∑ hw S(N,h,w)·F(h,w,D M )

[0069] Where S=(s1,s2,...,s N ),s l ∈{0,1} H×W represents the mask matrix, l represents the index number of the segmented area, each element represents whether the corresponding pixel belongs to a certain instance object, N represents the number of segmented areas in the image, and the number of each image is not fixed. M It represents the visual feature dimension corresponding to each segmented area, which is consistent with the feature dimension of the feature map F. h and w represent the row and column numbers corresponding to each segmented area. H and W represent the height and width of the input image respectively.

[0070] In step S121, any CNN or ViT backbone can be used to extract visual features.

[0071] Combination Figure 2,In step S100, since the grid and mask visual features have different visual representation characteristics, the two features are encoded separately using a Transformer encoder that does not share parameters, and richer and more targeted visual features are obtained after self-attention encoding.

[0072] In step S200, the text semantic feature acquisition method is:

[0073] Step S201, by adding the word embedding vector and the position code, the word feature X = (x1, x2, ..., x T ), where T is the sentence length, i.e. the number of words;

[0074] Step S202, the word features are sent to the decoder. In each decoding layer, the word features are encoded by the self-attention module. In order to ensure that the non-autoregressive decoding method during training and testing is consistent, a mask operation is performed on it so that each word can only observe the currently generated words, that is, Figure 1 The Masked Self-Attention operation in , and then the residual normalization operation (i.e. the first layer Add & Layer Norm) is performed to obtain the corresponding semantic feature X i :

[0075] X i =MultiHead(X i ,X i ,X i ) (2)

[0076] X i =X i +LayerNorm(X i ) (3)

[0077] Here, i represents the decoding layer index and MultiHead(·) represents the Transformer multi-head attention operation.

[0078] In step S200, the Transformer model is used to construct a decoding network, in which the text semantic features are cross-fused with the mask features M and the grid features G through two-stream cross attention, that is, Figure 1 Dual Cross-Attention layer operation. In the i-th decoding layer, the grid visual feature G i and the mask visual feature M i As key-value, the text semantic feature X is input into two multi-head attention modules respectively. i As the query vector. The two-stream cross-attention module outputs fusion features respectively and in is the mask visual feature M i and text semantic feature X i The fusion of is the grid visual feature G i and text semantic feature X i This dual-stream cross-learning method allows the model to learn the correlation between different features, improving the feature expression ability and semantic understanding ability.

[0079]

[0080]

[0081]

[0082]

[0083]

[0084] Among them, α M and α G is the dual-stream weight, W M , W G represents the learnable weight parameter, b M , b G represents the learnable bias parameter, which is calculated by the Sigmoid activation function σ(·), Represents the dot product operation. This weighted fusion method allows the model to better utilize the mask visual features and grid visual features, thereby improving the model's visual representation ability and description performance. Compared with the traditional cross-attention mechanism, this feature fusion method can more finely learn the relationship between text semantics and mask features and grid features, and can adaptively adjust the importance of different features, effectively improving the model's ability to represent image semantic information.

[0085] Dual stream fusion feature R i After residual normalization operation (i.e. Figure 1 After the second Add&Layer Norm operation in the process, it is fed into the Feed Forward layer to form the input of the next decoding layer:

[0086] X i+1 =FeedForward(X i +Layernorm(R i )) (9)

[0087] The output of the last layer of decoding blocks passes through the Generator, which mainly consists of a linear layer and a Softmax function. The linear layer is responsible for converting the dimension from dmodel (512 in the experiment, that is, the dimension of visual features and text features in the codec) is mapped to v, where v is the number of all words in the text set (9487 in the experiment). Finally, the Softmax function is responsible for converting it into the probability of each word in the vocabulary and predicting the word output X i .

[0088] Unlike sequential algorithms such as recurrent neural networks and convolutional neural networks, the multi-head attention in the Transformer model pays attention to the importance of words, but cannot capture the relative or absolute position information between words. Figure 1 , the decoding network introduces the positional encoding operation. Positional encoding uses sine and cosine functions to obtain the sequential position information of words, and the result is then added to the word embedding vector and sent to the multi-head attention as input. Since the input text is arranged in a certain order, the relative position between words can be effectively learned through positional encoding. In the encoding part, because there is no fixed order relationship between different areas of the image, the image feature encoding sequence models the relationship between features through the Transformer full attention mechanism without introducing positional encoding. Positional encoding uses sine and cosine functions to obtain the sequential position information of words, and the result is then added to the word embedding vector and sent to the multi-head attention as input. The definition of positional encoding is as follows:

[0089]

[0090] Where pos represents the position of the current word in the sequence, j is the index of the embedding dimension, and d is the embedding dimension. PE(pos,2j) is the position encoding value of position pos and even index 2j, and PE(pos,2j+1) is the position encoding value of position pos and odd index 2j+1.

[0091] The following is a detailed introduction to the principle and process of multi-head attention:

[0092] Multi-head attention divides the query matrix Q, key matrix K, and value vector matrix V of the high-dimensional space into h different subspaces to calculate the similarity separately, ensuring the independence of each subspace, and finally merges them by transformation method. The text semantic features encoded by self-attention are taken as Q, and the encoded grid features or mask features are taken as K and V. Cross-attention is performed, and then the respective weights are learned through linear layers and activation functions to integrate the respective advantages of the two types of visual features. The calculation formula is as follows:

[0093] MultiHead(Q,K,V)=W 0 [head1,…,head h ] (11)

[0094] head i =Attention(W i Q Q,W i K K,W i V V) (12)

[0095] in is a learnable parameter that maps each matrix to the i-th d k (d k =d / h)-dimensional subspace. [head1,…,head n ] represents the concatenation of h different subspaces, followed by a learnable linear transformation The final output is obtained. Multi-head attention is conducive to obtaining richer information from different subspaces, thereby enhancing the representation ability of the model.

[0096] The matrices Q, K, and V of the first multi-head attention input of the encoder and decoder all come from the same eigenvector, so they can be called self-attention. The query matrix Q of the second multi-head attention of the decoder comes from the final output of the encoder, which is different from the source of the key matrix K and the value matrix V, so it is called cross-attention. In general, cross-attention is often used for interaction between data of different modalities, which plays a very important role in cross-modal tasks such as image description generation.

[0097] Another component of the Transformer model is the Feed-Forward Network (FFN). Its main function is to provide nonlinear transformation and improve the model's ability to fit complex processes. The formula is as follows. It can be seen that the feed-forward network takes the output features of the multi-head attention as input and performs nonlinear transformation on the feature vector through two fully connected layers (FC). In addition, the ReLU activation function and random dropout are between the two layers.

[0098] FFN(x)=FC(Dropout(ReLU(FC(x,4d),t),d)) (13)

[0099] The dimension of the input feature x is d, it is first projected to 4d dimension and then mapped to d dimension, and t is the random activation rate.

[0100] An additional residual and normalization layer (Add&Norm) is added after each sublayer. It is mainly used after the multi-head attention layer and the feedforward network layer. Unlike sequential algorithms such as recurrent neural networks and convolutional neural networks, the multi-head attention in the Transformer model pays attention to the importance of words, but cannot capture the relative or absolute position information between words.

[0101] Comparative Example

[0102] The proposed method is evaluated on the widely used MS-COCO 2014 dataset, which consists of 123,287 images (82,783 for training and 40,504 for validation), each with 5 reference descriptions. According to the commonly used "Karpathy" partitioning method, the training and test datasets are repartitioned in the experiment, with 113,287 images for training, 5,000 images for validation, and 5,000 for offline evaluation. As usual, all description sentences are converted to lowercase and words that appear less than 6 times are deleted, resulting in a vocabulary of 9,487 words.

[0103] This embodiment is based on the Pytorch framework, and the graphics card used is NVIDIA RTX 3090. The Swin-L backbone network is used to extract the visual features of the image grid. The kMax-DeepLab panoramic segmentation network is used to extract the mask visual features of the image. The visual feature dimension is set to 512, the number of heads in the multi-head attention is set to 8, and the encoder and decoder transformer block size is set to 3. In the training phase, the model is first pre-trained using the cross entropy loss under the condition of a batch size of 10, and the number of iterations is set to 20. In the model training phase, this experiment uses the Adam optimization algorithm and the warmup learning rate preheating technique to optimize the proposed model, and the number of preheating steps is set to 1000. Subsequently, the SCST self-evaluation reinforcement learning strategy is used to continue to iterate the model for 30 times, and the learning rate is set to 5×10-6. In the model testing phase, this experiment adopts a beam search strategy with a beam width of 3.

[0104] The image description generation experiment in this embodiment uses the machine translation evaluation indicators CIDEr, BLEU, METEOR, ROUGE-L, and SPICE to evaluate the generated description. Table 1 shows the performance results of each method for generating description sentences under the "Karpathy" partition of the MSCOCO dataset. Among them, BN, M, R, C, and S are the abbreviations of BLEU-N, METEOR, ROUGE-L, CIDEr, and SPICE, respectively. The comparison models include: ORT, M2 Transformer, X-Transformer, RSTNet, Dual Global, DLCT

[13] , PureT (including PureT-standard and PureT-Swin), and RAS-FSG. ORT, X-Transformer, M2Transformer, and GET use the regional features extracted by Faster RCNN for experiments, RSTNet uses grid features, PureT-standard uses the standard Transformer as the visual encoder, and PureT-Swin uses the SwinTransformer as the visual encoder. Dual Global and DLCT combine grid features with regional features. RAS-FSG proposes a relationship-aware selection module, and uses semantic information as supervisory knowledge to filter noise from grid features. It can be seen that the CIDEr index of the proposed model reached the highest 138.5. Compared with the current best model PureT-Swin, the model of this embodiment obtains better BLEU-1, BLEU-4, ROUGH-L and CIDEr indicators, and the scores of METEOR and SPICE are slightly lower than PureT-Swin. The experimental results show that the method proposed in this embodiment uses panoramic segmentation mask features as visual features, which can effectively improve the accuracy of image description generation, and generally exceeds the existing methods based on grid features and object region features.

[0105] Table 1 Performance comparison of methods in the “Karpathy” partition of the MSCOCO dataset

[0106]

[0107]

[0108] In order to further analyze the effectiveness of panoptic segmentation mask features, Figure 4The description sentences generated by the standard Transformer model and the method of this embodiment are compared. Among them, the standard Transformer model uses the target features extracted by Faster RCNN. For the convenience of comparison, the real words are marked in bold, and the words generated by the method of this paper are marked in italics. It can be seen that the proposed method performs panoramic segmentation on the first row of pictures, which can accurately distinguish the three young people in the picture, thus facilitating accurate headcount and generating correct words. However, the standard Transformer model cannot accurately use the segmentation target features to generate accurate quantifiers. In the second row of pictures, there is an overlap problem when detecting the "cake" and "fork" objects, and the standard Transformer model has visual confusion and wrong associations, resulting in the generation of the incorrect word "fork". In contrast, the method of this embodiment uses the panoramic segmentation model to provide a clear visual outline for "spoon", thereby generating the correct word "spoon". Qualitative experiments show that the method of this embodiment can effectively use the panoramic segmentation mask features to capture more comprehensive and fine-grained visual information, effectively alleviating the problem of visual confusion.

Claims

1. A method for generating image descriptions by combining panoptic segmentation with multiple visual features, characterized in that: include: Extract the grid feature G of the input image and the mask feature M based on panoptic segmentation and encode them; The text semantic features are cross-fused with the grid features G and the mask features M through two-stream cross attention; The kMax-DeepLab panoptic segmentation network is used to generate mask features M, including: Step S121, obtaining an image feature F of an input image I through a pixel encoder; Step S122, using a pixel decoder to upsample the image feature F to extract pixel features; Step S123, using a kMax decoder to convert the randomly initialized cluster center into a mask embedding vector; Step S124, multiplying the mask embedding vector and the pixel feature to obtain the segmentation feature S; Step S125: pooling the image feature F and the segmentation feature S to generate the final mask feature M. M(N,D M )=∑ hw S(N,h,w)·F(h,w,D M ) (2) Where S=(s1,s2,...,s N ),s l ∈{0,1} H×W represents the mask matrix, l represents the index number of the segmented area, each element represents whether the corresponding pixel belongs to a certain instance object, N represents the number of segmented areas in the image, and the number of each image is not fixed. M Indicates the visual feature dimension corresponding to each segmented area, which is consistent with the feature dimension of the feature map F. h and w represent the row and column numbers of each segmented area. H and W represent the height and width of the input image respectively. Text SemanticsX i The feature acquisition methods include: Step S201, by adding the word embedding vector and the position code, the word feature X = (x1, x2, ..., x T ), where T is the sentence length, i.e. the number of words; Step S202: Send the word features to the decoder. In each decoding layer, the word features are encoded by the self-attention module to obtain the semantic features X i : X i =MultiHead(X i ,X i ,X i ) (3) X i =X i +LayerNorm(X i ) (4) Where i represents the decoding layer index, MultiHead(·) represents the Transformer multi-head attention operation, and LayerNorem(·) represents the normalization operation; The process of cross-attention of two streams to cross-fuse text semantic features with grid features G and mask features M includes: In the i-th decoding layer, the grid visual feature G i and the mask visual feature M i As key-value, the text semantic feature X is input into two multi-head attention modules respectively. i As the query vector, we get the mask visual feature M i and text semantic feature X i Fusion features And the grid visual feature G i and text semantic feature X i Fusion Among them, α M and α G is the dual-stream weight, W M , W G represents the learnable weight parameter, b M , b G represents the learnable bias parameter, which is calculated by the Sigmoid activation function σ(·); Get the dual-stream fusion feature R i Represents the dot product operation; Dual stream fusion feature R i After the residual normalization operation, it is input into the forward feedback layer to form the input of the next decoding layer X i+1 =FeedForward(X i +Layernorm(R i )) The output of the last layer of decoding blocks passes through the Generator to predict the word output X i .

2. The method according to claim 1, characterized in that The Swin attention mechanism is used to obtain the grid features G of the input image, including: Step S111, dividing the input image I into non-overlapping small blocks through the patch partition layer; Step S112, mapping each small block into a low-dimensional space with a dimension of C through a linear embedding layer to generate a block embedding vector; Step S113, perform four-stage downsampling on the block embedding vector to obtain a shape of H×W×D G The grid feature G 3. The method according to claim 1, characterized in that In step S202, the word feature is subjected to a masked self-attention mechanism operation, and then the semantic feature X is obtained through residual normalization operation. i .

4. The method according to claim 3, characterized in that Before performing the masked self-attention mechanism operation on the word features, a position encoding operation is performed to obtain the sequential position information pos of the word Among them, pos represents the position of the current word in the sequence, j is the index of the embedding dimension, and d is the dimension of the embedding.

Images