Contour extraction and detection method and system for chest lesion images

Abstract

The present invention discloses a method for contour extraction and detection of chest lesion images. The method comprises: dividing an input chest lesion image into blocks, flattening the blocks into one-dimensional vectors and adding position codes to obtain block embeddings, inputting the block embeddings into a Transformer encoder to extract image features, obtaining block feature codes, respectively decoding the block feature code sequences into chest lesion contour feature maps using a linear decoder and a Transformer decoder, fusing the two chest lesion contour feature maps obtained by the linear decoder and the Transformer decoder by connection, and performing interpolation upsampling to obtain a chest lesion contour map. The present invention also discloses a system for contour extraction and detection of chest lesion images. The present invention adopts a Transformer-based approach, and the attention mechanism of the Transformer can model long-term dependencies and effectively capture global features, thereby improving the accuracy. Moreover, the present invention does not rely on additional input data, has a fast calculation speed, and has a low model complexity.

Description

Technical Field

[0001] The present invention relates to computer vision and image processing, and in particular to a method and system for extracting and detecting the contour of a chest lesion image. Background Art

[0002] The thoracic cavity is an important part of the human body. It is formed by the diaphragm and the upper end of the diaphragm, with the upper opening of the diaphragm connecting to the neck. The lower boundary of the thoracic cavity is separated from the abdomen by the diaphragm. The thoracic cavity contains the mediastinum in the middle and the lungs and pleural cavity on the upper and lower sides. Part of the thoracic duct is an inner cavity formed by the scapula, lumbar vertebrae and ribs. The upper end is connected to the neck and the lower end is separated from the abdomen by the diaphragm. The heart, lungs and other organs are all in the thoracic cavity. There are many important organs in the thoracic cavity, including the alveoli on both sides, the heart mainly located on the left side, as well as important organs such as the esophagus and bronchi. The thoracic cavity is very important in medicine because diseases of the heart and lungs can easily pose a threat to life.

[0003] Extracting chest lesion contours is an important area of medical imaging research. Its goal is to extract the chest contour from medical images to facilitate physicians' assessment of a patient's health. However, existing technologies typically rely on the naked eye to determine lesions in medical images, which can sometimes lead to inaccurate detection. Therefore, there is an urgent need for a chest lesion detection method that can efficiently and accurately assist physicians in diagnosis.

[0004] With the development of deep learning technology, research on chest lesion contour extraction and detection has made significant progress and breakthroughs, and many models using convolutional neural networks have achieved certain results. However, because convolutional neural networks cannot effectively extract global features, simple convolutional neural networks cannot effectively achieve the purpose of chest lesion contour extraction and detection.

[0005] One existing technique is the gradient operator-based method described in the patented "Abdominal MRI Image Contour Extraction Method Based on Flow-XDoG Operator." This method approximates the gradient operator by performing vertical and horizontal segmentation on the image, using the gradient operator to extract the boundary between the object and the background in the image. However, the disadvantage of this approach is that the gradient operator is sensitive to noise in the image. After filtering to remove noise, the image edges become blurred, losing edge information.

[0006] The second existing technology is a directional morphology-based method described in the paper "Binary Handwriting Image Enhancement by Directional Field-Guided Morphology." This method performs a rolling rotation operation on a binary edge image and defines a directional objective function to determine the movement direction of the structural element, thereby obtaining the target's contour information. However, this method relies on the results of previous image edge extraction, is susceptible to noise, and can easily fall into local contour extraction when edges are discontinuous.

[0007] The third existing technique is the fully convolutional neural network-based method described in the paper "Fully convolutional networks for semantic segmentation." This method uses convolutional neural networks, such as ResNet and DeepLab, to extract features from images. It then uses deconvolution to upsample the image and generate a chest contour map. The entire training process is supervised by a cross-entropy loss function. However, this method suffers from the limitations of the receptive field of convolutional neural networks. A single convolution layer cannot capture long-range features, resulting in a low-accuracy chest contour map. Summary of the Invention

[0008] The purpose of the present invention is to overcome the shortcomings of existing methods and propose a method and system for contour extraction and detection of chest lesion images. The main problems solved by the present invention are: first, the existing method based on gradient operators is sensitive to noise in the image; second, the directional morphological method relies on the results of the previous image edge extraction and is prone to local contour extraction when the edges are discontinuous; third, the method based on the fully convolutional neural network is limited by the receptive field, which makes the accuracy of human body analysis images not high.

[0009] In order to solve the above problems, the present invention proposes a method for contour extraction and detection of chest lesion images, the method comprising:

[0010] Input chest lesion images and contour images, perform data augmentation on the input images, and process them into a uniform size;

[0011] Dividing the input chest lesion image into blocks, flattening each block into a one-dimensional vector, and adding a position code to the one-dimensional vector to obtain a block embedding;

[0012] The block embedding is input into the Transformer encoder to extract image features and obtain block feature encoding. The Transformer encoder contains L layers, and the input sequence is denoted as z0. The output is a block feature encoding sequence z containing rich contextual semantic information. L ;

[0013] The block feature encoding sequence z is encoded using a linear decoder and a Transformer decoder respectively. L Decoded into chest lesion contour feature map;

[0014] The chest lesion contour feature map obtained by the linear decoder and the chest lesion contour feature map obtained by the Transformer encoder are fused in a connection manner, and the chest lesion contour map is obtained by interpolation upsampling. The chest lesion contour map has the same size as the original input chest lesion image.

[0015] Preferably, the chest lesion image and the contour image are input, data enhancement is performed on the input images, and they are processed into a uniform size, specifically:

[0016] The input images of chest lesions and contour images are from real medical scenarios, while the contour images are manually labeled. To make the trained model more robust, the images are randomly rotated, horizontally mirrored, and randomly cropped for data enhancement. Finally, all images are scaled to a uniform size.

[0017] Preferably, the input chest lesion image is divided into blocks, each block is flattened into a one-dimensional vector, and a position code is added to the one-dimensional vector to obtain a block embedding, specifically:

[0018] The input image size is H×W×C, where H, W, and C refer to the number of channels, width, and height of the image, respectively. The input image is divided into N blocks, and the size of each block is P×P, that is, N=HW / P 2 , the number of channels of each block is the same as that of the image, which is C. Flattening each block into a one-dimensional vector will result in N vectors of dimension D, D = P 2 C;

[0019] In order to capture the position information in the image, the N×D block representation is added to the position embedding of dimension N×D to obtain the block embedding, which can be learned.

[0020] Preferably, the block embedding is input into the Transformer encoder to extract image features and obtain block feature coding. The Transformer encoder includes L layers, and the input sequence is z0, and the output is a block feature coding sequence z containing rich contextual semantic information. L , specifically:

[0021] A Transformer layer consists of a multi-head attention layer (MSA) and a two-layer fully connected layer (MLP). Layer normalization (LN) is applied before each block and residual connections are added. Let i∈{1,...,L} be calculated as follows:

[0022] a i-1 =MSA(LN(z i-1 ))+z i-1

[0023] z i =MLP(LN(a i-1 ))+a i-1

[0024] The self-attention mechanism in Transformer helps capture the global context information of the image. Let Q, K, and V represent the transformation matrices of query, key, and value. The calculation method of self-attention is as follows:

[0025]

[0026] Preferably, the block feature encoding sequence z is encoded using a linear decoder and a Transformer decoder respectively. L Decoded into chest lesion contour feature map, specifically:

[0027] The linear encoder uses a simple linear layer to decode the block encoding sequence, and the final linear layer outputs a 2D chest lesion contour feature;

[0028] The input of the Transformer decoder is the block encoding sequence and the corresponding query. The block encoding sequence and the query first calculate the cross attention:

[0029] X l =softmax(Q l K l )V l +X l-1

[0030] Among them, l is the subscript of the layer number, X l is the query feature of the lth layer, Q l is the query input to layer l, V l and K l The image features of the l-th layer input are transformed by two different linear transformation functions f V and f K The transformed matrix is then normalized by the cross-attention result and passed through a self-attention layer. The result calculated by the self-attention layer is normalized and then output through the feedforward layer to produce the final query feature.

[0031] Accordingly, the present invention also provides a system for extracting and detecting contours of chest lesion images, comprising:

[0032] An image preprocessing unit is used to input chest lesion images and contour images, perform data enhancement on the input images, and process them into a uniform size;

[0033] a block embedding unit, configured to divide the input chest lesion image into blocks, flatten each block into a one-dimensional vector, and add a position code to the one-dimensional vector to obtain a block embedding;

[0034] A block feature encoding unit, configured to embed the block into a Transformer encoder to extract image features and obtain block feature encoding;

[0035] A decoding unit is used to decode the block feature encoding sequence z using a linear decoder and a Transformer decoder respectively. L Decoded into chest lesion contour feature map;

[0036] The contour map acquisition unit is used to fuse the chest lesion contour feature map obtained by the linear decoder and the chest lesion contour feature map obtained by the Transformer encoder in a connection manner, and use interpolation upsampling to obtain a chest lesion contour map, and the chest lesion contour map is the same size as the original input chest lesion image.

[0037] The implementation of the present invention has the following beneficial effects:

[0038] The present invention does not rely on any additional input data and has the advantages of fast calculation speed and low model complexity compared with other methods. The present invention only uses the Transformer network, fully leveraging the ability of the attention mechanism to capture global features and maximizing the accuracy of the contour map. The input and output between the various parts of the present invention are interconnected to extract and integrate different features, thereby improving efficiency and making the generated contour map more in line with people's expectations. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is an overall flow chart of a method for contour extraction and detection of chest lesion images according to an embodiment of the present invention;

[0040] Figure 2 is a flow chart of a Transformer decoder according to an embodiment of the present invention;

[0041] Figure 3 4 is a structural diagram of a system for extracting and detecting contours of chest lesion images according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0043] Figure 1 FIG. 1 is a general flow chart of a method for extracting and detecting contours of chest lesion images according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0044] S1, input chest lesion images and contour images, perform data augmentation on the input images, and process them into a uniform size;

[0045] S2, dividing the input chest lesion image into blocks, flattening each block into a one-dimensional vector, and adding a position code to the one-dimensional vector to obtain a block embedding;

[0046] S3, embedding the block into the Transformer encoder to extract image features and obtain block feature encoding;

[0047] S4, respectively use the linear decoder and Transformer decoder to encode the block feature sequence z L Decoded into chest lesion contour feature map;

[0048] S5, fusing the chest lesion contour feature map obtained by the linear decoder and the chest lesion contour feature map obtained by the Transformer encoder in a concatenated manner, and obtaining a chest lesion contour map by interpolation upsampling, wherein the chest lesion contour map has the same size as the original input chest lesion image;

[0049] S6, using the chest lesion contour map and the input contour map to calculate the loss function, and repeatedly training the Transformer neural network using the gradient descent method until the loss function converges.

[0050] Step S1 is as follows:

[0051] S1-1, input chest lesion images and contour images. The chest lesion images are from real medical scenarios, and the contour images are formed by manual annotation. To make the trained model more robust, the images are randomly rotated, horizontally mirrored, and randomly cropped for data enhancement. Finally, all images are scaled to a uniform size.

[0052] Step S2 is as follows:

[0053] S2-1, let the input image size be H×W×C, where H, W, and C refer to the number of channels, width, and height of the image respectively. The input image is divided into N blocks, and the size of each block is P×P, that is, N=HW / P 2 , the number of channels of each block is the same as that of the image, which is C. Flattening each block into a one-dimensional vector will result in N vectors of dimension D, D = P 2 C;

[0054] S2-2, in order to capture the position information in the image, the N×D block representation is added to the position embedding with dimension N×D to obtain the block embedding, which can be learned.

[0055] Step S3 is as follows:

[0056] S3-1, a Transformer layer consists of a multi-head attention layer (MSA) and a two-layer fully connected layer (MLP). Layer normalization (LN) is applied before each block and residual connections are added. Let i∈{1,...,L} be calculated as follows:

[0057] a i-1 =MSA(LN(z i-1 ))+z i-1

[0058] z i =MLP(LN(a i-1 ))+a i-1 .

[0059] S3-2, the self-attention mechanism in Transformer is conducive to capturing the global context information of the image. Let Q, K and V represent the transformation matrix of query, key and value. The calculation method of self-attention is as follows:

[0060]

[0061] Step S4, as Figure 2 As shown, the details are as follows:

[0062] S4-1, the linear encoder uses a simple linear layer to decode the block encoding sequence, and the final linear layer output is a 2D chest lesion contour feature;

[0063] S4-2, the input of the Transformer decoder is the block encoding sequence and the corresponding query. The block encoding sequence and the query first calculate the cross attention:

[0064] X l =softmax(Q l K l )V l +X l-1

[0065] Among them, l is the subscript of the layer number, X l is the query feature of the lth layer, Q l is the query input to layer l, V l and K l The image features of the l-th layer input are transformed by two different linear transformation functions f V and f K The transformed matrix is then normalized by the cross-attention result and passed through a self-attention layer. The result calculated by the self-attention layer is normalized and then output through the feedforward layer to produce the final query feature.

[0066] Step S5 is as follows:

[0067] S5-1, the chest lesion contour feature map obtained by the linear decoder and the chest lesion contour feature map obtained by the Transformer encoder are fused by connection, and the chest lesion contour map is obtained by interpolation upsampling. The chest lesion contour map is the same size as the original input chest lesion image.

[0068] Step S6 is as follows:

[0069] S6-1 uses cross entropy as the loss function, which is calculated as follows:

[0070]

[0071] Among them, M represents the number of categories, y c It is a binary vector with only two values: 0 and 1. If the category is the same as the sample category, it takes 1, otherwise it takes 0. c It represents the probability that the predicted sample belongs to category c.

[0072] Accordingly, the present invention also provides a system for extracting and detecting the contour of a chest lesion image, such as Figure 3 Shown, including:

[0073] The image preprocessing unit 1 is used to input chest lesion images and contour images, perform data enhancement on the input images, and process them into a uniform size.

[0074] Specifically, chest lesion images and contour images are input. The chest lesion images are from real medical scenes, and the contour images are formed by manual annotation. In order to make the trained model more robust, the images are randomly rotated, horizontally mirrored, and randomly cropped for data enhancement. Finally, all the images are scaled to a uniform size.

[0075] The block embedding unit 2 is used to divide the input chest lesion image into blocks, flatten each block into a one-dimensional vector, and add a position code to the one-dimensional vector to obtain a block embedding.

[0076] Specifically, the input image size is H×W×C, where H, W, and C refer to the number of channels, width, and height of the image, respectively. The input image is divided into N blocks, and the size of each block is P×P, that is, N=HW / P 2 , the number of channels of each block is the same as that of the image, which is C. Flattening each block into a one-dimensional vector will result in N vectors of dimension D, D = P 2 C;

[0077] In order to capture the position information in the image, the N×D block representation is added to the position embedding of dimension N×D to obtain the block embedding, which can be learned.

[0078] The block feature encoding unit 3 is used to embed the block into the Transformer encoder to extract image features and obtain block feature encoding. The Transformer encoder includes L layers, and the input sequence is z0. The output is a block feature encoding sequence z containing rich contextual semantic information. L .

[0079] Specifically, a Transformer layer consists of a multi-head attention layer (MSA) and a two-layer fully connected layer (MLP). Layer normalization (LN) is applied before each block and a residual connection is added. Let i∈{1,...,L} be calculated as follows:

[0080] a i-1 =MSA(LN(z i-1 ))+z i-1

[0081] z i =MLP(LN(a i-1 ))+a i-1

[0082] The self-attention mechanism in Transformer helps capture the global context information of the image. Let Q, K, and V represent the transformation matrices of query, key, and value. The calculation method of self-attention is as follows:

[0083]

[0084] Decoding unit 4 is used to use linear decoder and Transformer decoder to decode the block feature encoding sequence z L Decoded into chest lesion contour feature map.

[0085] Specifically, the linear encoder uses a simple linear layer to decode the block encoding sequence, and the final linear layer outputs the 2D chest lesion contour features;

[0086] The input of the Transformer decoder is the block encoding sequence and the corresponding query. The block encoding sequence and the query first calculate the cross attention:

[0087] X l =softmax(Q l K l )V l +X l-1

[0088] Among them, l is the subscript of the layer number, X l is the query feature of the lth layer, Q l is the query input to layer l, V l and K l The image features of the l-th layer input are transformed by two different linear transformation functions f V and f K The transformed matrix is then normalized by the cross-attention result and passed through a self-attention layer. The result calculated by the self-attention layer is then normalized and passed through the feed-forward layer to output the final query feature.

[0089] The contour map acquisition unit 5 is used to fuse the chest lesion contour feature map obtained by the linear decoder and the chest lesion contour feature map obtained by the Transformer encoder in a connection manner, and use interpolation upsampling to obtain a chest lesion contour map, and the chest lesion contour map is the same size as the original input chest lesion image.

[0090] Therefore, the present invention adopts a contour extraction and detection method for chest lesion images. The present invention does not rely on any additional input data and has the advantages of fast calculation speed and low model complexity compared with other methods. The present invention only uses the Transformer network, which fully utilizes the ability of the attention mechanism to capture global features and maximizes the accuracy of the contour map. The input and output between each part of the present invention are interconnected to extract and integrate different features, thereby improving efficiency and making the generated contour map more in line with people's expectations.

[0091] The above is a detailed introduction to the contour extraction and detection method and system for chest lesion images provided by the embodiments of the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for extracting and detecting the contour of a chest lesion image, characterized in that: The method comprises: Input chest lesion images and contour images, perform data augmentation on the input images, and process them into a uniform size; Dividing the input chest lesion image into blocks, flattening each block into a one-dimensional vector, and adding a position code to the one-dimensional vector to obtain a block embedding; The block embedding is input into the Transformer encoder to extract image features and obtain block feature encoding. The Transformer encoder contains L layers, and the input sequence is denoted as z0. The output is a block feature encoding sequence z containing rich contextual semantic information. L ; The block feature encoding sequence z is encoded using a linear decoder and a Transformer decoder respectively. L Decoded into chest lesion contour feature map; The chest lesion contour feature map obtained by the linear decoder and the chest lesion contour feature map obtained by the Transformer encoder are fused in a concatenated manner, and the chest lesion contour map is obtained by interpolation upsampling. The chest lesion contour map has the same size as the original input chest lesion image. Calculating a loss function using the chest lesion contour map and the input contour map, and repeatedly training the Transformer neural network using a gradient descent method until the loss function converges; Wherein, the block feature encoding sequence z is encoded using a linear decoder and a Transformer decoder respectively. L Decoded into chest lesion contour feature map, specifically: The linear encoder uses a simple linear layer to decode the block encoding sequence, and the final linear layer outputs a 2D chest lesion contour feature; The input of the Transformer decoder is the block encoding sequence and the corresponding query. The block encoding sequence and the query first calculate the cross attention: X l =softmax(Q l K l )V l +X l-1 Among them, l is the subscript of the layer number, X l is the query feature of the lth layer, Q l is the query input to layer l, V l and K l The image features of the l-th layer input are transformed by two different linear transformation functions f V and f K The transformed matrix is then normalized by the cross-attention result and passed through a self-attention layer. The result calculated by the self-attention layer is normalized and then output through the feedforward layer to produce the final query feature.

2. The method for extracting and detecting the contour of a chest lesion image according to claim 1, wherein: The input chest lesion image and contour image are subjected to data enhancement and processed into a uniform size, specifically: The input images of chest lesions and contour images are from real medical scenarios, while the contour images are manually labeled. To make the trained model more robust, the images are randomly rotated, horizontally mirrored, and randomly cropped for data enhancement. Finally, all images are scaled to a uniform size.

3. The method for extracting and detecting the contour of a chest lesion image according to claim 1, wherein: The input chest lesion image is divided into blocks, each block is flattened into a one-dimensional vector, and a position code is added to the one-dimensional vector to obtain a block embedding, specifically: The input image size is H×W×C, where H, W, and C refer to the number of channels, width, and height of the image, respectively. The input image is divided into N blocks, and the size of each block is P×P, that is, N=HW / P 2 , the number of channels of each block is the same as that of the image, which is C. Flattening each block into a one-dimensional vector will result in N vectors of dimension D, D = P 2 C; In order to capture the position information in the image, the N×D block representation is added to the position embedding of dimension N×D to obtain the block embedding, which can be learned.

4. The method for extracting and detecting the contour of a chest lesion image according to claim 1, wherein: The block embedding is input into the Transformer encoder to extract image features and obtain block feature coding. The Transformer encoder includes L layers, and the input sequence is recorded as z0. The output is a block feature coding sequence z containing rich contextual semantic information. L , specifically: A Transformer layer consists of a multi-head attention layer (MSA) and a two-layer fully connected layer (MLP). Layer normalization (LN) is applied before each block and residual connections are added. Let i∈{1,...,L} be calculated as follows: a i-1 =MSA(LN(z i-1 ))+z i-1 z i =MLP(LN(a i-1 ))+a i-1 The self-attention mechanism in Transformer helps capture the global context information of the image. Let Q, K, and V represent the transformation matrices of query, key, and value. The calculation method of self-attention is as follows:

5. The method for contour extraction and detection of chest lesion images according to claim 1, wherein: The loss function is calculated using the chest lesion contour map and the input contour map, and the Transformer neural network is repeatedly trained using the gradient descent method until the loss function converges, specifically: Using cross entropy as the loss function, it is calculated as follows: Among them, M represents the number of categories, y c It is a binary vector with only two values: 0 and 1. If the category is the same as the sample category, it takes 1, otherwise it takes 0. c It represents the probability that the predicted sample belongs to category c.

6. A chest lesion image contour extraction and detection system, characterized in that: The system comprises: Image preprocessing unit, used to perform data enhancement on the input human body image and human body analysis image, and process them into a uniform size; a block embedding unit, configured to divide the input chest lesion image into blocks, flatten each block into a one-dimensional vector, and add a position code to the one-dimensional vector to obtain a block embedding; A block feature encoding unit, configured to embed the block into a Transformer encoder to extract image features and obtain block feature encoding; A decoding unit is used to decode the block feature encoding sequence z using a linear decoder and a Transformer decoder respectively. L Decoded into chest lesion contour feature map; A contour map acquisition unit is used to fuse the chest lesion contour feature map obtained by the linear decoder and the chest lesion contour feature map obtained by the Transformer encoder in a concatenated manner, and obtain a chest lesion contour map using interpolation upsampling. The chest lesion contour map has the same size as the original input chest lesion image; The decoding unit requires a linear encoder to use a simple linear layer to decode the block coding sequence, and the final linear layer output is a 2D chest lesion contour feature; The input of the Transformer decoder is the block encoding sequence and the corresponding query. The block encoding sequence and the query first calculate the cross attention: X l =softmax(Q l K l )V l +X l-1 Among them, l is the subscript of the layer number, X l is the query feature of the lth layer, Q l is the query input to layer l, V l and K l The image features of the l-th layer input are transformed by two different linear transformation functions f V and f K The transformed matrix is then normalized by the cross-attention result and passed through a self-attention layer. The result calculated by the self-attention layer is normalized and then output through the feedforward layer to produce the final query feature.

7. The chest lesion image contour extraction and detection system according to claim 6, wherein: The image preprocessing unit needs to input chest lesion images and contour images. The chest lesion images are derived from real medical scenes, and the contour images are formed by manual annotation. In order to make the trained model more robust, the images are randomly rotated, horizontally mirrored, and randomly cropped for data enhancement. Finally, all the images are scaled to a uniform size.

8. The chest lesion image contour extraction and detection system according to claim 6, wherein: The block embedding unit needs to remember that the input image size is H×W×C, where H, W, and C refer to the number of channels, width, and height of the image respectively. The input image is divided into N blocks, and the size of each block is P×P, that is, N=HW / P 2 , the number of channels of each block is the same as that of the image, which is C. Flattening each block into a one-dimensional vector will result in N vectors of dimension D, D = P 2 C; In order to capture the position information in the image, the N×D block representation is added to the position embedding of dimension N×D to obtain the block embedding, which can be learned.

9. The chest lesion image contour extraction and detection system according to claim 6, wherein: The block feature encoding unit requires a Transformer layer consisting of a multi-head attention layer (MSA) and a two-layer fully connected layer (MLP). Layer normalization (LN) is applied before each block and residual connections are added. Note that i∈{1,...,L} is calculated as follows: a i-1 =MSA(LN(z i-1 ))+z i-1 z i =MLP(LN(a i-1 ))+a i-1 The self-attention mechanism in Transformer helps capture the global context information of the image. Let Q, K, and V represent the transformation matrices of query, key, and value. The calculation method of self-attention is as follows:

Images