A structure of a cross-filter transformer, an image semantic segmentation model and method

Abstract

The application discloses a structure of a cross filtering transformer, which comprises feature embedding, multi-head cross attention and a feedforward neural network, obtains a feature map F1 of a current layer and a feature map F2 of a previous layer, the size of the feature map F1 is H, the channel is C, the size of the feature map F2 is H*2, and the channel is C / 2; the feature map F1 and the feature map F2 are respectively input; the feature map F1 and the feature map F2 of the two branches are converted into low-dimensional feature vectors through a 2D convolution layer, the feature map F1 and the feature map F2 are respectively flattened into 1D sequences, the input sequences are divided into multiple sub-sequences, the attention of the sub-sequences is calculated in different subspaces, then the output of each attention is taken as a head, and all the heads are spliced together; in the feedforward neural network, a 2D convolution is used to replace a full connection layer. The problem that an original skip connection in a U-Net cannot reduce the semantic gap between an encoder and a decoder is solved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a cross-filtering transformer structure, an image semantic segmentation model, and a method. Background Technology

[0002] Currently, numerous CNN models have been implemented for image semantic segmentation, with most being improvements upon U-Net. U-Net now has several variants, including U-Net++ and U-Net 3+. U-Net++ uses dense skip connections to bridge the gap between low-level and high-level semantic information, while U-Net 3+ combines multi-scale feature information, using the concept of full-scale connections to extract both fine-grained and coarse-grained information. However, the introduction of full-scale connections increases the computational complexity of the model, making U-Net3+'s computational complexity as high as 151.69G.

[0003] CNN models cannot preserve sufficient object space details, nor can they establish long-range dependencies for dense prediction tasks. CNNs can only extract local features from neighboring pixels, making it difficult to obtain global contextual information. For complex tasks, detailed information and global context are equally important; objects of the same type may have significant differences in texture and location. Relying solely on the CNN structure is insufficient to identify the similarity between segmented objects; a large amount of contextual information is required, which explains the necessity of the Transformer structure for segmentation tasks.

[0004] The Transformer was initially used in natural language processing for language translation. In the past two years, researchers have introduced the Transformer structure into computer vision segmentation tasks, achieving excellent segmentation results and sparking a surge of interest in the field. Most of this work is an improvement on the U-Net model. TransU-Net was the first model to use the Transformer structure in medical image segmentation. The Transformer structure was placed at the bottom of the U-Net encoder as a feature extractor. TransU-Net showed outstanding segmentation performance in multi-organ segmentation, but its parameter count was as high as 105.28M. Similarly, LeViT U-Net adopted the same combination approach, combining LeViT modules with U-Net, but this model still had 52.15M parameters. Swin U-Net was the first model to use only a pure Transformer structure (excluding CNN structures) for medical image segmentation. Swin U-Net utilized a window shifting mechanism to reduce computational complexity. UTNet improved the multi-head attention mechanism and added Transformer structures to the encoder and decoder of the U-shaped model.

[0005] However, some researchers pointed out that the encoder and decoder in the U-Net framework did not have major flaws in UCTransNet. They demonstrated that not all skip connections were effective in experiments. Some researchers redesigned skip connections using Transformers. UCTransNet introduced the CTrans (Channel Transformer) module in skip connections, but CTrans directly concatenates channels in parallel, which significantly increases the number of parameters and computational complexity. This model has 66.44M parameters and 32.93G computational complexity. Furthermore, in MISSFormer and TransBridge, Transformers were added to the original skip connections of U-Net. TransBTS and TransAttU-Net introduced Transformer structures to fill the bottleneck between the encoder and decoder used for brain tumor segmentation, achieving excellent segmentation results. Although TransAttU-Net reduced the number of model parameters, its computational complexity increased significantly, reaching 67.81G. One researcher designed an enhancement module that uses Transformers to replace skip connections, proposing a new network model called TransU-Net+, which introduces a Transformer structure at the bottom of the transformer encoder, inspired by TransU-Net and UCTransNet. In summary, many problems and challenges still remain:

[0006] (1) The simple skip connections in U-Net cannot effectively reduce the semantic gap between the encoder and decoder. The original skip connections can partially recover the spatial information lost due to successive downsampling, but they cannot solve this problem well for complex segmentation tasks.

[0007] (2) Although the Transformer structure can compensate for the deficiency of CNN models in establishing long-range dependencies, its global self-attention mechanism is difficult to capture local information. However, local information and global information are equally important for complex segmentation tasks.

[0008] (3) Applying Transformer to image tasks will generate a large amount of computational overhead and parameters. For example, the TransU-Net model has 105.28M parameters, UCTransNet has 66.44 parameters, and TransAttU-Net has a computational complexity of 67.81G.

[0009] (4) The Transformer model requires a large amount of training data to achieve good performance, but some tasks have limited data, which will lead to poor model performance.

[0010] These are all problems and shortcomings of existing technologies. Summary of the Invention

[0011] In view of the shortcomings of the prior art, the technical problem to be solved by the present invention is: how to provide a cross-filtering transformer structure that can solve the problem that the original skip connections in U-Net cannot reduce the semantic gap between the encoder and decoder.

[0012] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0013] A cross-filtering transformer architecture includes feature embedding, multi-head cross-attention, and a feedforward neural network. Feature embedding represents the transformation of input image data into a representation in a vector space, enabling the model to process semantic information between data. Multi-head cross-attention represents a cross-attention mechanism used in the transformer to handle local and global correlations between data. The feedforward neural network represents a fundamental component of the transformer that helps the model learn the nonlinear features of the data.

[0014] Obtain the feature map F1 of the current layer and the feature map F2 of the previous layer. Feature map F1 has a size of H and a channel of C, and feature map F2 has a size of H*2 and a channel of C / 2. Feature maps F1 and F2 are the inputs.

[0015] The feature maps F1 and F2 of the two branches are converted into low-dimensional feature vectors through 2D convolutional layers. Then, feature maps F1 and F2 are flattened into 1D sequences respectively. The feature embedding is represented as:

[0016] S i =Flatten(Conv(F i ))+Pos i (1)

[0017] X i =LN(S i (2)

[0018] Where i = 1, 2, Conv() is the convolution operation, Pos i It is F i The corresponding positional encoding, LN() is LayerNormalization, X i Representing the input sequence of MHCA, this invention still uses a multi-head approach to improve the parallelism and computational efficiency of the model; the input sequence is divided into multiple subsequences, and then the attention of each subsequence is calculated in different subspaces. The output of each attention is then used as the head, and all heads are concatenated together.

[0019]

[0020] MHCA O (X1,X2)=Linear(Concat(Head1,...,Head h (4)

[0021] Where h represents the number of headers, Head represents the header, Q, K, and V represent the query, key, and value, respectively, and d k The subspace dimension is represented by `Concat`, the join operation by `Linear`, and `MHCA` by `Linear`. O This is the output of MHCA. In a feedforward neural network, 2D convolutions are used to replace the connection layers. The overall structure is expressed as follows:

[0022] CFTrans = FNN(LN(MHCA) O +S1))+(MHCA O +S1) (5).

[0023] As an optimization, Q, K, and V can all be obtained in the following way:

[0024] First, perform a Fast Fourier Transform on the input sequence, as shown in the following mathematical expression:

[0025] T (b,u,v,c)=F(X) (6)

[0026] Where X∈R b×w×h×c b, h, w, and c represent the batch size, width, height, and number of channels, respectively, and (u, v) represents the position in the frequency domain. Then, a global filter W is applied and multiplied by T, where W ∈ R. b×(w / / 2+1)×(h / / 2+1)×c The mathematical expression is as follows:

[0027] T = T * W (7)

[0028] Finally, the spatial features of the image are recovered from the Fourier space, and the mathematical expression is as follows:

[0029] X (b,w,h,c) =F -1 (T) (8)

[0030] Where (w, h) represents the position in the spatial domain.

[0031] An image semantic segmentation model includes a U-Net image segmentation model and the aforementioned cross-filtering transformer structure; the cross-filtering transformer structure is used between the encoder and decoder in the U-Net structure to reduce the semantic gap between the decoders.

[0032] An image semantic segmentation method includes the following steps:

[0033] (a) Select an image dataset with the same characteristics as the image to be processed and perform batch data preprocessing, adjusting the window width and window level of the image data;

[0034] (b) The image dataset obtained in step (a) is randomly divided into a training set, a validation set and a test set. The training set is used to train the model. The trained model is placed on the validation set for evaluation and the optimal weights are selected. After training, the model is tested on the test set.

[0035] (c) Perform data augmentation on the divided training, validation and test sets, including horizontal flipping, vertical flipping and random rotation;

[0036] (d) The weighted BCE-Dice loss function is used. BCE loss is Binary CrossEntropy Loss, and Dice loss is segmentation loss. BCE loss treats each pixel as an independent sample for prediction, while Dice loss takes a more "holistic" view of the final prediction output. The two loss functions are combined in a weighted manner, with a weight of 0.5, to measure from the perspective of pixels and the whole. The total loss formula is as follows:

[0037] LOSS = W1 × L Dice+W2×L BCE (9)

[0038] Where W1 represents the weights corresponding to the Dice loss, and W2 represents the weights corresponding to the BCE loss, the expression for the Dice loss is as follows:

[0039]

[0040] Where n represents the number of pixels in the image, x i y represents the true value of the i-th pixel. i Let represent the predicted value of the i-th pixel, and t represent a constant. Furthermore, BCE (Binary CrossEntropy Loss) is commonly used to handle binary classification problems, and segmentation problems also perform binary classification on each pixel. This loss checks each pixel one by one, comparing the predicted result (probability distribution vector) for each pixel's class with the one-hot encoded label vector. BCE loss also works well for classification guidance modules. The expression for BCE loss is as follows:

[0041]

[0042] Where n represents the number of pixels in the image, x i y represents the true value of the i-th pixel. i (e) Represent the predicted value of the i-th pixel;

[0043] (f) Based on the BCE_Dice loss function in step (d), the image semantic segmentation model is trained. After the image semantic segmentation model is trained, the image semantic segmentation model is used to complete the image semantic segmentation.

[0044] Compared with the prior art, the present invention has the following advantages:

[0045] (1) Compared with the CNN model, the model structure proposed in this invention can establish long-distance dependencies and has better segmentation effect for complex segmentation tasks.

[0046] (2) Compared with the Transformer model, the Transformer structure designed in this invention can capture global-local information simultaneously. Furthermore, this invention enhances the feature representation in skip connections, better narrowing the semantic gap between the encoder and decoder. In addition, most Transformer models have a large number of parameters and high computational complexity, while this invention has both low parameter count and low computational complexity. Attached Figure Description

[0047] Figure 1This is a schematic diagram of the cross-filtering transformer in this invention;

[0048] Figure 2 This is a schematic diagram of the image semantic segmentation model in this invention;

[0049] Figure 3 This is a flowchart of the image semantic segmentation method in this invention;

[0050] Figure 4 A comparison chart of segmentation predictions using different models. Detailed Implementation

[0051] like Figure 1 As shown, the structure of the cross-filtering transformer in this specific embodiment includes feature embedding, multi-head cross attention, and a feedforward neural network. Feature embedding means converting the input image data into a representation in a vector space, enabling the model to process the semantic information between data. Multi-head cross attention means a cross-attention mechanism used in the transformer to handle the local and global correlations between data. The feedforward neural network means a basic component in the transformer that can help the model learn the nonlinear features of the data.

[0052] Obtain the feature map F1 of the current layer and the feature map F2 of the previous layer. Feature map F1 has a size of H and a channel of C, and feature map F2 has a size of H*2 and a channel of C / 2. Feature maps F1 and F2 are the inputs.

[0053] The feature maps F1 and F2 of the two branches are converted into low-dimensional feature vectors through 2D convolutional layers. Then, feature maps F1 and F2 are flattened into 1D sequences respectively. The feature embedding is represented as:

[0054] S i =Flatten(Conv(F i ))+Pos i (1)

[0055] X i =LN(S i (2)

[0056] Where i = 1, 2, Conv() is the convolution operation, Pos i It is F i The corresponding positional encoding, LN() is LayerNormalization, X i The input sequence for MHCA is divided into multiple subsequences. Attention is then calculated for each subsequence in a different subspace. The output of each attention is then used as a head, and all heads are concatenated together.

[0057]

[0058] MHCA O (X1,X2)=Linear(Concat(Head1,...,Head h (4)

[0059] Where h represents the number of headers, and in this embodiment, h = 4; Head represents the header; Q, K, and V represent the query, key, and value, respectively; and d... k The subspace dimension is represented by `Concat`, the join operation by `Linear`, and `MHCA` by `Linear`. O This is the output of MHCA. In a feedforward neural network, 2D convolutions are used to replace the connection layers. The overall structure is expressed as follows:

[0060] CFTrans = FNN(LN(MHCA) O +S1))+(MHCA O +S1) (5).

[0061] In this specific embodiment, Q, K, and V can all be obtained in the following way:

[0062] First, perform a Fast Fourier Transform on the input sequence, as shown in the following mathematical expression:

[0063] T (b,u,v,c) =F(X) (6)

[0064] Where X∈R b×w×h×c b, h, w, and c represent the batch size, width, height, and number of channels, respectively, and (u, v) represents the position in the frequency domain. Then, a global filter W is applied and multiplied by T, where W ∈ R. b×(w / / 2+1)×(h / / 2+1)×c The mathematical expression is as follows:

[0065] T = T * W (7)

[0066] Finally, the spatial features of the image are recovered from the Fourier space, and the mathematical expression is as follows:

[0067] X (b,w,h,c) =F -1 (T) (8)

[0068] Where (w, h) represents the position in the spatial domain.

[0069] like Figure 2As shown, an image semantic segmentation model includes a U-Net image segmentation model and the aforementioned cross-filtering transformer structure; the cross-filtering transformer structure is used between the encoder and decoder in the U-Net structure to reduce the semantic gap between the decoders.

[0070] Most Transformer models are improvements on U-Net. CNNs can capture details, while Transformer structures can establish long-range dependencies. Combining CNNs with Transformer structures can effectively handle complex segmentation tasks. The image semantic segmentation model of this invention is built on top of U-Net and includes an encoder, decoder, and skip connections. The encoder and decoder base blocks are two residual structures to prevent loss of detail due to network deepening. Specifically, this structure has two consecutive 3x3 convolutions to increase the feature channels. Subsequently, 1x1 convolutions merge information from the residual connections into the convolutional layer. Compared to U-Net, the image semantic segmentation model focuses on designing skip connections and bottlenecks.

[0071] The computational complexity of the Transformer exhibits a quadratic exponential relationship with image size; therefore, this work only modifies the skip connections in the third layer (feature map: 56×56) and the fourth layer (feature table: 28×28). The cross-filtering transformer structure addresses the shortcomings of simple skip connections in fusing low-level and high-level features, thereby improving model performance. Furthermore, the cross-filtering transformer structure is a dual-input module used to combine feature information from the current layer and the previous layer in the encoder. The main advantage of this structure lies in the redesigned Transformer architecture to facilitate cross-level feature fusion and capture global-local information. This structure not only enhances skip features but also establishes the model's ability to model long-range dependencies.

[0072] like Figure 3 As shown, an image semantic segmentation method includes the following steps:

[0073] (a) Select an image dataset with the same characteristics as the image to be processed, perform batch data preprocessing, adjust the window width and window level of the image data, and crop all images to 224×224 size;

[0074] (b) The image dataset obtained in step (a) is randomly divided into a training set, a validation set and a test set. The training set is used to train the model. The trained model is placed on the validation set for evaluation and the optimal weights are selected. After training, the model is tested on the test set.

[0075] (c) Perform data augmentation on the divided training, validation and test sets, including horizontal flipping, vertical flipping and random rotation;

[0076] (d) The weighted BCE_Dice loss function is used. BCE loss is the Binary CrossEntropy Loss, and Dice loss is the segmentation loss. The two loss functions are combined in a weighted manner, with a weight of 0.5. The total loss is measured from the perspective of pixels and the whole. The formula is as follows:

[0077] LOSS = W1 × L Dice +W2×L BCE (9)

[0078] Where W1 represents the weights corresponding to the Dice loss, and W2 represents the weights corresponding to the BCE loss, the expression for the Dice loss is as follows:

[0079]

[0080] Where n represents the number of pixels in the image, x i y represents the true value of the i-th pixel. i Let represent the predicted value of the i-th pixel, and t represent a constant. The expression for the BCE loss is as follows:

[0081]

[0082] Where n represents the number of pixels in the image, x i y represents the true value of the i-th pixel. i This represents the predicted value of the i-th pixel;

[0083] (e) Obtain the image semantic segmentation model described above;

[0084] (f) Based on the BCE_Dice loss function in step (d), train the image semantic segmentation model. After the image semantic segmentation model is trained, use the image semantic segmentation model to complete the image semantic segmentation.

[0085] After image semantic segmentation is completed, DSC (Dice Similarity Coffiliation) and IoU (Intersection over Union) can be used as segmentation metrics to evaluate the similarity between two sets. The values ​​range from [0,1]. The larger the value, the more similar the two sets are. The calculation expression is as follows:

[0086]

[0087] Where X pre This represents the segmentation area predicted by the model.X mask ∩ represents the area of ​​the segmentation of the real labels, ∪ represents the intersection, and ∪ represents the union.

[0088] This implementation uses lesion segmentation in CT images of non-small cell lung cancer as an example. A total of 5120 CT images were used in the experiment, divided into training, validation, and test sets in an 8:1:1 ratio. The algorithm performance was evaluated using the Dice coefficient (DSC), intersection-over-union ratio (IoU), number of parameters (#Param), and floating-point operations per second (FLOPs). The image semantic segmentation model (FilU-Net network model) of this invention was compared with various algorithms; the results are shown in Table 1. The segmentation mask is as follows: Figure 4 As shown in the figure. Experimental results show that the FilU-Net network model in this embodiment has the best segmentation performance.

[0089]

[0090] Table 1

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described with reference to preferred embodiments, those skilled in the art should understand that various changes in form and detail can be made without departing from the spirit and scope of the invention as defined in the appended claims.

Claims

1. A structure for a cross-filtering transformer, characterized in that: It includes feature embedding, multi-head cross attention, and feedforward neural networks. Feature embedding means converting the input image data into a representation in a vector space, enabling the model to process the semantic information between the data. Multi-head cross attention is a cross attention mechanism used in transformers to handle local and global correlations between data. Feedforward neural networks are a basic component of transformers that help the model learn the non-linear features of the data. Obtain the feature map F1 of the current layer and the feature map F2 of the previous layer. Feature map F1 has a size of H and a channel of C, and feature map F2 has a size of H*2 and a channel of C / 2. Feature maps F1 and F2 are the inputs. The feature maps F1 and F2 of the two branches are converted into low-dimensional feature vectors through 2D convolutional layers. Then, feature maps F1 and F2 are flattened into 1D sequences respectively. The feature embedding is represented as: S i =Flatten(Conv(F i ))+Pos i (1) X i =LN(S i ) (2) Where i = 1, 2, Conv() is the convolution operation, Pos i It is F i The corresponding positional encoding, LN() is LayerNormalization, X i The input sequence for MHCA is divided into multiple subsequences. Attention is then calculated for each subsequence in a different subspace. The output of each attention is then used as a head, and all heads are concatenated together. MHCA O (X1,X2)=Linear(Concat(Head1,...,Head h )) (4) Where h represents the number of headers, Head represents the header, Q, K, and V represent the query, key, and value, respectively, and d k The subspace dimension is represented by `Concat`, the join operation by `Linear`, and `MHCA` by `Linear`. O This is the output of MHCA. In a feedforward neural network, 2D convolutions are used to replace the connection layers. The overall structure is expressed as follows: CFTrans=FNN(LN(MHCA O +S1))+(MHCA O +S1) (5)。 2. The structure of a cross-filtering transformer according to claim 1, characterized in that: Q, K, and V can all be obtained using the following methods: First, perform a Fast Fourier Transform on the input sequence, as shown in the following mathematical expression: T (b,u,v,c) =F(X) (6) Where X∈R b×w×h×c b, h, w, and c represent the batch size, width, height, and number of channels, respectively, and (u, v) represents the position in the frequency domain. Then, a global filter W is applied and multiplied by T, where W ∈ R. b×(w / / 2+1)×(h / / 2+1)×c The mathematical expression is as follows: T = T * W (7) Finally, the spatial features of the image are recovered from the Fourier space, and the mathematical expression is as follows: X (b,w,h,c) =F -1 (T) (8) Where (w, h) represents the position in the spatial domain.

3. An image semantic segmentation model, characterized in that: The structure includes the U-Net image segmentation model and the cross-filtering transformer of any one of claims 1 and 2; the cross-filtering transformer structure is used between the encoder and decoder in the U-Net structure to reduce the semantic gap between the decoders.

4. An image semantic segmentation method, characterized in that: Includes the following steps: (a) Select an image dataset with the same characteristics as the image to be processed and perform batch data preprocessing, adjusting the window width and window level of the image data; (b) The image dataset obtained in step (a) is randomly divided into a training set, a validation set and a test set. The training set is used to train the model. The trained model is placed on the validation set for evaluation and the optimal weights are selected. After training, the model is tested on the test set. (c) Perform data augmentation on the divided training, validation and test sets, including horizontal flipping, vertical flipping and random rotation; (d) The weighted BCE_Dice loss function is used. BCE loss is the Binary CrossEntropy Loss, and Dice loss is the segmentation loss. The two loss functions are combined in a weighted manner, with a weight of 0.

5. The total loss is measured from the perspective of pixels and the whole. The formula is as follows: LOSS=W1×L Dice +W2×L BCE (9) Where W1 represents the weights corresponding to the Dice loss, and W2 represents the weights corresponding to the BCE loss, the expression for the Dice loss is as follows: Where n represents the number of pixels in the image, x i y represents the true value of the i-th pixel. i Let represent the predicted value of the i-th pixel, and t represent a constant. The expression for the BCE loss is as follows: Where n represents the number of pixels in the image, x i y represents the true value of the i-th pixel. i This represents the predicted value of the i-th pixel; (e) Obtain the image semantic segmentation model of claim 3; (f) Based on the BCE_Dice loss function in step (d), the image semantic segmentation model is trained. After the image semantic segmentation model is trained, the image semantic segmentation model is used to complete the image semantic segmentation.

Images