Medical image segmentation method and device based on hierarchical multi-scale CNN-Transform

Abstract

The invention discloses a medical image segmentation method and device based on hierarchical multi-scale CNN-Transform, and relates to the field of deep learning, a medical image segmentation model based on CNN-Transform is constructed and trained, a trained medical image segmentation model is obtained, a to-be-segmented medical image is obtained and input to the trained medical image segmentation model, and the to-be-segmented medical image is obtained. Inputting a to-be-segmented medical image into the stem layer for block segmentation to obtain a block image, and inputting the block image into a coding module to obtain a coding feature of a first stage and a coding feature of a third stage; inputting the coding feature of the first stage and the coding feature of the third stage into a double-layer feature fusion module to respectively obtain a calibrated coding feature of the first stage and a calibrated coding feature of the third stage; and inputting the calibrated coding characteristics of the first stage and the calibrated coding characteristics of the third stage into a decoding unit to obtain a prediction segmentation result. According to the method, the modeling capability of the model on the multi-scale long-range dependency relationship and the local space information can be improved.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a medical image segmentation method and device based on hierarchical multi-scale CNN-Transformer. Background Art

[0002] Deep learning technology is widely used in the field of medical imaging. Medical image segmentation is one of the key tasks, which provides the basis for the processing and analysis of medical images. By accurately segmenting the region of interest, many applications can be achieved, including but not limited to tissue volume quantification, disease diagnosis, regional localization, anatomical structure research, radiotherapy planning, and computer-assisted surgery.

[0003] Current medical image segmentation methods mainly rely on convolutional neural networks (CNNs) and Transformers, but the former lacks global features and the latter has limited local feature representation. Although the hybrid method based on CNN-Transformer attempts to fuse the two, it fails to ensure feature consistency and effectively fuse multi-scale information. Convolutional neural networks are good at extracting local spatial information of images, however, due to the limitations of their receptive field, they perform relatively poorly in modeling global information. In contrast, the Transformer model is good at capturing long-range dependencies, but may be lacking in processing local detail information. For the hybrid model, it mainly focuses on the tokenized self-attention mechanism, but fails to fully consider the importance of channel attention. In addition, it is also insufficient in effectively combining low-level and high-level features. Summary of the invention

[0004] The purpose of this application is to propose a medical image segmentation method and device based on hierarchical multi-scale CNN-Transformer to address the above-mentioned technical problems.

[0005] In a first aspect, the present invention provides a medical image segmentation method based on a hierarchical multi-scale CNN-Transformer, comprising the following steps:

[0006] A CNN-Transformer-based medical image segmentation model is constructed and trained to obtain a trained medical image segmentation model, wherein the trained medical image segmentation model includes a coding unit and a decoding unit, wherein the coding unit includes a stem layer, a coding module, and a double-layer feature fusion module, wherein the coding module includes a three-stage hybrid Transformer module and a block merging layer arranged between two adjacent stages of the hybrid Transformer modules, wherein the hybrid Transformer module includes a multi-scale channel attention layer, a CNN block, and a Transformer layer, wherein the Transformer layer includes a reshaping layer, a SwinTransformer module, and a reverse reshaping layer connected in sequence;

[0007] A medical image to be segmented is obtained and input into a trained medical image segmentation model, the medical image to be segmented is input into the stem layer for block segmentation to obtain a block image, the block image is input into the encoding module, in the hybrid Transformer module of the i-th stage in the encoding module, the input features of the hybrid Transformer module of the i-th stage are respectively input into the multi-scale channel attention layer and the CNN block in the hybrid Transformer module of the i-th stage, and the initial multi-scale features of the i-th stage and the local features of the i-th stage are obtained, and the initial multi-scale features of the i-th stage and the local features of the i-th stage are input into the i-th stage In the Transformer layer in the hybrid Transformer module, the coded features of the i-th stage are obtained; the coded features of the i-th stage are input into the hybrid Transformer module of the i+1-th stage after passing through the block merging layer, and finally the coded features of the first stage and the coded features of the third stage are obtained; the coded features of the first stage and the coded features of the third stage are input into the double-layer feature fusion module to obtain the calibrated coded features of the first stage and the calibrated coded features of the third stage respectively; the calibrated coded features of the first stage and the calibrated coded features of the third stage are input into the decoding unit to obtain the predicted segmentation result.

[0008] Preferably, the multi-scale channel attention layer includes a multi-scale feature extraction layer and a channel attention layer. The calculation process of the multi-scale channel attention layer in the hybrid Transformer module of the i-th stage is as follows:

[0009] ;

[0010] ;

[0011] in, represents the multi-scale feature extraction layer in the multi-scale channel attention layer in the hybrid Transformer module of the i-th stage, whose convolution kernel size is q×q, represents the output features of the multi-scale feature extraction layer in the multi-scale channel attention layer in the hybrid Transformer module of the i-th stage, represents the input features of the hybrid Transformer module at stage i, represents the initial multi-scale features of the i-th stage, Indicates a cascade operation;

[0012] The channel attention layer includes a global average pooling layer, a fast one-dimensional convolution layer, and a Sigmoid activation function layer. The initial multi-scale features of the i-th stage are input into the channel attention layer, and first pass through the global average pooling layer, the fast one-dimensional convolution layer, and the Sigmoid activation function layer in sequence to obtain the channel weight, and then multiply the channel weight with the initial multi-scale features of the i-th stage to obtain the final multi-scale features of the i-th stage, as shown in the following formula:

[0013] ;

[0014] in, represents the global average pooling layer, represents a fast one-dimensional convolutional layer, represents the Sigmoid activation function layer, represents the final multi-scale features of the i-th stage;

[0015] The CNN blocks in the three-stage hybrid Transformer module respectively use the second, third, and fourth convolution groups in the traditional ResNet-50 structure; the calculation process of the CNN block in the hybrid Transformer module in the i-th stage is as follows:

[0016] ;

[0017] in, represents the CNN block in the hybrid Transformer module of the i-th stage, represents the local features of the i-th stage;

[0018] In the Transformer layer in the hybrid Transformer module of the i-th stage, the initial multi-scale features of the i-th stage and the local features of the i-th stage are first added and then input into the reshaping layer to obtain the reshaped features, as shown in the following formula:

[0019] ;

[0020] in, It means reshaping the two-dimensional feature map into a one-dimensional token. Indicates remodeling characteristics;

[0021] The reshaped features are then passed through the Swin Transformer module and the reverse reshaping layer in turn to obtain the encoding features of the i-th stage, as shown in the following formula:

[0022] ;

[0023] in, represents the Swin Transformer module, It means reshaping the one-dimensional token into a two-dimensional feature map. Represents the encoding features of the i-th stage.

[0024] Preferably, the Swin Transformer module includes a plurality of Swin Transformer blocks stacked in sequence, the number of Swin Transformer blocks of the Swin Transformer module in the Transformer layer in the three-stage hybrid Transformer module is 2, 2 and 6 respectively, each Swin Transformer block includes a first encoder block and a second encoder block, the first encoder block and the second encoder block are obtained by replacing the multi-head self-attention layer in the encoder block of ViT with a window multi-head self-attention layer and a sliding window multi-head self-attention layer respectively, and the calculation process of the Swin Transformer block is as follows:

[0025] ;

[0026] ;

[0027] ;

[0028] ;

[0029] in, represents the input features of the Swin Transformer block, Representation layer normalization layer, represents the window multi-head self-attention layer, represents a multi-layer perceptron, represents the sliding window multi-head self-attention layer, represents the intermediate features of the first encoder block, represents the output features of the first encoder block, represents the intermediate features of the second encoder block, Represents the output features of the Swin Transformer block.

[0030] Preferably, the double-layer feature fusion module includes a first hierarchical structure, a second hierarchical structure and a cross-attention module, the coding features of the first stage pass through the first hierarchical structure to obtain the output features of the first hierarchical structure, the coding features of the third stage pass through the second hierarchical structure to obtain the output features of the second hierarchical structure, the output features of the first hierarchical structure and the output features of the second hierarchical structure are input into the cross-attention module to obtain the calibrated coding features of the first stage and the calibrated coding features of the third stage respectively;

[0031] Both the first-level structure and the second-level structure include a normalization layer, a global average pooling layer, and a Transformer encoder;

[0032] The encoding features of the first stage and the encoding features of the third stage are respectively passed through the normalization layer and the global average pooling layer to derive the corresponding class tokens, as shown in the following formula:

[0033] ;

[0034] ;

[0035] in, represents the encoding features of the first stage, represents the encoding features of the third stage, represents the normalization layer, represents the global average pooling layer, Indicates the class token corresponding to the encoding feature of the first stage, Indicates the class token corresponding to the encoding feature of the third stage;

[0036] The number of Transformer encoders in the first hierarchical structure is 2, and the number of Transformer encoders in the second hierarchical structure is 1. The class token corresponding to the encoding feature of the first stage passes through the two Transformer encoders in the first hierarchical structure to obtain the processed class token corresponding to the encoding feature of the first stage; the class token corresponding to the encoding feature of the third stage passes through the one Transformer encoder in the second hierarchical structure to obtain the processed class token corresponding to the encoding feature of the third stage;

[0037] The dimensionally adjusted features of the processed class tokens corresponding to the encoded features of the first stage are connected with the encoded features of the third stage as the key and value, and the dimensionally adjusted features of the processed class tokens corresponding to the encoded features of the first stage are used to perform the query to obtain the calibrated encoded features of the first stage, as shown in the following formula:

[0038] ;

[0039] ;

[0040] in, represents the first fusion feature, Indicates the processed class token corresponding to the encoding features of the first stage, represents the cross attention layer, Indicates that The operation of projecting to the dimension of the encoded features of the third stage, LN represents the layer normalization layer, Indicates that The operation of projecting to the dimension of the encoded features in stage 1, Indicates a connection operation. represents the coded features of the first stage after calibration;

[0041] The dimensionally adjusted features of the processed class tokens corresponding to the encoded features of the third stage are connected with the encoded features of the first stage as the key and value, and the dimensionally adjusted features of the processed class tokens corresponding to the encoded features of the third stage are used to perform the query to obtain the calibrated encoded features of the third stage, as shown in the following formula:

[0042] ;

[0043] ;

[0044] in, represents the second fusion feature, Indicates the processed class token corresponding to the encoding feature of the third stage, represents the cross attention layer, Indicates that The operation of projecting to the dimension of the encoded features of the first stage, LN represents the layer normalization layer, Indicates that Projection to the dimension of the encoded features in stage 3, Represents the coded features of the calibrated stage 3.

[0045] Preferably, the decoding unit includes a first upsampling convolution module, a convolution module, a second upsampling convolution module and a segmentation head; the first upsampling convolution module and the second upsampling convolution module both adopt an upsampling convolution structure, the upsampling convolution structure includes a convolution layer with a convolution kernel size of 3×3, two bilinear upsampling layers, a group normalization layer and a ReLU activation function layer, the convolution module includes a 3×3 convolution layer, a group normalization layer and a ReLU activation function layer, and the segmentation head adopts a convolution layer with a convolution kernel size of 3×3; in the decoding unit, the calibrated first stage encoding features and the calibrated third stage encoding features are respectively input into the first upsampling convolution module and the convolution module to obtain the first decoding features and the second decoding features, and the first decoding features and the second decoding features are added and sequentially passed through the second upsampling convolution module and the segmentation head to obtain the predicted segmentation result.

[0046] Preferably, the loss function used in the training process of the medical image segmentation model is for:

[0047] ;

[0048] in, represents the cross entropy loss, represents the Dice loss, represents the weight coefficient, and They represent the true segmentation results and the predicted segmentation results respectively.

[0049] In a second aspect, the present invention provides a medical image segmentation device based on a hierarchical multi-scale CNN-Transformer, comprising:

[0050] A model building module is configured to build and train a medical image segmentation model based on CNN-Transformer to obtain a trained medical image segmentation model, wherein the trained medical image segmentation model includes an encoding unit and a decoding unit, wherein the encoding unit includes a stem layer, an encoding module, and a double-layer feature fusion module, wherein the encoding module includes a three-stage hybrid Transformer module and a block merging layer arranged between the hybrid Transformer modules of two adjacent stages, wherein the hybrid Transformer module includes a multi-scale channel attention layer, a CNN block, and a Transformer layer, wherein the Transformer layer includes a reshaping layer, a Swin Transformer module, and a reverse reshaping layer connected in sequence;

[0051] The segmentation module is configured to obtain a medical image to be segmented and input it into a trained medical image segmentation model, input the medical image to be segmented into a stem layer for block segmentation to obtain a block image, and input the block image into an encoding module. In the hybrid Transformer module of the i-th stage in the encoding module, the input features of the hybrid Transformer module of the i-th stage are respectively input into the multi-scale channel attention layer and the CNN block in the hybrid Transformer module of the i-th stage to obtain the initial multi-scale features of the i-th stage and the local features of the i-th stage, and the initial multi-scale features of the i-th stage and the local features of the i-th stage are input into the i-th stage. The encoding features of the i-th stage are obtained in the Transformer layer in the hybrid Transformer module of the i-th stage; the encoding features of the i-th stage are input into the hybrid Transformer module of the i+1-th stage after passing through the block merging layer, and finally the encoding features of the first stage and the encoding features of the third stage are obtained; the encoding features of the first stage and the encoding features of the third stage are input into the double-layer feature fusion module to obtain the calibrated encoding features of the first stage and the calibrated encoding features of the third stage respectively; the calibrated encoding features of the first stage and the calibrated encoding features of the third stage are input into the decoding unit to obtain the predicted segmentation result.

[0052] In a third aspect, the present invention provides an electronic device comprising one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0053] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0054] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] (1) The medical image segmentation method based on hierarchical multi-scale CNN-Transformer proposed in the present invention adopts a medical image segmentation model based on CNN-Transformer, which mainly includes a coding unit and a decoding unit. In the hybrid Transformer module in the coding unit, the local features of the CNN block and the long-range modeling advantages of the Transformer layer are combined, and a multi-scale channel attention layer is introduced to capture multi-scale features using channel attention to mine more comprehensive feature maps, thereby solving the problems of CNN lacking global features, Transformer lacking local features, and difficulty in fusing multi-scale features.

[0057] (2) The medical image segmentation method based on hierarchical multi-scale CNN-Transformer proposed in the present invention utilizes the cross-attention mechanism in the double-layer feature fusion module in the encoding unit to more finely fuse low-level and high-level features, establish effective feature fusion between coarse-grained and fine-grained feature representations, and enhance the model's ability to model multi-scale long-range dependencies and local spatial information.

[0058] (3) The medical image segmentation method based on hierarchical multi-scale CNN-Transformer proposed in this paper can simultaneously fuse multi-scale information on the basis of effectively modeling local information and long-range relationships to maintain the feature richness and consistency of the 2D medical image segmentation task. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0060] Figure 1 A schematic diagram of a process of a medical image segmentation method based on a hierarchical multi-scale CNN-Transformer according to an embodiment of the present application;

[0061] Figure 2 A schematic diagram of a medical image segmentation model of a medical image segmentation method based on hierarchical multi-scale CNN-Transformer according to an embodiment of the present application;

[0062] Figure 3 A schematic diagram of a hybrid Transformer module of a medical image segmentation method based on hierarchical multi-scale CNN-Transformer according to an embodiment of the present application;

[0063] Figure 4A schematic diagram of a multi-scale channel attention layer of a medical image segmentation method based on a hierarchical multi-scale CNN-Transformer according to an embodiment of the present application;

[0064] Figure 5 A schematic diagram of a cross-attention module of a medical image segmentation method based on a hierarchical multi-scale CNN-Transformer according to an embodiment of the present application;

[0065] Figure 6 A schematic diagram of a medical image segmentation device based on a hierarchical multi-scale CNN-Transformer according to an embodiment of the present application;

[0066] Figure 7 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0068] Figure 1 A medical image segmentation method based on a hierarchical multi-scale CNN-Transformer provided in an embodiment of the present application is shown, comprising the following steps:

[0069] S1, construct and train a medical image segmentation model based on CNN-Transformer to obtain a trained medical image segmentation model, the trained medical image segmentation model includes a coding unit and a decoding unit, the coding unit includes a stem layer, a coding module and a double-layer feature fusion module, the coding module includes a three-stage hybrid Transformer module and a block merging layer arranged between the hybrid Transformer modules of two adjacent stages, the hybrid Transformer module includes a multi-scale channel attention layer, a CNN block and a Transformer layer, and the Transformer layer includes a reshaping layer, a SwinTransformer module and a reverse reshaping layer connected in sequence.

[0070] Specifically, the embodiment of the present application proposes a medical image segmentation model based on the CNN-Transformer architecture, which effectively combines global and local information to maintain the feature richness and consistency of the 2D medical image segmentation task. The medical image segmentation model is called HmHFormer. The medical image segmentation model mainly includes an encoding unit and a decoding unit, which cleverly combines the advantages of convolutional neural networks in local feature representation and multi-scale channel attention, and the strengths of Transformer in capturing global long-range dependencies, effectively aggregating feature extraction capabilities from different aspects, thereby achieving accurate medical image segmentation tasks. The encoding unit mainly includes a stem layer for extracting initial local features, a three-stage hybrid Transformer module, and a double-layer feature fusion (DLF) module. First, the medical image is input into the Stem layer for patch segmentation, and the result is fed into the three-stage encoding module to generate a hierarchical feature map. Secondly, the encoded features output from the first and third stages are input into the DLF module. The DLF module is a multi-scale visual converter that uses a cross-attention mechanism to fuse two encoded features to obtain two calibrated encoded features. Finally, the two calibrated encoded features are passed into the decoding unit to produce the final predicted segmentation result.

[0071] refer to Figure 2 , given an input image ,in Represent the height, width and number of channels respectively, represents a set of real numbers. First, use the Stem layer to perform patch segmentation. The Stem layer consists of a convolution kernel with a step size of 1 and a size of The convolution layer and stride 2, the pooling kernel size is The segmented image is divided into patches, represented by , the size of each patch is , , Indicates Then, a trainable linear projection is used to transform Embed in image tag . The embedded image tag It is fed into a three-stage encoding module to produce hierarchical features. The spatial resolution of the encoded features output by each stage is 1 / 2 of the original input resolution. . Use a step size of 2 and a convolution kernel size of The patch merging layer is implemented by a convolutional layer to expand the dimension between two adjacent stages. Each stage consists of a hybrid Transformer module that can capture multi-scale features, local spatial information, and long-range features through feature selection. , Represent the coding features of the first stage and the third stage respectively. , Input into the DLF module to recalibrate the encoding features. Finally, the decoding unit is used to decode and obtain the predicted segmentation result. , Indicates the number of categories.

[0072] In a specific embodiment, the loss function used in the training process of the medical image segmentation model is for:

[0073] ;

[0074] in, represents the cross entropy loss, represents the Dice loss, represents the weight coefficient, and They represent the true segmentation results and the predicted segmentation results respectively.

[0075] Specifically, the above loss function is used to supervise the training of the medical image segmentation model to obtain a trained medical image segmentation model. During the training process, the relative contribution of the cross entropy loss and the Dice loss in the loss function is controlled by the weight coefficient, and usually the sum of the weights of the two losses is 1. Combining the two losses can effectively alleviate the problem of class imbalance.

[0076] S2, obtain the medical image to be segmented and input it into the trained medical image segmentation model, input the medical image to be segmented into the stem layer for block segmentation to obtain the block image, input the block image into the encoding module, in the hybrid Transformer module of the i-th stage in the encoding module, the input features of the hybrid Transformer module of the i-th stage are respectively input into the multi-scale channel attention layer and the CNN block in the hybrid Transformer module of the i-th stage, and the initial multi-scale features of the i-th stage and the local features of the i-th stage are obtained, and the initial multi-scale features of the i-th stage and the local features of the i-th stage are input into the i-th stage The coding features of the i-th stage are input into the Transformer layer in the hybrid Transformer module of the ith stage; the coding features of the i-th stage are input into the hybrid Transformer module of the i+1-th stage after the block merging layer, and finally the coding features of the first stage and the coding features of the third stage are obtained; the coding features of the first stage and the coding features of the third stage are input into the double-layer feature fusion module to obtain the calibrated coding features of the first stage and the calibrated coding features of the third stage respectively; the calibrated coding features of the first stage and the calibrated coding features of the third stage are input into the decoding unit to obtain the predicted segmentation result.

[0077] In a specific embodiment, the multi-scale channel attention layer includes a multi-scale feature extraction layer and a channel attention layer. The calculation process of the multi-scale channel attention layer in the hybrid Transformer module of the i-th stage is as follows:

[0078] ;

[0079] ;

[0080] in, represents the multi-scale feature extraction layer in the multi-scale channel attention layer in the hybrid Transformer module of the i-th stage, whose convolution kernel size is q×q, represents the output features of the multi-scale feature extraction layer in the multi-scale channel attention layer in the hybrid Transformer module of the i-th stage, represents the input features of the hybrid Transformer module at stage i, represents the initial multi-scale features of the i-th stage, Indicates a cascade operation;

[0081] The channel attention layer includes a global average pooling layer, a fast one-dimensional convolution layer, and a Sigmoid activation function layer. The initial multi-scale features of the i-th stage are input into the channel attention layer, and first pass through the global average pooling layer, the fast one-dimensional convolution layer, and the Sigmoid activation function layer in sequence to obtain the channel weight, and then multiply the channel weight with the initial multi-scale features of the i-th stage to obtain the final multi-scale features of the i-th stage, as shown in the following formula:

[0082] ;

[0083] in, represents the global average pooling layer, represents a fast one-dimensional convolutional layer, represents the Sigmoid activation function layer, represents the final multi-scale features of the i-th stage;

[0084] The CNN blocks in the three-stage hybrid Transformer module respectively use the second, third, and fourth convolution groups in the traditional ResNet-50 structure; the calculation process of the CNN block in the hybrid Transformer module in the i-th stage is as follows:

[0085] ;

[0086] in, represents the CNN block in the hybrid Transformer module of the i-th stage, represents the local features of the i-th stage;

[0087] In the Transformer layer in the hybrid Transformer module of the i-th stage, the initial multi-scale features of the i-th stage and the local features of the i-th stage are first added and then input into the reshaping layer to obtain the reshaped features, as shown in the following formula:

[0088] ;

[0089] in, It means reshaping the two-dimensional feature map into a one-dimensional token. Indicates remodeling characteristics;

[0090] The reshaped features are then passed through the Swin Transformer module and the reverse reshaping layer in turn to obtain the encoding features of the i-th stage, as shown in the following formula:

[0091] ;

[0092] in, represents the Swin Transformer module, It means reshaping the one-dimensional token into a two-dimensional feature map. Represents the encoding features of the i-th stage.

[0093] In a specific embodiment, the Swin Transformer module includes a plurality of Swin Transformer blocks stacked in sequence, the number of Swin Transformer blocks of the Swin Transformer module in the Transformer layer in the three-stage hybrid Transformer module is 2, 2, and 6, respectively, each Swin Transformer block includes a first encoder block and a second encoder block, the first encoder block and the second encoder block are obtained by replacing the multi-head self-attention layer in the encoder block of ViT with a window multi-head self-attention layer and a sliding window multi-head self-attention layer, respectively, and the calculation process of the Swin Transformer block is as follows:

[0094] ;

[0095] ;

[0096] ;

[0097] ;

[0098] in, represents the input features of the Swin Transformer block, Representation layer normalization layer, represents the window multi-head self-attention layer, represents a multi-layer perceptron, represents the sliding window multi-head self-attention layer, represents the intermediate features of the first encoder block, represents the output features of the first encoder block, represents the intermediate features of the second encoder block, Represents the output features of the Swin Transformer block.

[0099] Specifically, with the advent of the era of deep learning, convolutional neural networks capture and collect local features in a hierarchical manner through convolution operations, and preserve these features in the form of feature maps. The visual transformer adopts another strategy. It aggregates compressed patch embeddings in a more flexible and dynamic way through cascaded self-attention modules to construct a global representation. These two methods complement each other and jointly promote the vigorous development of the field of computer vision. Due to the inherent locality of convolution, long-range dependencies cannot be modeled. The Transformer only generates single-scale features with self-attention, and it ignores the relationship between channels, so it cannot handle situations such as multi-scale lesion areas in medical images. Taking these issues into account, the embodiment of the present application introduces a hybrid Transformer module in the feature extraction process, which consists of a multi-scale channel attention (MSCA) layer, a CNN block and a Transformer layer, such as Figure 3 As shown, the input feature of the hybrid Transformer module in the i-th stage is the block image or the merged image of the encoded features in the i-1-th stage after the block merging layer.

[0100] Multi-scale features and channel attention are very important for the segmentation of medical images of different shapes and scales, so a multi-scale channel attention layer is introduced, such as Figure 4 As shown in Figure 2, channel attention is used to capture multi-scale feature dependencies, complementing the self-attention in Transformer.

[0101] Convolutional layers with different convolution kernel sizes and a step size of 1 are used to extract multi-scale features, and the extracted features are cascaded to form the initial multi-scale features. The initial multi-scale features of the stage are expressed as , and then channel attention is used to calibrate the initial multi-scale features. Among them, the global average pooling (GAP) layer is used to aggregate features from different sizes. Conv1d is a fast one-dimensional convolution layer with a convolution kernel size of 3, which is used to obtain channel relationships. Represents the Sigmoid activation function layer, which is used to obtain the weight indicating the importance of each channel. Then, the initial multi-scale features are calibrated by multiplying the channel weights and the initial multi-scale features to obtain the final multi-scale features. In the embodiment of the present application, a large-scale convolution kernel is used in low dimensions, and a small-scale convolution kernel is used in high dimensions to reduce the computational complexity of MSCA. For the low-dimensional high-resolution feature map of the first stage, four scales are used, namely For the low-resolution feature maps of the remaining two stages, use Two scales.

[0102] In order to better encode local spatial information, CNN blocks are used to extract local spatial features. Specifically, given the input feature , using CNN blocks to model local features. The embodiments of this application focus on the macroscopic combination of convolutional neural networks and Transformer, rather than the microscopic network structure design, so the existing convolutional network structure is used. The CNN blocks in the three stages are different from those in the original ResNet-50 architecture. Be consistent.

[0103] The encoder block of ViT mainly consists of a multi-head self-attention mechanism (MSA) and a multi-layer perceptron (MLP). ViT includes N identical encoder blocks. In each block, a layer normalization (LN) layer is applied before the MSA and MLP. In addition, a copy of the activation is added to the output of the MSA or MLP through a skip connection. Due to the fully connected attention mechanism, ViT has a high computational complexity, requires more computing resources, and may perform relatively poorly on small-scale datasets, so it is less efficient in high-resolution computer vision tasks such as image segmentation.

[0104] In order to solve this problem, the embodiment of the present application replaces MSA with a window multi-head self-attention (W-MSA) layer and a sliding window multi-head self-attention (SW-MSA) layer containing windows of different sizes in the SwinTransformer block of the Transformer layer. Through the MSA of the window and the sliding window, the SwinTransformer block improves the operation efficiency and ultimately improves the accuracy of the classification. In the SwinTransformer block, the MSA is divided into multiple fixed windows to form the W-MSA. The characteristic of W-MSA is that the self-attention mechanism is applied to the local fixed window, thereby greatly reducing the amount of calculation and improving the operation efficiency of the network. However, since the information cannot be exchanged between the windows, its receptive field is reduced, and it is impossible to obtain more globally accurate information, thereby affecting the accuracy of the network. In order to realize the interaction between different windows, the SwinTransformer block introduces SW-MSA. The principle of SW-MSA is to slide the window, offset the window to contain different pixels, and then perform W-MSA calculations, connect the results of the two W-MSA calculations, and combine the information contained in the pixels in the two different windows to realize the information interaction between different windows. Through the SW-MSA mechanism, the MSA calculation of the pixels in the offset window is completed, and the information exchange of pixels between different windows is realized. This further expands the "receptive field" of the network and improves the efficiency of information utilization.

[0105] In a specific embodiment, the double-layer feature fusion module includes a first hierarchical structure, a second hierarchical structure and a cross-attention module, the coding features of the first stage pass through the first hierarchical structure to obtain the output features of the first hierarchical structure, the coding features of the third stage pass through the second hierarchical structure to obtain the output features of the second hierarchical structure, the output features of the first hierarchical structure and the output features of the second hierarchical structure are input into the cross-attention module to obtain the calibrated coding features of the first stage and the calibrated coding features of the third stage respectively;

[0106] Both the first-level structure and the second-level structure include a normalization layer, a global average pooling layer, and a Transformer encoder;

[0107] The encoding features of the first stage and the encoding features of the third stage are respectively passed through the normalization layer and the global average pooling layer to derive the corresponding class tokens, as shown in the following formula:

[0108] ;

[0109] ;

[0110] in, represents the encoding features of the first stage, represents the encoding features of the third stage, represents the normalization layer, represents the global average pooling layer, Indicates the class token corresponding to the encoding feature of the first stage, Indicates the class token corresponding to the encoding feature of the third stage;

[0111] The number of Transformer encoders in the first hierarchical structure is 2, and the number of Transformer encoders in the second hierarchical structure is 1. The class token corresponding to the encoding feature of the first stage passes through the two Transformer encoders in the first hierarchical structure to obtain the processed class token corresponding to the encoding feature of the first stage; the class token corresponding to the encoding feature of the third stage passes through the one Transformer encoder in the second hierarchical structure to obtain the processed class token corresponding to the encoding feature of the third stage;

[0112] The dimensionally adjusted features of the processed class tokens corresponding to the encoded features of the first stage are connected with the encoded features of the third stage as the key and value, and the dimensionally adjusted features of the processed class tokens corresponding to the encoded features of the first stage are used to perform the query to obtain the calibrated encoded features of the first stage, as shown in the following formula:

[0113] ;

[0114] ;

[0115] in, represents the first fusion feature, Indicates the processed class token corresponding to the encoding features of the first stage, represents the cross attention layer, Indicates that The operation of projecting to the dimension of the encoded features of the third stage, LN represents the layer normalization layer, Indicates that The operation of projecting to the dimension of the encoded features in stage 1, Indicates a connection operation. represents the coded features of the first stage after calibration;

[0116] The dimensionally adjusted features of the processed class tokens corresponding to the encoded features of the third stage are connected with the encoded features of the first stage as the key and value, and the dimensionally adjusted features of the processed class tokens corresponding to the encoded features of the third stage are used to perform the query to obtain the calibrated encoded features of the third stage, as shown in the following formula:

[0117] ;

[0118] ;

[0119] in, represents the second fusion feature, Indicates the processed class token corresponding to the encoding feature of the third stage, represents the cross attention layer, Indicates that The operation of projecting to the dimension of the encoded features of the first stage, LN represents the layer normalization layer, Indicates that Projection to the dimension of the encoded features in stage 3, Represents the coded features of the calibrated stage 3.

[0120] Specifically, in order to effectively fuse multi-level features, a simple method is to directly sum the features of each level through the decoding unit and obtain the final segmentation map. However, this method cannot ensure the consistency of features between them, resulting in poor performance. In order to solve the above problems, an embodiment of the present application proposes a double-layer feature fusion (DLF) module, which takes the encoding features of the first stage and the encoding features of the third stage as input, and adopts a cross-attention mechanism to fuse information across scales.

[0121] Generally speaking, shallow layers have better localization information. As the number of layers increases, semantic information becomes more prevalent and more suitable for the decoding unit part. Faced with the dilemma of huge computational cost and the difficulty of mid-level feature maps in model accuracy, it is not considered to use mid-level feature maps in feature fusion to save computational cost. Therefore, it is encouraged to encode features in the first stage by multi-scaling and the encoding features of stage 3 to represent it while retaining the positioning information.

[0122] In the DLF module, the class token plays an important role because it summarizes all the information of the input features, so each level in the two-level structure is first assigned a class token derived from the global average pooling layer (GAP) layer on the norm of the level. The class token is then embedded into the associated level and passed to the Transformer encoder. The two stages are connected respectively. Transformer encoders, where S=2, L=1. After being processed by the Transformer encoder, the processed class tokens corresponding to the encoded features of the first stage are obtained respectively ( ) and the processed class token corresponding to the encoding features of the third stage ( ).

[0123] Finally, a cross-attention module is used to fuse the encoded features of each stage. Specifically, before fusion, the class tokens of the two stages are exchanged, which means that the class tokens of one stage are connected with the tokens of the other stage. Then, each new embedding is fused through the cross-attention module separately and finally returned to its own level. This interaction with tokens from other stages enables the class token to share rich information with it across stages. Figure 5 As shown, the processed class token corresponding to the encoding features of the first stage ( ) is used as an example. First, Project to The output is expressed as ,Right now . and Connection as key Sum In addition, Execute a query , combined with the query weight , key weight Sum weight Calculate attention. Since only class tokens are queried, the cross-attention mechanism runs in linear time and finally outputs the calibrated encoding features of the first stage. The processed class token corresponding to the encoding features of the third stage ( ) is processed in the same way as the encoded features in the first stage, and the processed class tokens ( ) is a process of mutual exchange of queries, which will not be elaborated here.

[0124] In a specific embodiment, the decoding unit includes a first upsampling convolution module, a convolution module, a second upsampling convolution module and a segmentation head; the first upsampling convolution module and the second upsampling convolution module both adopt an upsampling convolution structure, the upsampling convolution structure includes a convolution layer with a convolution kernel size of 3×3, two bilinear upsampling layers, a group normalization layer and a ReLU activation function layer, the convolution module includes a 3×3 convolution layer, a group normalization layer and a ReLU activation function layer, and the segmentation head adopts a convolution layer with a convolution kernel size of 3×3; in the decoding unit, the calibrated first stage encoding features and the calibrated third stage encoding features are respectively input into the first upsampling convolution module and the convolution module to obtain the first decoding features and the second decoding features, and the first decoding features and the second decoding features are added and sequentially passed through the second upsampling convolution module and the segmentation head to obtain the predicted segmentation result.

[0125] Specifically, the decoding unit in the embodiment of the present application first receives the calibrated first stage coding feature from the DLF module. and the coded features of the calibrated stage 3 , the coded features of the first stage after calibration The convolution kernel size is The upsampling convolution structure (ConvUp) consists of a convolution layer, two bilinear upsampling layers, a group normalization (GroupNorm) layer, and a ReLU activation function layer to obtain a resolution of The first decoded feature of . The encoded feature of the 3rd stage after calibration It is followed by a convolution module, which uses a convolution kernel size of The convolutional layer, group normalization (GroupNorm) layer and ReLU activation function layer are used to obtain a resolution of The sum of the first decoded feature and the second decoded feature is passed through another ConvUp to achieve a final size of The feature map obtained is passed through the convolution kernel size in the segmentation head. After the convolutional layer, the predicted segmentation result is generated.

[0126] Further references Figure 6 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a medical image segmentation device based on a hierarchical multi-scale CNN-Transformer. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0127] The embodiment of the present application provides a medical image segmentation device based on a hierarchical multi-scale CNN-Transformer, comprising:

[0128] A model building module 1 is configured to build and train a medical image segmentation model based on CNN-Transformer to obtain a trained medical image segmentation model, wherein the trained medical image segmentation model includes a coding unit and a decoding unit, wherein the coding unit includes a stem layer, a coding module, and a double-layer feature fusion module, wherein the coding module includes a three-stage hybrid Transformer module and a block merging layer arranged between the hybrid Transformer modules of two adjacent stages, wherein the hybrid Transformer module includes a multi-scale channel attention layer, a CNN block, and a Transformer layer, wherein the Transformer layer includes a reshaping layer, a Swin Transformer module, and a reverse reshaping layer connected in sequence;

[0129] The segmentation module 2 is configured to obtain the medical image to be segmented and input it into the trained medical image segmentation model, input the medical image to be segmented into the stem layer for block segmentation to obtain a block image, and input the block image into the encoding module. In the hybrid Transformer module of the i-th stage in the encoding module, the input features of the hybrid Transformer module of the i-th stage are respectively input into the multi-scale channel attention layer and the CNN block in the hybrid Transformer module of the i-th stage to obtain the initial multi-scale features of the i-th stage and the local features of the i-th stage, and the initial multi-scale features of the i-th stage and the local features of the i-th stage are input into The encoding features of the i-th stage are obtained in the Transformer layer in the hybrid Transformer module of the i-th stage; the encoding features of the i-th stage are input into the hybrid Transformer module of the i+1-th stage after passing through the block merging layer, and finally the encoding features of the first stage and the encoding features of the third stage are obtained; the encoding features of the first stage and the encoding features of the third stage are input into the double-layer feature fusion module to obtain the calibrated encoding features of the first stage and the calibrated encoding features of the third stage respectively; the calibrated encoding features of the first stage and the calibrated encoding features of the third stage are input into the decoding unit to obtain the predicted segmentation result.

[0130] Figure 7 Schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present invention. Figure 7As shown, the electronic device of this embodiment includes: a processor 701 and a memory 702; wherein the memory 702 is used to store computer-executable instructions; the processor 701 is used to execute the computer-executable instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0131] Optionally, the memory 702 may be independent or integrated with the processor 701 .

[0132] When the memory 702 is independently provided, the electronic device further includes a bus 703 for connecting the memory 702 and the processor 701 .

[0133] The embodiment of the present invention further provides a computer storage medium, in which computer execution instructions are stored. When the processor 701 executes the computer execution instructions, the above method is implemented.

[0134] The embodiment of the present invention further provides a computer program product, including a computer program. When the computer program is executed by the processor 701, the above method is implemented.

[0135] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0136] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of this embodiment.

[0137] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each module may exist physically separately, or two or more modules may be integrated into one unit. The unit formed by the above modules may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0138] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor 701 to perform some steps of the methods of various embodiments of the present application.

[0139] It should be understood that the processor 701 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or the processor 701 may be any conventional processor 701, etc. The steps of the method disclosed in the invention may be directly embodied in the execution of the hardware processor 701, or may be executed by a combination of hardware and software modules in the processor 701.

[0140] The memory 702 may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk, or an optical disk.

[0141] The bus 703 may be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 703 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus 703 in the drawings of the present application is not limited to only one bus 703 or one type of bus 703.

[0142] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0143] An exemplary storage medium is coupled to the processor 701, so that the processor 701 can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor 701. The processor 701 and the storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as ASIC). Of course, the processor 701 and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0144] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A medical image segmentation method based on hierarchical multi-scale CNN-Transformer, characterized in that: The following steps are involved: A CNN-Transformer-based medical image segmentation model is constructed and trained to obtain a trained medical image segmentation model, wherein the trained medical image segmentation model includes a coding unit and a decoding unit, wherein the coding unit includes a stem layer, a coding module, and a double-layer feature fusion module, wherein the coding module includes a three-stage hybrid Transformer module and a block merging layer arranged between two adjacent stages of the hybrid Transformer modules, wherein the hybrid Transformer module includes a multi-scale channel attention layer, a CNN block, and a Transformer layer, wherein the Transformer layer includes a reshaping layer, a Swin Transformer module, and a reverse reshaping layer connected in sequence; Acquire a medical image to be segmented and input it into the trained medical image segmentation model, input the medical image to be segmented into the stem layer for block segmentation to obtain a block image, input the block image into the encoding module, in the hybrid Transformer module of the i-th stage in the encoding module, input features of the hybrid Transformer module of the i-th stage into the multi-scale channel attention layer and the CNN block in the hybrid Transformer module of the i-th stage respectively, obtain the initial multi-scale features of the i-th stage and the local features of the i-th stage, the initial multi-scale features of the i-th stage and the local features of the i-th stage are input into the Transformer layer in the hybrid Transformer module of the i-th stage, and obtain the coding features of the i-th stage; The encoded features of the i-th stage are input into the hybrid Transformer module of the i+1-th stage after passing through the block merging layer, and finally the encoded features of the first stage and the encoded features of the third stage are obtained; Inputting the coding features of the first stage and the coding features of the third stage into the double-layer feature fusion module to obtain the calibrated coding features of the first stage and the calibrated coding features of the third stage respectively; The calibrated encoding features of the first stage and the calibrated encoding features of the third stage are input into the decoding unit to obtain a predicted segmentation result.

2. The medical image segmentation method based on hierarchical multi-scale CNN-Transformer according to claim 1, characterized in that: The multi-scale channel attention layer includes a multi-scale feature extraction layer and a channel attention layer. The calculation process of the multi-scale channel attention layer in the hybrid Transformer module of the i-th stage is as follows: ； ； in, represents the multi-scale feature extraction layer in the multi-scale channel attention layer in the hybrid Transformer module of the i-th stage, whose convolution kernel size is q×q, represents the output features of the multi-scale feature extraction layer in the multi-scale channel attention layer in the hybrid Transformer module of the i-th stage, represents the input features of the hybrid Transformer module at stage i, represents the initial multi-scale features of the i-th stage, Indicates a cascade operation; The channel attention layer includes a global average pooling layer, a fast one-dimensional convolution layer and a Sigmoid activation function layer. The initial multi-scale features of the i-th stage are input into the channel attention layer, and first pass through the global average pooling layer, the fast one-dimensional convolution layer and the Sigmoid activation function layer in sequence to obtain the channel weight, and then the channel weight is multiplied by the initial multi-scale features of the i-th stage to obtain the final multi-scale features of the i-th stage, as shown in the following formula: ； in, represents the global average pooling layer, represents a fast one-dimensional convolutional layer, represents the Sigmoid activation function layer, represents the final multi-scale features of the i-th stage; The CNN blocks in the hybrid Transformer modules of the three stages respectively use the second convolution group, the third convolution group and the fourth convolution group in the traditional ResNet-50 structure; the calculation process of the CNN block in the hybrid Transformer module of the i-th stage is as follows: ； in, represents the CNN block in the hybrid Transformer module of the i-th stage, represents the local features of the i-th stage; In the Transformer layer in the hybrid Transformer module of the i-th stage, the initial multi-scale features of the i-th stage and the local features of the i-th stage are first added and then input into the reshaping layer to obtain the reshaping features, as shown in the following formula: ； in, It means reshaping the two-dimensional feature map into a one-dimensional token. Indicates remodeling characteristics; The reshaped features then pass through the Swin Transformer module and the reverse reshaping layer in sequence to obtain the encoding features of the i-th stage, as shown in the following formula: ； in, represents the Swin Transformer module, It means reshaping the one-dimensional token into a two-dimensional feature map. Represents the encoding features of the i-th stage.

3. The medical image segmentation method based on hierarchical multi-scale CNN-Transformer according to claim 1, characterized in that: The Swin Transformer module includes a plurality of Swin Transformer blocks stacked in sequence, the number of Swin Transformer blocks of the Swin Transformer module in the Transformer layer in the three-stage hybrid Transformer module is 2, 2 and 6 respectively, each Swin Transformer block includes a first encoder block and a second encoder block, the first encoder block and the second encoder block are obtained by replacing the multi-head self-attention layer in the encoder block of ViT with a window multi-head self-attention layer and a sliding window multi-head self-attention layer respectively, and the calculation process of the Swin Transformer block is as follows: ； ； ； ； in, represents the input features of the Swin Transformer block, Representation layer normalization layer, represents the window multi-head self-attention layer, represents a multi-layer perceptron, represents the sliding window multi-head self-attention layer, represents the intermediate features of the first encoder block, represents the output features of the first encoder block, represents the intermediate features of the second encoder block, Represents the output features of the Swin Transformer block.

4. The medical image segmentation method based on hierarchical multi-scale CNN-Transformer according to claim 1, characterized in that: The two-layer feature fusion module includes a first hierarchical structure, a second hierarchical structure and a cross-attention module, the coding features of the first stage pass through the first hierarchical structure to obtain the output features of the first hierarchical structure, the coding features of the third stage pass through the second hierarchical structure to obtain the output features of the second hierarchical structure, the output features of the first hierarchical structure and the output features of the second hierarchical structure are input into the cross-attention module to obtain the calibrated coding features of the first stage and the calibrated coding features of the third stage respectively; The first hierarchical structure and the second hierarchical structure both include a normalization layer, a global average pooling layer, and a Transformer encoder; The encoding features of the first stage and the encoding features of the third stage are respectively passed through the normalization layer and the global average pooling layer to derive the corresponding class tokens, as shown in the following formula: ； ； in, represents the encoding features of the first stage, represents the encoding features of the third stage, represents the normalization layer, represents the global average pooling layer, Indicates the class token corresponding to the encoding feature of the first stage, Indicates the class token corresponding to the encoding feature of the third stage; The number of Transformer encoders in the first hierarchical structure is 2, and the number of Transformer encoders in the second hierarchical structure is 1. The class token corresponding to the encoding feature of the first stage passes through the two Transformer encoders in the first hierarchical structure to obtain the processed class token corresponding to the encoding feature of the first stage; the class token corresponding to the encoding feature of the third stage passes through the one Transformer encoder in the second hierarchical structure to obtain the processed class token corresponding to the encoding feature of the third stage; The dimensionally adjusted features of the processed class tokens corresponding to the coding features of the first stage are connected with the coding features of the third stage as the key and value, and the dimensionally adjusted features of the processed class tokens corresponding to the coding features of the first stage are used to perform the query to obtain the calibrated coding features of the first stage, as shown in the following formula: ； ； in, represents the first fusion feature, Indicates the processed class token corresponding to the encoding features of the first stage, represents the cross attention layer, Indicates that The operation of projecting to the dimension of the encoded features of the third stage, LN represents the layer normalization layer, Indicates that The operation of projecting to the dimension of the encoded features in stage 1, Indicates a connection operation. represents the coded features of the first stage after calibration; The dimensionally adjusted features of the processed class tokens corresponding to the coding features of the third stage are connected with the coding features of the first stage as the key and value, and the dimensionally adjusted features of the processed class tokens corresponding to the coding features of the third stage are used to perform the query to obtain the calibrated coding features of the third stage, as shown in the following formula: ； ； in, represents the second fusion feature, Indicates the processed class token corresponding to the encoding feature of the third stage, represents the cross attention layer, Indicates that The operation of projecting to the dimension of the encoded features of the first stage, LN represents the layer normalization layer, Indicates that The operation of projecting to the dimension of the encoded features in stage 3, Represents the coded features of the calibrated stage 3.

5. The medical image segmentation method based on hierarchical multi-scale CNN-Transformer according to claim 1, characterized in that: The decoding unit includes a first upsampling convolution module, a convolution module, a second upsampling convolution module and a segmentation head; the first upsampling convolution module and the second upsampling convolution module both adopt an upsampling convolution structure, the upsampling convolution structure includes a convolution layer with a convolution kernel size of 3×3, two bilinear upsampling layers, a group normalization layer and a ReLU activation function layer, the convolution module includes a 3×3 convolution layer, a group normalization layer and a ReLU activation function layer, and the segmentation head adopts a convolution layer with a convolution kernel size of 3×3; in the decoding unit, the calibrated first stage encoding features and the calibrated third stage encoding features are respectively input into the first upsampling convolution module and the convolution module to obtain a first decoding feature and a second decoding feature, and the first decoding feature and the second decoding feature are added and sequentially passed through the second upsampling convolution module and the segmentation head to obtain a predicted segmentation result.

6. The medical image segmentation method based on hierarchical multi-scale CNN-Transformer according to claim 1, characterized in that: The loss function used in the training process of the medical image segmentation model for: ； in, represents the cross entropy loss, represents the Dice loss, represents the weight coefficient, and They represent the true segmentation results and the predicted segmentation results respectively.

7. A medical image segmentation device based on hierarchical multi-scale CNN-Transformer, characterized in that: include: A model building module is configured to build and train a medical image segmentation model based on CNN-Transformer to obtain a trained medical image segmentation model, wherein the trained medical image segmentation model includes an encoding unit and a decoding unit, wherein the encoding unit includes a stem layer, an encoding module, and a double-layer feature fusion module, wherein the encoding module includes a three-stage hybrid Transformer module and a block merging layer arranged between the hybrid Transformer modules of two adjacent stages, wherein the hybrid Transformer module includes a multi-scale channel attention layer, a CNN block, and a Transformer layer, wherein the Transformer layer includes a reshaping layer, a Swin Transformer module, and a reverse reshaping layer connected in sequence; A segmentation module is configured to obtain a medical image to be segmented and input it into the trained medical image segmentation model, input the medical image to be segmented into the stem layer for block segmentation to obtain a block image, input the block image into the encoding module, in the hybrid Transformer module of the i-th stage in the encoding module, input features of the hybrid Transformer module of the i-th stage are respectively input into the multi-scale channel attention layer and the CNN block in the hybrid Transformer module of the i-th stage to obtain the initial multi-scale features of the i-th stage and the local features of the i-th stage, the initial multi-scale features of the i-th stage and the local features of the i-th stage are input into the Transformer layer in the hybrid Transformer module of the i-th stage to obtain the coding features of the i-th stage; The encoded features of the i-th stage are input into the hybrid Transformer module of the i+1-th stage after passing through the block merging layer, and finally the encoded features of the first stage and the encoded features of the third stage are obtained; Inputting the coding features of the first stage and the coding features of the third stage into the double-layer feature fusion module to obtain the calibrated coding features of the first stage and the calibrated coding features of the third stage respectively; The calibrated encoding features of the first stage and the calibrated encoding features of the third stage are input into the decoding unit to obtain a predicted segmentation result.

8. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images