ATFormer system and corresponding method for medical image segmentation

Abstract

The present invention provides an ATFormer architecture for medical image segmentation, comprising a patch partitioning layer, an encoder, a bottleneck layer, a decoder, and a linear projection layer; the patch partitioning layer divides an input medical image into non-overlapping patch images; the encoder generates feature maps at different resolutions from the non-overlapping patch images, wherein the resolution of the feature maps is reduced and the feature dimension is increased; the bottleneck layer performs deep feature learning on the feature maps; the decoder gradually upsamples the feature maps at different resolutions from the encoder under the joint action of a progressive guided fusion (PGF) module, wherein the resolution of the upsampled feature maps is increased and the feature dimension is reduced, and the resolution of the feature maps is restored to the original resolution; and the linear projection layer outputs the feature maps output by the decoder as pixel-by-pixel segmentation prediction results. The present invention provides a new ATFormer architecture that can be used for medical image segmentation and provides accurate segmentation results.

Description

Technical Field

[0001] The present invention relates to the field of medical image analysis, and more particularly, to an ATFormer system for medical image segmentation and a medical image segmentation method. Background Art

[0002] Accurate medical image segmentation results are crucial for medical image analysis, such as clinical diagnosis and preoperative assessment. CNN-based methods have long been a cornerstone for medical image segmentation due to their inductive biases, such as translation invariance. With the success of the Transformer in natural language processing (NLP), researchers have widely applied it to medical image processing. However, for medical image segmentation, Transformer-based methods are still in their infancy. Summary of the Invention

[0003] The present invention aims to solve the technical problems existing in the prior art and provides an ATFormer system for medical image segmentation and a medical image segmentation method.

[0004] According to a first aspect of the present invention, there is provided an ATFormer system for medical image segmentation, comprising a patch partitioning layer, an encoder, a bottleneck layer, a decoder, and a linear projection layer;

[0005] Patch segmentation layer, which divides the input medical image into non-overlapping patch images;

[0006] The encoder generates feature maps at different resolutions from non-overlapping patch images, with the resolution of the feature maps reduced and the feature dimension increased;

[0007] The bottleneck layer performs deep feature learning on the feature map;

[0008] The decoder gradually upsamples the feature maps of different resolutions from the encoder under the joint action of the PGF module. The resolution of the upsampled feature map is increased and the feature dimension is reduced, and the resolution of the feature map is restored to the original resolution.

[0009] The linear projection layer outputs the feature map output by the decoder as a pixel-by-pixel segmentation prediction result.

[0010] According to a second aspect of the present invention, a medical image segmentation method based on an ATFormer system is provided, wherein the ATFormer system includes a patch partitioning layer, an encoder, a bottleneck layer, a decoder, and a linear projection layer, and the method includes:

[0011] The patch segmentation layer divides the input medical image into non-overlapping patch images;

[0012] Generating feature maps of different resolutions from the non-overlapping patch images in an encoder, wherein the resolution of the feature maps is reduced and the feature dimension is increased;

[0013] The bottleneck layer performs deep feature learning on the feature map;

[0014] The decoder gradually upsamples the feature maps of different resolutions from the encoder under the joint action of the PGF module. The resolution of the upsampled feature map is increased and the feature dimension is reduced, and the resolution of the feature map is restored to the original resolution.

[0015] The feature map output by the decoder is output as a pixel-by-pixel segmentation prediction result through a linear projection layer.

[0016] The present invention provides an ATFormer system for medical image segmentation and a medical image segmentation method, providing a new ATFormer system that can be used for medical image segmentation and has accurate segmentation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of the ATFormer system for medical image segmentation provided by the present invention;

[0018] Figure 2 It is a structural diagram of the enhanced conversion block;

[0019] Figure 3(a) is a schematic diagram of the structure of a simplified AMix-FFN module;

[0020] FIG3( b ) is a schematic structural diagram of the AMix-FFN module of the present invention;

[0021] Figure 4 It is a structural diagram of the progressive guided fusion module;

[0022] Figure 5 This is a flowchart of a medical image segmentation method provided by the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. In addition, the technical features in the various embodiments or single embodiments provided by the present invention can be arbitrarily combined with each other to form a feasible technical solution. This combination is not restricted by the sequence of steps and / or structural composition mode, but must be based on the ability of ordinary technicians in this field to implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0024] Figure 1 The present invention provides an ATFormer system for medical image segmentation, which includes a patch partitioning layer, an encoder, a bottleneck layer, a decoder and a linear projection layer.

[0025] Among them, the patch division layer divides the input medical image into non-overlapping patch images; the encoder generates feature maps at different resolutions from the non-overlapping patch images, with the resolution of the feature maps reduced and the feature dimension increased; the bottleneck layer performs deep feature learning on the feature maps; the decoder gradually upsamples the feature maps at different resolutions from the encoder under the joint action of the PGF module, with the resolution of the upsampled feature maps increased and the feature dimension reduced, and the resolution of the feature maps is restored to the original resolution; the linear projection layer outputs the feature maps output by the decoder as pixel-by-pixel segmentation prediction results.

[0026] As an example, the encoder includes a patch embedding layer, three sets of Advanced Transformer Blocks (ATBs) connected in series, and a patch fusion layer. The patch embedding layer projects the dimensions of each patch image to a specified dimension. The three sets of ATBs connected in series and the patch fusion layer generate feature images with hierarchical feature representations based on the patch images of the specified dimensions. The ATBs learn long-range dependencies and local context with low computational complexity, and the patch fusion layer generates downsampled feature maps with reduced resolution and increased dimensionality.

[0027] The decoder includes three groups of patch expansion layers, PGF modules, enhanced conversion blocks and a separate patch expansion layer; the patch expansion layer in each group upsamples the deep feature map output by the encoder, reshapes the feature map of the adjacent dimension to twice the original resolution, and reduces the feature dimension to half the original dimension; the upsampled feature map is combined with the multi-scale feature map output by the enhanced conversion block of the corresponding group in the encoder. Figure 1 The image is input into the PGF module and undergoes feature representation learning in the enhanced conversion block. After being processed by multiple sets of patch expansion layers, PGF modules and enhanced conversion blocks, it is finally processed by a separate patch expansion layer to restore the resolution of the output feature map to the same resolution as the input medical image.

[0028] It can be understood that the overall system of ATFormer (Advanced Transformer) proposed in the present invention is as follows Figure 1 It maintains a U-shaped encoder-decoder system similar to U-Net. Specifically, given a medical image, in order to convert image data into sequence embedding, the patch partitioning layer first divides the input medical image into non-overlapping patch images of size The non-overlapping image patches are then fed into the encoder to generate features at different resolutions. In the encoder, a patch embedding layer is first applied to project the feature dimensions to a specified dimension (denoted as C). The transformed tokens (Transformer's name for transformed features) are passed through three interleaved enhanced transformation blocks and a patch fusion layer to generate a hierarchical feature representation. The enhanced transformation blocks learn long-range dependencies and local context with low computational complexity (less than quadratic complexity), while the patch fusion layer is responsible for generating downsampled features and expanding the dimensions. As the encoder progresses, the resolution of the feature maps decreases and the feature dimensions increase. Inspired by the U-Net, the present invention designs a decoder based on the symmetric Advanced Transformer and adds a PGF module. The PGF module guides the upsampling process based on semantic information from the encoder at different resolutions, which helps recover image details lost during downsampling. The decoder consists of three interleaved enhanced transformation blocks and a patch expansion layer, as well as a separate patch expansion layer. In contrast to the patch fusion layer, the patch expansion layer reshapes adjacent feature maps to twice their original resolution. Different from this, the last patch expansion layer performs a four-fold upsampling to restore the resolution of the feature map to the resolution of the input image ( The resolution and dimension of the feature map change in the reverse order of the encoder. Finally, a linear projection layer is applied to the resulting feature map to output pixel-by-pixel segmentation predictions. The following describes each module in the ATFormer system in detail.

[0029] As an embodiment, each group of enhanced conversion blocks includes two, each enhanced conversion block includes a multi-head self-attention module, two back-layer normalizations, an AMix-FFN module containing a GELU activation function, and two residual connections, wherein the multi-head self-attention module in one enhanced conversion block is a multi-head attention W-MSA module based on a window mechanism, and the multi-head self-attention module in the other enhanced conversion block is a multi-head attention SW-MSA module based on a shift window mechanism;

[0030] The W-MSA module, the first rear layer normalization, the first AMix-FFN module and the second rear layer normalization in the first enhanced conversion block are connected in sequence, and are connected through the first residual connection before the W-MSA and after the first rear layer normalization, and are connected through the second residual connection before the first AMix-FFN module and after the second rear layer normalization. The SW-MSA module, the third rear layer normalization, the second AMix-FFN module and the fourth rear layer normalization in the second enhanced conversion block are connected in sequence, and are connected through the third residual connection before the SW-MSA and after the third rear layer normalization, and are connected through the fourth residual connection before the second AMix-FFN module and after the fourth rear layer normalization. The output of the second rear layer normalization is connected to the input of the SW-MSA module.

[0031] See also Figure 2 , is a structural diagram of the enhanced conversion block. Each group of enhanced conversion blocks has two, and each enhanced conversion block consists of a multi-head self-attention module, residual connection, back-layer normalization, and AdvancedMixFFN with GELU activation function. In particular, the present invention replaces the traditional scaled dot product attention with scaled cosine attention for the calculation of attention scores in the multi-head self-attention module. Figure 2 Two consecutive enhanced conversion blocks based on the shifting window mechanism are shown.

[0032] The window-based multi-head attention (W-MSA) module and the shifted window-based multi-head self-attention (SW-MSA) module are applied to two consecutive enhanced transformation blocks respectively. Based on this window division mechanism, two consecutive enhanced transformation blocks can be expressed as:

[0033] (1);

[0034] (2);

[0035] (3);

[0036] (4);

[0037] in, and They represent the outputs of the multi-head self-attention module and the first AMix-FFN module in the first enhanced conversion block respectively; and denote the outputs of the multi-head self-attention module and the second AMix-FFN module in the second enhanced transformation block, respectively; W-MSA and SW-MSA denote multi-head self-attention using window partitioning and shifted window partitioning, respectively.

[0038] Among them, in the medical image segmentation task, the Transformer-based method uses scalable dot product attention. When the data is expanded to high dimensions, the minority weight values will be too large, and the front-layer normalization method will lead to training instability. Therefore, for the multi-head self-attention module, the present invention introduces scalable cosine attention to address the problem of large minority weight values in the model. The attention calculation formula is expressed as:

[0039] (5);

[0040] in, are query, key, and value matrices respectively, is the number of patch images in the window, B is the relative position deviation matrix, is a learnable scalar. The value of the cosine function is naturally normalized to the interval [-1, 1], so it has a lower attention weight.

[0041] Among them, in the process of using the front-layer normalization method, the output activation value of each residual connection is directly merged into the main branch, which causes the activation value amplitude of the main branch to become larger and larger in deeper layers, and the difference in activation value amplitude between layers will also become larger and larger, which makes training unstable. The present invention introduces the back-layer normalization method proposed by Ze Liu et al. The so-called back-layer normalization is to normalize the output results of the multi-head self-attention module. Layer normalization is performed before the output of each residual connection is merged into the main branch to solve the problem of activation value amplitude accumulation in the main branch.

[0042] Advanced Mixed Feedforward Neural Networks incorporate deep convolutions to effectively capture local contextual information, thereby providing discriminative representations. Figure 3(a) shows a simplified version of Advanced MixFFN. First, a deep convolution is added after the fully connected layer, and a skip connection is added before the deep convolution. The result after the deep convolution and the skip connection are then summed, and finally layer normalization is applied. This process can be expressed as:

[0043] (6);

[0044] (7);

[0045] in is the output of self-attention, The kernel size is The depth of convolution is , FC is the linear layer, LN is the layer normalization, and the activation function is GELU.

[0046] Pure Transformer-based methods lack the inherent inductive bias of CNNs and cannot effectively model local context. Based on this, the present invention extends the simplified Advanced MixFFN to make the model more expressive. As shown in Figure 3(b), recursive skip connections with layer normalization are added to the Mini Advanced MixFFN, which can be expressed as:

[0047] (8);

[0048] (9);

[0049] (10);

[0050] in, is the output of the depth convolution layer, is the output result of the first fully connected layer, is the result after normalization of the first layer, is the result after normalization of the i-th layer, is the output of the second fully connected layer.

[0051] Depending on the value of i, Advanced MixFFNs with varying numbers of recursive layers can be constructed. Experiments have shown that an i of 2 yields the best results, so this is used in this paper. Layer normalization and skip connections are used in tandem to construct different feature distributions and consistencies, enabling the model to more fully learn and thus enhancing representational capabilities.

[0052] See Figure 2 , first, the patch embedding layer in the encoder converts the resolution The patch is projected onto the specified dimension C, resulting in The patch images are then fed into two consecutive Advanced Transformer blocks for image representation learning, during which the feature dimension and resolution remain constant. Finally, the patch fusion layer downsamples by a factor of 2, halving the number of patch images and doubling the feature dimension. The above three steps are repeated three times.

[0053] The patch fusion layer connects each group Feature maps of adjacent patch images, where the number of patch images is reduced , the resolution is reduced by a factor of 2, and the dimension is increased by a factor of 4. To maintain consistency with the typical multi-scale design style of convolutional neural networks (CNNs), this paper applies a linear layer on the channel dimension of the connection (the linear layer is located in the patch fusion layer) to transform the feature dimension to twice the original dimension.

[0054] It should be noted that there is a bottleneck layer between the encoder and decoder. The bottleneck layer is relatively simple, using only two consecutive enhanced transformation blocks to learn deep feature representations. During this period, the resolution and dimensionality of the feature maps remain unchanged, just like the ordinary enhanced transformation blocks in the encoder and decoder.

[0055] The encoder reduces the resolution of the feature map and increases its dimensionality. After processing the bottleneck layer, the deep feature map is fed into the patch expansion layer in the decoder for a 2x upsampling, thereby reshaping the feature map of the adjacent dimension to twice the original resolution and reducing the feature dimensionality to half the original dimension. The resulting feature map is then fed into the PGF module along with the multi-scale features from the encoder. Next, it is fed into two consecutive enhanced transformation blocks to learn feature representations. Similar to the encoder, this process is repeated three times, and finally, the resolution of the feature map in the patch expansion layer is restored to the resolution of the input image.

[0056] Among them, traditional skip connection methods such as concatenation and summation cannot effectively utilize semantic information at different resolutions, which leads to poor results when generating pixel-level predictions. Therefore, a progressive guided fusion module is designed in the ATFormer system of the present invention to efficiently utilize semantic information at different resolutions to guide predictions.

[0057] Among them, see Figure 4 , the PGF module in the decoder consists of a linear projection layer, two enhanced transformation blocks ( Figure 4 Only the first Block is drawn, the second Block and Figure 2 Same as the second Block in ).

[0058] In CNN, the common skip connection methods are concatenation and summation. Based on the research of Ashish Vaswani et al. on skip connection and the application of Hong-Yu Zhou et al. on skip attention, the present invention proposes a PGF module to replace the common skip connection methods, such as Figure 4 Specifically, from the encoder Downsampled output of the layer patch fusion layer After a linear layer, they are projected into and :

[0059] (11);

[0060] Where LP represents the linear projection layer. Layer patch expands the upsampled output of the layer As The three are input to the first Block of two consecutive enhanced conversion blocks (such as Figure 4 The second Block (SW-MSA Block) is connected to the normal enhanced conversion block (such as Figure 2 As shown in the figure, it is not repeated in the figure. As with the enhanced conversion block, the attention calculation of the first Block in the PGF module can be expressed as:

[0061] (12);

[0062] in Indicates the layer number after linear projection layer processing, except Other symbols except have the same meanings as in formula (5).

[0063] The patch expansion layer and the patch fusion layer have opposite processing logic. First, the linear layer increases the feature dimension to twice the original dimension, and then a reshape operation is used to expand the resolution of the input features to twice the original resolution. The patch expansion layer and the patch fusion layer jointly perform the key operations required for multi-scale features: upsampling and downsampling.

[0064] See also Figure 5 , a medical image segmentation method based on the ATFormer system of the present invention is provided. The ATFormer system includes a patch partitioning layer, an encoder, a bottleneck layer, a decoder and a linear projection layer. The medical image segmentation method based on the ATFormer system includes: the patch partitioning layer divides the input medical image into non-overlapping patch images; the non-overlapping patch images are used to generate feature maps at different resolutions in the encoder, and the resolution of the feature maps is reduced and the feature dimension is increased; the bottleneck layer performs deep feature learning on the feature maps; the decoder, under the joint action of the PGF module, gradually upsamples the feature maps at different resolutions from the encoder, the resolution of the upsampled feature maps is increased and the feature dimension is reduced, and the resolution of the feature maps is restored to the original resolution; the feature maps output by the decoder are output as pixel-by-pixel segmentation prediction results through the linear projection layer.

[0065] It can be understood that the medical image segmentation method based on the ATFormer system provided by the present invention corresponds to the ATFormer system for medical image segmentation provided in the aforementioned embodiments. The relevant technical features of the medical image segmentation method based on the ATFormer system can refer to the relevant technical features of the ATFormer system for medical image segmentation, which will not be repeated here.

[0066] Different methods were experimented with and compared based on the experimental results. Table 1 shows the experimental data results on the Automatic Cardiac Diagnosis Challenge (ACDC) dataset. ATFormer achieved the best performance, with a DSC of 87.38% and an HD95 of 2.09. Compared to Swin-UNet, a method using only the Swin TransFormer, this method significantly improved, with DSC and HD95 increases of 1.6% and 0.23, respectively. Compared to Trans UNet, a hybrid system method using a CNN encoder, DSC and HD95 increased by 0.82% and 0.19, respectively. Compared to the newer method MISSFormer, the proposed method still improved HD95 by 0.19, demonstrating its superior performance in processing edge details.

[0067] Table 1 Experimental data of different methods on the ACDC dataset

[0068]

[0069] According to experimental phenomena, the other three methods cannot distinguish the heart and surrounding tissues well, while this method is closer to the true label and can learn feature representation well.

[0070] Experimental data on the COVID-19 CT Lung and Infection Segmentation Dataset (COVID-19 CT-Seg) is shown in Table 2. Taking into account both DSC and HD5 evaluation metrics, ATFormer still achieved the best results, with a DSC of 80.74% and an HD95 of 46.44. Given the focus of this dataset's task, segmentation of the COVID-19 infected region is more important than that of the lung region. This method achieved impressive results for infected region segmentation—DSC and HD95 scores of 70.47% and 69.24, respectively, representing improvements of 8.56% and 2.01% over Trans UNet. This demonstrates ATFormer's adaptability and efficiency for the COVID-19 infected region segmentation task.

[0071] Table 2 Experimental data of different methods on the COVID-19 CT-Seg dataset

[0072]

[0073] As shown in Table 3, Trans UNet has the highest computational and parameter requirements due to the high resource and parameter consumption of convolution. ATFormer, on the other hand, significantly improves computational efficiency by simply adding depthwise convolution to Advanced MixFFN, validating the high efficiency of depthwise convolution. Compared to the other three methods, ATFormer reduces computational and parameter requirements while improving segmentation performance.

[0074] Table 3 Comparison of four methods in FLOPs and Params

[0075]

[0076] In addition, we conduct ablation experiments on the ACDC dataset to verify the effectiveness of the key modules in this method. Swin-UNet is used as the baseline model. To ensure fair comparison, all experiments are conducted under the same hyperparameters.

[0077] Impact of Advanced MixFFN: Based on Swin-UNet, the present invention replaces the FFN in the Swin Transformer with an Advanced MixFFN containing depthwise convolution and recurrent residual connections. Table 4 shows that the addition of depthwise convolution and layer normalization significantly improves performance compared to the FFN containing two linear layers, with the best results achieving improvements of 2.67% and 0.11 on DSC and HD95, respectively. In particular, the addition of two layers of recurrent residual connections with layer normalization achieves the best performance, with DSC scores of 88.45% and HD95 scores of 2.21.

[0078] Table 4 Comparison of the effects of different numbers of recursive residual connection layers in Advanced MixFFN

[0079]

[0080] Among them, MixFFN1 is Advanced MixFFN with 1 layer of LayerNorm, and the others are similar.

[0081] Impact of Scalable Cosine Attention and Post-Layer Normalization: To validate the effectiveness of Scalable Cosine Attention and Post-Layer Normalization, we conducted comparative experiments against methods such as Scalable Dot Product Attention and Pre-Layer Normalization. As shown in Table 5, using Scalable Dot Product Attention and Pre-Layer Normalization as the baseline, we replaced them with three combinations of key modules: post-layer normalization, Scalable Cosine Attention, post-layer normalization, and Scalable Cosine Attention. DotAtt+posLN achieved improvements of 1.47% and 0.02% in DSC and 0.02% in HD95, respectively, compared to the baseline, demonstrating that post-layer normalization significantly enhances feature representation. CosAtt+preLN showed modest improvements compared to the baseline, but the combination of CosAtt+posLN achieved significant performance improvements—2.54% and 0.12% in DSC and HD95, respectively. This demonstrates that Scalable Cosine Attention achieves stronger performance only when combined with post-layer normalization.

[0082] Table 5 Comparison of the effects of scalable dot product attention, scalable cosine attention, and post-layer normalization

[0083]

[0084] Among them, Dot Att is scalable dot product attention, Cos Att is scalable cosine attention, preLN is front layer norm, and posLN is back layer norm.

[0085] Impact of the Progressive Guided Fusion Module: Inspired by Vaswani A, the commonly used skip connections, such as concatenation or summation, are replaced with the PGF module, making it more versatile. As shown in Table 6, although PGF is slightly inferior on HD95, it has significant effects on the segmentation of all three cardiac tissues on DSC.

[0086] Table 6 Comparison of the effects of series connection and PGF module

[0087]

[0088] Among them, CatCon represents series connection.

[0089] The impact of gradual module combination: The aforementioned ablation experiments only verified the effectiveness of one or two modules. This experiment combines modules from shallow to deep to verify the effectiveness of module stacking. As shown in Table 7, the first row shows the baseline configuration of Swin-UNet, while the second row combines previously proven effective methods (CosAtt+posLN and MixMLP2). As expected, both DSC and HD95 achieved significant improvements—1.3% and 0.15%, respectively. To explore the effectiveness of the enhanced transform block for the PGF module, the blocks in the PGF were set to the Swin Transformer block and the enhanced transform block, respectively, to design the PGF v1 and v2 versions. Clearly, the v1 version performs poorly on HD95 compared to the baseline. This indicates that the FFN in the Swin Transformer block fails to effectively utilize semantic information from the encoder at different resolutions, resulting in an inability to accurately recover edge details. In contrast, version v2 achieves the best performance in both evaluation metrics—DSC and HD95 improve by 1.6% and 0.23 respectively, which shows that the enhanced transformation block is not only effective in learning feature representations but also makes full use of contextual information.

[0090] Table 7 Comparison of the effects of several improvement measures combined

[0091]

[0092] Among them, v1 means that PGF uses Swin Transformer Block (the third row in the figure), v2 means that PGF uses enhanced transformation block, and the meanings of other symbols are the same as the previous table.

[0093] The ATFormer system and medical image segmentation method provided by the embodiments of the present invention, which combine a hybrid system of CNN and Transformer, have the following advantages:

[0094] First, ATFormer introduces scalable cosine attention to produce more moderate attention weights, and the post-layer normalization method limits the activation values to an acceptable range, stabilizing the training of the model.

[0095] Second, we use depthwise convolutions combined with residual connections with recurrent layer normalization to add inductive biases, which can effectively model local context.

[0096] Third, ATFormer designs a progressive guided fusion module to efficiently utilize semantic information at different resolutions to guide prediction.

[0097] It should be noted that, in the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0098] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0099] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0100] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0101] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0102] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0103] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. An ATFormer system for medical image segmentation, characterized in that: The ATFormer system includes a patch partitioning layer, an encoder, a bottleneck layer, a decoder, and a linear projection layer; Patch segmentation layer, which divides the input medical image into non-overlapping patch images; The encoder generates feature maps at different resolutions from non-overlapping patch images, with the resolution of the feature maps reduced and the feature dimension increased; The bottleneck layer performs deep feature learning on the feature map; The decoder gradually upsamples the feature maps of different resolutions from the encoder under the joint action of the PGF module. The resolution of the upsampled feature map is increased and the feature dimension is reduced, and the resolution of the feature map is restored to the original resolution. The linear projection layer outputs the feature map output by the decoder as a pixel-by-pixel segmentation prediction result; The decoder includes three sets of patch extension layers, PGF module and enhanced conversion block and a separate patch extension layer; The patch expansion layer in each group upsamples the deep feature map output by the encoder, reshapes the feature map of the adjacent dimension to twice the original resolution, and reduces the feature dimension to half of the original dimension; The upsampled feature map is input into the PGF module together with the multi-scale feature map output by the enhanced transformation block of the corresponding group in the encoder, and the feature representation is learned by the enhanced transformation block; After being processed by multiple sets of patch expansion layers, PGF modules and enhanced conversion blocks, and finally by a separate patch expansion layer, the resolution of the output feature map is restored to the resolution of the input medical image.

2. The ATFormer system according to claim 1, characterized in that: The encoder includes a patch embedding layer and three sets of enhanced transformation blocks and patch fusion layers connected in series; The patch embedding layer projects the dimension of each patch image to a specified dimension; The three sets of enhanced transformation blocks and patch fusion layers connected in series generate feature images of hierarchical feature representation based on patch images of specified dimensions; The enhanced transformation block learns long-range dependencies and local contexts with low computational complexity, and the patch fusion layer is responsible for generating downsampled feature maps with reduced resolution and increased dimensionality.

3. The ATFormer system according to claim 2, characterized in that: Each group of enhanced conversion blocks consists of two, each of which includes a multi-head self-attention module, two back-layer normalizations, an AMix-FFN module with a GELU activation function, and two residual connections. The multi-head self-attention module in one of the enhanced conversion blocks is a multi-head attention W-MSA module based on a window mechanism, and the multi-head self-attention module in the other enhanced conversion block is a multi-head attention SW-MSA module based on a shift window mechanism. The W-MSA module, the first rear layer normalization, the first AMix-FFN module and the second rear layer normalization in the first enhanced conversion block are connected in sequence, and are connected through the first residual connection before the W-MSA and after the first rear layer normalization, and are connected through the second residual connection before the first AMix-FFN module and after the second rear layer normalization. The SW-MSA module, the third rear layer normalization, the second AMix-FFN module and the fourth rear layer normalization in the second enhanced conversion block are connected in sequence, and are connected through the third residual connection before the SW-MSA and after the third rear layer normalization, and are connected through the fourth residual connection before the second AMix-FFN module and after the fourth rear layer normalization. The output of the second rear layer normalization is connected to the input of the SW-MSA module.

4. The ATFormer system according to claim 3, characterized in that: Two consecutive enhanced conversion blocks are represented as: ； ； ； ； in and They represent the outputs of the multi-head self-attention module and the first AMix-FFN module in the first enhanced conversion block respectively; and They represent the outputs of the multi-head self-attention module and the second AMix-FFN module in the second enhanced transformation block; W-MSA and SW-MSA represent multi-head self-attention using window partitioning and shifted window partitioning, respectively; The multi-head self-attention module is based on the scalable cosine attention mechanism, and the calculation formula of the scalable cosine attention mechanism is expressed as: ； in are query, key, and value matrices respectively, is the number of patch images in the window, B is the relative position deviation matrix, To learn scalars, the value of the cosine function is naturally normalized to the interval [-1,1].

5. The ATFormer system according to claim 3, characterized in that: The first AMix-FFN module and the second AMix-FFN module each include a first fully connected layer, a depthwise convolutional layer, several layer normalizations with residual connections, a GELU activation function, and a second fully connected layer connected in sequence. The processing of the first AMix-FFN module and the second AMix-FFN module is expressed as follows: ； ； ； in, is the output of the first fully connected layer, ) is the output result of the depth convolution layer, LN represents the output result of layer normalization, is the result after normalization of the first layer, is the normalized output of the i-th layer, is the output of the second fully connected layer.

6. The ATFormer system according to claim 1 or 5, characterized in that: The PGF module includes a linear projection layer and two enhanced conversion blocks connected in sequence; From the encoder Downsampled output of the layer patch fusion layer After a linear projection layer, they are projected into and : ； Among them, LP represents the linear projection layer, which comes from the Layer patch expands the upsampled output of the layer As ; The three ( and ) are input into the W-MSA module in the enhanced conversion block respectively. The attention calculation of the W-MSA module is expressed as: ； in Indicates the layer number after the linear projection layer is processed.

7. A medical image segmentation method based on the ATFormer system, characterized in that: The ATFormer system includes a patch partitioning layer, an encoder, a bottleneck layer, a decoder, and a linear projection layer. The method includes: The patch segmentation layer divides the input medical image into non-overlapping patch images; Generating feature maps of different resolutions from the non-overlapping patch images in an encoder, wherein the resolution of the feature maps is reduced and the feature dimension is increased; The bottleneck layer performs deep feature learning on the feature map; The decoder gradually upsamples the feature maps of different resolutions from the encoder under the joint action of the PGF module. The resolution of the upsampled feature map is increased and the feature dimension is reduced, and the resolution of the feature map is restored to the original resolution. Outputting the feature map output by the decoder as a prediction result of pixel-by-pixel segmentation through a linear projection layer; The decoder includes three sets of patch extension layers, PGF module and enhanced conversion block and a separate patch extension layer; The patch expansion layer in each group upsamples the deep feature map output by the encoder, reshapes the feature map of the adjacent dimension to twice the original resolution, and reduces the feature dimension to half of the original dimension; The upsampled feature map is input into the PGF module together with the multi-scale feature map output by the enhanced transformation block of the corresponding group in the encoder, and the feature representation is learned by the enhanced transformation block; After being processed by multiple sets of patch expansion layers, PGF modules and enhanced conversion blocks, and finally by a separate patch expansion layer, the resolution of the output feature map is restored to the resolution of the input medical image.

Images