A medical image segmentation method based on a plurality of parallel transformer branch neural networks

Abstract

This disclosure relates to a medical image segmentation method based on multiple parallel Transformer branch neural networks. This method utilizes the inherent multi-scale characteristics of convolutional networks, progressively reducing the spatial resolution of the output feature maps through pooling layers or strided convolutional layers to extract multi-scale local feature maps from sub-residual networks at different stages. Then, a Transformer branch network is constructed to acquire global contextual information using a self-attention mechanism. Multiple parallel Transformer branch networks are built on top of the multi-level local feature maps in the convolutional network portion to obtain multi-scale global feature maps. Finally, long skip connections and large-scale linear upsampling are used to add and fuse the local and global feature maps from the multiple parallel branches to obtain a target fused feature map. The fused target feature map contains multi-scale local and global features, thus contributing to the accurate segmentation of organs at multiple scales.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to a medical image segmentation method based on a neural network with multiple parallel Transformer branches. Background Technology

[0002] Medical image segmentation is an indispensable step in many modern clinical medical procedures. It separates organs or diseased tissues from medical scan images, thereby assisting clinicians in diagnosing conditions and developing treatment plans. However, manual visual segmentation of medical images is very time-consuming, highly subjective, and relies heavily on the professional knowledge and experience of physicians. This application proposes a technique for automatic segmentation of abdominal computed tomography (CT) images. Abdominal multi-organ segmentation can provide the interrelationships between organs and the spatial location and shape of target organs. Furthermore, due to the different shapes and sizes of various abdominal organs, there is a high demand for accurate segmentation of multi-scale targets.

[0003] In related technologies, a detection-then-segmentation approach is used to segment medical images. This involves first locating targets using the classic SSD (Single Shot MultiBox Detector) object detection network, and then performing pixel-level segmentation of the target regions. Furthermore, it performs detection on feature maps of different scales extracted using a VGG (Visual Geometry Group)-16 convolutional network to locate targets of varying sizes. While this method uses convolutional networks to extract multi-scale feature maps for detecting and segmenting organs or lesions of different sizes, due to the inherent locality of convolutional kernels, it can only acquire multi-scale local features and misses equally important global contextual features. Summary of the Invention

[0004] To avoid the shortcomings of existing technologies, this application provides a medical image segmentation method based on multiple parallel Transformer branch neural networks, which solves the problem that existing medical image segmentation algorithms based on convolutional networks cannot effectively extract multi-scale global context information and thus have difficulty in accurately segmenting organs or lesions of different sizes.

[0005] According to embodiments of this disclosure, a medical image segmentation method based on a neural network with multiple parallel Transformer branches is provided, the method comprising:

[0006] A convolutional network module is used to extract local features from a medical image dataset to obtain several local feature maps at different scales. The convolutional network module includes several cascaded residual neural networks ResNet-50.

[0007] A multi-head self-attention layer is constructed using the self-attention mechanism, and a Transformer branch is built;

[0008] Several parallel Transformer branches are built on each of the cascaded ResNet-50 residual neural networks, and multiple global feature maps of different scales are obtained using the multiple Transformer branches.

[0009] The local feature maps and global feature maps corresponding to each branch are fused using long skip connections to obtain multiple initial fused feature maps of different scales.

[0010] Multiple initial fusion feature maps are fused to obtain a target fusion feature map, and a segmentation map is obtained using a segmentation module.

[0011] Furthermore, the step of using a convolutional network module to extract local features from the medical image dataset to obtain several local feature maps at different scales, wherein the convolutional network module includes several cascaded ResNet-50 residual neural networks, includes:

[0012] A network was built on top of ResNet-50 to obtain the residual neural network ResNet-50; wherein, by removing the pooling layer and fully connected layer after the last residual module, the simplified convolutional network part contains 17 residual modules and 1 max pooling layer.

[0013] The size of the local feature maps output by each of the ResNet-50 residual neural networks differs.

[0014] Furthermore, the convolutional network module includes four ResNet-50 residual neural networks, which can obtain four local feature maps at different scales, namely, the first local feature map L1, the second local feature map L2, the third local feature map L3, and the fourth local feature map L4.

[0015] Furthermore, the step of constructing a multi-head self-attention layer using a self-attention mechanism and building Transformer branches includes:

[0016] The self-attention layer consists of a matrix dot product layer, a normalization layer, and a feedforward network built with a fully connected layer.

[0017] The Transformer branch is formed by connecting four self-attention layers in series.

[0018] Furthermore, the step of constructing a multi-head self-attention layer using a self-attention mechanism and building Transformer branches also includes:

[0019] The input local feature map is unfolded into a one-dimensional sequence. Then, three different linear projections are used to compute the first feature vector Query, the second feature vector Key, and the third feature vector Value after linear transformation from the sequence:

[0020]

[0021]

[0022]

[0023] in, Q is the Query value, K is the Key value, V is the Value value, and D is the Key value. h This represents the feature dimension, where N represents the number of pixels in the feature map. w Q Let w represent the first trainable parameter in the linear mapping. K w represents the second trainable parameter in the linear mapping. V denoted by the third trainable parameter in the linear mapping, h represents the h-th self-attention head, and Norm represents the normalization operation;

[0024] Calculate attention map A using dot product operation h Then use the attention map A h For V h Transformation:

[0025]

[0026] in, SA represents the scaling factor. h For the h-th self-attention head, Softmax represents the normalization function;

[0027] The outputs of a total of H self-attention heads are connected in parallel and projected. Remapping the feature dimensions back

[0028] MSA = W Project [SA 1 ;…;SA H ] T

[0029] The output of the multi-head self-attention module is passed through a normalization layer and a feedforward network (FFN) to output the global feature G:

[0030] G = FFN(Norm(MSA))

[0031] The self-attention layers are connected in series to obtain a Transformer branch.

[0032] Furthermore, the step of constructing several parallel Transformer branches on each of the cascaded ResNet-50 residual neural networks, and using the multiple Transformer branches to obtain multiple global feature maps of different scales, includes:

[0033] Four parallel Transformer branches are constructed on the local feature maps L1, L2, L3, and L4 at four different scales to extract global features at multiple scales:

[0034] G i =Branch i (L i i = 1, 2, 3, 4

[0035] Among them, Branch i This represents the i-th Transformer branch.

[0036] Furthermore, the step of fusing the local feature maps and the global feature maps corresponding to each branch using long skip connections to obtain multiple initial fused feature maps of different scales includes:

[0037] The long skip connection is used to fuse the local feature maps and the global feature maps of the same size in each branch:

[0038] F i =Conv 1×1 (L i +G i i = 1, 2, 3, 4

[0039] Among them, F i For the initial fused feature map, Conv 1×1 This indicates that a 1x1 convolutional layer is used to linearly fuse the local and global feature maps.

[0040] Furthermore, the step of fusing multiple initial fusion feature maps to obtain a target fusion feature map, and then using the segmentation module to obtain a segmentation map, includes:

[0041] The initial fused feature map output from the low-resolution Transformer branch is upsampled using bilinear interpolation.

[0042] The initial fusion feature map is fused by adding feature points one by one.

[0043] Furthermore, the features output from the four Transformer branches are fused:

[0044] F Multi =Conv 1×1 (F1+Up 2× (F2)+Up 4× (F3)+Up 8× (F4)),

[0045] Among them, F Multi For the target fused feature map, Up represents upsampling using bilinear interpolation;

[0046] In the target fusion feature map F Multi The segmentation result is obtained using the segmentation module constructed from a 1×1 convolutional layer:

[0047] S = Conv 1×1 (F Multi )

[0048] Where S is the segmentation map output by the network. The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0049] In the embodiments of this disclosure, the medical image segmentation method based on multiple parallel Transformer branch neural networks described above, on the one hand, utilizes the inherent multi-scale characteristics of convolutional networks to gradually reduce the spatial resolution of the output feature maps through pooling layers or strided convolutional layers, thereby extracting multi-scale local feature maps with different receptive fields and spatial resolutions from sub-residual networks at different stages; then, a Transformer branch network is constructed to acquire global contextual information using a self-attention mechanism. On the other hand, multiple parallel Transformer branch networks are built on the multi-level local feature maps of the convolutional network part to obtain multi-scale global feature maps; finally, long skip connections and large-scale linear upsampling are used to add and fuse the local and global feature maps from the multiple parallel branches to obtain the target fused feature map. The fused target feature map contains multi-scale local and global features, thereby contributing to the accurate segmentation of multi-scale organs. Attached Figure Description

[0050] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0051] Figure 1 This illustration shows a medical image segmentation method based on a neural network with multiple parallel Transformer branches, as shown in an exemplary embodiment of this disclosure.

[0052] Figure 2 This diagram illustrates the overall structure of the network in an exemplary embodiment of this disclosure.

[0053] Figure 3 A flowchart illustrating a medical image segmentation method based on multiple parallel Transformer branch neural networks in an exemplary embodiment of this disclosure is shown.

[0054] Figure 4 The figure shows the test results of the Synapse abdominal multi-organ segmentation dataset in an exemplary embodiment of this disclosure. Detailed Implementation

[0055] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0056] Furthermore, the accompanying drawings are merely illustrative diagrams of embodiments of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0057] This example implementation provides a medical image segmentation method based on a neural network with multiple parallel Transformer branches. (Reference) Figure 1 As shown, the medical image segmentation method based on multiple parallel Transformer branch neural networks may include steps S101 to S105.

[0058] Step S101: Use a convolutional network module to extract local features from the medical image dataset to obtain several local feature maps at different scales. The convolutional network module includes several cascaded residual neural networks ResNet-50.

[0059] Step S102: Construct a multi-head self-attention layer using the self-attention mechanism and build a Transformer branch;

[0060] Step S103: Construct several parallel Transformer branches on each of the cascaded ResNet-50 residual neural networks, and use the multiple Transformer branches to obtain multiple global feature maps of different scales.

[0061] Step S104: Use long skip connections to fuse the local feature maps and global feature maps corresponding to each branch to obtain multiple initial fused feature maps of different scales;

[0062] Step S105: Fuse multiple initial fusion feature maps to obtain a target fusion feature map, and use the segmentation module to obtain a segmentation map.

[0063] The medical image segmentation method based on multiple parallel Transformer branch neural networks described above achieves two main objectives. First, leveraging the inherent multi-scale characteristics of convolutional networks, the spatial resolution of the output feature maps is progressively reduced through pooling layers or strided convolutional layers. This allows for the extraction of multi-scale local feature maps with different receptive fields and spatial resolutions from sub-residual networks at different stages. Then, a Transformer branch network is constructed to utilize a self-attention mechanism to acquire global contextual information. Second, multiple parallel Transformer branch networks are built on top of the multi-level local feature maps in the convolutional network portion, thereby obtaining multi-scale global feature maps. Finally, long skip connections and large-scale linear upsampling are used to fuse the local and global feature maps from the multiple parallel branches to obtain the target fused feature map. The fused target feature map contains multi-scale local and global features, thus contributing to the accurate segmentation of organs at multiple scales.

[0064] Below, we will refer to Figures 1 to 4 The steps of the medical image segmentation method based on multiple parallel Transformer branch neural networks described in this example embodiment will be explained in more detail.

[0065] In step S101, as Figure 2The diagram shows the overall network structure. Because the ResNet-50 network model rapidly reduces the size of the output feature image through strided convolutions and pooling operations in the initial network layers, it helps reduce the computational complexity of the Transformer branch networks. Furthermore, the residual subnetwork uses skip connections to improve information flow, mitigating the gradient vanishing and degradation problems caused by excessive network depth. After removing the pooling layer and fully connected layer after the last residual module, the remaining convolutional network contains one max-pooling layer and 17 residual modules.

[0066] Based on the different feature map sizes output by each module, the convolutional network part is divided into... Figure 1 The diagram shows four stages (stages 1 through 4). Samples from the Synapse dataset are then input into the convolutional network for feature extraction. The output feature map size is reduced by half after each sub-residual network pass. This results in four local feature maps with different spatial resolutions: L1, L2, L3, and L4 (resolutions of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input image, respectively).

[0067] In step S102, a Transformer branch is constructed using a self-attention mechanism. Self-attention was first used in natural language processing and subsequently applied to image processing. It calculates the dependencies between input feature points through matrix dot product operations to obtain global contextual information. The self-attention layer used in this application consists of a matrix dot product layer, a normalization layer, and a feedforward network constructed using fully connected layers, with each Transformer branch constructed from four self-attention layers.

[0068] Unfold the input local feature map into a one-dimensional sequence. Then, three different linear projections are used to compute three linearly transformed feature vectors for Query, Key, and Value from the sequence:

[0069]

[0070]

[0071]

[0072] in, Q is the Query value, K is the Key value, V is the Value value, and D is the Key value. h This represents the feature dimension, where N represents the number of pixels in the feature map. w Q Let w represent the first trainable parameter in the linear mapping. Kw represents the second trainable parameter in the linear mapping. V denoted by the third trainable parameter in the linear mapping, h represents the h-th self-attention head, and Norm represents the normalization operation;

[0073] Calculate attention map A using dot product operation h Then use attention map A h For V h Transformation:

[0074]

[0075] in, SA represents the scaling factor. h For the h-th self-attention head, Softmax represents the normalization function;

[0076] The outputs of a total of H self-attention heads are connected in parallel and projected. Remapping the feature dimensions back

[0077] MSA = W Project [SA 1 ;…;SA H ] T

[0078] The output of the multi-head self-attention module is passed through a normalization layer and a feedforward network (FFN) to output the global feature G:

[0079] G = FFN(Norm(MSA))

[0080] The four self-attention layers constructed through the above steps are connected in series to obtain a Transformer branch used in this application.

[0081] In steps S103 and S104, as follows Figure 2 As shown, based on the convolutional network constructed in step S101, multiple parallel Transformer branch networks as described in step S102 are built. Four local feature maps L1, L2, L3, and L4 with different resolutions obtained from the convolutional network are each input into a Transformer branch to obtain global feature maps G1, G2, G3, and G4 at different scales. Then, long skip connections are used to add and fuse the local and global feature maps of the same size in each parallel branch to obtain fused feature maps F1, F2, F3, and F4.

[0082] Four parallel Transformer branches are built on four local feature maps at different scales (L1, L2, L3, and L4) to extract global features at multiple scales:

[0083] G i =Branch i (L i i = 1, 2, 3, 4

[0084] Among them, Branch i This represents the i-th Transformer branch.

[0085] Long skip connections are used to fuse local and global feature maps of the same size in each branch:

[0086] F i =Conv 1×1 (L i +G i i = 1, 2, 3, 4

[0087] Among them, F i For the initial fused feature map, Conv 1×1 This indicates that a 1x1 convolutional layer is used to linearly fuse the local and global feature maps.

[0088] In step S105, large-scale linear upsampling is used to align and fuse the outputs F1, F2, F3, and F4 from step S104. Bilinear interpolation is used to upsample the feature maps F2, F3, and F4 from the low-resolution Transformer branch outputs to the same size as F1. Then, the features at the four scales are fused using a feature-point-by-feature-point addition method, finally obtaining the output F that fuses multi-scale local and global features. Multi The final fused feature map is used to obtain the segmentation result through a segmentation module constructed from 1×1 convolutional layers.

[0089] The initial fused feature map output from the low-resolution Transformer branch is upsampled using bilinear interpolation; the initial fused feature map is then fused by adding feature points one by one.

[0090] The features output from the four Transformer branches are fused:

[0091] F Multi =Conv 1×1 (F1+Up 2× (F2)+Up 4× (F3)+Up 8× (F4)),

[0092] Among them, F Multi For the target fused feature map, Up represents upsampling using bilinear interpolation;

[0093] In the target fusion feature map F MultiThe segmentation result is obtained using a segmentation module constructed from 1×1 convolutional layers:

[0094] S = Conv 1×1 (F Multi )

[0095] Where S is the segmentation graph that the network finally outputs.

[0096] The constructed neural network model is trained using the dataset.

[0097] like Figure 3 The diagram shows a flowchart of a medical image segmentation method based on a neural network with multiple parallel Transformer branches. This method leverages the inherent multi-level characteristics of convolutional networks, constructing multiple parallel Transformer branches on local features at each level with different receptive fields and spatial resolutions extracted by sub-networks at each stage. This application utilizes the multi-scale characteristics of convolutional networks to construct multiple parallel neural network branches. By using multiple residual convolutional modules to progressively reduce the spatial resolution of the output feature map to obtain local features at different scales, the computational cost of the corresponding Transformer branches can also be reduced. Secondly, the parallel Transformer branches constructed on local features at different scales can obtain global contextual features at the corresponding scales. Finally, long connections are used to fuse local and global features, providing multi-level features for multi-scale target segmentation.

[0098] In one embodiment, using the Synapse dataset of abdominal multi-organ CT image segmentation as an example, the result is as follows: Figure 4 As shown.

[0099] The medical image segmentation method based on multiple parallel Transformer branch neural networks described above achieves two main objectives. First, leveraging the inherent multi-scale characteristics of convolutional networks, the spatial resolution of the output feature maps is progressively reduced through pooling layers or strided convolutional layers. This allows for the extraction of multi-scale local features with different receptive fields and spatial resolutions from sub-residual networks at different stages. Then, a Transformer network is constructed to utilize self-attention mechanisms to acquire global contextual information. Second, multiple parallel Transformer branch networks are built upon the multi-level local features of the convolutional network portion to obtain multi-scale global features. Finally, long skip connections and large-scale linear upsampling are used to fuse the local and global feature maps from the multiple parallel branches. The fused output contains multi-scale local and global features, thus contributing to the accurate segmentation of organs at multiple scales.

[0100] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, "a plurality of" means two or more, unless otherwise explicitly specified.

[0101] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0102] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. A medical image segmentation method based on a plurality of parallel Transformer branch neural networks, characterized in that, The method includes: A convolutional network module is used to extract local features from a medical image dataset to obtain several local feature maps at different scales. The convolutional network module includes several cascaded residual neural networks ResNet-50. A multi-head self-attention layer is constructed using the self-attention mechanism, and a Transformer branch is built; Several parallel Transformer branches are built on each of the cascaded ResNet-50 residual neural networks, and multiple global feature maps of different scales are obtained using the multiple Transformer branches. The local feature maps and global feature maps corresponding to each branch are fused using long skip connections to obtain multiple initial fused feature maps of different scales. Multiple initial fusion feature maps are fused to obtain a target fusion feature map, and a segmentation map is obtained using a segmentation module.

2. The method of claim 1, wherein, The step of using a convolutional network module to extract local features from a medical image dataset to obtain several local feature maps at different scales, wherein the convolutional network module includes several cascaded ResNet-50 residual neural networks, includes: A network was built on top of ResNet-50 to obtain the residual neural network ResNet-50; wherein, by removing the pooling layer and fully connected layer after the last residual module, the simplified convolutional network part contains 17 residual modules and 1 max pooling layer. The size of the local feature maps output by each of the ResNet-50 residual neural networks differs.

3. The method of claim 2, wherein, The convolution network module includes 4 residual neural networks ResNet-50, and 4 local feature maps of different scales, i.e., a first local feature map , a second local feature map , a third local feature map , and a fourth local feature map , can be obtained.

4. The method of claim 3, wherein, The steps of constructing a multi-head self-attention layer using a self-attention mechanism and building Transformer branches include: The self-attention layer consists of a matrix dot product layer, a normalization layer, and a feedforward network built with a fully connected layer. The Transformer branch is formed by connecting four self-attention layers in series.

5. The method of claim 4, wherein, The step of constructing a multi-head self-attention layer using a self-attention mechanism and building Transformer branches also includes: The input local feature map is unfolded into a one-dimensional sequence. Then, using three different linear projections, the first feature vector Query, the second feature vector Key, and the third feature vector Value, after linear transformation, are calculated from the sequence: in, , Q For Query value, K For the key value, V For the Value, Represents the feature dimension. This indicates the number of pixels in the feature map. , Denotes the first trainable parameter in a linear mapping. Denotes the second trainable parameter in a linear mapping. This represents the third trainable parameter in a linear mapping. This represents the h-th self-attention head. This indicates a normalization operation; Calculating an attention map using dot product operations Using the attention map Transforming : in, Indicates the scaling factor. This is the output of the h-th self-attention head. Represents the normalization function; The outputs of the H self-attention heads in total are concatenated in parallel and projected The feature dimension is remapped back to : The output of the multi-head self-attention module is passed through a normalization layer and a feedforward network The global feature is outputted after : The self-attention layers are connected in series to obtain a Transformer branch.

6. The method of claim 5, wherein, The step of constructing several parallel Transformer branches on each of the cascaded ResNet-50 residual neural networks and obtaining multiple global feature maps of different scales using the multiple Transformer branches includes: The local feature maps at four different scales , , and Four parallel Transformer branches are built on top to extract global features at multiple scales: wherein, represents the first i Transformer branch.

7. The method of claim 6, wherein, The step of fusing the local feature maps and the global feature maps corresponding to each branch using long skip connections to obtain multiple initial fused feature maps of different scales includes: The long skip connection is used to fuse the local feature maps and the global feature maps of the same size in each branch: in, F i This is the initial fused feature map. This indicates that a 1x1 convolutional layer is used to linearly fuse the local and global feature maps.

8. The method of claim 7, wherein, The step of fusing multiple initial fusion feature maps to obtain a target fusion feature map, and then using the segmentation module to obtain a segmentation map, includes: The initial fused feature map output from the low-resolution Transformer branch is upsampled using bilinear interpolation. The initial fusion feature map is fused by means of point-by-point addition.

9. The method of claim 8, wherein, The features output by the four Transform branches are fused: wherein, is a target fused feature map, denotes upsampling by bilinear interpolation. In the target fusion feature map Above, the segmentation result is obtained by using the segmentation module constructed by the 1x1 convolution layer. S is a segmentation map output by the network finally.

Images