Medical Image Segmentation Method and Related Devices Based on Transformer Context Information Fusion

Abstract

The present application provides a medical image segmentation method and related equipment based on Transformer context information fusion, wherein the Transformer includes multiple encoding layers and symmetrical decoding layers; the method includes: forming a contraction path for each group of medical images through multiple encoding layers, presetting a multi-level branch local context extraction model in the encoding layer, and obtaining local context relationships through spatial separation, void convolution, and residual connection; forming an expansion path through multiple decoding layers, presetting a multi-scale context information fusion module in the decoding layer, using self-attention calculation and cross-attention calculation to obtain the global context relationships of the current scale and multi-scale medical images respectively, combining the information of the two for extraction and fusion to obtain multi-scale context relationships; segmenting the medical image based on the above two context relationships to obtain a segmentation result. Through multi-scale context relationships and local context relationships, medical image segmentation can be effectively performed, thereby effectively identifying anatomical structures in medical images.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a medical image segmentation method and related equipment based on Transformer context information fusion. Background Art

[0002] With the rapid development of artificial intelligence and computer vision technology, image segmentation algorithms are becoming increasingly mature. In addition, with the advancement of medical knowledge and medical devices, more and more medical imaging methods have become auxiliary tools for diagnosis and treatment. The size and number of medical images have increased dramatically, and traditional manual segmentation and manual labeling methods are gradually unable to meet the growing demand.

[0003] Against this backdrop, segmentation methods suitable for medical images have emerged. Research has been conducted on high-performance neural network-based segmentation methods. Some studies have proposed that the Transformer structure can compensate for the shortcomings of CNN in terms of long-distance dependencies in the image field. However, experimental results have shown that using only the Transformer is insufficient to extract fine-grained local contextual information in images. To address this issue, other studies have proposed methods that fuse convolutional neural networks with the Transformer. However, these methods still suffer from the inflexibility of fusing the contextual information of the convolutional layer and the Transformer, the destruction of the continuity and transferability of features in the fully connected layer, and the inability to effectively utilize multi-scale contextual information. Summary of the Invention

[0004] In view of this, the purpose of this application is to propose a method to overcome the above problems or at least partially solve the above problems.

[0005] Based on the above purpose, the first aspect of the present application provides a medical image segmentation method based on Transformer context information fusion, wherein the Transformer includes multiple encoding layers and symmetrical decoding layers;

[0006] The method comprises:

[0007] Acquiring a medical image and grouping pixels of the medical image;

[0008] Performing contraction path downsampling on each group of the medical images through the multiple coding layers to obtain medical images of different scales, and extracting relationships between pixels in a local receptive field and surrounding pixels of the medical images of different scales through spatial separation, dilated convolution, and residual connection to obtain local contextual relationships;

[0009] Determine a minimum-scale medical image, perform dilation path upsampling on the minimum-scale medical image through each decoding layer, and splice the current decoding layer with a symmetrical encoding layer to obtain a splicing result; and perform self-attention calculation on the splicing result to obtain a global context relationship of the medical image at the current scale;

[0010] splicing each of the encoding layers with each of the decoding layers, and fusing the medical images of different scales with the global image of the medical image at the current scale using cross-attention calculation to obtain a global contextual relationship between the medical image at the current scale and the medical images at different scales;

[0011] splicing the global context relationship of the medical image at the current scale and the global context relationship of the multi-scale medical image to obtain a multi-scale context relationship;

[0012] The medical image is segmented based on the multi-scale contextual relationship and the local contextual relationship to obtain a segmentation result.

[0013] Optionally, each encoding layer is preset with a multi-level branch local context extraction model as a feedforward neural network in the encoding layer for feature extraction;

[0014] The multi-level branch local context extraction model includes a main branch, a first branch, a second branch, a third branch and a fourth branch;

[0015] The relationship between pixels in the local receptive field and surrounding pixels of the medical images of different scales is extracted through spatial separation, void convolution, and residual connection to obtain local contextual relationships, including:

[0016] Using 1×1 convolution to compress the number of channels of the medical image input to the main branch to obtain a first feature map, and inputting the first feature map into the first branch, the second branch, the third branch, and the fourth branch respectively;

[0017] Use spatial separation convolution to extract vertical and horizontal boundary information from the first feature map input to the first branch to obtain a second feature map;

[0018] splicing the first branch, the second branch, the third branch, and the fourth branch so that the first branch, the second branch, the third branch, and the fourth branch are sequentially connected in series;

[0019] Using dilated convolutions with 3×3 convolution kernels of different dilation rates, feature extraction is performed on the second branch, the third branch, and the fourth branch, respectively, to obtain a third feature map, a fourth feature map, and a fifth feature map, respectively; wherein the dilation rate increases from the second branch to the fourth branch;

[0020] The second feature map, the third feature map, the fourth feature map and the fifth feature map are merged to obtain fused local information, and then residual calculation is performed with the medical image input into the main branch to obtain the local context relationship.

[0021] Optionally, the coding layer includes a first-layer Transformer block, a second-layer Transformer block, a third-layer Transformer block, and a fourth-layer Transformer block;

[0022] The first-layer Transformer block, the second-layer Transformer block, the third-layer Transformer block, and the fourth-layer Transformer block are connected in series in sequence, and the downsampling resolution of the contraction path from the first-layer Transformer block to the fourth-layer Transformer block decreases in sequence.

[0023] Optionally, the medical images of different scales include a first-scale medical image, a second-scale medical image, a third-scale medical image, and a minimum-scale medical image;

[0024] The grouping of pixels of the medical image comprises:

[0025] Divide the pixels of the medical image into 7×7 two-dimensional groups using a two-dimensional convolution kernel, and input the grouping results into the first-layer Transformer block;

[0026] Divide the pixels of the first-scale medical image output by the first-layer Transformer block into 3×3 two-dimensional groups using a two-dimensional convolution kernel, and input the grouping results into the second-layer Transformer block;

[0027] Divide the pixels of the second-scale medical image output by the second-layer Transformer block into 3×3 two-dimensional groups using a two-dimensional convolution kernel, and input the grouping results into the third-layer Transformer block;

[0028] The pixels of the third-scale medical image output by the third-layer Transformer block are divided into 3×3 two-dimensional groups through a two-dimensional convolution kernel, and the grouping results are input into the fourth-layer Transformer block.

[0029] Optionally, the decoding layer includes a first-layer decoder block, a second-layer decoder block, a third-layer decoder block, and a fourth-layer decoder block; the fourth-layer decoder block, the third-layer decoder block, the second-layer decoder block, and the first-layer decoder block are sequentially connected in series, and the sampling resolution increases sequentially along the expansion path from the fourth-layer decoder block to the first-layer decoder block;

[0030] The splicing results include: a first splicing result, a second splicing result and a third splicing result;

[0031] The determining of the minimum-scale medical image, performing dilation path upsampling on the minimum-scale medical image through each decoding layer, and splicing the current decoding layer with the symmetrical encoding layer to obtain a splicing result; and performing self-attention calculation on the global image to obtain a global context relationship of the medical image at the current scale, including:

[0032] performing self-attention calculation on the minimum-scale medical image output by the fourth-layer Transformer block in the fourth-layer Decoder block to obtain a global context relationship of the minimum-scale medical image;

[0033] splicing, in the third-layer Decoder block, the third-scale medical image output by the fourth-layer Decoder block and the third-scale medical image output by the third-layer Transformer block to obtain a third splicing result, and performing self-attention calculation on the third splicing result to obtain a global context relationship of the third-scale medical image;

[0034] splicing, in the second-layer Decoder block, the second-scale medical image output by the third-layer Decoder block and the second-scale medical image output by the second-layer Transformer block to obtain a second splicing result, and performing self-attention calculation on the second splicing result to obtain a global context relationship of the second-scale medical image;

[0035] In the first-layer Decoder block, the first-scale medical image output by the second-layer Decoder block and the first-scale medical image output by the first-layer Transformer block are spliced to obtain a first splicing result, and self-attention calculation is performed on the first splicing result to obtain a global context relationship of the first-scale medical image.

[0036] Optionally, each encoding layer is spliced with each decoding layer, and the medical images of different scales are fused with the global image of the medical image at the current scale using cross-attention calculation to obtain a global contextual relationship of the multi-scale medical image, including:

[0037] In the fourth-layer decoder block, cross-attention calculations are performed on the first-scale medical image, the second-scale medical image, and the third-scale medical image, respectively, with the global image of the minimum-scale medical image to obtain a global contextual relationship between the minimum-scale medical image and the first-scale medical image, the second-scale medical image, and the third-scale medical image;

[0038] In the third-layer decoder block, cross-attention calculations are performed on the first-scale medical image, the second-scale medical image, the third-scale medical image, and the minimum-scale medical image, respectively, with a global image of the third-scale medical image to obtain a global contextual relationship between the third-scale medical image and the first-scale medical image, the second-scale medical image, the third-scale medical image, and the minimum-scale medical image;

[0039] In the second-layer decoder block, cross-attention calculation is performed on the first-scale medical image, the second-scale medical image, the third-scale medical image, and the minimum-scale medical image, respectively, with the global image of the second-scale medical image to obtain a global contextual relationship between the second-scale medical image and the first-scale medical image, the second-scale medical image, the third-scale medical image, and the minimum-scale medical image;

[0040] In the first-layer decoder block, cross-attention calculation is performed on the first-scale medical image, the second-scale medical image, the third-scale medical image, and the minimum-scale medical image, respectively, with the global image of the first-scale medical image to obtain the first-scale medical image and the first-scale medical image, the second-scale medical image, the third-scale medical image, and the minimum-scale medical image.

[0041] Optionally, splicing the global context relationship of the medical image at the current scale and the global context relationship of the multi-scale medical image to obtain the multi-scale context relationship includes:

[0042] splicing the global context relationship of the minimum-scale medical image and the global context relationship between the minimum-scale medical image and the first-scale medical image, the second-scale medical image, and the third-scale medical image to obtain a multi-scale context relationship of the minimum-scale medical image;

[0043] splicing the global context relationship of the third-scale medical image and the global context relationship between the third-scale medical image and the first-scale medical image, the second-scale medical image, the third-scale medical image, and the minimum-scale medical image to obtain a multi-scale context relationship of the third-scale medical image;

[0044] splicing the global context relationship of the second-scale medical image and the global context relationship of the second-scale medical image with the first-scale medical image, the second-scale medical image, the third-scale medical image, and the minimum-scale medical image to obtain a multi-scale context relationship of the second-scale medical image;

[0045] The global context relationship of the first-scale medical image and the global context relationships of the first-scale medical image, the first-scale medical image, the second-scale medical image, the third-scale medical image, and the minimum-scale medical image are stitched together to obtain a multi-scale context relationship of the first-scale medical image.

[0046] Optionally, using spatially separated convolution to extract vertical and horizontal boundary information from the first feature map input to the first branch to obtain a second feature map, including:

[0047] Convolving the first feature map using a first path first vertically and then horizontally to obtain a first result;

[0048] Convolving the first feature map using a second horizontal and vertical path to obtain a second result;

[0049] Concatenating the first result and the second result to obtain a second feature map;

[0050] The second feature map is expressed as:

[0051] F1=concat[χ(conv h (conv v (F0))),χ(conv v (conv h (F0)))],

[0052] Among them, conv v and conv h They are 3×1 and 1×3 vertical convolution and horizontal convolution respectively, F0 is the first feature map, and χ is the Batch Normalization and ReLU activation function.

[0053] In a second aspect of the present application, a medical image segmentation device based on Transformer context information fusion is provided, comprising:

[0054] A grouping module, for grouping pixels of a medical image;

[0055] The encoding module is used to perform contraction path downsampling on the medical image to obtain medical images of different scales;

[0056] A local context extraction module is used to extract the relationship between pixels in the local receptive field and surrounding pixels of the medical images of different scales through spatial separation, void convolution and residual connection to obtain local context relationship;

[0057] A decoding module is used to sequentially perform dilation path upsampling on the minimum-scale medical image, and splice the current decoding layer with the symmetrical encoding layer to obtain a splicing result; perform self-attention calculation on the global image to obtain the global context relationship of the medical image at the current scale;

[0058] a multi-scale context extraction module, configured to connect each encoding layer and each decoding layer, and utilize cross-attention calculation to combine the medical images of different scales with the global image of the medical image at the current scale with the global context relationship to obtain a multi-scale context relationship; and to splice the global context relationship of the medical image at the current scale with the global context relationship of the multi-scale medical image to obtain a multi-scale context relationship;

[0059] A segmentation module is used to segment the medical image based on the multi-scale context relationship and the local context relationship to obtain a segmentation result.

[0060] According to a third aspect of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-described method when executing the program.

[0061] From the above, it can be seen that the present application provides a medical image segmentation method and related equipment based on Transformer context information fusion. Through multiple coding layers, a contraction path is formed for each group of medical images to reduce the resolution and downsample to obtain medical images of different scales. A multi-level branch local context extraction model is preset in the coding layer. The local context relationship of medical images of different scales can be effectively obtained through separation and void convolution and residual connection. At the same time, each layer of Transformer block and Decoder block is connected through a multi-scale context extraction model. Self-attention calculation and cross-attention calculation are used to obtain the global context relationship of the current scale and multi-scale medical images respectively. The information of the two is combined for extraction and fusion to obtain a multi-scale context relationship, which can effectively refer to the medical image segmentation, thereby effectively identifying the anatomical structure in the medical image.

[0062] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in this application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are merely embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0064] Figure 1 This is a flowchart of an embodiment of the present application;

[0065] Figure 2 A schematic diagram of the technical solution of an embodiment of the present application;

[0066] Figure 3 A schematic diagram of multi-level branch local context extraction according to an embodiment of the present application;

[0067] Figure 4 This is a schematic diagram of multi-scale context information fusion according to an embodiment of the present application;

[0068] Figure 5 Schematic diagram of the main branch spatial separation convolution in an embodiment of the present application;

[0069] Figure 6 Schematic diagram of a Transformer-based medical image segmentation device according to an embodiment of the present application;

[0070] Figure 7This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0071] In order to make the objectives, technical solutions and advantages of this application more clear, this application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0072] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should have the usual meanings understood by people with ordinary skills in the field to which this application belongs. The "first", "second" and similar words used in the embodiments of the present application do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0073] In related technologies, the attention-based Transformer is good at modeling long-distance dependencies, but the global matrix operation causes it to ignore detailed local contextual information and does not take into account the local relationships and structural information within the group, which is very important for medical images.

[0074] Meanwhile, conventional convolutional networks and Transformer architectures in related technologies extract contextual information at the same scale. By stacking layers of feature pyramid networks, they learn a hierarchical structure that gradually expands channel information capacity while reducing spatial resolution to obtain multi-scale features. The sequential network topology they follow lacks sufficient cross-scale interaction, so the generated multi-scale contextual information is often not fully utilized, and high-quality, high-resolution representations cannot be generated.

[0075] That is to say, the research on the medical image segmentation method mentioned above, in related technologies, cannot fully extract local context information, and the current Transformer combined with convolution method cannot make good use of multi-scale context information.

[0076] For this problem, refer to Figure 1The first aspect of the present application provides a medical image segmentation method based on Transformer context information fusion, wherein the Transformer includes multiple encoding layers and symmetrical decoding layers, and the method includes:

[0077] S101: Acquire a medical image and group pixels of the medical image.

[0078] In this step, before the medical image enters the corresponding encoding layer, that is, the Transformer block, in order to reduce the calculation scope and complexity, that is, to reduce the amount of calculation, so that each layer of the Transformer block can more quickly obtain the medical image of the scale corresponding to the resolution of the corresponding contraction path, the pixels of the medical image input to the Transformer block are grouped. The grouping scheme is used to control the attention calculation scope and complexity:

[0079] The calculation method of the multi-head self-attention mechanism is shown in formula (1)

[0080]

[0081] Where Q, K, and V are the query matrix, index matrix, and content matrix respectively.

[0082] When the whole image is added to the attention calculation without grouping, that is, when the medical image is added to the whole image without grouping, the correlation between each pixel and other pixels in the whole image is calculated. The computational complexity is shown in formula (2)

[0083] FLOPs1 = 4HWC 2 +2H 2 W 2 C 2 (2)

[0084] Here, H, W, and C refer to the length, width, and number of channels of the two-dimensional medical image, respectively.

[0085] When the image is divided into N×N binary groups, the complexity of calculating the relationship between the groups is shown in formula (3)

[0086]

[0087] From the above calculations, we can see that after grouping, the relationship between pixels in the receptive field and surrounding pixels is analyzed, which has lower complexity and more efficient calculation.

[0088] Furthermore, pixels of the medical image are grouped, including:

[0089] Divide the pixels of the medical image into 7×7 two-dimensional groups using a two-dimensional convolution kernel, and input the grouping results into the first-layer Transformer block;

[0090] Divide the pixels of the first-scale medical image output by the first-layer Transformer block into 3×3 two-dimensional groups using a two-dimensional convolution kernel, and input the grouping results into the second-layer Transformer block;

[0091] Divide the pixels of the second-scale medical image output by the second-layer Transformer block into 3×3 two-dimensional groups using a two-dimensional convolution kernel, and input the grouping results into the third-layer Transformer block;

[0092] The pixels of the third-scale medical image output by the third-layer Transformer block are divided into 3×3 two-dimensional groups through a two-dimensional convolution kernel, and the grouping results are input into the fourth-layer Transformer block.

[0093] Specifically, the medical image is divided into 7×7 two-dimensional groups using a two-dimensional convolution kernel with a side length of 7, a stride of 4, and a padding of 3, and the relationship between different groups is established through overlapping grouping, and the grouping results are input into the first-layer Transformer block;

[0094] Divide the first-scale medical image output by the first-layer Transformer block into 3×3 two-dimensional groups using a two-dimensional convolution kernel with a side length of 3, a stride of 2, and a padding of 1, introduce the relationship between different groups by overlapping groups, and input the grouping results into the second-layer Transformer block;

[0095] Divide the second-scale medical image output by the second-layer Transformer block into 3×3 two-dimensional groups using a two-dimensional convolution kernel with a side length of 3, a stride of 2, and a padding of 1, introduce the relationship between different groups by overlapping groups, and input the grouping results into the third-layer Transformer block;

[0096] The third-scale medical image output by the third-layer Transformer block is divided into 3×3 two-dimensional groups by a two-dimensional convolution kernel with a side length of 3, a stride of 2, and a padding of 1, and the relationship between different groups is introduced by overlapping grouping, and the grouping results are input into the fourth-layer Transformer block.

[0097] That is to say, a medical image is first obtained, and the medical image is divided into 7×7 two-dimensional groups by a two-dimensional convolution kernel with a side length of 7, a stride of 4, and a padding of 3. The relationship between different groups is established through overlapping grouping, and the grouping result is input into the first-layer Transformer block to obtain a first-scale medical image. It can be understood that the local context relationship of the first-scale medical image is also obtained. Similarly, the minimum-scale medical image output by the fourth-layer Transformer block and the local context relationship of the minimum-scale medical image are obtained.

[0098] Introducing the relationship between different groups through overlapping grouping can be understood as, when grouping a certain pixel of a medical image, overlapping the above-mentioned certain pixel in each group, thereby introducing the relationship between different groups through the relationship between the certain pixel and the medical image in each group.

[0099] S102. Perform contraction path downsampling on each group of medical images through multiple coding layers to obtain medical images of different scales. Extract the relationship between pixels in the local receptive field and surrounding pixels of the medical images of different scales through spatial separation, void convolution, and residual connection to obtain local contextual relationships.

[0100] In some embodiments, reference Figure 2 , the encoding layer includes the first layer Transformer block, the second layer Transformer block, the third layer Transformer block and the fourth layer Transformer block;

[0101] The first-layer Transformer block, the second-layer Transformer block, the third-layer Transformer block, and the fourth-layer Transformer block are connected in series, and the downsampling resolution of the contraction path from the first layer Transformer block to the fourth layer Transformer block decreases successively. It can be understood that the process of downsampling in the contraction path from the first layer Transformer block to the fourth layer Transformer block is actually the process of downsampling the resolution of the medical image, so that medical images of different scales are extracted in each layer Transformer block. In other words, the scale of the medical image is determined by the sampling resolution of the medical image on the sampling path of each Transformer block during downsampling. From the first layer Transformer block to the fourth layer Transformer block, the relationship between the pixels in the local receptive field and the surrounding pixels of the medical image is gradually extracted from the shallow layer to the deep layer. Each layer Transformer block extracts the relationship between the pixels in the local receptive field and the surrounding pixels of the medical image at the corresponding resolution, and obtains the local context relationship at the corresponding resolution.

[0102] It can be understood that in this application, for the convenience of description, the coding layer in this application includes a first-layer Transformer block, a second-layer Transformer block, a third-layer Transformer block and a fourth-layer Transformer block, a total of four layers of Transformer blocks. However, in actual applications, the first, second, etc. in the text are not used as a restriction on the order of quantity. That is to say, there can be more or fewer Transformer blocks depending on the actual application.

[0103] Further, refer to Figure 2 In some embodiments, the medical images of different scales include a first-scale medical image, a second-scale medical image, a third-scale medical image, and a minimum-scale medical image. It is understood that, in the first to fourth-layer Transformer blocks, the medical images are correspondingly sampled with reduced resolution under a contraction path to obtain local context relationships, that is, local context relationships of the first-scale medical image, the second-scale medical image, the third-scale medical image, and the minimum-scale medical image are obtained.

[0104] Among them, the acquired medical image is input into the first layer Transformer block to obtain a first-scale medical image, the first-scale medical image is input into the second layer Transformer block to obtain a second-scale medical image, the second-scale medical image is input into the third layer Transformer block to obtain a third-scale medical image, and the third-scale medical image is input into the fourth layer Transformer block to obtain a minimum-scale medical image.

[0105] Further, refer to Figure 3 ,Each encoding layer presets a multi-level branch local context extraction model as the feedforward neural network in the encoding layer for feature extraction;

[0106] The multi-level branch local context extraction model includes a main branch, a first branch, a second branch, a third branch and a fourth branch;

[0107] Through spatial separation, dilated convolution, and residual connection, the relationship between pixels in the local receptive field and surrounding pixels of medical images of different scales is extracted to obtain local contextual relationships, including:

[0108] Taking the first layer Transformer block as an example, refer to Figure 3 、 Figure 5 , use 1×1 convolution to compress the number of channels of the medical image input to the main branch to obtain the first feature map, such as Figure 3 F0 in, the first feature map is input into the first branch, the second branch, the third branch and the fourth branch respectively;

[0109] It should be noted that in the second-layer Transformer block, the third-layer Transformer block, and the fourth-layer Transformer block, 1×1 convolution is used to compress the channel books of the first-scale medical image, the second-scale medical image, and the third-scale medical image of the input main branch to obtain the first feature map.

[0110] Use spatial separation convolution to extract vertical and horizontal boundary information from the first feature map of the first input branch to obtain the second feature map;

[0111] Further, convolution is performed on F0 using a first path that is first vertical and then horizontal to obtain a first result;

[0112] Convolve F0 using the second path horizontally and vertically to obtain the second result;

[0113] splicing the first result and the second result of the two parallel paths to obtain a second feature map;

[0114] The second feature map is expressed as:

[0115] F1=concat[χ(conv h (conv v (F0))),χ(conv v (conv h (F0)))],

[0116] Among them, conv v and conv h They are 3×1 and 1×3 vertical convolution and horizontal convolution respectively, F0 is the first feature map, and χ is the Batch Normalization and ReLU activation function.

[0117] Afterwards, the first branch, the second branch, the third branch and the fourth branch are spliced so that the first branch, the second branch, the third branch and the fourth branch are connected in series in sequence; that is, in addition to obtaining the first feature map F0 as input, the second branch also obtains the second feature map F1 output by the first branch as input, which is conducive to obtaining small-scale local information and obtaining the correlation between different scales.

[0118] Using dilated convolutions with 3×3 convolution kernels of different dilation rates, feature extraction is performed on the second branch, third branch, and fourth branch, respectively, to obtain the third feature map, fourth feature map, and fifth feature map, respectively; the dilation rate increases from the second branch to the fourth branch. It can be understood that within limited computing resources, without increasing the number or size of convolution kernels, dilated convolution can increase the receptive field without losing resolution, allowing each convolution to contain a larger range of information, that is, each convolution contains as much medical image information within the receptive field as possible. The third feature map, the fourth feature map, and the fifth feature map are expressed as:

[0119] F i =χ(conv i (concat[F0,F i-1 ]))

[0120] The second, third, fourth and fifth feature maps are combined to obtain the fused local information, which is then combined with the medical image input to the main branch for residual calculation to obtain the local context relationship. The fused local information is expressed as:

[0121] F=concat[F0,...,F i-1 ],

[0122] The local context relationship is expressed as:

[0123]

[0124] Among them, F in is the medical image of the current scale after grouping into the current Transformer block.

[0125] S103: Determine a minimum-scale medical image, perform dilation path upsampling on the minimum-scale medical image through each decoding layer, and concatenate the current decoding layer with the symmetrical encoding layer to obtain a concatenated result; perform self-attention calculation on the concatenated result to obtain a global context of the medical image at the current scale. The concatenated results include: a first concatenated result, a second concatenated result, and a third concatenated result.

[0126] Specifically, refer to Figure 2 、 Figure 3 The decoding layer includes a first-layer Decoder block, a second-layer Decoder block, a third-layer Decoder block, and a fourth-layer Decoder block; the fourth-layer Decoder block, the third-layer Decoder block, the second-layer Decoder block, and the first-layer Decoder block are connected in series in sequence, and the sampling resolution increases in sequence along the expansion path from the fourth-layer Decoder block to the first-layer Decoder block.

[0127] It can be understood that the decoding layer is set symmetrically with the coding layer, and the process of upsampling in the expansion path of the decoding layer is also symmetrical with the process of downsampling in the contraction path of the coding layer. That is to say, the process of upsampling in the decoding layer is the layer-by-layer recovery of the reduced downsampling resolution.

[0128] Specifically, refer to Figure 4 , in the fourth layer Decoder block, the minimum scale medical image output by the fourth layer Transformer block is subjected to self-attention calculation to obtain the global context relationship of the minimum scale medical image;

[0129] In the third-layer Decoder block, the third-scale medical image output by the fourth-layer Decoder block and the third-scale medical image output by the third-layer Transformer block are spliced to obtain a third splicing result, and self-attention calculation is performed on the third splicing result to obtain the global context relationship of the third-scale medical image;

[0130] In the second-layer Decoder block, the second-scale medical image output by the third-layer Decoder block and the second-scale medical image output by the second-layer Transformer block are spliced to obtain a second splicing result, and self-attention calculation is performed on the second splicing result to obtain the global context relationship of the second-scale medical image;

[0131] In the first-layer Decoder block, the first-scale medical image output by the second-layer Decoder block and the first-scale medical image output by the first-layer Transformer block are spliced to obtain a first splicing result, and self-attention calculation is performed on the first splicing result to obtain the global context relationship of the first-scale medical image.

[0132] Here, the fourth-layer Decoder block directly obtains the minimum-scale medical image output by the fourth-layer Transformer block, i.e. x3, and performs self-attention calculation on the minimum-scale medical image to obtain the global contextual relationship of the minimum-scale medical image. In addition to obtaining the third-scale medical image output after upsampling the fourth-layer Decoder expansion path, the third-layer Decoder block also splices the third-scale medical image output by the third-layer Transformer block to obtain the third splicing result, which is The self-attention calculation is performed on the third splicing result to obtain the global contextual relationship of the third-scale medical image. It can be understood that, similarly, the second-layer Decoder block obtains the second-scale medical image output after upsampling the third-layer Decoder expansion path, and simultaneously splices the second-scale medical image output by the second-layer Transformer block to obtain the second splicing result, which is The second splicing result is subjected to self-attention calculation to obtain the global contextual relationship of the second-scale medical image. Similarly, the first-layer Decoder block is also the same, and the global contextual relationship of the first-scale medical image is obtained. It can be understood that what is obtained from the third-layer Decoder block to the first-layer Decoder block is the splicing result of the previous layer, which is restored to the current-scale medical image through the expansion path. That is, the third-layer Decoder block obtains the output result of the fourth-layer Decoder block and splices it with the third-layer Transformer block to obtain the third splicing result. The second-layer Decoder block obtains the third splicing result output by the third-layer Decoder block and then splices it with the second-layer Transformer block to obtain the second splicing result. The first-layer Decoder block obtains the second splicing result output by the second-layer Decoder block and then splices it with the first-layer Transformer block to obtain the first splicing result. That is, the subsequent Decoder blocks obtain the output of the previous-layer Decoder block respectively. in The calculation formula is expressed as:

[0133]

[0134] where x i It is the current-scale medical image output by the encoding layer (Transformer block) symmetrical to the current decoding layer (Decoder block).

[0135] S104. Each encoding layer is spliced with each decoding layer, and the medical images of different scales are fused with the global image of the medical image at the current scale by using cross-attention calculation to obtain the global contextual relationship between the medical image at the current scale and the medical images of different scales.

[0136] Specifically, refer to Figure 2 、 Figure 3 In the fourth-layer Decoder block, the first-scale medical image x0, the second-scale medical image x1, and the third-scale medical image x2 are respectively subjected to cross-attention calculation with the global image of the minimum-scale medical image x3 to obtain the global contextual relationship between the minimum-scale medical image x3 and the first-scale medical image x0, the second-scale medical image x1, and the third-scale medical image x2;

[0137] In the third layer Decoder block, the first scale medical image x0, the second scale medical image x1, the third scale medical image x2 and the minimum scale medical image x3 are respectively combined with the third splicing result Perform cross-attention calculation to obtain the global contextual relationship between the third-scale medical image x2 and the first-scale medical image x0, the second-scale medical image x1, the third-scale medical image x2, and the minimum-scale medical image x3;

[0138] In the second layer Decoder block, the first scale medical image x0, the second scale medical image x1, the third scale medical image x2 and the minimum scale medical image x3 are respectively combined with the second splicing result Perform cross-attention calculation to obtain the global contextual relationship between the second-scale medical image x1 and the first-scale medical image x0, the second-scale medical image x1, the third-scale medical image x2, and the minimum-scale medical image x3;

[0139] In the first-layer Decoder block, the first-scale medical image x0, the second-scale medical image x1, the third-scale medical image x2 and the minimum-scale medical image x3 are respectively subjected to cross-attention calculation with the global image of the first-scale medical image x0 to obtain the global contextual relationship between the first-scale medical image x0 and the first-scale medical image x0, the second-scale medical image x1, the third-scale medical image x2 and the minimum-scale medical image x3.

[0140] refer to Figure 2 、 Figure 4 , using cross attention calculation, in the first layer Decoder block, the minimum scale medical image x3 after self-attention calculation is used as Q (query matrix), and x0, x1, x2 are used as K (index matrix) and V (content matrix) after matrix operation to perform cross attention calculation with Q respectively; in other Decoders, the self-attention calculation is used as As Q, x0, x1, x2, and x3 are used as K and V after matrix operation to perform cross-attention calculation with Q. Through cross-attention calculation, the correlation between the medical image of the current scale and the feature maps of different scales in the encoding layer is obtained, thereby enhancing the relationship of multi-scale context information.

[0141] S105 : Splicing the global context relationship of the medical image at the current scale and the global context relationship of the multi-scale medical image to obtain a multi-scale context relationship.

[0142] Specifically, the global context relationship of the third-scale medical image and the global context relationship of the third-scale medical image with the first-scale medical image, the second-scale medical image, the third-scale medical image, and the minimum-scale medical image are stitched together to obtain a multi-scale context relationship of the third-scale medical image;

[0143] splicing the global context relationship of the second-scale medical image and the global context relationship of the second-scale medical image with the first-scale medical image, the second-scale medical image, the third-scale medical image, and the minimum-scale medical image to obtain a multi-scale context relationship of the second-scale medical image;

[0144] The global context relationship of the first-scale medical image and the global context relationship between the first-scale medical image and the first-scale medical image, the second-scale medical image, the third-scale medical image and the minimum-scale medical image are stitched together to obtain a multi-scale context relationship of the first-scale medical image.

[0145] It should be noted that a horizontal branch local context extraction model is also preset in the encoding layer, which is used as a feedforward neural network for further fusion calculation to obtain the multi-scale context relationship of the current scale medical image.

[0146] S106 : Segment the medical image based on the multi-scale contextual relationship and the local contextual relationship to obtain a segmentation result.

[0147] Through the above description, it can be understood that the multi-level branch local context extraction model preset in the first-layer Transformer block divides the medical image into 7×7 two-dimensional groups through a two-dimensional convolution kernel with a side length of 7, a step length of 4, and a padding of 3, and then performs local relationship extraction to obtain a first-scale medical image. The first-scale medical image is divided into 3×3 two-dimensional groups through a two-dimensional convolution kernel with a side length of 3, a step length of 2, and a padding of 1, and enters the multi-level branch local context extraction model preset in the second-layer Transformer block to perform local relationship extraction on the grouped first-scale medical image, and so on.

[0148] Through the above-mentioned medical image segmentation method based on Transformer context information fusion, local contextual relationships within receptive fields of different scales can be effectively obtained. At the same time, the multi-scale contextual relationships are obtained by connecting each layer of Transformer blocks and Decoder blocks through a multi-scale context extraction model, which can effectively segment medical images and thus effectively identify anatomical structures in medical images.

[0149] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied in a distributed scenario and performed by multiple devices working together. In such a distributed scenario, one of the multiple devices may only perform one or more steps of the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the method.

[0150] It should be noted that the above description is limited to some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in an order different from that described in the above embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0151] Based on the same technical concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a medical image segmentation device based on Transformer context information fusion.

[0152] refer to Figure 6 The medical image segmentation device based on Transformer context information fusion includes:

[0153] A grouping module 601 is used to group pixels of a medical image;

[0154] The encoding module 602 is used to perform contraction path downsampling on the medical image to obtain medical images of different scales;

[0155] A local context module 603 is configured to extract the relationship between pixels in the local receptive field and surrounding pixels of the medical images of different scales through spatial separation, dilated convolution, and residual connection to obtain a local context relationship;

[0156] The decoding module 604 is configured to sequentially perform dilation path upsampling on the minimum-scale medical image, and splice the current decoding layer with the symmetrical encoding layer to obtain a splicing result; perform self-attention calculation on the global image to obtain a global context relationship of the medical image at the current scale;

[0157] The multi-scale context extraction module 605 is configured to connect each encoding layer and each decoding layer, and utilize cross-attention calculation to combine the medical images of different scales with the global image of the medical image at the current scale with the global context relationship to obtain a multi-scale context relationship; and to combine the global context relationship of the medical image at the current scale with the global context relationship of the multi-scale medical image to obtain a multi-scale context relationship;

[0158] The segmentation module 606 is configured to segment the medical image based on the multi-scale contextual relationship and the local contextual relationship to obtain a segmentation result.

[0159] For the convenience of description, the above devices are described as being divided into various modules according to their functions. Of course, when implementing this application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0160] The device of the above embodiment is used to implement the corresponding medical image segmentation method based on Transformer context information fusion in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0161] Based on the same technical concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the program, it implements the medical image segmentation method based on Transformer context information fusion described in any of the above embodiments.

[0162] Figure 7 10 is a schematic diagram showing a more specific hardware structure of an electronic device provided in this embodiment. The device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are communicatively connected to each other within the device via the bus 1050.

[0163] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0164] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage devices, dynamic storage devices, etc. The memory 1020 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0165] The input / output interface 1030 is used to connect input / output modules to implement information input and output. The input / output modules can be configured as components within the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and output devices may include a display, speaker, vibrator, indicator light, etc.

[0166] The communication interface 1040 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WiFi, Bluetooth, etc.).

[0167] The bus 1050 comprises a path for transmitting information between the various components of the device (eg, the processor 1010 , the memory 1020 , the input / output interface 1030 , and the communication interface 1040 ).

[0168] It should be noted that although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040, and the bus 1050, in a specific implementation, the device may also include other components necessary for normal operation. In addition, it will be understood by those skilled in the art that the above device may only include the components necessary to implement the embodiments of this specification, and does not necessarily include all the components shown in the figure.

[0169] The electronic device of the above embodiment is used to implement the corresponding medical image segmentation method based on Transformer context information fusion in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0170] Based on the same technical concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the medical image segmentation method based on Transformer context information fusion as described in any of the above embodiments.

[0171] The computer-readable media of this embodiment include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0172] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the medical image segmentation method based on Transformer context information fusion as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0173] Based on the same inventive concept, corresponding to the medical image segmentation method based on Transformer context information fusion described in any of the above embodiments, the present disclosure also provides a computer program product, which includes computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of a computer so that the computer and / or the processor executes the medical image segmentation method based on Transformer context information fusion. Corresponding to the execution subject corresponding to each step in each embodiment of the medical image segmentation method based on Transformer context information fusion, the processor that executes the corresponding step may belong to the corresponding execution subject.

[0174] The computer program product of the above embodiment is used to enable the computer and / or the processor to execute the medical image segmentation method based on Transformer context information fusion as described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0175] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. Within the scope of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0176] In addition, for simplicity of description and discussion, and in order not to make the embodiment of the application difficult to understand, the known power supply / ground connection with integrated circuit (IC) chip and other components may or may not be shown in the accompanying drawings provided. In addition, the device can be shown in the form of a block diagram to avoid making the embodiment of the application difficult to understand, and this also takes into account the following fact, that is, the details of the embodiment of these block diagram devices are highly dependent on the platform to be implemented in the embodiment of the application (that is, these details should be fully within the scope of understanding of those skilled in the art). When specific details (for example, circuit) are set forth to describe exemplary embodiments of the application, it will be apparent to those skilled in the art that the embodiment of the application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0177] Although the present invention has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may utilize the embodiments discussed.

[0178] The embodiments of the present application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of this application.

Claims

1. A medical image segmentation method based on Transformer context information fusion, characterized in that: The Transformer includes multiple encoding layers and symmetrical decoding layers; The method comprises: Acquiring a medical image and grouping pixels of the medical image; Performing contraction path downsampling on each group of the medical images through the multiple coding layers to obtain medical images of different scales, and extracting relationships between pixels in a local receptive field and surrounding pixels of the medical images of different scales through spatial separation, dilated convolution, and residual connection to obtain local contextual relationships; Determine a minimum-scale medical image, perform dilation path upsampling on the minimum-scale medical image through each decoding layer, and splice the current decoding layer with a symmetrical encoding layer to obtain a splicing result; perform self-attention calculation on the splicing result to obtain a global context relationship of the medical image at the current scale; splicing each of the encoding layers with each of the decoding layers, and fusing the medical images of different scales with the global image of the medical image at the current scale using cross-attention calculation to obtain a global contextual relationship between the medical image at the current scale and the medical images at different scales; splicing the global context relationship of the medical image at the current scale and the global context relationship of the medical images at different scales to obtain a multi-scale context relationship; The medical image is segmented based on the multi-scale contextual relationship and the local contextual relationship to obtain a segmentation result.

2. The method according to claim 1, characterized in that Each coding layer presets a multi-level branch local context extraction model as a feedforward neural network in the coding layer for feature extraction; The multi-level branch local context extraction model includes a main branch, a first branch, a second branch, a third branch and a fourth branch; The relationship between pixels in the local receptive field and surrounding pixels of the medical images of different scales is extracted through spatial separation, void convolution, and residual connection to obtain local contextual relationships, including: Using 1×1 convolution to compress the number of channels of the medical image input to the main branch to obtain a first feature map, and inputting the first feature map into the first branch, the second branch, the third branch, and the fourth branch respectively; Use spatial separation convolution to extract vertical and horizontal boundary information from the first feature map input to the first branch to obtain a second feature map; splicing the first branch, the second branch, the third branch, and the fourth branch so that the first branch, the second branch, the third branch, and the fourth branch are sequentially connected in series; Using dilated convolutions with 3×3 convolution kernels of different dilation rates, feature extraction is performed on the second branch, the third branch, and the fourth branch, respectively, to obtain a third feature map, a fourth feature map, and a fifth feature map, respectively; wherein the dilation rate increases from the second branch to the fourth branch; The second feature map, the third feature map, the fourth feature map and the fifth feature map are merged to obtain fused local information, and then residual calculation is performed with the medical image input into the main branch to obtain the local context relationship.

3. The method according to claim 1, characterized in that The coding layer includes a first-layer Transformer block, a second-layer Transformer block, a third-layer Transformer block, and a fourth-layer Transformer block; The first-layer Transformer block, the second-layer Transformer block, the third-layer Transformer block, and the fourth-layer Transformer block are connected in series in sequence, and the downsampling resolution of the contraction path from the first-layer Transformer block to the fourth-layer Transformer block decreases in sequence.

4. The method according to claim 3, characterized in that The medical images of different scales include a first-scale medical image, a second-scale medical image, a third-scale medical image and a minimum-scale medical image; The grouping of pixels of the medical image comprises: Divide the pixels of the medical image into 7×7 two-dimensional groups using a two-dimensional convolution kernel, establish relationships between different groups through overlapping grouping, and input the grouping results into the first-layer Transformer block; Divide the pixels of the first-scale medical image output by the first-layer Transformer block into 3×3 two-dimensional groups using a two-dimensional convolution kernel, introduce the relationship between different groups by overlapping groups, and input the grouping results into the second-layer Transformer block; Dividing the pixels of the second-scale medical image output by the second-layer Transformer block into 3×3 two-dimensional groups using a two-dimensional convolution kernel, introducing the relationship between different groups by overlapping groups, and inputting the grouping results into the third-layer Transformer block; The pixels of the third-scale medical image output by the third-layer Transformer block are divided into 3×3 two-dimensional groups through a two-dimensional convolution kernel, and the relationship between different groups is introduced through overlapping grouping, and the grouping results are input into the fourth-layer Transformer block.

5. The method according to claim 4, characterized in that The decoding layer includes a first-layer Decoder block, a second-layer Decoder block, a third-layer Decoder block, and a fourth-layer Decoder block; the fourth-layer Decoder block, the third-layer Decoder block, the second-layer Decoder block, and the first-layer Decoder block are sequentially connected in series, and the sampling resolution increases sequentially along the expansion path from the fourth-layer Decoder block to the first-layer Decoder block; The splicing results include: a first splicing result, a second splicing result and a third splicing result; The determining of the minimum-scale medical image, performing dilation path upsampling on the minimum-scale medical image through each decoding layer, and splicing the current decoding layer with the symmetrical encoding layer to obtain a splicing result; and performing self-attention calculation on the global image to obtain a global context relationship of the medical image at the current scale, including: performing self-attention calculation on the minimum-scale medical image output by the fourth-layer Transformer block in the fourth-layer Decoder block to obtain a global context relationship of the minimum-scale medical image; splicing, in the third-layer Decoder block, the third-scale medical image output by the fourth-layer Decoder block and the third-scale medical image output by the third-layer Transformer block to obtain a third splicing result, and performing self-attention calculation on the third splicing result to obtain a global context relationship of the third-scale medical image; splicing, in the second-layer Decoder block, the second-scale medical image output by the third-layer Decoder block and the second-scale medical image output by the second-layer Transformer block to obtain a second splicing result, and performing self-attention calculation on the second splicing result to obtain a global context relationship of the second-scale medical image; In the first-layer Decoder block, the first-scale medical image output by the second-layer Decoder block and the first-scale medical image output by the first-layer Transformer block are spliced to obtain a first splicing result, and self-attention calculation is performed on the first splicing result to obtain a global context relationship of the first-scale medical image.

6. The method according to claim 5, characterized in that Each of the encoding layers is spliced with each of the decoding layers, and the medical images of different scales are fused with the global image of the medical image at the current scale using cross-attention calculation to obtain a global contextual relationship of the multi-scale medical image, including: In the fourth-layer decoder block, cross-attention calculations are performed on the first-scale medical image, the second-scale medical image, and the third-scale medical image, respectively, with the global image of the minimum-scale medical image to obtain a global contextual relationship between the minimum-scale medical image and the first-scale medical image, the second-scale medical image, and the third-scale medical image; In the third-layer decoder block, cross-attention calculations are performed on the first-scale medical image, the second-scale medical image, the third-scale medical image, and the minimum-scale medical image, respectively, with a global image of the third-scale medical image to obtain a global contextual relationship between the third-scale medical image and the first-scale medical image, the second-scale medical image, the third-scale medical image, and the minimum-scale medical image; In the second-layer decoder block, cross-attention calculation is performed on the first-scale medical image, the second-scale medical image, the third-scale medical image, and the minimum-scale medical image, respectively, with the global image of the second-scale medical image to obtain a global contextual relationship between the second-scale medical image and the first-scale medical image, the second-scale medical image, the third-scale medical image, and the minimum-scale medical image; In the first-layer decoder block, cross-attention calculation is performed on the first-scale medical image, the second-scale medical image, the third-scale medical image, and the minimum-scale medical image, respectively, with the global image of the first-scale medical image to obtain the first-scale medical image and the first-scale medical image, the second-scale medical image, the third-scale medical image, and the minimum-scale medical image.

7. The method according to claim 6, characterized in that Splicing the global context relationship of the medical image at the current scale and the global context relationship of the medical images at different scales to obtain a multi-scale context relationship, including: splicing the global context relationship of the minimum-scale medical image and the global context relationship between the minimum-scale medical image and the first-scale medical image, the second-scale medical image, and the third-scale medical image to obtain a multi-scale context relationship of the minimum-scale medical image; splicing the global context relationship of the third-scale medical image and the global context relationship between the third-scale medical image and the first-scale medical image, the second-scale medical image, the third-scale medical image, and the minimum-scale medical image to obtain a multi-scale context relationship of the third-scale medical image; splicing the global context relationship of the second-scale medical image and the global context relationship of the second-scale medical image with the first-scale medical image, the second-scale medical image, the third-scale medical image, and the minimum-scale medical image to obtain a multi-scale context relationship of the second-scale medical image; The global context relationship of the first-scale medical image and the global context relationships of the first-scale medical image, the first-scale medical image, the second-scale medical image, the third-scale medical image, and the minimum-scale medical image are stitched together to obtain a multi-scale context relationship of the first-scale medical image.

8. The method according to claim 2, characterized in that Using spatial separation convolution to extract vertical and horizontal boundary information from the first feature map input to the first branch, a second feature map is obtained, including: Convolving the first feature map using a first path first vertically and then horizontally to obtain a first result; Convolving the first feature map using a second path first horizontally and second vertically to obtain a second result; Concatenating the first result and the second result to obtain a second feature map; The second feature map is expressed as: F1=concat[χ(conv h (conv. v (F0))),χ(conv v (conv. h (F0)))], Among them, conv v and conv h They are 3×1 and 1×3 vertical convolution and horizontal convolution respectively, F0 is the first feature map, and χ is the Batch Normalization and ReLU activation function.

9. A medical image segmentation device based on Transformer, characterized in that: The Transformer includes multiple encoding layers and symmetrical decoding layers; the device includes: A grouping module, for grouping pixels of a medical image; The encoding module is used to perform contraction path downsampling on the medical image to obtain medical images of different scales; A local context extraction module is used to extract the relationship between pixels in the local receptive field and surrounding pixels of the medical images of different scales through spatial separation, void convolution and residual connection to obtain local context relationship; A decoding module is used to sequentially upsample the minimum-scale medical image along the expansion path, and to splice the current decoding layer with the symmetrical encoding layer to obtain a splicing result; and to perform self-attention calculation on the splicing result to obtain the global context of the medical image at the current scale; a multi-scale context extraction module, configured to connect each encoding layer and each decoding layer, and fuse medical images of different scales with the global image of the medical image at the current scale using cross-attention calculation to obtain a global context relationship between the medical image at the current scale and the medical images at different scales; and to concatenate the global context relationship of the medical image at the current scale and the global context relationship of the medical images at different scales to obtain a multi-scale context relationship; A segmentation module is used to segment the medical image based on the multi-scale context relationship and the local context relationship to obtain a segmentation result.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

Images