Prompt-based edge enhanced medical image segmentation method and system

By constructing a Prompt-based edge enhancement medical image segmentation method, and utilizing a cross-attention mechanism and a feature alignment enhancement module, the problem of insufficient processing of bone edge features in bone CT images is solved, thereby improving segmentation accuracy and efficiency.

CN118918329AActive Publication Date: 2024-11-08SHANDONG NORMAL UNIV

Patent Information

Application Number
CN202411049574.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-01
Publication Date
2024-11-08
Estimated Expiration
2044-08-01

AI Technical Summary

Technical Problem

Existing Transformer-based medical image segmentation methods struggle to effectively handle local features of bone edges in bone CT images, leading to inaccurate segmentation, especially in cases of osteonecrosis where information about bone depressions is lacking.

Method used

We construct a Prompt-based edge enhancement medical image segmentation method. By using an encoder, bottleneck layer, decoder, and edge-aware learning module, we obtain edge information from bone CT images. We then use a cross-attention mechanism and a feature alignment enhancement module to fuse high- and low-resolution feature maps, thereby enhancing edge information and preserving local features.

Benefits of technology

It improves the accuracy of bone CT image segmentation, enabling more natural display of bone morphological abnormalities and saving medical staff time in reading and understanding CT images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918329A_ABST
    Figure CN118918329A_ABST
Patent Text Reader

Abstract

The invention belongs to the field of computer vision and medical images, and particularly relates to an edge enhancement medical image segmentation method and system based on Prompt, and the method comprises the steps: obtaining a skeleton CT image, carrying out the continuous down-sampling of the image through an encoder, and obtaining a segmentation result; a feature obtained by down-sampling of each layer is used as a high-resolution feature map input into an up-sampling layer in the corresponding decoder; edge information, frame information and mask information are obtained from a skeleton CT image through an edge perception learning module, and the obtained information and features output by an encoder are fused and then serve as input of a bottleneck layer; and in the encoder, the output of the bottleneck layer or the previous layer is used as a low-resolution feature map, the low-resolution feature map and the high-resolution feature map are subjected to feature alignment and feature enhancement and then are used as the output of the current layer, and the output of the last layer of the encoder is used as an image segmentation result. According to the method, the image segmentation effect is improved, so that the time spent on reading and understanding the CT image by medical personnel can be saved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the field of computer vision and medical images, and in particular relates to a Prompt-based edge enhancement medical image segmentation method and system. Background Art

[0002] Medical image segmentation is an important branch of image segmentation, and plays an important role in clinical applications such as diagnosis, surgical planning, and image-guided surgery. Especially in the clinical diagnosis of bone necrosis, the segmentation of bone CT images can facilitate doctors' diagnosis and analysis of the disease, thereby reducing the workload of doctors. Although bones have obvious contrast in CT images, their segmentation is still challenging. Because bones are different from pathological features such as tumors, the edges of different bones are unique, especially the bones of bone necrosis can cause different degrees of depression. Most of the current automatic methods lack the learning of partial information on the edge of the bones, resulting in inaccurate segmentation. In addition, the Transformer-based architecture has excellent ability to model global context, so it is widely popular in medical image segmentation tasks. However, the Transformer-based encoder method is prone to destroy some local features when modeling the global context, and the edge hints cannot handle the local information of the bones well. Summary of the invention

[0003] In view of the shortcomings of the prior art, the present invention proposes a Prompt-based edge-enhanced medical image segmentation method, constructs an image segmentation network, and the network includes an encoder, a bottleneck layer, a decoder, and an edge-aware learning module. The process of image segmentation includes the following steps:

[0004] Obtain a bone CT image, and continuously downsample the image through the encoder. The features obtained from each downsampling layer are used as high-resolution feature maps input to the upsampling layer in the corresponding decoder.

[0005] The edge information, frame information, and mask information are obtained from the bone CT image through the edge perception learning module. The obtained information is fused with the features output by the last downsampling layer of the encoder as the input of the bottleneck layer.

[0006] In the encoder, the output of the bottleneck layer or the previous layer is used as a low-resolution feature map, the low-resolution feature map and the high-resolution feature map are aligned and enhanced as the output of the current layer, and the output of the last layer of the encoder is used as the image segmentation result.

[0007] The present invention also proposes a prompt-based edge-enhanced medical image segmentation system, which is used to implement a prompt-based edge-enhanced medical image segmentation method, including a preprocessing module, an encoder, a decoder, an edge-aware prompt learning module and a bottleneck layer, and the feature maps of each layer in the encoder and the decoder are matched with each other; wherein:

[0008] A preprocessing module, used for preprocessing the input bone CT image, including converting the bone CT image into a feature map and performing enhancement and annotation;

[0009] The encoder is used to perform multiple downsampling operations on the input feature map;

[0010] The edge perception module is used to obtain the hint semantic information based on the fusion of the detection box and edge information;

[0011] The bottleneck layer is used to filter the hint semantic information and obtain the initial high-resolution feature map input to the decoder;

[0012] The decoder is used to perform multiple upsampling on the input feature map corresponding to the decoder. During the upsampling process, the output of the corresponding layer of the decoder is upsampled as a low-resolution feature map, the low-resolution feature map and the high-resolution feature map are aligned and enhanced as the output of the current layer, and the output of the current layer is used as the high-resolution feature map of the next layer. The feature map output by the last layer of the decoder is the segmentation result.

[0013] Furthermore, the edge-aware learning module processes the bone CT images, such as Figure 5 , specifically including the following steps:

[0014] The detection frame is obtained by manual annotation, and the edge information is extracted from the bone CT image using an edge detector;

[0015] The detection box and edge information are positionally encoded respectively, and then the corresponding weights are generated by MLP and multiplied with the position encoding to obtain the corresponding box embedding vector and edge embedding vector;

[0016] Concatenate the edge embedding vector and the box embedding vector together as a sparse embedding vector;

[0017] The manually annotated masks in the bone CT images are converted into dense embedding vectors through continuous downsampling;

[0018] The sparse embedding vector and the dense embedding vector are concatenated together as the prompt speech information;

[0019] The low-resolution feature map is fused with the hint semantic information through a cross-attention mechanism.

[0020] Furthermore, the mask in the image is continuously downsampled, that is, three cascaded downsampling modules are used for processing. The first and second downsampling modules are composed of cascaded 2×2 convolution units, layer normalization units and GELU activation functions; the third downsampling module is composed of 1×1 convolution units.

[0021] Furthermore, if the low-resolution feature map is S1 and the hint semantic information is S2, the fusion process through the cross-attention mechanism includes:

[0022] S3=softmax((W Q S2)(W K S1) T )W V S1

[0023] Among them, S3 is the feature map obtained by fusing the low-resolution feature map with the hint semantic information through the cross-attention mechanism; W Q , W K , W V is the trainable cross-attention mechanism weight matrix; W Q S2 is the query matrix in the cross-attention mechanism, W K S1 is the key matrix in the cross-attention mechanism, W V S1 is the value matrix in the cross-attention mechanism; (·) T Represents the transpose of a matrix.

[0024] Furthermore, the high-resolution feature map and the low-resolution feature map are aligned, such as Figure 6 , including the following steps:

[0025] The high-resolution feature map is processed by a 1×1 convolution module to obtain a corresponding high-resolution feature map, and the low-resolution feature map is upsampled by a 3×3 convolution module to obtain an upsampled low-resolution feature map, so that the size of the upsampled low-resolution feature map matches the high-resolution feature map; the upsampled low-resolution feature map is concatenated with the high-resolution feature map and processed by a 3×3 convolution module to obtain a combined feature map;

[0026] The combined feature map is processed using the optical flow estimation method to obtain an optical flow feature map;

[0027] The optical flow feature map and the high-resolution feature map are input into the warp network for local alignment to obtain a local aligned feature map;

[0028] The locally aligned feature map is added to the upsampled low resolution to obtain the aligned feature map.

[0029] Furthermore, the feature enhancement process is performed based on the feature map after feature alignment between the high-resolution image and the low-resolution image, such as Figure 7 , specifically including the following steps:

[0030] Based on the spatial attention mechanism, a spatial attention map is generated according to the aligned feature map, and the aligned feature map is weighted by using the spatial attention map to obtain a first weighted feature map;

[0031] Use global average pooling to generate channel statistics of the aligned feature maps, and generate the weight of each channel based on the channel attention mechanism and channel statistics;

[0032] Using the weight of each channel to weight each channel of the aligned feature map, a second weighted feature map is obtained;

[0033] The first weighted feature map and the second weighted feature map are fused by adding each other to obtain an enhanced feature map.

[0034] Further, weighting the aligned feature map using the spatial attention map to obtain a first weighted feature map includes the following steps:

[0035] A 1×1 convolutional layer with two activation functions is used to generate the spatial attention map. Right now:

[0036]

[0037] Using att SA For the input feature F h′ Perform spatial modulation to obtain the first weighted feature map, namely:

[0038]

[0039] Among them, σ(·) is the sigmoid activation function; represents the weights of two 1×1 convolutional layers; δ(·) represents the ReLU activation function; F h′ is the aligned feature map; represents the aligned feature map, Indicates H h ×W h ×C h dimensional real space, H h represents the height of the high-resolution feature map, W h represents the width of the high-resolution feature map, C h Represents the number of channels of the high-resolution feature map; represents the Kronecker product.

[0040] Further, using the weight of each channel to weight each channel of the aligned feature map, respectively, to obtain a second weighted feature map includes the following steps:

[0041] Use global average pooling to generate the aligned feature map F h′ Calculate its channel statistics G CA , G CA The calculation of the cth element of is expressed as:

[0042]

[0043] A simple gating mechanism with sigmoid activation is used to generate the channel attention map att CA , expressed as:

[0044]

[0045] Using channel attention map att CA To adjust the input feature F h′ ,Right now:

[0046]

[0047] in, represents the element at position (i, j) in the cth channel, i∈{1,2,…,H h}, j∈{1,2,…,W h}; Represents the weights of the two 1×1 convolutional layers in the channel attention; is the channel attention map att CA The attention map of the cth channel in, is the aligned feature map F h′ The feature map of the cth channel in .

[0048] The present invention also proposes a prompt-based edge-enhanced medical image segmentation system, which is used to implement a prompt-based edge-enhanced medical image segmentation method, including a preprocessing module, an encoder, a decoder, an edge-aware prompt learning module and a bottleneck layer, and the feature maps of each layer in the encoder and the decoder are matched with each other; wherein:

[0049] A preprocessing module, used for preprocessing the input bone CT image, including converting the bone CT image into a feature map and performing enhancement and annotation;

[0050] The encoder is used to perform multiple downsampling operations on the input feature map;

[0051] The edge perception module is used to obtain the hint semantic information based on the fusion of the detection box and edge information;

[0052] The bottleneck layer is used to filter the hint semantic information and obtain the initial high-resolution feature map input to the decoder;

[0053] The decoder is used to perform multiple upsampling on the input feature map corresponding to the decoder. During the upsampling process, the output of the corresponding layer of the decoder is upsampled as a low-resolution feature map, the low-resolution feature map and the high-resolution feature map are aligned and enhanced as the output of the current layer, and the output of the current layer is used as the high-resolution feature map of the next layer. The feature map output by the last layer of the decoder is the segmentation result.

[0054] In the medical image segmentation method, in view of the problem of insufficient edge processing and local information loss, compared with the prior art, the present invention has the following beneficial effects:

[0055] (1) The present invention obtains the edge of the real image through the edge perception module, first fuses the Box and Edge prompt information, processes the mask prompt information, and then combines Sparse_Embeddings and Dense_Embeddings with the image through the fusion module, which can effectively capture and enhance the edge information of the image and make the edge transition more natural and smooth;

[0056] (2) The present invention combines context information through a feature alignment enhancement module to ensure that local features are not lost during upsampling; and combines spatial information of the image to ensure that global features are not lost.

[0057] In summary, the present invention improves the effect of image segmentation, thereby saving the time that medical personnel spend on reading and understanding CT images, and intuitively displays abnormal bone morphology. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a framework diagram of a Prompt-based edge enhancement medical image segmentation method of the present invention;

[0059] Figure 2 A schematic diagram of the edge-aware prompt learning framework process in a Prompt-based edge enhancement medical image segmentation method of the present invention;

[0060] Figure 3 A schematic diagram of a context feature alignment enhancement process in a Prompt-based edge enhancement medical image segmentation method of the present invention;

[0061] Figure 4 A flowchart of a Prompt-based edge enhancement medical image segmentation method of the present invention;

[0062] Figure 5 A flowchart of a process of processing a bone CT image by an edge-aware learning module in a Prompt-based edge-enhanced medical image segmentation method of the present invention;

[0063] Figure 6 A flowchart of feature alignment between a high-resolution feature map and a low-resolution feature map in a Prompt-based edge enhancement medical image segmentation method of the present invention;

[0064] Figure 7 The present invention is a flowchart of performing feature enhancement on an aligned feature map in a Prompt-based edge enhancement medical image segmentation method. DETAILED DESCRIPTION

[0065] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] The present invention proposes a Prompt-based edge enhancement medical image segmentation method. Figure 4 , the process of image segmentation includes the following steps:

[0067] S0, construct an image segmentation network, which includes an encoder, a bottleneck layer, a decoder, and an edge-aware learning module;

[0068] S1. Obtain a bone CT image, and continuously downsample the image through the encoder. The features obtained from each downsampling layer are used as high-resolution feature maps input to the upsampling layer in the corresponding decoder.

[0069] S2, obtain edge information, frame information and mask information from the bone CT image through the edge perception learning module, and fuse the obtained information with the features output by the last downsampling layer of the encoder as the input of the bottleneck layer;

[0070] S3. In the encoder, the output of the bottleneck layer or the previous layer is used as a low-resolution feature map, the low-resolution feature map and the high-resolution feature map are aligned and enhanced as the output of the current layer, and the output of the last layer of the encoder is used as the image segmentation result.

[0071] In this embodiment, in order to implement a Prompt-based edge enhancement medical image segmentation method, the following processing is performed on the bone CT image in sequence:

[0072] First, the bone CT images are preprocessed;

[0073] Then get the edge information of the bone image, such as Figure 1In this embodiment, an edge detector (EdgeDetection) is used, and the prompt information generated by the prompt learning module (Prompt Encoder is used in this embodiment) is combined with the mask and box information to combine with the feature information processed by the encoder;

[0074] Finally, the decoder performs feature fusion with the encoder through jump connections (this embodiment uses a mixer for fusion), and combines the context information through the context feature alignment enhancement module to ensure that global and local information is not lost, and finally a bone segmentation image is generated.

[0075] In this embodiment, preprocessing the bone CT image includes labeling and enhancing the image, including labeling the bone part in the bone CT image, and enhancing the image data by rotation, horizontal flipping and vertical flipping.

[0076] The present invention uses an encoder to continuously downsample the image, such as Figure 1 In this embodiment, three SwinTransfamer downsampling blocks are used to continuously downsample the input bone CT image. Figure 1 The three Swin Transfamer downsampling blocks from left to right are defined as the first Swin Transfamer downsampling block, the second Swin Transfamer downsampling block, and the third Swin Transfamer downsampling block, each Swin Transfamer downsampling block outputs a high-resolution feature map of one size, and the three downsampling blocks from left to right output the first high-resolution feature map, the second high-resolution feature map, and the third high-resolution feature map, respectively.

[0077] In this embodiment, the edge information of the real image can be extracted by using the Sobel operator, and the extraction process includes the following steps:

[0078] 101. Set the Sobel convolution factor. The operator consists of two 3×3 matrices, corresponding to the horizontal and vertical directions of the image respectively. The Sobel convolution factor is expressed as:

[0079]

[0080] The two sets of matrices are convolved with the image to calculate the approximate values ​​of the brightness difference of the image in the horizontal and vertical directions. x and G y Respectively represent the grayscale values ​​of the image after horizontal and vertical edge detection, and the formulas are as follows:

[0081]

[0082] Among them, * represents the planar convolution operation.

[0083] 102. Calculate the grayscale of each pixel in the image. In this embodiment, the grayscale values ​​of the horizontal and vertical directions are comprehensively considered. This calculation process can be implemented by the following mathematical formula:

[0084]

[0085] In order to improve the calculation efficiency, an approximate method without square root operation is often used to estimate the gray value of the pixel. This method can effectively simplify the calculation process while maintaining sufficient accuracy, and is particularly suitable for scenarios with high requirements for calculation speed. As an optional implementation, the approximate calculation formula is as follows:

[0086] |G|=|G x |+|G y |

[0087] 203. Then we need to calculate the gradient direction of each pixel. This direction can be obtained by G x and G y , that is, the gray value change of the pixel in the horizontal and vertical directions. The calculation formula for the gradient direction is as follows:

[0088]

[0089] 204. After obtaining the gradient direction, this embodiment needs to determine whether the pixel is an edge point, which is determined by comparing the magnitude of the gradient (expressed by δ) with a preset threshold. If δ is greater than the threshold, the pixel (x, y) is considered to be an edge point.

[0090] In this embodiment, the preprocessed and enhanced images are used as inputs of the encoder and the edge-aware cue learning module, respectively. The encoder downsamples the image, and the edge-aware cue learning module extracts semantic information from the image. Figure 5 ,The process of extracting semantic information from the image by the edge-aware cue learning module specifically includes the following steps:

[0091] The detection frame is obtained by manual annotation, and the edge information is extracted from the bone CT image using an edge detector;

[0092] The detection box and edge information are positionally encoded respectively, and then the corresponding weights are generated by MLP and multiplied with the position encoding to obtain the corresponding box embedding vector and edge embedding vector;

[0093] Concatenate the edge embedding vector and the box embedding vector together as a sparse embedding vector;

[0094] The manually annotated masks in the bone CT images are converted into dense embedding vectors through continuous downsampling;

[0095] The sparse embedding vector and the dense embedding vector are concatenated together as the prompt speech information;

[0096] The low-resolution feature maps are fused with the hint semantic information via a cross-attention mechanism.

[0097] Specifically, the labeled box information and the detected edge information are converted into edge embedding vectors (EmbedEdges) and box embedding vectors (Embed Boxes). This step involves encoding the significant edges and frame structures in the image and converting them into numerical vectors that can be processed by the neural network.

[0098] The edge embedding vector and the box embedding vector are further combined into a sparse embedding vector (SparseEmbeddings). The purpose of this step is to merge the edge and box information to provide a more detailed and comprehensive image structure representation, such as Figure 2 , the process of synthesizing sparse embedding vectors includes the following steps:

[0099] The detection frame is obtained by manual annotation, and the edge information is extracted from the bone CT image using an edge detector;

[0100] The detection box and edge information are positionally encoded respectively, and then the corresponding weights are generated by MLP and multiplied with the position encoding to obtain the corresponding box embedding vector and edge embedding vector;

[0101] The edge embedding vector and the box embedding vector are concatenated together as a sparse embedding vector.

[0102] The mask in the image is converted into a dense embedding vector (DenseEmbeddings) through a series of downsampling steps. This transformation makes the mask information more compact and concentrated. The manually annotated mask in the bone CT image is converted into a dense embedding vector through continuous downsampling. The mask in the image is continuously downsampled, that is, it is processed by three cascaded downsampling modules. The first and second downsampling modules are composed of cascaded 2×2 convolution units, layer normalization units and GELU activation functions; the third downsampling module is composed of 1×1 convolution units.

[0103] The dense embedding vector and the sparse embedding vector are concatenated as the hint semantic information. In this embodiment, the high-resolution image is fused with the hint semantic information by using the cross-attention mechanism. In this embodiment, the third high-resolution image output by the third SwinTransfamer downsampling block is fused with the hint semantic information. If the low-resolution feature map is S1 and the hint semantic information is S2, the fusion process by the cross-attention mechanism includes:

[0104] S3=softmax((W Q S2)(W K S1) T )W V S1

[0105] Among them, S3 is the feature map obtained by fusing the high-resolution feature map with the hint semantic information through the cross-attention mechanism; W Q , W K 、V W is the trainable cross-attention mechanism weight matrix; W Q S2 is the query matrix in the cross-attention mechanism, W K S1 is the key matrix in the cross-attention mechanism, W V S1 is the value matrix in the cross-attention mechanism; (·) T Represents the transpose of a matrix.

[0106] The feature map S3 obtained by fusing the low-resolution feature map with the hint semantic information through the cross-attention mechanism is filtered through the bottleneck layer. In this embodiment, two cascaded Swin Transfamer blocks are used as the bottleneck layer.

[0107] In this implementation, the decoder corresponds to the encoder and includes a three-layer cascade structure, each structure is composed of a cascade of a SwinTransfamer upsampling block and a contextual feature alignment enhancement block, such as Figure 1 , from left to right in the decoder are the first hierarchical structure, the second hierarchical structure and the third hierarchical structure. The first hierarchical structure is composed of the cascaded first Swin Transfamer upsampling block and the first context feature alignment enhancement block, the second hierarchical structure is composed of the cascaded second Swin Transfamer upsampling block and the second context feature alignment enhancement block, and the third hierarchical structure is composed of the cascaded third Swin Transfamer upsampling block and the third context feature alignment enhancement block.

[0108] In the first hierarchical structure, the third high-resolution image output by the third Swin Transfamer downsampling block is input into the first Swin Transfamer upsampling block for upsampling to obtain a high-resolution feature map as the feature map Fh And the low-resolution feature map output by the bottleneck layer is used as the feature map F l Input the first context feature alignment enhancement block for feature alignment and local enhancement;

[0109] In the second hierarchical structure, the second high-resolution image output by the second Swin Transfamer downsampling block is input into the second Swin Transfamer upsampling block for upsampling to obtain a high-resolution feature map as the feature map F h The low-resolution feature map after feature alignment and local enhancement by the first context feature alignment enhancement block of the first layer structure is used as the feature map F l Input the second context feature alignment enhancement block for feature alignment and local enhancement;

[0110] In the third hierarchical structure, the first high-resolution image output by the first Swin Transfamer downsampling block is input into the third Swin Transfamer upsampling block for upsampling to obtain a high-resolution feature map as the feature map F l The low-resolution feature map after feature alignment and local enhancement by the second context feature alignment enhancement block of the second layer structure is used as the feature map F h Input the third context feature alignment enhancement block for feature alignment and local enhancement;

[0111] The feature map after feature alignment and local enhancement by the third context feature alignment enhancement block is used as the output of the decoder.

[0112] In this embodiment, the spatial information is combined by the decoder to ensure that the global features are not lost, and the context information is combined by the context feature alignment enhancement module to ensure that the local features are not lost while upsampling. The feature maps F of two different resolutions are l and F h It is used as the input of the context feature alignment enhancement module, which performs two feature maps F with different resolutions on the context feature alignment enhancement module. l and F h The processing process includes two parts: feature alignment and local enhancement, which specifically includes the following steps:

[0113] 301. High-resolution feature map From the corresponding stage of the decoder, Indicates latitude is H h ×W h ×C h The set of real numbers, H h , W h , C hare the height, width, and number of channels of the high-resolution feature map. This feature map contains more detailed information and is crucial for preserving the local features of the image; while the low-resolution image comes from the previous stage of the corresponding stage, Indicates latitude is H l ×W l ×C l The set of real numbers, H l , W l , C l The height, width, and number of channels of the low-resolution feature map are respectively; the high-resolution feature map is processed by a 1×1 convolution module. For the low-resolution feature map, a 3×3 convolution module is used to upsample it and adjust its size to match the high-resolution feature map.

[0114] 302. After splicing the first low-resolution feature map and the first high-resolution feature map, the 3×3 convolution module is used to process the combined feature map, and the optical flow estimation method is used to process the combined feature map to obtain an optical flow feature map. In this embodiment, the FlowNetSimple architecture can be used to obtain the corresponding flow field feature map.

[0115] 303. Input the optical flow feature map and the first high-resolution feature map into the warp network for local alignment to obtain a locally aligned feature map. Since there is a resolution gap between the feature and the flow field, the present invention reduces its offset by half through the warp network. The warp network used in the present invention, that is, the local alignment (Mesh Warps) in the prior art, is a matching method based on image blocks (patch), which divides the image into multiple grid-like regions, constructs geometric transformations of multiple regions by local feature matching, and each region is differentiated into a plane for alignment, which solves the problem of depth parallax to a certain extent.

[0116] Those skilled in the art may use any optical flow estimation method and local alignment to align the images, and the present invention does not impose any other limitations on the optical flow estimation method and local alignment.

[0117] 304. Add the locally aligned feature map to the first low-resolution map to obtain an aligned feature map.

[0118] After the alignment of the images is completed, the aligned images are input into the feature enhancement module for processing. In this embodiment, the aligned feature map is divided into two branches, which are processed by spatial attention and channel attention respectively. Then, the data processed by spatial attention and channel attention are fused by addition. The fused feature map is used as the output of the context feature alignment enhancement block, as shown in FIG. Figure 3 System processing block diagram and Figure 7As shown in the flowchart, the process of local enhancement specifically includes the following steps:

[0119] 401. Based on the spatial attention mechanism, a spatial attention map is generated according to the aligned feature map, and the aligned feature map is weighted by using the spatial attention map to obtain a first weighted feature map, which specifically includes the following steps:

[0120] A 1×1 convolutional layer with two activation functions is used to generate the spatial attention map. Right now:

[0121]

[0122] Using att SA For the input feature F h′ Perform spatial modulation to obtain the first weighted feature map, namely:

[0123]

[0124] Among them, σ(·) is the sigmoid activation function; represents the weights of two 1×1 convolutional layers; δ(·) represents the ReLU activation function; F h′ is the aligned feature map; represents the aligned feature map, Indicates H h ×W h ×C h dimensional real space, H h represents the height of the high-resolution feature map, W h represents the width of the high-resolution feature map, C h Represents the number of channels of the high-resolution feature map; represents the Kronecker product.

[0125] 402. Generate channel statistics of the aligned feature map using global average pooling, and generate weights of each channel based on the channel attention mechanism and the channel statistics, specifically including the following steps:

[0126] Use global average pooling to generate the aligned feature map F h′ Calculate its channel statistics G CA , G CA The calculation of the cth element of is expressed as:

[0127]

[0128] A simple gating mechanism with sigmoid activation is used to generate the channel attention map att CA , expressed as:

[0129]

[0130] Using channel attention map att CA To adjust the input feature F h′ ,Right now:

[0131]

[0132] in, represents the element at position (i, j) in the cth channel, i∈{1,2,…,H h}, j∈{1,2,…,W h}; Represents the weights of the two 1×1 convolutional layers in the channel attention; is the channel attention map att CA The attention map of the cth channel in, is the aligned feature map F h′ The feature map of the cth channel in .

[0133] 403. The first weighted feature map and the second weighted feature map are fused by adding each other to obtain an enhanced feature map.

[0134] In addition, Figure 3 In the figure, for the spatial attention branch, attention processing is performed and then processed by two cascaded 1×1 convolutional layers; correspondingly, in the spatial attention branch, the input feature map is first processed by two cascaded 1×1 convolutional layers and then channel attention is processed, and finally the outputs of the two branches are fused by adding them.

[0135] The present invention also provides a prompt-based edge-enhanced medical image segmentation system, which is used to implement a prompt-based edge-enhanced medical image segmentation method, and the system includes a preprocessing module, an encoder, a decoder, an edge-aware prompt learning module and a bottleneck layer, and the feature maps of each layer in the encoder and the decoder match each other; wherein: the preprocessing module is used to preprocess the input bone CT image, including converting the bone CT image into a feature map and enhancing and annotating the feature map;

[0136] The encoder is used to perform multiple downsampling operations on the input feature map;

[0137] The edge perception module is used to obtain hint semantic information by fusing the detection frame and edge information. The hint semantic information is obtained by fusing Sparse_Embeddings and Dense_Embeddings. The present invention fuses the hint semantic information with the image information (i.e., the feature map obtained by downsampling the decoder), effectively capturing and enhancing the edge information of the image, making the edge transition more natural and smooth.

[0138] The bottleneck layer is used to filter the hint semantic information and obtain the initial high-resolution feature map input to the decoder;

[0139] The decoder is used to perform multiple upsampling on the input feature map corresponding to the decoder. During the upsampling process, the output of the corresponding layer of the decoder is upsampled as a high-resolution feature map, the high-resolution feature map and the low-resolution feature map are aligned and enhanced as the output of the current layer, and the output of the current layer is used as the low-resolution feature map of the next layer. The feature map output by the last layer of the decoder is the segmentation result. Each upsampling process includes the SwinTransfomer upsampling block and the context feature alignment enhancement module to process the input high-resolution feature map and the low-resolution feature map. The upsampling process combines the spatial information and channel information with the original image to ensure that the global features are not lost and the local features are not lost during upsampling.

[0140] This embodiment also provides a one-minute calculation device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the following steps are implemented, including:

[0141] Input bone CT images, and preprocess and enhance the images;

[0142] The bone CT image is processed by the edge detector to obtain the edge information of the bone and generate the corresponding fragment; then the Box and Edge prompt information are fused, and the mask prompt information is processed; then Sparse_Embeddings and Dense_Embeddings are combined with the image through the fusion module to obtain the prompt information;

[0143] The bone CT image is downsampled through the encoder to obtain low-level feature information;

[0144] After downsampling, the hint semantic information is fused through the cross-attention fusion module;

[0145] The spatial information is combined through the decoder to ensure that the global features are not lost, and the contextual information is combined through the feature alignment enhancement module to ensure that the local features are not lost while upsampling.

[0146] The decoder and encoder perform feature fusion via skip connections to compensate for the loss of spatial information caused by downsampling.

[0147] The purpose of this embodiment is to provide a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the following steps are performed:

[0148] Input bone CT images, and preprocess and enhance the images;

[0149] The bone CT image is processed by the edge detector to obtain the edge information of the bone and generate the corresponding fragment; then the Box and Edge prompt information are fused, and the mask prompt information is processed; then Sparse_Embeddings and Dense_Embeddings are combined with the image through the fusion module to obtain the prompt information;

[0150] The bone CT image is downsampled through the encoder to obtain low-level feature information;

[0151] After downsampling, the hint semantic information is fused through the cross-attention fusion module;

[0152] The spatial information is combined through the decoder to ensure that the global features are not lost, and the contextual information is combined through the feature alignment enhancement module to ensure that the local features are not lost while upsampling.

[0153] The decoder and encoder perform feature fusion via skip connections to compensate for the loss of spatial information caused by downsampling.

[0154] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A Prompt-based edge enhancement medical image segmentation method, characterized in that: Construct an image segmentation network, which includes an encoder, a bottleneck layer, a decoder, and an edge-aware learning module. The image segmentation process includes the following steps: Obtain a bone CT image and continuously downsample the image through the encoder. The features obtained from each downsampling layer serve as the high-resolution feature map input to the upsampling layer in the corresponding decoder. The edge-aware learning module obtains edge information, frame information, and mask information from the bone CT image. The obtained information is fused with the features output by the last downsampling layer of the encoder and used as the input of the bottleneck layer. In the encoder, the output of the bottleneck layer or the previous layer is used as the low-resolution feature map, the low-resolution feature map and the high-resolution feature map are aligned and enhanced as the output of the current layer, and the output of the last layer of the encoder is used as the image segmentation result.

2. The edge enhancement medical image segmentation method based on Prompt according to claim 1, characterized in that: The edge-aware learning module processes bone CT images in the following ways: The detection frame is obtained by manual annotation, and the edge information is extracted from the bone CT image using an edge detector; The detection box and edge information are positionally encoded respectively, and then the corresponding weights are generated by MLP and multiplied by the position encoding to obtain the corresponding box embedding vector and edge embedding vector; Concatenate the edge embedding vector and the box embedding vector together as a sparse embedding vector; The manually annotated masks in the bone CT images are converted into dense embedding vectors through continuous downsampling; Concatenate the sparse embedding vector and the dense embedding vector together as the prompt speech information; The low-resolution feature map is fused with the hint semantic information through a cross-attention mechanism.

3. The edge enhancement medical image segmentation method based on Prompt according to claim 2, characterized in that: The mask in the image is continuously downsampled, that is, it is processed by three cascaded downsampling modules. The first and second downsampling modules are composed of cascaded 2×2 convolution units, layer normalization units and GELU activation functions; the third downsampling module is composed of 1×1 convolution units.

4. The edge enhancement medical image segmentation method based on Prompt according to claim 2, characterized in that: If the low-resolution feature map is S1 and the hint semantic information is S2, the fusion process through the cross-attention mechanism includes: S3=softmax((W Q S2)(W K S1) T )W V S1 Among them, S3 is the feature map obtained by fusing the low-resolution feature map with the hint semantic information through the cross-attention mechanism; W Q 、W K 、W V is the trainable cross-attention mechanism weight matrix; W Q S2 is the query matrix in the cross-attention mechanism, W K S1 is the key matrix in the cross-attention mechanism, W V S1 is the value matrix in the cross-attention mechanism; (·) T Represents the transpose of a matrix.

5. The edge enhancement medical image segmentation method based on Prompt according to claim 1, characterized in that: Aligning the high-resolution feature map with the low-resolution feature map includes the following steps: The high-resolution feature map is processed using a 1×1 convolution module to obtain a corresponding high-resolution feature map. The low-resolution feature map is upsampled using a 3×3 convolution module to obtain an upsampled low-resolution feature map, so that the size of the upsampled low-resolution feature map matches the high-resolution feature map. The upsampled low-resolution feature map and the high-resolution feature map are concatenated and processed using a 3×3 convolution module to obtain a combined feature map. The combined feature map is processed using the optical flow estimation method to obtain an optical flow feature map; Input the optical flow feature map and the high-resolution feature map into the warp network for local alignment to obtain a locally aligned feature map; The locally aligned feature map is added to the upsampled low resolution to obtain the aligned feature map.

6. The Prompt-based edge enhancement medical image segmentation method according to claim 1 or 5, characterized in that: The process of feature enhancement based on the feature map after feature alignment of the high-resolution image and the low-resolution image includes: Based on the spatial attention mechanism, a spatial attention map is generated according to the aligned feature map, and the aligned feature map is weighted using the spatial attention map to obtain a first weighted feature map; Use global average pooling to generate channel statistics of the aligned feature map, and generate the weight of each channel based on the channel attention mechanism and channel statistics; Use the weight of each channel to weight each channel of the aligned feature map to obtain a second weighted feature map; The first weighted feature map and the second weighted feature map are fused by addition to obtain an enhanced feature map.

7. The edge enhancement medical image segmentation method based on Prompt according to claim 6, characterized in that: Using the spatial attention map to weight the aligned feature map to obtain the first weighted feature map includes the following steps: A 1×1 convolutional layer with two activation functions is used to generate the spatial attention map. Right now: Use att SA For the input feature F h′ Perform spatial modulation to obtain the first weighted feature map, namely: Where σ(·) is the sigmoid activation function; represents the weights of two 1×1 convolutional layers; δ(·) represents the ReLU activation function; F h′ is the aligned feature map; represents the aligned feature map, Indicates H h ×W h ×C h dimensional real space, H h Represents the height of the high-resolution feature map, W h represents the width of the high-resolution feature map, C h The number of channels representing the high-resolution feature map; represents the Kronecker product.

8. The Prompt-based edge enhancement medical image segmentation method according to claim 6, characterized in that: Using the weight of each channel to weight each channel of the aligned feature map to obtain a second weighted feature map includes the following steps: Generate aligned feature maps using global average pooling F h 'Calculate its channel statistics G CA , G CA The calculation of the cth element is expressed as: A simple gating mechanism with sigmoid activation is used to generate the channel attention map att CA , expressed as: Using channel attention map att CA To adjust the input feature F h′ ,Right now: Among them, H h Represents the aligned feature map F h′ High; W h Represents the aligned feature map F h′ width; F h′c (i, j) represents the element at position (i, j) in the cth channel, i∈{1, 2, ..., Hh}, j∈{1, 2, ..., W h }; represents the weights of the two 1×1 convolutional layers in the channel attention; δ(·) represents the ReLU function; is the channel attention map att CA The attention map of the c-th channel in, is the aligned feature map F h′ The feature map of the cth channel in; represents the Kronecker product.

9. A Prompt-based edge-enhanced medical image segmentation system, characterized in that: The method for implementing the Prompt-based edge-enhanced medical image segmentation method of claim 1 comprises a preprocessing module, an encoder, a decoder, an edge-aware prompt learning module, and a bottleneck layer, wherein the feature maps of each layer in the encoder and the decoder are matched with each other; wherein: A preprocessing module is used to preprocess the input bone CT image, including converting the bone CT image into a feature map and performing enhancement and annotation; Encoder, used to perform multiple downsampling operations on the input feature map; The edge perception module is used to obtain hint semantic information based on the fusion of detection box and edge information; The bottleneck layer is used to filter the hint semantic information and obtain the initial high-resolution feature map input to the decoder; The decoder is used to perform multiple upsampling on the input feature map corresponding to the decoder. During the upsampling process, the output of the corresponding layer of the decoder is upsampled as a low-resolution feature map. The low-resolution feature map and the high-resolution feature map are aligned and enhanced as the output of the current layer. The output of the current layer is used as the high-resolution feature map of the next layer. The feature map output by the last layer of the decoder is the segmentation result.

Citation Information

Patent Citations

  • 3D medical image segmentation system and method

    CN116758100A

  • Arbitrary focus segmentation and tracking system based on medical ultrasonic image

    CN117350979A

  • Interaction system based on edge AIGC architecture and AR glasses

    CN117742497A

  • Automatic unlabeled pancreas image segmentation system based on adversarial learning

    WO2023098289A1

Cited By

  • Image detection method and system based on fusion medical image learning

    CN119478381A