Prompt-based Edge Enhancement Medical Image Segmentation Method and System

Abstract

The present invention belongs to the fields of computer vision and medical images, and particularly relates to a Prompt-based medical image segmentation method and system for edge enhancement. The method includes obtaining a skeletal CT image, continuously downsampling the image through an encoder, and using the features obtained by each layer of downsampling as the high-resolution feature maps input to the upsampling layer in the corresponding decoder; obtaining edge information, box information, and mask information from the skeletal CT image through an edge-aware learning module, and fusing the obtained information with the features output by the encoder as the input to the bottleneck layer; in the encoder, using the output of the bottleneck layer or the previous layer as the low-resolution feature map, performing feature alignment and feature enhancement on the low-resolution feature map and the high-resolution feature map as the output of the current layer, and using the output of the last layer of the encoder as the image segmentation result. The present invention improves the effect of image segmentation, thereby saving the time of medical staff spent on reading and understanding CT images.

Description

Technical Field

[0001] The present invention belongs to the fields of computer vision and medical images, and particularly relates to a Prompt-based medical image segmentation method and system for edge enhancement. Background Art

[0002] Medical image segmentation is an important branch of image segmentation and plays an important role in clinical applications such as diagnosis, surgical planning, and image-guided surgery. Especially in the clinical diagnosis of osteonecrosis, the segmentation of bone CT images can facilitate doctors' diagnosis and analysis of the condition, thus reducing the workload of doctors. Although bones have obvious contrast in CT images, their segmentation is still challenging. Since bones are different from pathological features such as tumors, the edges of different bones are unique. Especially, the bones with osteonecrosis will cause depressions of different degrees. Currently, most automatic methods lack the learning of information on the edge parts of bones, resulting in inaccurate segmentation. In addition, the Transformer-based architecture has excellent global context modeling ability, so it is widely popular in medical image segmentation tasks. However, the method based on the Transformer encoder is prone to destroying some local features when modeling the global context, and the edge prompts cannot handle the local information of bones well. Summary of the Invention

[0003] Aiming at the deficiencies in the prior art, the present invention proposes a Prompt-based medical image segmentation method for edge enhancement, constructs an image segmentation network, which includes an encoder, a bottleneck layer, a decoder, and an edge-aware learning module. The process of image segmentation includes the following steps:

[0004] Obtain a bone CT image, and continuously downsample the image through the encoder. The features obtained from each layer of downsampling are used as the high-resolution feature maps input to the upsampling layer in the corresponding decoder;

[0005] Obtain edge information, box information, and mask information from the bone CT image through the edge-aware learning module, and fuse the obtained information with the features output by the last downsampling layer of the encoder as the input to the bottleneck layer;

[0006] In the encoder, use the output of the bottleneck layer or the previous layer as the low-resolution feature map, perform feature alignment and feature enhancement on the low-resolution feature map and the high-resolution feature map, and use the output as the output of the current layer. The output of the last layer of the encoder is used as the image segmentation result.

[0007] The present invention also proposes a Prompt-based edge-enhanced medical image segmentation system for implementing a Prompt-based edge-enhanced medical image segmentation method, including a preprocessing module, an encoder, a decoder, an edge-aware prompt learning module, and a bottleneck layer, where the feature maps of each layer in the encoder and decoder match each other; wherein:

[0008] The preprocessing module is used to preprocess the input bone CT image, including converting the bone CT image into a feature map and performing enhancement and annotation;

[0009] The encoder is used to perform multiple downsampling operations on the input feature map;

[0010] The edge-aware module is used to fuse the detection box and edge information to obtain prompt semantic information;

[0011] The bottleneck layer is used to filter the semantic information of the prompt semantic information to obtain the initial high-resolution feature map input to the decoder;

[0012] The decoder is used to perform multiple upsamplings on the input feature map corresponding to the decoder. During the upsampling process, the output of the corresponding layer of the decoder is upsampled as the low-resolution feature map, and the low-resolution feature map and the high-resolution feature map are image-aligned and image-enhanced as the output of the current layer, and the output of the current layer is used as the high-resolution feature map of the next layer. The feature map output by the last layer of the decoder is the segmentation result.

[0013] Further, the processing process of the edge-aware learning module for the bone CT image, such as Figure 5 , specifically includes the following steps:

[0014] Obtain the detection box through manual annotation, and extract the edge information from the bone CT image using an edge detector;

[0015] Perform positional encoding on the detection box and edge information respectively, and then generate corresponding weights through MLP and multiply them with the positional encoding to obtain the corresponding box embedding vector and edge embedding vector;

[0016] Concatenate the edge embedding vector and the box embedding vector together as the sparse embedding vector;

[0017] Convert the manually annotated mask in the bone CT image into a dense embedding vector through continuous downsampling;

[0018] Concatenate the sparse embedding vector and the dense embedding vector together as the prompt speech information;

[0019] The low-resolution feature map is fused with the prompt semantic information through a cross-attention mechanism.

[0020] Further, the process of continuously downsampling the mask in the image, that is, processing it with three cascaded downsampling modules. The first and second downsampling modules are composed of a cascaded 2×2 convolutional unit, a layer normalization unit, and a GELU activation function; the third downsampling module is composed of a 1×1 convolutional unit.

[0021] Further, if the low-resolution feature map is S1 and the prompt semantic information is S2, the process of fusing them through the cross-attention mechanism includes:

[0022] S3 = softmax((W Q S2)(W K S1) T )W V S1

[0023] where S3 is the feature map obtained by fusing the low-resolution feature map with the prompt semantic information through the cross-attention mechanism; W Q 、W K 、W V are trainable cross-attention mechanism weight matrices; W Q S2 is the query matrix in the cross-attention mechanism, W K S1 is the key matrix in the cross-attention mechanism, W V S1 is the value matrix in the cross-attention mechanism; (·) T represents the transpose of the matrix.

[0024] Further, aligning the high-resolution feature map and the low-resolution feature map, such as Figure 6 , includes the following steps:

[0025] Using a 1×1 convolutional module to process the high-resolution feature map to obtain the corresponding high-resolution feature map, using a 3×3 convolutional module to upsample the low-resolution feature map to obtain an upsampled low-resolution feature map, and making the size of the upsampled low-resolution feature map match that of the high-resolution feature map; After concatenating the upsampled low-resolution feature map and the high-resolution feature map, using a 3×3 convolutional module to process them to obtain a combined feature map;

[0026] Using the optical flow estimation method to process the combined feature map to obtain an optical flow feature map;

[0027] Inputting the optical flow feature map and the high-resolution feature map into the warp network for local alignment to obtain a locally aligned feature map;

[0028] Adding the locally aligned feature map to the upsampled low-resolution to obtain an aligned feature map.

[0029] Further, the process of feature enhancement based on the feature map after feature alignment of the high-resolution image and the low-resolution image, such asFigure 7 , specifically including the following steps:

[0030] Based on the spatial attention mechanism, generate a spatial attention map according to the aligned feature map, and use the spatial attention map to weight the aligned feature map to obtain a first weighted feature map;

[0031] Use global average pooling to generate channel statistics of the aligned feature map, and generate the weight of each channel based on the channel attention mechanism and channel statistics;

[0032] Use the weight of each channel to weight each channel of the aligned feature map respectively to obtain a second weighted feature map;

[0033] Fuse the first weighted feature map and the second weighted feature map by addition to obtain an enhanced feature map.

[0034] Furthermore, using the spatial attention map to weight the aligned feature map to obtain the first weighted feature map includes the following steps:

[0035] Use a 1×1 convolutional layer with two activation functions to generate the spatial attention map That is:

[0036]

[0037] Use att SA Perform spatial modulation on the input feature F h′ to obtain the first weighted feature map, that is:

[0038]

[0039] where σ(·) is the sigmoid activation function; represent the weights of two 1×1 convolutional layers; δ(·) represents the ReLU activation function; F h′ is the aligned feature map; represents the aligned feature map, represents H h ×W h ×C h dimensional real space, H h represents the height of the high-resolution feature map, W h represents the width of the high-resolution feature map, C h represents the number of channels of the high-resolution feature map; represents the Kronecker product.

[0040] Furthermore, using the weight of each channel to weight each channel of the aligned feature map respectively to obtain the second weighted feature map includes the following steps:

[0041] Generate the aligned feature map F using global average pooling h′ Calculate its channel statistics G CA , G CA The calculation of the c-th element of G is expressed as:

[0042]

[0043] Use a simple gating mechanism with sigmoid activation to generate the channel attention map att CA , expressed as:

[0044]

[0045] Use the channel attention map att CA to adjust the input feature F h′ , that is:

[0046]

[0047] where, represents the element at the position (i, j) in the c-th channel, i ∈ {1, 2, …, H h} and j ∈ {1, 2, …, W h}; represents the weights of the two 1×1 convolutional layers in the channel attention; is the attention map of the c-th channel in the channel attention map att CA , is the feature map of the c-th channel in the aligned feature map F h′ .

[0048] The present invention also proposes a Prompt-based edge-enhanced medical image segmentation system for implementing a Prompt-based edge-enhanced medical image segmentation method, including a preprocessing module, an encoder, a decoder, an edge-aware prompt learning module, and a bottleneck layer, and the feature maps of each layer in the encoder and decoder match each other; where:

[0049] The preprocessing module is used to preprocess the input skeletal CT image, including converting the skeletal CT image into a feature map and performing enhancement and annotation;

[0050] The encoder is used to perform multiple downsampling operations on the input feature map;

[0051] The edge-aware module is used to fuse the detection box and edge information to obtain the prompt semantic information;

[0052] The bottleneck layer is used to filter the prompt semantic information to obtain the initial high-resolution feature map input to the decoder;

[0053] A decoder is used to perform multiple upsamplings on the corresponding decoder of the input feature map. During the upsampling process, the output of the corresponding layer of the decoder is upsampled as a low-resolution feature map. After image alignment and image enhancement of the low-resolution feature map and the high-resolution feature map, it is used as the output of the current layer, and the output of the current layer is used as the high-resolution feature map of the next layer. The feature map output by the last layer of the decoder is the segmentation result.

[0054] In the medical image segmentation method, in view of the problems of insufficient edge processing and local information loss, compared with the prior art, the present invention has the following beneficial effects:

[0055] (1) The present invention obtains the edge of the real image through the edge perception module. First, it fuses the Box and Edge hint information, processes the mask hint information, and then combines the Sparse_Embeddings and Dense_Embeddings with the picture through the fusion module, which can effectively capture and enhance the edge information of the image, making the edge transition more natural and smooth.

[0056] (2) The present invention combines the context information through the feature alignment enhancement module to ensure that local features are not lost during upsampling; by combining the spatial information of the image, it ensures that global features are not lost.

[0057] In summary, the present invention improves the effect of image segmentation, thereby saving the time spent by medical staff in reading and understanding CT images and intuitively displaying abnormal bone morphology. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a framework diagram of a Prompt-based edge-enhanced medical image segmentation method of the present invention;

[0059] Figure 2 It is a schematic diagram of the process of the edge perception prompt learning framework in a Prompt-based edge-enhanced medical image segmentation method of the present invention;

[0060] Figure 3 It is a schematic diagram of the context feature alignment enhancement process in a Prompt-based edge-enhanced medical image segmentation method of the present invention;

[0061] Figure 4 It is a flowchart of a Prompt-based edge-enhanced medical image segmentation method of the present invention;

[0062] Figure 5 It is a flowchart of the processing process of the edge perception learning module for bone CT images in a Prompt-based edge-enhanced medical image segmentation method of the present invention;

[0063] Figure 6 It is a flowchart for feature alignment of high - resolution feature maps and low - resolution feature maps in a Prompt - based edge - enhanced medical image segmentation method of the present invention;

[0064] Figure 7 It is a flowchart for feature enhancement of the aligned feature maps in a Prompt - based edge - enhanced medical image segmentation method of the present invention. Detailed implementation manners

[0065] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0066] The present invention proposes a Prompt - based edge - enhanced medical image segmentation method. As Figure 4 , the process of image segmentation includes the following steps:

[0067] S0. Construct an image segmentation network, which includes an encoder, a bottleneck layer, a decoder, and an edge - aware learning module;

[0068] S1. Obtain a skeletal CT image, and continuously downsample the image through the encoder. The features obtained by each layer of downsampling are used as the high - resolution feature maps input to the upsampling layer in the corresponding decoder;

[0069] S2. Obtain edge information, box information, and mask information from the skeletal CT image through the edge - aware learning module, and fuse the obtained information with the features output by the last downsampling layer of the encoder as the input to the bottleneck layer;

[0070] S3. In the encoder, use the output of the bottleneck layer or the previous layer as the low - resolution feature map, perform feature alignment and feature enhancement on the low - resolution feature map and the high - resolution feature map as the output of the current layer, and use the output of the last layer of the encoder as the image segmentation result.

[0071] In this embodiment, in order to implement a Prompt - based edge - enhanced medical image segmentation method, the skeletal CT image is processed as follows:

[0072] First, preprocess the skeletal CT image;

[0073] Then obtain the edge information of the skeletal image, as Figure 1, in this embodiment, an edge detector (Edge Detection) is used, and combined with mask and box information, prompt information is generated through a prompt learning module (Prompt Encoder is used in this embodiment) and combined with the feature information processed by the encoder;

[0074] Finally, the decoder performs feature fusion with the encoder through skip connections (Mixer is used for fusion in this embodiment), and combines the context information through the context feature alignment enhancement module to ensure that global and local information are not lost, and finally generates a bone segmentation image.

[0075] In this embodiment, the preprocessing of the bone CT image includes annotating and enhancing the image, including annotating the bone part in the bone CT image, and enhancing the image data by rotating, horizontally flipping, and vertically flipping.

[0076] The present invention uses an encoder to continuously downsample the image, such as Figure 1 , in this embodiment, three Swin Transfamer downsampling blocks are used to continuously downsample the input bone CT image. In this embodiment, Figure 1 the three Swin Transfamer downsampling blocks from left to right are respectively defined as the first Swin Transfamer downsampling block, the second Swin Transfamer downsampling block, and the third Swin Transfamer downsampling block. Each Swin Transfamer downsampling block outputs a high-resolution feature map of one size, and the three downsampling blocks from left to right respectively output the first high-resolution feature map, the second high-resolution feature map, and the third high-resolution feature map.

[0077] In this embodiment, the edge information of the real image can be extracted through the Sobel operator. The extraction process includes the following steps:

[0078] 101. Set the Sobel convolution factor. This operator consists of two 3×3 matrices, corresponding to the horizontal and vertical directions of the image respectively. The Sobel convolution factor is expressed as:

[0079]

[0080] These two groups of matrices perform planar convolution operations with the image, and the approximate values of the brightness differences in the horizontal and vertical directions of the image can be calculated respectively. If A represents the original image, G x and G y represent the gray values of the image after horizontal and vertical edge detection respectively, and the formula is as follows:

[0081]

[0082] Among them, * represents a planar convolution operation.

[0083] 102. Calculate the grayscale value of each pixel point in the image. In this embodiment, the grayscale values in the horizontal and vertical directions are comprehensively considered. This calculation process can be implemented through the following mathematical formula:

[0084]

[0085] To improve the calculation efficiency, an approximate method that does not perform square root operations is often used to estimate the grayscale value of pixel points. This method can effectively simplify the calculation process while still maintaining sufficient accuracy, especially suitable for scenarios with high requirements for calculation speed. As an alternative implementation, the approximate calculation formula is as follows:

[0086] |G| = |G x | + |G y |

[0087] 203. Then, it is necessary to calculate the gradient direction of each pixel point. This direction can be determined by G x and G y , that is, the change in grayscale values of the pixel point in the horizontal and vertical directions. The calculation formula for the gradient direction is as follows:

[0088]

[0089] 204. After obtaining the gradient direction, in this embodiment, it is necessary to determine whether the pixel point is an edge point. This is judged by comparing the magnitude of the gradient (denoted by δ) with a preset threshold. If δ is greater than this threshold, then the pixel point (x, y) is considered an edge point.

[0090] In this embodiment, the preprocessed and enhanced images are used as the inputs of the encoder and the edge-aware prompt learning module respectively. The encoder downsamples the image, and the edge-aware prompt learning module extracts semantic information from the image. As Figure 5 , the process of the edge-aware prompt learning module extracting semantic information from the image specifically includes the following steps:

[0091] Obtain detection boxes through manual annotation, and extract edge information from the skeletal CT image using an edge detector;

[0092] Perform position encoding on the detection boxes and edge information respectively, and then generate corresponding weights through MLP and multiply them with the position encoding to obtain corresponding box embedding vectors and edge embedding vectors;

[0093] Concatenate the edge embedding vector and the box embedding vector together as a sparse embedding vector;

[0094] Convert the manually annotated mask in the bone CT image into a dense embedding vector through continuous downsampling;

[0095] Concatenate the sparse embedding vector and the dense embedding vector together as the prompt voice information;

[0096] The low-resolution feature map is fused with the prompt semantic information through the cross-attention mechanism.

[0097] Specifically, the annotated box information and the detected edge information are converted into an edge embedding vector (EmbedEdges) and a box embedding vector (Embed Boxes). This step involves encoding the prominent edge and frame structures in the image, converting them into numerical vectors that can be processed by the neural network.

[0098] The edge embedding vector and the box embedding vector are further combined into a sparse embedding vector (SparseEmbeddings). The purpose of this step is to merge the edge and box information to provide a more detailed and comprehensive representation of the image structure, such as Figure 2 , and the process of synthesizing the sparse embedding vector includes the following steps:

[0099] Obtain the detection box through manual annotation, and extract the edge information from the bone CT image using an edge detector;

[0100] Perform positional encoding on the detection box and the edge information respectively, and then generate the corresponding weights through MLP to multiply with the positional encoding to obtain the corresponding box embedding vector and edge embedding vector;

[0101] Concatenate the edge embedding vector and the box embedding vector together as the sparse embedding vector.

[0102] The mask in the image is converted into a dense embedding vector (DenseEmbeddings) through a series of downsampling steps. This conversion makes the mask information more compact and concentrated. Convert the manually annotated mask in the bone CT image into a dense embedding vector through continuous downsampling. The process of continuous downsampling of the mask in the image is to process it using three cascaded downsampling modules. The first and second downsampling modules are composed of cascaded 2×2 convolutional units, layer normalization units, and GELU activation functions; the third downsampling module is composed of 1×1 convolutional units.

[0103] The dense embedding vector and the sparse embedding vector are concatenated as the prompt semantic information. In this embodiment, a cross-attention mechanism is used to fuse the high-resolution image with the prompt semantic information. In this embodiment, the third high-resolution image output by the third Swin Transfamer downsampling block is fused with the prompt semantic information. If the low-resolution feature map is S1 and the prompt semantic information is S2, the process of fusion through the cross-attention mechanism includes:

[0104] S3 = softmax((W Q S2)(W K S1) T )W V S1

[0105] where S3 is the feature map obtained by fusing the high-resolution feature map with the prompt semantic information through the cross-attention mechanism; W Q , W K , V W are trainable cross-attention mechanism weight matrices; W Q S2 is the query matrix in the cross-attention mechanism, W K S1 is the key matrix in the cross-attention mechanism, W V S1 is the value matrix in the cross-attention mechanism; (·) T represents the transpose of the matrix.

[0106] The feature map S3 obtained by fusing the low-resolution feature map with the prompt semantic information through the cross-attention mechanism is filtered through a bottleneck layer. In this embodiment, two cascaded Swin Transfamer blocks are used as the bottleneck layer.

[0107] In this implementation, the decoder corresponding to the encoder includes a three-level cascaded structure. Each structure is composed of a cascaded SwinTransfamer upsampling block and a context feature alignment enhancement block, as Figure 1 , from left to right in the decoder are the first-level structure, the second-level structure, and the third-level structure. The first-level structure is composed of a cascaded first SwinTransfamer upsampling block and a first context feature alignment enhancement block. The second-level structure is composed of a cascaded second Swin Transfamer upsampling block and a second context feature alignment enhancement block. The third-level structure is composed of a cascaded third Swin Transfamer upsampling block and a third context feature alignment enhancement block.

[0108] In the first-level structure, the high-resolution feature map obtained by upsampling the third high-resolution image output by the third Swin Transfamer downsampling block into the first Swin Transfamer upsampling block is used as the feature map Fh and the low-resolution feature map output by the bottleneck layer as feature map F l Input the first context feature alignment and enhancement block for feature alignment and local enhancement;

[0109] In the second hierarchical structure, the high-resolution feature map obtained by upsampling the second high-resolution image output by the second Swin Transfamer downsampling block through the second Swin Transfamer upsampling block is used as feature map F h and the low-resolution feature map after feature alignment and local enhancement by the first context feature alignment and enhancement block in the first layer structure as feature map F l Input the second context feature alignment and enhancement block for feature alignment and local enhancement;

[0110] In the third hierarchical structure, the high-resolution feature map obtained by upsampling the first high-resolution image output by the first Swin Transfamer downsampling block through the third Swin Transfamer upsampling block is used as feature map F l and the low-resolution feature map after feature alignment and local enhancement by the second context feature alignment and enhancement block in the second layer structure as feature map F h Input the third context feature alignment and enhancement block for feature alignment and local enhancement;

[0111] The feature map after feature alignment and local enhancement by the third context feature alignment and enhancement block is used as the output of the decoder.

[0112] In this embodiment, the spatial information is combined through the decoder to ensure that the global features are not lost, and the context information is combined through the context feature alignment and enhancement module to ensure that the local features are not lost during upsampling. Two feature maps F l and F h are used as the input of the context feature alignment and enhancement module. The processing process of the context feature alignment and enhancement module for the two feature maps F l and F h includes two parts: feature alignment and local enhancement, and specifically includes the following steps:

[0113] 301. The high-resolution feature map from the corresponding stage of the decoder, represents a set of real numbers with dimensions H h ×W h ×C h where H h 、W h 、C hThey are the height, width, and number of channels of the corresponding high-resolution feature map respectively. This feature map contains more detailed information and is crucial for retaining local features of the image; while the low-resolution image comes from the previous stage of the corresponding stage, represents a set of real numbers with dimensions H l ×W l ×C l where H l and W l and C l are the height, width, and number of channels of the corresponding low-resolution feature map respectively; the high-resolution feature map is processed through a 1×1 convolutional module. For the low-resolution feature map, it is upsampled through a 3×3 convolutional module to adjust its size to match that of the high-resolution feature map.

[0114] 302. After splicing the first low-resolution feature map and the first high-resolution feature map, they are processed using a 3×3 convolutional module to obtain a combined feature map, and the combined feature map is processed using an optical flow estimation method to obtain an optical flow feature map; in this embodiment, the FlowNetSimple architecture can be used to obtain the corresponding flow field feature map.

[0115] 303. Input the optical flow feature map and the first high-resolution feature map into the warp network for local alignment to obtain a locally aligned feature map. Since there is a resolution gap between the features and the flow field, the present invention halves the offset through the warp network. The warp network adopted in the present invention, namely local alignment (Mesh Warps) in the prior art, is a patch-based matching method. The image is divided into multiple grid-like regions, and geometric transformations of multiple regions are constructed through local feature matching. Each region is differentially divided into a plane for alignment, which solves the problem of depth disparity to a certain extent.

[0116] Those skilled in the art can use any optical flow estimation method and local alignment to perform alignment processing on the image, and the present invention does not make other limitations on the optical flow estimation method and local alignment.

[0117] 304. Add the locally aligned feature map to the first low-resolution feature map to obtain an aligned feature map.

[0118] After completing the alignment of the image, the aligned image is input into the feature enhancement module for processing. In this embodiment, the aligned feature map is divided into two branches and processed using spatial attention and channel attention respectively, and then the data processed by spatial attention and channel attention are fused by addition. The fused feature map is used as the output of the context feature alignment enhancement block, as shown in Figure 3 the system processing block diagram and Figure 7For the flowchart shown, the process of local enhancement specifically includes the following steps:

[0119] 401. Based on the spatial attention mechanism, generate a spatial attention map according to the aligned feature map, and use the spatial attention map to weight the aligned feature map to obtain the first weighted feature map. Specifically, it includes the following steps:

[0120] Use a 1×1 convolutional layer with two activation functions to generate the spatial attention map That is:

[0121]

[0122] Use att SA Perform spatial modulation on the input feature F h′ to obtain the first weighted feature map, that is:

[0123]

[0124] where σ(·) is the sigmoid activation function; represents the weights of two 1×1 convolutional layers; δ(·) represents the ReLU activation function; F h′ is the aligned feature map; represents the aligned feature map, represents H h ×W h ×C h dimensional real space, H h represents the height of the high-resolution feature map, W h represents the width of the high-resolution feature map, C h represents the number of channels of the high-resolution feature map; represents the Kronecker product.

[0125] 402. Use global average pooling to generate the channel statistics of the aligned feature map, and generate the weight of each channel based on the channel attention mechanism and channel statistics. Specifically, it includes the following steps:

[0126] Use global average pooling to generate the aligned feature map F h′ Calculate its channel statistics G CA , and the calculation of the c-th element of G CA is expressed as:

[0127]

[0128] Use a simple gating mechanism with sigmoid activation to generate the channel attention map att CA , which is expressed as:

[0129]

[0130] Using the channel attention map att CA to adjust the input feature F h′ , that is:

[0131]

[0132] wherein, represents the element at position (i, j) in the c-th channel, i ∈ {1, 2, …, H h}, j ∈ {1, 2, …, W h}; represents the weights of two 1×1 convolutional layers in the channel attention; is the attention map of the c-th channel in the channel attention map att CA , and is the feature map of the c-th channel in the aligned feature map F h′ .

[0133] 403. Fuse the first weighted feature map and the second weighted feature map by addition to obtain an enhanced feature map.

[0134] In addition, in Figure 3 , for the spatial attention branch, after performing attention processing, two cascaded 1×1 convolutional layers are used for processing; correspondingly, in the spatial attention branch, the input feature map is first processed by two cascaded 1×1 convolutional layers and then channel attention processing is performed, and finally the outputs of the two branches are fused by addition.

[0135] The present invention also provides a Prompt-based edge-enhanced medical image segmentation system, which is used to implement a Prompt-based edge-enhanced medical image segmentation method. The system includes a preprocessing module, an encoder, a decoder, an edge-aware prompt learning module, and a bottleneck layer, and the feature maps of each layer in the encoder and decoder match each other; wherein: the preprocessing module is used to preprocess the input skeletal CT image, including converting the skeletal CT image into a feature map and performing enhancement and annotation;

[0136] The encoder is used to perform multiple downsampling operations on the input feature map;

[0137] The edge-aware module is used to fuse the detection box and the edge information to obtain the prompt semantic information. The prompt semantic information is obtained by fusing Sparse_Embeddings and Dense_Embeddings, and the present invention fuses the prompt semantic information with the image information (i.e., the feature map obtained by downsampling the decoder), effectively capturing and enhancing the edge information of the image, making the edge transition more natural and smooth;​​

[0138] A bottleneck layer for filtering the hint semantic information to obtain an initial high-resolution feature map input to the decoder;

[0139] A decoder for performing multiple upsamplings on the input feature map corresponding to the decoder. During the upsampling process, the output of the corresponding layer of the decoder is upsampled as the high-resolution feature map, and the high-resolution feature map and the low-resolution feature map are subjected to image alignment and image enhancement as the output of the current layer, and the output of the current layer is used as the low-resolution feature map of the next layer. The feature map output by the last layer of the decoder is the segmentation result. Each upsampling process includes a Swin Transfomer upsampling block and a context feature alignment and enhancement module for processing the input high-resolution feature map and low-resolution feature map. The upsampling process combines spatial information, channel information with the original image to ensure that global features are not lost and local features are not lost during upsampling.

[0140] This embodiment also provides a computing device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the following steps are implemented, including:

[0141] Input a skeletal CT image and perform preprocessing and enhancement on the image;

[0142] Process the skeletal CT image through an edge detector to obtain the edge information of the bone and generate corresponding segments; then fuse the Box and Edge hint information and process the mask hint information; then combine the Sparse_Embeddings and Dense_Embeddings with the picture through a fusion module to obtain hint information;

[0143] Downsample the skeletal CT image through an encoder to obtain low-level feature information;

[0144] Fuse the hint semantic information through a cross-attention fusion module after downsampling;

[0145] Combine the spatial information through the decoder to ensure that global features are not lost, and combine the context information through a feature alignment and enhancement module to ensure that local features are not lost during upsampling.

[0146] The decoder and the encoder perform feature fusion through skip connections to supplement the spatial information loss caused by downsampling.

[0147] The purpose of this embodiment is to provide a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the following steps are performed:

[0148] Input the skeletal CT image and preprocess and enhance the image;

[0149] Process the skeletal CT image through an edge detector to obtain the edge information of the bone and generate corresponding segments; then fuse the Box and Edge hint information and process the mask hint information; then combine the Sparse_Embeddings and Dense_Embeddings with the picture through a fusion module to obtain hint information;

[0150] Downsample the skeletal CT image through an encoder to obtain low-level feature information;

[0151] Fuse the hint semantic information through a cross-attention fusion module after downsampling;

[0152] Combine the spatial information through a decoder to ensure that the global features are not lost, and combine the context information through a feature alignment enhancement module to ensure that the local features are not lost during upsampling.

[0153] The decoder and the encoder perform feature fusion through skip connections to supplement the loss of spatial information caused by downsampling.

[0154] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A prompt-based edge enhancement medical image segmentation method, characterized in that: Construct an image segmentation network, which includes an encoder, a bottleneck layer, a decoder, and an edge-aware learning module. The image segmentation process includes the following steps: Obtain a bone CT image, and continuously downsample the image through the encoder. The features obtained from each downsampling layer are used as high-resolution feature maps input to the upsampling layer in the corresponding decoder. The edge information, frame information, and mask information are obtained from the bone CT image through the edge perception learning module. The obtained information is fused with the features output by the last downsampling layer of the encoder as the input of the bottleneck layer. In the encoder, the output of the bottleneck layer or the previous layer is used as a low-resolution feature map, the low-resolution feature map and the high-resolution feature map are feature aligned and feature enhanced as the output of the current layer, and the output of the last layer of the encoder is used as the image segmentation result. The process of feature enhancement based on the feature map after feature alignment of the high-resolution image and the low-resolution image includes: Based on the spatial attention mechanism, a spatial attention map is generated according to the aligned feature map, and the aligned feature map is weighted by using the spatial attention map to obtain a first weighted feature map, including: A 1×1 convolutional layer with two activation functions is used to generate the spatial attention map. Right now: Using att SA For the input feature F h′ Perform spatial modulation to obtain the first weighted feature map, namely: Among them, σ(·) is the sigmoid activation function; represents the weights of two 1×1 convolutional layers; δ(·) represents the ReLU activation function; represents the aligned feature map, Indicates H h ×W h ×C h dimensional real space, H h Represents the aligned feature map F h′ High; W h Represents the aligned feature map F h′ Width, C h Represents the aligned feature map F h′ The number of channels; represents the Kronecker product; Use global average pooling to generate channel statistics of the aligned feature maps, and generate the weight of each channel based on the channel attention mechanism and channel statistics; Each channel of the aligned feature map is weighted using the weight of each channel to obtain a second weighted feature map, including: Use global average pooling to generate the aligned feature map F h′ Calculate its channel statistics G CA , G CA The calculation of the cth element of is expressed as: A simple gating mechanism with sigmoid activation is used to generate the channel attention map att CA , expressed as: Using channel attention map att CA To adjust the input feature F h′ ,Right now: Among them, F h′c (i, j) represents the element at position (i, j) in the cth channel, i∈{1, 2, ..., H h }, j∈{1, 2, ..., W h }; represents the weights of the two 1×1 convolutional layers in the channel attention; δ(·) represents the ReLU function; is the channel attention map att CA The attention map of the cth channel in F h′c is the aligned feature map F h′ The feature map of the cth channel in; The first weighted feature map and the second weighted feature map are fused by adding each other to obtain an enhanced feature map.

2. The Prompt-based edge enhancement medical image segmentation method according to claim 1, characterized in that: The processing process of the edge-aware learning module for bone CT images includes: The detection frame is obtained by manual annotation, and the edge information is extracted from the bone CT image using an edge detector; The detection box and edge information are positionally encoded respectively, and then the corresponding weights are generated by MLP and multiplied with the position encoding to obtain the corresponding box embedding vector and edge embedding vector; Concatenate the edge embedding vector and the box embedding vector together as a sparse embedding vector; The manually annotated masks in the bone CT images are converted into dense embedding vectors through continuous downsampling; The sparse embedding vector and the dense embedding vector are concatenated together as the prompt speech information; The low-resolution feature maps are fused with the hint semantic information via a cross-attention mechanism.

3. The edge enhancement medical image segmentation method based on Prompt according to claim 2, characterized in that: The process of continuous downsampling of the mask in the image is processed by three cascaded downsampling modules. The first and second downsampling modules are composed of cascaded 2×2 convolution units, layer normalization units and GELU activation functions; the third downsampling module is composed of 1×1 convolution units.

4. The Prompt-based edge enhancement medical image segmentation method according to claim 2, characterized in that: If the low-resolution feature map is S1 and the hint semantic information is S2, the fusion process through the cross-attention mechanism includes: <h2 style=";text-align:left;direction:ltr">S3 = softmax((W<h2 style=";text-align:left;direction:ltr"> Q <h2 style=";text-align:left;direction:ltr"> S2)(W<h2 style=";text-align:left;direction:ltr"> K <h2 style=";text-align:left;direction:ltr"> S1)<h2 style=";text-align:left;direction:ltr"> T <h2 style=";text-align:left;direction:ltr"> W)<h2 style=";text-align:left;direction:ltr"> V <h2 style=";text-align:left;direction:ltr"> S1 Among them, S3 is the feature map obtained by fusing the low-resolution feature map with the hint semantic information through the cross-attention mechanism; W Q , W K , W V is the trainable cross-attention mechanism weight matrix; W Q S2 is the query matrix in the cross-attention mechanism, W K S1 is the key matrix in the cross-attention mechanism, W V S1 is the value matrix in the cross-attention mechanism; (·) T Represents the transpose of a matrix.

5. The Prompt-based edge enhancement medical image segmentation method according to claim 1, characterized in that: Aligning the high-resolution feature map with the low-resolution feature map includes the following steps: The high-resolution feature map is processed by a 1×1 convolution module to obtain a corresponding high-resolution feature map, and the low-resolution feature map is upsampled by a 3×3 convolution module to obtain an upsampled low-resolution feature map, so that the size of the upsampled low-resolution feature map matches the high-resolution feature map; the upsampled low-resolution feature map is concatenated with the high-resolution feature map and processed by a 3×3 convolution module to obtain a combined feature map; The combined feature map is processed using the optical flow estimation method to obtain an optical flow feature map; The optical flow feature map and the high-resolution feature map are input into the warp network for local alignment to obtain a local aligned feature map; The locally aligned feature map is added to the upsampled low resolution to obtain the aligned feature map.

6. A prompt-based edge-enhanced medical image segmentation system, characterized in that: The method for implementing the Prompt-based edge-enhanced medical image segmentation method of claim 1 comprises a preprocessing module, an encoder, a decoder, an edge-aware prompt learning module and a bottleneck layer, wherein the feature maps of each layer in the encoder and the decoder match each other; wherein: A preprocessing module, used for preprocessing the input bone CT image, including converting the bone CT image into a feature map and performing enhancement and annotation; The encoder is used to perform multiple downsampling operations on the input feature map; The edge perception module is used to obtain the hint semantic information based on the fusion of the detection box and edge information; The bottleneck layer is used to filter the hint semantic information and obtain the initial high-resolution feature map input to the decoder; The decoder is used to perform multiple upsampling on the input feature map corresponding to the decoder. During the upsampling process, the output of the corresponding layer of the decoder is upsampled as a low-resolution feature map, the low-resolution feature map and the high-resolution feature map are aligned and enhanced as the output of the current layer, and the output of the current layer is used as the high-resolution feature map of the next layer. The feature map output by the last layer of the decoder is the segmentation result.

Images