Image segmentation method based on frame prompt enhanced SK-VM + + network

Abstract

The invention relates to the technical field of medical image processing, in particular to a medical image segmentation method based on a frame prompt enhanced SK-VM + + network, and the method comprises the steps: collecting a to-be-segmented image in a data set, and carrying out the three-channel copying of the to-be-segmented image; extracting focus bounding box information in the text annotation file to generate a binary mask; generating a box prompt visualization graph by using the binary box prompt mask; fusing the binary box prompt mask and the copied three-channel image to obtain a four-channel tensor; and taking the four-channel tensor as input, constructing an SK-VM + + model, and performing segmentation processing by using the SK-VM + + model. The medical image segmentation method solves the problem that the existing medical image segmentation method is difficult to accurately deduce the spatial context only by means of the pixel intensity in a low-contrast and high-noise environment, and is easy to cause false detection and missing detection.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image segmentation method based on box cueing enhancement using the SK-VM++ network. Background Technology

[0002] Automatic medical image segmentation plays an increasingly important role in assisting clinical diagnosis. FCN and UNet and their variants utilize encoder-decoder architecture and skip connections to achieve multi-scale feature fusion and improve segmentation accuracy. However, they have inherent limitations in receptive field and long-range dependency modeling, which can lead to problems such as blurred boundaries, missed detections or false detections in medical images with low contrast and severe noise interference.

[0003] Other methods, such as Hornet and GFN, have proposed introducing higher-order spatial interactions and global filtering mechanisms; attention mechanisms have been introduced into segmentation networks, such as TransUNet and Swin-UNet. Although these methods can improve long-range dependency modeling to some extent, they often rely heavily on large-scale labeled data and computational resources, limiting their application in medical environments where labels are scarce and equipment is limited.

[0004] Nevertheless, the above methods still mainly rely on blind inference of pixel or depth features, lacking explicit guidance on lesion areas. In complex scenes with high noise, low contrast, and lesions and background textures similar, without additional spatial priors or cue information, the network still cannot completely reduce the risk of false detection and false negatives, affecting the reliability of the segmentation results. Summary of the Invention

[0005] To address the shortcomings of existing methods, this invention solves the problem that current medical image segmentation methods struggle to accurately infer spatial context based solely on pixel intensity in low-contrast, high-noise environments, easily leading to false positives and false negatives. The technical solution adopted in this invention is an image segmentation method based on the SK-VM++ network with bounding box cueing enhancement, comprising the following steps: Step 1: Collect the images to be segmented from the dataset and perform three-channel copying on the images to be segmented; extract the lesion bounding box information from the text annotation file to generate a binary mask; fuse the binary bounding box cue mask with the copied three-channel image to obtain a four-channel tensor; As a preferred embodiment of the present invention, step one specifically includes: Step 11: Perform grayscale processing on the image to be segmented to obtain the original single-channel grayscale image. ;Will Copy as pseudo-RGB image ; Step 12: Read the coordinates of the lesion bounding box from the annotation file of the image to be segmented. ;in The top left corner of the bounding box. The bottom right corner of the bounding box; Step 13: Define a binary mask using the two corners of the bounding box. ; As a preferred embodiment of the present invention The formula is:

[0006] in, This represents the position index of a pixel in the image. x and y This represents the coordinate index of the pixel in the image matrix.

[0007] Step 14, and By concatenating the channels, a four-channel tensor is obtained. .

[0008] In a preferred embodiment of the present invention, the channel dimension is spliced ​​using... concat function.

[0009] As a preferred embodiment of the present invention, data preprocessing and enhancement are performed on the four-channel tensor.

[0010] In a preferred embodiment of the present invention, data preprocessing and enhancement includes: firstly, randomly rotating the four-channel tensor by an angle. to Take values ​​in intervals, then randomize. Rotate by an integer multiple of the original value, and finally flip horizontally and vertically randomly.

[0011] Step 2: Using a four-channel tensor as input, construct an SK-VM++ model and use the SK-VM++ model for segmentation.

[0012] In a preferred embodiment of the present invention, the number of receiving channels of the SK-VM++ model is set to 4.

[0013] As a preferred embodiment of the present invention, an image segmentation system based on box-cue enhancement SK-VM++ network includes: a memory for storing instructions executable by a processor; and a processor for executing the instructions to implement the image segmentation method based on box-cue enhancement SK-VM++ network.

[0014] As a preferred embodiment of the present invention, a computer-readable medium storing computer program code implements an image segmentation method based on box-cue enhancement SK-VM++ network when executed by a processor.

[0015] The beneficial effects of this invention are: 1. This method is the first to introduce a binary cue mask based on bounding box information extracted from text annotation files into a medical image segmentation network, and concatenates it with a pseudo-RGB image to form a four-channel tensor, enabling the network to have explicit spatial localization capabilities. 2. Based on this, by combining the Mamba long-range dependency module with UNet++-style skip connections, the detection accuracy and robustness of lesion areas are significantly improved by simultaneously utilizing prompt information during multi-scale feature fusion, which is achieved in low-contrast and high-noise environments. Attached Figure Description

[0016] Figure 1 This is a diagram of the BP-SK-VM++ network structure of the present invention; Figure 2 This is a BP four-channel splicing diagram of the present invention; Figure 3 This invention relates to the visualization of four-channel tensors; Figure 4 This is a comparison of the thyroid ultrasound image segmentation results of the present invention. Detailed Implementation

[0017] The present invention will be further described below with reference to the accompanying drawings and embodiments. The drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0018] like Figure 1 As shown, the image segmentation method based on box cue enhancement using the SK-VM++ network includes the following steps: Step 1: Collect the images to be segmented from the dataset and perform three-channel copying on the images to be segmented; extract the lesion bounding box information from the text annotation file to generate a binary mask; fuse the binary bounding box cue mask with the copied three-channel image to obtain a four-channel tensor; Binary box tooltip masks can be used to generate tooltip visualizations; First, pseudo-RGB image construction: The image to be segmented is processed into grayscale to obtain the original single-channel grayscale image. The pseudo-RGB image is obtained by copying it three times. It is used to simulate conventional three-channel color input to improve the network's initial perception of grayscale information.

[0019] Secondly, binary bounding box cue mask generation: read the coordinates of the lesion bounding box given in the annotation file corresponding to the image to be segmented. ;in The top left corner of the bounding box. The bottom right corner of the bounding box; the bounding box is usually a rectangle; Define a binary mask using the coordinates of two angles. The calculation formula is as follows:

[0020] in, Indicates the position index of a pixel in the image; x and y The coordinate index of the pixel in the image matrix. x Represents the horizontal coordinate. y Represents the vertical coordinate.

[0021] Secondly, the four-channel tensor is constructed and visualized as follows: Figure 3 As shown: Pseudo-RGB image With binary hint mask By concatenating the channels, a four-channel tensor is obtained. Its definition is as follows: ; in, H and W Both are used to represent the spatial dimensions of the input image, where H is the height of the image, i.e., the number of rows in the pixel matrix; W The width of the image is the number of columns in the pixel matrix.

[0022] Finally, data preprocessing and augmentation: During the training phase, the four-channel tensor... Perform random data augmentation; including: random rotation angles at which... to Take values ​​randomly between intervals. Rotation by integer multiples, and random horizontal and vertical flips, to increase the model's robustness to lesion regions under different transformations; During the verification and testing phase, the input image and its corresponding binary mask were uniformly scaled to 256×256 to maintain the consistent input size of the SK-VM++ model.

[0023] Tag import: Import actual segmentation tags Stored in binary form as a NumPy format file (".npy"), where During network training or inference, the label file corresponding to the image is loaded into memory and used as a supervision signal for loss calculation and evaluation metric statistics.

[0024] In summary, by using the four-channel tensor construction method, this invention can explicitly provide coarse location information of lesions during the input stage of the SK-VM++ model, effectively guiding the subsequent feature extraction and fusion process, and improving the detection accuracy and segmentation robustness of lesion regions in low-contrast, high-noise medical images.

[0025] Step 2: Using a four-channel tensor as input, construct an SK-VM++ model and use the SK-VM++ model for segmentation processing; The bounding box location information extracted from the text annotation file is used to generate bounding box hints, and the four-channel tensor obtained based on the bounding box hints is used as input. The SK-VM++ model includes: Encoder section (downsampling process): To match the pseudo-RGB + box cue four-channel tensor of the present invention, the number of input channels in the pre-trained weight file of the existing three-channel input of SK-VM++ is modified. The BP-SK-VM++ model receives an input tensor with 4 channels and a spatial size of 512×512. The number of receiver channels in the coding layer 1 of the Res2Net50 backbone in SK-VM++ is changed from 3 to 4. That is, the input channels of the first convolutional layer (conv1) of coding layer 1 are adjusted from the original 3 channels to 4 channels, so that the SK-VM++ model can receive an input tensor with 4 channels and a spatial size of 512×512. In the encoder, the four-channel tensor is first subjected to a 7×7 large convolution (input channel 4 → output channel 64, stride 2, padding 3), BatchNorm (bn1), and ReLU (relu), downsampling the feature map from 512×512 to 256×256; then it is further downsampled to 128×128 by 3×3 maxpooling (stride 2, padding 1), while maintaining the number of channels C at 64; then it sequentially enters the four Res2Net backbone stages for encoding, namely encoding layer 2 to encoding layer 5: Encoding layer 2: No downsampling is performed, while the spatial size remains unchanged at 128×128 and the number of channels is expanded to 256; Encoding layer 3: Spatial downsampling to 64×64 and channel expansion to 512 is achieved through convolution with stride 2; Encoding layer 4: Similarly, downsample to 32×32 and expand the channels to 1024; Encoding layer 5: Downsample to 16×16 and expand the channels to 2048; The encoder design is compatible with four-channel tensors and enables multi-scale feature extraction and channel expansion from shallow to deep, providing a rich semantic foundation for subsequent multi-scale fusion.

[0026] The encoder layers 2-5 all use a 3×3 convolution to project the original number of channels to a uniform 64, ensuring that the input channels are consistent when the subsequent layers are fused, which facilitates chained splicing and PVMlayer operations. The features of coding layer 5 are 16×16×64, the features of coding layer 4 are 32×32×64, the features of coding layer 3 are 64×64×64, the features of coding layer 2 are 128×128×64, and the features of coding layer 1 are 128×128×64.

[0027] Decoder section (upsampling and feature fusion process): Decoding layer 4: (1) Adjacent layer fusion: The 16×16×64 dimensionality reduction feature of coding layer 5 is upsampled to 32×32 by bilinear interpolation, and added element by element with the 32×32×64 dimensionality reduction feature of coding layer 4 and the absolute value is taken. The result is fed into PVMLayer → BatchNorm → ReLU to obtain the first-level fusion feature x5_4.

[0028] (2) Multi-level fusion and aggregation: x5_4 is added to the dimensionality reduction feature x4_dem of the encoding layer 4 again, and then fed into PVMLayer→BatchNorm→ReLU to output the final feature map of the decoding layer 4 (32×32×64).

[0029] Decoding layer 3: (1) Adjacent layer fusion: Upsample the 32×32×64 feature map of decoding layer 4 to 64×64, add it element by element with the 64×64×64 dimensionality reduction feature of coding layer 3 and take the absolute value, input PVMlayer→ BatchNorm→ReLU to obtain the first-level fusion feature x4_3.

[0030] (2) Cross-layer fusion: Add x5_4 (after upsampling from 32 to 64) and x4_3 and take the absolute value, then input it into PVMlayer→BatchNorm→ReLU to obtain the cross-layer secondary fusion feature x5_4_3.

[0031] (3) Fusion and aggregation: The three dimensionality reduction features x3_dem, x4_3 and x5_4_3 of the coding layer 3 are summed and fed into PVMLayer→BatchNorm→ReLU, and the feature map of the decoding layer 3 (64×64×64) is output.

[0032] Decoding layer 2: (1) Adjacent layer fusion: Upsample the 64×64×64 feature map of decoding layer 3 to 128×128, add it element by element with the 128×128×64 dimensionality reduction feature of coding layer 2 and take the absolute value, input PVMlayer → BatchNorm → ReLU to obtain the first-level fusion feature x3_2.

[0033] (2) Second-level cross-layer fusion: Add x4_3 (after upsampling from 32 to 128) to x3_2 and take the absolute value, then input it into PVMlayer→BatchNorm→ReLU to obtain the second-level fusion feature x4_3_2; add x5_4_3 (after upsampling from 64 to 128) to x4_3_2 and take the absolute value, then input it into PVMlayer→BatchNorm→ReLU to obtain the third-level fusion feature x5_4_3_2.

[0034] (3) Multi-level fusion and aggregation: The dimensionality reduction features x2_dem of the coding layer 2 are accumulated with x3_2, x4_3_2 and x5_4_3_2, and fed into PVMLayer→BatchNorm→ReLU to output the feature map of the decoding layer 2 (128×128×64).

[0035] Decoding layer 1: (1) Adjacent layer fusion: The 128×128×64 feature map of decoding layer 2 is upsampled by bilinear interpolation (keeping 128×128) and added element by element to the 128×128×64 dimensionality reduction feature of coding layer 1 and the absolute value is taken. The result is sent to PVMlayer→BatchNorm→ReLU to obtain the first-level fusion feature x2_1.

[0036] (2) Cross-layer fusion: Add x3_2 (upsampled from 64 to 128) to x2_1 and take the absolute value, then input it into PVMlayer→BatchNorm→ReLU to obtain the second-level fusion feature x3_2_1; add x4_3_2 (upsampled from 32 to 128) to x3_2_1 and take the absolute value, then input it into PVMlayer→BatchNorm→ReLU to obtain the third-level fusion feature x4_3_2_1; add x5_4_3_2 (upsampled from 16 to 128) to x4_3_2_1 and take the absolute value, then input it into PVMlayer→BatchNorm→ReLU to obtain the fourth-level fusion feature x5_4_3_2_1.

[0037] (3) Multi-level fusion and aggregation: The dimensionality reduction features x1_dem of coding layer 1 are accumulated with x2_1, x3_2_1, x4_3_2_1, and x5_4_3_2_1, and fed into PVMLayer→BatchNorm→ReLU to output the final feature map of decoding layer 1 (128×128×64).

[0038] Finally, the 128×128×64 feature map of decoding layer 1 is mapped to a single-channel 128×128×1 through a 3×3 convolution, and then upsampled to 512×512×1 by bilinear interpolation to generate the final segmentation mask with the same resolution as the original input.

[0039] The core of the BP-SK-VM++ model of this invention lies in the introduction of bounding box cueing. The binary bounding box cueing mask and the pseudo-RGB image are concatenated to generate a four-channel tensor. SK-VM++ replaces the traditional convolutional skip channels with parallel Vision Mamba (PVM) layers to achieve efficient fusion and accurate guidance of multi-scale features.

[0040] The specific technical solution is as follows: 1. Feature map of the input four-channel data after processing by the encoder After layer normalization (LN) processing, it is uniformly divided into four parallel branches, as shown in the following formula: ; Features of each branch (i=1,2,3,4) enter the subsequent parallel Vision Mamba (PVM) layer; where, This represents the four-channel data feature map after the encoder output; This represents the layer normalization operation. This means that the normalized features are uniformly divided into four parallel branches; They represent the first Parallel branch input features, This represents the number of branch channels.

[0041] 2. The features of each parallel branch are enhanced using Mamba within the PVM layer for long-range dependency modeling, as shown in the following formula: ; Each PVM module processes its corresponding input features, capturing long-range dependency information through augmentation; among them, The Mamba state-space model represents long-range dependency modeling of input features; A learnable scaling factor used for gain-adjusting branch residuals; For the first Output characteristics of parallel branch paths enhanced by Mamba.

[0042] 3. Enhanced features of the four parallel branches They are concatenated along the channel dimension to form the fused output features. The formula is as follows: ; Figure 1The output mapping and normalization operations further process the features enhanced by PVM, ultimately outputting them to the decoder. This cascaded operation not only preserves feature information at multiple scales but also enhances the richness of feature fusion, providing a more comprehensive feature representation for subsequent output mappings; among which: This indicates that the outputs of the four parallel branches are concatenated along the channel dimension to obtain the fused features. ; 4. Characteristics of cascaded connections After layer normalization, a linear mapping is used to project the features back to the original number of channels, generating the final output features. The formula is as follows: ; in, Perform a layer normalization on the cascaded features; This represents a linear mapping operation that projects the normalized features back to the original number of channels to generate the final output features of this skip layer. , This represents the number of output channels.

[0043] 5. Cross-layer feature fusion within the decoder: At each layer of the decoder, BP-SK-VM++ employs a multi-node hierarchical fusion strategy, achieving progressive aggregation of cross-scale features through multiple PVM operations; at each layer of the decoder... In this section, BP-SK-VM++ employs a multi-node hierarchical fusion strategy, achieving progressive aggregation of cross-scale features through multiple PVM operations. The specific process is as follows: ; in, Indicates the first Layer The fusion features corresponding to each node; when When =1, directly use the previous layer. Level characteristics ;when When the value is greater than 1, first select the features of the previous node in the same layer. Upsampled features of the corresponding node in the next layer Add them together, then input them into the parallel Vision Mamba (PVM layer) for fusion; This indicates an upsampling operation, used to adjust the feature size of the next layer node to the current layer; This represents the Parallel Vision Mamba layer, which contains several parallel Mamba modules for efficiently modeling long sequence dependencies and performing global feature interactions inside and outside the channel.

[0044] Experimental procedure: Hardware: Ubuntu 20.04 LTS; Intel Core i9-10900K; 64 GB DDR4; NVIDIA RTX 3080Ti (12 GB); Software: Python 3.8.10; PyTorch 1.13.0; CUDA 12.8; cuDNN 8.x.

[0045] Data loading: During training, data is loaded in parallel using 4 threads; during validation / testing, it is not parallelized. Optimization strategies and hyperparameters: Optimizer: AdamW ( = 0.9, = 0.999, ε = 1×1 Weight Decay = 1×1 ²), initial learning rate 1×1 ³; Learning rate scheduling: Cosine Annealing =450, = 1×1 The training process consists of 450 rounds, with the learning rate updated after each round. Batch size: Training phase Batch = 4; Validation / testing phase Batch = 1; Mixed Precision: Automatic Mixed Precision (AMP) training was used throughout the process, along with gradient scaling. Data augmentation probability: 90° flip is performed with a probability of 0.5; affine rotation is performed with a probability of 100%.

[0046] Dataset: The thyroid ultrasound image dataset consists of 1,426 images collected by the Second People's Hospital of Changzhou Affiliated to Nanjing Medical University (ethics review approval number: 2020_KY146-01), using three commercial ultrasound scanners from Philips Healthcare (Netherlands), Siemens Healthineers (Germany), and GE Healthcare (USA).

[0047] A single-channel grayscale image (512×512×1) is copied into a pseudo-RGB tensor (512 × 512 × 3), and text bounding box annotations are parsed into a binary guided mask, which is then used as the fourth channel for concatenation to generate a network input with an input size of 512 × 512 × 4. During the training phase, to enhance the robustness of the model, random 90° rotations and random rotations within the range of 20°–80° are applied respectively. During the validation and testing phases, only global normalization is performed on the input image, without geometric transformations. Finally, the training set, validation set, and test set are randomly divided in a 6:2:2 ratio, and each subset is stored in binary array format to accelerate data loading.

[0048] Comparative Analysis of Experimental Results

[0049] Quantitative Analysis: To verify the superiority of BP-SK-VM++, typical methods such as UNet, MSNet, M²SNet, Swin-UNet, VMUNet, and SK-VM++ were selected as controls. Evaluations were conducted under the same test set conditions with a 512×512×4 four-channel input. Table 1 summarizes the model categories and five segmentation performance metrics for each method. BP-SK-VM++ achieved an mIoU of 86.84%, a 13.35 percentage point improvement over SK-VM++; its F1 score was 92.95%, an 8.23 ​​percentage point improvement over SK-VM++; and its pixel classification accuracy reached 99.13%, specificity 99.55%, and sensitivity 92.74%, an 11.1 percentage point improvement over SK-VM++, all superior to all baseline methods. Swin-UNet, due to excessive smoothing and undersegmentation, had an mIoU of only 37.13%, significantly lower than other methods.

[0050] Table 1. Comparison and segmentation results of thyroid ultrasound images

[0051] Qualitative analysis: In Figure 4In the typical complex samples shown, because the lesion boundaries are highly similar to the background texture, BP-SK-VM++ can accurately reconstruct coherent edges that are highly consistent with the ground truth, and produces almost no noise artifacts. In contrast: SK-VM++ can capture the main contours in most areas, but it is slightly over-smoothed in the depressions of the first row of samples, making it difficult to preserve fine structures; VMUNet can recover the overall shape well, but there are edge blurring and slight missed detections in the raised details of the third row of samples; UNet is prone to misjudging background noise as foreground and produces obvious artifacts at the edges of the second row of samples; Swin-UNet shows isolated island-like artifacts in the second and third rows of samples; MSNet has continuous edge missed detections, such as the left area of ​​the first row of samples not being completely covered; M²SNet maintains a relatively complete contour, but still has a small number of noise false detections in complex texture areas. The significant advantage of the BP-SK-VM++ of this invention is due to the introduction of Box extracted from the text annotation file. The Prompt (BP) mechanism explicitly incorporates coarse lesion localization information into a four-channel tensor. BoxPrompt provides strong spatial constraints for the network during feature extraction, enabling the PVM layer to accurately focus on the target region and suppress background interference during cross-scale fusion, resulting in the following effects: 1. Details are fully preserved even at concave and convex boundaries; 2. Effectively eliminates misclassification of textured areas and noise artifacts; 3. Significantly reduces missed detections and isolated artifacts, ensuring smooth, coherent segmentation boundaries without false detections.

[0052] In summary, this invention achieves high-precision segmentation of complex lesions through the Box Prompt strategy, which is significantly superior to many existing mainstream methods.

[0053] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. An image segmentation method based on box cueing enhancement using the SK-VM++ network, characterized in that, Includes the following steps: Step 1: Collect the images to be segmented from the dataset and perform three-channel copying on the images to be segmented; Extract lesion bounding box information from text annotation files to generate a binary mask; The binary bounding box cue mask is fused with the copied three-channel image to obtain a four-channel tensor; Step 2: Using a four-channel tensor as input, construct an SK-VM++ model and use the SK-VM++ model for segmentation.

2. The image segmentation method based on box cueing enhancement using the SK-VM++ network according to claim 1, characterized in that, Step one specifically includes: Step 11: Perform grayscale processing on the image to be segmented to obtain the original single-channel grayscale image. ;Will Copy as pseudo-RGB image ; Step 12: Read the lesion bounding box coordinates from the text annotation file of the image to be segmented. ;in The top left corner of the bounding box. The bottom right corner of the bounding box; Step 13: Define a binary mask using the two corners of the bounding box. ; Step 14, and Concatenate along the channel dimension to obtain a four-channel input tensor. .

3. The image segmentation method based on box cueing enhancement using the SK-VM++ network according to claim 2, characterized in that, The formula is: in, x and y This represents the coordinate index of the pixel in the image matrix.

4. The image segmentation method based on box cueing enhancement using the SK-VM++ network according to claim 2, characterized in that, splicing is performed using channel dimensions. concat function.

5. The image segmentation method based on box cueing enhancement using the SK-VM++ network according to claim 2, characterized in that, Data preprocessing and enhancement are performed on the four-channel tensor.

6. The image segmentation method based on box cueing enhancement using the SK-VM++ network according to claim 5, characterized in that, Data preprocessing and augmentation include: firstly, randomly rotating the four-channel tensor by an angle. to Take values ​​in intervals, then randomize. Rotate by an integer multiple of the original value, and finally flip it horizontally and vertically randomly.

7. The image segmentation method based on box cueing enhancement using the SK-VM++ network according to claim 1, characterized in that, The number of receive channels for the SK-VM++ model is set to 4.

8. An image segmentation system based on box cueing enhancement of the SK-VM++ network, characterized in that, include: Memory is used to store instructions that can be executed by the processor; A processor for executing instructions to implement the image segmentation method based on the box-cue-enhanced SK-VM++ network as described in any one of claims 1-7.

9. A computer-readable medium storing computer program code, characterized in that, The computer program code implements the image segmentation method based on the box-cue-enhanced SK-VM++ network as described in any one of claims 1-7 when executed by a processor.

Images