Tumor image segmentation method and system of multi-scale feature fusion network based on boundary enhancement

Abstract

The invention discloses a tumor image segmentation method and system of a multi-scale feature fusion network based on boundary enhancement, and relates to the technical field of image segmentation, and the method comprises the steps: obtaining a to-be-segmented tumor image; constructing a tumor image segmentation model based on a pyramid visual converter PVTv2 backbone network; training the tumor image segmentation model through the to-be-segmented tumor image and the known tumor image to obtain an optimal tumor image segmentation model; and acquiring a real-time to-be-segmented tumor image and inputting the real-time to-be-segmented tumor image into the optimal tumor image segmentation model to obtain a tumor image segmentation result. Aiming at the endoscope image segmentation of the kidney tumor, the kidney tumor in the endoscope image is efficiently, stably and automatically segmented through the boundary-enhanced multi-scale feature fusion network, clinical doctors are helped to provide accurate tumor area positioning in endoscopy and surgical operations, and compared with the most advanced method, the method has the advantages that the accuracy is high, and the efficiency is high. According to the method, better segmentation capability and stronger generalization capability are obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image segmentation, and more particularly to a tumor image segmentation method and system based on a boundary-enhanced multi-scale feature fusion network. BACKGROUND

[0002] Kidney cancer is a disease with very high morbidity and mortality, and accurate automatic segmentation of the kidney and kidney tumor plays an extremely important guiding role for clinicians in the diagnosis and surgery of the disease. Surgical resection of kidney tumors is an effective method for treating kidney cancer, which requires radiologists to find the location of the kidney tumor through computed tomography (CT) images or magnetic resonance imaging (MRI) images before surgery. Then, the surgeon can accurately locate the kidney tumor for resection during surgery. However, due to the differences between CT and MRI images and actual tumor images, there may be endoscopic areas of tumor lesions missed in CT and MRI detection, and the surgeon must choose the appropriate resection point according to the actual boundary between the tumor area and normal kidney tissue in the endoscope, so accurate delineation of the kidney tumor in the endoscope image is a key step in the diagnosis and treatment of kidney cancer.

[0003] Current research on deep learning-based kidney tumor segmentation methods still focuses on analyzing CT and MRI images. At the same time, in similar endoscopic segmentation tasks, there are few algorithms optimized specifically for endoscopic images of kidney tumors, and the existing algorithms have low segmentation accuracy. Current automatic segmentation algorithms still have some problems when it comes to kidney tumor segmentation: fuzzy kidney tumor boundaries, multi-scale problems of kidney and kidney tumor regarding position and size, and similarity of tumor and surrounding normal tissue details.

[0004] Therefore, how to propose a tumor image segmentation method and system based on a boundary-enhanced multi-scale feature fusion network, which is aimed at endoscopic image segmentation of kidney tumors, and performs efficient and stable automatic segmentation of kidney tumors in endoscopic images through a boundary-enhanced multi-scale feature fusion network is a problem that those skilled in the art need to solve. SUMMARY

[0005] Therefore, the present application provides a tumor image segmentation method and system based on a boundary-enhanced multi-scale feature fusion network, which is aimed at endoscopic image segmentation of kidney tumors, and performs efficient and stable automatic segmentation of kidney tumors in endoscopic images through a boundary-enhanced multi-scale feature fusion network. In order to achieve the above purpose, the present application adopts the following technical solutions:

[0006] A tumor image segmentation method based on a boundary-enhanced multi-scale feature fusion network, comprising:

[0007] obtaining a tumor image to be segmented;

[0008] construct a tumor image segmentation model based on a pyramid vision transformer PVTv2 backbone network;

[0009] train the tumor image segmentation model by a to-be-segmented tumor image and a known tumor image to obtain an optimal tumor image segmentation model;

[0010] obtain a real-time to-be-segmented tumor image and input it into the optimal tumor image segmentation model to obtain a tumor image segmentation result.

[0011] Optionally, the tumor image segmentation model based on the pyramid vision transformer PVTv2 backbone network comprises a pyramid vision transformer PVTv2 backbone network module, a multi-scale feature fusion attention module, a detail enhancement module, a hybrid cross-modal attention module, and a boundary selective attention module; one end of the pyramid vision transformer PVTv2 backbone network module outputs a global mapping graph constructed by the multi-scale feature fusion attention module, and the other end combines the global mapping graph and the hybrid cross-modal attention module through the detail enhancement module to perform multiple upsampling, and the results of the multiple upsampling are coupled with the boundary selective attention module respectively to obtain binary masks of different scales.

[0012] Optionally, the one end of the pyramid vision transformer PVTv2 backbone network module outputs a global mapping graph constructed by the multi-scale feature fusion attention module, which comprises: inputting an image through the pyramid vision transformer PVTv2 backbone network module, extracting multiple multi-scale pyramid features, then sending each adjacent two features in the multiple features to the multi-scale feature fusion attention module for multi-scale context aggregation, obtaining multiple outputs, and obtaining a global mapping graph by the multiple outputs through a partial decoder.

[0013] Optionally, the other end combines the global mapping graph and the hybrid cross-modal attention module through the detail enhancement module to perform multiple upsampling, which comprises: combining the global mapping graph with the detail enhancement features output by the boundary selective attention module, performing upsampling on the combined features through a decoder and inputting the combined features to the hybrid cross-modal attention module, combining the combined features with the multi-scale pyramid features enhanced by the boundary selective attention module and the cross-semantic attention module, and obtaining multiple pyramid output features.

[0014] Optionally, the multiple upsampling results are coupled with the boundary selective attention module respectively to obtain binary masks of different scales, which further comprises:

[0015] After obtaining the multiple pyramid output features obtained in each stage, a plurality of parallel convolution layers are adopted to cooperate with an upsampling layer to generate binary masks of different scales.

[0016] The small full connection network is constructed to dynamically generate weight coefficients: a plurality of pyramid output features are compressed in space dimension by global average pooling, and then the features are aligned in dimension by a plurality of convolution layers with m output channels, the aligned features are fused by channel addition, and four weight coefficients are output by sigmoid activation function after coding by two serial full connection layers;

[0017] Finally, the final tumor image segmentation result is obtained according to the four weight coefficients and the binary mask of different scales.

[0018] Optionally, the coupled boundary selective attention module comprises:

[0019] The coupled boundary selective attention module receives a feature map from the backbone network and forms a channel mean value map as a predicted mask, and performs probability map processing to generate three types of spatial attention mask maps: background, foreground and boundary position.

[0020] The input feature map is divided into three sub-feature maps along the channel dimension, corresponding to the boundary, foreground and background respectively, and the divided sub-feature maps are multiplied by the corresponding attention mask maps respectively.

[0021] The results after multiplication are processed by a standard convolution block with regularization respectively, and the attention features obtained by processing are connected along the channel dimension to construct a connection feature.

[0022] The channel compression and excitation module is used for the connection feature, the module first obtains a channel descriptor by adaptive average pooling, outputs a channel weight vector after processing by a full connection layer, and finally recalibrates the original feature by channel multiplication, enhances important channels and suppresses irrelevant channels, and finally performs residual connection between the enhanced result and the input feature map.

[0023] Optionally, the multi-scale feature fusion attention module comprises:

[0024] The two adjacent feature layers are used by 1x1 convolution, and then cross attention mechanism is used to concentrate attention on information in different channels and positions, while suppressing noise and redundant information in the feature map, and establishing dependence on remote information to obtain O1.

[0025] The connected features are input into convolution module branches with four expansion rate convolution kernels respectively, to capture global and local features by different range of receptive fields.

[0026] A jump connection is used to connect the output of each convolution module.

[0027] The feature outputs of the four branches are connected to obtain O2. At the same time, polarized self-attention is introduced to coordinately optimize the two dimensions by deploying the channel polarization branch and the spatial polarization branch in parallel, and the spatial weight of O1 is calibrated to obtain O psa , and added to the multi-scale feature O2 to form a residual connection.

[0028] Optionally, the hybrid cross-modal attention module includes:

[0029] The first branch input features are passed through the fully connected layer to obtain Query (Q1), Key (K1) and Value (V1), which are input to the SA-FF module and the output O is obtained through the multi-head attention module. sa , the value V and O sa Add together and pass through the feedforward network module to get O sa1 , and perform standardization to obtain the first branch result;

[0030] The second branch uses convolution operation to model the local details of the features to obtain the second branch result;

[0031] The results of the two branches are input into the cross-modal attention module, and the features of the two branches are dynamically fused. The results of the first branch and the second branch are processed respectively through three independent 1×1 convolutional layers to generate Query (Q2), Key (K2) and Value (V2). Among them, Q2 comes from the result of the first branch and extracts semantic information as the query vector through channel compression. K2 and V2 both come from the result of the second branch. K2 is compressed with the same number of channels as Q2 and is used to calculate the similarity. V2 maintains the same number of channels as the original feature to retain complete detail information;

[0032] Then, the cross-modal spatial attention map of Q2 and K2 is calculated through matrix multiplication, and then the weight matrix is ​​obtained through SoftMax normalization. The weight matrix is ​​weighted summed with V2, and the summed result is fused with the result of the first branch through a residual connection.

[0033] Optionally, the method further includes: calculating the average similarity coefficient mDice and average intersection-over-union (mIoU) of the tumor image segmentation model on the Re-TMRS dataset to perform model verification.

[0034] Optionally, a tumor image segmentation system based on a multi-scale feature fusion network with boundary enhancement includes:

[0035] Acquisition module: used to obtain the tumor image to be segmented;

[0036] Model building module: used to build a tumor image segmentation model based on the Pyramid Visual Transformer PVTv2 backbone network;

[0037] The training module is configured to train the tumor image segmentation model by using the tumor image to be segmented and a known tumor image to obtain an optimal tumor image segmentation model.

[0038] The segmentation module is configured to obtain a real-time tumor image to be segmented, input the tumor image to be segmented into the optimal tumor image segmentation model, and obtain a tumor image segmentation result.

[0039] Compared with the prior art, the tumor image segmentation method and system based on the boundary-enhanced multi-scale feature fusion network have the following beneficial effects:

[0040] The tumor image segmentation method based on the boundary-enhanced multi-scale feature fusion network comprises the following steps: obtaining a tumor image to be segmented; constructing a tumor image segmentation model based on a pyramid vision transformer (PVTv2) backbone network; training the tumor image segmentation model by using the tumor image to be segmented and a known tumor image to obtain an optimal tumor image segmentation model; and obtaining a real-time tumor image to be segmented, inputting the tumor image to be segmented into the optimal tumor image segmentation model, and obtaining a tumor image segmentation result. The tumor image segmentation method based on the boundary-enhanced multi-scale feature fusion network provides an efficient and stable method for automatic segmentation of kidney tumors in endoscopic images. For endoscopic image segmentation of kidney tumors, the boundary-enhanced multi-scale feature fusion network is used to efficiently and stably automatically segment kidney tumors in endoscopic images, which helps clinicians to accurately locate the tumor area during endoscopic examination and surgical operation. Experimental results show that, compared with the most advanced method, the proposed method has better segmentation capability and stronger generalization capability, and the mDice (average similarity coefficient) and mIoU (average intersection over union) on the Re-TMRS dataset reach 91.2% and 84.9%, respectively. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the provided drawings.

[0042] Figure 1 A structure framework diagram of the tumor image segmentation method based on the boundary-enhanced multi-scale feature fusion network is provided.

[0043] Figure 2 A structure framework diagram of the multi-scale feature fusion attention module is provided.

[0044] Figure 3 A structure framework diagram of the boundary-selective attention module is provided.

[0045] Figure 4 A mixed cross-modal attention module structure framework provided by the present application. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0047] The present application discloses a tumor image segmentation method based on a boundary enhancement multi-scale feature fusion network, comprising:

[0048] Obtaining a tumor image to be segmented;

[0049] Constructing a tumor image segmentation model based on a pyramid vision transformer PVTv2 backbone network;

[0050] Training the tumor image segmentation model by using the tumor image to be segmented and a known tumor image to obtain an optimal tumor image segmentation model;

[0051] Obtaining a real-time tumor image to be segmented and inputting it into the optimal tumor image segmentation model to obtain a tumor image segmentation result.

[0052] In the specific embodiment, the tumor image segmentation method based on boundary enhancement multi-scale feature fusion network is a medical image segmentation model based on a Pyramid Vision Transformer (PVTv2) backbone network, which has the characteristics of multi-scale feature fusion and attention mechanism. The model selects PVTv2 as the encoder, which pays more attention to local information than the traditional convolutional neural network (CNN) method, and PVTv2 shows stronger global information extraction capability and better input interference robustness. In order to enhance the representation ability of local detail information features, a series of innovative modules are designed: multi-scale feature fusion attention module (MFA), boundary selective attention module (BSA), and hybrid cross-modal attention module (HCA). BSA is specifically designed to handle boundary regions and enhance the sensitivity of the model to boundaries. BSA is applied at different levels, from low-level to high-level features, and all have boundary perception capability. MFA is a multi-branch feature pyramid module, and its core function is to enhance feature expression ability through multi-scale atrous convolution and attention mechanism. HCA is used to capture long-range dependencies and local appearance details.

[0053] Further, the tumor image segmentation model based on the Pyramid Vision Transformer PVTv2 backbone network includes a Pyramid Vision Transformer PVTv2 backbone network module, a multi-scale feature fusion attention module, a detail enhancement module, a hybrid cross-modal attention module, and a boundary selective attention module. One end of the Pyramid Vision Transformer PVTv2 backbone network module outputs a global mapping graph constructed by the multi-scale feature fusion attention module, and the other end combines the global mapping graph and the hybrid cross-modal attention module through the detail enhancement module for multiple upsampling. The results of multiple upsampling are coupled with the boundary selective attention module to obtain binary masks of different scales.

[0054] Further, the output of one end of the Pyramid Vision Transformer PVTv2 backbone network module through the multi-scale feature fusion attention module to construct a global mapping graph includes: inputting an image through the Pyramid Vision Transformer PVTv2 backbone network module to extract multiple multi-scale pyramid features, then sending each adjacent two features in the multiple features to the multi-scale feature fusion attention module for multi-scale context aggregation, obtaining multiple outputs, and then obtaining a global mapping graph by passing the multiple outputs through a partial decoder.

[0055] Furthermore, the other end performs multiple upsamplings through a detail enhancement module in combination with a global map and a hybrid cross-modal attention module, including: combining the global map with the detail enhancement features output by the boundary selective attention module, upsampling the combined features through a decoder and transmitting them to the hybrid cross-modal attention module, and then combining them with the multi-scale pyramid features for detail enhancement through the boundary selective attention module and the cross-semantic attention module to obtain multiple pyramid output features.

[0056] Specifically, the cross-semantic attention module combines channel-level and spatial-level attention mechanisms, achieving multi-dimensional dynamic calibration of features through a parallel dual-branch architecture. Specifically, the module receives input features and first divides them into two sub-features along the channel dimension. In the spatial attention branch, 3×3 convolution is performed to reduce the dimensionality to a single channel, followed by sigmoid activation to generate a spatial weight map. This weight map effectively focuses on spatially salient regions of the target object. Simultaneously, in the channel-level attention branch, a linear ECA block is used to calculate channel weights, enhancing feature discriminability by capturing cross-channel interactions. Both branches modulate the original features through element-wise multiplication. The spatial attention branch multiplies the channel weights to enhance channel-level features, while the channel attention branch multiplies the spatial weights to focus on spatial regions. Finally, the processed dual-branch features are concatenated along the channel dimension, outputting features that retain the original dimensionality but possess enhanced representational power.

[0057] Furthermore, the steps of coupling the results of multiple upsampling steps with the boundary selective attention module to obtain binary masks of different scales further include:

[0058] After obtaining multiple pyramid output features at each stage, multiple sets of parallel convolutional layers are used in conjunction with upsampling layers to generate binary masks of different scales.

[0059] Build a small fully connected network to dynamically generate weight coefficients: compress the spatial dimensions of multiple pyramid output features through global average pooling, align the feature dimensions through multiple groups of convolutional layers with m output channels, add and fuse the aligned features, encode them through two series-connected fully connected layers, and output four weight coefficients through the sigmoid activation function;

[0060] Finally, the final tumor image segmentation result is obtained based on four weight coefficients and binary masks of different scales.

[0061] Specifically, the four weight coefficients are α1 to α4. Finally, according to the four weight coefficients and the binary masks of different scales, the multi-scale predicted binary masks are fused by weighted summation. mask =∑Up i (P i )α i , where Up i (·) indicates the corresponding P i The upsampling operation is performed, and for i∈{1, 2, 3, 4}, the final tumor image segmentation result is obtained.

[0062] In a specific embodiment, Figure 1 As shown, the PVTv2 backbone network includes:

[0063] The input image passes through PVTv2 as the backbone network of the encoder to extract four multi-scale pyramid features X i , i∈{1, 2, 3, 4}. Each adjacent feature layer, X1 and X2, X2 and X3, and X3 and X4, of these four features are then fed into the MFA for multi-scale context aggregation. The resulting three outputs, M1, M2, and M3, are then passed through a partial decoder (PD) to produce a preliminary global map. The global map primarily captures the approximate location of the kidney tumor, resulting in a rough segmentation result that often lacks specific structural details.

[0064] The PD feature D5 is then combined with the detail-enhanced feature L4 output by BSA to achieve clearer boundary segmentation. The combined feature is then continuously upsampled by the decoder and fed into the HCA module. It is then combined with the horizontal connections enhanced by the BSA and cross-semantic attention modules (CSA). This process generates four pyramid output features S1, S2, S3, and S4, which are then passed through the BSA module to obtain prediction maps, achieving clearer boundary segmentation.

[0065] A trainable weight coefficient W is designed, and adaptive weighted addition is used to obtain the final predicted output mask. Specifically, the weight coefficient W is trained by using four sets of parallel 1×1 convolution layers in conjunction with upsampling layers after the decoder obtains the output features Di of each stage to generate binary masks P1, P2, P3 and P4 of different scales. At the same time, a small fully connected network is constructed to dynamically generate weight coefficients αi: D1 to D4 are first compressed into spatial dimensions through global average pooling, and then the feature dimensions are aligned through four sets of 1×1 convolution layers with m output channels (m=64 is set in the experiment). These aligned global statistics are fused by channel addition, encoded by two series-connected fully connected layers, and then the four-dimensional weight coefficients are output through the sigmoid activation function.

[0066] Furthermore, the coupled boundary selective attention module includes:

[0067] The coupling boundary selective attention module receives a feature map from the backbone network and forms a channel mean value map therefrom as a predicted mask, and performs probability map processing to generate three types of spatial attention mask maps: background, foreground, and boundary position;

[0068] The input feature map is divided into three sub-feature maps along the channel dimension, corresponding to the boundary, foreground, and background, respectively. The divided sub-feature maps are multiplied by the corresponding attention mask maps, respectively.

[0069] The results after multiplication are processed using a standard convolution block with regularization, respectively, and the processed attention features are connected along the channel dimension to construct a connected feature.

[0070] The connected feature is processed using a channel compression and excitation module. The module first obtains a channel descriptor through adaptive average pooling, processes the channel descriptor through a fully connected layer to output a channel weight vector, and finally re-labels the original feature through channel multiplication to enhance important channels and suppress irrelevant channels. The enhanced result is finally connected in residual connection with the input feature map.

[0071] In the specific embodiment, as shown in Figure 3 The boundary selective attention module BSA includes:

[0072] The BSA module uses boundary features and reverse features to supplement relatively rough kidney tumor region features with boundary and local information, and realizes and balances the attention mechanisms of the three regions: background, foreground (kidney tumor), and boundary position, respectively. The BSA module receives a feature map from the backbone network and forms a channel mean value map therefrom as a predicted mask (pred) input into the module. The input probability map generates three types of spatial attention mask maps: background, foreground, and boundary position.

[0073] Specifically, the input feature map x is divided into three sub-feature maps along the channel dimension (dim=1), m u corresponding to the boundary, m f corresponding to the foreground, and m b corresponding to the background. The features of these three parts focus on enhancing the representation ability of the tumor edge region. The divided sub-feature maps are multiplied by the corresponding attention mask maps, respectively, to enhance the accuracy of the edge. Then, a standard 3x3 convolution block with regularization is used for processing, respectively. Then, the three attention features are connected along the channel dimension to obtain a connected feature Fcat. Finally, a Squeeze-and-Excitation Layer is used to enhance important channels and suppress irrelevant channels, and a residual connection is used to ensure that the network does not lose the original features due to excessive processing.

[0074] Further, the multi-scale feature fusion attention module comprises:

[0075] Two adjacent feature layers are used 1x1 convolution, and then cross attention mechanism is used to concentrate attention on information in different channels and positions, while suppressing noise and redundant information in the feature map, establishing dependence on remote information, and obtaining O1;

[0076] The connected features are respectively input into the convolution module branches of four expansion rate convolution kernels, and global and local features are captured through different range of receptive fields;

[0077] The output of each convolution module is connected by using a jump connection;

[0078] The feature outputs of the four branches are connected to obtain O2, and polarized self-attention is introduced to deploy channel polarization branches and spatial polarization branches in parallel, respectively optimize two dimensions, and calibrate the spatial weight of O1 to obtain O psa , and the multi-scale feature O2 is added to form a residual connection.

[0079] In the specific embodiment, as shown in Figure 2 , the multi-scale feature fusion attention module specifically comprises:

[0080] The size of the kidney tumor varies significantly, so the multi-scale feature extraction of the kidney tumor is crucial for accurate segmentation. Therefore, a multi-scale feature extraction module with multiple parallel branches of convolution layers with different expansion rates and integrated cross attention mechanism (Criss-Cross Attention, CCA) and polarized self-attention (Polarized Self-Attention, PSA) is proposed to extract tumor features of different sizes, and more detailed local and global feature information is obtained.

[0081] Specifically, first, 1x1 convolution is used on two adjacent feature layers to reduce the amount of calculation. Then, through the CCA module, row-column attention is used to concentrate attention on information in different channels and positions, while suppressing noise and redundant information in the feature map, establishing dependence on remote information, and obtaining O1. The connected features O1 are respectively input into four convolution kernels with expansion rates set to (1, 2, 4, 8), and global and local features are better captured through different range of receptive fields. And the output of each convolution module is connected by using a jump connection to eliminate blurred residual shadows. Finally, the features of the four branches are connected together to obtain O2, and PSA is introduced to deploy channel polarization branches and spatial polarization branches in parallel, respectively optimize two dimensions, and calibrate the spatial weight of O1 to obtain O psa , and the multi-scale feature O2 is added to form a residual connection.

[0082] The multi-scale feature fusion attention module improves the perception ability of the model to multi-scale targets through the diversity of the dilated convolution and the attention mechanism, and is especially suitable for dense prediction tasks such as semantic segmentation, while maintaining the computational efficiency and strengthening the discriminability and robustness of the features.

[0083] Further, the hybrid cross-modal attention module comprises:

[0084] The first branch input feature obtains Query (Q1), Key (K1) and Value (V1) through a full connection layer, and inputs them into the SA-FF module to obtain output O through the multi-head attention module sa , and the value V is added to O sa , and O sa1 is obtained through the feedforward network module, and the first branch result is obtained after normalization processing.

[0085] The second branch utilizes convolution operation to model the local details of the feature to obtain the second branch result.

[0086] The results of the two branches are input into the cross-modal attention module to dynamically fuse the features of the two branches, and three independent 1x1 convolution layers are used to process the first branch result and the second branch result respectively to generate Query (Q2), Key (K2) and Value (V2), wherein Q2 is obtained from the first branch result and is extracted as a query vector through channel compression, K2 and V2 are both obtained from the second branch result, K2 is compressed to the same channel number as Q2 to calculate the similarity, and V2 maintains the same channel number as the original feature to retain complete detail information.

[0087] Then, the cross-modal spatial attention map of Q2 and K2 is calculated through matrix multiplication, and the weight matrix is obtained through SoftMax normalization, and the weighted sum of the weight matrix and V2 is calculated, and the result after the sum is fused with the first branch result through residual connection.

[0088] In the specific embodiment, as shown in Figure 4 , the hybrid cross-modal attention module specifically comprises:

[0089] The decoder needs to understand the mapping from low-resolution features to high-resolution features, however, due to the difference between high-level and low-level semantics, the direct combination of the two will cause semantic mismatch problems.

[0090] Therefore, a HCA module is developed, in which branch 1 input features are passed through a fully connected layer to obtain Query (Q1), Key (K1) and Value (V1). Then they are input into the SA-FF module, which is mainly composed of a multi-head attention module and a feed-forward network module. Through the multi-head attention module (MSA) to obtain the output O sa , which can be summarized as:

[0091]

[0092] where D denotes the dimension of the feature embedding. This process reduces the high computational cost caused by the attention operation and reduces the complexity of the network. Then the value V1 is added to O sa , and through the feed-forward network module (ConvFFN), O sa1 is obtained, where the formula is

[0093] O sa1 = Norm ((Osa+V1)+ConvFFN(Osa+V1));

[0094] x_t= Norm (O sa1 ).

[0095] In addition, branch 2 uses pure convolutional operations to model the local details of the features:

[0096] x_c= δ (O S +F3( δ (F2( δ (F1(O S )))))) ;

[0097] Wherein, delta (·) represents the ReLU activation function, F1() and F3() represent 1x1 convolutional layers, and F2 represents a 3x3 convolutional layer. Finally, the results of the two branches are input into a cross-modal attention module (CMA) to dynamically fuse the features of the two branches and improve the detail recovery of small targets. In the CMA module, first, three independent 1x1 convolutional layers are used to process branch 1 and branch 2 respectively to generate Query (Q2), Key (K2) and Value (V2), wherein Q2 is derived from x_t and is compressed in channels (reduced to 1 / 8 of the original number of channels) to extract semantic information as a query vector, K2 and V2 are both derived from x_c but have different functions, K2 is compressed to the same number of channels as Q2 to calculate the similarity, and V2 retains the same number of channels as the original feature to retain complete detail information; then the cross-modal spatial attention map ([B, HW, HW]) of Q2 and K2 is calculated through matrix multiplication, and then normalized through SoftMax, and the weight matrix is used to weight and sum V2, and the aggregated result is fused with x_t through a residual connection, which helps to enhance the perception of local details while maintaining the original semantic understanding during feature fusion. The entire process maintains the same spatial size and only changes the channels to effectively fuse cross-modal features through channel transformation and attention mechanism.

[0098] Further, the average similarity coefficient mDice and the average intersection-over-union mIoU of the tumor image segmentation model are calculated on the Re-TMRS dataset to verify the model.

[0099] In the specific embodiment, a tumor image segmentation system based on a boundary-enhanced multi-scale feature fusion network includes:

[0100] The acquisition module is configured to acquire a tumor image to be segmented.

[0101] The model construction module is configured to construct a tumor image segmentation model based on a pyramid vision transformer (PVTv2) backbone network.

[0102] The training module is configured to train the tumor image segmentation model using the tumor image to be segmented and a known tumor image to obtain an optimal tumor image segmentation model.

[0103] The segmentation module is configured to input a real-time tumor image to be segmented into the optimal tumor image segmentation model to obtain a tumor image segmentation result.

[0104] The various embodiments described in this specification are implemented in a progressive manner, each embodiment focusing on the differences from other embodiments, and the same or similar parts between embodiments can be mutually referred to. For the apparatus disclosed by the embodiments, since it corresponds to the method disclosed by the embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method part.

[0105] The above description of disclosed embodiments enables one of ordinary skill in the art to make or use the application. Various modifications to these embodiments will be readily apparent to those of ordinary skill in the art, and the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Therefore, the application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A tumor image segmentation method based on a boundary enhancement-based multi-scale feature fusion network, characterized in that, The method comprises the following steps: Obtain a tumor image to be segmented; Construct a tumor image segmentation model based on a pyramid vision transformer PVTv2 backbone network; Train the tumor image segmentation model by using the tumor image to be segmented and a known tumor image to obtain an optimal tumor image segmentation model; Input a real-time tumor image to be segmented into the optimal tumor image segmentation model to obtain a tumor image segmentation result.

2. The tumor image segmentation method based on the boundary enhancement and multi-scale feature fusion network according to claim 1, characterized in that, The tumor image segmentation model based on the pyramid vision transformer PVTv2 backbone network comprises a pyramid vision transformer PVTv2 backbone network module, a multi-scale feature fusion attention module, a detail enhancement module, a hybrid cross-modal attention module and a boundary selective attention module; one end of the pyramid vision transformer PVTv2 backbone network module outputs a global mapping graph constructed by the multi-scale feature fusion attention module, and the other end of the pyramid vision transformer PVTv2 backbone network module combines the global mapping graph and the hybrid cross-modal attention module by using the detail enhancement module to perform multiple times of up-sampling, and the results of the multiple times of up-sampling are coupled with the boundary selective attention module to obtain binary masks of different scales.

3. The tumor image segmentation method based on the boundary enhancement and multi-scale feature fusion network according to claim 2, characterized in that, The one end of the pyramid vision transformer PVTv2 backbone network module outputs a global mapping graph constructed by the multi-scale feature fusion attention module, which comprises the following steps: inputting an image into the pyramid vision transformer PVTv2 backbone network module to extract multiple multi-scale pyramid features, then sending each adjacent two features in the multiple features to the multi-scale feature fusion attention module for multi-scale context aggregation to obtain multiple outputs, and then obtaining the global mapping graph by using the multiple outputs through a partial decoder.

4. The tumor image segmentation method based on the boundary enhancement and multi-scale feature fusion network according to claim 2, characterized in that, The other end of the pyramid vision transformer PVTv2 backbone network module combines the global mapping graph and the hybrid cross-modal attention module by using the detail enhancement module to perform multiple times of up-sampling, which comprises the following steps: combining the global mapping graph with the detail enhancement features output by the boundary selective attention module, up-sampling the combined features through a decoder and inputting the up-sampled features into the hybrid cross-modal attention module, combining the up-sampled features with the multi-scale pyramid features enhanced by the boundary selective attention module and the cross-semantic attention module to obtain multiple pyramid output features.

5. The tumor image segmentation method based on the boundary enhancement and multi-scale feature fusion network according to claim 2, characterized in that, The results of the multiple times of up-sampling are coupled with the boundary selective attention module to obtain binary masks of different scales, which further comprises the following steps: After obtaining the multiple pyramid output features at each stage, a plurality of parallel convolution layers are used to generate binary masks of different scales in cooperation with up-sampling layers; A small full connection network is constructed to dynamically generate weight coefficients: the multiple pyramid output features are compressed in spatial dimensions by using a global average pooling, then the features are aligned in feature dimensions by using a plurality of convolution layers with m output channels, the aligned features are fused by adding channels, and four weight coefficients are output by using a sigmoid activation function after the fused features are encoded by two serial full connection layers; Finally, the final tumor image segmentation result is obtained according to the four weight coefficients and the binary masks of different scales.

6. The tumor image segmentation method based on the boundary enhancement and multi-scale feature fusion network according to claim 5, characterized in that, The coupling boundary selective attention module comprises the following steps: The coupling boundary selective attention module receives a feature map from the backbone network and forms a channel mean value graph of the feature map as a prediction mask, and generates three types of spatial attention mask graphs: background, foreground and boundary position by probability graph processing. The input feature map is divided into three sub-feature maps along the channel dimension, corresponding to the boundary, foreground and background respectively, and the divided sub-feature maps are multiplied by the corresponding attention mask maps respectively; The results after multiplication are processed using standard convolution blocks with regularization respectively, and the attention features obtained by processing are connected along the channel dimension to construct connection features; The connection features are processed using a channel compression and excitation module. The module first obtains channel descriptors through adaptive average pooling, processes the channel descriptors through a fully connected layer to output a channel weight vector, and finally recalibrates the original features through channel multiplication to enhance important channels and suppress irrelevant channels. Finally, the enhanced results are connected with the input feature map in residual connection.

7. The tumor image segmentation method based on the boundary enhancement and multi-scale feature fusion network according to claim 1, characterized in that, The multi-scale feature fusion attention module comprises: The two adjacent feature layers are used for 1*1 convolution, and then cross attention mechanism is used to concentrate attention on information in different channels and positions, while suppressing noise and redundant information in the feature map, and establishing dependence on remote information to obtain O1; The connected features are input into convolution module branches with four different dilation rates, which capture global and local features through different receptive fields; The outputs of each convolution module are connected using a skip connection; The four-branch feature outputs are connected to obtain O2, while introducing polarized self-attention by deploying a channel polarization branch and a spatial polarization branch in parallel to cooperatively optimize two dimensions, respectively, and calibrating the spatial weight of O1 to obtain O psa and adding the multi-scale feature O2 to form a residual connection.

8. The tumor image segmentation method based on the boundary enhancement and multi-scale feature fusion network according to claim 2, characterized in that, The mixed cross-modal attention module comprises: The first branch input feature is transmitted through a full connection layer to obtain Query (Q1), Key (K1) and Value (V1), which are input into an SA-FF module, and an output O is obtained through a multi-head attention module sa The value V is added to O sa , and O sa1 is obtained through a feedforward network module, and the first branch result is obtained after standardization processing. The second branch uses convolution operation to model the local details of the features to obtain a second branch result; The results of the two branches are input into the cross-modal attention module to dynamically fuse the features of the two branches. Three independent 1*1 convolution layers are used to process the first branch result and the second branch result respectively to generate Query (Q2), Key (K2) and Value (V2). Q2 is obtained from the first branch result and is extracted as a query vector through channel compression, and K2 and V2 are obtained from the second branch result. K2 is compressed to the same number of channels as Q2 to calculate the similarity, and V2 retains the same number of channels as the original feature to retain complete detail information; Then, the cross-modal spatial attention map of Q2 and K2 is calculated through matrix multiplication, and the weight matrix is obtained through SoftMax normalization. The weight matrix and V2 are weighted and summed, and the summed result is fused with the first branch result through residual connection. 9.The tumor image segmentation method based on the boundary enhanced multi-scale feature fusion network according to claim 1, characterized in that, Further comprising: Calculate the average similarity coefficient mDice and the average intersection over union mIoU of the tumor image segmentation model on the Re-TMRS dataset to verify the model.

10. A tumor image segmentation system based on a boundary enhancement multi-scale feature fusion network, characterized in that, Comprise: The acquisition module is used to obtain the tumor image to be segmented; The model construction module is used to construct a tumor image segmentation model based on the PVTv2 backbone network of the pyramid visual transformer; The training module is used to train the tumor image segmentation model through the tumor image to be segmented and the known tumor image to obtain an optimal tumor image segmentation model; The segmentation module is used to input the real-time tumor image to be segmented into the optimal tumor image segmentation model to obtain the tumor image segmentation result.

Images