Liver tumor image real-time segmentation method and system based on YOLO algorithm

Abstract

The invention discloses a liver tumor image real-time segmentation method and system based on a YOLO algorithm, and the method comprises the steps: obtaining CT images and MRI images, and calculating the contrast indexes and signal-to-noise ratio indexes of a plurality of CT images; preprocessing the CT image, and dynamically adjusting an image enhancement strategy; a residual attention module is added on the basis of the YOLOv8 network, a multi-scale mask branch is introduced, and an optimized YOLO-Med segmentation network is constructed; inputting the enhanced CT image into a segmentation network, and training the segmentation network in combination with a loss function; and when an MRI image is input, through a multi-modal feature fusion mechanism, features of the MRI image and the enhanced CT image are aligned and fused and then are input into the segmentation network, and the segmentation network outputs a pixel-level segmentation mask for real-time segmentation of the liver and the tumor. According to the invention, by fusing the density characteristic of CT and the soft tissue resolution capability of MRI, the small tumor (diameter lt; 5 mm).

Description

Technical Field

[0001] The present invention relates to medical image processing technology, and in particular to a real-time liver tumor image segmentation method and system based on the YOLO algorithm. Background Art

[0002] The diagnosis and treatment planning of liver tumors rely on the accurate segmentation of medical images. In recent years, deep learning-based image segmentation technology has been widely used in this field. Among existing technologies, the YOLO (You Only Look Once) algorithm has been introduced into medical image processing due to its efficient target detection capabilities. For example, YOLOv8 has been used for liver and tumor segmentation tasks. Literature reports that its Dice score for liver segmentation on the LiTS dataset reached 89.54% and for tumor segmentation reached 80.55%. In addition, UNet and its variants (such as ResUNet) have also been used for liver tumor segmentation in CT images. Literature shows that their Dice scores range from 75.84% to 91.44%. These methods extract features and generate segmentation masks through convolutional neural networks, providing technical support for automated segmentation.

[0003] However, existing technologies have certain limitations in practical applications. The YOLO-based method has advantages in real-time performance, but its segmentation accuracy is limited by its ability to recognize complex tumor boundaries and tiny tumors (less than 5mm in diameter), especially in CT images with low contrast or noise interference, where the missegmentation rate is high. Although UNet-type methods have high accuracy, they are computationally complex and usually take more than 0.2 seconds to process a single frame of image, making it difficult to meet real-time requirements such as surgical navigation. In addition, existing methods mostly rely on single-modality data (such as CT or MRI only) and do not fully utilize the complementary information of multi-modal imaging, resulting in unstable segmentation results in cases with diverse tumor characteristics. These problems limit the wide applicability of existing technologies in clinical scenarios. Summary of the Invention

[0004] Purpose of the Invention: The purpose of the present invention is to provide a real-time liver tumor image segmentation method and system based on the YOLO algorithm, which can improve the segmentation accuracy of small tumors (less than 5mm in diameter) while reducing the time required to process a single frame of image. In addition, the present invention can also improve the problem that existing methods rely mostly on single-modality data (such as CT or MRI alone) and fail to fully utilize the complementary information of multimodal imaging, resulting in unstable segmentation results in cases with diverse tumor characteristics.

[0005] Technical solution: The present invention provides a real-time liver tumor image segmentation method based on the YOLO algorithm, comprising:

[0006] Acquire CT images and MRI images containing liver tumor images, calculate the contrast index and signal-to-noise ratio index of multiple CT images; perform adaptive image enhancement preprocessing on the CT images, and dynamically adjust the image enhancement strategy based on the contrast index and signal-to-noise ratio index to obtain enhanced CT images;

[0007] A residual attention module was added to the YOLOv8 network to enhance feature extraction of tumor regions. A multi-scale mask branch was introduced into the detection head of the YOLOv8 network to construct an optimized YOLO-Med segmentation network. The enhanced CT images were input into the optimized YOLO-Med segmentation network, which was trained using a combination of Dice loss and cross-entropy loss functions.

[0008] When an MRI image is input, the features of the MRI image and the enhanced CT image are aligned and fused through a multimodal feature fusion mechanism; the fused feature map is input into a trained optimized YOLO-Med segmentation network, and a modality loss robustness strategy is introduced to ensure single-modality segmentation performance. The trained optimized YOLO-Med segmentation network outputs a pixel-level segmentation mask, which is used for real-time segmentation of the liver and tumor.

[0009] Furthermore, the contrast index of the multiple CT images is calculated as follows:

[0010] For each of the multiple CT images, calculate the average signal frequency μ of region I inside the lesion i and the average signal frequency μ in the region outside the lesion O o , the calculation formula is as follows:

[0011] μ i =E{|s i | 2}

[0012] μ o =E{|s o | 2}

[0013] Calculate the signal power variance σ within the lesion i And the signal power variance outside the lesion σ0 is calculated as follows:

[0014]

[0015] Among them, s i is the random variable of the region I inside the lesion; s o is the random variable for the region O outside the lesion;

[0016] The contrast index CNR is calculated based on the average signal frequency and signal power variance of the lesion area. The calculation formula is as follows:

[0017]

[0018] Among them, σ i represents the signal power variance inside the lesion; σ0 represents the signal power variance outside the lesion.

[0019] Furthermore, the signal-to-noise ratio index of the multiple CT images is calculated as follows:

[0020] For each of the multiple CT images, the signal power is calculated. The signal power is usually calculated by the mean square value RMS of the signal. For a discrete signal x[n], the signal power P signal Expressed as:

[0021]

[0022] Where N represents the number of samples of the signal;

[0023] Calculate the noise power, noise power P noise The calculation method is similar to the signal power. Usually, it is necessary to separate the noise component from the total signal first and then calculate its mean square value. The calculation formula is as follows:

[0024]

[0025] The signal-to-noise ratio index is usually expressed in decibels and is calculated as follows:

[0026]

[0027] Furthermore, the residual attention module includes a 1×1 dimensionality reduction convolution layer, a 3×3 feature extraction layer, and a 1×1 dimensionality increase convolution layer;

[0028] The dimension reduction convolution layer uses a 1×1 convolution kernel to transform the input channel C in Reduced to half of the output channel, that is, C mid =C in / 2, the formula is as follows:

[0029]

[0030] Among them, X' represents the image after being processed by the 1×1 convolution kernel and the RELU function; Conv is the convolution kernel; RELU represents the mathematical function; x represents the input image; H is the height; W is the width; C mid C in Half of the output channels; R is a real number;

[0031] The feature extraction layer captures the local spatial features of the tumor area through a 3×3 convolution kernel. The formula is as follows:

[0032]

[0033] Where X” represents the image processed by the 3×3 convolution kernel;

[0034] The dimension-raising convolution layer restores the channel to C through 1×1 convolution in , the formula is as follows:

[0035]

[0036] Where X'' is the output image after dimensionality increase by 1×1 convolution kernel;

[0037] The attention weight is generated by global average pooling GAP and Sigmoid function, where the global average pooling formula is as follows:

[0038]

[0039] Combined with the attention weight generated by the Sigmoid function, the Sigmoid function formula is as follows:

[0040]

[0041] Multiply σ(x) by the original feature map element-wise, focus on the tumor area, and obtain the residual output through feature weighting and residual connection. The formula is as follows:

[0042]

[0043] Y=x+X att ∈R H×W×C

[0044] Among them, X att is the feature weighting; Y is the final residual output.

[0045] Furthermore, the multi-scale mask branch includes a 16×16 small-scale feature map for capturing the global liver contour, a 32×32 medium-scale feature map for locating the approximate tumor area, and a 64×64 large-scale feature map for refining tumor boundary details.

[0046] Furthermore, the enhanced CT images are input into the optimized YOLO-Med segmentation network, and the optimized YOLO-Med segmentation network is trained by combining the Dice loss and cross entropy loss functions, including:

[0047] The enhanced CT image is input into the optimized YOLO-Med segmentation network, and the optimized YOLO-Med segmentation network is trained using a medical loss function that combines the Dice loss and the cross entropy loss function. The formula of the Dice loss function is as follows:

[0048]

[0049] Among them, A is the prediction result; B is the true label; |A∩B| is the intersection of the prediction and the true label; |A|+|B| is the sum of the number of pixels of the prediction and the true label;

[0050] The formula for the cross entropy loss function is as follows:

[0051]

[0052] Among them, y i is the true label; is the prediction probability; M is the total number of pixels;

[0053] The formula of the medical loss function is as follows:

[0054] L Dice =0.7×L_Dice+0.3×L CE .

[0055] Furthermore, when an MRI image is input, the features of the MRI image and the enhanced CT image are aligned and fused through a multimodal feature fusion mechanism, including:

[0056] Multi-layer convolutional layers are used to process enhanced CT images and MRI images respectively. The multi-layer convolutional layers compress the spatial resolution in a hierarchical progressive manner while expanding the feature depth. The multi-layer convolutional layers extract feature maps containing density difference features from the enhanced CT images, and the multi-layer convolutional layers extract feature maps containing soft tissue contrast features from the MRI images. The dual-path parallel extraction ensures the independence of modality-specific information.

[0057] The feature maps are mapped to a unified feature space through a feature alignment network. The feature alignment network compresses the CT and MRI feature maps into a global description vector based on a global average pooling operation. Subsequently, the channel distribution is adjusted through a convolutional layer, and the CT feature maps and MRI feature maps are mapped to a unified feature space to achieve feature alignment. After alignment, upsampling is used to restore the CT feature maps and MRI feature maps to their original resolution.

[0058] The CT feature map and MRI feature map restored to their original resolution are fused at the element level, and the fused feature maps are normalized to eliminate distribution differences.

[0059] Furthermore, in the process of element-wise addition and fusion of the CT feature map and the MRI feature map restored to their original resolution, a channel attention mechanism is introduced to dynamically adjust the modal contribution ratio.

[0060] Furthermore, the modality loss robustness strategy is implemented through a random number generator, which sets the discarding probability of the CT feature map and the MRI feature map. If the MRI feature map is discarded during training, an all-zero feature map is input instead. If the CT feature map is discarded, it is compensated by weighted enhancement of the MRI features.

[0061] Based on the same inventive concept, the present invention provides a real-time liver tumor image segmentation system based on the YOLO algorithm, comprising:

[0062] An adaptive image enhancement preprocessing module is used to acquire CT images and MRI images containing liver tumor images, calculate the contrast index and signal-to-noise ratio index of multiple CT images, perform adaptive image enhancement preprocessing on the CT images, and dynamically adjust the image enhancement strategy based on the contrast index and signal-to-noise ratio index to obtain an enhanced CT image;

[0063] The optimized YOLO-Med segmentation module adds a residual attention module to the YOLOv8 network to enhance feature extraction of tumor regions. A multi-scale mask branch is introduced into the detection head of the YOLOv8 network to construct an optimized YOLO-Med segmentation network. The enhanced CT images are fed into the optimized YOLO-Med segmentation network, which is trained using a combination of Dice loss and cross-entropy loss functions.

[0064] The multimodal feature fusion module is used to align and fuse the features of the MRI image and the enhanced CT image through a multimodal feature fusion mechanism when the MRI image is input. The fused feature map is input into a trained optimized YOLO-Med segmentation network, and a modality loss robustness strategy is introduced to ensure the single-modality segmentation performance. The trained optimized YOLO-Med segmentation network outputs a pixel-level segmentation mask, which is used for real-time segmentation of the liver and tumor.

[0065] Beneficial effects: Compared with the existing technology, the significant technical effects of the present invention are as follows: (1) By aligning and fusing the features of CT images and MRI images, the segmentation accuracy of small tumors (less than 5mm in diameter) is improved. The Dice score under multimodal input is increased from 87% of a single CT modality to 94%, and the small tumor detection rate is increased by 15%. This technology utilizes the density information of CT and the soft tissue resolution ability of MRI to solve the limitations of single modality data in complex tumor segmentation; (2) By adding a residual attention module and a multi-scale mask branch to the YOLOv8 network, this method improves the Dice score of liver segmentation to 92% and tumor segmentation to 90%, with a processing speed of 0.08 seconds per frame, which is an improvement over YOLOv8's 89.54% and 0.15 seconds. The network reduces the missed detection rate of small tumors (by about 20%) by enhancing the feature extraction and pixel-level segmentation capabilities of tumor regions, and meets the needs of real-time segmentation while maintaining computational efficiency. (3) Adaptive image enhancement preprocessing is used to adjust the enhancement strategy according to the contrast index and signal-to-noise ratio of CT images, which increases the segmentation success rate of low-quality images with a noise ratio exceeding 30% from 60% to 90%. This module provides stable input data for subsequent segmentation through local adaptive contrast enhancement (CLAHE) and wavelet transform denoising (processing time is less than 0.01 seconds / frame). (4) By introducing a modality-missing strategy in training, the Dice score under single-modality input (such as CT or MRI only) is maintained above 88%, which is an improvement over the 75% when there is no strategy. This strategy enhances the model's adaptability to modality-incomplete scenarios by randomly discarding certain modality data and combining it with data augmentation (rotation ±15°, translation ±10 pixels); (5) Through model pruning and TensorRT acceleration, this method reduces the number of parameters in the optimized YOLO-Med network to 70% of the original, reduces memory usage to 2GB, increases inference speed to 15 frames / second, and controls power consumption within 5W. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 This is a flowchart of a method for real-time segmentation of liver tumor images based on the YOLO algorithm disclosed in an embodiment of the present invention;

[0067] Figure 2 Flowchart for adaptive preprocessing of input images;

[0068] Figure 3 This is a segmentation effect diagram of the present invention;

[0069] Figure 4 This is a schematic diagram of the structure of a real-time liver tumor image segmentation system based on the YOLO algorithm disclosed in an embodiment of the present invention;

[0070] Figure 5Schematic diagram of the structure of the optimized YOLO-Med segmentation module;

[0071] Figure 6 Schematic diagram of the structure of the multimodal feature fusion module. DETAILED DESCRIPTION

[0072] The technical solution of the present invention is described in detail below in conjunction with specific implementation methods and the accompanying drawings.

[0073] Example 1

[0074] like Figure 1 As shown, the present invention provides a real-time liver tumor image segmentation method based on the YOLO algorithm, comprising the following steps:

[0075] S1. Acquire CT images and MRI images containing liver tumor images, calculate the contrast ratio (CNR) and signal-to-noise ratio (SNR) of multiple CT images; perform adaptive image enhancement preprocessing on the CT images, and dynamically adjust the image enhancement strategy based on the contrast ratio and signal-to-noise ratio of the CT images to obtain enhanced CT images.

[0076] like Figure 2 As shown, the specific implementation process of step S1 is as follows:

[0077] S1.1. Obtain CT images and MRI images containing liver tumor images, and calculate the contrast ratio (CNR) and signal-to-noise ratio (SNR) of multiple CT images. The details are as follows:

[0078] For each of the multiple CT images, calculate the average signal frequency μ of region I inside the lesion i and the average signal frequency μ in the region outside the lesion O o , the calculation formula is as follows:

[0079] μ i =E{|s i | 2}

[0080] μ o =E{|s o | 2}

[0081] Calculate the signal power variance σ within the lesion i And the signal power variance outside the lesion σ0 is calculated as follows:

[0082]

[0083] Among them, s i is the random variable of the region I inside the lesion; s ois a random variable in the region O outside the lesion.

[0084] The contrast index CNR is calculated based on the average signal frequency and signal power variance of the lesion area. The calculation formula is as follows:

[0085]

[0086] Among them, σ i represents the signal power variance inside the lesion; σ0 represents the signal power variance outside the lesion.

[0087] For each of the multiple CT images, the signal power is calculated. The signal power is usually calculated by the mean square value (RMS) of the signal. For a discrete signal x[n], the signal power P signal It can be expressed as:

[0088]

[0089] Where N represents the number of samples of the signal;

[0090] Calculate the noise power, noise power P noise The calculation method is similar to the signal power. Usually, it is necessary to separate the noise component from the total signal first, and then calculate its mean square value. The calculation formula is as follows:

[0091]

[0092] The signal-to-noise ratio index is usually expressed in decibels and is calculated as follows:

[0093]

[0094] S1.2, perform adaptive image enhancement preprocessing on the CT image, and dynamically adjust the image enhancement strategy based on the contrast index and signal-to-noise ratio index of the CT image to obtain an enhanced CT image. The details are as follows:

[0095] The contrast index of each of the multiple CT images is compared with a pre-set first threshold. When the contrast index of a CT image is lower than the pre-set first threshold (CNR < 0.3), the input image is subjected to local adaptive contrast enhancement (CLAHE) with a grid size of 8×8 and a contrast limit of 2.0. The enhanced image is designated as a Category I image. When the contrast index of the input image is greater than or equal to the first threshold (CNR ≥ 0.3), no processing is performed and the image is designated as a Category II image. For each of the Category I and Category II images, when the signal-to-noise ratio (SNR) index is lower than a pre-set second threshold (SNR < 20 dB), a wavelet-based denoising algorithm is applied. In this embodiment, a three-layer Daubechies wavelet-based decomposition is used for denoising. The denoised image is designated as a Category III image. When the SNR index is greater than or equal to the second threshold (SNR ≥ 20 dB), no processing is performed and the image is combined with the Category III enhanced CT image and fed into YOLO-Med for processing.

[0096] In step S1.2, the first threshold is set to 0.3. Based on statistical analysis of clinical CT images, the local adaptive contrast enhancement algorithm uses CLAHE, limiting the contrast threshold to 4, the grid size to 8×8, and the processing time to less than 0.01 seconds per frame. The second threshold is set to 20 dB, and the Daubechies wavelet basis is used for wavelet transform with 3 decomposition levels. The image is reconstructed after soft threshold denoising. CLAHE is significant in that it improves image visual quality while avoiding noise and artifacts caused by excessive contrast enhancement in local areas. It has wide applications in medical imaging, autonomous driving, image enhancement, and other fields.

[0097] The principle of CLAHE is as follows: (1) Divide the image into non-overlapping small blocks (called tiles); (2) In each small block, calculate the histogram; (3) Equalize the histogram of each small block to ensure that the pixel distribution within the small block is more uniform; (4) When performing histogram equalization, introduce a limit parameter that controls the degree of contrast enhancement. If the histogram of a small block, then the contrast enhancement will be limited to avoid over-enhancement. The mathematical representation of CLAHE is as follows: (1) Let I(x,y) represent the pixel value at coordinate (x,y) on the original image, I clahe (x, y) represents the pixel value at the coordinate (x, y) on the image after CLAHE processing; (2) For each small block I block (x,y), calculate the histogram H block (I block ); (3) Calculate the cumulative distribution function (CDF): CDF block (I block), and perform histogram equalization; (4) limit the contrast enhancement threshold to C, when performing histogram equalization, block (I block ) to limit the degree of enhancement.

[0098] The precise calculation of contrast index and signal-to-noise ratio improves the targetedness of preprocessing, increasing the segmentation success rate of low-quality images (such as images with a noise ratio exceeding 30%) from 60% to 90%; the fast processing of CLAHE and wavelet transform ensures real-time performance, with a delay of less than 0.01 second per frame, providing high-quality input for subsequent segmentation and significantly reducing the missegmentation rate.

[0099] S2. A residual attention module is added to the YOLOv8 network to enhance feature extraction of tumor areas, and a multi-scale mask branch is introduced into the detection head of the YOLOv8 network to construct an optimized YOLO-Med segmentation network. The enhanced CT image is input into the optimized YOLO-Med segmentation network, and the optimized YOLO-Med segmentation network is trained by combining the Dice loss and cross-entropy loss functions.

[0100] The specific implementation process of step S2 is as follows:

[0101] S2.1. Based on the YOLOv8 network, the residual attention module is added to enhance the feature extraction of the tumor region.

[0102] In this example, the residual attention module consists of three convolutional layers: a 1×1 dimensionality reduction convolution layer (reducing the number of channels by half), a 3×3 feature extraction layer, and a 1×1 dimensionality increase convolution layer. Attention weights are generated using a sigmoid function in the range [0, 1]. The residual attention module is embedded in the Backbone layer of YOLOv8.

[0103] First, the dimensionality reduction convolution layer passes through the 1×1 convolution kernel to convert the input channel C in Reduced to half of the output channel, that is, C mid =C in / 2, fill with 0, stride 1. The formula is as follows:

[0104]

[0105] Among them, X' represents the image after being processed by the 1×1 convolution kernel and the RELU function; Conv is the convolution kernel; RELU represents the mathematical function; x represents the input image; H is the height; W is the width; C mid C in Half of the output channels; R is a real number.

[0106] Next, the feature extraction layer captures the local spatial features of the tumor area through a 3×3 convolution kernel, with zero padding and a stride of 1. The formula is as follows:

[0107]

[0108] Where "X" represents the image processed by the 3×3 convolution kernel.

[0109] Secondly, the dimension-raising convolution layer restores the channel to C through 1×1 convolution in The formula is as follows:

[0110]

[0111] Among them, X'' is the output image after dimensionality increase by 1×1 convolution kernel.

[0112] Then, the attention weight is generated by global average pooling (GAP) and Sigmoid function, where the global average pooling formula is as follows:

[0113]

[0114] Combined with the attention weights (range [0,1]) generated by the Sigmoid function, the Sigmoid function formula is as follows:

[0115]

[0116] Finally, σ(x) is element-wise multiplied with the original feature map to focus on the tumor area. The residual output is obtained through feature weighting and residual connection. The formula is as follows:

[0117]

[0118] Y=x+X att ∈R H×W×C .

[0119] Among them, X att is the feature weighting; Y is the final residual output.

[0120] The Sigmoid function is a nonlinear function widely used in mathematics, machine learning, and neural networks. It is named because of its "S"-shaped curve whose output range is between (0, 1).

[0121] S2.2. A multi-scale mask branch is introduced into the detection head of the YOLOv8 network. The multi-scale mask branch includes a 16×16 small-scale feature map, a 32×32 medium-scale feature map, and a 64×64 large-scale feature map. The 16×16 small-scale feature map is used to capture the global liver contour, the 32×32 medium-scale feature map is used to locate the approximate tumor area, and the 64×64 large-scale feature map is used to refine the tumor boundary details. Finally, a pixel-level segmentation mask is output.

[0122] In step S2.2, the multi-scale mask branch sets three output layers, corresponding to small (16×16), medium (32×32), and large (64×64) scale feature maps, and the mask resolution is restored to the original image size by upsampling.

[0123] S2.3. Input the enhanced CT image into the optimized YOLO-Med segmentation network and train the optimized YOLO-Med segmentation network by combining the Dice loss and cross entropy loss functions. The details are as follows:

[0124] The enhanced CT image is input into the optimized YOLO-Med segmentation network, and the optimized YOLO-Med segmentation network is trained using a medical loss function that combines the Dice loss (weight 0.7) and the cross entropy loss (weight 0.3) functions. The formula of the Dice loss function is as follows:

[0125]

[0126] Among them, A is the prediction result; B is the true label; |A∩B| is the intersection of the prediction and the true label (the number of correctly predicted positive samples); |A|+|B| is the sum of the number of pixels of the prediction and the true label.

[0127] The formula for the cross entropy loss function is as follows:

[0128]

[0129] Among them, y i is the true label (0 or 1); is the prediction probability; M is the total number of pixels.

[0130] The formula of the medical loss function is as follows:

[0131] L Dice =0.7×L_Dice+0.3×L CE .

[0132] Dice loss is a widely used loss function in medical image segmentation tasks, particularly well-suited for addressing class imbalance (e.g., small object segmentation). Its core is based on the Dice coefficient, which optimizes the model by measuring the overlap between the predicted results and the true labels. The cross-entropy loss function is designed for pixel classification, with a weight ratio of 0.7:0.3. The training dataset is LiTS, containing 131 samples, with 200 training epochs, a learning rate of 0.001, and a batch size of 8.

[0133] The residual attention module enhances the detection capability of tiny tumors (less than 5mm in diameter), reducing the missed detection rate by about 20%; the multi-scale mask branch improves the segmentation accuracy of tumors with complex boundaries, increasing the Dice score from 87% to 90%; the medical loss function optimizes the model's adaptability to unbalanced data, increasing the segmentation accuracy of tumor areas by 5% and enhancing training stability.

[0134] S3. When an MRI image is input, the features of the MRI image and the enhanced CT image are aligned and fused through a multimodal feature fusion mechanism. The fused feature map is input into a trained optimized YOLO-Med segmentation network. At the same time, a modality loss robustness strategy is introduced to ensure the single-modality segmentation performance. The trained optimized YOLO-Med segmentation network outputs a pixel-level segmentation mask, which is used for real-time segmentation of the liver and tumor.

[0135] The specific implementation process of step S3 is as follows:

[0136] S3.1. Use convolutional layers to extract feature maps of enhanced CT images and MRI images respectively.

[0137] In this embodiment, 5 convolutional layers are used to process the enhanced CT images and MRI images respectively. Each layer is configured with a 3×3 convolution kernel, a stride of 2, a padding of 1, and a ReLU activation function. The 5 convolutional layers compress the spatial resolution in a hierarchical manner (such as gradually downsampling the input 512×512 image to 16×16), while expanding the feature depth to 256 channels, effectively capturing multi-scale information. Multi-layer convolutional layers extract feature maps containing density difference features (such as tumor calcification areas) from the enhanced CT images, and multi-layer convolutional layers extract feature maps containing soft tissue contrast features (such as tumor edge texture) from the MRI images. The dual-path parallel extraction ensures the independence of modality-specific information.

[0138] S3.2. Map the feature map to a unified feature space through a feature alignment network.

[0139] In this embodiment, the feature map is mapped to a unified feature space through a feature alignment network. The feature alignment network compresses the CT and MRI feature maps into a global description vector of 1×1×128 based on the global average pooling operation (GAP) to eliminate the difference in spatial dimension. Subsequently, the channel distribution is adjusted through a 1×1 convolutional layer, and the CT feature map and the MRI feature map are forcibly mapped to a unified feature space to solve the problem of feature distribution offset between modalities and achieve feature alignment. After alignment, upsampling is used to restore the CT feature map and the MRI feature map to the original resolution to ensure the consistency of spatial information.

[0140] S3.3. Perform feature fusion and input the fused features into the optimized YOLO-Med segmentation network for segmentation.

[0141] In this embodiment, the CT feature map and the MRI feature map restored to the original resolution are fused at the element level to enhance the complementary information (such as the calcification features of CT + the edge texture of MRI). The fused feature map is normalized by BatchNormalization to eliminate the distribution difference.

[0142] During the element-wise fusion of the CT and MRI feature maps restored to their original resolution, a channel attention mechanism (weights generated by a sigmoid function) is introduced to dynamically adjust the modality contribution ratio (e.g., enhancing MRI weights in noisy CT images). The fused feature maps are fed into a trained optimized YOLO-Med segmentation network, and a modality loss robustness strategy is introduced to ensure single-modality segmentation performance. The trained optimized YOLO-Med segmentation network outputs pixel-level segmentation masks.

[0143] A modality missing robustness strategy is introduced to ensure the segmentation performance under single modality input by randomly discarding a certain modality data.

[0144] The modality-missing robustness strategy is implemented using a random number generator. The probability of dropping both CT and MRI feature maps is set to 0.3. If an MRI feature map is dropped during training, an all-zero feature map is input instead. If a CT feature map is dropped, it is compensated by weighted augmentation of the MRI features. Training data augmentation includes random rotations (±15°), translations (±10 pixels), and flips. Training uses the Adam optimizer with a momentum parameter of 0.9 and a weight decay of 0.0005. A validation set accounts for 20% of the training set. Single-modality performance is tested on the 3Dircadb dataset, which contains 20 samples.

[0145] The modality-missing strategy enables the system to maintain a Dice score above 88% in a single modality (only CT or MRI), a significant improvement compared to 75% without a strategy; data augmentation and optimizer selection improve the model's generalization ability, adapting to imaging equipment in different hospitals, and enhancing robustness by approximately 30%, laying the foundation for the system's promotion and application.

[0146] In step S3.1, feature extraction uses a 5-layer convolutional network, each with a convolution kernel size of 3×3, a stride of 2, a padding of 1, and an activation function of ReLU. The feature map depth extracted from CT images is 256, and that of MRI images is 256. In step S3.2, the feature alignment network is based on global average pooling and convolution operations. It first compresses the CT and MRI feature maps into a global vector of 1×1×256, adjusts the channel consistency through 1×1 convolution, and then restores the spatial resolution through upsampling. In step S3.3, the fusion method is element-wise addition. The fused feature map is normalized by Batch Normalization and input into the Neck layer of YOLO-Med. The fusion process takes less than 0.02 seconds per frame.

[0147] Multimodal feature extraction fully utilizes the density information of CT and the soft tissue contrast of MRI, increasing the detection rate of small tumors by 15% and the overall segmentation Dice score to 93%. Feature alignment and rapid fusion ensure real-time performance, and the system maintains a processing speed of 10 frames per second under multimodal input, providing higher-precision support for comprehensive clinical diagnosis.

[0148] According to the technical solution of the present invention, in step S1, the input CT image resolution is 512×512 pixels, the grayscale range is 0-255, and the pixel values ​​are normalized to the interval [0,1] by image normalization before preprocessing. In step S2, the number of input channels of the optimized YOLO-Med segmentation network is adjusted to 3, the convolution kernel size is 3×3, the stride is 1, and the feature map output resolution is maintained at 1 / 8 of the input image, supporting multi-scale feature extraction. In step S3, the MRI image adopts a T2-weighted sequence, the resolution is aligned with the enhanced CT image, the size is adjusted by bilinear interpolation before fusion, and the depth of the fused feature map is 256. The segmentation mask is output as a binary image, the liver area is marked as 1, the tumor area is marked as 2, and the background is 0. The output result is saved in PNG format to support subsequent clinical analysis.

[0149] Through standardization and multi-scale feature extraction, the system's compatibility is enhanced and it is suitable for CT equipment from different sources; the binarization and multi-region labeling of the mask output improve the visualization of the segmentation results, enabling doctors to quickly locate the liver and tumors during surgical planning, reducing diagnosis time by approximately 50%, and improving segmentation accuracy to over 92%.

[0150] like Figure 3As shown, Figure 3 Description of different positions: (1) Axial view (upper left), in the axial view, the segmentation result of the liver (grey) is displayed, and the position of the tumor (white) is similar to the standard segmentation. The overall shape of the liver is consistent with the standard segmentation; (2) Sagittal view (upper right), the sagittal view shows the liver and tumor from the side. The shape of the liver is well aligned with the standard segmentation, but the segmentation accuracy of the tumor is lower; 3. Coronal view (lower right), in the coronal view, the segmentation of the liver is consistent with the standard segmentation in terms of overall shape and position. The tumor appears in the correct position, but its size and shape again appear slightly deviated, and some areas may be missed or incorrectly segmented; (4) 3D view (lower left) provides an overall view of the segmentation results. The 3D shape of the liver is consistent with the standard segmentation, but the 3D representation of the tumor appears less detailed.

[0151] This method significantly improves the segmentation accuracy of small tumors (diameter <5 mm) by fusing the density features of CT with the soft tissue resolution capabilities of MRI. Experiments show that the Dice score for multimodal input reaches 94%, a 7% improvement over single CT modality, and increases the small tumor detection rate by 15%. The processing speed reaches 15 frames per second, meeting real-time clinical requirements.

[0152] The following is a specific example to verify the effectiveness of the method for real-time segmentation of liver tumor images based on the YOLO algorithm described in the present invention.

[0153] Table 1: Ablation experiment table

[0154]

[0155]

[0156] Experimental conditions: Input images were CT (512 × 512 pixels), with some experimental groups also including MRI (T2-weighted sequences, aligned to 512 × 512). The training set consisted of 104 samples and the test set consisted of 27 samples.

[0157] Evaluation metrics: Dice score (measures segmentation accuracy) and processing speed (frames per second, a measure of real-time performance). Experimental objective: To verify the effectiveness of the three innovative features of this method (adaptive image enhancement preprocessing, optimized YOLO-Med network, and multimodal feature fusion), by gradually adding each component and analyzing its contribution to performance.

[0158] Analysis: When adaptive preprocessing was added alone, liver Dice increased by 0.7% and tumor Dice increased by 1.5%, indicating enhanced processing capabilities for low-quality images (noise ratio > 30%), with a slight decrease in speed (0.2 frames / second). When optimized YOLO-Med was used alone, liver Dice increased by 1.5% and tumor Dice increased by 6.4%, with the speed increased to 12.5 frames / second, thanks to the residual attention module and multi-scale mask branch. Multimodal fusion in the complete method further improved the Dice score (liver 1.8%, tumor 1.5%). Due to the complementary CT and MRI features, the detection rate of small tumors increased by 15%, and the speed dropped to 10 frames / second.

[0159] Table 2: Performance comparison of different algorithms

[0160]

[0161] Experimental conditions:

[0162] UNet and ResUNet are classic segmentation networks and are not optimized for real-time performance. YOLOv3+ 3D segmentation combines object detection and volume segmentation, offering faster speed but lower accuracy. YOLOv8 serves as the benchmark for this method, using single-modal input. The test conditions for this method are the same as those used in the ablation experiment, using multimodal data including CT and MRI.

[0163] analyze:

[0164] Accuracy: The liver Dice (93.8%) and tumor Dice (91.5%) of this method are higher than those of UNet (91.4%, 75.8%), YOLOv8 (89.5%, 80.6%), etc., which is attributed to multimodal fusion and optimization network.

[0165] Speed: The processing speed (10 frames / second) is better than UNet (5 frames / second) and ResUNet (4.5 frames / second), slightly lower than the optimized YOLO-Med single modality (12.5 frames / second), and still meets real-time requirements (>10 frames / second).

[0166] Modality: This method supports multimodal input and performs more stably in complex cases compared to single-modality methods.

[0167] Example 2

[0168] like Figure 4 As shown, the present invention provides a real-time liver tumor image segmentation system based on the YOLO algorithm, comprising:

[0169] An adaptive image enhancement preprocessing module is used to acquire CT images and MRI images containing liver tumor images, calculate the contrast index and signal-to-noise ratio index of multiple CT images, perform adaptive image enhancement preprocessing on the CT images, and dynamically adjust the image enhancement strategy based on the contrast index and signal-to-noise ratio index to obtain an enhanced CT image;

[0170] The optimized YOLO-Med segmentation module adds a residual attention module to the YOLOv8 network to enhance feature extraction of tumor regions. A multi-scale mask branch is introduced into the detection head of the YOLOv8 network to construct an optimized YOLO-Med segmentation network. The enhanced CT images are fed into the optimized YOLO-Med segmentation network, which is trained using a combination of Dice loss and cross-entropy loss functions.

[0171] The multimodal feature fusion module is used to align and fuse the features of the MRI image and the enhanced CT image through a multimodal feature fusion mechanism when the MRI image is input. The fused feature map is input into a trained optimized YOLO-Med segmentation network, and a modality loss robustness strategy is introduced to ensure the single-modality segmentation performance. The trained optimized YOLO-Med segmentation network outputs a pixel-level segmentation mask, which is used for real-time segmentation of the liver and tumor.

[0172] In an optional embodiment, a real-time segmentation method for liver tumor images based on the YOLO algorithm includes: a) acquiring a CT image and an MRI image containing a liver tumor image, and calculating the contrast index and signal-to-noise ratio index of multiple CT images; performing adaptive image enhancement preprocessing on the CT image, dynamically adjusting the image enhancement strategy, and obtaining an enhanced CT image; b) adding a residual attention module on the basis of the YOLOv8 network, and introducing a multi-scale mask branch in the detection head of the YOLOv8 network to construct an optimized YOLO-Med segmentation network; inputting the enhanced CT image into the optimized YOLO-Med segmentation network, and training the segmentation network by combining the Dice loss and the cross-entropy loss function; c) when inputting an MRI image, aligning and fusion of the features of the MRI image and the enhanced CT image through a multimodal feature fusion mechanism, and inputting the trained optimized YOLO-Med segmentation network, introducing a modality loss robustness strategy, and the segmentation network outputs a pixel-level segmentation mask, which is used for real-time segmentation of the liver and tumor.

[0173] The adaptive image enhancement preprocessing module is implemented based on FPGA hardware, supports parallel processing of 512×512 images, and consumes less than 5W of power. The optimized YOLO-Med segmentation module is deployed on the NVIDIA Jetson AGX Xavier platform, occupies approximately 2GB of memory, and has an inference speed of 12 frames per second. The multimodal feature fusion module supports dual input interfaces (DICOM format for CT and NIfTI format for MRI) with a data transmission rate of 1GB / s. The output module is equipped with an HDMI interface with a resolution of 1080p. The mask is superimposed on the original image and displayed at a refresh rate of 60Hz. A USB interface is also provided to export results to a PC.

[0174] Hardware optimization reduces system power consumption and size, making it easier to integrate into portable medical devices; high inference speed and dual-channel input improve the flexibility of clinical use, allowing doctors to view segmentation results in real time during surgery, improving efficiency by 60%; diversified output methods facilitate data sharing and subsequent analysis, meeting the needs of different medical scenarios.

[0175] like Figure 5 As shown, the optimized YOLO-Med segmentation module includes a residual attention unit, a multi-scale mask branch unit and a medical loss function unit; wherein, the residual attention unit is used to execute step S2.1 to enhance the feature extraction of the tumor area; the multi-scale mask branch unit is used to execute step S2.2 to output the pixel-level segmentation mask; the medical loss function unit is used to execute step S2.3 to optimize the segmentation accuracy by combining Dice loss and cross entropy loss.

[0176] The residual attention unit has 500K convolutional layer parameters and an inference delay of less than 0.005 seconds. The multi-scale mask branch unit supports FP16 precision calculations, reducing the amount of computation by about 50%. The output mask is upsampled through bilinear interpolation, and 3×3 mean filtering is used for edge smoothing. The medical loss function unit records the loss curve during training, and the loss value is less than 0.1 after convergence. After model pruning, the number of parameters is compressed to 70% of the original. TensorRT acceleration is supported during deployment, and the inference speed is increased to 15 frames per second.

[0177] Low latency and high-precision computing make the system suitable for real-time surgical navigation, and tumor edge smoothing reduces the misjudgment rate by approximately 10%; model pruning and acceleration technology reduce hardware requirements, enabling the system to run on mid- and low-end devices, reducing costs by 30%, increasing inference speed by 25%, and enhancing practicality.

[0178] like Figure 6As shown, the multimodal feature fusion module includes a feature extraction unit, a feature alignment unit and a feature fusion unit; wherein the feature extraction unit is used to execute step S3.1 to extract feature maps of CT images and MRI images through a convolutional layer; the feature alignment unit is used to execute step S3.2 to map the feature maps to a unified feature space; the feature fusion unit is used to execute step S3.3 to generate fusion features and input them into the optimized YOLO-Med segmentation module.

[0179] The convolutional network depth of the feature extraction unit is configurable (3-7 layers), with a default of 5 layers and a parameter size of 1.2M. It supports INT8 quantization to reduce memory usage to 500MB. The feature alignment unit uses an attention mechanism for weighted fusion, with weights calculated using a softmax function. During training, the parameters of the first three layers are frozen to accelerate convergence. The feature fusion unit supports channel-level splicing as an optional mode. The resolution of the fused feature map is optional (128×128 or 256×256), and a dropout layer (discarding rate 0.2) is passed before output to prevent overfitting.

[0180] Configurable depth and quantization technology enables the module to adapt to devices with different computing power, reducing memory usage by 50%, making it easier to use in small hospitals; the attention mechanism improves fusion efficiency, and the multimodal segmentation accuracy is increased to 94%. Dropout enhances model robustness and reduces the risk of overfitting by approximately 15%, providing reliable support for complex cases.

Claims

1. A real-time liver tumor image segmentation method based on the YOLO algorithm, characterized in that: include: Acquire CT images and MRI images containing liver tumor images, and calculate contrast indices and signal-to-noise ratio indices of multiple CT images; Adaptive image enhancement preprocessing is performed on the CT image, and the image enhancement strategy is dynamically adjusted based on the contrast index and signal-to-noise ratio index of the CT image to obtain the enhanced CT image; A residual attention module was added to the YOLOv8 network to enhance feature extraction of tumor regions. A multi-scale mask branch was introduced into the detection head of the YOLOv8 network to construct an optimized YOLO-Med segmentation network. The enhanced CT images were input into the optimized YOLO-Med segmentation network, which was trained using a combination of Dice loss and cross-entropy loss functions. When an MRI image is input, the features of the MRI image and the enhanced CT image are aligned and fused through a multimodal feature fusion mechanism; The fused feature maps are input into a trained optimized YOLO-Med segmentation network. At the same time, a modality loss robustness strategy is introduced to ensure the single-modality segmentation performance. The trained optimized YOLO-Med segmentation network outputs a pixel-level segmentation mask, which is used for real-time segmentation of the liver and tumor.

2. The method for real-time segmentation of liver tumor images based on the YOLO algorithm according to claim 1, characterized in that: The contrast index of the multiple CT images is calculated as follows: For each of the multiple CT images, calculate the average signal frequency μ of region I inside the lesion i and the average signal frequency μ in the region outside the lesion O o , the calculation formula is as follows: m i =E{|s i | 2 } m o =E{|s o | 2 } Calculate the signal power variance σ within the lesion i And the signal power variance outside the lesion σ0 is calculated as follows: Among them, s i is the random variable of the region I inside the lesion; s o is the random variable for the region O outside the lesion; The contrast index CNR is calculated based on the average signal frequency and signal power variance of the lesion area. The calculation formula is as follows: Among them, σ i represents the signal power variance inside the lesion; σ0 represents the signal power variance outside the lesion.

3. The method for real-time segmentation of liver tumor images based on the YOLO algorithm according to claim 1, characterized in that: The signal-to-noise ratio index of the multiple CT images is calculated as follows: For each of the multiple CT images, the signal power is calculated. The signal power is usually calculated by the mean square value RMS of the signal. For a discrete signal x[n], the signal power P signal Expressed as: Where N represents the number of samples of the signal; Calculate the noise power, noise power P noise The calculation method is similar to the signal power. Usually, it is necessary to separate the noise component from the total signal first and then calculate its mean square value. The calculation formula is as follows: The signal-to-noise ratio index is usually expressed in decibels and is calculated as follows:

4. The method for real-time segmentation of liver tumor images based on the YOLO algorithm according to claim 1, characterized in that: The residual attention module includes a 1×1 dimensionality reduction convolution layer, a 3×3 feature extraction layer, and a 1×1 dimensionality increase convolution layer; The dimension reduction convolution layer uses a 1×1 convolution kernel to transform the input channel C in Reduced to half of the output channel, that is, C mid =C in / 2, the formula is as follows: Among them, X' represents the image after being processed by the 1×1 convolution kernel and the RELU function; Conv is the convolution kernel; RELU represents the mathematical function; x represents the input image; H is the height; W is the width; C mid C in Half of the output channels; R is a real number; The feature extraction layer captures the local spatial features of the tumor area through a 3×3 convolution kernel. The formula is as follows: Where X” represents the image processed by the 3×3 convolution kernel; The dimension-raising convolution layer restores the channel to C through 1×1 convolution in , the formula is as follows: Where X'' is the output image after dimensionality increase by 1×1 convolution kernel; The attention weight is generated by global average pooling GAP and Sigmoid function, where the global average pooling formula is as follows: Combined with the attention weight generated by the Sigmoid function, the Sigmoid function formula is as follows: Multiply σ(x) by the original feature map element-wise, focus on the tumor area, and obtain the residual output through feature weighting and residual connection. The formula is as follows: Y=X+X att ∈R H×W×C Among them, X att is the feature weighting; Y is the final residual output.

5. The method for real-time segmentation of liver tumor images based on the YOLO algorithm according to claim 1, characterized in that: The multi-scale mask branch includes a 16×16 small-scale feature map for capturing the global liver contour, a 32×32 medium-scale feature map for locating the approximate tumor area, and a 64×64 large-scale feature map for refining the tumor boundary details.

6. The method for real-time segmentation of liver tumor images based on the YOLO algorithm according to claim 1, characterized in that: The enhanced CT image is input into the optimized YOLO-Med segmentation network, and the optimized YOLO-Med segmentation network is trained by combining the Dice loss and cross entropy loss functions, including: The enhanced CT image is input into the optimized YOLO-Med segmentation network, and the optimized YOLO-Med segmentation network is trained using a medical loss function that combines the Dice loss and the cross entropy loss function. The formula of the Dice loss function is as follows: Among them, A is the prediction result; B is the true label; |A∩B| is the intersection of the prediction and the true label; |A|+|B| is the sum of the number of pixels of the prediction and the true label; The formula for the cross entropy loss function is as follows: Among them, y i is the true label; is the prediction probability; M is the total number of pixels; The formula of the medical loss function is as follows: THE Dice =0.7×L_Dice+0.3×L CE 。 7. The method for real-time segmentation of liver tumor images based on the YOLO algorithm according to claim 1, characterized in that: When an MRI image is input, the features of the MRI image and the enhanced CT image are aligned and fused through a multimodal feature fusion mechanism, including: Multi-layer convolutional layers are used to process enhanced CT images and MRI images respectively. The multi-layer convolutional layers compress the spatial resolution in a hierarchical progressive manner while expanding the feature depth. The multi-layer convolutional layers extract feature maps containing density difference features from the enhanced CT images, and the multi-layer convolutional layers extract feature maps containing soft tissue contrast features from the MRI images. The dual-path parallel extraction ensures the independence of modality-specific information. The feature maps are mapped to a unified feature space through a feature alignment network. The feature alignment network compresses the CT and MRI feature maps into a global description vector based on a global average pooling operation. Subsequently, the channel distribution is adjusted through a convolutional layer, and the CT feature maps and MRI feature maps are mapped to a unified feature space to achieve feature alignment. After alignment, upsampling is used to restore the CT feature maps and MRI feature maps to their original resolution. The CT feature map and MRI feature map restored to their original resolution are fused at the element level, and the fused feature maps are normalized to eliminate distribution differences.

8. The method for real-time segmentation of liver tumor images based on the YOLO algorithm according to claim 7, characterized in that: In the process of element-wise addition and fusion of CT feature maps and MRI feature maps restored to their original resolution, a channel attention mechanism is introduced to dynamically adjust the modal contribution ratio.

9. The method for real-time segmentation of liver tumor images based on the YOLO algorithm according to claim 1, characterized in that: The modality loss robustness strategy is implemented through a random number generator, which sets the discarding probability of CT feature maps and MRI feature maps. If the MRI feature map is discarded during training, an all-zero feature map is input instead. If the CT feature map is discarded, it is compensated by weighted enhancement of MRI features.

10. A real-time liver tumor image segmentation system based on the YOLO algorithm, characterized in that: include: An adaptive image enhancement preprocessing module is used to acquire CT images and MRI images containing liver tumor images and calculate the contrast index and signal-to-noise ratio index of multiple CT images; Adaptive image enhancement preprocessing is performed on CT images, and the image enhancement strategy is dynamically adjusted based on the contrast index and signal-to-noise ratio index to obtain enhanced CT images; The optimized YOLO-Med segmentation module adds a residual attention module to the YOLOv8 network to enhance feature extraction of tumor regions. A multi-scale mask branch is introduced into the detection head of the YOLOv8 network to construct an optimized YOLO-Med segmentation network. The enhanced CT images are fed into the optimized YOLO-Med segmentation network, which is trained using a combination of Dice loss and cross-entropy loss functions. The multimodal feature fusion module is used to align and fuse the features of the MRI image and the enhanced CT image through the multimodal feature fusion mechanism when the MRI image is input; The fused feature maps are input into a trained optimized YOLO-Med segmentation network. At the same time, a modality loss robustness strategy is introduced to ensure the single-modality segmentation performance. The trained optimized YOLO-Med segmentation network outputs a pixel-level segmentation mask, which is used for real-time segmentation of the liver and tumor.

Images