Medical image segmentation method and system based on multimodal data

Abstract

The present invention relates to the field of medical image segmentation technology, and in particular to a medical image segmentation method and system based on multimodal data. The method comprises: obtaining a medical image to be segmented; inputting the medical image to be segmented into a trained medical image segmentation model to obtain a medical image segmentation result; wherein the trained medical image segmentation model comprises: an encoder and a decoder connected in sequence, wherein the encoder extracts multi-scale features from the medical image to be segmented and enhances the details of the extracted features using a self-attention mechanism; the decoder improves feature expression capabilities through spatial and channel attention mechanisms; and the decoder uses the results of a dual segmentation head to recognize and differentiate cells. The method improves the accuracy and efficiency of medical image segmentation and is suitable for processing large-scale medical image datasets.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image segmentation, and in particular to a medical image segmentation method and system based on multimodal data. Background Art

[0002] Medical image segmentation plays a vital role in modern medical diagnosis and treatment, particularly in tumor detection, organ segmentation, and lesion region identification. Traditional image segmentation methods, such as those based on region growing, threshold segmentation, and edge detection, while effective in some simple scenarios, often perform poorly when dealing with medical images with complex structures, low contrast, and noise.

[0003] With the rapid development of deep learning, convolutional neural network-based segmentation models, particularly U-Net and its variants, have become the mainstream method for medical image segmentation. U-Net successfully achieves an effective fusion of global and local features through its encoder-decoder structure and skip connections. However, when processing multimodal medical images, traditional U-Net models still face problems such as high model complexity, high computational resource consumption during training, and insufficient capture of detailed and global features. Furthermore, with the continuous growth of medical image datasets, improving segmentation efficiency and reducing computational overhead while ensuring segmentation accuracy has become a pressing challenge.

[0004] To address these issues, researchers have proposed a number of improvement strategies in recent years, such as introducing attention mechanisms, multi-scale feature extraction modules, and depthwise separable convolutions. These technologies have improved model performance to varying degrees, but still suffer from issues such as high model complexity and slow inference speed. Summary of the Invention

[0005] In order to address the shortcomings of the existing technology, the present invention provides a medical image segmentation method and system based on multimodal data; proposes a lightweight U-Net optimization model, and combines multimodal data processing, attention mechanism, dual segmentation head structure and mixed precision training technologies to improve the accuracy and efficiency of medical image segmentation, which is suitable for the processing requirements of large-scale medical image datasets.

[0006] On the one hand, a medical image segmentation method based on multimodal data is provided, comprising:

[0007] Acquire a medical image to be segmented;

[0008] Input the medical image to be segmented into the trained medical image segmentation model to obtain the medical image segmentation result;

[0009] Among them, the trained medical image segmentation model includes: an encoder and a decoder connected in sequence, the encoder performs multi-scale feature extraction on the medical image to be segmented, and uses a self-attention mechanism to enhance the details of the extracted features; the decoder improves the feature expression capability through a spatial attention mechanism and a channel attention mechanism; the decoder uses the dual segmentation head results to realize cell recognition and cell differentiation.

[0010] On the other hand, a medical image segmentation system based on multimodal data is provided, comprising:

[0011] An acquisition module is configured to: acquire a medical image to be segmented;

[0012] a segmentation module configured to: input the medical image to be segmented into the trained medical image segmentation model to obtain a medical image segmentation result;

[0013] Among them, the trained medical image segmentation model includes: an encoder and a decoder connected in sequence, the encoder performs multi-scale feature extraction on the medical image to be segmented, and uses a self-attention mechanism to enhance the details of the extracted features; the decoder improves the feature expression capability through a spatial attention mechanism and a channel attention mechanism; the decoder uses the dual segmentation head results to realize cell recognition and cell differentiation.

[0014] In another aspect, an electronic device is provided, comprising:

[0015] a memory for non-transitory storage of computer-readable instructions; and

[0016] a processor for executing said computer-readable instructions,

[0017] When the computer-readable instructions are executed by the processor, the method described in the first aspect is executed.

[0018] On the other hand, a storage medium is provided, which non-temporarily stores computer-readable instructions, wherein when the non-temporary computer-readable instructions are executed by a computer, the method described in the first aspect is executed.

[0019] On the other hand, a computer program product is provided, comprising a computer program, wherein the computer program is configured to implement the method described in the first aspect when running on one or more processors.

[0020] The above technical solution has the following advantages or beneficial effects:

[0021] This invention improves the accuracy and efficiency of medical image segmentation while reducing computational complexity and video memory usage, and enhancing the model's clinical interpretability. Based on the traditional U-Net architecture, it incorporates several optimizations to improve segmentation capabilities when processing multimodal data, making it suitable for processing large-scale medical image datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0023] Figure 1 This is a step diagram of the overall design scheme of an embodiment of the present invention;

[0024] Figure 2 This is the structure diagram of the cell probability segmentation head;

[0025] Figure 3 Segmentation head structure diagram for gradient flow tracking;

[0026] Figure 4 This is a structural diagram of the optimized U-Net model according to an embodiment of the present invention;

[0027] Figure 5 This is a model structure diagram of the MiT Block in the SegFormer encoder according to an embodiment of the present invention;

[0028] Figure 6 This is a model structure diagram of the pixel-level attention block PAB Block according to an embodiment of the present invention;

[0029] Figure 7 Schematic diagram of the model structure of the channel and spatial attention block scSE according to an embodiment of the present invention;

[0030] Figure 8 This is a diagram of the preliminary output results of the optimized U-Net according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0032] Cell images are often obtained through different technical means and therefore have multimodal characteristics. At present, the acquisition of cell images in the medical field mainly relies on a variety of advanced microscopic imaging technologies, including optical microscopy, electron microscopy, confocal microscopy, super-resolution microscopy, and atomic force microscopy. Although the multimodal information in microscope images mainly comes from the type of microscope used for imaging and technical differences, there is still a large amount of modal information coming from the differences in the observed cell and tissue types. Moreover, in actual work, it is impossible to completely separate different types of cells from a tissue slice, so the data faced by the cell segmentation task in actual application must be multimodal.

[0033] Example 1

[0034] This embodiment provides a medical image segmentation method based on multimodal data;

[0035] like Figure 1 As shown, the medical image segmentation method based on multimodal data includes:

[0036] S101: Acquire a medical image to be segmented;

[0037] S102: Input the medical image to be segmented into the trained medical image segmentation model to obtain the medical image segmentation result; wherein the trained medical image segmentation model includes: an encoder and a decoder connected in sequence, the encoder performs multi-scale feature extraction on the medical image to be segmented, and uses a self-attention mechanism to enhance the details of the extracted features; the decoder improves the feature expression capability through a spatial attention mechanism and a channel attention mechanism; the decoder uses the dual segmentation head results to realize cell recognition and cell differentiation.

[0038] Furthermore, the medical image to be segmented is obtained, and the medical image is a medical image containing cells.

[0039] Furthermore, S102: the medical image to be segmented is input into a trained medical image segmentation model to obtain a medical image segmentation result; the training process of the trained medical image segmentation model includes:

[0040] Constructing a training set, wherein the training set is medical images with known image segmentation results;

[0041] Input the training set into the medical image segmentation model to train the segmentation model. When the loss function value of the model no longer decreases or the number of iterations reaches a set number, stop training to obtain the trained medical image segmentation model.

[0042] The loss function of the model, L(x,y), is:

[0043] L(x,y)=L BCE (h CR (f θ (x),y cell )+λ·L MSE (h CD (f θ (x),y hrafirnt );

[0044]

[0045] Furthermore, constructing a training set includes:

[0046] Medical images in various formats are uniformly converted to RGB images, normalized, and subjected to a series of spatial and intensity transformations to improve data quality and enhance their diversity, including random scaling, cropping, flipping, rotation, Gaussian noise, contrast adjustment, smoothing, histogram equalization, and sharpening.

[0047] Furthermore, if Figure 4 As shown, the encoder includes: an Overlap Patch Embeddings module, a first MiT Block module, a second MiT Block module, a third MiT Block module, and a fourth MiT Block module connected in sequence; the input end of the Overlap Patch Embeddings module is the input end of the encoder; the output end of the fourth MiT Block module is the output end of the encoder;

[0048] The decoder includes: a PAB Blocks module, a first upsampling layer, a first scSE module, a first adder, a second upsampling layer, a second scSE module, a second adder, a third upsampling layer, a third scSE module, a third adder, and a dual splitting head connected in sequence; the input end of the PAB Blocks module is the input end of the decoder; the output end of the dual splitting head is the output end of the decoder;

[0049] Among them, the output end of the fourth MiT Block module is connected to the input end of the PAB Blocks module; the output end of the third MiTBlock module is connected to the input end of the first adder; the output end of the second MiT Block module is connected to the input end of the second adder; the output end of the first MiT Block module is connected to the input end of the third adder.

[0050] Furthermore, the encoder is used to extract cell features. It uses the traditional U-Net as the backbone structure, retains the original skip connection mechanism, introduces SegFormer as the encoder, and makes full use of the efficient self-supervised attention mechanism (Efficient Self-Attention) designed based on the lightweight Transformer structure in its MiT Blocks to achieve accurate extraction of cell features and enhance the original U-Net model's perception and understanding of space.

[0051] Furthermore, the decoder is used to enhance feature extraction. The scSE attention mechanism module is introduced, which is a mechanism that combines spatial and channel attention. The scSE attention mechanism module uses parallel sSE (Channel Squeeze and Spatial Excitation) and cSE (SpatialSqueeze and Channel Excitation) modules to enhance the feature expression ability of the model, especially for visual tasks such as segmentation and classification. The outputs of the two are fused by element-by-element addition to ensure that the model can fully capture the spatial and channel importance information of the input features to improve the performance of visual tasks.

[0052] Furthermore, the Overlap Patch Embeddings module includes: a block partitioning layer, a convolutional neural network, a feature splicing layer, and an output layer. Overlap Patch Embeddings is an image coding method used to divide an image into multiple overlapping image blocks and encode each image block into a vector representation. First, the input image is divided into multiple overlapping image blocks of fixed size, with each block having a certain overlap with adjacent blocks to ensure that key information is not lost. Then, each image block is extracted using a pre-trained convolutional neural network to generate a high-dimensional vector representation. Next, the feature vectors of all image blocks are spliced ​​together to form an encoding of the entire image. This encoding not only retains the local information of each image block, but also reflects the contextual relationship between blocks, comprehensively considering local details and global information. Ultimately, the output image encoding can be used for computer vision tasks such as image segmentation and target detection to provide more accurate representation.

[0053] Further, if Figure 5 As shown, the internal structures of the first MiT Block module, the second MiT Block module, the third MiTBlock module and the fourth MiT Block module are consistent. The first MiT Block module includes:

[0054] The first attention module, the second attention module, the third attention module...the Nth attention module and the Overlap Patch Merging module are connected in sequence.

[0055] The first attention module, the second attention module, the third attention module, ... the Nth attention module have the same internal structure. The first attention module includes:

[0056] The Efficient self-Attention module and Mix-FFN module are connected in sequence.

[0057] Furthermore, the Efficient self-Attention module includes:

[0058] Perform linear transformation on the input features to obtain the Query, Key, and Value matrices respectively. When calculating Attention, first calculate the similarity between the Query and Key through the dot product, then calculate the similarity between the Query and Key through the Softmax activation function, and generate the attention weight. Multiply the elements one by one, apply the weight to the Value, and finally obtain the final output through addition or splicing.

[0059] Furthermore, the Efficient Self-Attention module is used to capture global dependencies. Through the self-attention mechanism, the model is able to capture long-range dependencies in the input sequence, effectively connecting even long-distance information through attention weights. This enables the model to better understand and process complex contextual information. The design of the Efficient Self-Attention module makes the computational process more efficient than traditional global self-attention, making it suitable for processing large-scale data and long sequences. Through various optimization methods, this module reduces computation time and memory overhead, thereby improving the overall operational efficiency of the model.

[0060] Furthermore, the Mixed Feed-Forward Network (MFFN) module includes two different feedforward neural network structures: a global feedforward neural network (Global FFN) and a local feedforward neural network (Local FFN). The global feedforward neural network has a larger receptive field and can better capture global context information. The local feedforward neural network has a smaller receptive field and can better capture local details.

[0061] Furthermore, the Mix-FFN module is used for feature extraction. The Mix-FFN module provides a more flexible feature extraction method by introducing different feedforward neural network structures. Traditional FFN uses the same nonlinear transformation at each position, while Mix-FFN allows different nonlinear transformations to be used at different positions, thereby increasing the expressive power of the model. By using global FFN and local FFN at the same time, the Mix-FFN module is more flexible and accurate in processing features at different positions. Global FFN can help the model capture longer-range dependencies and semantic information, while local FFN can better handle local details and subtle changes.

[0062] Furthermore, the Overlap Patch Merging module includes:

[0063] First, the original image is divided into overlapping blocks, and there is an overlapping area between adjacent blocks; the size of the block is a set size;

[0064] Then, feature extraction is performed on each image block through a convolutional neural network;

[0065] Next, the image blocks in the overlapping area are merged, and the merging methods include taking the average value or the weighted average value;

[0066] Finally, the processed image blocks are merged into a complete image by copying the pixel values ​​of the blocks to the corresponding positions.

[0067] Furthermore, the Overlap Patch Merging module is used to merge multiple overlapping image blocks into a complete, non-overlapping image, thereby restoring the details and integrity of the original image. This technology is commonly used in applications such as image stitching and image reconstruction, and can improve the accuracy and effectiveness of image processing. In image processing, dividing an image into small blocks for processing can bring some benefits, such as reducing computational complexity and improving processing efficiency. However, when there is overlap between image blocks, they need to be merged to restore the image's integrity.

[0068] Further, if Figure 6 As shown, the PAB Blocks module includes:

[0069] First, the query, key, and value feature maps are generated through convolution operations.

[0070] Then, the similarity between the query and the key is calculated to generate the attention weight matrix;

[0071] Subsequently, the attention weight matrix is ​​weighted and summed with the value feature map to obtain the enhanced feature map;

[0072] Finally, the enhanced feature map is fused with the original feature map to output the final feature map.

[0073] Furthermore, if Figure 6 As shown, the PAB Blocks module includes: an input end of the PAB Blocks module; an input end of the PABBlocks module, which is respectively connected to the input end of the first convolutional layer, the input end of the second convolutional layer, and the input end of the third convolutional layer; the output end of the first convolutional layer and the output end of the second convolutional layer are both connected to the input end of the first multiplier; the output end of the first multiplier is connected to the input end of the activation function layer, and the output end of the activation function layer and the output end of the third convolutional layer are both connected to the input end of the second multiplier; the output end of the second multiplier is connected to the input end of the fourth adder; the output end of the fourth adder is connected to the input end of the fourth convolutional layer; the output end of the fourth convolutional layer is the output end of the PABBlocks module.

[0074] Furthermore, the PAB Blocks module is used to enhance spatial attention. Using a self-attention mechanism, it calculates the relative importance of different locations in the image, thereby increasing the model's focus on important areas. PAB can effectively enhance the features of important locations in the image, improving the accuracy and robustness of cell segmentation.

[0075] Furthermore, if Figure 7 As shown, the internal structures of the first scSE module, the second scSE module and the third scSE module are consistent. The first scSE module includes: a parallel spatial attention module and a channel attention module;

[0076] The spatial attention module uses a channel compression and spatial excitation mechanism. It adjusts the weights of the spatial dimensions of the feature map to focus on important spatial regions. Its working process is as follows:

[0077] (11): Input tensor slicing: The input feature map U is split into U = [u 1,1 ,u 1,2 ,…,u i,j ,…,u H,W ], where u i,j ∈R 1×1×C ,u i,j Represents all channel features at the spatial position (i, j);

[0078] (12): Channel compression: compress the channel dimension of the input tensor U through convolution operation q = W sq ×U; where W sq ∈R 1×1×C×1, generate the projection tensor q∈R H×W , each element q of the tensor q i,j Represents the linear combination of all channels at spatial position (i, j);

[0079] (13): Spatial excitation: The projection tensor q is input into the Sigmoid activation function σ(.) to obtain the scalar value σ(q i,j ), which is used to recalibrate the spatial excitation of the input feature U;

[0080]

[0081] (14): Feature recalibration: Generate a new feature map U sSE , the formula is as follows:

[0082] U sSE =F sSE (U)=[σ(q 1,1 )u 1,1 ,…,σ(q i,j )u i,j ,…,σ(q H,W )u H,W ]

[0083] Among them, σ(q i,j ) represents the relative importance of a specific spatial position (i, j).

[0084] The spatial attention module recalibrates the input feature map to give higher weights to important spatial locations and ignore irrelevant locations.

[0085] Furthermore, the channel attention module includes: spatial compression and channel excitation mechanism. The spatial compression and channel excitation mechanism is to strengthen important channel features and suppress unimportant channels by adjusting the weight of each channel on the feature map. Its working process is as follows:

[0086] (21): Input feature map: Input feature map U∈R H×W×C Contains C channels, each channel u i ∈R H×W Represents the spatial characteristics of an image.

[0087] (22): Spatial Squeeze: Through the global average pooling operation, the spatial information of each channel is compressed into a scalar.

[0088] For the kth channel, the pooling operation is as follows:

[0089]

[0090] The vector z∈R 1×1×C Contains global information for each channel.

[0091] (23): Channel-dependent encoding: The pooled vector z is transformed through two fully connected layers. First, a linear transformation is performed:

[0092]

[0093] Where W1∈R C×C / 2 ,W2∈R C / 2×C is the weight of the fully connected layer, and δ(·) is the ReLU activation function. This step is used to encode the dependencies between channels.

[0094] (24): Activation function and normalization: Then, the Sigmoid activation function is used to The values ​​are mapped to the interval [0,1]:

[0095]

[0096] The vector obtained in this way Represents the importance score of each channel.

[0097] (25): Channel Excitation: Finally, the input feature map U is recalibrated using the obtained excitation value, and the activation value of each channel is Will be the corresponding channel feature map u k Multiply them to get the recalibrated output feature map:

[0098]

[0099] in, It represents the relative importance of the k-th channel. The recalibration result will strengthen important channels and suppress unimportant channels.

[0100] The first scSE module is used to adaptively recalibrate both spatially and channel-wise, thereby improving the performance of convolutional neural networks in image processing tasks. This module introduces an importance weighting mechanism across different spatial regions and channel dimensions, allowing the network to more accurately focus on meaningful features, thereby improving the model's discriminative ability and performance.

[0101] Specifically, the scSE module combines spatial excitation and channel excitation mechanisms, first compressing the spatial and channel information in the input feature map through global pooling and convolution operations, and then generating importance weights in the spatial and channel dimensions using the sigmoid activation function. These weights are used to recalibrate the input feature map, both strengthening the focus on useful channels in the channel dimension and enhancing the saliency of key regions in the spatial dimension, while suppressing irrelevant features. In this way, the scSE module can effectively enhance the model's ability to learn fine-grained features, especially in complex scenarios, with significant performance improvements.

[0102] Furthermore, if Figure 2 , Figure 3 As shown, the dual segmentation heads include: a cell probability segmentation head and a gradient flow tracking segmentation head; the formula for calculating the loss function using the results of the two segmentation heads is as follows:

[0103] L(x,y)=L BCE (h CR (f θ (x),y cell )+λ·L MSE (h CD (f θ (x),y gradient )

[0104] The cell probability segmentation head includes: a first convolution layer, a first activation function layer, a first batch of normalization layers, a second convolution layer and a first upsampling layer connected in sequence;

[0105] The first convolutional layer receives the final feature map from the decoder, extracts the feature information of the cell through a convolution operation, and generates a binary mask of the cell.

[0106] The first activation function layer uses the Mish activation function to introduce nonlinearity into the model and enhance the learning ability of cell features.

[0107] The first batch normalization layer performs batch normalization on the convolution output to stabilize the training process and accelerate convergence.

[0108] The second convolutional layer converts the feature map into a final binary mask through convolution, which represents the boundary of the cell.

[0109] The first upsampling layer uses bilinear interpolation for upsampling.

[0110] The cell probability segmentation head is used to generate a binary mask of the cell, identify the cell boundaries in the image, and ensure accurate segmentation of the cell position.

[0111] Furthermore, the gradient flow tracking segmentation head includes: a third convolution layer, a second activation function layer, a second batch normalization layer, a fourth convolution layer and a second upsampling layer connected in sequence;

[0112] The third convolutional layer receives the final feature map from the decoder and extracts the gradient flow information of the cell boundary through a convolution operation. The function of the convolutional layer is to extract detailed information about the cell boundary from the feature map.

[0113] The second activation function layer uses the Mish activation function to help the network learn nonlinear features and enhance the representation ability of cell boundaries.

[0114] The second batch normalization layer normalizes the convolution output through batch normalization, making the training more stable and accelerating convergence.

[0115] The fourth convolutional layer further processes the convolution output and converts it into a cell gradient map, outputting the gradient flow information representing the cell boundary, usually two channels, representing the gradient in the horizontal and vertical directions respectively.

[0116] In the second upsampling layer, if the spatial resolution of the original image needs to be restored, bilinear interpolation is used for upsampling to restore the feature map to a higher resolution.

[0117] The gradient flow tracking segmentation head is used to accurately distinguish the boundaries of adjacent cell regions through a gradient flow tracking method, ensure the accuracy of adjacent cell segmentation, and optimize the processing of inter-cell boundaries.

[0118] The present invention aims to improve the accuracy and efficiency of medical image segmentation. The method first preprocesses medical images in various formats, uniformly converts the images into RGB images, and improves data quality and diversity through spatial transformation and intensity transformation. Then, a lightweight segmentation model (Seg-LightUNet) is established with U-Net as the backbone network. Specifically, we use SegFormer as the encoder, retaining the jump connection mechanism of traditional U-Net. SegFormer enhances the capture of image details and global features through multi-scale feature extraction and self-attention mechanism. In the decoder, the ConcurrentSpatial and Channel Squeeze and Channel Excitation (scSE) attention mechanism is introduced to improve feature expression ability through spatial attention and channel attention modules. The model adopts a dual segmentation head structure, combined with the pseudo diffusion field method and gradient flow tracking method to further optimize the segmentation results. In addition, the model uses depthwise separable convolution and mixed precision training to reduce computational complexity and video memory occupancy, and is suitable for efficient processing of large-scale medical image datasets.

[0119] The image is segmented through a lightweight U-Net model based on the SegFormer encoder. SegFormer is used as the encoder of the U-Net network framework, retaining its skip connection mechanism, and the SCSE attention mechanism is introduced in the decoder. The preprocessed image is input for target area segmentation.

[0120] Use pre-trained models for transfer learning, adopt the pre-trained weights provided by the MEDIAR project team, and further enrich the semantic information of the model and enhance the generalization ability of the model through transfer learning.

[0121] Model performance is optimized through mixed precision training, and a mixed precision training strategy is adopted. Specifically, the computing power of 16-bit floating point numbers (FP16) and 32-bit floating point numbers (FP32) is comprehensively utilized. By reducing some non-critical operations in the model to FP16 precision processing, the use of video memory is significantly optimized. This strategy allows larger batch training under the same hardware conditions, thereby significantly accelerating the training process. At the same time, the calculation operations that have a key impact on the segmentation results are still maintained at FP32 precision to ensure the prediction accuracy and stability of the model details. This flexible application of mixed precision enables the model to reduce video memory usage and accelerate calculations while keeping the hardware resources unchanged, so that the model can achieve a good balance between training accuracy and efficiency. This optimizes the segmentation results and improves processing efficiency.

[0122] The segmentation results are further optimized through a dual segmentation head structure. The dual segmentation head structure of cell probability and gradient flow tracking method and the binary cross entropy loss function are used to perform detailed segmentation of the target area and identify adjacent areas.

[0123] Specifically, the cell probability recognition head provides a probability map of each pixel belonging to a cell region, ensuring accurate positioning of cell boundaries and contours. This method uses the binary mask generated by the cell recognition head to perform cell segmentation, accurately distinguishing cells from background areas while creating clear markers at cell boundaries to avoid over- or under-segmentation.

[0124] At the same time, the differentiation head uses gradient flow tracking to further improve segmentation accuracy, especially when cell boundaries are blurred or adjacent cells have similar morphologies. By calculating the gradient flow map of the image, the differentiation head can track the details of cell edges and optimize the segmentation of adjacent cell regions, especially to address issues where cell boundaries touch or overlap. The gradient flow information helps accurately identify the topological structure between cells and ensures that the boundaries of adjacent cells are accurately distinguished during the segmentation process.

[0125] In terms of loss function, combined with the binary cross entropy loss (L BCE ) and mean square error loss (L MSEThe cell recognition head uses a binary cross-entropy loss to optimize the segmentation of cell boundaries, while the discrimination head uses a mean squared error loss to optimize the accuracy of the gradient flow map. By introducing a weight parameter λ, the influence of these two loss functions is balanced during training, ensuring that the tasks of cell recognition and adjacent region segmentation are properly optimized.

[0126] The trained model is deployed as a system for clinical use, generating cell segmentation maps and CAM (Class Activation Mapping) attention visualization maps to enhance the clinical interpretability of the model.

[0127] The encoder adopts a SegFormer structure and consists of a stack of multiple layers of Transformer encoders. The encoder obtains rich image context information through multi-scale feature extraction modules (MSPP) such as Overlap Patch Embeddings modules and MiT Blocks. Each layer contains an efficient self-attention module (EfficientSelf-Attention) and a feedforward neural network (Mix-FFN) based on the self-attention mechanism, thereby accurately capturing the details and global features of medical images. These modules effectively capture long-range dependencies through the self-attention mechanism and optimize the feature processing method by utilizing the global and local feature extraction capabilities of Mix-FFN. At the same time, the encoder retains the skip connection mechanism of the traditional U-Net model, and fuses the low-level features of the encoder with the high-level features of the decoder through skip connections to enhance the ability to recover spatial detail information and improve segmentation accuracy.

[0128] The decoder introduces the Concurrent Spatial and Channel Squeeze and Channel Excitation (scSE) attention mechanism module, which includes the spatial attention module (Channel Squeeze and SpatialExcitation, sSE) and the channel attention module (Spatial Squeeze and Channel Excitation, cSE).

[0129] The spatial attention module generates a spatial weight map through 1×1 convolution to enhance attention to important spatial locations. Specifically, the spatial attention module first divides the input feature map U into multiple small blocks u i,j , each small block contains all channel features of a specific spatial location. Then, through a 1×1 convolution operation q=W sq ×U compresses the channel and generates a projection tensor q, each element q of the tensor i,jRepresents the linear combination of all channel features of the spatial position (i, j). Then, this projection tensor q is input to the Sigmoid activation function σ(.) to generate a scalar σ(q i,j ), which represents the relative importance of spatial position (i, j). Finally, by performing an element-wise product of the generated spatial weight map with the original feature map, the module can focus on important spatial regions, improve the model's ability to perceive key information, and ignore irrelevant locations.

[0130] The channel attention module models the importance of each channel through global average pooling and fully connected layers to further enhance the feature expression capability. In this module, the input feature map U is compressed by global average pooling, and the spatial information of each channel is aggregated into a scalar to generate a vector z∈R 1×1×C , where each element z k Represents the global information of the kth channel. Next, two fully connected layers are used to transform the vector and generate the weight of each channel through the Sigmoid activation function. Indicates the importance of the channel. Finally, these weights are multiplied by each channel of the input feature map, strengthening important channel features while suppressing unimportant channel features, thereby optimizing the expressive power of the feature map.

[0131] In the decoder, the scSE module is applied after each upsampling layer to improve segmentation accuracy. Specifically, by combining spatial excitation and channel excitation, the scSE module optimizes the spatial and channel information in the decoder feature map after each upsampling step, allowing the model to accurately restore the spatial details of the image while focusing on key spatial and channel features, further improving segmentation accuracy and performance.

[0132] The model uses depthwise separable convolution technology in both the encoder and decoder designs, decomposing the standard convolution into depthwise convolution and pointwise convolution to reduce the number of parameters and computational complexity while maintaining the model's efficient feature extraction capabilities. Depthwise separable convolution is mainly used in the following two key locations:

[0133] 1) Convolution operation after upsampling layer: In the decoder, the convolution operation after each upsampling layer is replaced by depthwise separable convolution to restore the resolution of the feature map and optimize the detail information.

[0134] 2) Feature Fusion Convolution after Skip Connection: In the skip connection mechanism between the encoder and decoder, low-level features and high-level features need to be fused through convolution. Here, depthwise separable convolution is used instead of standard convolution to efficiently fuse feature information at different levels.

[0135] In addition, the Mixed Precision Training strategy combines 16-bit floating-point and 32-bit floating-point calculations. By using both types of precision in the model, the memory usage can be effectively reduced and training efficiency can be improved. Without affecting the segmentation accuracy of the model, some weights and gradients in the calculation process are stored as 16-bit floating-point numbers to reduce memory requirements, enabling the model to process larger batches of data or run on hardware with limited memory resources. At the same time, the key calculation parts still maintain 32-bit precision to ensure the accuracy and stability of the model. Mixed precision training can not only improve training speed, but also optimize segmentation results, so that the model can generate high-quality segmented images more quickly in practical applications. It effectively reduces memory usage and improves training speed, and is particularly suitable for processing the computing needs of large-scale medical image datasets.

[0136] The model adopts a dual segmentation head structure, in which the cell probability segmentation head generates a cell probability map through a convolution layer and a nonlinear activation function (such as ReLU) to accurately segment the boundaries of cells. Figure 2 As shown in the figure, the gradient flow tracking segmentation head calculates the gradient flow features and uses convolutional layers, batch normalization layers, and bilinear interpolation to identify and separate adjacent cells. The structure is as follows: Figure 3 The output results of the two segmentation heads are optimized by calculating their difference and setting an error threshold to optimize the segmentation accuracy.

[0137] Data preprocessing: Medical images in various formats are preprocessed and uniformly converted into RGB images. A series of spatial and intensity transformations (such as random scaling, cropping, flipping, rotation, Gaussian noise, contrast adjustment, smoothing, histogram equalization, and sharpening) are performed to enhance data quality and diversity.

[0138] Image segmentation based on a lightweight U-Net model: This model uses SegFormer as the encoder and replaces ordinary convolutions with depthwise separable convolutions, significantly reducing the number of model parameters and computational complexity. The model retains the traditional U-Net skip connection mechanism and introduces the SCSE attention mechanism in the decoder to enhance feature expression capabilities.

[0139] Use pre-trained models for transfer learning: During the model training process, a pre-training mechanism is introduced and pre-trained weights are used for transfer learning to further enrich the semantic information of the model and enhance the model's generalization ability.

[0140] Mixed-precision training: This strategy uses a mixed-precision training strategy that combines 16-bit floating-point and 32-bit floating-point calculations to effectively reduce video memory usage, increase training speed, and optimize the model's segmentation results and processing efficiency.

[0141] The segmentation results are further optimized through the dual segmentation head structure: the dual segmentation head structure of the cell probability and gradient flow tracking method and the binary cross entropy loss function are used to perform detailed segmentation of the target area and identification of adjacent areas to improve the segmentation accuracy. The output results are as follows Figure 8 shown.

[0142] Model Deployment and Inference: Deploy the trained model as a system for clinical use. During the inference phase, the model generates not only cell segmentation maps but also Class Activation Mapping (CAM) attention visualizations to enhance the model's clinical interpretability and help physicians better understand and interpret the model's predictions. The core idea of ​​CAM is to combine the feature maps (activation maps) of the last convolutional layer of the neural network with the weights to generate a heat map that indicates the contribution of different regions to the final classification.

[0143] Furthermore, the method further includes generating a Class Activation Mapping (CAM) attention visualization graph to enhance the clinical interpretability of the model; the specific implementation is as follows:

[0144] (31): Obtaining feature maps from the last convolutional layer: The output of the last convolutional layer of a trained medical image segmentation model is a set of feature maps that capture the high-level features of the image. When the image is forward propagated, the network generates these feature maps, which contain visual information about each region in the input image.

[0145] (32): Apply global average pooling to each feature map: In CAM, global average pooling (GAP) is first performed on each convolutional feature map. Through the GAP operation, each feature map of the network is converted into a single number that represents the overall information of the feature map. For each feature map A k (Assuming there are K feature maps), the result of global average pooling z k is the average value of the pixel value of the feature map:

[0146]

[0147] Among them, H×W is the spatial size of the feature map, It is the feature map A k The value at position (i, j) in .

[0148] (33): Weighted synthetic feature map: The weight of the last fully connected layer in the network (usually the weight in the classification task) will be combined with each feature map A k The global average pooling result z k Multiplication is used to weight the feature map. Assuming the output category is C, for each category c, the final CAM image is calculated as follows:

[0149]

[0150] in, is the weight of the last fully connected layer, representing the feature map A k The contribution to category c.

[0151] (34): Generate a heat map for visualization: Remap the weighted feature map back to the spatial domain of the input image to generate a heat map. Each value in the heat map represents the importance of that location in the predicted category.

[0152] (35): Overlay heatmap and original image: Overlay the CAM heatmap onto the original image to more intuitively show the areas that the model focuses on when processing the image.

[0153] In this way, doctors or other users can observe which image regions have a high influence on the model's final prediction.

[0154] Collect open-source multimodal cell segmentation datasets: To improve the model's segmentation capabilities under different microscopic image modes, the model training dataset collects multiple open-source multimodal cell segmentation datasets. These datasets cover a variety of microscopic imaging techniques, including fluorescence microscopy, confocal laser scanning microscopy (CLSM), transmission electron microscopy (TEM), scanning electron microscopy (SEM), phase contrast microscopy, confocal phase contrast microscopy (DIC), fusion protein expression microscopy, super-resolution microscopy, etc. The multimodal nature of these datasets provides the model with rich training samples, covering different contrasts, resolutions, cell structure features, etc. This diversity of input data can help the model accurately segment cells under different imaging conditions, improving the model's generalization ability and applicability in practical applications.

[0155] After converting all images in the dataset to RGB images, we use padding to expand the image boundaries to the target size and then randomly crop them to a final size of 512x512. A series of data augmentation operations are then applied to the cropped images, including random scaling, flipping, rotation, Gaussian noise addition, contrast adjustment, smoothing, histogram equalization, and sharpening. These processed images are then directly fed into the model for training.

[0156] The encoder of the lightweight U-Net optimization model adopts the MiT-B5 structure, which consists of four levels of MiT Block.

[0157] The encoder first divides the input 512×512 RGB image into overlapping image patches through Overlap Patch Embeddings and maps them to a 64-channel feature space.

[0158] Next, the feature map passes through four MiT Blocks in sequence. Each Block uses the self-attention mechanism and the multi-head attention layer to process the feature map, so that its spatial size gradually decreases and the number of channels gradually increases: 64, 128, 320 and 512 respectively.

[0159] The first layer of the decoder is the Position-wise Attention Block (PAB), which receives the feature map from the last layer of the encoder, MiT Block, and enhances the important spatial information in the feature map through the position attention mechanism while keeping the resolution of the feature map unchanged.

[0160] In the subsequent decoder layers, the spatial size of the feature map is gradually expanded through upsampling operations, and the feature maps of the corresponding layers of the encoder are fused with the feature maps in the decoder using the skip connection mechanism.

[0161] The first upsampling layer receives the output of PAB, and after processing by the first upsampling layer and the first scSE module, it is fused with the feature map output by the third MiT Block of the encoder.

[0162] The second upsampling layer receives the output of the first adder, and after processing by the second upsampling layer and the second scSE module, it is fused with the feature map output by the second MiT Block of the encoder.

[0163] The third upsampling layer receives the output of the second adder, and after processing by the third upsampling layer and the third scSE module, it is fused with the feature map output by the first MiT Block of the encoder.

[0164] The scSE module is introduced after each decoder layer to further enhance the expression of feature maps through spatial attention and channel attention mechanisms, ensuring that important semantic information is retained while restoring the resolution.

[0165] Finally, the decoder outputs a feature map of size 512×512, which is used to generate segmentation results.

[0166] In order to achieve a lightweight design of the model, the model uses depthwise separable convolutional layers to replace ordinary convolutional layers in the decoder and segmentation head. Specifically, the convolutional layer of the decoder: In each layer of the decoder, especially in the convolution processing after the upsampling operation, depthwise separable convolutional layers are used. These convolutional layers are responsible for restoring spatial resolution and extracting detail features. By introducing depthwise separable convolutions, the computational complexity and number of parameters are effectively reduced. Convolutional layer in the segmentation head: Depthwise separable convolutional layers are also used in the cell recognition head (Recognition Head) and the cell distinction head (Distinction Head). In the process of generating binary masks and gradient flow maps, these segmentation heads improve the processing speed and efficiency of the model in the inference stage through the application of depthwise separable convolutions.

[0167] A pre-training mechanism has been incorporated into the training module, allowing easy activation and management of pre-trained parameters through a unified configuration file. For pre-trained models, the pre-trained weights provided by the original MEDIAR project team were used for transfer learning. These pre-trained weights incorporate a rich set of feature representations, particularly fundamental knowledge in the field of medical imaging. These weights provide effective initial weights for the model, accelerating the training process and reducing model convergence time. They also enhance the model's ability to capture complex regions such as cellular structures, improving segmentation accuracy and generalization capabilities, enabling it to perform well across medical images of different modalities.

[0168] A mixed-precision training strategy is used to optimize the model training process. Mixed-precision training combines 16-bit and 32-bit floating-point calculations. By using both types of precision in the model, it can effectively reduce video memory usage and improve training efficiency. Without affecting the model's segmentation accuracy, some weights and gradients in the calculation process are stored as 16-bit floating-point numbers to reduce video memory requirements, enabling the model to process larger batches of data or run on hardware with limited video memory resources. At the same time, key calculations still maintain 32-bit precision to ensure model accuracy and stability. Mixed-precision training not only speeds up training but also optimizes segmentation results, enabling the model to generate high-quality segmented images more quickly in practical applications.

[0169] The decoder output feature map is processed by a dual segmentation head structure. The model uses two segmentation heads to achieve a more refined segmentation task:

[0170] Recognition Head: Receives the final feature map from the decoder and generates a binary mask of the cell after convolution operation, which is used to identify the cell boundaries and contours in the image.

[0171] Distinction Head: Another segmentation head uses the gradient flow tracking method provided by Cellpose. It receives the final feature map from the decoder and calculates the gradient flow information to generate a gradient map, which helps the model better identify and distinguish adjacent cell regions. This method uses the cell topology to ensure that the boundaries between adjacent cells are accurately handled during the segmentation process, improving the overall accuracy and reliability of the segmentation.

[0172] In terms of loss function, the model uses a combined loss function to optimize the segmentation performance. The combined loss function combines the binary cross-entropy loss (L BCE ) and mean square error loss (mean square error loss, L MSE ) to balance the optimization requirements of different tasks. The calculation formula is as follows:

[0173] L(x,y)=L BCE (h CR (f θ (x),y cell )+λ·L MSE (h CD (f θ (x),y gradient )

[0174]

[0175] Among them, L BCE The loss calculation for the cell recognition head is performed by comparing the cell binary mask h generated by the model CR (f θ (x),y cell ) and the true label y cell The cross entropy between them is used to guide model learning.

[0176] L MSE It is used for cell differentiation head (DistinctionHead), which generates the gradient flow graph h by computing the model CD (f θ (x),y gradient ) and the true gradient flow y gradient The mean square error between them helps the model to more accurately identify and distinguish adjacent cell regions.

[0177] True gradient flow y gradient The generation method is as follows: First, based on the binary mask of the cell, the cell boundaries are identified by calculating the neighborhood relationship of each pixel in the mask. Then, by expanding and iterating these boundaries, the gradient information of each pixel is generated, which represents the relative change of the pixel and the adjacent cells. This gradient information reflects the direction and strength of the cell boundary, and finally forms a complete gradient flow map (y gradient The weight parameter λ is used to balance the contribution of the two losses. In experiments, λ is usually set to 0.5 to ensure that the losses of cell recognition and discrimination tasks are reasonably balanced.

[0178] To adapt to clinical applications, the model is integrated into a system that can not only perform cell segmentation but also generate CAM images for interpretable analysis. In the inference stage, the system first generates cell segmentation masks and gradient flow maps through the trained model. These segmentation results can be directly used for cell identification and analysis. To enhance the clinical interpretability of the model, the system then generates CAM images through Grad-CAM technology. Specifically, in the process of generating CAM, the system captures important activation information in the last convolutional layer of the model and calculates weighted images related to the segmentation results based on this activation information. Grad-CAM generates an image that highlights the model's focus area by superimposing these weighted maps with the original input image, allowing clinicians to intuitively understand the basis for the model's decision.

[0179] like Figure 3 As shown in the figure, the feature map passes through four MiT Blocks in sequence. Each MiT Block mainly includes three components: efficient self-supervised attention mechanism (Efficient Self-Attention), hybrid feedforward network (Mix-FFN), and overlap patch mapping (Overlap Patch Mapping). Using MiT Block to process the feature map, its spatial size is gradually reduced, while the number of channels is gradually increased: 64, 128, 320, and 512 respectively. At the same time, it strengthens the model's understanding of the spatial structure of the image, helps retain more edge information and contextual information, and thus improves segmentation or classification performance.

[0180] Furthermore, Efficient Self-Attention is a method to optimize the self-attention mechanism, aiming to reduce computational and memory overhead, especially when processing long sequences or large-scale data. The computational complexity of the traditional self-attention mechanism is O(N 2), where N is the length of the input sequence. This makes computation and memory consumption very high when processing long sequences or large-scale data. The self-supervised attention mechanism can reduce computational complexity through methods such as sparse attention, low-rank approximation, or factorized attention. For example, it only calculates attention locally or between adjacent elements, or uses matrix decomposition techniques to simplify computation. These optimizations enable the self-supervised attention mechanism to significantly improve processing efficiency while maintaining expressiveness, especially in large-scale tasks.

[0181] The Mixed Feedforward Network (Mix-FFN) is a design approach that combines multiple feedforward network structures, aiming to improve the model's expressiveness and efficiency through a diverse set of layers and operations. Unlike traditional feedforward networks that use only a single type of layer, Mix-FFN fuses different layers, such as standard fully connected layers, convolutional layers, and self-attention layers, to capture multi-scale features and optimize information flow. This design approach is particularly well-suited for processing complex tasks such as large-scale data in natural language processing and computer vision. Through this hybrid approach, Mix-FFN enhances the model's feature learning capabilities while maintaining low computational complexity, improving its performance and generalization.

[0182] Overlap Patch Mapping is an image processing technique that reduces boundary effects and improves the quality of image reconstruction or segmentation by segmenting an image into multiple small patches with overlapping regions. The overlapping regions allow the model to better capture local and global features, particularly image edges and details. It is commonly used in image stitching, denoising, and segmentation tasks.

[0183] Example 2: This embodiment provides a medical image segmentation system based on multimodal data, including: an acquisition module, which is configured to: acquire a medical image to be segmented; a segmentation module, which is configured to: input the medical image to be segmented into a trained medical image segmentation model to obtain a medical image segmentation result; wherein the trained medical image segmentation model includes: an encoder and a decoder connected in sequence, the encoder performs multi-scale feature extraction on the medical image to be segmented, and uses a self-attention mechanism to enhance the details of the extracted features; the decoder improves the feature expression capability through a spatial attention mechanism and a channel attention mechanism; the decoder uses the dual segmentation head results to achieve cell recognition and cell differentiation. The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may be subject to various modifications and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A medical image segmentation method based on multimodal data, characterized by: include: Acquire a medical image to be segmented; Input the medical image to be segmented into the trained medical image segmentation model to obtain the medical image segmentation result; The trained medical image segmentation model includes an encoder and a decoder connected in sequence. The encoder extracts multi-scale features from the medical image to be segmented and enhances the details of the extracted features using a self-attention mechanism. The decoder improves the feature expression capability through a spatial attention mechanism and a channel attention mechanism. The decoder uses the dual segmentation head results to realize cell recognition and differentiation. The encoder includes: an Overlap Patch Embeddings module, a first MiT Block module, a second MiT Block module, a third MiT Block module, and a fourth MiT Block module connected in sequence; the input end of the Overlap Patch Embeddings module is the input end of the encoder; the output end of the fourth MiT Block module is the output end of the encoder; The decoder includes: a PAB Blocks module, a first upsampling layer, a first scSE module, a first adder, a second upsampling layer, a second scSE module, a second adder, a third upsampling layer, a third scSE module, a third adder, and a dual splitting head connected in sequence; the input end of the PAB Blocks module is the input end of the decoder; the output end of the dual splitting head is the output end of the decoder; Among them, the output end of the fourth MiT Block module is connected to the input end of the PAB Blocks module; the output end of the third MiT Block module is connected to the input end of the first adder; the output end of the second MiT Block module is connected to the input end of the second adder; the output end of the first MiT Block module is connected to the input end of the third adder.

2. The medical image segmentation method based on multimodal data according to claim 1, wherein: Input the medical image to be segmented into the trained medical image segmentation model to obtain the medical image segmentation result; The training process of the trained medical image segmentation model includes: Constructing a training set, wherein the training set is medical images with known image segmentation results; Input the training set into the medical image segmentation model to train the segmentation model. When the loss function value of the model no longer decreases or the number of iterations reaches a set number, stop training to obtain the trained medical image segmentation model. The loss function of the model ,for: ； ； 。 3. The medical image segmentation method based on multimodal data according to claim 1, wherein: The internal structures of the first MiTBlock module, the second MiT Block module, the third MiT Block module and the fourth MiT Block module are consistent. The first MiT Block module includes: The first attention module, the second attention module, the third attention module...the Nth attention module and the Overlap Patch Merging module are connected in sequence; The internal structures of the first attention module, the second attention module, the third attention module, ... the Nth attention module are consistent. The first attention module includes: an Efficient self-Attention module and a Mix-FFN module connected in sequence; The Efficient self-Attention module works by performing linear transformations on the input features to obtain query, key, and value matrices. When calculating attention, it first calculates the similarity between the query and the key through dot product, then calculates the similarity between the query and the key through the softmax activation function, generates attention weights, multiplies them element-by-element, applies the weights to the value, and finally obtains the final output through addition or concatenation. The Overlap Patch Merging module operates as follows: first, the original image is divided into overlapping blocks with overlapping areas between adjacent blocks; the block size is a set size; then, features are extracted from each image block using a convolutional neural network; then, image blocks in the overlapping area are merged, including by taking an average; finally, the processed image blocks are merged into a complete image by copying the pixel values ​​of the blocks to corresponding locations.

4. The medical image segmentation method based on multimodal data according to claim 1, wherein: The PABBlocks module includes: First, the query, key, and value feature maps are generated through convolution operations; Then, the similarity between the query and the key is calculated to generate the attention weight matrix; Subsequently, the attention weight matrix is ​​weighted and summed with the value feature map to obtain the enhanced feature map; Finally, the enhanced feature map is fused with the original feature map to output the final feature map.

5. The medical image segmentation method based on multimodal data according to claim 4, characterized in that: The PABBlocks module includes: an input end of the PAB Blocks module; the input end of the PAB Blocks module is respectively connected to the input end of the first convolutional layer, the input end of the second convolutional layer, and the input end of the third convolutional layer; the output end of the first convolutional layer and the output end of the second convolutional layer are both connected to the input end of the first multiplier; the output end of the first multiplier is connected to the input end of the activation function layer, and the output end of the activation function layer and the output end of the third convolutional layer are both connected to the input end of the second multiplier; the output end of the second multiplier is connected to the input end of the fourth adder; the output end of the fourth adder is connected to the input end of the fourth convolutional layer; the output end of the fourth convolutional layer is the output end of the PAB Blocks module; The PAB Blocks module is used to enhance spatial attention. It calculates the relative importance of different positions in the image through the self-attention mechanism, thereby enhancing the model's attention to important areas.

6. The medical image segmentation method based on multimodal data according to claim 1, wherein: The internal structures of the first scSE module, the second scSE module and the third scSE module are consistent. The first scSE module includes: a parallel spatial attention module and a channel attention module; The spatial attention module uses a channel compression and spatial excitation mechanism. The spatial attention module focuses on important spatial regions by adjusting the weights of the spatial dimensions of the feature map. Its working process is as follows: (11): Input tensor slice: input feature map Divided into ,in, , Indicates spatial location All channel features at ; (12): Channel compression: for input tensors The channel dimension is compressed by convolution operation ;in , generate the projection tensor , tensor Each element of Indicates spatial location Linear combination of all channels; (13): Spatial excitation: Projection tensor Input to Sigmoid activation function , we get the scalar value , which is used to recalibrate the input features spatial incentives; ； (14): Feature recalibration: generating new feature maps , the formula is as follows: in, Represents a specific spatial location the relative importance of .

7. The medical image segmentation method based on multimodal data according to claim 6, wherein: The channel attention module includes: spatial compression and channel excitation mechanism. The spatial compression and channel excitation mechanism is to strengthen important channel features and suppress unimportant channels by adjusting the weight of each channel on the feature map. Its working process is as follows: (21): Input feature map: Input feature map Include channels, each channel ∈R H×W Represents the spatial characteristics of the image; (22): Spatial Squeeze: Through the global average pooling operation, the spatial information of each channel is compressed into a scalar. The pooling operation is as follows: The resulting vector z∈R 1×1×C Contains global information for each channel; (23): Channel-dependent encoding: The pooled vector is encoded by two fully connected layers Perform the transformation; first, perform a linear transformation: Where W1∈R C×C / 2 , W2∈R C / 2×C is the weight of the fully connected layer, is the ReLU activation function; (24): Activation function and normalization: Then, the Sigmoid activation function is used to The values ​​of are mapped to the interval [0, 1]: The resulting vector ∈R 1×1×C Represents the importance score of each channel; (25): Channel Excitation: Finally, the input feature map U is recalibrated using the obtained excitation value, and the activation value of each channel is Will be compared with the corresponding channel feature map Multiply them to get the recalibrated output feature map: in, Indicates the The recalibration result will strengthen the important channels and suppress the unimportant channels.

8. The medical image segmentation method based on multimodal data according to claim 1, wherein: The dual segmentation head includes: a cell probability segmentation head and a gradient flow tracking segmentation head; The cell probability segmentation head includes: a first convolution layer, a first activation function layer, a first normalization layer, a second convolution layer and a first upsampling layer connected in sequence; the first convolution layer receives the final feature map from the decoder, extracts the feature information of the cell through a convolution operation, and is used to generate a binary mask of the cell; the first activation function layer uses a Mish activation function to introduce nonlinearity into the model and enhance the learning ability of the cell features; the first normalization layer performs batch normalization on the convolution output to stabilize the training process and accelerate convergence; the second convolution layer converts the feature map into a final binary mask through convolution to represent the boundary of the cell; the first upsampling layer uses bilinear interpolation for upsampling; the cell probability segmentation head is used to generate a binary mask of the cell, identify the cell boundary in the image, and ensure accurate segmentation of the cell position; The gradient flow tracking segmentation head includes: a third convolutional layer, a second activation function layer, a second batch normalization layer, a fourth convolutional layer and a second upsampling layer connected in sequence; the third convolutional layer receives the final feature map from the decoder and extracts the gradient flow information of the cell boundary through a convolution operation; the function of the convolution layer is to extract detailed information of the cell boundary from the feature map; the second activation function layer uses the Mish activation function to help the network learn nonlinear features and enhance the representation ability of the cell boundary; the second batch normalization layer normalizes the convolution output through batch normalization; the fourth convolutional layer further processes the convolution output, converts it into a cell gradient map, and outputs gradient flow information representing the cell boundary, usually two channels, representing the gradient in the horizontal and vertical directions respectively; the second upsampling layer, if it is necessary to restore the spatial resolution of the original image, uses bilinear interpolation for upsampling to restore the feature map to a higher resolution; the gradient flow tracking segmentation head is used to accurately distinguish the boundaries of adjacent cell areas through the gradient flow tracking method, ensure the accuracy of adjacent cell segmentation, and optimize the processing of cell boundaries.

9. A medical image segmentation system based on multimodal data, characterized by: include: An acquisition module is configured to: acquire a medical image to be segmented; a segmentation module configured to: input the medical image to be segmented into the trained medical image segmentation model to obtain a medical image segmentation result; The trained medical image segmentation model includes an encoder and a decoder connected in sequence. The encoder extracts multi-scale features from the medical image to be segmented and enhances the details of the extracted features using a self-attention mechanism. The decoder improves the feature expression capability through a spatial attention mechanism and a channel attention mechanism. The decoder uses the dual segmentation head results to realize cell recognition and differentiation. The encoder includes: an Overlap Patch Embeddings module, a first MiT Block module, a second MiT Block module, a third MiT Block module, and a fourth MiT Block module connected in sequence; the input end of the Overlap Patch Embeddings module is the input end of the encoder; the output end of the fourth MiT Block module is the output end of the encoder; The decoder includes: a PAB Blocks module, a first upsampling layer, a first scSE module, a first adder, a second upsampling layer, a second scSE module, a second adder, a third upsampling layer, a third scSE module, a third adder, and a dual splitting head connected in sequence; the input end of the PAB Blocks module is the input end of the decoder; the output end of the dual splitting head is the output end of the decoder; Among them, the output end of the fourth MiT Block module is connected to the input end of the PAB Blocks module; the output end of the third MiT Block module is connected to the input end of the first adder; the output end of the second MiT Block module is connected to the input end of the second adder; the output end of the first MiT Block module is connected to the input end of the third adder.

Images