Medical image segmentation method and system based on multi-modal data

Abstract

The invention relates to the technical field of medical image segmentation, in particular to a medical image segmentation method and system based on multi-modal data, and the method comprises the steps: obtaining a to-be-segmented medical image; inputting a medical image to be segmented into the trained medical image segmentation model to obtain a medical image segmentation result; wherein the trained medical image segmentation model comprises an encoder and a decoder which are connected in sequence, and the encoder performs multi-scale feature extraction on a medical image to be segmented and performs detail enhancement on the extracted features by adopting a self-attention mechanism; the decoder improves the feature expression ability through a space attention mechanism and a channel attention mechanism; and the decoder realizes cell recognition and cell distinguishing by adopting a double-segmentation head result. The medical image segmentation precision and efficiency are improved, and the method is suitable for processing requirements of large-scale medical image data sets.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image segmentation, and in particular to a medical image segmentation method and system based on multimodal data. Background Art

[0002] Medical image segmentation plays a vital role in modern medical diagnosis and treatment, especially in tumor detection, organ segmentation and lesion area identification. Traditional image segmentation methods, such as algorithms based on region growing, threshold segmentation and edge detection, although they can provide effective segmentation results in some simple scenarios, often perform poorly when dealing with medical images with complex structures, low contrast and noise interference.

[0003] With the rapid development of deep learning, segmentation models based on convolutional neural networks, especially U-Net and its variants, have become the mainstream methods for medical image segmentation. U-Net has successfully achieved effective fusion between global and local features through the encoder-decoder structure and skip connections. However, when processing multimodal medical images, traditional U-Net models still face problems such as high model complexity, large amount of computing resources occupied during the training process, and insufficient capture of details and global features. In addition, with the continuous growth of the size of medical image datasets, how to improve segmentation efficiency and reduce computational overhead while ensuring segmentation accuracy has become a challenge that needs to be solved urgently.

[0004] In order to solve these problems, researchers have proposed many improvement strategies in recent years, such as introducing attention mechanisms, multi-scale feature extraction modules, and deep separable convolutions. These technologies have improved the performance of the model to varying degrees, but there are still problems such as high model complexity and slow inference speed. Summary of the invention

[0005] In order to address the shortcomings of the prior art, the present invention provides a medical image segmentation method and system based on multimodal data; proposes a lightweight U-Net optimization model, and combines multimodal data processing, attention mechanism, dual segmentation head structure and mixed precision training technologies to improve the accuracy and efficiency of medical image segmentation, which is suitable for the processing requirements of large-scale medical image datasets.

[0006] On the one hand, a medical image segmentation method based on multimodal data is provided, comprising:

[0007] Acquire a medical image to be segmented;

[0008] Inputting the medical image to be segmented into the trained medical image segmentation model to obtain the medical image segmentation result;

[0009] Among them, the trained medical image segmentation model includes: an encoder and a decoder connected in sequence, the encoder performs multi-scale feature extraction on the medical image to be segmented, and uses a self-attention mechanism to enhance the details of the extracted features; the decoder improves the feature expression ability through a spatial attention mechanism and a channel attention mechanism; the decoder uses a dual segmentation head result to realize cell recognition and cell differentiation.

[0010] On the other hand, a medical image segmentation system based on multimodal data is provided, comprising:

[0011] An acquisition module is configured to: acquire a medical image to be segmented;

[0012] A segmentation module is configured to: input the medical image to be segmented into a trained medical image segmentation model to obtain a medical image segmentation result;

[0013] Among them, the trained medical image segmentation model includes: an encoder and a decoder connected in sequence, the encoder performs multi-scale feature extraction on the medical image to be segmented, and uses a self-attention mechanism to enhance the details of the extracted features; the decoder improves the feature expression ability through a spatial attention mechanism and a channel attention mechanism; the decoder uses a dual segmentation head result to realize cell recognition and cell differentiation.

[0014] On the other hand, there is also provided an electronic device, comprising:

[0015] a memory for non-transitory storage of computer-readable instructions; and

[0016] a processor for executing the computer readable instructions,

[0017] When the computer-readable instructions are executed by the processor, the method described in the first aspect is executed.

[0018] On the other hand, a storage medium is provided, which non-temporarily stores computer-readable instructions, wherein when the non-temporary computer-readable instructions are executed by a computer, the method described in the first aspect is executed.

[0019] On the other hand, a computer program product is provided, comprising a computer program, wherein the computer program is used to implement the method described in the first aspect when running on one or more processors.

[0020] The above technical solution has the following advantages or beneficial effects:

[0021] The present invention improves the accuracy and efficiency of medical image segmentation, reduces computational complexity and video memory usage, and enhances the clinical interpretability of the model. Based on the traditional U-Net structure, a number of optimization improvements have been made, so that the model has better segmentation capabilities when processing multimodal data, and is suitable for the processing needs of large-scale medical image data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0023] Figure 1 It is a step diagram of the overall design scheme of an embodiment of the present invention;

[0024] Figure 2 This is the structure diagram of the cell probability segmentation head;

[0025] Figure 3 Segmentation head structure diagram for gradient flow tracking;

[0026] Figure 4 This is a structural diagram of the optimized U-Net model according to an embodiment of the present invention;

[0027] Figure 5 : is a model structure diagram of MiT Block in the SegFormer encoder of an embodiment of the present invention;

[0028] Figure 6 2 is a model structure diagram of a pixel-level attention block PAB Block according to an embodiment of the present invention;

[0029] Figure 7 Schematic diagram of the model structure of the channel and spatial attention block scSE of an embodiment of the present invention;

[0030] Figure 8 This is a preliminary output result diagram of the optimized U-Net according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0032] Cell images are often obtained through different technical means and therefore have multimodal characteristics. At present, the acquisition of cell images in the medical field mainly relies on a variety of advanced microscopic imaging technologies, including optical microscopy, electron microscopy, confocal microscopy, super-resolution microscopy, and atomic force microscopy. Although the multimodal information in microscope images mainly comes from the type of microscope used for imaging and technical differences, there is still a large amount of modal information derived from the differences in the observed cell and tissue types. Moreover, in actual work, it is impossible to completely separate different types of cells from a tissue slice, so the data faced by the cell segmentation task in actual application must be multimodal.

[0033] Embodiment 1

[0034] This embodiment provides a medical image segmentation method based on multimodal data;

[0035] like Figure 1 As shown, the medical image segmentation method based on multimodal data includes:

[0036] S101: Acquire a medical image to be segmented;

[0037] S102: Input the medical image to be segmented into the trained medical image segmentation model to obtain the medical image segmentation result; wherein the trained medical image segmentation model includes: an encoder and a decoder connected in sequence, the encoder performs multi-scale feature extraction on the medical image to be segmented, and uses a self-attention mechanism to enhance the details of the extracted features; the decoder improves the feature expression capability through a spatial attention mechanism and a channel attention mechanism; the decoder uses a dual segmentation head result to realize cell recognition and cell differentiation.

[0038] Furthermore, the medical image to be segmented is obtained, and the medical image is a medical image containing cells.

[0039] Further, S102: inputting the medical image to be segmented into a trained medical image segmentation model to obtain a medical image segmentation result; the training process of the trained medical image segmentation model includes:

[0040] Constructing a training set, wherein the training set is medical images with known image segmentation results;

[0041] Input the training set into the medical image segmentation model to train the segmentation model. When the loss function value of the model no longer decreases or the number of iterations reaches a set number, stop the training to obtain the trained medical image segmentation model.

[0042] The loss function of the model, L(x,y), is:

[0043] L(x,y)=L BCE (h CR (f θ (x),y cell )+λ·L MSE (h CD (f θ (x),y hrafirnt );

[0044]

[0045] Furthermore, constructing a training set includes:

[0046] Medical images in various formats are uniformly converted to RGB images, normalized, and improved in data quality and diversity through a series of spatial and intensity transformations, including random scaling, cropping, flipping, rotation, Gaussian noise, contrast adjustment, smoothing, histogram equalization, and sharpening.

[0047] Furthermore, if Figure 4 As shown, the encoder includes: an Overlap Patch Embeddings module, a first MiT Block module, a second MiT Block module, a third MiT Block module and a fourth MiT Block module connected in sequence; the input end of the Overlap Patch Embeddings module is the input end of the encoder; the output end of the fourth MiT Block module is the output end of the encoder;

[0048] The decoder comprises: a PAB Blocks module, a first upsampling layer, a first scSE module, a first adder, a second upsampling layer, a second scSE module, a second adder, a third upsampling layer, a third scSE module, a third adder and a double segmentation head connected in sequence; the input end of the PAB Blocks module is the input end of the decoder; the output end of the double segmentation head is the output end of the decoder;

[0049] Among them, the output end of the fourth MiT Block module is connected to the input end of the PAB Blocks module; the output end of the third MiTBlock module is connected to the input end of the first adder; the output end of the second MiT Block module is connected to the input end of the second adder; the output end of the first MiT Block module is connected to the input end of the third adder.

[0050] Furthermore, the encoder is used to extract cell features. It uses the traditional U-Net as the backbone structure, retains the original skip connection mechanism, introduces SegFormer as the encoder, and makes full use of the efficient self-supervised attention mechanism (Efficient Self-Attention) designed based on the lightweight Transformer structure in its MiT Blocks to achieve accurate extraction of cell features and enhance the original U-Net model's perception and understanding of space.

[0051] Furthermore, the decoder is used to enhance feature extraction. The scSE attention mechanism module is introduced, which is a mechanism that combines spatial and channel attention. The scSE attention mechanism module uses parallel sSE (Channel Squeeze and Spatial Excitation) and cSE (SpatialSqueeze and Channel Excitation) modules to enhance the feature expression ability of the model, especially for visual tasks such as segmentation and classification. The outputs of the two are fused by element-by-element addition to ensure that the model can fully capture the spatial and channel importance information of the input features to improve the performance of visual tasks.

[0052] Furthermore, the Overlap Patch Embeddings module includes: a stacking block division layer, a convolutional neural network, a feature splicing layer and an output layer. Overlap Patch Embeddings is an image encoding method for dividing an image into multiple overlapping image blocks and encoding each image block into a vector representation. First, the input image is divided into multiple overlapping image blocks of fixed size, and each block has a certain overlap with the adjacent blocks to ensure that key information is not lost. Then, each image block is extracted through a pre-trained convolutional neural network to generate a high-dimensional vector representation. Next, the feature vectors of all image blocks are spliced ​​together to form the encoding of the entire image. This encoding not only retains the local information of each image block, but also reflects the contextual relationship between blocks, and comprehensively considers local details and global information. Finally, the output image encoding can be used for computer vision tasks such as image segmentation and target detection to provide more accurate representation.

[0053] Furthermore, if Figure 5 As shown, the internal structures of the first MiT Block module, the second MiT Block module, the third MiTBlock module and the fourth MiT Block module are consistent, and the first MiT Block module includes:

[0054] The first attention module, the second attention module, the third attention module...the Nth attention module and the Overlap Patch Merging module are connected in sequence.

[0055] Among them, the internal structures of the first attention module, the second attention module, the third attention module, ... the Nth attention module are consistent, and the first attention module includes:

[0056] Efficient self-Attention module and Mix-FFN module connected in sequence.

[0057] Furthermore, the Efficient self-Attention module includes:

[0058] Perform linear transformation on the input features to obtain the Query, Key, and Value matrices respectively. When calculating Attention, first calculate the similarity between Query and Key through dot product, then calculate the similarity between Query and Key through Softmax activation function, generate attention weights, perform element-by-element multiplication, apply the weights to Value, and finally get the final output through addition or concatenation.

[0059] Furthermore, the Efficient self-Attention module is used to capture global dependencies. Through the self-attention mechanism, the model can capture long-range dependencies in the input sequence, and even long-distance information can be effectively connected through attention weights. This enables the model to better understand and process complex contextual information. The design of the Efficient Self-Attention module makes the calculation process more efficient than traditional global self-attention, and is suitable for processing large-scale data and long sequences. Through various optimization methods, the module reduces computing time and memory overhead, thereby improving the operating efficiency of the entire model.

[0060] Furthermore, the Mix-FFN module, full name: Mixed Feed-Forward Network, includes two different feed-forward neural network structures: global feed-forward neural network (Global FFN) and local feed-forward neural network (Local FFN). The global feed-forward neural network is a feed-forward neural network with a larger receptive field, which can better capture global context information. The local feed-forward neural network is a feed-forward neural network with a smaller receptive field, which can better capture local detail information.

[0061] Furthermore, the Mix-FFN module is used for feature extraction. The Mix-FFN module provides a more flexible feature extraction method by introducing different feedforward neural network structures. Traditional FFN uses the same nonlinear transformation at each position, while Mix-FFN allows different nonlinear transformations to be used at different positions, thereby increasing the expressive power of the model. By using global FFN and local FFN at the same time, the Mix-FFN module is more flexible and accurate in processing features at different positions. Global FFN can help the model capture longer-range dependencies and semantic information, while local FFN can better handle local details and subtle changes.

[0062] Furthermore, the Overlap Patch Merging module includes:

[0063] First, the original image is divided into overlapping blocks, and there are overlapping areas between adjacent blocks; the size of the block is a set size;

[0064] Then, feature extraction is performed on each image block through a convolutional neural network;

[0065] Next, the image blocks in the overlapping area are merged, and the merging methods include taking an average value or a weighted average value;

[0066] Finally, the processed image blocks are merged into a complete image by copying the pixel values ​​of the blocks to the corresponding positions.

[0067] Furthermore, the Overlap Patch Merging module is used to merge multiple overlapping image blocks into a complete image without overlap, so as to restore the details and integrity of the original image. This technology is often used in applications such as image stitching and image reconstruction, and can improve the accuracy and effect of image processing. In image processing, dividing an image into small blocks for processing can bring some benefits, such as reducing computational complexity and improving processing efficiency. However, when there is overlap between image blocks, they need to be merged to restore the integrity of the image.

[0068] Furthermore, if Figure 6 As shown, the PAB Blocks module includes:

[0069] First, the query, key, and value feature maps are generated through convolution operations;

[0070] Then, the similarity between the query and the key is calculated to generate the attention weight matrix;

[0071] Then, the attention weight matrix is ​​weighted and summed with the value feature map to obtain the enhanced feature map;

[0072] Finally, the enhanced feature map is fused with the original feature map to output the final feature map.

[0073] Furthermore, if Figure 6 As shown, the PAB Blocks module includes: an input end of the PAB Blocks module; an input end of the PABBlocks module, which is respectively connected to the input end of the first convolutional layer, the input end of the second convolutional layer, and the input end of the third convolutional layer; the output end of the first convolutional layer and the output end of the second convolutional layer are both connected to the input end of the first multiplier; the output end of the first multiplier is connected to the input end of the activation function layer, and the output end of the activation function layer and the output end of the third convolutional layer are both connected to the input end of the second multiplier; the output end of the second multiplier is connected to the input end of the fourth adder; the output end of the fourth adder is connected to the input end of the fourth convolutional layer; the output end of the fourth convolutional layer is the output end of the PABBlocks module.

[0074] Furthermore, the PAB Blocks module is used to enhance spatial attention. It calculates the relative importance of different positions in the image through the self-attention mechanism, thereby enhancing the model's attention to important areas. PAB can effectively enhance the features of important positions in the image and improve the accuracy and robustness of cell segmentation.

[0075] Furthermore, if Figure 7 As shown, the internal structures of the first scSE module, the second scSE module and the third scSE module are consistent, and the first scSE module includes: a parallel spatial attention module and a channel attention module;

[0076] The spatial attention module adopts channel compression and spatial excitation mechanism; the spatial attention module adjusts the weight of the spatial dimension of the feature map to achieve attention to important spatial areas. Its working process is as follows:

[0077] (11): Input tensor slicing: The input feature map U is split into U = [u 1,1 ,u 1,2 ,…,u i,j ,…,u H,W ], where u i,j ∈R 1×1×C ,u i,j Represents all channel features at the spatial position (i, j);

[0078] (12): Channel compression: compress the channel dimension of the input tensor U through convolution operation q = W sq ×U; where W sq ∈R 1×1×C×1, generate the projection tensor q∈R H×W , each element q of the tensor q i,j Represents the linear combination of all channels at spatial position (i, j);

[0079] (13): Spatial excitation: The projection tensor q is input into the Sigmoid activation function σ(.) to obtain the scalar value σ(q i,j ), which is used to recalibrate the spatial excitation of the input feature U;

[0080]

[0081] (14): Feature recalibration: Generate a new feature map U sSE , the formula is as follows:

[0082] U sSE =F sSE (U) = [σ(q 1,1 ) 1,1 ,…,σ(q i,j ) i,j ,…,σ(q H,W ) H,W ]

[0083] Among them, σ(q i,j ) represents the relative importance of a specific spatial position (i, j).

[0084] The spatial attention module recalibrates the input feature map to give higher weights to important spatial locations and ignore irrelevant locations.

[0085] Furthermore, the channel attention module includes: a spatial compression and channel excitation mechanism, which is to strengthen important channel features and suppress unimportant channels by adjusting the weight of each channel on the feature map. The working process is as follows:

[0086] (21): Input feature map: Input feature map U∈R H×W×C Contains C channels, each channel u i ∈R H×W Represents the spatial characteristics of an image.

[0087] (22): Spatial Squeeze: Through the global average pooling operation, the spatial information of each channel is compressed into a scalar.

[0088] For the kth channel, the pooling operation is as follows:

[0089]

[0090] The resulting vector z∈R 1×1×C Contains global information for each channel.

[0091] (23): Channel-dependent encoding: The pooled vector z is transformed through two fully connected layers. First, a linear transformation is performed:

[0092]

[0093] Among them, W 1 ∈R C×C / 2 ,W 2 ∈R C / 2×C is the weight of the fully connected layer, and δ(·) is the ReLU activation function. This step is used to encode the dependencies between channels.

[0094] (24): Activation function and normalization: Then, the Sigmoid activation function is used to convert The values ​​are mapped to the interval [0,1]:

[0095]

[0096] The vector obtained in this way Represents the importance score of each channel.

[0097] (25): Channel Excitation: Finally, the input feature map U is recalibrated using the obtained excitation value, and the activation value of each channel is Will be with the corresponding channel feature map u k Multiply them to get the recalibrated output feature map:

[0098]

[0099] in, It represents the relative importance of the kth channel. The recalibration result will strengthen important channels and suppress unimportant channels.

[0100] The first scSE module is used to adaptively recalibrate space and channels at the same time, thereby improving the performance of convolutional neural networks in image processing tasks. This module can introduce an importance weighting mechanism in different spatial regions and channel dimensions, so that the network can focus more accurately on meaningful features, thereby improving the model's discrimination ability and performance.

[0101] Specifically, the scSE module uses a combination of spatial excitation and channel excitation mechanisms to first compress the spatial and channel information in the input feature map through global pooling and convolution operations, and then generates importance weights in the spatial and channel dimensions through the Sigmoid activation function. These weights are used to recalibrate the input feature map, which can not only strengthen the focus on useful channels in the channel dimension, but also enhance the significance of key areas in the spatial dimension and suppress irrelevant features. In this way, the scSE module can effectively enhance the model's ability to learn fine-grained features, especially in complex scenarios.

[0102] Furthermore, if Figure 2 , Figure 3 As shown, the dual segmentation heads include: a cell probability segmentation head and a gradient flow tracking segmentation head; the formula for calculating the loss function using the results of the two segmentation heads is as follows:

[0103] L(x,y)=L BCE (h CR (f θ (x),y cell )+λ·L MSE (h CD (f θ (x),y gradient )

[0104] The cell probability segmentation head comprises: a first convolution layer, a first activation function layer, a first batch of normalization layers, a second convolution layer and a first upsampling layer connected in sequence;

[0105] The first convolutional layer receives the final feature map from the decoder, extracts the feature information of the cell through a convolution operation, and generates a binary mask of the cell.

[0106] The first activation function layer uses the Mish activation function to introduce nonlinearity into the model and enhance the learning ability of cell features.

[0107] The first batch of normalization layers performs batch normalization on the convolution outputs to stabilize the training process and accelerate convergence.

[0108] The second convolutional layer converts the feature map into a final binary mask through convolution, which represents the boundary of the cell.

[0109] The first upsampling layer uses bilinear interpolation for upsampling.

[0110] The cell probability segmentation head is used to generate a binary mask of the cell, identify the cell boundary in the image, and ensure accurate segmentation of the cell position.

[0111] Furthermore, the gradient flow tracking segmentation head includes: a third convolution layer, a second activation function layer, a second batch normalization layer, a fourth convolution layer and a second upsampling layer connected in sequence;

[0112] The third convolutional layer receives the final feature map from the decoder and extracts the gradient flow information of the cell boundary through the convolution operation. The function of the convolutional layer is to extract detailed information of the cell boundary from the feature map.

[0113] The second activation function layer uses the Mish activation function to help the network learn nonlinear features and enhance the representation ability of cell boundaries.

[0114] The second batch normalization layer normalizes the convolution output through batch normalization, making the training more stable and accelerating convergence.

[0115] The fourth convolutional layer further processes the convolution output and converts it into a cell gradient map, outputting the gradient flow information representing the cell boundary, usually in two channels, representing the gradients in the horizontal and vertical directions respectively.

[0116] In the second upsampling layer, if the spatial resolution of the original image needs to be restored, bilinear interpolation is used for upsampling to restore the feature map to a higher resolution.

[0117] The gradient flow tracking segmentation head is used to accurately distinguish the boundaries of adjacent cell regions through a gradient flow tracking method, ensure the accuracy of adjacent cell segmentation, and optimize the processing of inter-cell boundaries.

[0118] The present invention aims to improve the accuracy and efficiency of medical image segmentation. The method first performs data preprocessing on medical images of various formats, uniformly converts the images into RGB images, and improves the data quality and diversity through spatial transformation and intensity transformation. Then, a lightweight segmentation model (Seg-LightUNet) is established with U-Net as the backbone network. Specifically, we use SegFormer as the encoder and retain the jump connection mechanism of the traditional U-Net. SegFormer enhances the capture of image details and global features through multi-scale feature extraction and self-attention mechanism. In the decoder, the ConcurrentSpatial and Channel Squeeze and Channel Excitation (scSE) attention mechanism is introduced to improve the feature expression ability through spatial attention and channel attention modules. The model adopts a dual segmentation head structure, combined with a pseudo diffusion field method and a gradient flow tracking method, to further optimize the segmentation results. In addition, the model uses deep separable convolution and mixed precision training to reduce the computational complexity and video memory occupancy, and is suitable for efficient processing of large-scale medical image datasets.

[0119] The image is segmented through a lightweight U-Net model based on the SegFormer encoder. SegFormer is used as the encoder of the U-Net network framework, its skip connection mechanism is retained, and the SCSE attention mechanism is introduced in the decoder. The preprocessed image is input for target area segmentation.

[0120] Use pre-trained models for transfer learning, adopt the pre-trained weights provided by the MEDIAR project team, and use transfer learning to further enrich the semantic information of the model and enhance the generalization ability of the model.

[0121] Optimize model performance through mixed precision training, and adopt a mixed precision training strategy. Specifically, the computing power of 16-bit floating point numbers (FP16) and 32-bit floating point numbers (FP32) is comprehensively utilized. By reducing some non-critical operations in the model to FP16 precision processing, the use of video memory is significantly optimized. This strategy allows larger batch training under the same hardware conditions, thereby significantly accelerating the training process. At the same time, the calculation operations that have a critical impact on the segmentation results are still maintained at FP32 precision to ensure the prediction accuracy and stability of the model details. This flexible application of mixed precision enables the model to reduce video memory usage and accelerate calculations while keeping the hardware resources unchanged, so that the model can achieve a good balance between training accuracy and efficiency. Thereby optimizing the segmentation results and improving processing efficiency.

[0122] The segmentation results are further optimized through a dual segmentation head structure, which uses a dual segmentation head structure of cell probability and gradient flow tracking method and a binary cross entropy loss function to perform detailed segmentation of the target area and identify adjacent areas.

[0123] Specifically, the cell probability provides a probability map of each pixel belonging to the cell area through the recognition head, ensuring the accurate positioning of the cell boundary and contour. This method performs cell segmentation through the binary mask generated by the cell recognition head, accurately distinguishing cells from the background area, and forming a clearer mark on the cell boundary to avoid excessive or missed segmentation.

[0124] At the same time, the differentiation head uses the gradient flow tracking method to further improve the segmentation accuracy, especially when the cell boundaries are blurred or the adjacent cells have similar morphology. By calculating the gradient flow map of the image, the differentiation head can track the details of the cell edges and optimize the segmentation of adjacent cell areas, especially to solve the problem of boundary contact or overlap between cells. The gradient flow information helps to accurately identify the topological structure between cells and ensure that the boundaries of adjacent cells are accurately distinguished during the segmentation process.

[0125] In terms of loss function, combined with the binary cross entropy loss (L BCE ) and mean square error loss (L MSE). The cell recognition head optimizes the segmentation effect of cell boundaries through binary cross entropy loss, while the differentiation head uses mean square error loss to optimize the accuracy of the gradient flow map. By introducing the weight parameter λ, the influence of these two loss functions is balanced during the training process to ensure that the tasks of cell recognition and adjacent region segmentation are reasonably optimized.

[0126] The trained model is deployed as a system for clinical use to generate cell segmentation maps and CAM (Class Activation Mapping) attention visualization maps to enhance the clinical interpretability of the model.

[0127] The encoder adopts a SegFormer structure, which is composed of a stack of multiple layers of Transformer encoders. The encoder obtains rich image context information through multi-scale feature extraction modules (MSPP) such as Overlap Patch Embeddings modules and MiT Blocks. Each layer contains an efficient self-attention module (EfficientSelf-Attention) and a feedforward neural network (Mix-FFN) based on the self-attention mechanism, so as to accurately capture the details and global features of medical images. These modules effectively capture long-range dependencies through the self-attention mechanism, and optimize the feature processing method by utilizing the global and local feature extraction capabilities of Mix-FFN. At the same time, the encoder retains the jump connection mechanism of the traditional U-Net model, and fuses the low-level features of the encoder with the high-level features of the decoder through jump connections to enhance the ability to recover spatial detail information and improve segmentation accuracy.

[0128] The decoder introduces the Concurrent Spatial and Channel Squeeze and Channel Excitation (scSE) attention mechanism module, which includes a spatial attention module (Channel Squeeze and SpatialExcitation, sSE) and a channel attention module (Spatial Squeeze and Channel Excitation, cSE).

[0129] The spatial attention module generates a spatial weight map through 1×1 convolution to enhance the attention to important spatial locations. Specifically, the spatial attention module first divides the input feature map U into multiple small blocks u i,j , each small block contains all channel features of a specific spatial position. Then, through a 1×1 convolution operation q=W sq ×U compresses the channel and generates a projection tensor q. Each element q of this tensor i,jRepresents the linear combination of all channel features at the spatial position (i, j). Then, this projection tensor q is input to the Sigmoid activation function σ(.) to generate a scalar σ(q i,j ), which represents the relative importance of the spatial position (i, j). Finally, by performing an element-by-element product of the generated spatial weight map with the original feature map, the module can focus on important spatial regions, improve the model's ability to perceive key information, and ignore irrelevant positions.

[0130] The channel attention module models the importance of each channel through global average pooling and fully connected layers to further enhance the feature expression capability. In this module, the input feature map U is compressed through global average pooling, and the spatial information of each channel is aggregated into a scalar to generate a vector z∈R 1×1×C , where each element z k Represents the global information of the kth channel. Next, two fully connected layers are used to transform the vector and generate the weight of each channel through the Sigmoid activation function. Indicates the importance of the channel. Finally, these weights are multiplied channel by channel with each channel of the input feature map to strengthen important channel features while suppressing unimportant channel features and optimize the expressiveness of the feature map.

[0131] In the decoder, the scSE module is applied after each up-sampling layer to improve segmentation accuracy. Specifically, by combining spatial excitation and channel excitation, the scSE module optimizes the spatial and channel information in the decoder feature map after each up-sampling step, so that the model can accurately restore the spatial details of the image while focusing on key spatial and channel features, further improving the accuracy and effect of segmentation.

[0132] The model uses the depthwise separable convolution technique in the design of both the encoder and the decoder, decomposing the standard convolution into depthwise convolution and pointwise convolution to reduce the number of parameters and computational complexity while maintaining the model's efficient feature extraction capabilities. Depthwise separable convolution is mainly used in the following two key locations:

[0133] 1) Convolution operation after upsampling layer: In the decoder, the convolution operation after each upsampling layer is replaced by a depth-wise separable convolution to restore the resolution of the feature map and optimize the detail information.

[0134] 2) Feature fusion convolution after skip connection: In the skip connection mechanism between the encoder and decoder, low-level features and high-level features need to be fused through convolution. Here, depthwise separable convolution is used instead of standard convolution, which can efficiently fuse feature information at different levels.

[0135] In addition, the Mixed Precision Training strategy combines 16-bit floating-point and 32-bit floating-point calculations. By using both precisions in the model, the memory usage can be effectively reduced and the training efficiency can be improved. Without affecting the segmentation accuracy of the model, some weights and gradients in the calculation process are stored as 16-bit floating-point numbers to reduce the memory requirements, allowing the model to process larger batches of data or run on hardware with limited memory resources. At the same time, the key calculation parts still maintain 32-bit precision to ensure the accuracy and stability of the model. Mixed precision training can not only increase the training speed, but also optimize the segmentation results, so that the model can generate high-quality segmented images more quickly in practical applications. It effectively reduces the memory usage and increases the training speed, which is especially suitable for processing the computing needs of large-scale medical image data sets.

[0136] The model adopts a dual segmentation head structure, in which the cell probability segmentation head generates a cell probability map through a convolution layer and a nonlinear activation function (such as ReLU) to accurately segment the boundaries of cells. Figure 2 As shown in the figure, the gradient flow tracking segmentation head calculates the gradient flow features and uses convolutional layers, batch normalization layers, and bilinear interpolation to identify and separate adjacent cells. The structure is as follows: Figure 3 The output results of the two segmentation heads are optimized by calculating their difference and setting the error threshold to optimize the segmentation accuracy.

[0137] Data preprocessing: Medical images in various formats are preprocessed and uniformly converted into RGB images. A series of spatial and intensity transformations (such as random scaling, cropping, flipping, rotation, Gaussian noise, contrast adjustment, smoothing, histogram equalization, and sharpening) are performed to enhance data quality and diversity.

[0138] Image segmentation based on lightweight U-Net model: This model uses SegFormer as the encoder and uses depthwise separable convolution instead of ordinary convolution, which significantly reduces the number of model parameters and computational complexity. The model retains the traditional U-Net skip connection mechanism and introduces the SCSE attention mechanism in the decoder to enhance feature expression capabilities.

[0139] Use pre-trained models for transfer learning: During the model training process, introduce a pre-training mechanism and use pre-trained weights for transfer learning to further enrich the semantic information of the model and enhance the generalization ability of the model.

[0140] Mixed-precision training: A mixed-precision training strategy is used, combining the calculation of 16-bit floating-point numbers and 32-bit floating-point numbers, which effectively reduces the memory usage, improves the training speed, and optimizes the segmentation results and processing efficiency of the model.

[0141] The segmentation results are further optimized through the dual segmentation head structure: the dual segmentation head structure of the cell probability and gradient flow tracking method and the binary cross entropy loss function are used to perform detailed segmentation of the target area and identification of adjacent areas to improve the segmentation accuracy. The output results are as follows Figure 8 shown.

[0142] Model deployment and inference: Deploy the trained model as a system for clinical use. In the inference stage, the model not only generates a cell segmentation map, but also generates a class activation mapping (CAM) attention visualization map to enhance the clinical interpretability of the model and help doctors better understand and interpret the model's prediction results. The core idea of ​​CAM is to combine the feature map (activation maps) of the last convolutional layer of the neural network with the weights to generate a heat map to indicate the contribution of different regions to the final classification.

[0143] Furthermore, the method further includes: generating a Class Activation Mapping (CAM) attention visualization diagram to enhance the clinical interpretability of the model; the specific implementation is as follows:

[0144] (31): Obtaining feature maps of the last convolutional layer: The output of the last convolutional layer of the trained medical image segmentation model is a set of feature maps that capture the high-level features of the image. When the image is forward propagated, the network will generate these feature maps, which contain the visual information of each region in the input image.

[0145] (32): Apply global average pooling to each feature map: In CAM, global average pooling (GAP) is first performed on each convolutional feature map. Through the GAP operation, each feature map of the network is converted into a single number that represents the overall information of the feature map. For each feature map A k (Assuming there are K feature maps), the result of global average pooling z k is the average value of the pixel value of the feature map:

[0146]

[0147] Among them, H×W is the spatial size of the feature map, It is the feature map A k The value at position (i,j) in .

[0148] (33): Weighted synthetic feature map: The weights of the last fully connected layer in the network (usually the weights in the classification task) are combined with each feature map A k The global average pooling result z k Multiply them together to weight the feature map. Assuming the output category is C, for each category c, the final CAM image is calculated as follows:

[0149]

[0150] in, is the weight of the last fully connected layer, representing the feature map A k The contribution of to category c.

[0151] (34): Generate a heat map for visualization: Remap the weighted feature map back to the spatial domain of the input image to generate a heat map. Each value in the heat map represents the importance of the position in the predicted category.

[0152] (35): Overlay heatmap and original image: Overlay the CAM heatmap onto the original image to more intuitively show the areas that the model focuses on when processing the image.

[0153] In this way, doctors or other users can observe which image regions have a high influence on the model's final prediction.

[0154] Collect open-source multimodal cell segmentation datasets: In order to improve the model's segmentation capabilities under different microscopic image modes, the model training dataset collects multiple open-source multimodal cell segmentation datasets. These datasets cover a variety of microscopic imaging techniques, including fluorescence microscopy, confocal laser scanning microscopy (CLSM), transmission electron microscopy (TEM), scanning electron microscopy (SEM), phase contrast microscopy, confocal phase contrast microscopy (DIC), fusion protein expression microscopy, super-resolution microscopy, etc. The multimodal nature of these datasets provides the model with rich training samples, covering different contrasts, resolutions, cell structure features, etc. This diversity of input data can help the model accurately segment cells under different imaging conditions, improving the model's generalization ability and applicability in practical applications.

[0155] After all images in the dataset are uniformly converted to RGB images, the image boundaries are expanded to the target size using padding operations, and then randomly cropped to a final size of 512x512. On this basis, a series of data augmentation operations are applied to the cropped images, including random scaling, flipping, rotation, Gaussian noise addition, contrast adjustment, smoothing, histogram equalization, and sharpening. After these processing, the images are directly input into the model for training.

[0156] The encoder of the lightweight U-Net optimization model adopts the MiT-B5 structure, which consists of four levels of MiT Block.

[0157] The encoder first divides the input 512×512 RGB image into overlapping image patches through Overlap Patch Embeddings and maps them to a feature space of 64 channels.

[0158] Next, the feature map passes through four MiT Blocks in turn. Each Block uses the self-attention mechanism and the multi-head attention layer to process the feature map, so that its spatial size gradually decreases, while the number of channels gradually increases: 64, 128, 320 and 512 respectively.

[0159] The first layer of the decoder is the Position-wise Attention Block (PAB), which receives the feature map from the last layer of the encoder, MiT Block, and enhances the important spatial information in the feature map through the position attention mechanism while keeping the resolution of the feature map unchanged.

[0160] In the subsequent decoder layers, the spatial size of the feature map is gradually expanded through upsampling operations, and the feature maps of the corresponding levels of the encoder are fused with the feature maps in the decoder using the skip connection mechanism.

[0161] The first upsampling layer receives the output of PAB, and after being processed by the first upsampling layer and the first scSE module, it is fused with the feature map output by the third MiT Block of the encoder.

[0162] The second upsampling layer receives the output of the first adder, and after being processed by the second upsampling layer and the second scSE module, it is fused with the feature map output by the second MiT Block of the encoder.

[0163] The third upsampling layer receives the output of the second adder, and after being processed by the third upsampling layer and the third scSE module, it is fused with the feature map output by the first MiT Block of the encoder.

[0164] The scSE module is introduced after each decoder level to further enhance the expression of feature maps through spatial attention and channel attention mechanisms, ensuring that important semantic information is retained while restoring the resolution.

[0165] Finally, the decoder outputs a feature map of size 512×512, which is used to generate segmentation results.

[0166] In order to achieve a lightweight design of the model, the model uses depthwise separable convolutional layers to replace ordinary convolutional layers in the decoder and segmentation head. Specifically, the convolutional layer of the decoder: In each layer of the decoder, especially in the convolution processing after the upsampling operation, depthwise separable convolutional layers are used. These convolutional layers are responsible for restoring spatial resolution and extracting detail features. By introducing depthwise separable convolutions, the computational complexity and the number of parameters are effectively reduced. Convolutional layers in the segmentation head: Depthwise separable convolutional layers are also used in the cell recognition head (Recognition Head) and the cell distinction head (Distinction Head). In the process of generating binary masks and gradient flow graphs, these segmentation heads improve the processing speed and efficiency of the model in the inference stage through the application of depthwise separable convolutions.

[0167] A pre-training mechanism has been added to the training module, and pre-training parameters can be easily activated and managed through a unified configuration file. In terms of pre-training models, the pre-training weights provided by the original MEDIAR project team are used for transfer learning. These pre-training weights contain rich feature representations, especially basic knowledge in the field of medical images, which can provide effective initial weights for the model, thereby accelerating the training process and reducing the model convergence time. At the same time, it enhances the model's ability to capture complex areas such as cell structures, improves segmentation accuracy and generalization ability, and enables it to perform well in medical images of different modalities.

[0168] A mixed precision training strategy is used to optimize the model training process. Mixed precision training combines the calculation of 16-bit and 32-bit floating point numbers. By using both precisions in the model, the memory usage can be effectively reduced and the training efficiency can be improved. Without affecting the segmentation accuracy of the model, some weights and gradients in the calculation process are stored as 16-bit floating point numbers to reduce the memory requirements, allowing the model to process larger batches of data or run on hardware with limited memory resources. At the same time, the key calculation parts still maintain 32-bit precision to ensure the accuracy and stability of the model. Mixed precision training can not only improve the training speed, but also optimize the segmentation results, so that the model can generate high-quality segmented images more quickly in practical applications.

[0169] The decoder output feature map is processed by a dual segmentation head structure. The model uses two segmentation heads to achieve a more refined segmentation task:

[0170] Recognition Head: Receives the final feature map from the decoder and generates a binary mask of the cell after convolution operation, which is used to identify the cell boundaries and contours in the image.

[0171] Distinction Head: Another segmentation head uses the gradient flow tracking method provided by Cellpose. It receives the final feature map of the decoder and generates a gradient map by calculating the gradient flow information to help the model better identify and distinguish adjacent cell regions. This method uses the topological structure information of the cell to ensure that the boundaries of adjacent cells are accurately processed during the segmentation process, improving the overall accuracy and reliability of the segmentation.

[0172] In terms of loss function, the model uses a combined loss function to optimize the segmentation performance. The combined loss function combines the binary cross-entropy loss (L BCE ) and mean square error loss (mean square error loss, L MSE ) to balance the optimization requirements of different tasks. The calculation formula is as follows:

[0173] L(x,y)=L BCE (h CR (f θ (x),y cell )+λ·L MSE (h CD (f θ (x),y gradient )

[0174]

[0175] Among them, L BCE The loss calculation for the cell recognition head is performed by comparing the cell binary mask h generated by the model CR (f θ (x),y cell ) and the true label y cell The cross entropy between them is used to guide model learning.

[0176] L MSE It is used for cell differentiation head (DistinctionHead), which generates the gradient flow graph h by calculating the model CD (f θ (x),y gradient ) and the true gradient flow y gradient The mean square error between them helps the model to more accurately identify and distinguish adjacent cell regions.

[0177] True gradient flow y gradient The generation method is as follows: First, based on the binary mask of the cell, the cell boundaries are identified by calculating the neighborhood relationship of each pixel in the mask. Then, by expanding and iterating these boundaries, the gradient information of each pixel is generated, which represents the relative change of the pixel and the adjacent cells. This gradient information reflects the direction and strength of the cell boundary, and finally constitutes a complete gradient flow map (y gradient ). The weight parameter λ is used to balance the contribution of the two losses. In experiments, λ is usually set to 0.5 to ensure that the losses of cell recognition and discrimination tasks are reasonably balanced.

[0178] To adapt to clinical applications, the model is integrated into a system that can not only perform cell segmentation but also generate CAM images for interpretable analysis. In the inference stage, the system first generates cell segmentation masks and gradient flow maps through the trained model. These segmentation results can be directly used for cell identification and analysis. To enhance the clinical interpretability of the model, the system then generates CAM images through Grad-CAM technology. Specifically, in the process of generating CAM, the system captures important activation information in the last convolutional layer of the model and calculates weighted images related to the segmentation results based on these activation information. Grad-CAM generates an image that highlights the model's focus area by superimposing these weighted maps with the original input image, allowing clinicians to intuitively understand the decision basis of the model.

[0179] like Figure 3 As shown in the figure, the feature map passes through four MiT Blocks in sequence. Each MiT Block mainly includes three components: efficient self-supervised attention mechanism (Efficient Self-Attention), hybrid feedforward network (Mix-FFN), and overlap patch mapping (Overlap Patch Mapping). Using MiT Block to process the feature map, its spatial size is gradually reduced, while the number of channels is gradually increased: 64, 128, 320 and 512 respectively. At the same time, the model's understanding of the spatial structure of the image is strengthened, which helps to retain more edge information and context information, thereby improving the performance of segmentation or classification.

[0180] Furthermore, Efficient Self-Attention is a method to optimize the self-attention mechanism, aiming to reduce computational and memory overhead, especially when processing long sequences or large-scale data. The computational complexity of the traditional self-attention mechanism is O(N 2), where N is the length of the input sequence, which makes the computation and memory consumption very high when processing long sequences or large-sized data. The self-supervised attention mechanism can reduce the amount of computation through methods such as sparse attention, low-rank approximation, or decomposed attention. For example, it only calculates the attention between local or adjacent elements, or uses matrix decomposition techniques to simplify the calculation. These optimizations enable the self-supervised attention mechanism to significantly improve processing efficiency while ensuring expressiveness, especially in large-scale tasks.

[0181] Mixed feedforward network (Mix-FFN) is a design method that combines multiple feedforward network structures to improve the expressiveness and efficiency of the model through diversified layers and operations. Unlike traditional feedforward networks that only use a single type of layer, Mix-FFN captures multi-scale features and optimizes information flow by fusing different layers, such as standard fully connected layers, convolutional layers, and self-attention layers. This design method is particularly suitable for processing complex tasks such as large-scale data in natural language processing and computer vision. Through this hybrid approach, Mix-FFN can enhance the model's feature learning ability while maintaining low computational complexity, improving the model's performance and generalization ability.

[0182] Overlap Patch Mapping is an image processing technique that reduces boundary effects and improves the quality of image reconstruction or segmentation by segmenting an image into multiple small patches with overlapping areas. Through overlapping areas, the model can better capture local and global features, especially image edges and details, and is often used in image stitching, denoising, and segmentation tasks.

[0183] Embodiment 2: This embodiment provides a medical image segmentation system based on multimodal data, including: an acquisition module, which is configured to: acquire a medical image to be segmented; a segmentation module, which is configured to: input the medical image to be segmented into a trained medical image segmentation model to obtain a medical image segmentation result; wherein the trained medical image segmentation model includes: connecting an encoder and a decoder in sequence, the encoder extracts multi-scale features of the medical image to be segmented, and uses a self-attention mechanism to enhance the details of the extracted features; the decoder improves the feature expression capability through a spatial attention mechanism and a channel attention mechanism; the decoder uses a dual segmentation head result to realize cell recognition and cell differentiation. The above is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A medical image segmentation method based on multimodal data, characterized in that: include: Acquire a medical image to be segmented; Inputting the medical image to be segmented into the trained medical image segmentation model to obtain the medical image segmentation result; Among them, the trained medical image segmentation model includes: an encoder and a decoder connected in sequence, the encoder performs multi-scale feature extraction on the medical image to be segmented, and uses a self-attention mechanism to enhance the details of the extracted features; the decoder improves the feature expression ability through a spatial attention mechanism and a channel attention mechanism; the decoder uses a dual segmentation head result to realize cell recognition and cell differentiation.

2. The medical image segmentation method based on multimodal data as claimed in claim 1, characterized in that: Inputting the medical image to be segmented into the trained medical image segmentation model to obtain the medical image segmentation result; The training process of the trained medical image segmentation model includes: Constructing a training set, wherein the training set is medical images with known image segmentation results; Input the training set into the medical image segmentation model to train the segmentation model. When the loss function value of the model no longer decreases or the number of iterations reaches a set number, stop the training to obtain the trained medical image segmentation model. The loss function of the model, L(x,y), is: L(x,y)=L BCE (h CR (f θ (x),y cell )+λ·L MSE (h CD (f θ (x),y gtadient ); 3. The medical image segmentation method based on multimodal data as claimed in claim 1, characterized in that: The encoder comprises: an Overlap Patch Embeddings module, a first MiT Block module, a second MiT Block module, a third MiT Block module and a fourth MiT Block module connected in sequence; the input end of the Overlap Patch Embeddings module is the input end of the encoder; the output end of the fourth MiT Block module is the output end of the encoder; The decoder comprises: a PAB Blocks module, a first upsampling layer, a first scSE module, a first adder, a second upsampling layer, a second scSE module, a second adder, a third upsampling layer, a third scSE module, a third adder and a double segmentation head connected in sequence; the input end of the PAB Blocks module is the input end of the decoder; the output end of the double segmentation head is the output end of the decoder; Among them, the output end of the fourth MiT Block module is connected to the input end of the PAB Blocks module; the output end of the third MiT Block module is connected to the input end of the first adder; the output end of the second MiT Block module is connected to the input end of the second adder; the output end of the first MiT Block module is connected to the input end of the third adder.

4. The medical image segmentation method based on multimodal data as claimed in claim 3, characterized in that: The internal structures of the first MiTBlock module, the second MiT Block module, the third MiT Block module and the fourth MiT Block module are consistent. The first MiT Block module includes: The first attention module, the second attention module, the third attention module, ... the Nth attention module and the Overlap Patch Merging module are connected in sequence; Among them, the internal structures of the first attention module, the second attention module, the third attention module...the Nth attention module are consistent, and the first attention module includes: an Efficient self-Attention module and a Mix-FFN module connected in sequence; The Efficient self-Attention module includes: performing linear transformation on input features to obtain query, key and value matrices respectively, and when calculating Attention, firstly calculating the similarity between query and key by dot product, then calculating the similarity between query and key by softmax activation function, and generating attention weight, performing element-by-element multiplication operation, applying the weight to value, and finally obtaining the final output by addition or concatenation; The Overlap Patch Merging module includes: first, dividing the original image into overlapping blocks, and there are overlapping areas between adjacent blocks; the size of the block is a set size; then, feature extraction is performed on each image block through a convolutional neural network; then, the image blocks in the overlapping area are merged, and the merging method includes taking an average value and a weighted average value; finally, the processed image blocks are merged into a complete image, which can be achieved by copying the pixel values ​​of the blocks to corresponding positions.

5. The medical image segmentation method based on multimodal data as claimed in claim 3, characterized in that: The PABBlocks module includes: First, the query, key, and value feature maps are generated through convolution operations; Then, the similarity between the query and the key is calculated to generate the attention weight matrix; Then, the attention weight matrix is ​​weighted and summed with the value feature map to obtain the enhanced feature map; Finally, the enhanced feature map is fused with the original feature map to output the final feature map.

6. The medical image segmentation method based on multimodal data as claimed in claim 5, characterized in that: The PABBlocks module includes: an input end of the PAB Blocks module; the input end of the PAB Blocks module is respectively connected to the input end of the first convolutional layer, the input end of the second convolutional layer and the input end of the third convolutional layer; the output end of the first convolutional layer and the output end of the second convolutional layer are both connected to the input end of the first multiplier; the output end of the first multiplier is connected to the input end of the activation function layer, and the output end of the activation function layer and the output end of the third convolutional layer are both connected to the input end of the second multiplier; the output end of the second multiplier is connected to the input end of the fourth adder; the output end of the fourth adder is connected to the input end of the fourth convolutional layer; the output end of the fourth convolutional layer is the output end of the PAB Blocks module; The PAB Blocks module is used to enhance spatial attention. It calculates the relative importance of different positions in the image through the self-attention mechanism, thereby enhancing the model's attention to important areas.

7. The medical image segmentation method based on multimodal data as claimed in claim 3, characterized in that: The internal structures of the first scSE module, the second scSE module and the third scSE module are consistent, and the first scSE module includes: a parallel spatial attention module and a channel attention module; The spatial attention module adopts channel compression and spatial excitation mechanism; the spatial attention module achieves attention to important spatial areas by adjusting the weights of the spatial dimensions of the feature map; its working process is as follows: (11): Input tensor slicing: The input feature map U is split into U = [u 1,1 ,u 1,2 ,…,u i,j ,…,u H,W ], where u i,j ∈R 1×1×C ,u i,j Represents all channel features at the spatial position (i, j); (12): Channel compression: compress the channel dimension of the input tensor U through convolution operation q = W sq ×U; where W sq ∈R 1×1×C×1 , generate the projection tensor q∈R H×W , each element q of the tensor q i,j Represents the linear combination of all channels at spatial position (i, j); (13): Spatial excitation: The projection tensor q is input into the Sigmoid activation function σ(.) to obtain the scalar value σ(q i,j ), which is used to recalibrate the spatial excitation of the input feature U; (14): Feature recalibration: Generate a new feature map U sSE , the formula is as follows: U sSE =F sSE (U)=[σ(q 1,1 )u 1,1 ,…,σ(q i,j )u i,j ,…,σ(q H,W )u H,W ] Among them, σ(q i,j ) represents the relative importance of a specific spatial position (i, j).

8. The medical image segmentation method based on multimodal data as claimed in claim 7, characterized in that: The channel attention module includes: space compression and channel excitation mechanism. The space compression and channel excitation mechanism is to strengthen important channel features and suppress unimportant channels by adjusting the weight of each channel on the feature map. The working process is as follows: (21): Input feature map: Input feature map U∈R H×W×C Contains C channels, each channel u i ∈R H×W Represents the spatial characteristics of the image; (22): Spatial Squeeze: Through the global average pooling operation, the spatial information of each channel is compressed into a scalar. For the kth channel, the pooling operation is as follows: The resulting vector z∈R 1×1×C Contains global information for each channel; (23): Channel-dependent encoding: The pooled vector z is transformed through two fully connected layers; first, a linear transformation is performed: Where W1∈R C×C / 2 ,W2∈R C / 2×C is the weight of the fully connected layer, δ(·) is the ReLU activation function; (24): Activation function and normalization: Then, the Sigmoid activation function is used to convert The values ​​are mapped to the interval [0,1]: The resulting vector Represents the importance score of each channel; (25): Channel Excitation: Finally, the input feature map U is recalibrated using the obtained excitation value, and the activation value of each channel is Will be the corresponding channel feature map u k Multiply them to get the recalibrated output feature map: in, It represents the relative importance of the kth channel. The recalibration result will strengthen important channels and suppress unimportant channels.

9. The medical image segmentation method based on multimodal data as claimed in claim 3, characterized in that: The dual segmentation heads include: a cell probability segmentation head and a gradient flow tracking segmentation head; The cell probability segmentation head comprises: a first convolution layer, a first activation function layer, a first normalization layer, a second convolution layer and a first upsampling layer connected in sequence; the first convolution layer receives the final feature map from the decoder, extracts the feature information of the cell through the convolution operation, and is used to generate a binary mask of the cell; the first activation function layer uses the Mish activation function to introduce nonlinearity into the model and enhance the learning ability of the cell features; the first normalization layer performs batch normalization on the convolution output to stabilize the training process and accelerate convergence; the second convolution layer converts the feature map into a final binary mask through convolution to represent the boundary of the cell; the first upsampling layer uses bilinear interpolation for upsampling; the cell probability segmentation head is used to generate a binary mask of the cell, identify the cell boundary in the image, and ensure the accurate segmentation of the cell position; The gradient flow tracking segmentation head comprises: a third convolution layer, a second activation function layer, a second batch normalization layer, a fourth convolution layer and a second upsampling layer connected in sequence; the third convolution layer receives the final feature map from the decoder and extracts the gradient flow information of the cell boundary through the convolution operation; the function of the convolution layer is to extract the detailed information of the cell boundary from the feature map; the second activation function layer uses the Mish activation function to help the network learn nonlinear features and enhance the representation ability of the cell boundary; the second batch normalization layer normalizes the convolution output through batch normalization; the fourth convolution layer further processes the convolution output, converts it into a cell gradient map, and outputs the gradient flow information representing the cell boundary, which is usually two channels, representing the gradients in the horizontal direction and the vertical direction respectively; the second upsampling layer, if it is necessary to restore the spatial resolution of the original image, uses bilinear interpolation for upsampling to restore the feature map to a higher resolution; the gradient flow tracking segmentation head is used to accurately distinguish the boundaries of adjacent cell areas through the gradient flow tracking method, ensure the accuracy of adjacent cell segmentation, and optimize the processing of cell boundaries.

10. A medical image segmentation system based on multimodal data, characterized in that: include: An acquisition module is configured to: acquire a medical image to be segmented; A segmentation module is configured to: input the medical image to be segmented into a trained medical image segmentation model to obtain a medical image segmentation result; Among them, the trained medical image segmentation model includes: an encoder and a decoder connected in sequence, the encoder performs multi-scale feature extraction on the medical image to be segmented, and uses a self-attention mechanism to enhance the details of the extracted features; the decoder improves the feature expression ability through a spatial attention mechanism and a channel attention mechanism; the decoder uses a dual segmentation head result to realize cell recognition and cell differentiation.

Images