Rectal cancer medical image focus segmentation method and device based on deep learning

Abstract

The invention discloses a rectal cancer medical image lesion segmentation method and device based on deep learning, and aims to solve the problem that in the prior art, only a medical image is used as a single input mode, and potential semantic association among multi-mode data is neglected, so that the segmentation effect is poor. The method comprises the following steps: acquiring a to-be-detected rectal cancer medical image focus image and corresponding clinical text information, and inputting the two images into a segmentation model for processing; the clinical text information is processed through a text multi-scale feature extraction module, and target text multi-scale features are generated; encoding the text features and the medical image by using a feature encoder, and outputting target text image fusion encoding features; inputting the fusion coding feature into a scale attention module, and outputting a target text image fusion attention feature; processing the fusion attention feature and the fusion coding feature through a feature decoder to obtain a target text image decoding feature; and finally, segmenting the decoding features by using a segmentation head, and outputting a final target segmentation result.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a method and device for rectal cancer medical image lesion segmentation based on deep learning. Background Art

[0002] Medical image segmentation technology can extract imaging features, which play an important role in assessing patient prognosis and assisting doctors in formulating treatment plans. They provide information about the structure, density, size and potentially useful biological information of the target area.

[0003] Imaging feature extraction is generally categorized into two approaches: traditional methods and deep learning. Traditional image processing-based segmentation methods can extract features such as shape, texture, and density. Deep learning methods utilize deep neural networks to automatically learn and extract more complex and advanced feature representations from input data (primarily images), thereby achieving more accurate and highly automated medical image segmentation.

[0004] Automatic medical image segmentation technology is playing an increasingly important role in intelligent healthcare. It automatically captures imaging features and generates annotations for target regions (i.e., the specific shapes of tissues or organs in medical images). Traditional image processing-based segmentation methods are gradually being replaced by more efficient automatic segmentation models based on deep learning. Deep learning methods demonstrate greater robustness and accuracy in complex scenarios.

[0005] Existing deep learning-based methods for rectal cancer lesion segmentation in medical images primarily employ convolutional neural networks (CNNs) or Transformer architectures. CNNs, with their local perception, effectively extract local features of lesions, but they are limited in modeling long-range dependencies. While Transformers can overcome this limitation by capturing global contextual information through a self-attention mechanism, they significantly increase computational complexity and resource consumption. Furthermore, most current models use only medical images as a single input modality, failing to fully integrate auxiliary information such as clinical text reports and neglecting the underlying semantic connections between multimodal data, resulting in suboptimal segmentation results. Summary of the Invention

[0006] The present invention provides a method and device for rectal cancer medical image lesion segmentation based on deep learning, which is used to solve the technical problem that the existing rectal cancer medical image lesion segmentation method based on deep learning only uses medical images as a single input modality, fails to fully integrate auxiliary information such as clinical text reports, ignores the potential semantic associations between multimodal data, and leads to poor segmentation effect.

[0007] The first aspect of the present invention provides a method for segmenting rectal cancer medical image lesions based on deep learning, comprising:

[0008] Acquire a rectal cancer medical imaging lesion image to be detected and clinical text information corresponding to the rectal cancer medical imaging lesion image to be detected;

[0009] Inputting the rectal cancer medical imaging lesion image to be tested and the clinical text information into a rectal cancer medical imaging lesion segmentation model based on deep learning, wherein the rectal cancer medical imaging lesion segmentation model based on deep learning includes a text multi-scale feature extraction module, a feature encoder, a feature decoder, a scale attention module and a segmentation head;

[0010] Using the text multi-scale feature extraction module to extract text multi-scale features from the clinical text information to generate target text multi-scale features;

[0011] Encoding the target text multi-scale features and the rectal cancer medical imaging lesion image to be detected by the feature encoder, and outputting the target text image fusion coding features;

[0012] Taking the target text image fusion coding feature as the input of the scale attention module, and outputting the target text image fusion attention feature;

[0013] Inputting the target text image fusion attention feature and the target text image fusion encoding feature into the feature decoder for decoding, and outputting the target text image decoding feature;

[0014] The segmentation head is used to segment the target text image decoding features and output a target segmentation result.

[0015] Optionally, the text multi-scale feature extraction module includes a text encoder, a first multi-layer perceptron, a first state space model, a second multi-layer perceptron, a second state space model, a third multi-layer perceptron, a third state space model, a fourth multi-layer perceptron, and a fourth state space model; the target text multi-scale features include first text multi-scale features, second text multi-scale features, third text multi-scale features, and fourth text multi-scale features; and using the text multi-scale feature extraction module to extract text multi-scale features from the clinical text information to generate target text multi-scale features includes:

[0016] Using the text encoder to perform text encoding on the clinical text information, and outputting clinical text encoding features;

[0017] Using the clinical text coding features as inputs of the first multi-layer perceptron, the second multi-layer perceptron, the third multi-layer perceptron, and the fourth multi-layer perceptron, respectively, and outputting first text-perceptual features, second text-perceptual features, third text-perceptual features, and fourth text-perceptual features;

[0018] Performing spatial modeling on the first text perception feature by using the first state space model to generate a first text state feature;

[0019] Performing spatial modeling on the second text perception feature by using the second state space model to generate a second text state feature;

[0020] Performing spatial modeling on the third text perception feature by using the third state space model to generate a third text state feature;

[0021] Performing spatial modeling on the fourth text perception feature by using the fourth state space model to generate a fourth text state feature;

[0022] The first text state feature, the second text state feature, the third text state feature, and the fourth text state feature are respectively broadcast pixel by pixel to generate a first text multi-scale feature, a second text multi-scale feature, a third text multi-scale feature, and a fourth text multi-scale feature.

[0023] Optionally, the feature encoder includes a first multimodal feature extraction and fusion module, a second multimodal feature extraction and fusion module, a third multimodal feature extraction and fusion module, and a fourth multimodal feature extraction and fusion module; the target text image fusion coding feature includes a first text image fusion coding feature, a second text image fusion coding feature, a third text image fusion coding feature, and a fourth text image fusion coding feature; encoding the target text multi-scale feature and the rectal cancer medical imaging lesion image to be detected by the feature encoder and outputting the target text image fusion coding feature includes:

[0024] Using a first multimodal feature extraction and fusion module to perform multimodal feature extraction and fusion on the first text multi-scale feature and the rectal cancer medical imaging lesion image to be detected, and output a first text image fusion coding feature;

[0025] Downsampling the first text image fusion encoding feature to generate a first text image fusion downsampled feature;

[0026] Using a second multimodal feature extraction and fusion module to perform multimodal feature extraction and fusion on the second text multi-scale feature and the first text image fusion downsampling feature, and output a second text image fusion coding feature;

[0027] Downsampling the second text image fusion encoding feature to generate a second text image fusion downsampled feature;

[0028] Using a third multimodal feature extraction and fusion module to perform multimodal feature extraction and fusion on the third text multi-scale feature and the second text image fusion downsampling feature, and output a third text image fusion coding feature;

[0029] Downsampling the third text image fusion coding feature to generate a third text image fusion downsampled feature;

[0030] A fourth multimodal feature extraction and fusion module is used to perform multimodal feature extraction and fusion on the fourth text multi-scale feature and the third text image fusion downsampling feature, and output a fourth text image fusion coding feature.

[0031] Optionally, the first multimodal feature extraction and fusion module includes a visual state space feature extraction module, a multi-core spatial attention module, a 3×3 convolutional layer, a cross attention module, a multi-layer perceptron, and a channel attention module; the first multimodal feature extraction and fusion module is used to perform multimodal feature extraction and fusion on the first text multi-scale feature and the rectal cancer medical imaging lesion image to be tested, and output a first text image fusion coding feature, including:

[0032] The rectal cancer medical imaging lesion image to be tested is used as the input of the visual state space feature extraction module and the multi-core spatial attention module respectively, and the visual state space image features corresponding to the visual state space feature extraction module and the spatial attention image features corresponding to the multi-core spatial attention module are output;

[0033] performing element-by-element addition on the visual state space image feature and the spatial attention image feature, and outputting an image fusion feature;

[0034] Performing feature splicing on the image fusion feature and the first text multi-scale feature, and outputting a first image-text splicing feature;

[0035] Performing a convolution operation on the first image-text splicing feature through a 3×3 convolution layer to output an image-text convolution feature;

[0036] Using the image fusion feature and the first text multi-scale feature as inputs of a cross attention module, outputting an image-text cross feature, and performing nonlinear mapping on the image-text convolution feature to output an image-text nonlinear feature;

[0037] Performing feature splicing on the image-text nonlinear feature and the image-text cross feature to output a second image-text splicing feature, and using the second image-text splicing feature as an input of a multi-layer perceptron to output an image-text perception feature;

[0038] performing element-by-element multiplication on the image-text perception feature and the image fusion feature, and outputting an image-text multiplication feature;

[0039] The image-text multiplication feature is used as the input of the channel attention module, and the image-text channel attention feature is output. The image-text channel attention feature, the image-text perception feature, and the rectal cancer medical imaging lesion image to be tested are added element by element to output the first text-image fusion coding feature.

[0040] Optionally, the visual state space feature extraction module includes a fully connected layer, a depthwise separable convolution module, and a two-dimensional state space feature extraction module; the step of using the rectal cancer medical imaging lesion image to be detected as the input of the visual state space feature extraction module and outputting the visual state space image features corresponding to the visual state space feature extraction module includes:

[0041] performing layer normalization on the rectal cancer medical imaging lesion image to be detected, and outputting a normalized image;

[0042] Using the standardized image as the input of a fully connected layer, outputting a first fully connected feature, and performing a depthwise separable convolution on the first fully connected feature using a depthwise separable convolution module, outputting a depthwise separable convolution feature;

[0043] Performing nonlinear mapping on the depthwise separable convolution feature to output a first image nonlinear feature, and using the first image nonlinear feature as input to a two-dimensional state space feature extraction module to output a two-dimensional state space feature;

[0044] performing layer normalization on the two-dimensional state space feature to generate a normalized two-dimensional state space feature;

[0045] Using the rectal cancer medical imaging lesion image to be tested as the input of the fully connected layer, outputting a second fully connected feature, and performing nonlinear mapping on the second fully connected feature to output a second image nonlinear feature;

[0046] Performing element-by-element multiplication on the second image nonlinear feature and the standardized two-dimensional state space feature, outputting a multiplied feature, and using the multiplied feature as an input of a fully connected layer to output a third fully connected feature;

[0047] The third fully connected feature and the rectal cancer medical imaging lesion image to be detected are added element by element to output a visual state space image feature.

[0048] Optionally, the feature decoder includes a first multimodal feature extraction and fusion module, a second multimodal feature extraction and fusion module, a third multimodal feature extraction and fusion module, and a fourth multimodal feature extraction and fusion module; the target text image fusion attention feature includes a first text image fusion attention feature, a second text image fusion attention feature, a third text image fusion attention feature, and a fourth text image fusion attention feature; inputting the target text image fusion attention feature and the target text image fusion encoding feature into the feature decoder for decoding, and outputting the target text image decoded feature, includes:

[0049] Using a fourth multimodal feature extraction and fusion module to perform multimodal feature extraction and fusion on the fourth text multi-scale feature and the fourth text image fusion coding feature, and outputting a first text image decoding feature;

[0050] performing element-by-element addition on the first text image decoding feature and the fourth text image fused attention feature to generate a first text image added feature, and upsampling the first text image added feature to output a first text image fused upsampled feature;

[0051] Using a third multimodal feature extraction and fusion module to perform multimodal feature extraction and fusion on the third text multi-scale feature and the first text image fusion upsampling feature, and output a second text image decoding feature;

[0052] performing element-by-element addition on the second text image decoding feature and the third text image fused attention feature to generate a second text image added feature, and upsampling the second text image added feature to output a second text image fused upsampled feature;

[0053] Using a second multimodal feature extraction and fusion module to perform multimodal feature extraction and fusion on the second text multi-scale feature and the second text image fusion upsampling feature, and output a third text image decoding feature;

[0054] performing element-by-element addition on the third text image decoding feature and the second text image fusion attention feature to generate a third text image addition feature, and upsampling the third text image addition feature to output a third text image fusion upsampled feature;

[0055] Using a first multimodal feature extraction and fusion module to perform multimodal feature extraction and fusion on the first text multi-scale feature and the third text image fusion upsampling feature, and output a fourth text image decoding feature;

[0056] The fourth text image decoding feature and the first text image fusion attention feature are added element by element to generate a target text image decoding feature.

[0057] Optionally, the model training process of the deep learning-based rectal cancer medical imaging lesion segmentation model is specifically as follows:

[0058] Acquire a training rectal cancer medical imaging lesion image and training clinical text information corresponding to the training rectal cancer medical imaging lesion image, perform data enhancement on the training rectal cancer medical imaging lesion image, and output a target training rectal cancer medical imaging lesion image;

[0059] Inputting the target training rectal cancer medical imaging lesion image and the training clinical text information into a rectal cancer medical imaging lesion segmentation model to be trained based on deep learning, and outputting a training segmentation result and a training clinical text encoding feature;

[0060] Substituting the training segmentation result and the training clinical text coding feature into a preset loss function and taking derivatives, outputting a model gradient;

[0061] Using the model gradient to update the model parameters of the deep learning-based rectal cancer medical image lesion segmentation model to be trained, outputting an intermediate deep learning-based rectal cancer medical image lesion segmentation model, and counting the number of model updates in real time;

[0062] Determine whether the model update times reaches the preset training times;

[0063] If so, the intermediate rectal cancer medical image lesion segmentation model based on deep learning is used as the trained rectal cancer medical image lesion segmentation model based on deep learning.

[0064] The second aspect of the present invention provides a deep learning-based rectal cancer medical image lesion segmentation device, comprising:

[0065] An acquisition module, configured to acquire a rectal cancer medical imaging lesion image to be detected and clinical text information corresponding to the rectal cancer medical imaging lesion image to be detected;

[0066] An input module is used to input the rectal cancer medical imaging lesion image to be tested and the clinical text information into a rectal cancer medical imaging lesion segmentation model based on deep learning, wherein the rectal cancer medical imaging lesion segmentation model based on deep learning includes a text multi-scale feature extraction module, a feature encoder, a feature decoder, a scale attention module and a segmentation head;

[0067] An extraction module, configured to extract multi-scale features from the clinical text information using the multi-scale feature extraction module to generate multi-scale features of the target text;

[0068] An encoding module, configured to encode the target text multi-scale features and the rectal cancer medical imaging lesion image to be detected through the feature encoder, and output a target text image fusion encoding feature;

[0069] a scale module, configured to take the target text image fused coding features as input to the scale attention module and output the target text image fused attention features;

[0070] A decoding module, configured to input the target text image fusion attention feature and the target text image fusion coding feature into the feature decoder for decoding, and output a target text image decoding feature;

[0071] The segmentation module is used to segment the target text image decoding features using the segmentation head and output a target segmentation result.

[0072] A third aspect of the present invention provides a computer device comprising a memory and a processor, wherein a computer program is stored in the memory. When the computer program is executed by the processor, the processor executes the steps of the deep learning-based rectal cancer medical imaging lesion segmentation method as described in any one of the above items.

[0073] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the steps of the method for rectal cancer medical image lesion segmentation based on deep learning as described in any one of the above.

[0074] It can be seen from the above technical solutions that the present invention has the following advantages:

[0075] The above technical solution of the present invention provides a method for rectal cancer medical image lesion segmentation based on deep learning, which obtains the rectal cancer medical image lesion image to be tested and the clinical text information corresponding to the rectal cancer medical image lesion image to be tested; the rectal cancer medical image lesion image to be tested and the clinical text information are input into a rectal cancer medical image lesion segmentation model based on deep learning, and the rectal cancer medical image lesion segmentation model based on deep learning includes a text multi-scale feature extraction module, a feature encoder, a feature decoder, a scale attention module and a segmentation head; the text multi-scale feature extraction module is used to extract text multi-scale features from the clinical text information to generate target text multi-scale features; the feature encoder is used to extract the target text multi-scale features and the rectal cancer medical image lesion to be tested. The image is encoded and the target text image fusion coding features are output; the target text image fusion coding features are used as the input of the scale attention module, and the target text image fusion attention features are output; the target text image fusion attention features and the target text image fusion coding features are input into the feature decoder for decoding, and the target text image decoding features are output; the target text image decoding features are segmented using a segmentation head, and the target segmentation results are output; based on the above scheme, the present invention utilizes a rectal cancer medical image lesion segmentation model based on deep learning to process the rectal cancer medical image lesion images and clinical text information to be tested, which can fully utilize medical image data and clinical text data in combination, effectively explore the potential correlation between different modalities, and thus improve the segmentation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0077] Figure 1 A flowchart of the steps of a deep learning-based rectal cancer medical image lesion segmentation method provided in Example 1 of the present invention;

[0078] Figure 2 A schematic diagram of the structure of a rectal cancer medical imaging lesion segmentation model based on deep learning provided in Example 1 of the present invention;

[0079] Figure 3 A schematic diagram of the structure of a text multi-scale feature extraction module provided in the first embodiment of the present invention;

[0080] Figure 4 A schematic diagram of the structure of a multimodal feature extraction and fusion module provided in Example 1 of the present invention;

[0081] Figure 5 A schematic diagram of the structure of the visual state space feature extraction module provided in the first embodiment of the present invention;

[0082] Figure 6 A schematic structural diagram of a two-dimensional state space feature extraction module provided in Example 1 of the present invention;

[0083] Figure 7 A schematic diagram of the structure of the cross attention module provided in the first embodiment of the present invention;

[0084] Figure 8 This is a schematic diagram of the structure of the channel attention module provided in Example 1 of the present invention;

[0085] Figure 9 This is a schematic diagram of the structure of the scale attention module provided in Example 1 of the present invention;

[0086] Figure 10 A schematic diagram of the structure of the multi-core spatial attention module provided in Example 1 of the present invention;

[0087] Figure 11 A flowchart of the steps of the model training process of the deep learning-based rectal cancer medical imaging lesion segmentation model provided in the second embodiment of the present invention;

[0088] Figure 12 This is a structural block diagram of a deep learning-based rectal cancer medical image lesion segmentation device provided in Example 3 of the present invention.

[0089] Among them, the reference numerals Expressed as matrix multiplication, Expressed as element-by-element addition, Expressed as element-wise multiplication, Represented as feature concatenation, It is represented as activation function. DETAILED DESCRIPTION

[0090] The embodiments of the present invention provide a method and device for rectal cancer medical image lesion segmentation based on deep learning, which is used to solve the technical problem that the existing rectal cancer medical image lesion segmentation method based on deep learning only uses medical images as a single input modality, fails to fully integrate auxiliary information such as clinical text reports, ignores the potential semantic associations between multimodal data, and leads to poor segmentation effect.

[0091] In order to make the purpose, features, and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0092] See also Figure 1 , Figure 1 This is a flowchart of the steps of a deep learning-based method for rectal cancer medical image lesion segmentation provided in Example 1 of the present invention.

[0093] The present invention provides a deep learning-based rectal cancer medical imaging lesion segmentation method, comprising:

[0094] Step 101: Obtain a rectal cancer medical imaging lesion image to be detected and clinical text information corresponding to the rectal cancer medical imaging lesion image to be detected.

[0095] Clinical text information includes patient information, alpha-embryonic protein levels, carcinoembryonic antigen values ​​and other data.

[0096] Step 102: Input the rectal cancer medical imaging lesion image and clinical text information to be tested into a rectal cancer medical imaging lesion segmentation model based on deep learning. The rectal cancer medical imaging lesion segmentation model based on deep learning includes a text multi-scale feature extraction module, a feature encoder, a feature decoder, a scale attention module and a segmentation head.

[0097] Please note that Figure 2 The deep learning-based rectal cancer medical image lesion segmentation model proposed in this paper consists of a text multi-scale feature extraction module, a feature encoder, a feature decoder, a scale attention module, and a segmentation head (final segmentation head). Both the feature encoder and feature decoder are composed of four identical multimodal feature extraction and fusion modules (i.e., the first, second, third, and fourth multimodal feature extraction and fusion modules).

[0098] Step 103: Use a text multi-scale feature extraction module to extract text multi-scale features from clinical text information to generate target text multi-scale features.

[0099] The text multi-scale feature extraction module includes a text encoder, a first multi-layer perceptron, a first state space model, a second multi-layer perceptron, a second state space model, a third multi-layer perceptron, a third state space model, a fourth multi-layer perceptron, and a fourth state space model.

[0100] The target text multi-scale features include the first text multi-scale features (scale is H1×W1×C1), the second text multi-scale features (scale is H2×W2×C2), the third text multi-scale features (scale is H3×W3×C3) and the fourth text multi-scale features (scale is H4×W4×C4), where H represents height, W represents width, and C represents the number of channels.

[0101] Specifically, step 103 may include the following sub-steps S31-S36:

[0102] Step S31: using a text encoder to perform text encoding on the clinical text information and output clinical text encoding features;

[0103] Step S32: using the clinical text coding features as inputs to the first multi-layer perceptron, the second multi-layer perceptron, the third multi-layer perceptron, and the fourth multi-layer perceptron, respectively, and outputting the first text perception feature, the second text perception feature, the third text perception feature, and the fourth text perception feature;

[0104] Step S33: performing spatial modeling on the first text perception feature through a first state space model to generate a first text state feature;

[0105] Step S34: spatially modeling the second text perception feature using a second state space model to generate a second text state feature;

[0106] Step S35: spatially modeling the third text perception feature using a third state space model to generate a third text state feature;

[0107] Step S36: spatially modeling the fourth text perception feature using a fourth state space model to generate a fourth text state feature;

[0108] Step S37: broadcast the first text state feature, the second text state feature, the third text state feature, and the fourth text state feature pixel by pixel to generate the first text multi-scale feature, the second text multi-scale feature, the third text multi-scale feature, and the fourth text multi-scale feature.

[0109] Please note that Figure 3The text multi-scale feature extraction module consists of a text encoder (CLIP, Contrastive Language–Image Pretraining), four multi-layer perceptrons (i.e., the first multi-layer perceptron, the second multi-layer perceptron, the third multi-layer perceptron, and the fourth multi-layer perceptron), and four state-space models (the first state-space model, the second state-space model, the third state-space model, and the fourth state-space model). The scale (number of model parameters) of each multi-layer perceptron is different, and the directional sensitivity of each state-space model is different. Those skilled in the art can set them as needed. Figure 3 H in i ×W i ×C i Indicates the height, width, and number of channels corresponding to the i-th scale.

[0110] Furthermore, in the text multi-scale feature extraction module, medical clinical text data (clinical text information) is encoded using the Clip large language pre-trained model to obtain text embeddings, i.e., clinical text encoding features. These text embeddings are processed through four different-scale text multi-scale feature extraction modules. Each module consists of a learnable multi-layer perceptron and a state-space model to improve the expressive power of features at different scales (high-level global semantics and low-level local details), enhancing the model's ability to match image and text information at different scales.

[0111] It is worth mentioning that by using the state space model, this computational component enables the model to capture global image information while keeping the number of parameters and computational cost low. The processing principle of the state space model can be expressed as:

[0112] ;

[0113] Among them, A, B, and C are learnable parameters. are input features, is the output feature, is the current hidden state (system memory), is the hidden state after the next update. A represents the state transition matrix: it describes how the system's memory evolves and determines how the previous state affects the next state. B is the input action matrix: it describes how input affects the system state and controls the relationship between input and state. C is the state readout matrix: it transforms the internal hidden state into the final output.

[0114] Step 104: Encode the target text multi-scale features and the rectal cancer medical imaging lesion image to be detected through a feature encoder, and output the target text image fusion encoding features.

[0115] The feature encoder includes a first multimodal feature extraction and fusion module, a second multimodal feature extraction and fusion module, a third multimodal feature extraction and fusion module, and a fourth multimodal feature extraction and fusion module.

[0116] The target text image fusion coding feature includes a first text image fusion coding feature, a second text image fusion coding feature, a third text image fusion coding feature, and a fourth text image fusion coding feature.

[0117] It should be noted that the main purpose of the feature encoder is to extract features at different scales from image features through layered downsampling. The encoder accepts medical images (images of colorectal cancer lesions to be detected) as image input and text features (target text multi-scale features) obtained by the text multi-scale feature extraction module as text input. It then uses the multimodal feature extraction and fusion module to fuse image and text features at different scales. It then passes these features at different scales to the scale attention module via skip connections for further feature selection before sending them to the corresponding layers of the feature decoder. This improves feature reuse, accelerates training convergence, and enhances overall model performance.

[0118] Specifically, step 104 may include the following sub-steps S41-S47:

[0119] Step S41: using a first multimodal feature extraction and fusion module to perform multimodal feature extraction and fusion on the first text multi-scale feature and the rectal cancer medical imaging lesion image to be detected, and outputting a first text image fusion coding feature;

[0120] The first multimodal feature extraction and fusion module includes a visual state space feature extraction module, a multi-core spatial attention module, a 3×3 convolutional layer, a cross attention module, a multi-layer perceptron, and a channel attention module.

[0121] Please note that Figure 4 The structure of each multimodal feature extraction and fusion module in the feature encoder and feature decoder is the same, consisting of a visual state space feature extraction module, a multi-core spatial attention module, a 3×3 convolutional layer, a cross attention module, a multi-layer perceptron, and a channel attention module; among them, Figure 4 The image features / input image in the image may refer to the rectal cancer medical imaging lesion image to be tested, the first text image fusion downsampling feature, the second text image fusion downsampling feature or the third text image fusion downsampling feature. Figure 4 The text feature in may refer to the first text multi-scale feature, the second text multi-scale feature, the third text multi-scale feature or the fourth text multi-scale feature.

[0122] Furthermore, the main purpose of the multimodal feature extraction and fusion module is to extract and fuse multimodal features (image and text), using the visual state space model based on state space modeling as the main feature extraction and calculation unit. Figure 4 As shown in the figure, taking the processing of the first multimodal feature extraction and fusion module as an example, image features are divided into three streams. The image of the rectal cancer lesion is processed through the visual state space model to extract global image features (i.e., visual state space image features). The image of the rectal cancer lesion is processed through the multi-core spatial attention module to extract local images (i.e., spatial attention image features). The image of the rectal cancer lesion is concatenated to form a residual. The global and local features are fused to produce an image fusion feature, which is then divided into three streams. The text features are divided into two streams. The image fusion feature is concatenated with the first text multi-scale feature and input into a learnable multi-layer convolution to produce fusion feature A (image-text convolution feature). The image fusion feature and the first text multi-scale feature are input into a cross-attention process to learn the image-text correlation, producing fusion feature B (image-text cross-feature). The concatenation of fusion features A and B is then fed into a multi-layer perceptron to produce image-text perceptual features. The image-text perceptual features and the image fusion feature are fused (element-wise multiplication) and then processed through channel attention to produce fusion feature E (image-text multiplication feature). Finally, the image-text channel attention features, image-text perception features, and the colorectal cancer lesion image are fused (element-by-element addition) to produce the output graph features (the first text-image fusion encoding feature). Note that the multimodal feature extraction and fusion module is performed at different scales to capture features at different resolutions.

[0123] Furthermore, step S41 may include the following sub-steps S411-S418:

[0124] Step S411: using the rectal cancer medical imaging lesion image to be tested as input to the visual state space feature extraction module and the multi-core spatial attention module, respectively, and outputting the visual state space image features corresponding to the visual state space feature extraction module and the spatial attention image features corresponding to the multi-core spatial attention module;

[0125] It should be noted that the processing principle of the multi-core spatial attention module is as follows: the input features input to the multi-core spatial attention module are subjected to maximum pooling and average pooling operations to obtain corresponding features. After the two sets of features are spliced, they are subjected to convolution operations with different kernel sizes (7×7, 5×5, 3×3), and then added and fused, and nonlinear mapping is performed using the activation function to obtain spatial attention features. Finally, the spatial attention features are combined with the original input features (element-wise multiplication) to output features adjusted by spatial attention, thereby enhancing the attention of the input features in the spatial dimension. Based on the above principles, the medical imaging lesion image of rectal cancer to be tested is input into the multi-core spatial attention module for processing to obtain spatial attention image features.

[0126] Optionally, the visual state space feature extraction module includes a fully connected layer, a depthwise separable convolution module, and a two-dimensional state space feature extraction module; the process of using the rectal cancer medical imaging lesion image to be detected as the input of the visual state space feature extraction module and outputting the visual state space image features corresponding to the visual state space feature extraction module can be performed by the following steps S1-S7:

[0127] Step S1, performing layer normalization on the rectal cancer medical imaging lesion image to be tested, and outputting a normalized image;

[0128] Step S2: Using the normalized image as the input of the fully connected layer, outputting the first fully connected feature, and performing depthwise separable convolution on the first fully connected feature using a depthwise separable convolution module, outputting the depthwise separable convolution feature;

[0129] Step S3: performing nonlinear mapping on the depthwise separable convolutional features to output a first image nonlinear feature, and using the first image nonlinear feature as input to a two-dimensional state space feature extraction module to output a two-dimensional state space feature;

[0130] Step S4: performing layer normalization on the two-dimensional state space features to generate normalized two-dimensional state space features;

[0131] Step S5: using the rectal cancer medical imaging lesion image to be detected as the input of the fully connected layer, outputting a second fully connected feature, and performing nonlinear mapping on the second fully connected feature to output a second image nonlinear feature;

[0132] Step S6: perform element-by-element multiplication on the second image nonlinear feature and the standardized two-dimensional state space feature, output the multiplied feature, and use the multiplied feature as the input of the fully connected layer to output the third fully connected feature;

[0133] Step S7: perform element-by-element addition on the third fully connected feature and the rectal cancer medical imaging lesion image to be detected, and output a visual state space image feature.

[0134] Please note that Figure 5 The visual state space feature extraction module is mainly composed of three fully connected layers, a depth-wise separable convolution module, and a two-dimensional state space feature extraction module; among them, an activation function is used in the visual state space feature extraction module to perform nonlinear mapping of features. Those skilled in the art can select the corresponding activation function as needed, and the present invention is not limited to this.

[0135] Further, see Figure 6 The two-dimensional state space feature extraction module is mainly composed of four state space models with different directional sensitivities. As for the processing principle of the two-dimensional state space feature extraction module, the features input to the two-dimensional state space feature extraction module are first transformed in different ways to obtain four feature data with different arrangements. Then, the four feature data with different arrangements are respectively input into the corresponding state space models for processing, and then added element by element to obtain the output features processed by the two-dimensional state space feature extraction module; among them, directional sensitivity refers to the difference in the model or system's ability to capture, respond to or process specific directional features in the input data.

[0136] Step S412: Add the visual state space image features and the spatial attention image features element by element, and output an image fusion feature;

[0137] Step S413: performing feature splicing on the image fusion feature and the first text multi-scale feature, and outputting a first image-text splicing feature;

[0138] Step S414: performing a convolution operation on the first image-text splicing feature through a 3×3 convolution layer to output an image-text convolution feature;

[0139] Step S415: Using the image fusion feature and the first text multi-scale feature as input to the cross attention module, outputting the image-text cross feature, and performing nonlinear mapping on the image-text convolution feature to output the image-text nonlinear feature;

[0140] Please note that Figure 7The processing principle of the cross-attention module is as follows: the text features input to the cross-attention module are used as the query (q), and the image fusion features input to the cross-attention module are used as the key (k) and value (v), respectively. Matrix multiplication is performed on q and k, and then nonlinearly mapped through an activation function. Then, matrix multiplication is performed with v. Finally, processing is performed by a multilayer perceptron (MLP) to achieve interaction and information integration between the text features and the image fusion features, thereby capturing the correlation between the two. Based on this processing principle, the cross-attention module is used to process the image fusion features and the first text multi-scale features to obtain the image-text cross features.

[0141] Step S416: performing feature splicing on the image-text nonlinear features and the image-text cross features to output a second image-text splicing feature, and using the second image-text splicing feature as input to a multi-layer perceptron to output an image-text perceptual feature;

[0142] Step S417: performing element-by-element multiplication of the image-text perception feature and the image fusion feature, and outputting the image-text multiplication feature;

[0143] Step S418: Use the image-text multiplication feature as the input of the channel attention module, output the image-text channel attention feature, and add the image-text channel attention feature, the image-text perception feature, and the rectal cancer medical imaging lesion image to be tested element by element to output the first text-image fusion coding feature.

[0144] Please note that Figure 8 Regarding the processing principle of the channel attention module, the input features of the channel attention module are respectively subjected to maximum pooling and average pooling operations to obtain corresponding features. After these features pass through a shared multi-layer perceptron, the output maximum pooling features and average pooling features are added and nonlinear mapping is performed using an activation function to obtain channel attention features*. Finally, they are element-wise multiplied with the input features to obtain the final output features, realizing attention adjustment of the input features in the channel dimension and highlighting important channel information. Based on this processing principle, the present invention uses the channel attention module to process the image-text multiplication features to obtain image-text channel attention features.

[0145] Step S42: down-sample the first text image fusion coding feature to generate a first text image fusion down-sampled feature;

[0146] Step S43: using a second multimodal feature extraction and fusion module to perform multimodal feature extraction and fusion on the second text multi-scale feature and the first text image fusion downsampling feature, and outputting a second text image fusion coding feature;

[0147] Step S44: downsample the second text image fusion coding feature to generate a second text image fusion downsampled feature;

[0148] Step S45: Using a third multimodal feature extraction and fusion module to perform multimodal feature extraction and fusion on the third text multi-scale feature and the second text image fusion downsampling feature, and output a third text image fusion coding feature;

[0149] Step S46: down-sample the third text image fusion coding feature to generate a third text image fusion down-sampled feature;

[0150] Step S47: Use a fourth multimodal feature extraction and fusion module to perform multimodal feature extraction and fusion on the fourth text multi-scale feature and the third text image fusion downsampling feature, and output a fourth text image fusion coding feature.

[0151] It should be noted that the feature processing principles of the second multimodal feature extraction and fusion module, the third multimodal feature extraction and fusion module, and the fourth multimodal feature extraction and fusion module are consistent with the feature processing principles of the above-mentioned first multimodal feature extraction and fusion module, and the present invention will not go into details.

[0152] Step 105: Use the target text image fusion coding features as the input of the scale attention module, and output the target text image fusion attention features.

[0153] The target text image fusion attention feature includes a first text image fusion attention feature, a second text image fusion attention feature, a third text image fusion attention feature, and a fourth text image fusion attention feature.

[0154] Please note that Figure 9The scale attention module is mainly composed of four multi-core spatial attention modules and four channel attention modules. The scale attention module selectively enhances the skip connection features from the encoder. Specifically, through the two sub-modules of channel attention and spatial attention connected in series, the importance of information is modeled from the channel dimension and spatial dimension of the feature map respectively, to achieve the prominence of key semantic information and the suppression of redundant noise. The channel attention module uses global average pooling and maximum pooling to globally describe the feature map, and generates channel weights through a shared multi-layer perceptron, thereby emphasizing channels that are more discriminative in the global context; while the multi-core spatial attention module generates a spatial attention map by fusing statistical information of the channel dimension, guiding the model to focus on salient areas. By integrating this scale attention mechanism in the skip connection, the model can adaptively screen important features in the multi-scale semantic fusion process, effectively improving the quality of feature transfer and the reconstruction capability of the decoder. Specifically, each feature (of different sizes) from different layers of the encoder is subjected to channel attention and multi-core spatial attention respectively to obtain four features* of different sizes (corresponding to the feature sizes of the corresponding layers of the encoder). The four features* are upsampled, downsampled and directly mapped at the same time. Each feature* will obtain four new features# of different sizes, totaling 16 features# (there are 4 scales, each with 4 features#). Then, the features# of the same scale are multiplied element-by-element, and then added element-by-element with the original features of the same size from the encoder, and finally passed to the corresponding decoder layer.

[0155] Specifically, the processing process of the scale attention module is as follows: a multi-core spatial attention module is used to perform spatial dimension feature capture operations on the first text image fusion coding features, the second text image fusion coding features, the third text image fusion coding features, and the fourth text image fusion coding features, and output the first spatial dimension features, the second spatial dimension features, the third spatial dimension features, and the fourth spatial dimension features; a channel attention module is used to perform channel dimension importance weight learning operations on the first spatial dimension features, the second spatial dimension features, the third spatial dimension features, and the fourth spatial dimension features, and output the first channel dimension importance features, the second channel dimension importance features, the third channel dimension importance features, and the fourth channel dimension importance features.

[0156] Next, the important features of the first channel dimension are directly mapped and downsampled at three different scales to obtain four features of different sizes: A, feature A', feature A'', and feature A'''. The important features of the second channel dimension are upsampled, directly mapped, and downsampled at two different scales to obtain four features of different sizes: B, feature B', feature B'', and feature B'''.

[0157] The importance features of the third channel dimension are upsampled, directly mapped, and downsampled at two different scales to obtain four features C, feature C', feature C'', and feature C''' of different sizes. The importance features of the fourth channel dimension are upsampled and directly mapped at three different scales to obtain four features D, feature D', feature D'', and feature D''' of different sizes. Then, the features A, feature B, feature C, and feature D of the same size are smoothed convolved and multiplied element by element. After multiplication, they are added element by element with the original features of the same size from the encoder to obtain the first text image fusion attention feature. Similarly, the second text image fusion attention feature, the third text image fusion attention feature, and the fourth text image fusion attention feature can be obtained.

[0158] Further, see Figure 10 The processing principle of the multi-core spatial attention module is as follows: the input features of the multi-core spatial attention module are subjected to maximum pooling and average pooling operations to obtain corresponding features. After the two sets of features are spliced, they are subjected to convolution operations with different kernel sizes (7×7, 5×5, 3×3), and then added and fused. The activation function is used for nonlinear mapping to obtain spatial attention features. Finally, the spatial attention features are combined with the original input features (element-wise multiplication) to output features adjusted by spatial attention, thereby enhancing the attention of the input features in the spatial dimension.

[0159] Step 106: Input the target text image fusion attention feature and the target text image fusion encoding feature into the feature decoder for decoding, and output the target text image decoding feature.

[0160] The feature decoder includes a first multimodal feature extraction and fusion module, a second multimodal feature extraction and fusion module, a third multimodal feature extraction and fusion module, and a fourth multimodal feature extraction and fusion module.

[0161] Specifically, step 106 may include the following sub-steps S61-S68:

[0162] Step S61: Using a fourth multimodal feature extraction and fusion module to perform multimodal feature extraction and fusion on the fourth text multi-scale feature and the fourth text image fusion coding feature, and outputting the first text image decoding feature;

[0163] Step S62: adding the first text image decoding feature and the fourth text image fusion attention feature element by element to generate a first text image addition feature, upsampling the first text image addition feature, and outputting a first text image fusion upsampled feature;

[0164] Step S63: Using a third multimodal feature extraction and fusion module to perform multimodal feature extraction and fusion on the third text multi-scale feature and the first text image fusion upsampling feature, and output a second text image decoding feature;

[0165] Step S64: adding the second text image decoding feature and the third text image fused attention feature element by element to generate a second text image added feature, upsampling the second text image added feature, and outputting a second text image fused upsampled feature;

[0166] Step S65: Using a second multimodal feature extraction and fusion module to perform multimodal feature extraction and fusion on the second text multi-scale feature and the second text image fusion upsampling feature, and output a third text image decoding feature;

[0167] Step S66: performing element-by-element addition on the third text image decoding feature and the second text image fused attention feature to generate a third text image added feature, and upsampling the third text image added feature to output a third text image fused upsampled feature;

[0168] Step S67: using the first multimodal feature extraction and fusion module to perform multimodal feature extraction and fusion on the first text multi-scale feature and the third text image fusion upsampling feature, and outputting a fourth text image decoding feature;

[0169] Step S68: Add the fourth text image decoding feature and the first text image fusion attention feature element by element to generate the target text image decoding feature.

[0170] It should be noted that the structure of the feature decoder is similar to that of the feature encoder. Its main purpose is to gradually restore the size of the feature to the same as the input through layered upsampling. At the same time, it accepts the skip-connected features from the encoder after scale attention processing to prevent gradient disappearance, enhance model robustness, and output the target area segmentation annotation in the final output layer.

[0171] Step 107: Use a segmentation head to segment the target text image decoding features and output the target segmentation result.

[0172] It should be noted that the target text image decoding features are segmented by the segmentation head and the target segmentation result is output.

[0173] For comparison purposes, existing technologies can be used as a reference. Among these technologies, the CNN (Convolutional Neural Network)-based Unet and the Transformer-based transUnet are two implementations that are relatively close to the present invention. The Unet model pioneered a symmetrical encoder-decoder structure and skip connections. The encoder captures features at different scales through progressive downsampling, while the decoder recovers spatial information through progressive upsampling. The skip connection design avoids feature loss during downsampling, helping the model preserve finer spatial information. The inherent characteristics of CNNs allow them to effectively capture local information but are not adept at capturing long-range global information. transUnet, on the other hand, builds on Unet by introducing a Transformer module, enhancing its ability to capture long-range global information. However, the inherent characteristics of the Transformer increase the model's computational complexity and cost. Furthermore, existing technologies only use image data as model input, which means they only utilize image information and ignore the potential correlations between clinical text and multimodal image data.

[0174] Based on the above foundations, the shortcomings of existing methods for rectal cancer medical image lesion segmentation are: 1) It is difficult to strike a balance between low parameter count and high performance: In existing technologies, models with a low number of parameters usually limit the model's expressiveness and performance, resulting in poor performance on complex tasks. While increasing the number of parameters can improve performance, it also brings higher computational costs and the risk of overfitting, making it challenging to balance the two. 2) Inability to fully utilize multimodal information: Existing technologies cannot fully utilize medical imaging data and clinical text data in combination, fail to effectively explore the potential correlations between different modalities, and fail to extract and fuse multimodal features, resulting in insufficient information utilization or inefficient processing. This limits the performance of the model in actual medical scenarios and fails to realize the potential of multimodal data.

[0175] To address the above problems, the present invention proposes a deep learning-based method for rectal cancer medical image lesion segmentation, which can solve the technical problems that existing methods cannot fully utilize multimodal data in real medical scenarios and achieve low-parameter and high-performance segmentation of target areas. The present invention can fully utilize medical image data and clinical text data, explore the potential correlation between medical text information and image information, and extract multimodal fusion features to improve the accuracy and generalization of the rectal cancer lesion automatic segmentation model, thereby better meeting the real medical scenario's demand for rapid and automatic annotation of rectal cancer lesions. Specifically, the present invention uses a new visual state space model module based on state space modeling as the main feature extraction and calculation unit, which can effectively extract visual features at different scales and fuse visual and text features at different scales to enhance feature expression capabilities, balancing low parameters and high performance. In addition, the text information is encoded by a large language pre-training model to obtain a universal text embedding. After passing through a learnable text multi-scale feature extraction module, the text embedding continuously approaches the image features in a high-dimensional space, learning the correlation between image and text.

[0176] In an embodiment of the present invention, the present invention provides a method for rectal cancer medical image lesion segmentation based on deep learning, obtaining a rectal cancer medical image lesion image to be tested and clinical text information corresponding to the rectal cancer medical image lesion image to be tested; inputting the rectal cancer medical image lesion image to be tested and the clinical text information into a rectal cancer medical image lesion segmentation model based on deep learning, the rectal cancer medical image lesion segmentation model based on deep learning includes a text multi-scale feature extraction module, a feature encoder, a feature decoder, a scale attention module and a segmentation head; using the text multi-scale feature extraction module to extract text multi-scale features from the clinical text information to generate target text multi-scale features; using the feature encoder to perform the target text multi-scale features on the rectal cancer medical image lesion image to be tested. The lesion image is encoded and the target text image fusion coding feature is output; the target text image fusion coding feature is used as the input of the scale attention module, and the target text image fusion attention feature is output; the target text image fusion attention feature and the target text image fusion coding feature are input into the feature decoder for decoding, and the target text image decoding feature is output; the target text image decoding feature is segmented using a segmentation head, and the target segmentation result is output; based on the above scheme, the present invention utilizes a rectal cancer medical image lesion segmentation model based on deep learning to process the rectal cancer medical image lesion image to be tested and clinical text information, which can fully utilize medical image data and clinical text data in combination, effectively explore the potential correlation between different modalities, and thus improve the segmentation effect.

[0177] For better explanation, refer to Figure 11, shows a flowchart of the steps of the model training process of the rectal cancer medical image lesion segmentation model based on deep learning provided by the second embodiment of the present invention. The process may include the following steps:

[0178] Step 1101: obtain training rectal cancer medical imaging lesion images and training clinical text information corresponding to the training rectal cancer medical imaging lesion images, perform data enhancement on the training rectal cancer medical imaging lesion images, and output target training rectal cancer medical imaging lesion images.

[0179] It should be noted that in order to simulate the light spots and jitters that may appear in the real medical imaging process, the present invention uses the difusion diffusion model to randomly enhance some data samples to simulate these noises, and at the same time applies random rotation, flipping, occlusion, scaling and other data enhancement methods to enhance the robustness of the model.

[0180] Step 1102: Input the target training rectal cancer medical imaging lesion image and training clinical text information into the deep learning-based rectal cancer medical imaging lesion segmentation model to be trained, and output the training segmentation results and training clinical text encoding features.

[0181] Step 1103: Substitute the training segmentation results and the training clinical text coding features into the preset loss function and derive them to output the model gradient.

[0182] Step 1104: Use model gradients to update the model parameters of the deep learning-based rectal cancer medical image lesion segmentation model to be trained, output the intermediate deep learning-based rectal cancer medical image lesion segmentation model, and count the number of model updates in real time.

[0183] Step 1105: Determine whether the number of model updates reaches the preset number of training times.

[0184] Step 1106: If yes, use the intermediate rectal cancer medical image lesion segmentation model based on deep learning as the trained rectal cancer medical image lesion segmentation model based on deep learning.

[0185] The model gradient includes a first model gradient and a second model gradient.

[0186] It should be noted that this invention follows a training strategy of progressively more difficult data, progressing from large, simple, and noise-free data to small, complex, and noisy data. This strategy aims to more quickly learn simple, general features before moving on to more difficult, specialized features, thereby improving the model's convergence speed and robustness.

[0187] Furthermore, the present invention uses 0.2*Cross Entropy + 0.3*Dice + 0.5*InfoNCE in the text multi-scale feature extraction module to better perceive the matching between image and text. The cross attention module and multi-layer perceptron in the multimodal feature extraction and fusion module use 0.3*Cross Entropy + 0.5*Dice + 0.2*InfoNCE to ensure that the matching between image and text is not forgotten during the multimodal fusion process.

[0188] Specifically, the preset loss function consists of two parts: Formula 1: 0.2*Cross-Entropy + 0.3*Dice + 0.5*InfoNCE, and Formula 2: 0.3*Cross-Entropy + 0.5*Dice + 0.2*InfoNCE. Cross-Entropy is one of the most commonly used loss functions in classification tasks, measuring the difference between two probability distributions. It is particularly commonly used for the gap between the classification output of a neural network and the true label. Dice loss is commonly used in image segmentation tasks, especially in medical images to address class imbalance. The goal is to maximize the overlap between the predicted and true regions. InfoNCE is a loss function used in contrastive learning. By constructing a contrast objective, it allows the model to learn to bring positive pairs closer together while pushing negative pairs further apart.

[0189] Furthermore, the training data is input into the deep learning-based rectal cancer medical image lesion segmentation model to be trained, and the training segmentation results and training clinical text encoding features are output. The training segmentation results and training clinical text encoding features are substituted into formula 1 and differentiated to obtain the first model gradient. The first model gradient is used to update the model parameters of the text multi-scale feature extraction module in the deep learning-based rectal cancer medical image lesion segmentation model to be trained. The training segmentation results and training clinical text encoding features are substituted into formula 2 and differentiated to obtain the second model gradient. The second model gradient is used to update the model parameters of all multimodal feature extraction and fusion modules in the deep learning-based rectal cancer medical image lesion segmentation model to be trained to obtain the intermediate deep learning-based rectal cancer medical image lesion segmentation model. When the number of model updates reaches the preset training number, the intermediate deep learning-based rectal cancer medical image lesion segmentation model is used as the trained deep learning-based rectal cancer medical image lesion segmentation model. When the number of model updates does not reach the preset number of training times, the intermediate deep learning-based rectal cancer medical image lesion segmentation model is used as a new deep learning-based rectal cancer medical image lesion segmentation model to be trained, and the process jumps to step 1102 until the number of model updates reaches the preset number of training times. The intermediate deep learning-based rectal cancer medical image lesion segmentation model determined when the number of model updates reaches the preset number of training times is used as the trained deep learning-based rectal cancer medical image lesion segmentation model.

[0190] It is worth mentioning that for cross entropy The calculation principle can be expressed as:

[0191] ;

[0192] in, is the true label; is the probability distribution of the model output (that is, the output predicted by the model), corresponding to the training segmentation result; C is the number of categories.

[0193] For the Dice loss function The calculation principle can be expressed as:

[0194] ;

[0195] Where C is the number of categories; is the predicted probability of the i-th pixel for the c-th class, corresponding to the training segmentation result; is the c-th class true label of the i-th pixel; is a small constant used for numerical stability.

[0196] For InfoNCE loss function The calculation principle can be expressed as:

[0197] ;

[0198] ;

[0199] Among them, t is the text feature, corresponding to the training clinical text coding feature, , C is the feature dimension; is the average pixel feature of the mask area in the image, corresponding to the training segmentation result, ; is the pixel feature set of the non-mask area in the image, and K represents the number of pixel features in the non-mask area; is the temperature coefficient used to scale the similarity, N is the batch size; is the similarity; a, b are the inputs for calculating similarity; is the norm.

[0200] In an embodiment of the present invention, the present invention enhances the model feature extraction capability and improves the model performance by designing a multimodal feature extraction and fusion module and a hybrid attention module, and introducing a joint loss function of cross entropy-Dice similarity. Using a trained deep learning-based rectal cancer medical image lesion segmentation model for rectal cancer medical image lesion segmentation can effectively capture local image information while capturing global image information with high precision and low computational cost, thereby reducing information loss during the feature extraction process and retaining more sufficient and comprehensive image information. In addition, the present invention is more in line with actual medical scenarios, can make full use of the data and information in medical images and clinical texts in actual medical scenarios, and can be generalized to the automatic segmentation of medical images of various tissues and organs.

[0201] See also Figure 12 , Figure 12 This is a structural block diagram of a deep learning-based rectal cancer medical image lesion segmentation device provided in Example 3 of the present invention.

[0202] The present invention provides a deep learning-based rectal cancer medical image lesion segmentation device, comprising:

[0203] An acquisition module 1201 is configured to acquire a rectal cancer medical imaging lesion image to be detected and clinical text information corresponding to the rectal cancer medical imaging lesion image to be detected;

[0204] An input module 1202 is configured to input the colorectal cancer medical imaging lesion image and clinical text information to be tested into a colorectal cancer medical imaging lesion segmentation model based on deep learning, wherein the colorectal cancer medical imaging lesion segmentation model based on deep learning includes a text multi-scale feature extraction module, a feature encoder, a feature decoder, a scale attention module, and a segmentation head;

[0205] Extraction module 1203, configured to extract multi-scale features from clinical text information using a text multi-scale feature extraction module to generate target text multi-scale features;

[0206] The encoding module 1204 is used to encode the multi-scale features of the target text and the image of the rectal cancer medical imaging lesion to be detected through a feature encoder, and output the target text image fusion encoding features;

[0207] The scaling module 1205 is configured to take the target text image fused coding features as input to the scaling attention module and output the target text image fused attention features;

[0208] A decoding module 1206 is used to input the target text image fusion attention feature and the target text image fusion coding feature into a feature decoder for decoding, and output the target text image decoding feature;

[0209] The segmentation module 1207 is used to segment the target text image decoding features using a segmentation head and output a target segmentation result.

[0210] Furthermore, the text multi-scale feature extraction module includes a text encoder, a first multi-layer perceptron, a first state space model, a second multi-layer perceptron, a second state space model, a third multi-layer perceptron, a third state space model, a fourth multi-layer perceptron, and a fourth state space model; the target text multi-scale feature includes the first text multi-scale feature, the second text multi-scale feature, the third text multi-scale feature, and the fourth text multi-scale feature; the extraction module 1203 is specifically used to:

[0211] A text encoder is used to encode clinical text information and output clinical text encoding features;

[0212] The clinical text coding features are used as the input of the first multi-layer perceptron, the second multi-layer perceptron, the third multi-layer perceptron, and the fourth multi-layer perceptron, respectively, and the first text perception feature, the second text perception feature, the third text perception feature, and the fourth text perception feature are output;

[0213] Performing spatial modeling on the first text perception feature through a first state space model to generate a first text state feature;

[0214] Performing spatial modeling on the second text perception feature through a second state space model to generate a second text state feature;

[0215] Performing spatial modeling on the third text perception feature through a third state space model to generate a third text state feature;

[0216] Performing spatial modeling on the fourth text perception feature through a fourth state space model to generate a fourth text state feature;

[0217] The first text state feature, the second text state feature, the third text state feature, and the fourth text state feature are respectively broadcast pixel by pixel to generate the first text multi-scale feature, the second text multi-scale feature, the third text multi-scale feature, and the fourth text multi-scale feature.

[0218] Furthermore, the feature encoder includes a first multimodal feature extraction and fusion module, a second multimodal feature extraction and fusion module, a third multimodal feature extraction and fusion module, and a fourth multimodal feature extraction and fusion module; the target text image fusion coding feature includes a first text image fusion coding feature, a second text image fusion coding feature, a third text image fusion coding feature, and a fourth text image fusion coding feature; the encoding module 1204 includes:

[0219] The first submodule is configured to perform multimodal feature extraction and fusion on the first text multi-scale feature and the rectal cancer medical imaging lesion image to be detected using a first multimodal feature extraction and fusion module, and output a first text image fusion coding feature;

[0220] The second submodule is used to downsample the first text image fusion encoding feature to generate a first text image fusion downsampled feature;

[0221] The third submodule is used to use the second multimodal feature extraction and fusion module to perform multimodal feature extraction and fusion on the second text multi-scale feature and the first text image fusion downsampling feature, and output the second text image fusion coding feature;

[0222] A fourth submodule is configured to downsample the second text image fusion encoding feature to generate a second text image fusion downsampled feature;

[0223] A fifth submodule is configured to perform multimodal feature extraction and fusion on the third text multi-scale feature and the second text image fusion downsampling feature using a third multimodal feature extraction and fusion module, and output a third text image fusion coding feature;

[0224] A sixth submodule is configured to downsample the third text image fusion coding feature to generate a third text image fusion downsampled feature;

[0225] The seventh submodule is used to use the fourth multimodal feature extraction and fusion module to perform multimodal feature extraction and fusion on the fourth text multi-scale feature and the third text image fusion downsampling feature, and output the fourth text image fusion coding feature.

[0226] Furthermore, the first multimodal feature extraction and fusion module includes a visual state space feature extraction module, a multi-core spatial attention module, a 3×3 convolutional layer, a cross attention module, a multi-layer perceptron, and a channel attention module; the first submodule includes:

[0227] The first unit is used to use the rectal cancer medical imaging lesion image to be tested as the input of the visual state space feature extraction module and the multi-core spatial attention module, and output the visual state space image features corresponding to the visual state space feature extraction module and the spatial attention image features corresponding to the multi-core spatial attention module;

[0228] The second unit is used to add the visual state space image features and the spatial attention image features element by element and output the image fusion features;

[0229] The third unit is used to perform feature splicing on the image fusion feature and the first text multi-scale feature, and output the first image text splicing feature;

[0230] The fourth unit is used to perform a convolution operation on the first image-text splicing feature through a 3×3 convolution layer to output the image-text convolution feature;

[0231] The fifth unit is used to take the image fusion feature and the first text multi-scale feature as the input of the cross attention module, output the image-text cross feature, and perform nonlinear mapping on the image-text convolution feature to output the image-text nonlinear feature;

[0232] The sixth unit is used to perform feature splicing on the image-text nonlinear features and the image-text cross features, output a second image-text splicing feature, and use the second image-text splicing feature as an input of a multi-layer perceptron to output an image-text perception feature;

[0233] The seventh unit is used to perform element-by-element multiplication of the image text perception feature and the image fusion feature, and output the image text multiplication feature;

[0234] The eighth unit is used to use the image-text multiplication feature as the input of the channel attention module, output the image-text channel attention feature, and add the image-text channel attention feature, the image-text perception feature, and the rectal cancer medical imaging lesion image to be tested element by element to output the first text-image fusion coding feature.

[0235] Furthermore, the visual state space feature extraction module includes a fully connected layer, a depthwise separable convolution module, and a two-dimensional state space feature extraction module; the first unit is specifically used to:

[0236] Perform layer standardization on the image of the rectal cancer lesion to be tested and output a standardized image;

[0237] The standardized image is used as the input of the fully connected layer, the first fully connected feature is output, and the depth-wise separable convolution module is used to perform depth-wise separable convolution on the first fully connected feature, outputting the depth-wise separable convolution feature;

[0238] Performing nonlinear mapping on the depthwise separable convolutional features to output a first image nonlinear feature, and using the first image nonlinear feature as an input to a two-dimensional state space feature extraction module to output a two-dimensional state space feature;

[0239] Perform layer-by-layer normalization on the two-dimensional state space features to generate standardized two-dimensional state space features;

[0240] The rectal cancer medical imaging lesion image to be tested is used as the input of the fully connected layer, the second fully connected feature is output, and the second fully connected feature is nonlinearly mapped to output the second image nonlinear feature;

[0241] Perform element-wise multiplication on the second image nonlinear feature and the standardized two-dimensional state space feature, output the multiplied feature, and use the multiplied feature as the input of the fully connected layer to output the third fully connected feature;

[0242] The third fully connected feature and the rectal cancer medical imaging lesion image to be tested are added element by element to output the visual state space image feature.

[0243] Furthermore, the feature decoder includes a first multimodal feature extraction and fusion module, a second multimodal feature extraction and fusion module, a third multimodal feature extraction and fusion module, and a fourth multimodal feature extraction and fusion module; the target text image fusion attention feature includes a first text image fusion attention feature, a second text image fusion attention feature, a third text image fusion attention feature, and a fourth text image fusion attention feature; the decoding module 1206 is specifically used to:

[0244] Performing multimodal feature extraction and fusion on the fourth text multi-scale feature and the fourth text image fusion coding feature using a fourth multimodal feature extraction and fusion module, and outputting the first text image decoding feature;

[0245] Adding the first text image decoding feature and the fourth text image fused attention feature element by element to generate a first text image added feature, upsampling the first text image added feature, and outputting a first text image fused upsampled feature;

[0246] A third multimodal feature extraction and fusion module is used to perform multimodal feature extraction and fusion on the third text multi-scale feature and the first text image fusion upsampling feature, and output a second text image decoding feature;

[0247] Add the second text image decoding feature and the third text image fusion attention feature element by element to generate a second text image addition feature, upsample the second text image addition feature, and output a second text image fusion upsampled feature;

[0248] Using a second multimodal feature extraction and fusion module to perform multimodal feature extraction and fusion on the second text multi-scale feature and the second text image fusion upsampling feature, and output a third text image decoding feature;

[0249] Adding the third text image decoding feature and the second text image fused attention feature element-by-element to generate a third text image added feature, upsampling the third text image added feature, and outputting a third text image fused upsampled feature;

[0250] Using the first multimodal feature extraction and fusion module to perform multimodal feature extraction and fusion on the first text multi-scale feature and the third text image fusion upsampling feature, and output a fourth text image decoding feature;

[0251] The fourth text image decoding feature and the first text image fusion attention feature are added element by element to generate the target text image decoding feature.

[0252] In an optional embodiment of the device, the device further comprises:

[0253] The first module is used to obtain training rectal cancer medical imaging lesion images and training clinical text information corresponding to the training rectal cancer medical imaging lesion images, perform data enhancement on the training rectal cancer medical imaging lesion images, and output target training rectal cancer medical imaging lesion images;

[0254] The second module is used to input the target training rectal cancer medical imaging lesion image and training clinical text information into the rectal cancer medical imaging lesion segmentation model to be trained based on deep learning, and output the training segmentation results and training clinical text encoding features;

[0255] The third module is used to substitute the training segmentation results and the training clinical text encoding features into the preset loss function and derive them, and output the model gradient;

[0256] The fourth module is used to update the model parameters of the deep learning-based rectal cancer medical image lesion segmentation model to be trained using the model gradient, output the intermediate deep learning-based rectal cancer medical image lesion segmentation model, and count the number of model updates in real time;

[0257] The fifth module is used to determine whether the number of model updates reaches the preset number of training times;

[0258] The sixth module is used to use the intermediate rectal cancer medical image lesion segmentation model based on deep learning as the trained rectal cancer medical image lesion segmentation model based on deep learning.

[0259] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, modules, sub-modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0260] An embodiment of the present invention also provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory; when the computer program is executed by the processor, the processor executes the steps of the deep learning-based rectal cancer medical image lesion segmentation method as described in any of the above embodiments.

[0261] An embodiment of the present invention also provides a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, the steps of the method for segmenting rectal cancer medical images based on deep learning as in any of the above embodiments are implemented.

[0262] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A deep learning-based rectal cancer medical image lesion segmentation method, characterized in that: include: Acquire a rectal cancer medical imaging lesion image to be detected and clinical text information corresponding to the rectal cancer medical imaging lesion image to be detected; Inputting the rectal cancer medical imaging lesion image to be tested and the clinical text information into a rectal cancer medical imaging lesion segmentation model based on deep learning, wherein the rectal cancer medical imaging lesion segmentation model based on deep learning includes a text multi-scale feature extraction module, a feature encoder, a feature decoder, a scale attention module and a segmentation head; Using the text multi-scale feature extraction module to extract text multi-scale features from the clinical text information to generate target text multi-scale features; Encoding the target text multi-scale features and the rectal cancer medical imaging lesion image to be detected by the feature encoder, and outputting the target text image fusion coding features; Taking the target text image fusion coding feature as the input of the scale attention module, and outputting the target text image fusion attention feature; Inputting the target text image fusion attention feature and the target text image fusion encoding feature into the feature decoder for decoding, and outputting the target text image decoding feature; The segmentation head is used to segment the target text image decoding features and output a target segmentation result.

2. The method for rectal cancer medical image lesion segmentation based on deep learning according to claim 1, characterized in that: The text multi-scale feature extraction module includes a text encoder, a first multi-layer perceptron, a first state space model, a second multi-layer perceptron, a second state space model, a third multi-layer perceptron, a third state space model, a fourth multi-layer perceptron, and a fourth state space model; the target text multi-scale feature includes a first text multi-scale feature, a second text multi-scale feature, a third text multi-scale feature, and a fourth text multi-scale feature; The method of extracting multi-scale features from the clinical text information using the multi-scale feature extraction module to generate multi-scale features of the target text includes: Using the text encoder to perform text encoding on the clinical text information, and outputting clinical text encoding features; Using the clinical text coding features as inputs of the first multi-layer perceptron, the second multi-layer perceptron, the third multi-layer perceptron, and the fourth multi-layer perceptron, respectively, and outputting first text-perceptual features, second text-perceptual features, third text-perceptual features, and fourth text-perceptual features; Performing spatial modeling on the first text perception feature by using the first state space model to generate a first text state feature; Performing spatial modeling on the second text perception feature by using the second state space model to generate a second text state feature; Performing spatial modeling on the third text perception feature by using the third state space model to generate a third text state feature; Performing spatial modeling on the fourth text perception feature by using the fourth state space model to generate a fourth text state feature; The first text state feature, the second text state feature, the third text state feature, and the fourth text state feature are respectively broadcast pixel by pixel to generate a first text multi-scale feature, a second text multi-scale feature, a third text multi-scale feature, and a fourth text multi-scale feature.

3. The method for rectal cancer medical image lesion segmentation based on deep learning according to claim 2, characterized in that: The feature encoder includes a first multimodal feature extraction and fusion module, a second multimodal feature extraction and fusion module, a third multimodal feature extraction and fusion module, and a fourth multimodal feature extraction and fusion module; the target text image fusion coding feature includes a first text image fusion coding feature, a second text image fusion coding feature, a third text image fusion coding feature, and a fourth text image fusion coding feature; encoding the target text multi-scale feature and the rectal cancer medical imaging lesion image to be detected by the feature encoder, and outputting the target text image fusion coding feature, includes: Using a first multimodal feature extraction and fusion module to perform multimodal feature extraction and fusion on the first text multi-scale feature and the rectal cancer medical imaging lesion image to be detected, and output a first text image fusion coding feature; Downsampling the first text image fusion encoding feature to generate a first text image fusion downsampled feature; Using a second multimodal feature extraction and fusion module to perform multimodal feature extraction and fusion on the second text multi-scale feature and the first text image fusion downsampling feature, and output a second text image fusion coding feature; Downsampling the second text image fusion encoding feature to generate a second text image fusion downsampled feature; Using a third multimodal feature extraction and fusion module to perform multimodal feature extraction and fusion on the third text multi-scale feature and the second text image fusion downsampling feature, and output a third text image fusion coding feature; Downsampling the third text image fusion coding feature to generate a third text image fusion downsampled feature; A fourth multimodal feature extraction and fusion module is used to perform multimodal feature extraction and fusion on the fourth text multi-scale feature and the third text image fusion downsampling feature, and output a fourth text image fusion coding feature.

4. The method for rectal cancer medical image lesion segmentation based on deep learning according to claim 3, characterized in that: The first multimodal feature extraction and fusion module includes a visual state space feature extraction module, a multi-core spatial attention module, a 3×3 convolutional layer, a cross attention module, a multi-layer perceptron, and a channel attention module; the first multimodal feature extraction and fusion module is used to perform multimodal feature extraction and fusion on the first text multi-scale feature and the rectal cancer medical imaging lesion image to be tested, and output a first text image fusion coding feature, including: The rectal cancer medical imaging lesion image to be tested is used as the input of the visual state space feature extraction module and the multi-core spatial attention module respectively, and the visual state space image features corresponding to the visual state space feature extraction module and the spatial attention image features corresponding to the multi-core spatial attention module are output; performing element-by-element addition on the visual state space image feature and the spatial attention image feature, and outputting an image fusion feature; Performing feature splicing on the image fusion feature and the first text multi-scale feature, and outputting a first image-text splicing feature; Performing a convolution operation on the first image-text splicing feature through a 3×3 convolution layer to output an image-text convolution feature; Using the image fusion feature and the first text multi-scale feature as inputs of a cross attention module, outputting an image-text cross feature, and performing nonlinear mapping on the image-text convolution feature to output an image-text nonlinear feature; Performing feature splicing on the image-text nonlinear feature and the image-text cross feature to output a second image-text splicing feature, and using the second image-text splicing feature as an input of a multi-layer perceptron to output an image-text perception feature; performing element-by-element multiplication on the image-text perception feature and the image fusion feature, and outputting an image-text multiplication feature; The image-text multiplication feature is used as the input of the channel attention module, and the image-text channel attention feature is output. The image-text channel attention feature, the image-text perception feature, and the rectal cancer medical imaging lesion image to be tested are added element by element to output the first text-image fusion coding feature.

5. The method for rectal cancer medical image lesion segmentation based on deep learning according to claim 4, characterized in that: The visual state space feature extraction module includes a fully connected layer, a depthwise separable convolution module, and a two-dimensional state space feature extraction module; the method uses the rectal cancer medical imaging lesion image to be detected as the input of the visual state space feature extraction module and outputs the visual state space image features corresponding to the visual state space feature extraction module, including: performing layer normalization on the rectal cancer medical imaging lesion image to be detected, and outputting a normalized image; Using the standardized image as the input of a fully connected layer, outputting a first fully connected feature, and performing a depthwise separable convolution on the first fully connected feature using a depthwise separable convolution module, outputting a depthwise separable convolution feature; Performing nonlinear mapping on the depthwise separable convolution feature to output a first image nonlinear feature, and using the first image nonlinear feature as input to a two-dimensional state space feature extraction module to output a two-dimensional state space feature; performing layer normalization on the two-dimensional state space feature to generate a normalized two-dimensional state space feature; Using the rectal cancer medical imaging lesion image to be tested as the input of the fully connected layer, outputting a second fully connected feature, and performing nonlinear mapping on the second fully connected feature to output a second image nonlinear feature; Performing element-by-element multiplication on the second image nonlinear feature and the standardized two-dimensional state space feature, outputting a multiplied feature, and using the multiplied feature as an input of a fully connected layer to output a third fully connected feature; The third fully connected feature and the rectal cancer medical imaging lesion image to be detected are added element by element to output a visual state space image feature.

6. The method for rectal cancer medical image lesion segmentation based on deep learning according to claim 3, characterized in that: The feature decoder includes a first multimodal feature extraction and fusion module, a second multimodal feature extraction and fusion module, a third multimodal feature extraction and fusion module, and a fourth multimodal feature extraction and fusion module; the target text image fusion attention feature includes a first text image fusion attention feature, a second text image fusion attention feature, a third text image fusion attention feature, and a fourth text image fusion attention feature; the target text image fusion attention feature and the target text image fusion encoding feature are input into the feature decoder for decoding, and the target text image decoding feature is output, including: Using a fourth multimodal feature extraction and fusion module to perform multimodal feature extraction and fusion on the fourth text multi-scale feature and the fourth text image fusion coding feature, and outputting a first text image decoding feature; performing element-by-element addition on the first text image decoding feature and the fourth text image fused attention feature to generate a first text image added feature, and upsampling the first text image added feature to output a first text image fused upsampled feature; Using a third multimodal feature extraction and fusion module to perform multimodal feature extraction and fusion on the third text multi-scale feature and the first text image fusion upsampling feature, and output a second text image decoding feature; performing element-by-element addition on the second text image decoding feature and the third text image fused attention feature to generate a second text image added feature, and upsampling the second text image added feature to output a second text image fused upsampled feature; Using a second multimodal feature extraction and fusion module to perform multimodal feature extraction and fusion on the second text multi-scale feature and the second text image fusion upsampling feature, and output a third text image decoding feature; performing element-by-element addition on the third text image decoding feature and the second text image fusion attention feature to generate a third text image addition feature, and upsampling the third text image addition feature to output a third text image fusion upsampled feature; Using a first multimodal feature extraction and fusion module to perform multimodal feature extraction and fusion on the first text multi-scale feature and the third text image fusion upsampling feature, and output a fourth text image decoding feature; The fourth text image decoding feature and the first text image fusion attention feature are added element by element to generate a target text image decoding feature.

7. The method for rectal cancer medical image lesion segmentation based on deep learning according to claim 1, characterized in that: The model training process of the deep learning-based rectal cancer medical imaging lesion segmentation model is specifically as follows: Acquire a training rectal cancer medical imaging lesion image and training clinical text information corresponding to the training rectal cancer medical imaging lesion image, perform data enhancement on the training rectal cancer medical imaging lesion image, and output a target training rectal cancer medical imaging lesion image; Inputting the target training rectal cancer medical imaging lesion image and the training clinical text information into a rectal cancer medical imaging lesion segmentation model to be trained based on deep learning, and outputting a training segmentation result and a training clinical text encoding feature; Substituting the training segmentation result and the training clinical text coding feature into a preset loss function and taking derivatives, outputting a model gradient; Using the model gradient to update the model parameters of the deep learning-based rectal cancer medical image lesion segmentation model to be trained, outputting an intermediate deep learning-based rectal cancer medical image lesion segmentation model, and counting the number of model updates in real time; Determine whether the model update times reaches the preset training times; If so, the intermediate rectal cancer medical image lesion segmentation model based on deep learning is used as the trained rectal cancer medical image lesion segmentation model based on deep learning.

8. A rectal cancer medical image lesion segmentation device based on deep learning, characterized in that: include: An acquisition module, configured to acquire a rectal cancer medical imaging lesion image to be detected and clinical text information corresponding to the rectal cancer medical imaging lesion image to be detected; An input module is used to input the rectal cancer medical imaging lesion image to be tested and the clinical text information into a rectal cancer medical imaging lesion segmentation model based on deep learning, wherein the rectal cancer medical imaging lesion segmentation model based on deep learning includes a text multi-scale feature extraction module, a feature encoder, a feature decoder, a scale attention module and a segmentation head; An extraction module, configured to extract multi-scale features from the clinical text information using the multi-scale feature extraction module to generate multi-scale features of the target text; An encoding module, configured to encode the target text multi-scale features and the rectal cancer medical imaging lesion image to be detected through the feature encoder, and output a target text image fusion encoding feature; a scale module, configured to take the target text image fused coding features as input to the scale attention module and output the target text image fused attention features; A decoding module, configured to input the target text image fusion attention feature and the target text image fusion coding feature into the feature decoder for decoding, and output a target text image decoding feature; The segmentation module is used to segment the target text image decoding features using the segmentation head and output a target segmentation result.

9. A computer device, characterized in that: It includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor performs the steps of the deep learning-based rectal cancer medical imaging lesion segmentation method as described in any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed, the deep learning-based rectal cancer medical image lesion segmentation method according to any one of claims 1 to 6 is implemented.

Images