A rectal cancer medical image lesion segmentation method and device based on deep learning

Abstract

The application discloses a kind of rectal cancer medical image lesion segmentation method and device based on deep learning, to solve the problem that prior art only takes medical image as single input mode, ignores potential semantic association between multi-modal data, leading to poor segmentation effect. The method comprises obtaining rectal cancer medical image lesion image and its corresponding clinical text information, and inputting the two into a segmentation model for processing. The clinical text information is processed by a text multi-scale feature extraction module to generate target text multi-scale features. The text features and medical images are encoded using a feature encoder to output target text image fusion encoding features. The fusion encoding features are input into a scale attention module to output target text image fusion attention features. The fusion attention features and fusion encoding features are processed by a feature decoder to obtain target text image decoding features. Finally, the decoding features are segmented using a segmentation head to output the final target segmentation result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing, and in particular to a rectal cancer medical image lesion segmentation method and device based on deep learning. BACKGROUND

[0002] Medical image segmentation technology can extract imaging features, which play an important role in evaluating patient prognosis and assisting doctors in developing treatment plans, as they provide structural, density, size, and possibly useful biological information about the target region.

[0003] The extraction of imaging features is usually divided into traditional methods and deep learning methods. Traditional segmentation methods based on image processing can extract features such as shape, texture, and density. Deep learning methods use deep neural networks to automatically learn and extract more complex and advanced feature representations from input data (mainly images), achieving more accurate and highly automated medical image segmentation.

[0004] Medical image automatic segmentation technology plays an increasingly important role in intelligent medical care. Medical image automatic segmentation technology can automatically obtain imaging features and automatically generate annotations of target regions (i.e., the specific shape of tissues or organs in medical images). Traditional image processing-based segmentation methods are gradually being replaced by more efficient deep learning-based automatic segmentation models, and deep learning methods have shown greater robustness and accuracy in complex scenarios.

[0005] Existing deep learning-based rectal cancer medical image lesion segmentation methods mainly use convolutional neural networks (CNN) or Transformer architecture. CNNs can effectively extract local features of lesions due to their local perception characteristics, but have limitations in modeling long-range dependencies; while Transformers can compensate for this deficiency by capturing global context information through self-attention mechanisms, but significantly increase computational complexity and resource consumption. In addition, most current models only use medical images as a single input modality, failing to fully integrate auxiliary information such as clinical text reports, ignoring the potential semantic associations between multi-modal data, resulting in suboptimal segmentation results. SUMMARY

[0006] The present application provides a deep learning-based rectal cancer medical image lesion segmentation method and device, which solves the technical problem that existing deep learning-based rectal cancer medical image lesion segmentation methods only use medical images as a single input modality, fail to fully integrate auxiliary information such as clinical text reports, ignore the potential semantic associations between multi-modal data, and result in suboptimal segmentation results.

[0007] The application provides a deep learning-based colorectal cancer medical image lesion segmentation method, which comprises the following steps:

[0008] Obtaining a to-be-tested colorectal cancer medical image lesion image and clinical text information corresponding to the to-be-tested colorectal cancer medical image lesion image;

[0009] Inputting the to-be-tested colorectal cancer medical image lesion image and the clinical text information into a deep learning-based colorectal cancer medical image lesion segmentation model, wherein the deep learning-based colorectal cancer medical image lesion segmentation model comprises a text multi-scale feature extraction module, a feature encoder, a feature decoder, a scale attention module and a segmentation head;

[0010] Extracting text multi-scale features of the clinical text information by using the text multi-scale feature extraction module to generate target text multi-scale features;

[0011] Encoding the target text multi-scale features and the to-be-tested colorectal cancer medical image lesion image by using the feature encoder to output target text image fusion encoding features;

[0012] Taking the target text image fusion encoding features as inputs of the scale attention module to output target text image fusion attention features;

[0013] Inputting the target text image fusion attention features and the target text image fusion encoding features into the feature decoder for decoding to output target text image decoding features;

[0014] Segmenting the target text image decoding features by using the segmentation head to output a target segmentation result.

[0015] Optionally, the text multi-scale feature extraction module comprises a text encoder, a first multi-layer perceptron, a first state space model, a second multi-layer perceptron, a second state space model, a third multi-layer perceptron, a third state space model, a fourth multi-layer perceptron and a fourth state space model; the target text multi-scale features comprise first text multi-scale features, second text multi-scale features, third text multi-scale features and fourth text multi-scale features; and the text multi-scale feature extraction module is used to extract text multi-scale features of the clinical text information to generate target text multi-scale features, which comprises the following steps:

[0016] Text encoding the clinical text information by using the text encoder to output clinical text encoding features;

[0017] The clinical text coding features are respectively taken as inputs of the first multi-layer perception, the second multi-layer perception, the third multi-layer perception, and the fourth multi-layer perception, and first text perception features, second text perception features, third text perception features, and fourth text perception features are outputted;

[0018] The first text perception features are spatially modeled by the first state space model to generate first text state features;

[0019] The second text perception features are spatially modeled by the second state space model to generate second text state features;

[0020] The third text perception features are spatially modeled by the third state space model to generate third text state features;

[0021] The fourth text perception features are spatially modeled by the fourth state space model to generate fourth text state features;

[0022] The first text state features, the second text state features, the third text state features, and the fourth text state features are respectively pixel-by-pixel broadcasted to generate first text multi-scale features, second text multi-scale features, third text multi-scale features, and fourth text multi-scale features.

[0023] Optionally, the feature encoder comprises a first multi-modal feature extraction and fusion module, a second multi-modal feature extraction and fusion module, a third multi-modal feature extraction and fusion module, and a fourth multi-modal feature extraction and fusion module; the target text image fusion coding features comprise first text image fusion coding features, second text image fusion coding features, third text image fusion coding features, and fourth text image fusion coding features; and the feature encoder is used for coding the target text multi-scale features and the to-be-detected rectal cancer medical image lesion image to output the target text image fusion coding features, comprising:

[0024] The first multi-modal feature extraction and fusion module is used for multi-modal feature extraction and fusion on the first text multi-scale features and the to-be-detected rectal cancer medical image lesion image to output the first text image fusion coding features;

[0025] The first text image fusion coding features are down-sampled to generate first text image fusion down-sampling features;

[0026] The second multi-modal feature extraction and fusion module is used for multi-modal feature extraction and fusion on the second text multi-scale features and the first text image fusion down-sampling features to output the second text image fusion coding features;

[0027] down-sample the second text image fusion encoding feature to generate a second text image fusion down-sampling feature;

[0028] extract and fuse the third text multi-scale feature and the second text image fusion down-sampling feature by using a third multi-modal feature extraction and fusion module to output a third text image fusion encoding feature;

[0029] down-sample the third text image fusion encoding feature to generate a third text image fusion down-sampling feature;

[0030] extract and fuse the fourth text multi-scale feature and the third text image fusion down-sampling feature by using a fourth multi-modal feature extraction and fusion module to output a fourth text image fusion encoding feature.

[0031] Optionally, the first multi-modal feature extraction and fusion module comprises a visual state space feature extraction module, a multi-core space attention module, a 3x3 convolution layer, a cross attention module, a multi-layer perception, and a channel attention module; and the multi-modal feature extraction and fusion of the first text multi-scale feature and the to-be-tested rectal cancer medical image lesion image by using the first multi-modal feature extraction and fusion module to output the first text image fusion encoding feature comprises:

[0032] inputting the to-be-tested rectal cancer medical image lesion image into the visual state space feature extraction module and the multi-core space attention module as inputs to output visual state space image features corresponding to the visual state space feature extraction module and spatial attention image features corresponding to the multi-core space attention module;

[0033] performing element-wise addition on the visual state space image features and the spatial attention image features to output image fusion features;

[0034] performing feature splicing on the image fusion features and the first text multi-scale feature to output first image-text splicing features;

[0035] performing convolution operation on the first image-text splicing features by using the 3x3 convolution layer to output image-text convolution features;

[0036] inputting the image fusion features and the first text multi-scale feature into the cross attention module as inputs to output image-text cross features, and performing nonlinear mapping on the image-text convolution features to output image-text nonlinear features;

[0037] concatenate the image text nonlinear feature and the image text cross feature, output a second image text concatenation feature, and take the second image text concatenation feature as an input of a multilayer perception machine, output an image text perception feature;

[0038] element-wise multiply the image text perception feature and the image fusion feature, output an image text multiplication feature;

[0039] take the image text multiplication feature as an input of a channel attention module, output an image text channel attention feature, and element-wise add the image text channel attention feature, the image text perception feature, and the to-be-tested rectal cancer medical image lesion image, output a first text image fusion encoding feature.

[0040] Optionally, the visual state space feature extraction module includes a full connection layer, a depth separable convolution module, and a two-dimensional state space feature extraction module; taking the to-be-tested rectal cancer medical image lesion image as an input of the visual state space feature extraction module, outputting a visual state space image feature corresponding to the visual state space feature extraction module, includes:

[0041] perform layer normalization on the to-be-tested rectal cancer medical image lesion image, output a normalized image;

[0042] take the normalized image as an input of the full connection layer, output a first full connection feature, and use the depth separable convolution module to perform depth separable convolution on the first full connection feature, output a depth separable convolution feature;

[0043] perform nonlinear mapping on the depth separable convolution feature, output a first image nonlinear feature, and take the first image nonlinear feature as an input of the two-dimensional state space feature extraction module, output a two-dimensional state space feature;

[0044] perform layer normalization on the two-dimensional state space feature, output a normalized two-dimensional state space feature;

[0045] take the to-be-tested rectal cancer medical image lesion image as an input of the full connection layer, output a second full connection feature, and perform nonlinear mapping on the second full connection feature, output a second image nonlinear feature;

[0046] element-wise multiply the second image nonlinear feature and the normalized two-dimensional state space feature, output a multiplication feature, and take the multiplication feature as an input of the full connection layer, output a third full connection feature;

[0047] element-wise add the third full connection feature and the to-be-tested rectal cancer medical image lesion image, output a visual state space image feature.

[0048] Optionally, the feature decoder comprises a first multi-modal feature extraction and fusion module, a second multi-modal feature extraction and fusion module, a third multi-modal feature extraction and fusion module, and a fourth multi-modal feature extraction and fusion module; the target text image fusion attention feature comprises a first text image fusion attention feature, a second text image fusion attention feature, a third text image fusion attention feature, and a fourth text image fusion attention feature; the target text image fusion attention feature and the target text image fusion encoding feature are input into the feature decoder for decoding to output a target text image decoding feature, comprising:

[0049] The fourth multi-modal feature extraction and fusion module is adopted to perform multi-modal feature extraction and fusion on the fourth text multi-scale feature and the fourth text image fusion encoding feature to output a first text image decoding feature;

[0050] The first text image decoding feature and the fourth text image fusion attention feature are element-wise added to generate a first text image addition feature, and the first text image addition feature is up-sampled to output a first text image fusion up-sampling feature;

[0051] The third multi-modal feature extraction and fusion module is adopted to perform multi-modal feature extraction and fusion on the third text multi-scale feature and the first text image fusion up-sampling feature to output a second text image decoding feature;

[0052] The second text image decoding feature and the third text image fusion attention feature are element-wise added to generate a second text image addition feature, and the second text image addition feature is up-sampled to output a second text image fusion up-sampling feature;

[0053] The second multi-modal feature extraction and fusion module is adopted to perform multi-modal feature extraction and fusion on the second text multi-scale feature and the second text image fusion up-sampling feature to output a third text image decoding feature;

[0054] The third text image decoding feature and the second text image fusion attention feature are element-wise added to generate a third text image addition feature, and the third text image addition feature is up-sampled to output a third text image fusion up-sampling feature;

[0055] The first multi-modal feature extraction and fusion module is adopted to perform multi-modal feature extraction and fusion on the first text multi-scale feature and the third text image fusion up-sampling feature to output a fourth text image decoding feature;

[0056] Element-wise addition is performed on the fourth text image decoding feature and the first text image fusion attention feature to generate a target text image decoding feature.

[0057] Optionally, the model training process of the deep learning-based rectal cancer medical image lesion segmentation model specifically includes:

[0058] The training rectal cancer medical image lesion image and the training clinical text information corresponding to the training rectal cancer medical image lesion image are acquired, and the training rectal cancer medical image lesion image is subjected to data enhancement to output a target training rectal cancer medical image lesion image.

[0059] The target training rectal cancer medical image lesion image and the training clinical text information are input into the deep learning-based rectal cancer medical image lesion segmentation model to be trained to output a training segmentation result and a training clinical text encoding feature.

[0060] The training segmentation result and the training clinical text encoding feature are substituted into a preset loss function and are derived to output a model gradient.

[0061] The model gradient is used to update the model parameters of the deep learning-based rectal cancer medical image lesion segmentation model to be trained to output an intermediate deep learning-based rectal cancer medical image lesion segmentation model, and the number of model updates is counted in real time.

[0062] It is judged whether the number of model updates reaches a preset training number.

[0063] If yes, the intermediate deep learning-based rectal cancer medical image lesion segmentation model is taken as a trained deep learning-based rectal cancer medical image lesion segmentation model.

[0064] The second aspect of the present application provides a deep learning-based rectal cancer medical image lesion segmentation device, which comprises:

[0065] The acquisition module is used to acquire a to-be-tested rectal cancer medical image lesion image and clinical text information corresponding to the to-be-tested rectal cancer medical image lesion image.

[0066] The input module is used to input the to-be-tested rectal cancer medical image lesion image and the clinical text information into a deep learning-based rectal cancer medical image lesion segmentation model, wherein the deep learning-based rectal cancer medical image lesion segmentation model comprises a text multi-scale feature extraction module, a feature encoder, a feature decoder, a scale attention module and a segmentation head.

[0067] The extraction module is used to perform text multi-scale feature extraction on the clinical text information by using the text multi-scale feature extraction module to generate a target text multi-scale feature.

[0068] an encoding module configured to encode the target text multi-scale feature and the rectal cancer medical image lesion image to be tested by the feature encoder, and output a target text image fusion encoding feature;

[0069] a scale module configured to take the target text image fusion encoding feature as an input of the scale attention module, and output a target text image fusion attention feature;

[0070] a decoding module configured to input the target text image fusion attention feature and the target text image fusion encoding feature into the feature decoder for decoding, and output a target text image decoding feature;

[0071] a segmentation module configured to segment the target text image decoding feature by using the segmentation head, and output a target segmentation result.

[0072] A computer device provided in the third aspect of the present application comprises a memory and a processor, the memory stores a computer program, and the computer program is executed by the processor to make the processor execute the steps of the rectal cancer medical image lesion segmentation method based on deep learning.

[0073] A computer readable storage medium provided in the fourth aspect of the present application stores a computer program, and the computer program is executed to realize the steps of the rectal cancer medical image lesion segmentation method based on deep learning.

[0074] From the above technical solutions, it can be seen that the present application has the following advantages:

[0075] The technical scheme of the present application provides a deep learning-based colorectal cancer medical image lesion segmentation method, which obtains a to-be-tested colorectal cancer medical image lesion image and clinical text information corresponding to the to-be-tested colorectal cancer medical image lesion image; inputs the to-be-tested colorectal cancer medical image lesion image and the clinical text information into a deep learning-based colorectal cancer medical image lesion segmentation model, which includes a text multi-scale feature extraction module, a feature encoder, a feature decoder, a scale attention module, and a segmentation head; the text multi-scale feature extraction module is used to perform text multi-scale feature extraction on the clinical text information to generate target text multi-scale features; the feature encoder is used to encode the target text multi-scale features and the to-be-tested colorectal cancer medical image lesion image to output target text image fusion encoding features; the target text image fusion encoding features are input into the scale attention module as input to output target text image fusion attention features; the target text image fusion attention features and the target text image fusion encoding features are input into the feature decoder for decoding to output target text image decoding features; the segmentation head is used to segment the target text image decoding features to output a target segmentation result; based on the above scheme, the deep learning-based colorectal cancer medical image lesion segmentation model is used to process the to-be-tested colorectal cancer medical image lesion image and the clinical text information, which can fully utilize medical image data and clinical text data and effectively explore potential correlations between different modalities, thereby improving the segmentation effect. BRIEF DESCRIPTION OF DRAWINGS

[0076] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0077] Figure 1 A step flow chart of a deep learning-based colorectal cancer medical image lesion segmentation method provided for the first embodiment of the present application;

[0078] Figure 2 A structure schematic diagram of a deep learning-based colorectal cancer medical image lesion segmentation model provided for the first embodiment of the present application;

[0079] Figure 3 A structure schematic diagram of a text multi-scale feature extraction module provided for the first embodiment of the present application;

[0080] Figure 4 A structure schematic diagram of a multi-modal feature extraction and fusion module provided for the first embodiment of the present application;

[0081] Figure 5 A structural schematic diagram of a visual state space feature extraction module provided for the first embodiment of the present application is shown in FIG. 1.

[0082] Figure 6 A structural schematic diagram of a two-dimensional state space feature extraction module provided for the first embodiment of the present application is shown in FIG. 2.

[0083] Figure 7 A structural schematic diagram of a cross-attention module provided for the first embodiment of the present application is shown in FIG. 3.

[0084] Figure 8 A structural schematic diagram of a channel attention module provided for the first embodiment of the present application is shown in FIG. 4.

[0085] Figure 9 A structural schematic diagram of a scale attention module provided for the first embodiment of the present application is shown in FIG. 5.

[0086] Figure 10 A structural schematic diagram of a multi-core space attention module provided for the first embodiment of the present application is shown in FIG. 6.

[0087] Figure 11 A step flowchart of a model training process of a rectal cancer medical image lesion segmentation model based on deep learning provided for the second embodiment of the present application is shown in FIG. 7.

[0088] Figure 12 A structural block diagram of a rectal cancer medical image lesion segmentation device based on deep learning provided for the third embodiment of the present application is shown in FIG. 8.

[0089] In the figure, the represents matrix multiplication, represents element-wise addition, represents element-wise multiplication, represents feature concatenation, represents an activation function. DETAILED DESCRIPTION

[0090] The embodiments of the present application provide a rectal cancer medical image lesion segmentation method and device based on deep learning, which are used to solve the technical problem that the existing rectal cancer medical image lesion segmentation method based on deep learning only takes medical images as a single input modality, fails to fully fuse auxiliary information such as clinical text reports, ignores the potential semantic association between multi-modal data, and leads to poor segmentation effect.

[0091] In order to make the application purposes, features and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present application. Obviously, the following described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0092] Please refer to Figure 1 , Figure 1 The step flow chart of a deep learning-based rectal cancer medical image lesion segmentation method provided for Embodiment One of the present application.

[0093] The deep learning-based rectal cancer medical image lesion segmentation method provided by the present application comprises:

[0094] Step 101: Obtain the rectal cancer medical image lesion image to be tested and the clinical text information corresponding to the rectal cancer medical image lesion image to be tested.

[0095] The clinical text information includes patient information, alpha-fetoprotein level, carcinoembryonic antigen value and other data.

[0096] Step 102: Input the rectal cancer medical image lesion image to be tested and the clinical text information into the deep learning-based rectal cancer medical image lesion segmentation model, which comprises a text multi-scale feature extraction module, a feature encoder, a feature decoder, a scale attention module and a segmentation head.

[0097] It should be noted that, please refer to Figure 2 The deep learning-based rectal cancer medical image lesion segmentation model proposed by the present application is composed of a text multi-scale feature extraction module, a feature encoder, a feature decoder, a scale attention module and a segmentation head (final segmentation head). The feature encoder and the feature decoder are both composed of four identical multi-modal feature extraction and fusion modules (i.e. a first multi-modal feature extraction and fusion module, a second multi-modal feature extraction and fusion module, a third multi-modal feature extraction and fusion module and a fourth multi-modal feature extraction and fusion module).

[0098] Step 103: The text multi-scale feature extraction module is used to perform text multi-scale feature extraction on the clinical text information to generate target text multi-scale features.

[0099] The text multi-scale feature extraction module comprises a text encoder, a first multi-layer perceptron, a first state space model, a second multi-layer perceptron, a second state space model, a third multi-layer perceptron, a third state space model, a fourth multi-layer perceptron and a fourth state space model.

[0100] The target text multi-scale features include a first text multi-scale feature (scale of H1xW1xC1), a second text multi-scale feature (scale of H2xW2xC2), a third text multi-scale feature (scale of H3xW3xC3), and a fourth text multi-scale feature (scale of H4xW4xC4), wherein H represents height, W represents width, and C represents the number of channels.

[0101] Specifically, the step 103 can include the following sub-steps S31-S36:

[0102] Step S31, text encoding is performed on the clinical text information by using a text encoder, and clinical text encoding features are output;

[0103] Step S32, the clinical text encoding features are taken as inputs of a first multi-layer perception, a second multi-layer perception, a third multi-layer perception, and a fourth multi-layer perception respectively, and first text perception features, second text perception features, third text perception features, and fourth text perception features are output;

[0104] Step S33, spatial modeling is performed on the first text perception features by using a first state space model, and first text state features are generated;

[0105] Step S34, spatial modeling is performed on the second text perception features by using a second state space model, and second text state features are generated;

[0106] Step S35, spatial modeling is performed on the third text perception features by using a third state space model, and third text state features are generated;

[0107] Step S36, spatial modeling is performed on the fourth text perception features by using a fourth state space model, and fourth text state features are generated;

[0108] Step S37, the first text state features, the second text state features, the third text state features, and the fourth text state features are broadcasted pixel by pixel respectively, and first text multi-scale features, second text multi-scale features, third text multi-scale features, and fourth text multi-scale features are generated.

[0109] It should be noted that, please refer to Figure 3The text multi-scale feature extraction module is composed of a text encoder (CLIP, Contrastive Language-Image Pretraining), four multi-layer perceptrons (i.e., a first multi-layer perceptron, a second multi-layer perceptron, a third multi-layer perceptron, and a fourth multi-layer perceptron), and four state space models (i.e., a first state space model, a second state space model, a third state space model, and a fourth state space model). The scales (model parameter quantities) of each multi-layer perceptron are different from each other, and the direction sensitivity of each state space model is different from each other, which can be set by a person skilled in the art as needed. In the formula, H Figure 3 i ×W i ×C i represents the height, width, and channel number corresponding to the i-th scale.

[0110] Further, for the text multi-scale feature extraction module, the medical clinical text data (clinical text information) is processed by the Clip large language pre-training model to obtain text embedding (embedding), i.e., clinical text encoding features. The text embedding is processed by four text multi-scale feature extraction modules with different scales, respectively. Each text multi-scale feature extraction module is composed of a learnable multi-layer perceptron and a state space model to improve the feature expression ability (high-level global semantics and low-level local details) of different scales and enhance the matching ability of the model to image information and text information at different scales.

[0111] It is worth mentioning that the state space model is used for modeling, and the calculation component enables the model to capture global image information while maintaining low parameter quantity and calculation cost. For the processing principle of the state space model, it can be represented as:

[0112] ;

[0113] wherein A, B, and C are learnable parameters. is an input feature, is an output feature, is a current hidden state (system memory), is a hidden state updated next time. A represents a state transition matrix: describing the evolution mode of system memory, determining how the previous state affects the next state. B is an input action matrix: describing how the input affects the system state, controlling the relationship between the input and the state. C is a state readout matrix: converting the internal hidden state into the final output.

[0114] In step 104, the target text multi-scale feature and the to-be-detected rectal cancer medical image lesion image are encoded by the feature encoder to output a target text image fusion encoding feature.

[0115] ​The feature encoder comprises a first multi-modal feature extraction and fusion module, a second multi-modal feature extraction and fusion module, a third multi-modal feature extraction and fusion module, and a fourth multi-modal feature extraction and fusion module.

[0116] The target text image fusion coding features comprise first text image fusion coding features, second text image fusion coding features, third text image fusion coding features, and fourth text image fusion coding features.

[0117] It should be noted that for the feature encoder: the main purpose of this part is to extract features of different scales for the image features through hierarchical downsampling. The encoder accepts medical image pictures (rectal cancer medical image lesion images to be measured) as image input, and accepts text features obtained through the text multi-scale feature extraction module (target text multi-scale features) as text input, uses the multi-modal feature extraction and fusion module to fuse the image features and the text features at different scales, and transmits the features at different scales to the scale attention module through the skip connection for further feature selection, and then transmits them to the corresponding layers of the feature decoder, thereby improving the feature reuse rate, accelerating the training convergence speed, and improving the overall performance of the model.

[0118] Specifically, step 104 can include the following sub-steps S41-S47:

[0119] Step S41, using the first multi-modal feature extraction and fusion module to perform multi-modal feature extraction and fusion on the first text multi-scale feature and the rectal cancer medical image lesion image to be measured, and outputting the first text image fusion coding feature;

[0120] The first multi-modal feature extraction and fusion module comprises a visual state space feature extraction module, a multi-core space attention module, a 3*3 convolution layer, a cross attention module, a multi-layer perception, and a channel attention module.

[0121] It should be noted that please refer to Figure 4 The structure of each multi-modal feature extraction and fusion module in the feature encoder and the feature decoder is the same, and is composed of a visual state space feature extraction module, a multi-core space attention module, a 3*3 convolution layer, a cross attention module, a multi-layer perception, and a channel attention module; wherein, Figure 4 The image features / input images in the above can refer to the rectal cancer medical image lesion image to be measured, the first text image fusion down-sampling feature, the second text image fusion down-sampling feature, or the third text image fusion down-sampling feature, Figure 4 The text features in the above can refer to the first text multi-scale feature, the second text multi-scale feature, the third text multi-scale feature, or the fourth text multi-scale feature.

[0122] Further, the main purpose of the multi-modal feature extraction and fusion module is to extract and fuse multi-modal features (images and texts) as the main feature extraction and calculation unit of the visual state space model based on state space modeling. As shown in Figure 4 For example, taking the processing process of the first multi-modal feature extraction and fusion module as an example, the image features are divided into three branches. The image of the test rectal cancer medical image lesion is extracted by the visual state space model to obtain the global image features (i.e., visual state space image features), and the image of the test rectal cancer medical image lesion is extracted by the multi-kernel spatial attention module to obtain the local image features (spatial attention image features). The test rectal cancer medical image lesion is connected to the last as a residual. The global and local features are fused to obtain image fusion features, which are then divided into three branches, and the text features are divided into two branches. After the image fusion features and the first text multi-scale features are spliced, they are input into a learnable multi-layer convolution to obtain fusion features A (image-text convolution features). The image fusion features and the first text multi-scale features are input into the cross-attention to learn the correlation between the image and the text, and obtain fusion features B (image-text cross-attention features). After the fusion features A and the fusion features B are spliced, they are sent to a multi-layer perceptron to obtain image-text perception features. After the image-text perception features and the image fusion features are fused (element-wise multiplication), they are further processed by the channel attention to obtain fusion features E (image-text multiplication features). Finally, the image-text channel attention features, the image-text perception features, and the image of the test rectal cancer medical image lesion are fused (element-wise addition) to obtain the output image features (first text-image fusion encoding features). It is noted that the multi-modal feature extraction and fusion module will be performed at different sizes to capture features at different resolutions.

[0123] Further, the step S41 can include the following sub-steps S411-S418:

[0124] Step S411, the image of the test rectal cancer medical image lesion is input into the visual state space feature extraction module and the multi-kernel spatial attention module respectively, and the visual state space image features corresponding to the visual state space feature extraction module and the spatial attention image features corresponding to the multi-kernel spatial attention module are output.

[0125] It should be noted that the processing principle of the multi-core spatial attention module is that: the input features input into the multi-core spatial attention module are respectively subjected to maximum pooling and average pooling operations to obtain corresponding features, the two groups of features are subjected to feature splicing, then subjected to convolution operations with different core sizes (7x7, 5x5, 3x3), subjected to addition fusion, and subjected to nonlinear mapping by using an activation function to obtain spatial attention features, and finally the spatial attention features are combined with the original input features (element-wise multiplication) to output features subjected to spatial attention adjustment, so as to realize attention enhancement of the input features in the spatial dimension; based on the foregoing principle, the rectal cancer medical image lesion image to be tested is input into the multi-core spatial attention module for processing to obtain spatial attention image features.

[0126] Optionally, the visual state space feature extraction module includes a full connection layer, a depth separable convolution module, and a two-dimensional state space feature extraction module; the process of taking the rectal cancer medical image lesion image to be tested as the input of the visual state space feature extraction module and outputting the visual state space image features corresponding to the visual state space feature extraction module can be executed through the following steps S1-S7:

[0127] Step S1, performing layer normalization on the rectal cancer medical image lesion image to be tested to output a normalized image;

[0128] Step S2, taking the normalized image as the input of the full connection layer to output first full connection features, and adopting the depth separable convolution module to perform depth separable convolution on the first full connection features to output depth separable convolution features;

[0129] Step S3, performing nonlinear mapping on the depth separable convolution features to output first image nonlinear features, and taking the first image nonlinear features as the input of the two-dimensional state space feature extraction module to output two-dimensional state space features;

[0130] Step S4, performing layer normalization on the two-dimensional state space features to generate normalized two-dimensional state space features;

[0131] Step S5, taking the rectal cancer medical image lesion image to be tested as the input of the full connection layer to output second full connection features, and performing nonlinear mapping on the second full connection features to output second image nonlinear features;

[0132] Step S6, performing element-wise multiplication on the second image nonlinear features and the normalized two-dimensional state space features to output multiplication features, and taking the multiplication features as the input of the full connection layer to output third full connection features;

[0133] Step S7, performing element-wise addition on the third full connection features and the rectal cancer medical image lesion image to be tested to output visual state space image features.

[0134] It should be noted that, please refer to Figure 5 The visual state space feature extraction module is mainly composed of three full connection layers, a deep separable convolution module and a two-dimensional state space feature extraction module; wherein, in the visual state space feature extraction module, an activation function is used for nonlinear mapping of the features, and a person skilled in the art can select a corresponding activation function according to the needs, and the present application does not limit this.

[0135] Further, please refer to Figure 6 The two-dimensional state space feature extraction module is mainly composed of four state space models with different directional sensitivities. For the processing principle of the two-dimensional state space feature extraction module, firstly, the features input to the two-dimensional state space feature extraction module are transformed in different ways to obtain four different arrangements of feature data, then the four different arrangements of feature data are input to the corresponding state space models for processing, and then element-by-element addition is performed to obtain the output features processed by the two-dimensional state space feature extraction module; wherein, the directional sensitivity refers to the difference in the capturing, responding or processing ability of a model or system to the specific direction features in the input data.

[0136] Step S412, element-by-element addition is performed on the visual state space image features and the spatial attention image features, and image fusion features are output;

[0137] Step S413, feature splicing is performed on the image fusion features and the first text multi-scale features, and first image text splicing features are output;

[0138] Step S414, 3x3 convolution layers are used to perform convolution operation on the first image text splicing features, and image text convolution features are output;

[0139] Step S415, the image fusion features and the first text multi-scale features are taken as inputs of the cross attention module, image text cross features are output, and the image text convolution features are nonlinearly mapped, and image text nonlinear features are output;

[0140] It should be noted that, please refer to Figure 7For the processing principle of the cross attention module, the text feature input into the cross attention module is taken as a query (q), and the image fusion feature input into the cross attention module is taken as a key (k) and a value (v) respectively. The matrix multiplication operation is performed on q and k, and then the non-linear mapping is performed on the result through an activation function, and then the matrix multiplication operation is performed on the result and v, and finally the multilayer perceptron (MLP, Multilayer Perceptron) processing is performed, so as to realize the interaction and information integration between the text feature and the image fusion feature, and capture the correlation between them. Based on the processing principle, the cross attention module is used to process the image fusion feature and the first text multi-scale feature, so as to obtain the image-text cross feature.

[0141] In step S416, the image-text nonlinear feature and the image-text cross feature are spliced, the second image-text spliced feature is output, and the second image-text spliced feature is taken as the input of the multilayer perceptron, and the image-text perception feature is output.

[0142] In step S417, the image-text perception feature and the image fusion feature are multiplied element by element, and the image-text multiplication feature is output.

[0143] In step S418, the image-text multiplication feature is taken as the input of the channel attention module, the image-text channel attention feature is output, and the image-text channel attention feature, the image-text perception feature and the to-be-detected rectal cancer medical image lesion image are added element by element, and the first text-image fusion encoding feature is output.

[0144] It should be noted that, please refer to Figure 8 For the processing principle of the channel attention module, the input features input into the channel attention module are subjected to the maximum pooling and average pooling operations to obtain corresponding features, the maximum pooled features and the average pooled features are added and subjected to the non-linear mapping through the activation function after being processed by the shared multilayer perceptron, the channel attention feature is obtained, and finally the input features are multiplied element by element to obtain the final output feature, so as to realize the attention adjustment of the input features in the channel dimension and highlight the important channel information. Based on the processing principle, the channel attention module is used to process the image-text multiplication feature, and the image-text channel attention feature is obtained.

[0145] In step S42, the first text-image fusion encoding feature is down-sampled to generate the first text-image fusion down-sampled feature.

[0146] In step S43, the second multi-modal feature extraction and fusion module is used to perform the multi-modal feature extraction and fusion on the second text multi-scale feature and the first text-image fusion down-sampled feature, and the second text-image fusion encoding feature is output.

[0147] Step S44, the second text image fusion coding features are down-sampled to generate second text image fusion down-sampling features;

[0148] Step S45, the third text multi-scale features and the second text image fusion down-sampling features are subjected to multi-modal feature extraction and fusion by using a third multi-modal feature extraction and fusion module to output third text image fusion coding features;

[0149] Step S46, the third text image fusion coding features are down-sampled to generate third text image fusion down-sampling features;

[0150] Step S47, the fourth text multi-scale features and the third text image fusion down-sampling features are subjected to multi-modal feature extraction and fusion by using a fourth multi-modal feature extraction and fusion module to output fourth text image fusion coding features.

[0151] It should be noted that the feature processing principles of the second multi-modal feature extraction and fusion module, the third multi-modal feature extraction and fusion module and the fourth multi-modal feature extraction and fusion module are consistent with the feature processing principle of the first multi-modal feature extraction and fusion module, and the present application will not be described in more detail.

[0152] Step 105, the target text image fusion coding features are taken as the input of the scale attention module to output target text image fusion attention features.

[0153] The target text image fusion attention features include first text image fusion attention features, second text image fusion attention features, third text image fusion attention features and fourth text image fusion attention features.

[0154] It should be noted that please refer to Figure 9The scale attention module is mainly composed of four multi-core spatial attention modules and four channel attention modules. The scale attention module selectively enhances the features from the encoder through the two sub-modules of the channel attention and the spatial attention connected in series, respectively models the information importance from the channel dimension and the spatial dimension of the feature map, and highlights the key semantic information and suppresses the redundant noise. The channel attention module uses global average pooling and maximum pooling to globally describe the feature map, and generates channel weights through a shared multi-layer perceptron, thereby emphasizing the more discriminative channels in the global context. The multi-core spatial attention module generates a spatial attention map by fusing the statistical information in the channel dimension, guiding the model to focus on the salient regions. By integrating the scale attention mechanism in the skip connection, the model can adaptively filter important features during multi-scale semantic fusion, effectively improving the quality of feature transmission and the reconstruction ability of the decoder. Specifically, each feature from different layers of the encoder (different sizes) is respectively processed by the channel attention and the multi-core spatial attention, and four features of different sizes are obtained. The four features are simultaneously upsampled, downsampled, and directly mapped, and each feature obtains four new features of different sizes. A total of 16 features are obtained (four scales and four features for each scale), then the features of the same scale are multiplied element by element, and the multiplied features are added element by element with the original features of the same size from the encoder, and finally transmitted to the corresponding decoder layer.

[0155] Specifically, for the processing process of the scale attention module, the multi-core spatial attention module is used to perform spatial dimension feature capturing operations on the first text image fusion encoding feature, the second text image fusion encoding feature, the third text image fusion encoding feature, and the fourth text image fusion encoding feature, respectively, to output the first spatial dimension feature, the second spatial dimension feature, the third spatial dimension feature, and the fourth spatial dimension feature. The channel attention module is used to perform channel dimension importance weight learning operations on the first spatial dimension feature, the second spatial dimension feature, the third spatial dimension feature, and the fourth spatial dimension feature, respectively, to output the first channel dimension importance feature, the second channel dimension importance feature, the third channel dimension importance feature, and the fourth channel dimension importance feature.

[0156] Then, the first channel dimension importance feature is directly mapped and downsampled in three different scales to obtain four features of different sizes A, A', A'', and A''', and the second channel dimension importance feature is upsampled, directly mapped, and downsampled in two different scales to obtain four features of different sizes B, B', B'', and B''',

[0157] The third channel dimension importance feature is up-sampled, directly mapped and down-sampled in two different scales to obtain four features C, C', C'' and C''' of different sizes, the fourth channel dimension importance feature is up-sampled in three different scales and directly mapped to obtain four features D, D', D'' and D''' of different sizes, then the features A, B, C and D of the same size are subjected to smoothing convolution, and then element-wise multiplication is performed, and then the multiplied result is added to the original feature of the same size from the encoder to obtain the first text image fusion attention feature, and the second text image fusion attention feature, the third text image fusion attention feature and the fourth text image fusion attention feature can be obtained in the same way.

[0158] Further, please refer to Figure 10 For the processing principle of the multi-core spatial attention module, the input features input into the multi-core spatial attention module are subjected to maximum pooling and average pooling operations to obtain corresponding features, the two groups of features are subjected to feature splicing, then subjected to convolution operations with different core sizes (7x7, 5x5 and 3x3), added and fused, and subjected to nonlinear mapping by using an activation function to obtain a spatial attention feature, and finally the spatial attention feature is combined with the original input feature (element-wise multiplication) to output a feature adjusted by spatial attention, thereby achieving attention enhancement of the input feature in the spatial dimension.

[0159] Step 106, input the target text image fusion attention feature and the target text image fusion encoding feature into the feature decoder for decoding to output a target text image decoding feature.

[0160] The feature decoder includes a first multi-modal feature extraction and fusion module, a second multi-modal feature extraction and fusion module, a third multi-modal feature extraction and fusion module and a fourth multi-modal feature extraction and fusion module.

[0161] Specifically, step 106 can include the following sub-steps S61-S68:

[0162] Step S61, the fourth multi-modal feature extraction and fusion module is used to perform multi-modal feature extraction and fusion on the fourth text multi-scale feature and the fourth text image fusion encoding feature to output a first text image decoding feature;

[0163] Step S62, element-wise addition is performed on the first text image decoding feature and the fourth text image fusion attention feature to generate a first text image addition feature, and the first text image addition feature is up-sampled to output a first text image fusion up-sampling feature;

[0164] Step S63, the third multi-modal feature extraction and fusion module is adopted to perform multi-modal feature extraction and fusion on the third text multi-scale feature and the first text image fusion up-sampling feature, and second text image decoding feature is output.

[0165] Step S64, the second text image decoding feature and the third text image fusion attention feature are added element by element to generate a second text image addition feature, and the second text image addition feature is up-sampled to output a second text image fusion up-sampling feature.

[0166] Step S65, the second multi-modal feature extraction and fusion module is adopted to perform multi-modal feature extraction and fusion on the second text multi-scale feature and the second text image fusion up-sampling feature, and third text image decoding feature is output.

[0167] Step S66, the third text image decoding feature and the second text image fusion attention feature are added element by element to generate a third text image addition feature, and the third text image addition feature is up-sampled to output a third text image fusion up-sampling feature.

[0168] Step S67, the first multi-modal feature extraction and fusion module is adopted to perform multi-modal feature extraction and fusion on the first text multi-scale feature and the third text image fusion up-sampling feature, and fourth text image decoding feature is output.

[0169] Step S68, the fourth text image decoding feature and the first text image fusion attention feature are added element by element to generate a target text image decoding feature.

[0170] It should be noted that the structure of the feature decoder is similar to that of the feature encoder. The main purpose is to gradually restore the size of the feature to the same size as the input through hierarchical up-sampling, while accepting the skip connection features from the encoder after scale attention processing to prevent gradient disappearance, enhance model robustness, and output the target region segmentation annotation in the last output layer.

[0171] Step 107, the segmentation head is adopted to segment the target text image decoding feature to output a target segmentation result.

[0172] It should be noted that the segmentation head is adopted to segment the target text image decoding feature to output a target segmentation result.

[0173] As a comparison of technical effects, reference can be made in combination with the prior art. In the existing technology, Unet based on CNN (Convolutional Neural Network) and transunet based on Transformer are two implementation schemes close to the present application. The Unet model first proposes a symmetrical encoder-decoder structure and a skip connection. The encoder can capture features of different scales through stepwise downsampling, the decoder restores spatial information through stepwise upsampling, and the skip connection design avoids feature loss in the downsampling process, helping the model to retain finer spatial information. The essential characteristics of CNN enable the network to effectively capture local information, but it is not good at capturing long-range global information. The transUnet introduces a Transformer module based on the Unet to enhance the ability to capture long-range global information, but the essential characteristics of the Transformer increase the computational complexity and cost of the model. At the same time, the existing technology only takes image data as model input, meaning that only image information is used, and the potential correlation between clinical text information and image information multimodal data is ignored.

[0174] Based on the above, the existing rectal cancer medical image lesion segmentation method has the following shortcomings: 1) It is difficult to balance low parameter quantity and high performance: In the existing technology, a low parameter quantity model usually limits the expression ability and performance of the model, so that the model performs poorly on complex tasks. While increasing the parameter quantity can improve the performance, it also brings higher computational cost and overfitting risk, and it is challenging to balance the two. 2) Cannot fully utilize multimodal information: The existing technology cannot fully utilize medical image data and clinical text data, and cannot effectively explore the potential correlation between different modalities and extract and fuse multimodal features, resulting in insufficient or inefficient information utilization. This limits the performance of the model in actual medical scenarios and fails to realize the potential of multimodal data.

[0175] To solve the above problems, the present application provides a rectal cancer medical image lesion segmentation method based on deep learning, which can solve the technical problems that the existing methods cannot fully utilize the multi-modal data in the real medical scene and realize the low-parameter high-performance segmentation of the target region. The present application can fully utilize medical image data and clinical text data, explore the potential correlation between medical text information and image information, and extract multi-modal fusion features to improve the accuracy and generalization of the rectal cancer lesion automatic segmentation model, thereby better meeting the demand for fast automatic labeling of rectal cancer lesions in real medical scenarios. Specifically, the present application uses a new visual state space model module based on state space modeling as the main feature extraction and calculation unit, which can effectively extract visual features at different scales and fuse visual and text features at different scales to enhance the feature expression ability and balance low parameters and high performance. In addition, the text information is encoded by a large language pre-training model to obtain a general text embedding. After the text embedding is processed by a learnable text multi-scale feature extraction module, it is constantly close to the image features in high-dimensional space, and the correlation between the image and the text is learned.

[0176] In the embodiment of the present application, a rectal cancer medical image lesion segmentation method based on deep learning is provided, which obtains a to-be-measured rectal cancer medical image lesion image and clinical text information corresponding to the to-be-measured rectal cancer medical image lesion image; inputs the to-be-measured rectal cancer medical image lesion image and the clinical text information into a rectal cancer medical image lesion segmentation model based on deep learning, the rectal cancer medical image lesion segmentation model based on deep learning includes a text multi-scale feature extraction module, a feature encoder, a feature decoder, a scale attention module and a segmentation head; the text multi-scale feature extraction module is used for text multi-scale feature extraction of the clinical text information to generate target text multi-scale features; the feature encoder is used for encoding the target text multi-scale features and the to-be-measured rectal cancer medical image lesion image to output target text image fusion encoding features; the target text image fusion encoding features are input into the scale attention module as input to output target text image fusion attention features; the target text image fusion attention features and the target text image fusion encoding features are input into the feature decoder for decoding to output target text image decoding features; the segmentation head is used for segmenting the target text image decoding features to output a target segmentation result; based on the above scheme, the rectal cancer medical image lesion segmentation model based on deep learning is used to process the to-be-measured rectal cancer medical image lesion image and the clinical text information, which can fully utilize medical image data and clinical text data, effectively explore the potential correlation between different modalities, and thus improve the segmentation effect.

[0177] For better illustration, refer to Figure 11, a step flow chart of a model training process of the rectal cancer medical image lesion segmentation model based on deep learning provided by the second embodiment of the present application is shown, which can include the following steps:

[0178] Step 1101, obtain training rectal cancer medical image lesion images and training clinical text information corresponding to the training rectal cancer medical image lesion images, and perform data enhancement on the training rectal cancer medical image lesion images to output target training rectal cancer medical image lesion images.

[0179] It should be noted that, in order to simulate the speckle, jitter that may occur in the real medical image imaging process, the present application uses the difusion diffusion model to randomly perform data enhancement on part of the data samples to simulate these noises, and at the same time applies random rotation, flipping, occlusion, scaling and other data enhancement methods to enhance the robustness of the model.

[0180] Step 1102, input the target training rectal cancer medical image lesion images and the training clinical text information into the rectal cancer medical image lesion segmentation model based on deep learning to be trained, and output the training segmentation result and the training clinical text encoding feature.

[0181] Step 1103, substitute the training segmentation result and the training clinical text encoding feature into the preset loss function and derive to output the model gradient.

[0182] Step 1104, update the model parameters of the rectal cancer medical image lesion segmentation model based on deep learning to be trained using the model gradient, output the intermediate rectal cancer medical image lesion segmentation model based on deep learning, and real-time statistics the number of model updates.

[0183] Step 1105, determine whether the number of model updates reaches the preset training number.

[0184] Step 1106, if yes, the intermediate rectal cancer medical image lesion segmentation model based on deep learning is used as the trained rectal cancer medical image lesion segmentation model based on deep learning.

[0185] The model gradient includes a first model gradient and a second model gradient.

[0186] It should be noted that, according to the training strategy from easy to difficult, according to the strategy of large area, simple structure, no noise→small area, complex structure, noise, more difficult training data is gradually put in. The purpose is to learn simple general features faster, and then learn difficult special features, so as to improve the convergence speed and robustness of the model.

[0187] Further, the present application applies 0.2*cross-entropy+0.3*Dice+0.5*InfoNCE in the text multi-scale feature extraction module to better perceive the matching of images and texts. The cross-attention module and the multi-layer perception in the multi-modal feature extraction and fusion module apply 0.3*cross-entropy+0.5*Dice+0.2*InfoNCE, and the matching of images and texts is not forgotten in the multi-modal fusion process.

[0188] Specifically, the preset loss function includes two parts, i.e., formula 1: 0.2*cross-entropy+0.3*Dice+0.5*InfoNCE, and formula 2: 0.3*cross-entropy+0.5*Dice+0.2*InfoNCE, wherein the cross-entropy (Cross-Entropy) is one of the most commonly used loss functions in classification tasks, which measures the difference between two probability distributions, and is commonly used to measure the gap between the classification output of the neural network and the true label. The Dice loss function (Dice Loss) is commonly used in image segmentation tasks, especially in medical images to deal with the problem of class imbalance, and the goal is to maximize the overlap of the predicted area and the real area. InfoNCE is a loss function for contrastive learning (contrastive learning), which constructs a contrastive target to let the model learn to bring the positive sample pair closer, while pushing the negative sample pair away.

[0189] Further, the training data is input into the deep learning-based rectal cancer medical image lesion segmentation model to be trained, and a training segmentation result and a training clinical text encoding feature are output. The training segmentation result and the training clinical text encoding feature are substituted into formula 1 and differentiated to obtain a first model gradient. The first model gradient is used to update the model parameters of the text multi-scale feature extraction module in the deep learning-based rectal cancer medical image lesion segmentation model to be trained. The training segmentation result and the training clinical text encoding feature are substituted into formula 2 and differentiated to obtain a second model gradient. The second model gradient is used to update the model parameters of all multi-modal feature extraction and fusion modules in the deep learning-based rectal cancer medical image lesion segmentation model to be trained, to obtain an intermediate deep learning-based rectal cancer medical image lesion segmentation model. When the number of model updates reaches the preset training number, the intermediate deep learning-based rectal cancer medical image lesion segmentation model is used as the trained deep learning-based rectal cancer medical image lesion segmentation model. When the number of model updates does not reach the preset training number, the intermediate deep learning-based rectal cancer medical image lesion segmentation model is used as a new deep learning-based rectal cancer medical image lesion segmentation model to be trained, and step 1102 is jumped to be executed. Until the number of model updates reaches the preset training number, the intermediate deep learning-based rectal cancer medical image lesion segmentation model determined when the number of model updates reaches the preset training number is used as the trained deep learning-based rectal cancer medical image lesion segmentation model.

[0190] It is worth mentioning that, for the calculation principle of cross-entropy , it can be expressed as:

[0191] ;

[0192] wherein, y is the true label; is the probability distribution output by the model (that is, the output result predicted by the model), corresponding to the training segmentation result; and C is the number of categories. For the calculation principle of Dice loss function

[0193] , it can be expressed as:

[0194] ;

[0195] wherein, C is the number of categories; is the prediction probability of the i-th pixel to the c-th category, corresponding to the training segmentation result; is the true label of the i-th pixel to the c-th category; is a small constant for numerical stability.

[0196] For the calculation principle of InfoNCE loss function ​The calculation principle of the similarity can be represented as:

[0197] ;

[0198] ;

[0199] Wherein, t is a text feature, corresponding to a training clinical text encoding feature, , C is a feature dimension; is the average of the pixel features of the mask region in the image, corresponding to the training segmentation result, ; is the pixel feature set of the non-mask region in the image, and K represents the number of pixel features of the non-mask region; is a temperature coefficient for scaling the similarity, and N is a batch size; is a similarity; a and b are inputs for calculating the similarity; is a norm.

[0200] In the embodiment of the present application, the multi-modal feature extraction and fusion module and the mixed attention module are designed, and the cross-entropy-Dice similarity joint loss function is introduced, which enhances the feature extraction capability of the model and improves the performance of the model. The trained deep learning-based rectal cancer medical image lesion segmentation model is used for rectal cancer medical image lesion segmentation, which can effectively capture local image information while capturing global image information with high precision and low computational cost, thereby reducing information loss in the feature extraction process and retaining more sufficient and comprehensive image information. In addition, the present application is more suitable for actual medical scenarios, and can make full use of the data and information in medical images and clinical texts in actual medical scenarios, and can be generalized to automatic segmentation of medical images of various tissues and organs.

[0201] Please refer to Figure 12 , Figure 12 is a structural block diagram of a deep learning-based rectal cancer medical image lesion segmentation device provided in the third embodiment of the present application.

[0202] The deep learning-based rectal cancer medical image lesion segmentation device provided by the present application comprises:

[0203] The acquisition module 1201 is configured to acquire a rectal cancer medical image lesion image to be tested and clinical text information corresponding to the rectal cancer medical image lesion image to be tested.

[0204] The input module 1202 is configured to input the rectal cancer medical image lesion image to be tested and the clinical text information into a deep learning-based rectal cancer medical image lesion segmentation model, wherein the deep learning-based rectal cancer medical image lesion segmentation model comprises a text multi-scale feature extraction module, a feature encoder, a feature decoder, a scale attention module and a segmentation head.

[0205] The extraction module 1203 is configured to perform text multi-scale feature extraction on the clinical text information by using a text multi-scale feature extraction module to generate target text multi-scale features.

[0206] The encoding module 1204 is configured to encode the target text multi-scale features and the to-be-detected rectal cancer medical image lesion image by using a feature encoder to output target text image fusion encoding features.

[0207] The scale module 1205 is configured to input the target text image fusion encoding features into a scale attention module to output target text image fusion attention features.

[0208] The decoding module 1206 is configured to input the target text image fusion attention features and the target text image fusion encoding features into a feature decoder for decoding to output target text image decoding features.

[0209] The segmentation module 1207 is configured to perform segmentation on the target text image decoding features by using a segmentation head to output a target segmentation result.

[0210] Further, the text multi-scale feature extraction module includes a text encoder, a first multi-layer perceptron, a first state space model, a second multi-layer perceptron, a second state space model, a third multi-layer perceptron, a third state space model, a fourth multi-layer perceptron, and a fourth state space model. The target text multi-scale features include first text multi-scale features, second text multi-scale features, third text multi-scale features, and fourth text multi-scale features. The extraction module 1203 is specifically configured to:

[0211] perform text encoding on the clinical text information by using the text encoder to output clinical text encoding features;

[0212] input the clinical text encoding features into the first multi-layer perceptron, the second multi-layer perceptron, the third multi-layer perceptron, and the fourth multi-layer perceptron respectively to output first text perception features, second text perception features, third text perception features, and fourth text perception features;

[0213] perform spatial modeling on the first text perception features by using the first state space model to generate first text state features;

[0214] perform spatial modeling on the second text perception features by using the second state space model to generate second text state features;

[0215] perform spatial modeling on the third text perception features by using the third state space model to generate third text state features;

[0216] perform spatial modeling on the fourth text perception features by using the fourth state space model to generate fourth text state features;

[0217] The first text state feature, the second text state feature, the third text state feature, and the fourth text state feature are broadcasted pixel by pixel respectively to generate a first text multi-scale feature, a second text multi-scale feature, a third text multi-scale feature, and a fourth text multi-scale feature.

[0218] Further, the feature encoder comprises a first multi-modal feature extraction and fusion module, a second multi-modal feature extraction and fusion module, a third multi-modal feature extraction and fusion module, and a fourth multi-modal feature extraction and fusion module; the target text image fusion encoding feature comprises a first text image fusion encoding feature, a second text image fusion encoding feature, a third text image fusion encoding feature, and a fourth text image fusion encoding feature; the encoding module 1204 comprises:

[0219] The first sub-module is configured to perform multi-modal feature extraction and fusion on the first text multi-scale feature and the to-be-detected rectal cancer medical image lesion image by using the first multi-modal feature extraction and fusion module, and output the first text image fusion encoding feature;

[0220] The second sub-module is configured to perform down-sampling on the first text image fusion encoding feature to generate a first text image fusion down-sampling feature;

[0221] The third sub-module is configured to perform multi-modal feature extraction and fusion on the second text multi-scale feature and the first text image fusion down-sampling feature by using the second multi-modal feature extraction and fusion module, and output the second text image fusion encoding feature;

[0222] The fourth sub-module is configured to perform down-sampling on the second text image fusion encoding feature to generate a second text image fusion down-sampling feature;

[0223] The fifth sub-module is configured to perform multi-modal feature extraction and fusion on the third text multi-scale feature and the second text image fusion down-sampling feature by using the third multi-modal feature extraction and fusion module, and output the third text image fusion encoding feature;

[0224] The sixth sub-module is configured to perform down-sampling on the third text image fusion encoding feature to generate a third text image fusion down-sampling feature;

[0225] The seventh sub-module is configured to perform multi-modal feature extraction and fusion on the fourth text multi-scale feature and the third text image fusion down-sampling feature by using the fourth multi-modal feature extraction and fusion module, and output the fourth text image fusion encoding feature.

[0226] Further, the first multi-modal feature extraction and fusion module comprises a visual state space feature extraction module, a multi-core space attention module, a 3x3 convolutional layer, a cross-attention module, a multi-layer perceptron, and a channel attention module; the first sub-module comprises:

[0227] The first unit is configured to input the to-be-tested rectal cancer medical image lesion image into the visual state space feature extraction module and the multi-core space attention module as inputs, and output visual state space image features corresponding to the visual state space feature extraction module and spatial attention image features corresponding to the multi-core space attention module.

[0228] The second unit is configured to perform element-wise addition on the visual state space image features and the spatial attention image features, and output image fusion features.

[0229] The third unit is configured to perform feature concatenation on the image fusion features and the first text multi-scale features, and output first image-text concatenated features.

[0230] The fourth unit is configured to perform convolutional operation on the first image-text concatenated features through the 3x3 convolutional layer, and output image-text convolutional features.

[0231] The fifth unit is configured to input the image fusion features and the first text multi-scale features into the cross-attention module as inputs, output image-text cross features, and perform nonlinear mapping on the image-text convolutional features, and output image-text nonlinear features.

[0232] The sixth unit is configured to perform feature concatenation on the image-text nonlinear features and the image-text cross features, output second image-text concatenated features, and input the second image-text concatenated features into the multi-layer perceptron as inputs, and output image-text perception features.

[0233] The seventh unit is configured to perform element-wise multiplication on the image-text perception features and the image fusion features, and output image-text multiplication features.

[0234] The eighth unit is configured to input the image-text multiplication features into the channel attention module as inputs, output image-text channel attention features, and perform element-wise addition on the image-text channel attention features, the image-text perception features, and the to-be-tested rectal cancer medical image lesion image, and output first text-image fusion encoding features.

[0235] Further, the visual state space feature extraction module comprises a full connection layer, a depth separable convolution module, and a two-dimensional state space feature extraction module; the first unit is specifically configured to:

[0236] perform layer normalization on the to-be-tested rectal cancer medical image lesion image, and output a normalized image;

[0237] The standardized image is input into a full connection layer to output first full connection features, and a depth separable convolution module is used to perform depth separable convolution on the first full connection features to output depth separable convolution features;

[0238] The depth separable convolution features are subjected to nonlinear mapping to output first image nonlinear features, and the first image nonlinear features are input into a two-dimensional state space feature extraction module to output two-dimensional state space features;

[0239] The two-dimensional state space features are subjected to layer normalization to generate normalized two-dimensional state space features;

[0240] The rectal cancer medical image lesion image to be tested is input into a full connection layer to output second full connection features, and the second full connection features are subjected to nonlinear mapping to output second image nonlinear features;

[0241] The second image nonlinear features and the normalized two-dimensional state space features are subjected to element-wise multiplication to output multiplication features, and the multiplication features are input into a full connection layer to output third full connection features;

[0242] The third full connection features and the rectal cancer medical image lesion image to be tested are subjected to element-wise addition to output visual state space image features.

[0243] Further, the feature decoder includes a first multi-modal feature extraction and fusion module, a second multi-modal feature extraction and fusion module, a third multi-modal feature extraction and fusion module, and a fourth multi-modal feature extraction and fusion module; the target text image fusion attention features include first text image fusion attention features, second text image fusion attention features, third text image fusion attention features, and fourth text image fusion attention features; the decoding module 1206 is specifically configured to:

[0244] The fourth multi-modal feature extraction and fusion module is used to perform multi-modal feature extraction and fusion on the fourth text multi-scale features and the fourth text image fusion encoding features to output first text image decoding features;

[0245] The first text image decoding features and the fourth text image fusion attention features are subjected to element-wise addition to generate first text image addition features, and the first text image addition features are subjected to upsampling to output first text image fusion upsampling features;

[0246] The third multi-modal feature extraction and fusion module is used to perform multi-modal feature extraction and fusion on the third text multi-scale features and the first text image fusion upsampling features to output second text image decoding features;

[0247] perform element-wise addition on the third text image decoding feature and the second text image fusion attention feature to generate a third text image addition feature, and perform upsampling on the third text image addition feature to output a third text image fusion upsampling feature;

[0248] perform element-wise addition on the third text image decoding feature and the second text image fusion attention feature to generate a third text image addition feature, and perform upsampling on the third text image addition feature to output a third text image fusion upsampling feature;

[0249] perform element-wise addition on the third text image decoding feature and the second text image fusion attention feature to generate a third text image addition feature, and perform upsampling on the third text image addition feature to output a third text image fusion upsampling feature;

[0250] perform element-wise addition on the third text image decoding feature and the second text image fusion attention feature to generate a third text image addition feature, and perform upsampling on the third text image addition feature to output a third text image fusion upsampling feature;

[0251] perform element-wise addition on the third text image decoding feature and the second text image fusion attention feature to generate a third text image addition feature, and perform upsampling on the third text image addition feature to output a third text image fusion upsampling feature.

[0252] In an alternative device embodiment, further comprising:

[0253] The first module is configured to obtain a training rectal cancer medical image lesion image and training clinical text information corresponding to the training rectal cancer medical image lesion image, and perform data enhancement on the training rectal cancer medical image lesion image to output a target training rectal cancer medical image lesion image.

[0254] The second module is configured to input the target training rectal cancer medical image lesion image and the training clinical text information into a to-be-trained rectal cancer medical image lesion segmentation model based on deep learning to output a training segmentation result and a training clinical text encoding feature.

[0255] The third module is configured to substitute the training segmentation result and the training clinical text encoding feature into a preset loss function and derive to output a model gradient.

[0256] The fourth module is configured to update model parameters of the to-be-trained rectal cancer medical image lesion segmentation model based on deep learning using the model gradient to output an intermediate rectal cancer medical image lesion segmentation model based on deep learning, and to count a model update number in real time.

[0257] The fifth module is configured to determine whether the model update number reaches a preset training number.

[0258] The sixth module is configured to, if yes, take the intermediate deep learning based rectal cancer medical image lesion segmentation model as the trained deep learning based rectal cancer medical image lesion segmentation model.

[0259] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, modules, sub-modules and units can refer to the corresponding processes in the foregoing method embodiments, and will not be described here.

[0260] The embodiment of the present application further provides a computer device, comprising a memory and a processor, the memory stores a computer program; the computer program is executed by the processor, so that the processor executes the steps of the deep learning based rectal cancer medical image lesion segmentation method according to any one of the foregoing embodiments.

[0261] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program / instruction, and the computer program / instruction is executed by a processor to implement the steps of the deep learning based rectal cancer medical image lesion segmentation method according to any one of the foregoing embodiments.

[0262] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A deep learning-based medical image lesion segmentation method for rectal cancer, characterized by, The method comprises the following steps: obtaining a medical image lesion image of a to-be-tested rectal cancer and clinical text information corresponding to the medical image lesion image of the to-be-tested rectal cancer; inputting the medical image lesion image of the to-be-tested rectal cancer and the clinical text information into a deep learning-based rectal cancer medical image lesion segmentation model, wherein the deep learning-based rectal cancer medical image lesion segmentation model comprises a text multi-scale feature extraction module, a feature encoder, a feature decoder, a scale attention module, and a segmentation head; extracting text multi-scale features of the clinical text information by using the text multi-scale feature extraction module to generate target text multi-scale features; encoding the target text multi-scale features and the medical image lesion image of the to-be-tested rectal cancer by using the feature encoder to output target text image fusion encoding features; the feature encoder comprises a first multi-modal feature extraction and fusion module, a second multi-modal feature extraction and fusion module, a third multi-modal feature extraction and fusion module, and a fourth multi-modal feature extraction and fusion module; the target text image fusion encoding features comprise first text image fusion encoding features, second text image fusion encoding features, third text image fusion encoding features, and fourth text image fusion encoding features; inputting the target text image fusion encoding features into the scale attention module as inputs to output target text image fusion attention features; decoding the target text image fusion attention features and the target text image fusion encoding features by using the feature decoder to output target text image decoding features; segmenting the target text image decoding features by using the segmentation head to output a target segmentation result; the first multi-modal feature extraction and fusion module comprises a visual state space feature extraction module, a multi-core space attention module, a 3x3 convolution layer, a cross-attention module, a multi-layer perceptron, and a channel attention module; the visual state space feature extraction module comprises a full connection layer, a depth separable convolution module, and a two-dimensional state space feature extraction module; the feature decoder comprises a first multi-modal feature extraction and fusion module, a second multi-modal feature extraction and fusion module, a third multi-modal feature extraction and fusion module, and a fourth multi-modal feature extraction and fusion module. 2.The rectal cancer medical image lesion segmentation method based on deep learning according to claim 1, characterized in that, The text multi-scale feature extraction module comprises a text encoder, a first multi-layer perceptron, a first state space model, a second multi-layer perceptron, a second state space model, a third multi-layer perceptron, a third state space model, a fourth multi-layer perceptron, and a fourth state space model; the target text multi-scale features comprise first text multi-scale features, second text multi-scale features, third text multi-scale features, and fourth text multi-scale features; the method comprises the following steps: text encoding the clinical text information by using the text encoder to output clinical text encoding features; The clinical text coding features are respectively taken as inputs of the first multi-layer perceptron, the second multi-layer perceptron, the third multi-layer perceptron, and the fourth multi-layer perceptron, and first text perception features, second text perception features, third text perception features, and fourth text perception features are outputted; The first text perception features are spatially modeled by the first state space model to generate first text state features; The second text perception features are spatially modeled by the second state space model to generate second text state features; The third text perception features are spatially modeled by the third state space model to generate third text state features; The fourth text perception features are spatially modeled by the fourth state space model to generate fourth text state features; The first text state features, the second text state features, the third text state features, and the fourth text state features are respectively pixel-by-pixel broadcasted to generate first text multi-scale features, second text multi-scale features, third text multi-scale features, and fourth text multi-scale features. 3.The rectal cancer medical image lesion segmentation method based on deep learning according to claim 2, characterized in that, The target text multi-scale features and the to-be-detected rectal cancer medical image lesion image are encoded by the feature encoder to output target text image fusion coding features, including: The first text multi-scale features and the to-be-detected rectal cancer medical image lesion image are subjected to multi-modal feature extraction and fusion by a first multi-modal feature extraction and fusion module to output first text image fusion coding features; The first text image fusion coding features are down-sampled to generate first text image fusion down-sampling features; The second text multi-scale features and the first text image fusion down-sampling features are subjected to multi-modal feature extraction and fusion by a second multi-modal feature extraction and fusion module to output second text image fusion coding features; The second text image fusion coding features are down-sampled to generate second text image fusion down-sampling features; The third text multi-scale features and the second text image fusion down-sampling features are subjected to multi-modal feature extraction and fusion by a third multi-modal feature extraction and fusion module to output third text image fusion coding features; The third text image fusion coding features are down-sampled to generate third text image fusion down-sampling features; The fourth text multi-scale features and the third text image fusion down-sampling features are subjected to multi-modal feature extraction and fusion by a fourth multi-modal feature extraction and fusion module to output fourth text image fusion coding features. 4.The rectal cancer medical image lesion segmentation method based on deep learning according to claim 3, characterized in that, The first text multi-scale features and the to-be-detected rectal cancer medical image lesion image are subjected to multi-modal feature extraction and fusion by the first multi-modal feature extraction and fusion module to output first text image fusion coding features, including: The to-be-detected rectal cancer medical image lesion image is respectively taken as input of a visual state space feature extraction module and a multi-core spatial attention module to output visual state space image features corresponding to the visual state space feature extraction module and spatial attention image features corresponding to the multi-core spatial attention module; The image fusion feature and the first text multi-scale feature are element-wise added, and an image-text splicing feature is output; The image fusion feature and the first text multi-scale feature are spliced, and a first image-text splicing feature is output; The first image-text splicing feature is convoluted through a 3x3 convolution layer, and an image-text convolution feature is output; The image fusion feature and the first text multi-scale feature are input into a cross-attention module, and an image-text cross feature is output, and the image-text convolution feature is nonlinearly mapped, and an image-text nonlinear feature is output; The image-text nonlinear feature and the image-text cross feature are spliced, and a second image-text splicing feature is output, and the second image-text splicing feature is input into a multi-layer perception, and an image-text perception feature is output; The image-text perception feature and the image fusion feature are element-wise multiplied, and an image-text multiplication feature is output; The image-text multiplication feature is input into a channel attention module, and an image-text channel attention feature is output, and the image-text channel attention feature, the image-text perception feature, and the to-be-detected rectal cancer medical image lesion image are element-wise added, and a first text-image fusion encoding feature is output. 5.The rectal cancer medical image lesion segmentation method based on deep learning according to claim 4, characterized in that, The to-be-detected rectal cancer medical image lesion image is input into a visual state space feature extraction module, and a visual state space image feature corresponding to the visual state space feature extraction module is output, including: The to-be-detected rectal cancer medical image lesion image is layer standardized, and a standardized image is output; The standardized image is input into a full connection layer, and a first full connection feature is output, and a depth separable convolution module is used to perform depth separable convolution on the first full connection feature, and a depth separable convolution feature is output; The depth separable convolution feature is nonlinearly mapped, and a first image nonlinear feature is output, and the first image nonlinear feature is input into a two-dimensional state space feature extraction module, and a two-dimensional state space feature is output; The two-dimensional state space feature is layer standardized to generate a standardized two-dimensional state space feature; The to-be-detected rectal cancer medical image lesion image is input into a full connection layer, and a second full connection feature is output, and the second full connection feature is nonlinearly mapped, and a second image nonlinear feature is output; The second image nonlinear feature and the standardized two-dimensional state space feature are element-wise multiplied, and a multiplication feature is output, and the multiplication feature is input into a full connection layer, and a third full connection feature is output; The third full connection feature and the to-be-detected rectal cancer medical image lesion image are element-wise added, and a visual state space image feature is output. 6.The rectal cancer medical image lesion segmentation method based on deep learning according to claim 3, characterized in that, The target text image fusion attention feature includes a first text image fusion attention feature, a second text image fusion attention feature, a third text image fusion attention feature and a fourth text image fusion attention feature; the target text image fusion attention feature and the target text image fusion encoding feature are input into the feature decoder for decoding, and a target text image decoding feature is output, which includes: The fourth multi-modal feature extraction and fusion module is adopted to perform multi-modal feature extraction and fusion on the fourth text multi-scale feature and the fourth text image fusion encoding feature, and a first text image decoding feature is output; The first text image decoding feature and the fourth text image fusion attention feature are element-wise added to generate a first text image addition feature, and the first text image addition feature is up-sampled to output a first text image fusion up-sampling feature; The third multi-modal feature extraction and fusion module is adopted to perform multi-modal feature extraction and fusion on the third text multi-scale feature and the first text image fusion up-sampling feature, and a second text image decoding feature is output; The second text image decoding feature and the third text image fusion attention feature are element-wise added to generate a second text image addition feature, and the second text image addition feature is up-sampled to output a second text image fusion up-sampling feature; The second multi-modal feature extraction and fusion module is adopted to perform multi-modal feature extraction and fusion on the second text multi-scale feature and the second text image fusion up-sampling feature, and a third text image decoding feature is output; The third text image decoding feature and the second text image fusion attention feature are element-wise added to generate a third text image addition feature, and the third text image addition feature is up-sampled to output a third text image fusion up-sampling feature; The first multi-modal feature extraction and fusion module is adopted to perform multi-modal feature extraction and fusion on the first text multi-scale feature and the third text image fusion up-sampling feature, and a fourth text image decoding feature is output; The fourth text image decoding feature and the first text image fusion attention feature are element-wise added to generate a target text image decoding feature. 7.The rectal cancer medical image lesion segmentation method based on deep learning according to claim 1, characterized in that, The model training process of the deep learning-based rectal cancer medical image lesion segmentation model is specifically: training rectal cancer medical image lesion images and training clinical text information corresponding to the training rectal cancer medical image lesion images are obtained, and the training rectal cancer medical image lesion images are data enhanced to output target training rectal cancer medical image lesion images; the target training rectal cancer medical image lesion images and the training clinical text information are input into the deep learning-based rectal cancer medical image lesion segmentation model to be trained to output training segmentation results and training clinical text encoding features; the training segmentation results and the training clinical text encoding features are substituted into a preset loss function and derived to output model gradients; The model gradient is used to update model parameters of the to-be-trained deep learning-based rectal cancer medical image lesion segmentation model, an intermediate deep learning-based rectal cancer medical image lesion segmentation model is output, and the number of model updates is counted in real time; It is judged whether the number of model updates reaches a preset training number; If yes, the intermediate deep learning-based rectal cancer medical image lesion segmentation model is taken as a trained deep learning-based rectal cancer medical image lesion segmentation model.

8. A deep learning-based medical image lesion segmentation apparatus for rectal cancer, applied to the deep learning-based medical image lesion segmentation method for rectal cancer of claim 1, characterized in that, Comprise: An acquisition module is configured to acquire a to-be-tested rectal cancer medical image lesion image and clinical text information corresponding to the to-be-tested rectal cancer medical image lesion image; An input module is configured to input the to-be-tested rectal cancer medical image lesion image and the clinical text information into a deep learning-based rectal cancer medical image lesion segmentation model, wherein the deep learning-based rectal cancer medical image lesion segmentation model comprises a text multi-scale feature extraction module, a feature encoder, a feature decoder, a scale attention module, and a segmentation head; An extraction module is configured to perform text multi-scale feature extraction on the clinical text information by using the text multi-scale feature extraction module to generate target text multi-scale features; An encoding module is configured to encode the target text multi-scale features and the to-be-tested rectal cancer medical image lesion image by using the feature encoder to output target text image fusion encoding features; A scale module is configured to take the target text image fusion encoding features as input of the scale attention module to output target text image fusion attention features; A decoding module is configured to input the target text image fusion attention features and the target text image fusion encoding features into the feature decoder for decoding to output target text image decoding features; A segmentation module is configured to segment the target text image decoding features by using the segmentation head to output a target segmentation result.

9. A computer device, comprising: The computer program is executed to implement the deep learning-based rectal cancer medical image lesion segmentation method according to any one of claims 1-6.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed to implement the deep learning-based rectal cancer medical image lesion segmentation method according to any one of claims 1-6.

Images