Medical image reference segmentation method and device based on multi-scale deep learning

Abstract

The application discloses a medical image reference segmentation method and device based on multi-scale deep learning, which comprises the following steps: obtaining a tumor CT image of a patient and corresponding text description; embedding the text description into several tokens and then inputting the tokens into a language encoder based on a transformer to form high-dimensional word vectors and obtain language features; according to the tumor CT image and the language features, visual feature coding and multi-modal feature fusion are performed through four stages with decreasing spatial resolution by using MSFA; for the fused visual features output by each stage, upsampling is applied by using bilinear interpolation, the multi-modal feature maps output by each stage are scaled in the number of channels to a uniform dimension, and then the multi-modal feature maps are spliced to obtain a final feature map, which is then input into a full-scale decoder for decoding to obtain a pixel-level segmentation mask of a landmark target object. The application can be used for assisting in tumor identification in medical diagnosis, thereby simplifying the process.

Description

Technical Field

[0001] This application relates to the field of image segmentation technology, and in particular to a medical image reference segmentation method and apparatus based on multi-scale deep learning. Background Technology

[0002] Image segmentation refers to the process of subdividing a digital image into multiple image sub-regions (sets of pixels). The purpose of image segmentation is to simplify or change the representation of an image, making it easier to understand and analyze. Image segmentation is commonly used to locate objects and boundaries (lines, curves, etc.) in an image.

[0003] Image segmentation with reference is a classic image-text multimodal task. Given an image and a natural language expression describing a specific object in the image, the goal is to find the object referred to by the expression and provide a segmentation mask for that object.

[0004] A key challenge behind this task is leveraging natural language expressions to highlight relevant locations within an image. One paradigm for addressing this is utilizing a powerful vision-language decoder to fuse features independently extracted by the visual and language encoders. Recent methods have made significant progress in this paradigm by utilizing the Transformer as a cross-modal decoder, while the Transformer has achieved overwhelming success in many other vision-language tasks. Analyzing the approach in this work reveals that significantly better cross-modal alignment can be achieved by fusing language and visual features earlier in the intermediate layers of the vision-transformer encoder network. Cross-modal feature fusion during the visual feature encoding stage has been shown to mine rich multimodal context, leading to more accurate segmentation results.

[0005] Image segmentation has attracted increasing attention in the research community. A traditional pipeline consists of two main processes: (1) extracting features from text and image inputs respectively, and (2) fusing multimodal features to locate the target object and predict the segmentation mask. In the first process, previous methods used recurrent neural networks and language transducers to encode the language input. For encoding visual input, DeeplabV3 and DarkNet have been used in previous methods to learn discriminative representations.

[0006] Multimodal feature fusion modules are a key component of existing technologies. For example, based on cascaded operations, various attention mechanisms are used to establish cross-modal relationships between language and visual features.

[0007] Existing image segmentation models are designed for segmentation tasks on natural images, and while they achieve good results on natural image tests, they are not well-suited for the specialized field of medical imaging. They fail to adequately understand medical images and medical terminology, thus hindering their ability to achieve satisfactory results on medical image segmentation tasks.

[0008] While existing models for tumor segmentation in medical images are effective for most regularly shaped and moderately sized tumors, their localization capabilities and segmentation accuracy decrease significantly when segmenting smaller or irregularly shaped tumors. Summary of the Invention

[0009] The purpose of this application is to provide a medical image reference segmentation method and apparatus based on multi-scale deep learning, so as to solve the technical problem that existing models perform poorly in the field of medical image segmentation.

[0010] According to a first aspect of the embodiments of this application, a method for medical image reference segmentation is provided, comprising: Obtain the patient's tumor CT images and corresponding text descriptions; The text description is embedded into several tokens, which are then input into a Transformer-based language encoder to form high-dimensional word vectors, thus obtaining language features. Based on the tumor CT image and the language features, visual feature encoding and multimodal feature fusion are performed using MSFA through four stages with decreasing spatial resolution. Specifically, this includes: taking the fused visual features output from the previous stage as input, using convolution kernels to extract deep local features, outputting the extracted visual features, simultaneously performing two multi-branch strip convolutions on the extracted visual features and fusing language features using a cross-attention mechanism, summing the results of the four branches, and then using convolution to model the relationship between different channels, which is used as the input for reweighting the attention weights of MSFA, and serving as the fused visual features output in this stage. For the fused visual features output at each stage, bilinear interpolation upsampling was applied to scale the number of channels of the multimodal feature map output at each stage to a uniform dimension, and then they were stitched together to obtain the final feature map. The final feature map is input into a full-scale decoder for decoding to obtain a pixel-level segmentation mask that marks the target object.

[0011] The formula used for visual feature encoding and multimodal feature fusion using MSFA is as follows:

[0012]

[0013]

[0014] in, This represents the previous stage features of the input. and These represent the fused visual features of the attention map and the output, respectively. Represents the term-by-term product of matrices. Represents Depth-wise convolution. Represents the first in the diagram Convolution branches, In the convolution branch, two strip convolutions are used to simulate two-dimensional convolution, and the kernel sizes of the three branches are 7, 13, and 21, respectively.

[0015] The multimodal feature map outputs from each stage are scaled to a uniform dimension, and then concatenated to obtain the final feature map, including:

[0016] in, These represent intermediate values ​​generated during the feature map stitching process at each stage. ; Represents the final feature map; This indicates feature cascading along the channel dimension; It is a projection function implemented by two 3×3 convolutions with batch normalization and ReLU nonlinear connection.

[0017] According to a second aspect of the embodiments of this application, a medical image reference image segmentation apparatus is provided, comprising: The acquisition module is used to acquire the patient's tumor CT images and corresponding text descriptions; The processing module is used to embed the text description into several tokens, and then input them into a Transformer-based language encoder to form high-dimensional word vectors and obtain language features; The fusion module is used to perform visual feature encoding and multimodal feature fusion using MSFA through four stages with decreasing spatial resolution, based on the tumor CT image and the language features. Specifically, it includes: taking the fused visual features output from the previous stage as input, using convolution kernels to extract deep local features, outputting the extracted visual features, simultaneously performing two multi-branch strip convolutions on the extracted visual features and fusing language features using a cross-attention mechanism, summing the results of the four branches, and then using convolution to model the relationship between different channels, which is used as the input for reweighting the attention weights of MSFA, and as the fused visual features output in this stage. The unified stitching module is used to fuse the visual features output from each stage. It applies bilinear interpolation upsampling to scale the number of channels of the multimodal feature map output from each stage to a uniform dimension, and then stitches them together to obtain the final feature map. The decoding output module is used to input the final feature map into the full-scale decoder for decoding to obtain a pixel-level segmentation mask that marks the target object.

[0018] According to a third aspect of the embodiments of this application, an electronic device is provided, comprising: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors perform the method as described in the first aspect.

[0019] According to a third aspect of the embodiments of this application, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the method as described in the first aspect.

[0020] The technical solutions provided by the embodiments of this application may include the following beneficial effects: The most relevant methods to this invention are VLT and EFN. VLT designs a Transformer decoder that fuses language and visual features, while EFN uses a convolutional visual backbone network to encode language information. Unlike VLT, this invention employs an early fusion scheme that effectively utilizes a Transformer encoder to model multimodal context, designs a multi-scale fusion attention module for vision-language fusion, and integrates a full-scale analysis decoder for segmentation prediction. Compared to EFN, this invention does not rely on a complex cross-modal encoder, thus obtaining a clearer and more efficient framework. In a fair comparison, the method of this invention significantly outperforms the previous two methods.

[0021] This invention designs a multi-scale fusion attention module for visual-language fusion, which improves the ability to identify tumor edges; at the same time, it uses a full-scale analysis decoder to enhance the ability to locate target tumors based on text, effectively improving the segmentation effect of difficult tumors.

[0022] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0024] Figure 1 This is a flowchart illustrating a medical image reference image segmentation method according to an exemplary embodiment.

[0025] Figure 2 This is a flowchart illustrating a medical image reference image segmentation method according to an exemplary embodiment.

[0026] Figure 3 This is a schematic diagram of an MSCA structure according to an exemplary embodiment.

[0027] Figure 4 This is a block diagram illustrating a medical image reference image segmentation device according to an exemplary embodiment. Detailed Implementation

[0028] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0029] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0030] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0031] Figure 1 and Figure 2 This is a flowchart illustrating a medical image reference image segmentation method according to an exemplary embodiment, such as... Figure 1 and Figure 2 As shown, the method may include the following steps: S1: Obtain the patient's tumor CT images and corresponding text descriptions; Specifically, the acquisition of CT images relies on the hospital's examination and recording system. These images are typically generated directly by medical equipment and stored in the hospital's database. Simultaneously, the corresponding textual descriptions originate from diagnostic reports prepared by doctors based on these CT images. In this process, particular attention is paid to handling the correspondence between text and images. Doctors' diagnostic reports are often made for each tumor visible in the image; therefore, for CT images containing multiple tumors, the textual descriptions need to be subdivided to ensure that each tumor and its corresponding description match precisely. This design allows, in practical applications, for multiple tumors in a single CT image, doctors to obtain detailed analysis results for that specific tumor by inputting a description, thereby improving the accuracy and efficiency of diagnosis.

[0032] S2: Embed the text description into several tokens, and then input them into a Transformer-based language encoder to form high-dimensional word vectors and obtain language features; Specifically, in order to extract text features, this invention uses the traditional BERT model to transform text descriptions into high-dimensional word vectors. ,in Represents the number of channels. The number of representative words is used to complete the language feature encoding. The original intention of this method is to leverage the powerful language understanding capabilities of the BERT model to accurately capture subtle semantic differences in text descriptions, which plays a crucial role in tumor localization capabilities in subsequent cross-modal fusion.

[0033] The advantage of this design is that it allows the system to better understand the doctor's diagnostic intent, combining the doctor's rich experience with the processing power of the machine learning model, thereby improving the ability to locate tumors.

[0034] S3: Based on the tumor CT image and the language features, through four stages with decreasing spatial resolution, visual feature encoding and multimodal feature fusion are performed using MSFA. Specifically, this includes: taking the fused visual features output from the previous stage as input, using convolution kernels to extract deep local features, outputting the extracted visual features, simultaneously performing two multi-branch strip convolutions on the extracted visual features and fusing language features using a cross-attention mechanism, summing the results of the four branches, and then using convolution to model the relationship between different channels, which is used as the input for reweighting the attention weights of MSFA, and as the fused visual features output in this stage. Specifically, this invention employs an MSFA structure for extracting visual features and performing multimodal fusion. More specifically, the MSFA structure used in this invention is as follows: Figure 3 As shown, it is divided into three parts. The first part uses the fused visual features from the output of the previous stage. For input (in stage 1, input initial visual features) Using a 5×5 convolution kernel Perform deep local feature extraction and output the extracted visual features. The second part is the output of the first part. The process involves two multi-branch strip convolutions. Simultaneously, the output of the first part also utilizes a cross-attention mechanism. Integrating language features This is also used as the output of this stage. In the third part, the results from the four branches of the second part are summed and then passed through a 1×1 convolution to model the relationship between different channels. This sum is used as the input to the attention weight reweighted MSFA and serves as the output of this stage. .

[0035] In formulaic form, the MSFA used in this invention can be written as:

[0036]

[0037]

[0038] in, This represents the attention map obtained. Represents the fused visual features of stage i. Represents the term-by-term product of matrices. Represents Depth-wise convolution. Represents the first in the diagram Strip convolution branches (from left to right) In the convolution branches, this invention employs two bar convolutions to simulate two-dimensional convolution, with kernel sizes of 7, 13, and 21 for the three branches, respectively. The advantages of this approach are: 1) lighter computational complexity; 2) better extraction of bar features from the image.

[0039] Employing strip convolutions instead of traditional 2D convolutions reduces the model's computational complexity, making it more lightweight and accelerating training and inference processes, while maintaining its ability to extract multiple features. Multi-branch strip convolutions can extract image features from different scales, enhancing the model's adaptability to the diversity of tumor structures. This is of great value in handling tumors of different sizes and shapes. Through a cross-attention mechanism, the model can effectively integrate visual and linguistic features, automatically identifying and strengthening the features most relevant to the target tumor, thus improving the accuracy of image segmentation.

[0040] S4: For the fused visual features output at each stage, scale the number of channels to a uniform dimension, and then stitch them together to obtain the final feature map; Specifically, because each stage in the multi-stage processing may generate feature maps with different dimensions (i.e., different numbers of channels), directly decoding these feature maps may lead to uneven distribution of information, thus affecting the final prediction results. Therefore, it is necessary to first adjust the output feature maps of different stages to a uniform dimension through a scaling operation. This operation is implemented through convolutional layers, aiming to maintain the density and quality of information while reducing the differences caused by dimensionality.

[0041] Subsequently, these feature maps with a uniform number of channels are concatenated to form a richer and more comprehensive feature representation. This concatenation operation helps to aggregate information from different stages, so that the final feature map not only retains the characteristics of each stage but also integrates cross-modal information from multiple stages, providing more comprehensive information support and increasing the understanding of the visual environment. The final feature map is obtained by scaling the multimodal feature map outputs from each stage to a uniform dimension and then concatenating them, as expressed by the following formula:

[0042]

[0043] in, Represents the stage features of scaling the number of channels to a uniform dimension, i∈{1,2,3,4}; Represents full-scale multimodal features; It is a projection function implemented by convolutional layers, dimension mapping ([C,H,W] compressed to [C,HW]), and batch normalization.

[0044] By unifying the number of channels and stitching feature maps from different stages, it is ensured that no information from any stage is lost in the final feature representation, thus fully utilizing all the effective information extracted from the original image. Secondly, this design helps to comprehensively consider multimodal information at different scales, enabling a better understanding of the visual environment at different scales and improving the model's ability to capture details and globally locate tumors.

[0045] S5: Input the final feature map into the full-scale decoder for decoding to obtain a pixel-level segmentation mask that marks the target object.

[0046] Specifically, this invention uses linear transformation and matrix factorization techniques to efficiently decode accurate pixel-level segmentation masks from fused full-scale multimodal features. A Hamburger architecture is employed for global decoding, with the input feature map... Mapping to a linear transformation The feature space is then solved using matrix decomposition M to obtain the low-rank signal subspace, and finally another linear transformation is used. The extracted signal is converted into an output. This is expressed as a formula:

[0047]

[0048] Where H(Z) represents the analysis process of the Hamburger architecture, M represents matrix decomposition, BN represents batch normalization, and Y represents the final output.

[0049] Through sophisticated feature processing and signal extraction mechanisms, and by integrating multimodal information from all scales, the global encoder can more accurately identify and segment target objects in images, especially in complex tumor segmentation tasks in medical images. Through linear transformation and matrix factorization, the full-scale decoder can effectively process and simplify high-dimensional feature data, reducing computational complexity while maintaining high accuracy in segmentation tasks, making it suitable for medical image processing scenarios requiring rapid response.

[0050] This application proposes a technique for globally processing multi-scale fusion features. It involves extracting features obtained at different scales and globally processing and fusing these multi-scale features, enabling features at different scales to play a role globally. This significantly improves the ability to locate target lesions based on text.

[0051] This application introduces multi-size convolutional kernels. The size of the convolutional kernel has a decisive impact on the performance of convolutional neural networks; different sized kernels can capture information at different scales. Most common neural networks use single-size convolutional kernels, a design that cannot meet all application scenarios, especially those requiring a more precise understanding of image details, such as edge detection and object segmentation. Multi-size convolutional kernels can process the same image information from multiple angles and at multiple scales, greatly enriching the model's perspective and depth of understanding. This improves the model's understanding of the visual environment and optimizes the segmentation accuracy of lesion edges.

[0052] For the medical image representation and segmentation task, we established a corresponding dataset for testing and training. This dataset contains 230 cases, totaling 1876 samples (including patient abdominal CT scans, text pointing to a lesion, and segmentation masks specifying the lesion). We use mIoU and oIoU, common metrics for natural image representation and segmentation tasks, as evaluation metrics to measure a model's performance in terms of segmentation results. mIoU (Mean Intersection over Union) is a metric used to evaluate the performance of image segmentation models. It first calculates the IoU (Intersection over Union) for each class, and then averages the IoU of all classes to obtain mIoU. IoU refers to the ratio of the intersection area to the union area of ​​the predicted segmented region and the ground truth segmented region, used to measure the degree of overlap between the predicted and ground truth results. mIoU, on the other hand, averages the IoU of all classes to comprehensively evaluate the model's segmentation accuracy for all classes. oIoU (Overall Intersection over Union) is a variant of IoU, which calculates the IoU of the entire image rather than for each class. oIoU directly calculates the ratio of the intersection area to the union area of ​​the predicted segmented region and the actual segmented region, which is used to measure the segmentation accuracy of the entire image.

[0053] Table 1 presents the experimental results. Compared with existing models, the method proposed in this invention significantly improves performance in medical image representation and segmentation tasks compared to pure image methods and existing vision-language methods. Existing vision-language methods are designed for natural image tasks.

[0054] Table 1. Medical image referencing and segmentation results on the self-built medical image dataset.

[0055] [1] Liu Z, Lin Y, Cao Y, et al. Swin transformer: Hierarchical visiontransformer using shifted windows[C] / / Proceedings of the IEEE / CVFinternational conference on computer vision. 2021: 10012-10022. [2] Yang Z, Wang J, Tang Y, et al. Lavt: Language-aware visiontransformer for referring image segmentation[C] / / Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition. 2022: 18155-18165. In tests on natural images, the model not only achieved excellent results, but also demonstrated outstanding performance compared to current state-of-the-art techniques in the specific field of medical imaging. It can understand the various structures and medical terminology of medical images more deeply and accurately, resulting in superior performance in medical image segmentation tasks and significantly improving the accuracy and precision of image segmentation.

[0056] Another advantage is that, compared to the best existing technologies, the model of this invention performs exceptionally well in tumor segmentation, covering the vast majority of tumor types. From irregularly shaped tumors to small tumors, the model can segment them with high accuracy, which has crucial application value in tumor diagnosis and treatment. This is because, on the one hand, the diverse shapes of tumors present complex and varied segmentation problems, which this invention can effectively solve. On the other hand, for small tumors, early identification and treatment are crucial for patient recovery, and the precise segmentation of the model of this invention makes this possible.

[0057] Corresponding to the aforementioned embodiments of the medical image reference image segmentation method, this application also provides embodiments of a medical image reference image segmentation apparatus.

[0058] Figure 4 This is a block diagram illustrating a medical image reference segmentation apparatus according to an exemplary embodiment. (Refer to...) Figure 4 The device includes: Module 1 is used to acquire the patient's tumor CT images and corresponding text descriptions; Processing module 2 embeds the text description into several tokens, and then inputs them into a Transformer-based language encoder to form high-dimensional word vectors and obtain language features; Fusion module 3 is used to perform visual feature encoding and multimodal feature fusion using MSFA through four stages with decreasing spatial resolution, based on the tumor CT image and the language features. Specifically, it includes: taking the fused visual features output from the previous stage as input, using convolution kernels to extract deep local features, outputting the extracted visual features, simultaneously performing two multi-branch strip convolutions on the extracted visual features and fusing language features using a cross-attention mechanism, summing the results of the four branches, and then using convolution to model the relationship between different channels, which is used as the input for reweighting the attention weights of MSFA, and as the fused visual features output in this stage. The unified stitching module 4 is used to apply bilinear interpolation upsampling to the fused visual features output from each stage. It scales the number of channels of the multimodal feature map output from each stage to a uniform dimension, and then stitches them together to obtain the final feature map. The decoding output module 5 is used to input the final feature map into the full-scale decoder for decoding to obtain a pixel-level segmentation mask that marks the target object.

[0059] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0060] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0061] Accordingly, this application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the medical image reference image segmentation method as described above.

[0062] Accordingly, this application also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the medical image reference image segmentation method described above.

[0063] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0064] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for segmenting medical image reference images, characterized in that, include: Obtain the patient's tumor CT images and corresponding text descriptions; The text description is embedded into several tokens, which are then input into a Transformer-based language encoder to form high-dimensional word vectors, thus obtaining language features. Based on the tumor CT image and the language features, visual feature encoding and multimodal feature fusion are performed using MSFA through four stages with decreasing spatial resolution. Specifically, this includes: taking the fused visual features output from the previous stage as input, using convolution kernels to extract deep local features, outputting the extracted visual features, simultaneously performing two multi-branch strip convolutions on the extracted visual features and fusing language features using a cross-attention mechanism, summing the results of the four branches, and then using convolution to model the relationship between different channels, which is used as the input for reweighting the attention weights of MSFA, and serving as the fused visual features output in this stage. For the fused visual features output at each stage, bilinear interpolation upsampling was applied to scale the number of channels of the multimodal feature map output at each stage to a uniform dimension, and then they were stitched together to obtain the final feature map. The final feature map is input into a full-scale decoder for decoding to obtain a pixel-level segmentation mask that identifies the target object. The formula used for visual feature encoding and multimodal feature fusion using MSFA is as follows: ； ； ； in, This represents the previous stage features of the input. and These represent the fused visual features of the attention map and the output, respectively. Represents the term-by-term product of matrices. Represents Depth-wise convolution. Represents the first in the diagram Convolution branches, In the convolution branch, two strip convolutions are used to simulate two-dimensional convolution, and the kernel sizes of the three branches are 7, 13, and 21, respectively. The multimodal feature map outputs from each stage are scaled to a uniform dimension, and then concatenated to obtain the final feature map, including: ； in, These represent intermediate values ​​generated during the feature map stitching process at each stage. ; Represents the final feature map; This indicates feature cascading along the channel dimension; It is a projection function implemented by two 3×3 convolutions with batch normalization and ReLU nonlinear connection.

2. A medical image segmentation device, characterized in that, To implement the medical image reference image segmentation method as described in claim 1, the method includes: The acquisition module is used to acquire the patient's tumor CT images and corresponding text descriptions; The processing module is used to embed the text description into several tokens, and then input them into a Transformer-based language encoder to form high-dimensional word vectors and obtain language features; The fusion module is used to perform visual feature encoding and multimodal feature fusion using MSFA through four stages with decreasing spatial resolution, based on the tumor CT image and the language features. Specifically, it includes: taking the fused visual features output from the previous stage as input, using convolution kernels to extract deep local features, outputting the extracted visual features, simultaneously performing two multi-branch strip convolutions on the extracted visual features and fusing language features using a cross-attention mechanism, summing the results of the four branches, and then using convolution to model the relationship between different channels, which is used as the input for reweighting the attention weights of MSFA, and as the fused visual features output in this stage. The unified stitching module is used to fuse the visual features output from each stage. It applies bilinear interpolation upsampling to scale the number of channels of the multimodal feature map output from each stage to a uniform dimension, and then stitches them together to obtain the final feature map. The decoding output module is used to input the final feature map into the full-scale decoder for decoding to obtain a pixel-level segmentation mask that marks the target object.

3. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in claim 1.

4. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by the processor, this instruction implements the steps of the method as described in claim 1.

Images