Medical image segmentation method and system based on image-text interaction

Abstract

The application discloses a medical image segmentation method and system based on picture-text interaction, and relates to the technical field of image segmentation. Firstly, the medical image is acquired and multi-scale visual features are extracted. The user's text description is deeply analyzed by using a medical knowledge graph, and knowledge is enhanced by using a medical anatomy knowledge graph, so that an enhanced text vector is constructed, which fuses deep semantics and accurate anatomical context. Further, through a multi-granularity semantic interface and a collaborative fusion mechanism, the enhanced text vector and the multi-scale visual features are progressively cross-modal aligned and information-interacted, the model is guided to focus and accurately locate the target area, and finally a high-precision segmentation mask is generated by the decoder. In this way, the flexibility of natural language and the accuracy of medical prior knowledge are combined, which can effectively overcome the segmentation challenge in complex or ambiguous scenarios, and significantly improve the accuracy and robustness of the segmentation task.

Description

A Medical Image Segmentation Method and System Based on Graphical-Text Interaction Technical Field

[0001] This application relates to the field of image segmentation technology, and more specifically, to a medical image segmentation method and system based on graphic-text interaction. Background Technology

[0002] With the rapid development of precision medicine and medical imaging technology, clinical practice has placed higher demands on the intelligence and precision of medical image analysis and recognition. As a core basis for disease diagnosis, the accurate segmentation of lesion areas in medical imaging directly affects diagnostic accuracy, treatment plan formulation, and efficacy evaluation. Especially in the diagnosis and treatment of complex diseases such as tumors, neurodegenerative diseases, and vascular lesions, high-precision image segmentation is a key link in improving the efficiency of medical decision-making and achieving personalized treatment.

[0003] In existing technologies, medical image segmentation mainly relies on manual annotation or automated algorithms based on a single image modality (such as MRI, CT, or ultrasound only). Manual segmentation not only consumes a significant amount of time and effort from experts but also suffers from strong subjectivity and poor repeatability, making it difficult to meet the processing needs of massive amounts of image data. While traditional automated segmentation methods can provide assistance in standardized scenarios, their processing flow typically focuses on low-order feature analysis at the pixel level (such as grayscale, texture, and edges), often neglecting the high-order semantic knowledge contained in rich textual information such as clinical reports and physician-annotated textual descriptions (such as lesion descriptions, boundary indicators, and structural annotations). This lack of semantic information leads to severely insufficient robustness of algorithms when facing complex situations such as image artifact interference, blurred boundaries between lesions and normal tissues (such as invasive tumors), or highly variable anatomical structures. This results in a significant decrease in segmentation accuracy and a high likelihood of missed segmentation, over-segmentation, or boundary localization errors.

[0004] Therefore, there is an urgent need for a medical image segmentation method and system based on image-text interaction. Summary of the Invention

[0005] This application is made in order to solve the above-mentioned technical problems.

[0006] According to one aspect of this application, a medical image segmentation method based on image-text interaction is provided, comprising: acquiring an original medical image; inputting the original medical image into a medical-specific visual encoder to obtain a multi-scale visual feature map of the medical image; performing text feature encoding and knowledge enhancement on the acquired user text description to obtain an enhanced text feature vector containing semantic information and anatomical structural context; performing multi-granularity semantic grounding and collaborative fusion on the multi-scale visual feature map of the medical image and the enhanced text feature vector containing semantic information and anatomical structural context to obtain a multi-scale cross-modal semantic grounding feature map of the medical image; inputting the multi-scale cross-modal semantic grounding feature map of the medical image into a decoder to obtain a segmentation mask; and applying the segmentation mask to the original medical image to obtain a medical image segmentation result.

[0007] According to another aspect of this application, a medical image segmentation system based on image-text interaction is provided, comprising: a raw medical image acquisition module for acquiring raw medical images; a visual feature extraction module for inputting the raw medical images into a medical-specific visual encoder to obtain a multi-scale visual feature map of the medical images; a text feature enhancement module for performing text feature encoding and knowledge enhancement on the acquired user text description to obtain an enhanced text feature vector containing semantic information and anatomical structural context; a multimodal fusion module for performing multi-granular semantic grounding and collaborative fusion on the multi-scale visual feature map of the medical images and the enhanced text feature vector containing semantic information and anatomical structural context to obtain a multi-scale cross-modal semantic grounding feature map of the medical images; a segmentation decoding module for inputting the multi-scale cross-modal semantic grounding feature map of the medical images into a decoder to obtain a segmentation mask; and a segmentation result generation module for applying the segmentation mask to the raw medical images to obtain a medical image segmentation result.

[0008] Compared with existing technologies, this application provides a medical image segmentation method and system based on image-text interaction. It first acquires medical images and extracts multi-scale visual features, then uses a medical knowledge graph for deep parsing of user text descriptions and utilizes a medical anatomy knowledge graph for knowledge enhancement, thereby constructing an enhanced text vector that integrates deep semantics and precise anatomical context. Furthermore, through a multi-granularity semantic grounding and collaborative fusion mechanism, the enhanced text vector and multi-scale visual features are progressively aligned and interact across modalities, guiding the model to focus on and accurately locate the target region. Finally, the decoder generates a high-precision segmentation mask. This effectively combines the flexibility of natural language with the accuracy of prior medical knowledge, effectively overcoming segmentation challenges in complex or ambiguous scenarios and significantly improving the accuracy and robustness of the segmentation task. Attached Figure Description

[0009] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0010] Figure 1 is a flowchart of a medical image segmentation method based on graphic-text interaction according to an embodiment of this application.

[0011] Figure 2 is a data flow diagram of a medical image segmentation method based on graphic-text interaction according to an embodiment of this application.

[0012] Figure 3 is a flowchart of sub-step S3 of the medical image segmentation method based on graphic-text interaction according to an embodiment of this application.

[0013] Figure 4 is a flowchart of sub-step S4 of the medical image segmentation method based on graphic-text interaction according to an embodiment of this application.

[0014] Figure 5 is a block diagram of a medical image segmentation system based on graphic-text interaction according to an embodiment of this application. Detailed Implementation

[0015] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0016] To address the problems in the aforementioned background technology, this application proposes a medical image segmentation method based on image-text interaction. Figure 1 is a flowchart of the medical image segmentation method based on image-text interaction according to an embodiment of this application. Figure 2 is a data flow diagram of the medical image segmentation method based on image-text interaction according to an embodiment of this application. As shown in Figures 1 and 2, the medical image segmentation method based on image-text interaction includes the following steps: S1, acquiring the original medical image; S2, inputting the original medical image into a medical-specific visual encoder to obtain a multi-scale visual feature map of the medical image; S3, performing text feature encoding and knowledge enhancement on the acquired user text description to obtain an enhanced text feature vector containing semantic information and anatomical structural context; S4, performing multi-granularity semantic grounding and collaborative fusion on the multi-scale visual feature map of the medical image and the enhanced text feature vector containing semantic information and anatomical structural context to obtain a multi-scale cross-modal semantic grounding feature map of the medical image; S5, inputting the multi-scale cross-modal semantic grounding feature map of the medical image into a decoder to obtain a segmentation mask; S6, applying the segmentation mask to the original medical image to obtain a medical image segmentation result.

[0017] In the aforementioned medical image segmentation method based on image-text interaction, step S1 involves acquiring the original medical images. This means accurately acquiring original medical images containing lesion areas from medical imaging equipment such as CT, MRI, and ultrasound, ensuring the integrity, clarity, and format standardization of the data. This provides high-quality original material for subsequent cross-modal semantic fusion and accurate segmentation, while also providing more reliable data support for subsequent extraction of multi-scale visual features and fusion of textual semantic information.

[0018] Specifically, in one possible embodiment, step S1 is implemented as follows: First, a standardized device access interface needs to be established, compatible with different types of medical imaging equipment. Communication between the device and the system is achieved through medical digital imaging and communication standard protocols, ensuring stable transmission of image data to the system server. Next, the image data import operation is performed, batch importing historical image data and acquiring newly acquired images in real time. During the import process, metadata such as patient basic information, examination time, and equipment model are recorded simultaneously for subsequent data management and traceability. Then, rigorous data verification is performed. On the one hand, algorithms automatically detect quality indicators such as image resolution, grayscale range, and the presence of artifacts or noise. On the other hand, manual visual verification of key images is allowed to eliminate data that does not meet requirements, ensuring the reliability of the original medical images. After verification, non-standard format images output from different devices are converted to a unified medical digital imaging and communication standard format recognizable by the system. Simultaneously, image data compression and encryption are completed, reducing storage space usage and improving security while ensuring data accuracy. Finally, the processed raw medical images are indexed according to dimensions such as patient ID and examination type, and stored in a distributed database or storage server to form a structured image data pool. This provides efficient data access support for subsequent rapid retrieval of raw medical images and association processing with user text descriptions, ensuring the accuracy and usability of the entire medical image segmentation process at the raw data level.

[0019] In the aforementioned medical image segmentation method based on image-text interaction, step S2 involves inputting the original medical image into a medical-specific visual encoder to obtain a multi-scale visual feature map of the medical image. It should be understood that the segmentation accuracy of an image significantly decreases when faced with complex situations such as image artifact interference, blurred boundaries between lesions and normal tissues, or highly variable anatomical structures. Therefore, this application extracts a multi-scale visual feature map of the medical image containing visual information at different scales using a medical-specific visual encoder. This not only preserves the detailed texture of the original image but also extracts high-level semantic information, providing a rich visual information foundation for subsequent cross-modal feature fusion. Specifically, this application inputs the original medical image into a medical-specific visual encoder based on the Unet encoder to obtain a high-resolution feature map, a medium-resolution feature map, and a low-resolution feature map of the medical image. These three feature maps constitute the multi-scale visual feature map of the medical image.

[0020] Specifically, in one possible embodiment, step S2 is implemented as follows: First, the original medical image is input into a medical-specific visual encoder based on the Unet encoder. Feature extraction is performed on the image through multi-layer convolution operations. Each convolution layer aims to capture visual features at different levels, while downsampling operations enable the model to understand image content at different scales, laying the foundation for subsequent multi-granularity semantic grounding and collaborative fusion. Specifically, the Unet encoder adopts a classic encoder-decoder architecture. Its encoder part gradually reduces the spatial resolution of the feature map and increases the number of channels through continuous convolution and downsampling operations, thereby sequentially extracting high-resolution feature maps, medium-resolution feature maps, and low-resolution feature maps of the medical image. The high-resolution feature map retains detailed information such as the edges and textures of lesions, while the medium-resolution feature map considers both local structure and semantic information. The low-resolution feature map contains the overall semantic and contextual information of the image. These three different resolution feature maps together constitute the multi-scale visual feature map of the medical image.

[0021] In the aforementioned medical image segmentation method based on image-text interaction, step S3 involves encoding and enhancing the acquired user text description to obtain an enhanced text feature vector containing semantic information and anatomical structural context. By encoding and enhancing the user text description, semantic information and medical anatomical structural knowledge are integrated into the model. This effectively constructs an enhanced text feature vector that fuses deep semantics and precise anatomical context. It not only preserves the semantic information in the user text description but also incorporates structured knowledge from the medical anatomical knowledge graph. This provides a rich semantic foundation for subsequent cross-modal fusion with multi-scale visual features of medical images, significantly improving the accuracy and robustness of the segmentation task.

[0022] In particular, in one possible embodiment, Figure 3 is a flowchart of sub-step S3 of the medical image segmentation method based on image-text interaction according to an embodiment of this application. As shown in Figure 3, step S3 includes: S31, inputting the user text description into a medical-specific language encoder based on the BioClinical BERT model to obtain an initial text embedding encoding vector; S32, extracting medical anatomical entities from the user text description, and using the medical anatomical entities to query a predefined medical anatomical structure knowledge graph to obtain relevant anatomical structure information; S33, encoding the relevant anatomical structure information to obtain an anatomical knowledge embedding encoding vector; S34, inputting the initial text embedding encoding vector and the anatomical knowledge embedding encoding vector into a knowledge fusion module to obtain the enhanced text feature vector containing semantic information and anatomical structure context.

[0023] Specifically, in step S31, the user text description is input into a medical-specific language encoder based on the BioClinical BERT model to obtain an initial text embedding encoding vector. It should be understood that the BioClinical BERT model is a pre-trained language model based on the BERT architecture, specifically optimized for clinical text. It inherits the bidirectional encoder structure of BERT and performs excellently in processing medical text through pre-training on a large amount of clinical text. Specifically, this application leverages the advantages of the BioClinical BERT model in medical text understanding to transform the user text description into an initial text embedding encoding vector containing basic semantic information. This preserves the original semantic information of the user text description, providing a semantic foundation for subsequent extraction of medical anatomical entities from the text. This more accurately maps the relationship between clinical needs and anatomical structures, providing high-quality semantic input for subsequent knowledge enhancement using a medical anatomical knowledge graph. This improves the entire segmentation model's ability to understand text semantics and lays a solid foundation for subsequent cross-modal fusion.

[0024] Specifically, in one possible embodiment, step S31 is implemented as follows: First, the user-input text description, such as lesion feature descriptions and anatomical structure annotations in a clinical report, is obtained and preprocessed to obtain a sequence conforming to the model input format, such as standardization processing including word segmentation and removal of special characters. Second, the preprocessed text sequence is input into a medical-specific language encoder based on the BioClinical BERT model. This encoder uses a multi-layer bidirectional Transformer architecture to perform contextual semantic encoding on each token in the text sequence, utilizing a self-attention mechanism to capture long-distance dependencies between words, especially targeting polysemous words and technical terms commonly found in medical texts for targeted semantic disambiguation and feature extraction. During the encoding process, the model strengthens the representation of key semantic information such as anatomical structures and lesion properties, and suppresses irrelevant noise, based on the characteristics of the medical text. Finally, the embedded representation of the text sequence is extracted from the model's output layer to form an initial text embedding encoding vector, which not only includes the lexical and syntactic features of the text but also integrates semantic associations unique to the medical field.

[0025] Specifically, step S32 involves extracting medical anatomical entities from the user's text description and querying these entities in a predefined medical anatomical structure knowledge graph to obtain relevant anatomical structure information. That is, by utilizing the predefined medical anatomical structure knowledge graph, the medical anatomical entities in the user's text are transformed into structured anatomical context information, thereby providing semantic enhancement in the anatomical dimension for the initial text embedding encoding vector. Specifically, by extracting anatomical entities, such as the left lobe of the liver and the pituitary gland, and querying their anatomical location, adjacent relationships, and blood supply system in the knowledge graph, enhanced text features that integrate text semantics and anatomical structure can be constructed. This provides anatomical prior knowledge support for the subsequent multimodal fusion process, enabling the model to understand the spatial relationship between lesions and surrounding tissues based on anatomical relationships when processing images, thus more accurately focusing on the target area.

[0026] Specifically, step S33 involves encoding the relevant anatomical structure information to obtain an anatomical knowledge embedding encoding vector. It should be understood that while the medical anatomical entities extracted from user text descriptions can initially indicate the segmentation target, they lack deeper knowledge such as spatial relationships and physiological characteristics between anatomical structures. Therefore, this application transforms relevant anatomical structure information, such as the adjacency relationships, anatomical layers, and morphological features of target organs, into a dense anatomical knowledge embedding encoding vector through a specific information encoding mechanism, thereby providing a computable anatomical semantic representation for subsequent knowledge fusion modules.

[0027] Specifically, in one possible embodiment, step S33 is implemented as follows: Since the relevant anatomical structure information exists in the form of structured data, including the attributes, relationships, and topological structures of anatomical entities, when performing feature extraction and vectorization processing on the relevant anatomical structure information, the attribute information in the graph can be encoded using a multilayer perceptron. The multilayer perceptron can process such multi-dimensional and multi-type structured data through nonlinear transformation. It should be understood that a multilayer perceptron is a feedforward artificial neural network composed of multiple fully connected neuron layers, typically including an input layer, one or more hidden layers, and an output layer. Specifically, the anatomical structure information is first preprocessed: textual attributes are converted into vectors through word embeddings, numerical attributes are normalized, and relational information is converted into numerical features through graph structure feature extraction. These features are then concatenated into a hybrid vector. Then, the hidden layers of the multilayer perceptron adopt a multi-layer fully connected structure. Each neuron transforms the input through a weight matrix and bias terms, and an activation function is introduced to enhance nonlinear expressive power. For example, two hidden layers can be set up. The first layer maps the hybrid vector to a higher-dimensional space, such as increasing it from 100 dimensions to 256 dimensions, to capture the deep correlations of anatomical structure information. The second layer then reduces the dimensionality to the target dimension, such as 128 dimensions, to ensure that the vector retains anatomical knowledge details while meeting the dimensionality requirements of subsequent fusion modules. Finally, the output layer maps the hidden layer features to anatomical knowledge embedding encoding vectors of a specified dimension through a fully connected layer, such as 768 dimensions, which is consistent with the dimension of the initial text embedding vector. This ensures that the generated anatomical knowledge embedding encoding vectors can be processed in the subsequent knowledge fusion module along with the initial text embedding encoding vectors.

[0028] Specifically, in step S34, the initial text embedding encoding vector and the anatomical knowledge embedding encoding vector are input into the knowledge fusion module to obtain the enhanced text feature vector containing semantic information and anatomical structural context. The knowledge fusion module performs fully connected encoding on the initial text embedding encoding vector and the anatomical knowledge embedding encoding vector to obtain the enhanced text feature vector containing semantic information and anatomical structural context. Specifically, the initial text embedding encoding vector and the anatomical knowledge embedding encoding vector are fused across dimensions, enabling the text features to reflect both the semantics of the lesion described by the user and prior knowledge such as the spatial location and adjacency relationships of the anatomical structure. This provides a text feature foundation with both semantic understanding and anatomical constraints for subsequent cross-modal fusion, thereby guiding the model to accurately locate the target region during image segmentation. The enhanced text feature vector generated by feature fusion through fully connected encoding not only retains the semantic flexibility of the text description but also introduces precise anatomical structural constraints from the medical field through knowledge enhancement from the anatomical knowledge graph. This allows the model to leverage anatomical knowledge context to assist semantic understanding and reduce segmentation bias when facing complex scenarios such as image artifacts and blurred boundaries.

[0029] Specifically, in one possible embodiment, step S34 is implemented as follows: The knowledge fusion module learns the parameters of the fully connected layer. The number of neurons in the fully connected layer can be set according to task requirements, such as 768 dimensions. A linear transformation is performed on the initial input text embedding encoding vector and the anatomical knowledge embedding encoding vector using a weight matrix and bias terms, assigning weights to the two types of features. During the linear transformation, the weights of the fully connected layer learn how to efficiently combine semantic information and anatomical knowledge information, for example, strengthening the weights of anatomical structure features relevant to the current segmentation task. After activation function processing, the output fusion vector is then normalized to finally generate an enhanced text feature vector containing semantic information and anatomical structure context.

[0030] In the aforementioned medical image segmentation method based on image-text interaction, step S4 involves performing multi-granular semantic grounding and collaborative fusion on the multi-scale visual feature map of the medical image and the enhanced text feature vector containing semantic information and anatomical structural context to obtain a multi-scale cross-modal semantic grounding feature map of the medical image. It should be understood that through the multi-granular cross-modal fusion mechanism, textual semantic information can be progressively grounded into image features, compensating for segmentation biases caused by modal differences. Specifically, firstly, low-resolution visual features and text features are coarsely aligned to locate the global position of the target region. Then, medium-resolution features are used to refine the region boundaries. Finally, high-resolution features are used to capture fine structures, thereby constructing a multi-scale cross-modal semantic grounding feature map of the medical image that integrates cross-modal semantics. This hierarchical fusion mechanism enables the model to combine textual semantics and anatomical knowledge at different scales, thereby gradually focusing on the target region and solving the problem that single-scale features cannot simultaneously address global localization and detailed segmentation. Through multi-granular cross-modal fusion, the model can utilize anatomical prior knowledge in textual semantics to guide the semantic understanding of image features at different resolutions.

[0031] In particular, in one possible embodiment, FIG4 is a flowchart of sub-step S4 of the medical image segmentation method based on graphic-text interaction according to an embodiment of the present application. As shown in Figure 4, step S4 includes: S41, constructing a query vector based on the enhanced text feature vector containing semantic information and anatomical structural context, and constructing a value vector and a key vector based on the low-resolution feature map of the medical image, and inputting the query vector, the value vector, and the key vector into a coarse-grained grounding module to obtain a low-resolution cross-modal semantic grounding feature map of the medical image; S42, constructing a query vector based on the low-resolution cross-modal semantic grounding feature map of the medical image, and constructing a value vector and a key vector based on a medium-resolution feature map of the medical image, and inputting the query vector, the value vector, and the key vector into a medium-grained grounding module to obtain a medium-resolution cross-modal semantic grounding feature map of the medical image; S43, constructing a query vector based on the medium-resolution cross-modal semantic grounding feature map of the medical image, and constructing a value vector and a key vector based on a high-resolution feature map of the medical image, and inputting the query vector, the value vector, and the key vector into a fine-grained grounding module to obtain a multi-scale cross-modal semantic grounding feature map of the medical image.

[0032] Specifically, in step S41, a query vector is constructed based on the enhanced text feature vector containing semantic information and anatomical structural context, and a value vector and a key vector are constructed based on the low-resolution feature map of the medical image. The query vector, the value vector, and the key vector are then input into the coarse-grained grounding module to obtain the low-resolution cross-modal semantic grounding feature map of the medical image. In one embodiment of this application, the coarse-grained grounding module processes the query vector, the value vector, and the key vector using the following formula to obtain the low-resolution cross-modal semantic grounding feature map of the medical image, wherein the formula is: ;in, Cross-modal semantic grounding feature vectors For query vectors, For key vectors, For value vectors, This is the transpose of the vector. For scaling factor, For the characteristic shape reshaping function, This is a low-resolution cross-modal semantic grounding feature map for medical images.

[0033] In other words, global localization of the target region is achieved through coarse-grained cross-modal interaction, using the enhanced text feature vector as the query vector, while the key and value vectors generated from the low-resolution feature map represent the global structural information of the image. Through the attention mechanism of the coarse-grained grounding module, the model can calculate the association weights between text semantics and the global structure of the image, thereby locating the region matching the text description in the low-resolution cross-modal semantic grounding feature map of the medical image. For example, it can quickly pinpoint the approximate range of the right lobe of the liver in a whole-liver CT image, providing an initial semantic anchor point for subsequent medium- and fine-grained feature fusion. Specifically, the low-resolution cross-modal semantic grounding feature map of the medical image generated after coarse-grained cross-modal fusion preserves the global structure of the image while embedding the anatomical constraints of the text semantics. In this way, in complex multi-organ contexts, the focus of subsequent feature processing can be effectively narrowed, computational redundancy reduced, and a more accurate initial region provided for medium-grained boundary refinement is provided, significantly enhancing the robustness of the overall segmentation process.

[0034] Specifically, in step S42, a query vector is constructed based on the low-resolution cross-modal semantic grounding feature map of the medical image, and a value vector and a key vector are constructed based on the medium-resolution feature map of the medical image. The query vector, the value vector, and the key vector are then input into the medium-granularity grounding module to obtain the medium-resolution cross-modal semantic grounding feature map of the medical image. It should be understood that through medium-resolution cross-modal fusion, the semantic alignment of details in the medium-resolution features can be guided, compensating for semantic uncertainties during boundary refinement. Specifically, through the attention mechanism of the medium-granularity grounding module, the model can calculate the association weights between coarse-grained semantics and medium-resolution structures, thereby strengthening the boundary details matched with coarse-grained semantics, such as refining the contact boundary between the tumor and the right branch of the portal vein, providing transitional features for subsequent fine-grained capture of texture details. The generated medical image cross-modal semantic grounding feature map can not only preserve the local structural details of medium-resolution visual features, such as the direction of the main blood vessel and the outline of organ lobes, but also embed boundary constraints guided by coarse-grained semantics, such as the preliminary boundary of tumor invasion of blood vessels. This effectively reduces the segmentation error caused by boundary blurring, provides a more accurate boundary feature basis for fine-grained feature fusion, and significantly improves the contour fit of the overall segmentation result.

[0035] Specifically, in one possible embodiment, step S42 is implemented as follows: First, the low-resolution cross-modal semantic grounding feature map of the medical image is transformed into a query vector with the same dimension as the enhanced text feature vector, such as 768 dimensions, through global average pooling. Simultaneously, the medium-resolution feature map of the medical image is subjected to two linear transformations to generate key and value vectors. Next, the query vector, key vector, and value vector are input into the medium-granularity grounding module. This module employs a scaling dot product attention mechanism similar to the coarse-granularity module, first calculating the transpose matrix product of the query vector and the key vector, then dividing by a scaling factor to stabilize the gradient, where the scaling factor is the dimension of the query vector. This is then multiplied by the value vector to obtain the cross-modal semantic grounding feature vector. Subsequently, a feature shape reshaping operation is used to convert the cross-modal semantic grounding feature vector into the same spatial size as the medium-resolution feature map, generating the medium-resolution cross-modal semantic grounding feature map of the medical image.

[0036] Specifically, in step S43, a query vector is constructed based on the mid-resolution cross-modal semantic grounding feature map of the medical image, and a value vector and a key vector are constructed based on the high-resolution feature map of the medical image. The query vector, the value vector, and the key vector are then input into the fine-grained grounding module to obtain the multi-scale cross-modal semantic grounding feature map of the medical image. It should be understood that the attention mechanism of the fine-grained grounding module calculates the association weights between mid-resolution semantics and high-resolution details, strengthening the microstructure that matches the semantics, and providing the decoder with feature input that combines anatomical semantics and pixel-level details. The generated multi-scale cross-modal semantic grounding feature map of the medical image retains the texture details of high-resolution visual features while embedding detail semantic constraints guided by mid-resolution semantics, effectively enhancing the model's semantic perception ability of fine structures.

[0037] Specifically, in one possible embodiment, step S43 is implemented as follows: First, the high-resolution cross-modal semantic grounding feature map of the medical image is processed by global average pooling to obtain a query vector. Simultaneously, the high-resolution feature map of the medical image is subjected to two linear transformations to generate a key vector and a value vector. Next, the query vector, key vector, and value vector are input into the fine-grained grounding module. This module employs a scaling dot product attention mechanism similar to the coarse-grained approach, first calculating the transpose of the query vector and the key vector, then dividing by a scaling factor to stabilize the gradient, where the scaling factor is the dimension of the query vector. This is then multiplied by the value vector to obtain the cross-modal semantic grounding feature vector. Subsequently, a feature shape reshaping operation is used to convert the cross-modal semantic grounding feature vector to the same spatial size as the high-resolution feature map of the medical image, generating a multi-scale cross-modal semantic grounding feature map of the medical image.

[0038] Here, by using low-resolution image semantics as the query vector, query-key value reshaping of high-resolution image semantics is performed. This allows the boundary saliency of high-resolution image semantics to be reshaped using low-resolution image semantics as the prediction boundary, thereby achieving the collaborative fusion of the key value structure of image semantics with respect to query initialization.

[0039] However, considering the potential lack of boundary saliency in low-resolution image semantics as the prediction boundary, the initialization through key-value structure queries may result in cross-boundary segmentation between the semantics of images at different resolutions, which affects the overall collaborative fusion effect.

[0040] Based on this, in another embodiment, a query vector is constructed based on the resolution cross-modal semantic grounding feature map in the medical image, and a value vector and a key vector are constructed based on the high-resolution feature map of the medical image. The query vector, the value vector, and the key vector are input into a fine-grained grounding module to obtain the multi-scale cross-modal semantic grounding feature map of the medical image. This includes: constructing a query vector based on the resolution cross-modal semantic grounding feature map in the medical image, and constructing a value vector and a key vector based on the high-resolution feature map of the medical image. The query vector, the value vector, and the key vector are input into a fine-grained grounding module to obtain an initial multi-scale cross-modal semantic grounding feature map of the medical image; and performing collaborative fusion optimization on the initial multi-scale cross-modal semantic grounding feature map of the medical image based on the hierarchical boundary segmentation representation of the low-resolution cross-modal semantic grounding feature map of the medical image, the resolution cross-modal semantic grounding feature map in the medical image, and the initial multi-scale cross-modal semantic grounding feature map of the medical image to obtain the multi-scale cross-modal semantic grounding feature map of the medical image.

[0041] Specifically, based on the hierarchical boundary segmentation representations of the low-resolution cross-modal semantic grounding feature map of the medical image, the mid-resolution cross-modal semantic grounding feature map of the medical image, and the initial multi-scale cross-modal semantic grounding feature map of the medical image, the initial multi-scale cross-modal semantic grounding feature map of the medical image is collaboratively fused and optimized to obtain the multi-scale cross-modal semantic grounding feature map of the medical image. This includes: calculating the boundary segmentation representation of the mid-resolution cross-modal semantic grounding feature map of the medical image relative to the low-resolution cross-modal semantic grounding feature map of the medical image to obtain a first boundary segmentation representation map, represented as: ;in, To calculate the gradient, This is a low-resolution cross-modal semantic grounding feature map for medical images. This is a high-resolution cross-modal semantic grounding feature map for medical images. This indicates subtraction by position. The diagram shows the first boundary segmentation.

[0042] In other words, to capture the differences in boundary details between feature maps of different resolutions, the boundary segmentation representation of the high-resolution cross-modal semantic grounding feature map in medical images relative to the low-resolution cross-modal semantic grounding feature map in medical images was calculated, thus obtaining the first boundary segmentation representation map. In actual medical imaging scenarios, such as tumors in liver CT images, the low-resolution feature map may only roughly identify a space-occupying lesion in the liver, but its specific boundaries and morphology are not yet clear. The medium-resolution feature map, on the other hand, begins to show the local texture and general outline of the tumor. By calculating the boundary segmentation representation between the two, the refinement trend and differences of the lesion boundary or anatomical structure edge during the transition from global overview to local details can be quantified and extracted. In this way, the difference between the model's grasp of the overall outline of the tumor and the depiction of local details during the semantic grounding process can be initially revealed, such as the preliminary distinction between the tumor and the liver parenchyma, providing a basis for subsequent fine-tuning.

[0043] The boundary segmentation representation of the initial medical image multi-scale cross-modal semantic grounding feature map relative to the resolution cross-modal semantic grounding feature map in the medical image is calculated to obtain the second boundary segmentation representation map, which is represented as follows: ;in, To calculate the gradient, This is the initial multi-scale cross-modal semantic grounding feature map of medical images. This is a high-resolution cross-modal semantic grounding feature map for medical images. This indicates subtraction by position. This is a diagram representing the second boundary segmentation.

[0044] In other words, to further improve the accuracy of the boundary, the boundary segmentation representation of the initial medical image multi-scale cross-modal semantic grounding feature map is calculated relative to the medium-resolution cross-modal semantic grounding feature map of the medical image, thus obtaining a second boundary segmentation representation map. It should be understood that in clinical practice, many lesions, especially invasive tumors or early lesions, may exhibit highly similar grayscale values ​​or textures to the surrounding normal tissue at their boundaries, making them difficult to distinguish accurately at a single resolution. Although the initial medical image multi-scale cross-modal semantic grounding feature map has already incorporated high-resolution visual information, its boundaries may still be coarsely affected by the medium-resolution feature map, or uncertainties may exist when facing lesions with blurred boundaries. Therefore, the overall contour direction difference of the feature manifold between successive feature maps is used as the boundary segmentation representation to measure the cross-boundary directional distribution difference of the feature manifold. This further captures the further refinement and correction information in boundary characterization from intermediate granularity to the finest granularity feature maps, especially when dealing with lesion edges that are similar in grayscale to normal tissue and difficult to distinguish; the quantification of this difference is particularly crucial. This allows for a more precise identification of the direction and extent of the model's fine-tuning of the tumor margins during the refinement process. For example, it distinguishes between the tiny invasive margins of the tumor and normal vascular structures, providing more detailed guidance for the final boundary optimization.

[0045] Then, the peak-valley slope derivatives between boundary segments at each level are further determined. That is, if the step-by-step boundary segmentation is represented as a two-way cross-boundary slope distribution, the differentiable peak-valley derivatives of its difference information can be extracted as the overall cross-boundary segmentation correlation between levels. That is, based on the first boundary segmentation representation map and the second boundary segmentation representation map, the overall cross-boundary segmentation correlation map between the first and second levels is constructed, represented as follows: ;in, This is a diagram representing the first boundary segmentation. This is a diagram representing the second boundary segmentation. This is a diagram representing the second boundary segmentation. This is a cross-boundary segmentation and correlation diagram for the first level. This is a cross-boundary segmentation relationship diagram between the second level.

[0046] It is understandable that relying solely on boundary differences between single levels is insufficient to fully reflect the complexity of lesion boundaries, especially in medical imaging, where lesion boundaries may exhibit different characteristics at different scales, such as the macroscopic morphology of a tumor versus the roughness of its microscopic edges. By treating the hierarchical boundary segmentation as a two-path cross-boundary slope distribution and extracting its differentiable peak-valley derivatives of the difference information, we can analyze the intrinsic correlation and directional distribution differences of boundary information between feature maps of different granularities, thereby gaining a more comprehensive understanding of the significant loss and directional characteristics of lesion edges. This helps the model understand why the boundaries of certain regions are blurred at low resolution but can be enhanced by textual descriptions (such as boundary blurring) at high resolution, thus more accurately capturing the true boundaries of lesions.

[0047] Finally, based on the overall cross-boundary segmentation association diagram between the first level... The overall cross-boundary segmentation relationship diagram between the second level and the second level The weighted sum is used as a cross-boundary segmentation correction map. The initial medical image multi-scale cross-modal semantic grounding feature map is then multiplied and fused to obtain the medical image multi-scale cross-modal semantic grounding feature map, expressed as: ;in, This is the initial multi-scale cross-modal semantic grounding feature map of medical images. This is a cross-boundary segmentation and correlation diagram for the first level. This is a cross-boundary segmentation relationship diagram between the second level. and Different weighting coefficients are determined through adaptive learning via end-to-end training, for example, when dealing with large, well-defined tumors. Of course, this is just an example and does not constitute a specific limitation. This is a positional dot product, which means multiplying each element of the feature map by its corresponding coefficient or the value corresponding to the feature map. To add by position, This is an enhanced low-resolution cross-modal semantic grounding feature map for medical images.

[0048] In other words, by introducing the peak-valley distribution features of boundary segmentation during the cross-boundary segmentation process, edge enhancement of the query initialization structure can be performed to address the lack of saliency at the boundary. Adaptive collaboration is then utilized by leveraging the sensitivity to directional distribution differences, matching the directionality and difference of the cross-boundary edge. This improves the collaborative fusion effect of the medical image multi-scale cross-modal semantic grounding feature map in query initialization of the key-value structure of image semantics at different granularities. Specifically, the first-level and second-level overall cross-boundary segmentation association maps have accurately captured the differences and correction information in boundary details between feature maps of different granularities. Now, this correction information is effectively applied to the current initial medical image multi-scale cross-modal semantic grounding feature map. In medical image segmentation, this means accurately mapping coarse-to-fine boundary correction information, such as the semantic guidance provided by the textual description of a lesion in the right lobe of the liver to the blurred boundary of hepatocellular carcinoma, to the pixel level. By using weighted summation to form a comprehensive cross-boundary segmentation correction map, the aim is to integrate boundary correction information from different levels and adaptively enhance the initial fine-grained feature map at the pixel level through dot-multiplication fusion. This dot-multiplication fusion approach enables query-initialized structural edge enhancement for regions with significantly missing boundaries and adaptive collaboration based on the sensitivity to differences in boundary directionality. This results in a final multi-scale, cross-modal semantic grounding feature map of medical images that retains high-order semantic information while exhibiting more accurate and clearer lesion boundary delineation. This significantly improves the segmentation accuracy and robustness in complex medical imaging scenarios, such as those with blurred tumor infiltration boundaries or small lesions.

[0049] In the aforementioned medical image segmentation method based on image-text interaction, step S5 involves inputting the multi-scale cross-modal semantic grounding feature map of the medical image into a decoder to obtain a segmentation mask. The decoder is a U-Net decoder. It should be understood that the U-Net decoder employs an encoder-decoder symmetrical structure and a skip connection mechanism, enabling it to restore spatial resolution while fusing cross-modal semantics at different scales. This is particularly suitable for scenarios in medical images where organ boundaries are complex and lesion details are varied. Specifically, the spatial dimensions of the multi-scale cross-modal semantic grounding feature map of the medical image are gradually restored through the upsampling path of U-Net. Simultaneously, skip connections are used to fuse high-resolution visual features from the encoder with cross-modal semantic features from the decoder. This allows the segmentation mask to capture both the fine structure of lesions, such as spiculation in high-resolution features, and to adhere to the anatomical constraints of textual semantics, such as the positional relationship between tumors and adjacent blood vessels. Ultimately, a high-precision segmentation mask that meets clinical needs is generated, solving the mapping problem from multi-modal features to segmentation results. This ensures that the segmentation mask maintains boundary accuracy and semantic consistency even in complex scenarios.

[0050] Specifically, in one possible embodiment, step S5 is implemented as follows: First, the multi-scale cross-modal semantic grounding feature maps are input into the corresponding layers of the U-Net decoder in ascending order of resolution. The downsampling path of the decoder has already been completed in the visual feature extraction stage. At this point, the input low-resolution cross-modal features first pass through an upsampling layer, such as deconvolution, to double their size. Then, they are concatenated with the corresponding resolution visual feature map in the encoder to fuse cross-modal semantics and original visual details. Next, two sets of 3×3 convolutional layers are used to semantically integrate the concatenated feature maps, reducing information noise caused by upsampling. The above upsampling-concatenation-convolution operation is repeated, fusing with the visual features of the corresponding layer of the encoder after each upsampling. Finally, at the highest resolution layer, a 1×1 convolutional layer maps the number of feature channels to the number of segmentation categories, and then a Sigmoid activation function is used to generate the final pixel-level segmentation mask.

[0051] In the aforementioned medical image segmentation method based on image-text interaction, step S6 involves applying the segmentation mask to the original medical image to obtain a medical image segmentation result. Specifically, through the geometric overlay and visualization rendering of the segmentation mask and the original medical image, a medical image segmentation result with both semantic accuracy and visual readability is generated. The medical image segmentation result after masking retains the anatomical details of the original medical image, such as the grayscale density of CT and the signal intensity of MRI, and through the visualization overlay of semantic tags, the target area is highlighted with bright colors or outlines, significantly improving the efficiency and accuracy of doctors in identifying lesions.

[0052] Specifically, in one possible embodiment, step S6 is implemented as follows: First, the segmentation mask output by the decoder is post-processed, including binarization or multi-class label mapping, and the mask resolution is adjusted to be completely consistent with the original medical image through interpolation operations to ensure pixel-level spatial alignment. Next, a spatial overlay algorithm is used to fuse the mask with the original image. For grayscale images, the pixels of the target area in the mask are displayed in pseudo-color overlay, while the background area retains its original grayscale. By setting transparency parameters, such as 30% mask opacity, semantic prominence and preservation of details in the original medical image are balanced. For color images, such as ultrasound pseudo-color images, the boundary of the target area is outlined by drawing contour lines to avoid color conflicts affecting readability. Then, clinical annotation information is added to the image overlay result to form a standardized segmentation report image. Finally, the output format is selected according to the application scenario. For example, the DICOM format retains medical metadata, while the JPEG format is used for report display, and the segmentation results are stored in the medical image management system.

[0053] In summary, the medical image segmentation method based on image-text interaction, as described in this application, is explained. It first acquires medical images and extracts multi-scale visual features, then uses a medical knowledge graph for deep parsing of user text descriptions, and further utilizes a medical anatomy knowledge graph for knowledge enhancement, thereby constructing an enhanced text vector that integrates deep semantics and precise anatomical context. Furthermore, through a multi-granularity semantic grounding and collaborative fusion mechanism, the enhanced text vector and multi-scale visual features are progressively aligned and interact across modalities, guiding the model to focus on and accurately locate the target region. Finally, the decoder generates a high-precision segmentation mask. This method effectively combines the flexibility of natural language with the accuracy of prior medical knowledge, effectively overcoming segmentation challenges in complex or ambiguous scenarios, and significantly improving the accuracy and robustness of the segmentation task.

[0054] Figure 5 is a block diagram of a medical image segmentation system based on text-image interaction according to an embodiment of this application. As shown in Figure 5, the medical image segmentation system 100 based on text-image interaction according to an embodiment of this application includes: a raw medical image acquisition module 110, used to acquire raw medical images; a visual feature extraction module 120, used to input the raw medical images into a medical-specific visual encoder to obtain a multi-scale visual feature map of the medical images; a text feature enhancement module 130, used to perform text feature encoding and knowledge enhancement on the acquired user text description to obtain an enhanced text feature vector containing semantic information and anatomical structural context; a multi-modal fusion module 140, used to perform multi-granularity semantic grounding and collaborative fusion on the multi-scale visual feature map of the medical images and the enhanced text feature vector containing semantic information and anatomical structural context to obtain a multi-scale cross-modal semantic grounding feature map of the medical images; a segmentation decoding module 150, used to input the multi-scale cross-modal semantic grounding feature map of the medical images into a decoder to obtain a segmentation mask; and a segmentation result generation module 160, used to apply the segmentation mask to the raw medical images to obtain a medical image segmentation result.

[0055] As described above, the image segmentation system 100 based on graphic-text interaction according to the embodiments of this application can be implemented in various wireless terminals, such as servers with intelligent network-based equipment asset operation and maintenance management algorithms. In one possible implementation, the image segmentation system 100 based on graphic-text interaction according to the embodiments of this application can be integrated into the wireless terminal as a software module and / or hardware module. For example, the image segmentation system 100 based on graphic-text interaction can be a software module in the operating system of the wireless terminal, or it can be an application developed for the wireless terminal; of course, the image segmentation system 100 based on graphic-text interaction can also be one of many hardware modules of the wireless terminal.

[0056] Alternatively, in another example, the image segmentation system 100 based on text-to-image interaction and the wireless terminal can also be separate devices, and the image segmentation system 100 based on text-to-image interaction can be connected to the wireless terminal via wired and / or wireless networks, and transmit interactive information in accordance with an agreed data format.

[0057] Here, those skilled in the art will understand that the specific operations of each step in the above-described medical image segmentation system based on graphic-text interaction have been described in detail in the description of the medical image segmentation method based on graphic-text interaction with reference to Figures 1 to 4 above, and therefore, their repeated description will be omitted.

Claims

1. A medical image segmentation method based on image-text interaction, characterized in that, include: Acquire raw medical images; input the raw medical images into a medical-specific visual encoder to obtain multi-scale visual feature maps of the medical images; The acquired user text description is subjected to text feature encoding and knowledge enhancement to obtain an enhanced text feature vector containing semantic information and anatomical structural context. Multi-granularity semantic grounding and collaborative fusion are performed on the multi-scale visual feature map of the medical image and the enhanced text feature vector containing semantic information and anatomical structural context to obtain a multi-scale cross-modal semantic grounding feature map of the medical image. This includes: constructing a query vector based on the enhanced text feature vector containing semantic information and anatomical structural context, and constructing a value vector and a key vector based on the low-resolution feature map of the medical image; inputting the query vector, the value vector, and the key vector into a coarse-grained grounding module to obtain a low-resolution cross-modal semantic grounding feature map of the medical image. A query vector is constructed based on the low-resolution cross-modal semantic grounding feature map of the medical image, and a value vector and a key vector are constructed based on the medium-resolution feature map of the medical image. The query vector, the value vector, and the key vector are input into the medium-granularity grounding module to obtain the medium-resolution cross-modal semantic grounding feature map of the medical image. A query vector is constructed based on the medium-resolution cross-modal semantic grounding feature map of the medical image, and a value vector and a key vector are constructed based on the high-resolution feature map of the medical image. The query vector, the value vector, and the key vector are input into the fine-granularity grounding module to obtain the multi-scale cross-modal semantic grounding feature map of the medical image. The multi-scale cross-modal semantic grounding feature map of the medical image is input into the decoder to obtain the segmentation mask. The segmentation mask is applied to the original medical image to obtain the medical image segmentation result.

2. The medical image segmentation method based on image-text interaction according to claim 1, characterized in that, Inputting the original medical image into a medical-specific visual encoder to obtain a multi-scale visual feature map of the medical image includes: inputting the original medical image into a medical-specific visual encoder based on a Unet encoder to obtain a high-resolution feature map, a medium-resolution feature map, and a low-resolution feature map of the medical image, wherein the high-resolution feature map, the medium-resolution feature map, and the low-resolution feature map constitute the multi-scale visual feature map of the medical image.

3. The medical image segmentation method based on image-text interaction according to claim 1, characterized in that, The process involves text feature encoding and knowledge enhancement of the acquired user text description to obtain an enhanced text feature vector containing semantic information and anatomical structural context. This includes: inputting the user text description into a medical-specific language encoder based on the BioClinical BERT model to obtain an initial text embedding encoding vector; extracting medical anatomical entities from the user text description and using these entities to query a predefined medical anatomical structural knowledge graph to obtain relevant anatomical structural information; encoding the relevant anatomical structural information to obtain an anatomical knowledge embedding encoding vector; and inputting the initial text embedding encoding vector and the anatomical knowledge embedding encoding vector into a knowledge fusion module to obtain the enhanced text feature vector containing semantic information and anatomical structural context.

4. The medical image segmentation method based on image-text interaction according to claim 3, characterized in that, The process of inputting the initial text embedding encoding vector and the anatomical knowledge embedding encoding vector into the knowledge fusion module to obtain the enhanced text feature vector containing semantic information and anatomical structural context includes: the knowledge fusion module performing fully connected encoding on the initial text embedding encoding vector and the anatomical knowledge embedding encoding vector to obtain the enhanced text feature vector containing semantic information and anatomical structural context.

5. The medical image segmentation method based on image-text interaction according to claim 2, characterized in that, Based on the resolution cross-modal semantic grounding feature map of the medical image, a query vector is constructed, and a value vector and a key vector are constructed based on the high-resolution feature map of the medical image. The query vector, the value vector, and the key vector are input into a fine-grained grounding module to obtain the multi-scale cross-modal semantic grounding feature map of the medical image. This includes: constructing a query vector based on the resolution cross-modal semantic grounding feature map of the medical image, and constructing a value vector and a key vector based on the high-resolution feature map of the medical image; inputting the query vector, the value vector, and the key vector into a fine-grained grounding module to obtain an initial multi-scale cross-modal semantic grounding feature map of the medical image; and performing collaborative fusion optimization on the initial multi-scale cross-modal semantic grounding feature map of the medical image based on the hierarchical boundary segmentation representation of the low-resolution cross-modal semantic grounding feature map of the medical image, the resolution cross-modal semantic grounding feature map of the medical image, and the initial multi-scale cross-modal semantic grounding feature map of the medical image to obtain the multi-scale cross-modal semantic grounding feature map of the medical image.

6. The medical image segmentation method based on image-text interaction according to claim 5, characterized in that, Based on the hierarchical boundary segmentation representations of the low-resolution cross-modal semantic grounding feature map of the medical image, the mid-resolution cross-modal semantic grounding feature map of the medical image, and the initial multi-scale cross-modal semantic grounding feature map of the medical image, the initial multi-scale cross-modal semantic grounding feature map of the medical image is collaboratively fused and optimized to obtain the multi-scale cross-modal semantic grounding feature map of the medical image. This includes: calculating the boundary segmentation representation of the mid-resolution cross-modal semantic grounding feature map of the medical image relative to the low-resolution cross-modal semantic grounding feature map of the medical image to obtain a first boundary segmentation representation map; calculating the initial multi-scale cross-modal semantic grounding feature map of the medical image... A second boundary segmentation representation map is obtained by dividing the boundary segmentation representation of the semantic grounding feature map relative to the resolution cross-modal semantic grounding feature map in the medical image; based on the first boundary segmentation representation map and the second boundary segmentation representation map, a first-level overall cross-boundary segmentation association map and a second-level overall cross-boundary segmentation association map are constructed; based on the weighted sum of the first-level overall cross-boundary segmentation association map and the second-level overall cross-boundary segmentation association map as the cross-boundary segmentation correction map, the initial medical image multi-scale cross-modal semantic grounding feature map is multiplied and fused to obtain the medical image multi-scale cross-modal semantic grounding feature map.

7. The medical image segmentation method based on image-text interaction according to claim 6, characterized in that, A query vector is constructed based on the enhanced text feature vector containing semantic information and anatomical structural context, and a value vector and a key vector are constructed based on the low-resolution feature map of the medical image. The query vector, the value vector, and the key vector are input into a coarse-grained grounding module to obtain a low-resolution cross-modal semantic grounding feature map of the medical image. This includes: the coarse-grained grounding module processes the query vector, the value vector, and the key vector using the following formula to obtain the low-resolution cross-modal semantic grounding feature map of the medical image, wherein the formula is: ;in, This is a cross-modal semantic grounding feature vector. For query vectors, For key vectors, For value vectors, This is the transpose of the vector. For scaling factor, For the characteristic shape reshaping function, This is a low-resolution cross-modal semantic grounding feature map for medical images.

8. The medical image segmentation method based on image-text interaction according to claim 1, characterized in that, The decoder is a U-Net decoder.

9. A medical image segmentation system based on image-text interaction, characterized in that, include: The raw medical image acquisition module is used to acquire raw medical images; The visual feature extraction module is used to input the original medical image into a medical-specific visual encoder to obtain a multi-scale visual feature map of the medical image; The text feature enhancement module is used to encode and enhance the acquired user text description to obtain an enhanced text feature vector containing semantic information and anatomical structural context. The multimodal fusion module is used to perform multi-granular semantic grounding and collaborative fusion of the multi-scale visual feature map of the medical image and the enhanced text feature vector containing semantic information and anatomical structural context to obtain a multi-scale cross-modal semantic grounding feature map of the medical image. This includes: constructing a query vector based on the enhanced text feature vector containing semantic information and anatomical structural context, and constructing a value vector and a key vector based on the low-resolution feature map of the medical image; inputting the query vector, the value vector, and the key vector into the coarse-grained grounding module to obtain a low-resolution cross-modal semantic grounding feature map of the medical image; and based on the low-resolution cross-modal feature map of the medical image... A query vector is constructed from the semantic grounding feature map of the medical image, and a value vector and a key vector are constructed based on the mid-resolution feature map of the medical image. The query vector, the value vector, and the key vector are input into the mid-granular grounding module to obtain the mid-resolution cross-modal semantic grounding feature map of the medical image. A query vector is constructed based on the mid-resolution cross-modal semantic grounding feature map of the medical image, and a value vector and a key vector are constructed based on the high-resolution feature map of the medical image. The query vector, the value vector, and the key vector are input into the fine-granular grounding module to obtain the multi-scale cross-modal semantic grounding feature map of the medical image. A segmentation decoding module is used to input the multi-scale cross-modal semantic grounding feature map of the medical image into a decoder to obtain a segmentation mask. A segmentation result generation module is used to apply the segmentation mask to the original medical image to obtain the medical image segmentation result.

Images