Medical image segmentation method and system based on image-text interaction

Abstract

The invention discloses a medical image segmentation method and system based on image-text interaction, and relates to the technical field of image segmentation, and the method comprises the steps: firstly obtaining a medical image, extracting a multi-scale visual feature, carrying out the deep analysis of a user text description through a medical knowledge graph, and carrying out the knowledge enhancement through a medical anatomical knowledge graph; and thus, an enhanced text vector fusing deep semantics and precise anatomical context is constructed. Furthermore, through a multi-granularity semantic grounding and collaborative fusion mechanism, progressive cross-modal alignment and information interaction are carried out on enhanced text vectors and multi-scale visual features, model focusing is guided, a target area is accurately positioned, and finally a high-precision segmentation mask is generated by a decoder. Therefore, the flexibility of the natural language and the accuracy of the medical priori knowledge are combined, the segmentation challenge in a complex or fuzzy scene can be effectively overcome, and the accuracy and robustness of the segmentation task are remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of image segmentation technology, and more specifically, to a medical image segmentation method and system based on image-text interaction. Background Art

[0002] With the rapid development of precision medicine and medical imaging technologies, clinical diagnosis and treatment practices are placing higher demands on the intelligent and precise analysis and recognition of medical images. As the core basis for disease diagnosis, the precise segmentation of lesions in medical images is directly related to diagnostic accuracy, treatment plan formulation, and efficacy evaluation. Especially in the diagnosis and treatment of complex diseases such as tumors, neurodegenerative diseases, and vascular lesions, high-precision image segmentation is a key step in improving medical decision-making efficiency and achieving personalized treatment.

[0003] In existing technologies, medical image segmentation primarily relies on manual annotation or automated algorithms based on a single image modality (such as MRI, CT, or ultrasound). Manual segmentation not only consumes a significant amount of expert time and effort, but is also subject to high subjectivity and poor repeatability, making it difficult to meet the demands of processing massive amounts of image data. While traditional automated segmentation methods can provide assistance in standardized scenarios, their processing pipelines typically focus on low-level pixel-level feature analysis (such as grayscale, texture, and edges), often overlooking the high-level semantic knowledge contained in rich textual information such as clinical reports and physician-annotated illustrations (such as lesion descriptions, boundary indications, and structural annotations). This lack of semantic information results in a significant lack of robustness in algorithms when faced with complex situations such as image artifacts, blurred boundaries between lesions and normal tissue (such as invasive tumors), or highly variable anatomical structures. This significantly reduces segmentation accuracy and makes it prone to under-segmentation, over-segmentation, and boundary positioning errors.

[0004] Therefore, there is an urgent need for a medical image segmentation method and system based on image-text interaction. Summary of the Invention

[0005] In order to solve the above technical problems, this application is proposed.

[0006] According to one aspect of the present application, a medical image segmentation method based on image-text interaction is provided, which includes: obtaining an original medical image; inputting the original medical image into a medical-specific visual encoder to obtain a multi-scale visual feature map of the medical image; performing text feature encoding and knowledge enhancement on the obtained user text description to obtain an enhanced text feature vector containing semantic information and anatomical structure context; performing multi-granularity semantic grounding and collaborative fusion on the multi-scale visual feature map of the medical image and the enhanced text feature vector containing semantic information and anatomical structure context to obtain a multi-scale cross-modal semantic grounding feature map of the medical image; inputting the multi-scale cross-modal semantic grounding feature map of the medical image into a decoder to obtain a segmentation mask; and applying the segmentation mask to the original medical image to obtain a medical image segmentation result.

[0007] According to another aspect of the present application, a medical image segmentation system based on image-text interaction is provided, which includes: an original medical image acquisition module for acquiring original medical images; a visual feature extraction module for inputting the original medical image into a medical-specific visual encoder to obtain a multi-scale visual feature map of the medical image; a text feature enhancement module for performing text feature encoding and knowledge enhancement on the acquired user text description to obtain an enhanced text feature vector containing semantic information and anatomical structure context; a multimodal fusion module for performing multi-granularity semantic grounding and collaborative fusion on the multi-scale visual feature map of the medical image and the enhanced text feature vector containing semantic information and anatomical structure context to obtain a multi-scale cross-modal semantic grounding feature map of the medical image; a segmentation decoding module for inputting the multi-scale cross-modal semantic grounding feature map of the medical image into a decoder to obtain a segmentation mask; and a segmentation result generation module for applying the segmentation mask to the original medical image to obtain a medical image segmentation result.

[0008] Compared with the existing technology, the present application provides a medical image segmentation method and system based on image-text interaction, which first acquires medical images and extracts multi-scale visual features, uses medical knowledge graphs to deeply analyze user text descriptions, and uses medical anatomical knowledge graphs for knowledge enhancement, thereby constructing an enhanced text vector that combines deep semantics and precise anatomical context. Furthermore, through multi-granular semantic grounding and collaborative fusion mechanisms, the enhanced text vectors and multi-scale visual features are progressively cross-modally aligned and information interacted, guiding the model to focus on and accurately locate the target area, and finally the decoder generates a high-precision segmentation mask. In this way, the flexibility of natural language and the accuracy of medical prior knowledge are effectively combined, which can effectively overcome the segmentation challenges in complex or ambiguous scenes and significantly improve the accuracy and robustness of the segmentation task. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other purposes, features, and advantages of the present application will become more apparent through a more detailed description of the embodiments of the present application in conjunction with the accompanying drawings. The accompanying drawings are intended to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the drawings, the same reference numerals generally represent the same components or steps.

[0010] Figure 1 Flowchart of a medical image segmentation method based on image-text interaction according to an embodiment of the present application.

[0011] Figure 2 This is a data flow diagram of a medical image segmentation method based on image-text interaction according to an embodiment of the present application.

[0012] Figure 3 This is a flowchart of sub-step S3 of the medical image segmentation method based on image-text interaction according to an embodiment of the present application.

[0013] Figure 4 This is a flowchart of sub-step S4 of the medical image segmentation method based on image-text interaction according to an embodiment of the present application.

[0014] Figure 5 4 is a block diagram of a medical image segmentation system based on image-text interaction according to an embodiment of the present application. DETAILED DESCRIPTION

[0015] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. While the drawings illustrate certain embodiments of the present disclosure, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0016] In response to the problems in the above background technology, this application proposes a medical image segmentation method based on image-text interaction. Figure 1 Flowchart of a medical image segmentation method based on image-text interaction according to an embodiment of the present application. Figure 2 Figure 1 is a data flow diagram of a medical image segmentation method based on image-text interaction according to an embodiment of the present application. Figure 1 and Figure 2As shown, the medical image segmentation method based on image-text interaction includes the following steps: S1, obtaining an original medical image; S2, inputting the original medical image into a medical-specific visual encoder to obtain a multi-scale visual feature map of the medical image; S3, performing text feature encoding and knowledge enhancement on the obtained user text description to obtain an enhanced text feature vector containing semantic information and anatomical structure context; S4, performing multi-granularity semantic grounding and collaborative fusion on the multi-scale visual feature map of the medical image and the enhanced text feature vector containing semantic information and anatomical structure context to obtain a multi-scale cross-modal semantic grounding feature map of the medical image; S5, inputting the multi-scale cross-modal semantic grounding feature map of the medical image into a decoder to obtain a segmentation mask; S6, applying the segmentation mask to the original medical image to obtain a medical image segmentation result.

[0017] In the above-mentioned medical image segmentation method based on image-text interaction, step S1, obtaining the original medical image, should be understood to be accurately collecting original medical images containing the lesion area from medical imaging equipment such as CT, MRI, and ultrasound, ensuring the integrity, clarity, and format standardization of the data, providing high-quality original materials for subsequent cross-modal semantic fusion and precise segmentation, and providing more reliable data support for the subsequent extraction of multi-scale visual features and fusion of text semantic information.

[0018] In particular, in one possible embodiment, step S1 is implemented as follows: First, a standardized device access interface is established that is compatible with different types of medical imaging devices. Communication between the device and the system is achieved through standard medical digital imaging and communication protocols, ensuring stable transmission of image data to the system server. Next, an image data import operation is performed, batch importing historical image data and acquiring newly acquired images in real time. During the import process, metadata such as patient basic information, examination time, and device model are simultaneously recorded to facilitate subsequent data management and traceability. Subsequently, rigorous data verification is performed. Algorithms automatically detect image quality indicators such as resolution, grayscale value range, and the presence of artifacts or noise. Furthermore, manual visual verification of key images is allowed to eliminate non-compliant data, ensuring the quality and reliability of the original medical images. After verification, non-standard image formats output by different devices are converted to a standard medical digital imaging and communication format recognized by the system. Image data is also compressed and encrypted, reducing storage space and improving security while ensuring data accuracy. Finally, the processed original medical images are indexed according to dimensions such as patient ID and examination type, and stored in a distributed database or storage server to form a structured image data pool. This provides efficient data access support for subsequent rapid call-up of original medical images and association processing with user text descriptions, ensuring the accuracy and availability of the entire medical image segmentation process at the original data level.

[0019] In the above-mentioned medical image segmentation method based on image-text interaction, in step S2, the original medical image is input into a medical-specific visual encoder to obtain a multi-scale visual feature map of the medical image. It should be understood that due to complex situations such as interference from image artifacts, blurred boundaries between lesions and normal tissues, or highly variable anatomical structures, the image segmentation accuracy is significantly reduced. Therefore, the present application extracts a multi-scale visual feature map of the medical image containing visual information of different scales through a medical-specific visual encoder, which not only retains the detailed texture of the original image, but also extracts high-level semantic information, providing a rich visual information foundation for subsequent feature cross-modal fusion. Specifically, the present application inputs the original medical image into a medical-specific visual encoder based on a Unet encoder to obtain a high-resolution feature map of the medical image, a medium-resolution feature map of the medical image, and a low-resolution feature map of the medical image. The high-resolution feature map of the medical image, the medium-resolution feature map of the medical image, and the low-resolution feature map of the medical image constitute the multi-scale visual feature map of the medical image.

[0020] In particular, in one possible embodiment, the implementation process of step S2 is as follows: first, the original medical image is input into a medical-specific visual encoder based on the Unet encoder, and the image is subjected to feature extraction through multi-layer convolution operations. Each layer of convolution is dedicated to capturing visual features at different levels, and the downsampling operation enables the model to understand the image content from different scales, laying the foundation for subsequent multi-granularity semantic grounding and collaborative fusion. Specifically, the Unet encoder adopts a classic encoder-decoder architecture. Its encoder part gradually reduces the spatial resolution of the feature map and increases the number of channels through continuous convolution and downsampling operations, thereby extracting high-resolution feature maps of medical images, medium-resolution feature maps of medical images, and low-resolution feature maps of medical images in turn. Among them, the high-resolution feature map of medical images retains the detailed information of the image, such as the edge and texture of the lesion, while the medium-resolution feature map of medical images takes into account both local structure and semantic information. The low-resolution feature map of medical images contains the overall semantics and contextual information of the image. The three feature maps of different resolutions together constitute the multi-scale visual feature map of medical images.

[0021] In the above-mentioned medical image segmentation method based on image-text interaction, step S3 performs text feature encoding and knowledge enhancement on the acquired user text description to obtain an enhanced text feature vector containing semantic information and anatomical structure context. By encoding and knowledge-enhancing the user text description, the semantic information in the text and medical anatomical structure knowledge are integrated into the model, effectively constructing an enhanced text feature vector that integrates deep semantics and precise anatomical context. This not only retains the semantic information in the user text description, but also incorporates the structured knowledge of the medical anatomical knowledge graph. This provides a rich semantic foundation for subsequent cross-modal fusion with multi-scale visual features of medical images, significantly improving the accuracy and robustness of the segmentation task.

[0022] In particular, in one possible embodiment, Figure 3 FIG is a flowchart of sub-step S3 of the medical image segmentation method based on image-text interaction according to an embodiment of the present application. Figure 3 As shown, the step S3 includes: S31, inputting the user text description into a medical-specific language encoder based on the BioClinical BERT model to obtain an initial text embedding coding vector; S32, extracting medical anatomical entities from the user text description, and using the medical anatomical entities to query in a predefined medical anatomical structure knowledge graph to obtain relevant anatomical structure information; S33, encoding the relevant anatomical structure information to obtain an anatomical knowledge embedding coding vector; S34, inputting the initial text embedding coding vector and the anatomical knowledge embedding coding vector into a knowledge fusion module to obtain the enhanced text feature vector containing semantic information and anatomical structure context.

[0023] Specifically, in step S31, the user text description is input into a medical-specific language encoder based on the BioClinical BERT model to obtain an initial text embedding encoding vector. It should be understood that the BioClinical BERT model is a pre-trained language model based on the BERT architecture, which is specially optimized for clinical texts, inherits the bidirectional encoder structure of BERT, and performs well in processing medical texts by pre-training on a large amount of clinical texts. Specifically, this application utilizes the advantages of the BioClinical BERT model in medical text understanding to convert the user text description into an initial text embedding encoding vector containing basic semantic information, retaining the original semantic information of the user text description, and providing a semantic basis for the subsequent extraction of medical anatomical entities from the text, thereby more accurately mapping the relationship between clinical needs and anatomical structures, and providing high-quality semantic input for subsequent knowledge enhancement in combination with the medical anatomical knowledge graph, thereby improving the entire segmentation model's ability to understand text semantics and laying a solid foundation for subsequent cross-modal fusion.

[0024] In particular, in a possible embodiment, the implementation process of step S31 is as follows: first, obtain the text description input by the user, such as the description of lesion features and anatomical structure annotations in the clinical report, and preprocess it to obtain a sequence that conforms to the model input format, such as performing standardization processing such as word segmentation and removal of special characters. Secondly, the preprocessed text sequence is input into the medical-specific language encoder based on the BioClinical BERT model, which uses a multi-layer bidirectional Transformer architecture to perform contextual semantic encoding on each token in the text sequence, and uses the self-attention mechanism to capture the long-distance dependency between words, especially for polysemous words and professional terms commonly found in medical texts. Targeted semantic disambiguation and feature extraction are performed. During the encoding process, the model will strengthen the representation of key semantic information such as anatomical structure and lesion properties according to the characteristics of the medical text, and suppress irrelevant noise. Finally, the embedded representation of the text sequence is extracted from the output layer of the model to form an initial text embedding encoding vector, which not only contains the lexical and syntactic features of the text, but also integrates the semantic associations unique to the medical field.

[0025] Specifically, the step S32 extracts medical anatomical entities from the user text description, and uses the medical anatomical entities to query in a predefined medical anatomical structure knowledge graph to obtain relevant anatomical structure information. That is, using the predefined medical anatomical structure knowledge graph, the medical anatomical entities in the user text are converted into structured anatomical context information, thereby providing semantic enhancement of the anatomical dimension for the initial text embedding coding vector. Specifically, by extracting anatomical entities, such as the left lobe of the liver, pituitary gland, etc., and querying their anatomical position, adjacent relationship, blood supply system and other information in the knowledge graph, it is possible to construct an enhanced text feature that integrates text semantics and anatomical structure, provide anatomical prior knowledge support for the subsequent multimodal fusion process, and enable the model to understand the spatial relationship between the lesion and the surrounding tissue based on the anatomical relationship when processing the image, and thus focus on the target area more accurately.

[0026] Specifically, in step S33, the relevant anatomical structure information is encoded to obtain an anatomical knowledge embedding coding vector. It should be understood that although the medical anatomical entities extracted from the user's text description can preliminarily indicate the segmentation target, they lack deep knowledge such as the spatial relationship and physiological characteristics between anatomical structures. Based on this, the present application converts relevant anatomical structure information, such as the adjacent relationship of the target organ, anatomical hierarchy, morphological characteristics, etc., into a dense anatomical knowledge embedding coding vector through a specific information encoding mechanism, thereby providing a computable anatomical semantic representation for the subsequent knowledge fusion module.

[0027] In particular, in a possible embodiment, the implementation process of step S33 is as follows: Since the relevant anatomical structure information exists in the form of structured data, including the attributes, relationships and topological structures of anatomical entities, when the relevant anatomical structure information is subjected to feature extraction and vectorization processing, the attribute information in the atlas can be encoded by a multi-layer perceptron, and the multi-layer perceptron can process such multi-dimensional and multi-type structured data through nonlinear transformation. It should be understood that the multi-layer perceptron is a feedforward artificial neural network composed of multiple fully connected neuron layers, usually including an input layer, one or more hidden layers and an output layer. Specifically, the anatomical structure information is first preprocessed, the text attributes are converted into vectors through word embedding, the numerical attributes are normalized, the relational information is converted into numerical features through graph structure feature extraction, and the various features are spliced ​​into a mixed vector. Then, the hidden layer of the multi-layer perceptron adopts a multi-layer fully connected structure, and each layer of neurons transforms the input through a weight matrix and a bias term, and an activation function is introduced to enhance the nonlinear expression capability. For example, two hidden layers are set up. The first layer maps the mixed vector to a higher-dimensional space, such as from 100 to 256 dimensions, to capture the deep associations of anatomical structural information. The second layer then reduces the dimensionality to the target dimension, such as 128, to ensure that the vector can both retain anatomical knowledge details and meet the dimensionality requirements of the subsequent fusion module. Finally, the output layer maps the hidden layer features to an anatomical knowledge embedding encoding vector of a specified dimension through a fully connected layer. For example, the dimension is consistent with the initial text embedding vector, assuming it is 768 dimensions, to ensure that the generated anatomical knowledge embedding encoding vector can be processed with the initial text embedding encoding vector in the subsequent knowledge fusion module.

[0028] Specifically, in step S34, the initial text embedding coding vector and the anatomical knowledge embedding coding vector are input into the knowledge fusion module to obtain the enhanced text feature vector containing semantic information and anatomical structure context. Wherein, the knowledge fusion module performs full-connection coding on the initial text embedding coding vector and the anatomical knowledge embedding coding vector to obtain the enhanced text feature vector containing semantic information and anatomical structure context. Specifically, the initial text embedding coding vector and the anatomical knowledge embedding coding vector are fused across dimensions so that the text features can not only reflect the semantics of the lesion described by the user, but also be associated with prior knowledge such as the spatial position and adjacent relationship of the anatomical structure, providing a text feature basis with both semantic understanding and anatomical constraints for subsequent cross-modal fusion, thereby guiding the model to accurately locate the target area during image segmentation. The feature fusion achieved by full-connection coding generates a fused enhanced text feature vector, which not only retains the semantic flexibility of the text description, but also introduces precise anatomical structure constraints in the medical field through knowledge enhancement of the anatomical knowledge graph, so that the model can use the anatomical knowledge context to assist semantic understanding and reduce segmentation deviation when facing complex scenes such as image artifacts and blurred boundaries.

[0029] In particular, in a possible embodiment, the implementation process of step S34 is as follows: the knowledge fusion module learns the parameters of the fully connected layer, the number of neurons in the fully connected layer can be set according to the task requirements, such as 768 dimensions, and the input initial text embedding coding vector and the anatomical knowledge embedding coding vector are linearly transformed through the weight matrix and the bias term, and the weights of the two types of features are assigned. During the linear transformation process, the weights of the fully connected layer will learn how to efficiently combine semantic information and anatomical knowledge information, such as strengthening the weights of anatomical structure features related to the current segmentation task. After being processed by the activation function, the output fusion vector is normalized and other operations are performed to finally generate an enhanced text feature vector containing semantic information and anatomical structure context.

[0030] In the above-mentioned medical image segmentation method based on image-text interaction, step S4 performs multi-granular semantic grounding and collaborative fusion on the multi-scale visual feature map of the medical image and the enhanced text feature vector containing semantic information and anatomical structure context to obtain a multi-scale cross-modal semantic grounding feature map of the medical image. It should be understood that through the multi-granular cross-modal fusion mechanism, text semantic information can be gradually grounded into image features, compensating for segmentation deviations caused by modality differences. Specifically, low-resolution visual features and text features are first coarsely aligned to locate the global position of the target region. Then, medium-resolution features are used to refine the region boundaries. Finally, high-resolution features are used to capture fine structures, thereby constructing a multi-scale cross-modal semantic grounding feature map of the medical image that integrates cross-modal semantics. This layered fusion mechanism enables the model to combine text semantics and anatomical knowledge at different scales, thereby gradually focusing on the target region, solving the problem that single-scale features cannot balance global positioning and detailed segmentation. Through multi-granular cross-modal fusion, the model can use the anatomical prior knowledge in text semantics to guide the semantic understanding of image features at different resolutions.

[0031] In particular, in one possible embodiment, Figure 4 FIG is a flowchart of sub-step S4 of the medical image segmentation method based on image-text interaction according to an embodiment of the present application. Figure 4As shown, the step S4 includes: S41, constructing a query vector based on the enhanced text feature vector containing semantic information and anatomical structure context and constructing a value vector and a key vector based on the low-resolution feature map of the medical image, and inputting the query vector, the value vector and the key vector into a coarse-grained grounding module to obtain a low-resolution cross-modal semantic grounding feature map of the medical image; S42, constructing a query vector based on the low-resolution cross-modal semantic grounding feature map of the medical image and constructing a value vector and a key vector based on the medium-resolution feature map of the medical image, and inputting the query vector, the value vector and the key vector into a medium-grained grounding module to obtain a medium-resolution cross-modal semantic grounding feature map of the medical image; S43, constructing a query vector based on the medium-resolution cross-modal semantic grounding feature map of the medical image and constructing a value vector and a key vector based on the high-resolution feature map of the medical image, and inputting the query vector, the value vector and the key vector into a fine-grained grounding module to obtain the multi-scale cross-modal semantic grounding feature map of the medical image.

[0032] Specifically, step S41 constructs a query vector based on the enhanced text feature vector containing semantic information and anatomical structure context, and constructs a value vector and a key vector based on the low-resolution feature map of the medical image, and inputs the query vector, the value vector, and the key vector into a coarse-grained grounding module to obtain a low-resolution cross-modal semantic grounding feature map of the medical image. In one embodiment of the present application, the coarse-grained grounding module processes the query vector, the value vector, and the key vector according to the following formula to obtain the low-resolution cross-modal semantic grounding feature map of the medical image, wherein the formula is: ;in, Cross-modal semantically grounded feature vector, is the query vector, is the key vector, is a value vector, is the transpose of the vector, is the scaling factor, Reshape function for the feature, Low-resolution cross-modal semantically grounded feature maps for medical images.

[0033] That is, the global positioning of the target area is achieved through coarse-grained cross-modal interaction, with the enhanced text feature vector as the query vector, while the key vector and value vector generated by the low-resolution feature map represent the global structural information of the image. Through the attention mechanism of the coarse-grained grounding module, the model can calculate the association weights between text semantics and the global structure of the image, thereby locating the area matching the text description in the low-resolution cross-modal semantic grounding feature map of the medical image, such as quickly locking the approximate range of the right lobe of the liver in the whole liver CT image, thereby providing an initial semantic anchor for subsequent medium and fine-grained feature fusion. Specifically, the low-resolution cross-modal semantic grounding feature map of the medical image generated after coarse-grained cross-modal fusion can not only retain the global structure of the image, but also embed the anatomical constraints of the text semantics. In this way, in the complex background of multiple organs, the focus of subsequent feature processing can be effectively narrowed, computational redundancy can be reduced, and a more accurate initial area can be provided for medium-grained boundary refinement, which significantly enhances the robustness of the overall segmentation process.

[0034] Specifically, the step S42 constructs a query vector based on the low-resolution cross-modal semantic grounding feature map of the medical image and constructs a value vector and a key vector based on the medium-resolution feature map of the medical image, and inputs the query vector, the value vector and the key vector into the medium-granularity grounding module to obtain the medium-resolution cross-modal semantic grounding feature map of the medical image. It should be understood that through medium-resolution cross-modal fusion, it is possible to guide the semantic alignment of the details of the medium-resolution features and compensate for the semantic uncertainty during boundary refinement. Specifically, through the attention mechanism of the medium-granularity grounding module, the model can calculate the association weights between coarse-grained semantics and medium-resolution structures, thereby strengthening the boundary details that match the coarse-grained semantics, such as refining the contact boundary between the tumor and the right branch of the portal vein, and providing transitional features for the subsequent fine-grained capture of texture details. The generated medium-resolution cross-modal semantic grounded feature map of medical images can not only retain the local structural details of medium-resolution visual features, such as the direction of blood vessel trunks and the outline of organ lobes, but also embed boundary constraints guided by coarse-grained semantics, such as the preliminary boundaries of tumors invading blood vessels, effectively reducing the segmentation error caused by boundary blur, providing a more accurate boundary feature basis for fine-grained feature fusion, and significantly improving the contour fit of the overall segmentation result.

[0035] In particular, in a possible embodiment, the implementation process of step S42 is as follows: First, the low-resolution cross-modal semantic grounding feature map of the medical image is global average pooled into a query vector of the same dimension as the enhanced text feature vector, such as 768 dimensions. At the same time, the medium-resolution feature map of the medical image is linearly transformed twice to generate a key vector and a value vector. Then, the query vector, key vector and value vector are input into the medium-granularity grounding module. The module adopts a scaled dot product attention mechanism similar to the coarse-grained one. First, the transposed matrix product of the query vector and the key vector is calculated, and then divided by the scaling factor to stabilize the gradient, where the scaling factor is the dimension of the query vector. Then, it is multiplied with the value vector to obtain a cross-modal semantic grounding feature vector. Subsequently, the cross-modal semantic grounding feature vector is converted to the same spatial size as the medium-resolution feature map through a feature reshaping operation to generate a medium-resolution cross-modal semantic grounding feature map of the medical image.

[0036] Specifically, the step S43 constructs a query vector based on the medium-resolution cross-modal semantic grounding feature map of the medical image and constructs a value vector and a key vector based on the high-resolution feature map of the medical image, and inputs the query vector, the value vector and the key vector into the fine-grained grounding module to obtain the multi-scale cross-modal semantic grounding feature map of the medical image. It should be understood that through the attention mechanism of the fine-grained grounding module, the association weights of the medium-resolution semantics and the high-resolution details are calculated, the microstructure that matches the semantics is strengthened, and the decoder is provided with feature inputs that have both anatomical semantics and pixel-level details. The generated multi-scale cross-modal semantic grounding feature map of medical images not only retains the texture details of the high-resolution visual features, but also embeds the detailed semantic constraints guided by the medium-resolution semantics, effectively enhancing the model's semantic perception ability of subtle structures.

[0037] In particular, in a possible embodiment, the implementation process of step S43 is as follows: first, the mid-resolution cross-modal semantic grounding feature map of the medical image is subjected to global average pooling to obtain a query vector. At the same time, the high-resolution feature map of the medical image is linearly transformed twice to generate a key vector and a value vector. Then, the query vector, key vector and value vector are input into the fine-grained grounding module. The module adopts a scaled dot product attention mechanism similar to the coarse-grained one. First, the transposed matrix product of the query vector and the key vector is calculated, and then divided by the scaling factor to stabilize the gradient, where the scaling factor is the dimension of the query vector. Then, it is multiplied with the value vector to obtain a cross-modal semantic grounding feature vector. Subsequently, the cross-modal semantic grounding feature vector is converted to the same spatial size as the high-resolution feature map of the medical image through a feature reshaping operation to generate a multi-scale cross-modal semantic grounding feature map of the medical image.

[0038] Here, by using the low-resolution image semantics as the query vector, the query-key-value reshaping of the high-resolution image semantics can be performed. This can use the low-resolution image semantics as the predicted boundary to reshape the boundary saliency of the high-resolution image semantics, thereby achieving the collaborative fusion of the key-value structure of the image semantics relative to the query initialization.

[0039] However, considering the possible lack of boundary saliency of low-resolution image semantics as predicted boundaries, initialization through key-value structure query may result in cross-boundary disconnection between semantics of images of different resolutions, which affects the overall collaborative fusion effect.

[0040] Based on this, in another embodiment, a query vector is constructed based on the medium-resolution cross-modal semantic grounding feature map of the medical image and a value vector and a key vector are constructed based on the high-resolution feature map of the medical image, and the query vector, the value vector and the key vector are input into a fine-grained grounding module to obtain the multi-scale cross-modal semantic grounding feature map of the medical image, including: constructing a query vector based on the medium-resolution cross-modal semantic grounding feature map of the medical image and a value vector and a key vector based on the high-resolution feature map of the medical image, and inputting the query vector, the value vector and the key vector into a fine-grained grounding module to obtain an initial multi-scale cross-modal semantic grounding feature map of the medical image; based on the step-by-step boundary segmentation representation of the low-resolution cross-modal semantic grounding feature map of the medical image, the medium-resolution cross-modal semantic grounding feature map of the medical image and the initial multi-scale cross-modal semantic grounding feature map of the medical image, the initial multi-scale cross-modal semantic grounding feature map of the medical image is collaboratively fused and optimized to obtain the multi-scale cross-modal semantic grounding feature map of the medical image.

[0041] Specifically, based on the step-by-step boundary segmentation representation of the low-resolution cross-modal semantic grounding feature map of the medical image, the medium-resolution cross-modal semantic grounding feature map of the medical image, and the initial multi-scale cross-modal semantic grounding feature map of the medical image, the initial multi-scale cross-modal semantic grounding feature map of the medical image is collaboratively fused and optimized to obtain the multi-scale cross-modal semantic grounding feature map of the medical image, including: calculating the boundary segmentation representation of the medium-resolution cross-modal semantic grounding feature map of the medical image relative to the low-resolution cross-modal semantic grounding feature map of the medical image to obtain a first boundary segmentation representation map, expressed as: ;in, To calculate the gradient, It is a low-resolution cross-modal semantic grounding feature map for medical images. is a high-resolution cross-modal semantic grounding feature map for medical images, Indicates subtraction by position, This is the first boundary segmentation representation diagram.

[0042] Specifically, to capture the differences in boundary details between feature maps of different resolutions, a boundary segmentation representation is calculated for the medium-resolution cross-modal semantic grounding feature map of the medical image relative to the low-resolution cross-modal semantic grounding feature map of the medical image, thereby obtaining a first boundary segmentation representation. In real-world medical imaging scenarios, such as a tumor in a liver CT image, a low-resolution feature map may only roughly identify a space-occupying lesion within the liver, but its specific boundaries and morphology are not yet clear. In contrast, a medium-resolution feature map begins to reveal the local texture and general outline of the tumor. By calculating the boundary segmentation representation between the two, the refinement trends and differences in the edges of lesions or anatomical structures during the transition from a global overview to local details can be quantified and extracted. This can initially reveal the difference between the model's grasp of the overall tumor contour and its depiction of local details during the semantic grounding process, for example, the initial distinction between the tumor and the liver parenchyma, providing a basis for subsequent fine-tuning.

[0043] Calculate the boundary segmentation representation of the initial medical image multi-scale cross-modal semantic grounding feature map relative to the medical image medium-resolution cross-modal semantic grounding feature map to obtain a second boundary segmentation representation map, which is expressed as: ;in, To calculate the gradient, is the initial medical image multi-scale cross-modal semantic grounding feature map, is a high-resolution cross-modal semantic grounding feature map for medical images, Indicates subtraction by position, This is the second boundary segmentation representation diagram.

[0044] Specifically, to further improve boundary accuracy, a boundary segmentation representation is calculated for the initial multi-scale cross-modal semantic grounded feature map of the medical image relative to the medium-resolution cross-modal semantic grounded feature map of the medical image, thereby obtaining a second boundary segmentation representation. It should be understood that in clinical practice, many lesions, especially invasive tumors or early-stage lesions, may have boundaries that exhibit highly similar grayscale values ​​or textures to surrounding normal tissue, making them difficult to accurately distinguish at a single resolution. Although the initial multi-scale cross-modal semantic grounded feature map of the medical image incorporates high-resolution visual information, its boundaries may still be affected by the coarseness of the medium-resolution feature map, or uncertainty may remain when dealing with lesions with blurred boundaries. Therefore, the overall contour directional difference between the feature manifolds at each level is used as a boundary segmentation representation to measure the cross-boundary directional distribution difference of the feature manifold. This further captures the further refinement and correction of boundary delineation from the intermediate to the finest granularity feature maps. Quantifying this difference is particularly critical when dealing with lesions whose grayscale values ​​are similar to those of normal tissue and difficult to distinguish. In this way, the direction and degree of fine-tuning of the tumor edge by the model during the refinement process can be more accurately identified, for example, distinguishing the tiny infiltration edge of the tumor from the normal vascular structure, providing more detailed guidance for the final boundary optimization.

[0045] Then, the peak-valley slope derivatives between the boundary breaks at each level are further determined. That is, if the boundary breaks at each level are represented as a two-way cross-boundary slope distribution, the differentiable peak-valley derivatives of their difference information can be extracted as the overall cross-boundary break association between the levels. That is, based on the first boundary break representation graph and the second boundary break representation graph, the overall cross-boundary break association graph between the first level and the overall cross-boundary break association graph between the second level are constructed, which can be expressed as: ;in, is the first boundary segmentation representation, is the second boundary segmentation representation diagram, is the second boundary segmentation representation diagram, It is the overall cross-boundary segmentation association diagram between the first level, This is the overall cross-boundary segmentation association diagram between the second level.

[0046] It should be understood that relying solely on boundary differences between a single level is insufficient to fully reflect the complexity of lesion boundaries, especially in medical imaging, where lesion boundaries may exhibit different characteristics at different scales, such as the macroscopic morphology of a tumor and the roughness of its microscopic edges. By viewing the level-by-level boundary segmentation representation as a two-way cross-boundary slope distribution and extracting the differentiable peak-to-valley derivatives of its difference information, we analyze the intrinsic correlation and directional distribution differences of boundary information between feature maps of different granularities, thereby gaining a more comprehensive understanding of the significant loss and directional characteristics of lesion edges. This helps the model understand why the boundaries of certain regions are blurred at low resolution but can be enhanced by textual descriptions (such as boundary blur) at high resolution, thereby more accurately capturing the true boundaries of the lesion.

[0047] Finally, based on the overall cross-boundary segmentation association diagram between the first level The overall cross-boundary disconnection diagram between the second level The weighted sum of is used as the cross-boundary segmentation correction map, and the initial medical image multi-scale cross-modal semantic grounding feature map is point-multiplied and fused to obtain the medical image multi-scale cross-modal semantic grounding feature map, which is expressed as: ;in, is the initial medical image multi-scale cross-modal semantic grounding feature map, It is the overall cross-boundary segmentation association diagram between the first level, It is the overall cross-boundary segmentation association diagram between the second level, and are different weight coefficients, which are determined by adaptive learning through end-to-end training. For example, when dealing with large, well-defined tumors, Of course, this is just an example and does not constitute a specific limitation. It is point multiplication by position, that is, each element of the feature map is multiplied by the corresponding coefficient or the value corresponding to the feature map. To add by position, Enhanced low-resolution cross-modal semantic grounding feature map of medical images.

[0048] That is, by introducing the peak-valley distribution characteristics of boundary segmentation during the cross-boundary segmentation process, query-initialized structural edge reinforcement can be performed for missing boundary saliency, and adaptive collaboration can be achieved using directional distribution difference sensitivity, that is, matching cross-boundary edge directionality with difference, thereby improving the collaborative fusion effect of the medical image multi-scale cross-modal semantic grounding feature map for query initialization based on the key-value structure of image semantics at different granularities. Specifically, the first-level inter-level overall cross-boundary segmentation association map and the second-level inter-level overall cross-boundary segmentation association map have accurately captured the differences and correction information in boundary details between feature maps of different granularities, and now this correction information is effectively applied to the current initial medical image multi-scale cross-modal semantic grounding feature map. In medical image segmentation, this means that coarse-to-fine boundary correction information, such as the semantic guidance provided by the text description of the blurred boundary of the right lobe of the liver and hepatocellular carcinoma, is accurately mapped to the pixel level. A comprehensive cross-boundary segmentation correction map is formed through weighted summation. Its purpose is to integrate boundary correction information between different levels and perform pixel-level adaptive enhancement on the initial fine-grained feature map through point product fusion. The execution effect of this point product fusion is that it can query the initialization structure edge enhancement in areas where boundary saliency is missing, and use the sensitivity of the directional distribution difference of the boundary for adaptive collaboration. As a result, the final multi-scale cross-modal semantic grounding feature map of medical images retains high-order semantic information while depicting the lesion boundary more accurately and clearly, significantly improving the segmentation accuracy and robustness in complex medical imaging scenarios, such as those with blurred tumor infiltration boundaries or tiny lesions.

[0049] In the above-mentioned medical image segmentation method based on image-text interaction, step S5 inputs the multi-scale cross-modal semantic grounded feature map of the medical image into a decoder to obtain a segmentation mask. The decoder is a U-Net decoder. It should be understood that the U-Net decoder adopts an encoder-decoder symmetric structure and a skip connection mechanism, which can restore spatial resolution while integrating cross-modal semantics of different scales. It is particularly suitable for medical image scenes with complex organ boundaries and variable lesion details. Specifically, the spatial dimensions of the multi-scale cross-modal semantic grounded feature map of the medical image are gradually restored through the upsampling path of the U-Net. At the same time, the high-resolution visual features in the encoder are integrated with the cross-modal semantic features in the decoder using skip connections. This enables the segmentation mask to capture the subtle structure of the lesion, such as the burr signs in the high-resolution features, while also adhering to the anatomical constraints of the text semantics, such as the positional relationship between the tumor and adjacent blood vessels. Ultimately, a high-precision segmentation mask that meets clinical needs is generated, solving the problem of mapping multi-modal features to segmentation results, so that the segmentation mask can still maintain boundary accuracy and semantic consistency in complex scenes.

[0050] In particular, in one possible embodiment, step S5 is implemented as follows: First, the multi-scale cross-modal semantic grounded feature maps are sequentially input into the corresponding layers of the U-Net decoder in ascending order of resolution. The decoder's downsampling path has already been completed in the visual feature extraction stage. At this point, the input low-resolution cross-modal features first pass through an upsampling layer, such as deconvolution, to double their size. They are then channel-wise spliced ​​with the visual feature maps of the corresponding resolution in the encoder to fuse the cross-modal semantics with the original visual details. Next, the spliced ​​feature maps are semantically integrated through two sets of 3×3 convolutional layers to reduce the information noise introduced by upsampling. This upsampling-splicing-convolution operation is repeated, with each upsampling step being fused with the visual features of the corresponding encoder layer. Finally, at the highest resolution layer, a 1×1 convolutional layer is used to map the number of feature channels to the number of segmentation categories. The final pixel-level segmentation mask is then generated using a sigmoid activation function.

[0051] In the above-mentioned medical image segmentation method based on image-text interaction, in step S6, the segmentation mask is applied to the original medical image to obtain a medical image segmentation result. Specifically, by geometrically superimposing and visually rendering the segmentation mask and the original medical image, a medical image segmentation result with both semantic accuracy and visual readability is generated. The medical image segmentation result after the masking can not only retain the anatomical structure details of the original medical image, such as the grayscale density of CT and the signal intensity of MRI, but also highlight the target area in the form of a bright color or contour line through the visual superposition of semantic labels, significantly improving the doctor's efficiency and accuracy in identifying lesions.

[0052] In particular, in one possible embodiment, the implementation process of step S6 is as follows: first, the segmentation mask output by the decoder is post-processed, including binarization or multi-category label mapping, and the mask resolution is adjusted to be completely consistent with the original medical image through interpolation operations to ensure pixel-level spatial alignment. Then, the mask is fused with the original image using a spatial overlay algorithm. For grayscale images, the pixels of the target area in the mask are displayed in pseudo-color overlay, and the background area retains the original grayscale. By setting transparency parameters, such as 30% mask opacity, a balance is achieved between semantic prominence and retention of original medical image details. For color images, such as ultrasound pseudo-color images, the boundaries of the target area are outlined by contour drawing to avoid color conflicts that affect readability. Then, clinical annotation information is added to the image overlay results to form a standardized segmentation report image. Finally, the output format is selected according to the application scenario, such as DICOM format to retain medical metadata, JPEG format for report display, and the segmentation results are stored in the medical image management system.

[0053] In summary, a medical image segmentation method based on image-text interaction based on the embodiment of the present application is explained, which first acquires medical images and extracts multi-scale visual features, uses medical knowledge graphs to deeply analyze user text descriptions, and uses medical anatomical knowledge graphs for knowledge enhancement, thereby constructing an enhanced text vector that combines deep semantics and precise anatomical context. Furthermore, through multi-granular semantic grounding and collaborative fusion mechanisms, the enhanced text vectors and multi-scale visual features are progressively cross-modally aligned and information interacted, guiding the model to focus on and accurately locate the target area, and finally the decoder generates a high-precision segmentation mask. This method effectively combines the flexibility of natural language with the accuracy of medical prior knowledge, can effectively overcome the segmentation challenges in complex or ambiguous scenes, and significantly improves the accuracy and robustness of the segmentation task.

[0054] Figure 5 FIG is a block diagram of a medical image segmentation system based on image-text interaction according to an embodiment of the present application. Figure 5 As shown, according to an embodiment of the present application, a medical image segmentation system 100 based on image-text interaction includes: an original medical image acquisition module 110 for acquiring original medical images; a visual feature extraction module 120 for inputting the original medical image into a medical-specific visual encoder to obtain a multi-scale visual feature map of the medical image; a text feature enhancement module 130 for performing text feature encoding and knowledge enhancement on the acquired user text description to obtain an enhanced text feature vector containing semantic information and anatomical structure context; a multimodal fusion module 140 for performing multi-granularity semantic grounding and collaborative fusion on the multi-scale visual feature map of the medical image and the enhanced text feature vector containing semantic information and anatomical structure context to obtain a multi-scale cross-modal semantic grounding feature map of the medical image; a segmentation decoding module 150 for inputting the multi-scale cross-modal semantic grounding feature map of the medical image into a decoder to obtain a segmentation mask; and a segmentation result generation module 160 for applying the segmentation mask to the original medical image to obtain a medical image segmentation result.

[0055] As described above, the medical image segmentation system 100 based on image-text interaction according to the embodiment of the present application can be implemented in various wireless terminals, such as a server with an equipment asset operation and maintenance management algorithm based on intelligent networking. In one possible implementation, the medical image segmentation system 100 based on image-text interaction according to the embodiment of the present application can be integrated into the wireless terminal as a software module and / or hardware module. For example, the medical image segmentation system 100 based on image-text interaction can be a software module in the operating system of the wireless terminal, or it can be an application developed for the wireless terminal; of course, the medical image segmentation system 100 based on image-text interaction can also be one of the many hardware modules of the wireless terminal.

[0056] Alternatively, in another example, the medical image segmentation system 100 based on image-text interaction and the wireless terminal may also be separate devices, and the medical image segmentation system 100 based on image-text interaction may be connected to the wireless terminal via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0057] Here, those skilled in the art will appreciate that the specific operations of each step in the above-mentioned medical image segmentation system based on image-text interaction have been described in detail in the above reference. Figures 1 to 4 It has been introduced in detail in the description of the medical image segmentation method based on image-text interaction, and therefore, its repeated description will be omitted.

Claims

1. A medical image segmentation method based on image-text interaction, characterized in that: include: Obtaining an original medical image; inputting the original medical image into a medical-specific visual encoder to obtain a multi-scale visual feature map of the medical image; Performing text feature encoding and knowledge enhancement on the acquired user text description to obtain an enhanced text feature vector containing semantic information and anatomical structure context; performing multi-granularity semantic grounding and collaborative fusion on the multi-scale visual feature map of the medical image and the enhanced text feature vector containing semantic information and anatomical structure context to obtain a multi-scale cross-modal semantic grounding feature map of the medical image; inputting the multi-scale cross-modal semantic grounding feature map of the medical image into a decoder to obtain a segmentation mask; The segmentation mask is applied to the original medical image to obtain a medical image segmentation result.

2. The medical image segmentation method based on image-text interaction according to claim 1, characterized in that: Inputting the original medical image into a medical-specific visual encoder to obtain a multi-scale visual feature map of the medical image, including: inputting the original medical image into a medical-specific visual encoder based on a Unet encoder to obtain a medical image high-resolution feature map, a medical image medium-resolution feature map, and a medical image low-resolution feature map, wherein the medical image high-resolution feature map, the medical image medium-resolution feature map, and the medical image low-resolution feature map constitute the medical image multi-scale visual feature map.

3. The medical image segmentation method based on image-text interaction according to claim 1, characterized in that: The obtained user text description is subjected to text feature encoding and knowledge enhancement to obtain an enhanced text feature vector containing semantic information and anatomical structure context, including: inputting the user text description into a medical-specific language encoder based on the BioClinical BERT model to obtain an initial text embedding encoding vector; extracting medical anatomical entities from the user text description, and using the medical anatomical entities to query in a predefined medical anatomical structure knowledge graph to obtain relevant anatomical structure information; information encoding is performed on the relevant anatomical structure information to obtain an anatomical knowledge embedding encoding vector; and the initial text embedding encoding vector and the anatomical knowledge embedding encoding vector are input into a knowledge fusion module to obtain the enhanced text feature vector containing semantic information and anatomical structure context.

4. The medical image segmentation method based on image-text interaction according to claim 3, characterized in that: The initial text embedding coding vector and the anatomical knowledge embedding coding vector are input into a knowledge fusion module to obtain the enhanced text feature vector containing semantic information and anatomical structure context, including: the knowledge fusion module performs full-connection encoding on the initial text embedding coding vector and the anatomical knowledge embedding coding vector to obtain the enhanced text feature vector containing semantic information and anatomical structure context.

5. The medical image segmentation method based on image-text interaction according to claim 2, characterized in that: The multi-scale visual feature map of the medical image and the enhanced text feature vector containing semantic information and anatomical structure context are subjected to multi-granularity semantic grounding and collaborative fusion to obtain a multi-scale cross-modal semantic grounding feature map of the medical image, including: constructing a query vector based on the enhanced text feature vector containing semantic information and anatomical structure context and constructing a value vector and a key vector based on the low-resolution feature map of the medical image, and inputting the query vector, the value vector and the key vector into a coarse-grained grounding module to obtain a low-resolution cross-modal semantic grounding feature map of the medical image; constructing a query vector based on the low-resolution cross-modal semantic grounding feature map of the medical image and constructing a value vector and a key vector based on the medium-resolution feature map of the medical image, and inputting the query vector, the value vector and the key vector into a medium-granularity grounding module to obtain a medium-resolution cross-modal semantic grounding feature map of the medical image; constructing a query vector based on the medium-resolution cross-modal semantic grounding feature map of the medical image and constructing a value vector and a key vector based on the high-resolution feature map of the medical image, and inputting the query vector, the value vector and the key vector into a fine-granularity grounding module to obtain the multi-scale cross-modal semantic grounding feature map of the medical image.

6. The medical image segmentation method based on image-text interaction according to claim 5, characterized in that: A query vector is constructed based on the medium-resolution cross-modal semantic grounding feature map of the medical image, and a value vector and a key vector are constructed based on the high-resolution feature map of the medical image, and the query vector, the value vector and the key vector are input into a fine-grained grounding module to obtain the multi-scale cross-modal semantic grounding feature map of the medical image, including: a query vector is constructed based on the medium-resolution cross-modal semantic grounding feature map of the medical image, and a value vector and a key vector are constructed based on the high-resolution feature map of the medical image, and the query vector, the value vector and the key vector are input into a fine-grained grounding module to obtain an initial multi-scale cross-modal semantic grounding feature map of the medical image; based on the step-by-step boundary segmentation representation of the low-resolution cross-modal semantic grounding feature map of the medical image, the medium-resolution cross-modal semantic grounding feature map of the medical image and the initial multi-scale cross-modal semantic grounding feature map of the medical image, the initial multi-scale cross-modal semantic grounding feature map of the medical image is collaboratively fused and optimized to obtain the multi-scale cross-modal semantic grounding feature map of the medical image.

7. The medical image segmentation method based on image-text interaction according to claim 6, characterized in that: Based on the step-by-step boundary segmentation representation of the low-resolution cross-modal semantic grounding feature map of the medical image, the medium-resolution cross-modal semantic grounding feature map of the medical image and the initial multi-scale cross-modal semantic grounding feature map of the medical image, the initial multi-scale cross-modal semantic grounding feature map of the medical image is collaboratively fused and optimized to obtain the multi-scale cross-modal semantic grounding feature map of the medical image, including: calculating the boundary segmentation representation of the medium-resolution cross-modal semantic grounding feature map of the medical image relative to the low-resolution cross-modal semantic grounding feature map of the medical image to obtain a first boundary segmentation representation map; calculating the boundary segmentation representation of the initial multi-scale cross-modal semantic grounding feature map of the medical image The boundary segmentation representation of the semantic grounding feature map relative to the high-resolution cross-modal semantic grounding feature map of the medical image is obtained to obtain a second boundary segmentation representation map; based on the first boundary segmentation representation map and the second boundary segmentation representation map, a first-level overall cross-boundary segmentation association map and a second-level overall cross-boundary segmentation association map are constructed; based on the weighted sum of the first-level overall cross-boundary segmentation association map and the second-level overall cross-boundary segmentation association map as a cross-boundary segmentation correction map, the initial medical image multi-scale cross-modal semantic grounding feature map is point-multiplied and fused to obtain the medical image multi-scale cross-modal semantic grounding feature map.

8. The medical image segmentation method based on image-text interaction according to claim 7, characterized in that: Constructing a query vector based on the enhanced text feature vector containing semantic information and anatomical structure context, and constructing a value vector and a key vector based on the low-resolution feature map of the medical image, and inputting the query vector, the value vector, and the key vector into a coarse-grained grounding module to obtain a low-resolution cross-modal semantic grounding feature map of the medical image, including: the coarse-grained grounding module processes the query vector, the value vector, and the key vector according to the following formula to obtain the low-resolution cross-modal semantic grounding feature map of the medical image, wherein the formula is: ;in, Cross-modal semantically grounded feature vectors, is the query vector, is the key vector, is a value vector, is the transpose of the vector, is the scaling factor, is the characteristic reshape function, Low-resolution cross-modal semantically grounded feature maps for medical images.

9. The medical image segmentation method based on image-text interaction according to claim 1, characterized in that: The decoder is a U-Net decoder.

10. A medical image segmentation system based on image-text interaction, characterized in that: include: An original medical image acquisition module, used for acquiring original medical images; a visual feature extraction module, configured to input the original medical image into a medical-specific visual encoder to obtain a multi-scale visual feature map of the medical image; A text feature enhancement module is used to perform text feature encoding and knowledge enhancement on the acquired user text description to obtain an enhanced text feature vector containing semantic information and anatomical structure context; a multimodal fusion module is used to perform multi-granularity semantic grounding and collaborative fusion on the multi-scale visual feature map of the medical image and the enhanced text feature vector containing semantic information and anatomical structure context to obtain a multi-scale cross-modal semantic grounding feature map of the medical image; a segmentation decoding module, configured to input the multi-scale cross-modal semantic grounded feature map of the medical image into a decoder to obtain a segmentation mask; The segmentation result generating module is used to apply the segmentation mask to the original medical image to obtain a medical image segmentation result.

Images