Multimodal brain tumor segmentation method based on medical knowledge graph guidance enhancement

Abstract

The invention discloses a multi-modal brain tumor segmentation method based on medical knowledge map guide enhancement, and relates to the field of medical image segmentation. According to the technical scheme, an anatomical map is used as priori knowledge to carry out affinity matching calculation with potential target features, and importance to a target is weighed; and obtaining potential target features by context updating. Through an interactive method of multi-scale feature fusion, potential target features and extracted features are subjected to information sharing, and the most important scale is selected as a query set and introduced into a network structure to pay attention to an important local area. Brain tumor text knowledge is injected into an anatomical map to jointly train a text-visual encoder, multi-modal medical knowledge is aligned in space, and a knowledge map is formed to guide accurate segmentation of a brain tumor. And finally, the training process is supervised by discriminating a loss function according to the similarity of the extracted features and the multi-scale features. And the multi-scale feature similarity discrimination loss function supervises the training process at the pixel level through the similarity discrimination of the basic features and the multi-scale features. Through the technology, the brain tumor segmentation precision and credibility are improved, and diagnosis and treatment application in a real clinical scene is realized.

Description

Technical Field

[0001] This invention relates to the field of medical image segmentation, and in particular to a multimodal brain tumor segmentation method based on medical knowledge graph-guided enhancement. Background Technology

[0002] Brain tumor segmentation is a crucial research area in medical image processing, aiming to accurately segment lesions or abnormal regions in brain tumor images and quantify their location, size, and shape for accurate diagnosis and treatment. In brain tumor segmentation, compared to natural images, brain tumor images exhibit lower contrast, uncertain location and morphology, weak boundary information, and scattered and highly irregular targets, leading to significant variations in the morphology of the segmented targets. Incorrect or unstable segmentation directly impacts subsequent diagnosis and treatment, thus negating the original purpose of medical image segmentation. Furthermore, current methods tend to introduce more complex modules to improve segmentation performance, neglecting the relatively small differences between brain tumor MR image data samples, where significant differences exist only within the samples. They fail to fully utilize medical textual knowledge and anatomical atlases related to brain tumors. Therefore, it is necessary to combine medical textual knowledge and anatomical atlases to form an atlas segmentation concept, which is then injected into a neural network and integrated with brain tumor MR images for multimodal medical knowledge alignment. This guides the network to focus on potential lesion regions, achieving accurate brain tumor segmentation. Summary of the Invention

[0003] To address the aforementioned technical problems, this invention provides a multimodal brain tumor segmentation method guided by a medical knowledge graph. The visual concepts of anatomical atlases are used to generate representations of anatomical targets on an image via a visual encoder. These representations are then combined to form atlas segmentation concept pairs describing anatomical structures or lesions on the image. Multimodal medical knowledge is spatially aligned to construct a knowledge graph that guides accurate brain tumor segmentation. A multi-scale feature similarity discriminant loss function supervises the training process at the pixel level through similarity discrimination of basic features and multi-scale features.

[0004] The technical solution adopted by this invention to solve its technical problem is: a multimodal brain tumor segmentation method based on medical knowledge graph-guided enhancement, comprising the following steps:

[0005] S1, acquire brain tumor MR images and perform data preprocessing;

[0006] S2 uses brain tumor anatomical atlases as prior knowledge to perform affinity matching calculations and weigh the importance of potential targets to the features of potential targets and the associated information in the atlases; and obtains potential target features by performing convolution operations in the graph domain to update the context, guiding the network to focus on the potential target features of lesions.

[0007] S3. To achieve fine-grained and accurate segmentation of brain tumors, an interactive method for multi-scale feature fusion is designed. This method can simultaneously select important features at multiple scales and select the most important scale query set as the final scale information to be injected into the network structure, guiding the network to pay more attention to important local regions.

[0008] S4. Inject brain tumor textual knowledge into the brain tumor anatomical atlas, and train the text encoder using textual anatomical knowledge and atlas segmentation of specific anatomical structures; align the textual descriptions with the visual features of the anatomical atlas in the latent space to form a brain tumor knowledge graph.

[0009] S5, network training, is designed for brain tumor segmentation methods that use knowledge graph-guided enhancement to achieve multimodal medical knowledge alignment. It consists of two parts: the loss of the semantic segmentation module and the similarity discrimination loss function between basic tokens and multi-scale tokens. The training process is supervised at the pixel level.

[0010] Preferably, step S1 is as follows:

[0011] S1.1, Acquisition of MR images of brain tumors;

[0012] The patient's head was scanned in different directions and planes using an MRI machine to obtain images from different angles and positions; four different modal MR images were acquired, including T1-weighted, T2-weighted, contrast-enhanced T1-weighted, and liquid-attenuated inversion recovery pulsed images. The different modal MR images focused on sub-regions of different pathological information.

[0013] S1.2, Brain Tumor MRI Annotation: The acquired brain tumor MR images are annotated, marking the location, extent, shape, and size of tumor edema, enhanced tumor, necrosis, and non-enhanced tumor; finally, the annotated Ground Truth is paired with the original MR images.

[0014] S1.3, Data preprocessing: The above data is processed by image reconstruction, grayscale normalization, image registration and data augmentation to a specified size; finally, it is divided into training set, validation set and test set, which are used as input images for network training, tuning and evaluation.

[0015] Preferably, step S2 is as follows:

[0016] S2.1, based on prior knowledge relationships provided by the morphology, location, size, and grayscale contrast information of lesions in the brain tumor anatomical segmentation atlas; construct the atlas relationship R =<D,K> , where D represents the location of the brain tumor lesion and K represents the prior knowledge relationship in the anatomical atlas;

[0017] S2.2, Based on the features of potential targets, retrieve the associated information of the targets, so that the associated information is expressed in the feature space; construct the associated feature map S =<P,E> Where P represents a feature node, i.e., {p i ,...,p N}=P∈R N×F p, E represents the association information corresponding to the node;

[0018] S2.3, use the value of the association information in the feature node P and the prior knowledge K in the brain tumor anatomical atlas to calculate the feature relative affinity matching degree W;

[0019] S2.4 After obtaining the value of W, perform a context update operation to ensure that the node features in the graph are aggregated.

[0020] Preferably, step S2.4 is as follows:

[0021] Input target feature X R and spectral features X A Mapping from the spatial domain X to the graph domain to generate semantically aware graph features G R and G A ;

[0022]

[0023]

[0024] Where v(·) represents the convolution operation for graph projection, and w(·) represents the convolution operation for feature dimensionality reduction. v and W w Represent the learnable kernels of v(·) and w(·), respectively; symbol Represents matrix multiplication;

[0025] After projection, G R and G A By summing and performing graph convolution operations, the relationships between relevant node features in the semantic graph and the edge graph are learned, enabling inference on the fully connected graph:

[0026] G = G R +G A

[0027] It is implemented by two 1D convolutions in the channel direction and the node direction; the output can be represented as:

[0028]

[0029] Where I∈R N×N Let A represent the identity matrix. g ∈R N×N Let W represent the adjacency matrix.g Indicates updating parameters; A g and W g All of them are randomly initialized during training and optimized using gradient descent;

[0030] Finally, the mapping will... Remapping back to the original spatial domain Key local features that receive special attention:

[0031]

[0032] Preferably, step S3 specifically includes the following:

[0033] a) To improve the multi-scale performance of interactive segmentation, the original input image and the image of the region of interest (ROI) updated with prior knowledge and graph domain context are subjected to adaptive patch embedding to obtain tokens of different scales. The original input image is divided into basic tokens with a patch size of 16×16, denoted as f. b The region of interest image is then divided into multi-scale tokens with patch sizes of 8×8 and 28×28, denoted as f. t and f l ;

[0034] b) A similarity-based differentiable top-k token selection algorithm. This algorithm selects only important regions of interest from multi-scale tokens to update the base token, enabling information sharing among tokens of different sizes; it calculates f of the multi-scale tokens. t f l with f b The importance of a token relative to its base token is measured by the similarity between the two tokens.

[0035] s = sigmoid(cos(f) b ,f t ))+sigmoid(cos(f b ,f l ))

[0036] After calculating the similarity score between the multi-scale token and the basic token, a differentiable top-k selection method is proposed, which selects the top k tokens with the highest scores:

[0037] s k idx k =torch.topk(s)

[0038] c) Select the highest-scoring important token as the fine-tuning interaction information. bBased on this, cross-attention is used to fuse multi-scale information, and the basic f b f with important local regional features t and f l Scaling cross-attention fusion enables rapid updates of multi-scale features.

[0039] Preferably, step S4 specifically includes the following;

[0040] a) Use a pre-trained text encoder to generate neural embeddings as text cues for brain tumor anatomy target terms for segmentation:

[0041] z i =Φ text (t i ),z i ∈R d

[0042] Φ text It is a brain tumor medical knowledge text encoder, t i It contains all medical terminology related to brain tumors, where d is the feature dimension; after pre-training the text encoder using domain knowledge injection, z i It should include both textual background information and visual information from the atlas samples;

[0043] b) For the visual concept of anatomical atlases, a visual encoder Φ is used. visual For the extracted image embedding, the aggregated features are considered as a representation of the anatomical target in the image:

[0044]

[0045] x i It is a modal scan image of the anatomical region, V i H represents the multi-scale feature maps of each layer of the visual encoder. s W s D s The spatial resolution of each layer;

[0046] c) Combining text encoders and visual encoders to align textual anatomical knowledge with anatomical atlases for multimodal medical knowledge:

[0047] Φ Seg =(Φ visual (x i ),Φ text (t i )),

[0048] Forming text-visual medical concept pairs, that is, pairs D = {(x1, y1; t1), ..., (x K ,yK ;t K )}, where x i ∈R H×W×D×C Input the original image of the brain tumor, y i ∈R H×W×D×M This is a binary segmentation annotation of the anatomical target in the image, t i ={t1,t2,...,t M} represents the corresponding medical terminology set;

[0049] d) Finally, insert a Transformer-based query module to further enhance the text cues with visual cues; embedding the text cues as the query and the encoder's high-level latent visual embeddings as the key and value, represented as:

[0050] q i =Φ query (V i ,z i ),z i ∈R d

[0051] q i This can be viewed as the anatomical target in a specific image scan x i An adaptive representation; enabling the model to selectively extract brain tumor-specific features for accurate segmentation of the queried brain tumor target.

[0052] Preferably, step S5 specifically includes the following:

[0053] a)L seg It is the loss of the semantic segmentation module used to learn semantic features; BCEDiceLoss is a combination of binary cross-entropy (BCE) loss and Dice loss, used to define L seg for:

[0054]

[0055] Where y represents GT, Indicates the prediction result;

[0056] b) In the proposed token selection algorithm, the similarity between the base token and the multi-scale token is the key to selecting important tokens. A loss function is proposed to distinguish the similarity between the two:

[0057]

[0058] Where N represents the number of elements involved in the calculation; d p This indicates a positive match, d nThis indicates a negative match; when d p >d n When d, the loss function imposes a penalty; when d p <<d n At that time, the loss function applies a penalty L. c →0;

[0059] c) During the model training phase, this L c Loss and image segmentation loss L seg Combined training:

[0060] L = L seg +L c .

[0061] Advantages of this invention:

[0062] This patent proposes a multimodal brain tumor segmentation method based on medical knowledge graph-guided enhancement.

[0063] First, the acquired brain tumor MR images are professionally annotated and then preprocessed before being input into the network. Next, lesion location information is extracted from anatomical atlases as prior knowledge to construct atlas relationships. Target association information is retrieved from the MR images to construct an association feature map. Affinity matching degree is calculated, and context is updated to obtain potential target features. The original input image and potential targets are divided into different scales, and their similarity is calculated based on a similarity-differentiable selection algorithm. Cross-attention is used to fuse multi-scale information to achieve feature information sharing and updating, guiding the network to focus on important local regions. Anatomical target terminology for brain tumors is used to generate segmentation text prompts via a text encoder, and visual concepts from anatomical atlases are used to generate representations of anatomical targets on the image via a visual encoder. These two are combined to form a knowledge graph, aligning multimodal medical knowledge and guiding accurate brain tumor segmentation. A multi-scale feature similarity discriminant loss function supervises the training process at the pixel level through similarity discrimination of basic features and multi-scale features. A multimodal brain tumor segmentation method guided by medical knowledge graphs can integrate multimodal medical knowledge, including textual knowledge and anatomical atlases, into MR images, thereby improving the accuracy and reliability of brain tumor segmentation and enabling its application in real-world clinical scenarios. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only four of the drawings in this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1Flowchart of a multimodal brain tumor segmentation method guided by medical knowledge graph;

[0066] Figure 2 Latent target feature extraction based on prior knowledge of anatomical atlases;

[0067] Figure 3 An interactive method for multi-scale feature fusion;

[0068] Figure 4 A knowledge graph that aligns textual knowledge with multimodal knowledge of anatomical atlases. Detailed Implementation

[0069] To enhance understanding of the present invention, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. These embodiments are only used to explain the invention and do not limit the scope of protection of the invention.

[0070] Example

[0071] like Figure 1As shown, this patent proposes a brain tumor segmentation method based on knowledge graph-guided enhancement to achieve multimodal medical knowledge alignment. It mainly includes five modules: brain tumor MR image preprocessing, latent target feature extraction based on prior knowledge from anatomical atlases, an interactive method for multi-scale feature fusion, multimodal medical knowledge alignment guidance between textual knowledge and anatomical atlases, and a multi-scale feature similarity discrimination loss function. The acquired brain tumor MR images are professionally annotated and then processed to a specified size through image reconstruction, grayscale normalization, image registration, and data augmentation before being input into the network. The latent target features based on prior knowledge from anatomical atlases first extract information such as the morphology, location, size, and grayscale contrast of the lesion site from the segmentation atlas to construct atlas relationships. This information is then used as prior knowledge to retrieve target association information from the MR images input into the network, constructing an association feature map, calculating affinity matching degree, weighing the importance to the target, and finally performing a context update operation in the graph domain to obtain latent target features. An interactive method for multi-scale feature fusion divides the original input image and potential lesion targets into different scales. Based on a similarity-differentiable selection algorithm, it calculates the similarity between multi-scale features, selects the highest-scoring labeled feature, and utilizes cross-attention to fuse multi-scale information to achieve feature information sharing and rapid updates, guiding the network to focus more on important local regions. Multimodal medical knowledge alignment guidance using textual knowledge and anatomical atlases involves generating segmentation text prompts from brain tumor anatomical target terms using a text encoder, and generating representations of anatomical targets on the image from visual concepts of anatomical atlases using a visual encoder. These two are combined into atlas segmentation concept pairs describing anatomical structures or lesions on the image, aligning multimodal medical knowledge in space to construct a knowledge graph to guide accurate brain tumor segmentation. A multi-scale feature similarity discriminant loss function supervises the training process at the pixel level through similarity discrimination of basic features and multi-scale features.

[0072] The multimodal brain tumor segmentation method based on medical knowledge graph-guided enhancement includes the following steps:

[0073] S1, acquire brain tumor MR images and perform data preprocessing;

[0074] Step S1 is as follows:

[0075] S1.1, Acquisition of MR images of brain tumors;

[0076] MR image acquisition for brain tumors. The patient's head is scanned in different directions and planes using an MRI scanner to obtain images from different angles and locations; four different modalities of MR images are acquired, including T1-weighted (T1), T2-weighted (T2), contrast-enhanced T1-weighted (T1ce), and fluid-attenuated inversion recovery pulse (FLAIR), etc. Different modalities of MR images emphasize different sub-regions of pathological information.

[0077] S1.2, Brain Tumor MRI Annotation: Personnel with professional medical knowledge annotate the acquired brain tumor MR images, marking the location, severity, shape, and size of tumor edema (ED), enhanced tumor (ET), necrosis, and non-enhanced tumor (NCR / NET). Finally, the annotated Ground Truth is paired with the original MR images and saved in the corresponding data format.

[0078] S1.3, Data preprocessing: The above data is processed by image reconstruction, grayscale normalization, image registration and data augmentation to a specified size; finally, it is divided into training set, validation set and test set, which are used as input images for network training, tuning and evaluation.

[0079] S2 uses brain tumor anatomical atlases as prior knowledge to perform affinity matching calculations and weigh the importance of potential targets to the features of potential targets and the associated information in the atlases; and obtains potential target features by performing convolution operations in the graph domain to update the context, guiding the network to focus on the potential target features of lesions.

[0080] Step S2 is as follows:

[0081] S2.1, based on prior knowledge relationships provided by the morphology, location, size, and grayscale contrast information of lesions in the brain tumor anatomical segmentation atlas; construct the atlas relationship R =<D,K> , where D represents the location of the brain tumor lesion and K represents the prior knowledge relationship in the anatomical atlas;

[0082] S2.2, Based on the features of potential targets, retrieve the associated information of the targets, so that the associated information is expressed in the feature space; construct the associated feature map S =<P,E> Where P represents a feature node, i.e., {p i ,…,p N}=P∈R N×F p, E represents the association information corresponding to the node;

[0083] S2.3 calculates the relative affinity matching degree W of features using the values ​​of the association information in feature node P and the prior knowledge K in the brain tumor anatomical atlas; weights are applied based on the matching degree between two adjacent nodes, and the edge values ​​in R are aggregated. Features in the node neighborhood are stacked, and a linear transformation is performed using a shared weight matrix. Matrix E weighs the importance of the proposed object to other proposed objects in the scene graph based on the prior knowledge of the relationships between object combinations appearing in the prior knowledge graph, and performs feature enhancement.

[0084] S2.4, after obtaining the value of W, perform a context update operation; ensure that the node features in the graph are aggregated. To better achieve semantic and feature integration, step S2.4 is as follows:

[0085] Input target feature X R and spectral features X A Mapping from the spatial domain X to the graph domain to generate semantically aware graph features G R and G A ;

[0086]

[0087]

[0088] Where v(·) represents the convolution operation for graph projection, and w(·) represents the convolution operation for feature dimensionality reduction. v and W w Represent the learnable kernels of v(·) and w(·), respectively; symbol Represents matrix multiplication;

[0089] After projection, G R and G A By summing and performing graph convolution operations, the relationships between relevant node features in the semantic graph and the edge graph are learned, enabling inference on the fully connected graph:

[0090] G = G R +G A

[0091] It is implemented by two 1D convolutions in the channel direction and the node direction; the output can be represented as:

[0092]

[0093] Where I∈R N×N Let A represent the identity matrix. g ∈R N×N Let W represent the adjacency matrix. g Indicates updating parameters; A g and W g All of them are randomly initialized during training and optimized using gradient descent;

[0094] Finally, the mapping will... Remapping back to the original spatial domain Key local features that receive special attention:

[0095]

[0096] The above operations provide each node with more context about the entire brain tumor and its prior knowledge-based relationships with other nodes in this context.

[0097] S3. To achieve fine-grained and accurate segmentation of brain tumors, an interactive method for multi-scale feature fusion is designed. This method can simultaneously select important features at multiple scales and select the most important scale query set as the final scale information to be injected into the network structure, guiding the network to pay more attention to important local regions.

[0098] Step S3 specifically includes the following:

[0099] a) To improve the multi-scale performance of interactive segmentation, the original input image and the image of the region of interest (ROI) updated with prior knowledge and graph domain context are subjected to adaptive patch embedding to obtain tokens of different scales. The original input image is divided into basic tokens with a patch size of 16×16, denoted as f. b The region of interest image is then divided into multi-scale tokens with patch sizes of 8×8 and 28×28, denoted as f. t and f l ;

[0100] b) A similarity-based differentiable top-k token selection algorithm. This algorithm selects only important regions of interest from multi-scale tokens to update the base token, enabling information sharing among tokens of different sizes; it calculates f of the multi-scale tokens. t f l with f b The importance of a token relative to its base token is measured by the similarity between the two tokens.

[0101] s = sigmoid(cos(f) b ,f t ))+sigmoid(cos(f b ,f l ))

[0102] After calculating the similarity score between the multi-scale token and the basic token, a differentiable top-k selection method is proposed, which selects the top k tokens with the highest scores:

[0103] s k idx k =torch.topk(s)

[0104] c) Select the highest-scoring important token as the fine-tuning interaction information. b Based on this, cross-attention is used to fuse multi-scale information, and the basic f bf with important local regional features t and f l Scaling cross-attention fusion enables rapid updates of multi-scale features.

[0105] S4. Brain tumor textual knowledge is injected into the brain tumor anatomical atlas. The text encoder is trained using textual anatomical knowledge and atlas segmentation of specific anatomical structures. Multimodal medical knowledge is aligned in the latent space with the visual features of the anatomical atlas in textual form to form a brain tumor knowledge graph. The knowledge graph helps to use text embeddings as queries in a Transformer-based architecture, enabling the model to selectively extract brain tumor-specific features for accurate segmentation of the queried brain tumor target.

[0106] Step S4 specifically includes the following;

[0107] a) Use a pre-trained text encoder to generate neural embeddings as text cues for brain tumor anatomy target terms for segmentation:

[0108] z i =Φ text (t i ),z i ∈R d

[0109] Φ text It is a brain tumor medical knowledge text encoder, t i It contains all medical terminology related to brain tumors, where d is the feature dimension; after pre-training the text encoder using domain knowledge injection, z i It should include both textual background information and visual information from the atlas samples;

[0110] b) For the visual concept of anatomical atlases, a visual encoder Φ is used. visual For the extracted image embedding, the aggregated features are considered as a representation of the anatomical target in the image:

[0111]

[0112] x i It is a modal scan image of the anatomical region, V i H represents the multi-scale feature maps of each layer of the visual encoder. s W s D s The spatial resolution of each layer;

[0113] c) Combining text encoders and visual encoders to align textual anatomical knowledge with anatomical atlases for multimodal medical knowledge:

[0114] Φ Seg =(Φvisual (x i ),Φ text (t i )),

[0115] Forming text-visual medical concept pairs, that is, pairs D = {(x1, y1; t1), ..., (x K ,y K ;t K )}, where x i ∈R H×W×D×C Input the original image of the brain tumor, y i ∈R H×W×D×M This is a binary segmentation annotation of the anatomical target in the image, t i ={t1,t2,...,t M} represents the corresponding medical terminology set;

[0116] d) Finally, insert a Transformer-based query module to further enhance the text cues with visual cues; embedding the text cues as the query and the encoder's high-level latent visual embeddings as the key and value, represented as:

[0117] q i =Φ query (V i ,z i ),z i ∈R d

[0118] q i This can be viewed as the anatomical target in a specific image scan x i An adaptive representation; enabling the model to selectively extract brain tumor-specific features for accurate segmentation of the queried brain tumor target.

[0119] S5, network training, is designed for brain tumor segmentation methods that use knowledge graph-guided enhancement to achieve multimodal medical knowledge alignment. It consists of two parts: the loss of the semantic segmentation module and the similarity discrimination loss function between basic tokens and multi-scale tokens. The training process is supervised at the pixel level.

[0120] Step S5 specifically includes the following:

[0121] a)L seg It is the loss of the semantic segmentation module used to learn semantic features; BCEDiceLoss is a combination of binary cross-entropy (BCE) loss and Dice loss, used to define L seg for:

[0122]

[0123] Where y represents GT, Indicates the prediction result;

[0124] b) In the proposed token selection algorithm, the similarity between the base token and the multi-scale token is the key to selecting important tokens. A loss function is proposed to distinguish the similarity between the two:

[0125]

[0126] Where N represents the number of elements involved in the calculation; d p This indicates a positive match, d n This indicates a negative match; when d p >d n When d, the loss function imposes a penalty; when d p <<d n At that time, the loss function applies a penalty L. c →0;

[0127] c) During the model training phase, this L c Loss and image segmentation loss L seg Combined training:

[0128] L = L seg +L c .

[0129] Advantages of this invention:

[0130] This patent proposes a multimodal brain tumor segmentation method based on medical knowledge graph-guided enhancement.

[0131] First, the acquired brain tumor MR images are professionally annotated and then preprocessed before being input into the network. Next, lesion location information is extracted from anatomical atlases as prior knowledge to construct atlas relationships. Target association information is retrieved from the MR images to construct an association feature map. Affinity matching degree is calculated, and context is updated to obtain potential target features. The original input image and potential targets are divided into different scales, and their similarity is calculated based on a similarity-differentiable selection algorithm. Cross-attention is used to fuse multi-scale information to achieve feature information sharing and updating, guiding the network to focus on important local regions. Anatomical target terminology for brain tumors is used to generate segmentation text prompts via a text encoder, and visual concepts from anatomical atlases are used to generate representations of anatomical targets on the image via a visual encoder. These two are combined to form a knowledge graph, aligning multimodal medical knowledge and guiding accurate brain tumor segmentation. A multi-scale feature similarity discriminant loss function supervises the training process at the pixel level through similarity discrimination of basic features and multi-scale features. A multimodal brain tumor segmentation method guided by medical knowledge graphs can integrate multimodal medical knowledge, including textual knowledge and anatomical atlases, into MR images, thereby improving the accuracy and reliability of brain tumor segmentation and enabling its application in real-world clinical scenarios.

[0132] The specific implementation details of each part of this invention are as follows:

[0133] 1. Obtain MR images of the patient's brain tumor through MRI scans. After professional annotation, perform image reconstruction, grayscale standardization, image registration, and data enhancement, and process the images to a specified size for use as network input images.

[0134] 2. Using brain tumor anatomical atlases, information such as morphology, location, size, and grayscale contrast of lesions is extracted to construct atlas relationships. Then, the association information of the target is retrieved from the MR images in the input network to construct an association feature map and calculate the affinity matching degree between the two. In this way, context update operations are performed in the graph domain to obtain potential target features.

[0135] 3. The original input image is divided into basic tokens with a patch size of 16×16, and potential lesion targets are divided into multi-scale tokens with patch sizes of 8×8 and 28×28. Based on a top-k similarity token selection algorithm, the similarity between multi-scale features is calculated, the highest-scoring labeled feature is selected, and cross-attention is used to fuse multi-scale information to achieve feature information sharing and rapid updating.

[0136] 4. The anatomical target terms of brain tumors are generated into segmentation text prompts through a text encoder. The visual concepts of the anatomical atlas are generated into representations of the anatomical targets on the image through a visual encoder. The two are combined into atlas segmentation concept pairs of anatomical structures or lesion descriptions on the image. Multimodal medical knowledge is aligned in space to form a knowledge graph to guide the accurate segmentation of brain tumors.

[0137] 5. Finally, a multi-scale feature similarity discriminant loss function L is designed. c The training process is supervised at the pixel level by using similarity judgments of basic features and multi-scale features.

[0138]

[0139] Where N represents the number of elements involved in the calculation. p This indicates a positive match, d n This indicates a negative match. When d p >d n When d, the loss function imposes a penalty; when d p <<d n At that time, the loss function applies a penalty L. c →0.

[0140] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0141] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A multimodal brain tumor segmentation method based on medical knowledge graph-guided enhancement, characterized in that, Includes the following steps: S1, acquire MR images of brain tumors and perform data preprocessing; S2 uses brain tumor anatomical atlases as prior knowledge to perform affinity matching calculations and weigh the importance of potential targets to the features of potential targets and the associated information in the atlases; and obtains potential target features by performing convolution operations in the graph domain to update the context, guiding the network to focus on the potential target features of lesions. S3. To achieve fine-grained and accurate segmentation of brain tumors, an interactive method for multi-scale feature fusion is designed. This method can simultaneously select important features at multiple scales and select the most important scale query set as the final scale information to be injected into the network structure, guiding the network to pay more attention to important local regions. S4. Inject brain tumor textual knowledge into the brain tumor anatomical atlas, and train the text encoder using textual anatomical knowledge and atlas segmentation of specific anatomical structures; align the textual descriptions with the visual features of the anatomical atlas in the latent space to form a brain tumor knowledge graph. S5, network training, is designed for brain tumor segmentation methods that use knowledge graph-guided enhancement to achieve multimodal medical knowledge alignment. It consists of two parts: the loss of the semantic segmentation module and the similarity discrimination loss function between basic tokens and multi-scale tokens. The training process is supervised at the pixel level.

2. The multimodal brain tumor segmentation method based on medical knowledge graph-guided enhancement according to claim 1, characterized in that: Step S1 is as follows: S1.1, Acquisition of MR images of brain tumors; The patient's head was scanned in different directions and planes using an MRI machine to obtain images from different angles and positions; four different modal MR images were acquired, including T1-weighted, T2-weighted, contrast-enhanced T1-weighted, and liquid-attenuated inversion recovery pulsed images. The different modal MR images focused on sub-regions of different pathological information. S1.2, Brain Tumor MRI Annotation: The acquired brain tumor MR images are annotated, marking the location, extent, shape, and size of tumor edema, enhanced tumor, necrosis, and non-enhanced tumor; finally, the annotated Ground Truth is paired with the original MR images; S1.3, Data preprocessing: The above data is processed by image reconstruction, grayscale normalization, image registration and data augmentation to a specified size; finally, it is divided into training set, validation set and test set, which are used as input images for network training, tuning and evaluation.

3. The multimodal brain tumor segmentation method based on medical knowledge graph-guided enhancement according to claim 1, characterized in that: Step S2 is as follows: S2.1, based on prior knowledge relationships provided by the morphology, location, size, and grayscale contrast information of lesions in the brain tumor anatomical segmentation atlas; construct the atlas relationship R =<D,K> , where D represents the location of the brain tumor lesion and K represents the prior knowledge relationship in the anatomical atlas; S2.2, Based on the features of potential targets, retrieve the associated information of the targets, so that the associated information is expressed in the feature space; construct the associated feature map S =<P,E> Where P represents a feature node, i.e., {p i ,...,p N }=P∈R N×F E represents the associated information corresponding to the node; S2.3, use the value of the association information in the feature node P and the prior knowledge K in the brain tumor anatomical atlas to calculate the feature relative affinity matching degree W; S2.4 After obtaining the value of W, perform a context update operation; Ensure that the node features in the graph are aggregated.

4. The multimodal brain tumor segmentation method based on medical knowledge graph-guided enhancement according to claim 1, characterized in that: Step S2.4 is as follows: Input target feature X R and spectral features X A Mapping from the spatial domain X to the graph domain to generate semantically aware graph features G R and G A ; Where v(·) represents the convolution operation for graph projection, and w(·) represents the convolution operation for feature dimensionality reduction. v and W w Represent the learnable kernels of v(·) and w(·), respectively; symbol Represents matrix multiplication; After projection, G R and G A By summing and performing graph convolution operations, the relationships between relevant node features in the semantic graph and the edge graph are learned, enabling inference on the fully connected graph: G=G R +G A It is implemented by two 1D convolutions in the channel direction and the node direction; the output can be represented as: Where I∈R N×N Let A represent the identity matrix. g ∈R N×N Let W represent the adjacency matrix. g Indicates updating parameters; A g and W g All of them are randomly initialized during training and optimized using gradient descent; Finally, the mapping will... Remapping back to the original spatial domain Key local features that receive special attention:

5. The multimodal brain tumor segmentation method based on medical knowledge graph-guided enhancement according to claim 1, characterized in that: Step S3 specifically includes the following: a) To improve the multi-scale performance of interactive segmentation, the original input image and the image of the region of interest (ROI) updated with prior knowledge and graph domain context are subjected to adaptive patch embedding to obtain tokens of different scales. The original input image is divided into basic tokens with a patch size of 16×16, denoted as f. b The region of interest image is then divided into multi-scale tokens with patch sizes of 8×8 and 28×28, denoted as f. t and f l ; b) A similarity-based differentiable top-k token selection algorithm. This algorithm selects only important regions of interest from multi-scale tokens to update the base token, enabling information sharing among tokens of different sizes; it calculates f of the multi-scale tokens. t f l with f b The similarity between them is used to measure their importance relative to the base token: s=sigmoid(cos(f b ,f t ))+sigmoid(cos(f b ,f l )) After calculating the similarity scores between the multi-scale token and the basic token, a differentiable top-k selection method is proposed, which selects the top k tokens with the highest scores: s k ,idx k =torch.topk(s) c) Select the highest-scoring important token as the fine-tuning interaction information. b Based on this, cross-attention is used to fuse multi-scale information, and the basic f b f with important local regional features t and f l Scaling cross-attention fusion enables rapid updates of multi-scale features.

6. The multimodal brain tumor segmentation method based on medical knowledge graph-guided enhancement according to claim 1, characterized in that: Step S4 specifically includes the following; a) Use a pre-trained text encoder to generate neural embeddings as text cues for brain tumor anatomy target terms for segmentation: z i =Φ text (t i ),z i ∈R d Φ text It is a brain tumor medical knowledge text encoder, t i It contains all the textual medical terms related to brain tumors, where d is the feature dimension; After pre-training the text encoder using domain knowledge injection, z i It should include both textual background information and visual information from the atlas samples; b) For the visual concept of anatomical atlases, a visual encoder Φ is used. visual For the extracted image embedding, the aggregated features are considered as a representation of the anatomical target in the image: x i It is a modal scan image of the anatomical region, V i H represents the multi-scale feature maps of each layer of the visual encoder. s W s D s The spatial resolution of each layer; c) Combining text encoders and visual encoders to align textual anatomical knowledge with anatomical atlases for multimodal medical knowledge: F Seg =(Φ visual (x i ),F text (t i )), Forming text-visual medical concept pairs, that is, pairs D = {(x1, y1; t1), ..., (x K ,y K ;t K )}, where x i ∈R H×W×D×C Input the original image of the brain tumor, y i ∈R H×W×D×M This is a binary segmentation annotation of the anatomical target in the image, t i ={t1,t2,...,t M } represents the corresponding medical terminology set; d) Finally, insert a Transformer-based query module to further enhance the text cues with visual cues; The text prompt embedding is used as the query, and the encoder's high-level latent visual embedding is used as the key and value, represented as follows: q i =Φ query (V i ,z i ),z i ∈R d q i This can be viewed as the anatomical target in a specific image scan x i An adaptive representation; enabling the model to selectively extract brain tumor-specific features for accurate segmentation of the queried brain tumor target.

7. The multimodal brain tumor segmentation method based on medical knowledge graph-guided enhancement according to claim 1, characterized in that: Step S5 specifically includes the following: a)L seg It is the loss of the semantic segmentation module used to learn semantic features; BCEDiceLoss is a combination of binary cross-entropy (BCE) loss and Dice loss, used to define L seg for: Where y represents GT, Indicates the prediction result; b) In the proposed token selection algorithm, the similarity between the base token and the multi-scale token is the key to selecting important tokens. A loss function is proposed to distinguish the similarity between the two: Where N represents the number of elements involved in the calculation; d p This indicates a positive match, d n This indicates a negative match; when d p >d n When d, the loss function imposes a penalty; when d p <<d n At that time, the loss function applies a penalty L. c →0; c) During the model training phase, this L c Loss and image segmentation loss L seg Combined training: L=L seg +L c 。

Images