Method and system for identifying high-grade glioma and single brain metastases based on multi-modal data fusion

Abstract

The invention provides a method and a system for identifying high-grade glioma and single brain metastases based on multi-modal data fusion. The method comprises the following steps: acquiring an MRI image and clinical data of a patient; extracting a cross section bit slice from the MRI image, and extracting image features of the cross section bit slice; arranging the clinical data into a patient state descriptive text, and extracting semantic features of the patient state descriptive text; and fusing the image features and the semantic features based on a bidirectional attention mechanism, and generating an identification result based on the fused features. Through a multi-modal feature extraction and fusion mechanism, MRI image features and patient clinical data are effectively integrated, and the limitation of a traditional single-modal method is overcome. Compared with a traditional prediction method which only depends on images or clinical data, the method has the advantages that spatial features of the image data and semantic information of the clinical data can be fully utilized, and the recovery process of the patient is comprehensively captured, so that the accuracy and comprehensiveness of tumor identification are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical fields of medical artificial intelligence and medical image analysis, and particularly relates to a method and system for differentiating high-grade glioma and solitary brain metastasis based on multimodal data fusion. Background Art

[0002] HGG (High-Grade Glioma) and SBM (Solitary Brain Metastasis) are two common malignant brain tumors with relatively high fatality and disability rates. HGG usually grows rapidly, has strong invasiveness, and is prone to recurrence, making it difficult to treat and resulting in a relatively short survival period for patients. Although SBM is relatively rare, with the occurrence of tumor metastasis, the patient's condition deteriorates rapidly and the prognosis is poor. Although existing treatment methods such as surgery, radiotherapy, and chemotherapy can, to a certain extent, extend the survival period of patients, accurately differentiating between these two types of tumors remains a challenge in clinical diagnosis due to their significant similarities in imaging features and clinical manifestations.

[0003] Currently, the differentiation of HGG and SBM in clinical practice mainly relies on imaging data (such as anatomical and functional information provided by MRI) and the patient's basic clinical data information (such as age, past medical history, tumor marker levels, etc.). However, these methods usually have certain limitations. For example, imaging examinations can only provide limited structural information and are difficult to comprehensively reflect the molecular and functional characteristics of tumors; traditional differentiation methods based on statistical models often ignore the complex relationships and semantic features in the patient's clinical data. At the same time, clinical data usually contains a large amount of unstructured information (such as doctors' diagnostic records, treatment plans, etc.), and existing models are often unable to effectively process this unstructured data, restricting their accuracy and adaptability in practical applications.

[0004] To address these problems, in recent years, medical artificial intelligence (AI) technology has been widely applied. Although significant progress has been made in medical image analysis using deep learning methods, most existing models still focus on the processing of single data modalities (such as image data or structured clinical data) and ignore the fusion of different types of data. Summary of the Invention

[0005] In order to improve the accuracy of differentiating HGG and SBM and thus provide strong support for formulating personalized treatment plans and patient management strategies, the present invention proposes a method and system for differentiating high-grade glioma and solitary brain metastasis based on multimodal data fusion.

[0006] In a first aspect, the present invention provides a method for differentiating high-grade glioma and solitary brain metastasis based on multimodal data fusion, comprising:

[0007] Obtain the MRI images and clinical data of the patient;

[0008] Extract the transverse slices from the MRI images and extract the image features of the transverse slices;

[0009] Organize the clinical data into a descriptive text of the patient's status and extract the semantic features of the descriptive text of the patient's status;

[0010] Fuse the image features and the semantic features based on a bidirectional attention mechanism and generate a discrimination result based on the fused features.

[0011] Further, extracting the image features of the transverse slices specifically includes:

[0012] Use a patch embedding layer to generate a serialized representation of the transverse slices;

[0013] Input the serialized representation of the transverse slices into a feature extraction module based on a state space model to generate the image features of the transverse slices; wherein, the feature extraction module based on the state space model includes multiple stacked SSM modules; the SSM module includes a first branch and a second branch in parallel, and the outputs of the first branch and the second branch are concatenated as the final output; the first branch includes a layer normalization, a linear layer, a depth convolution layer, a state space layer, and a linear layer connected in sequence, and the output of the layer normalization is added to the output of the state space layer through a skip connection and used as the input of the subsequent linear layer; the second branch includes two 3×3 convolution layers and a point convolution layer connected in sequence.

[0014] When extracting the image features of the MRI images in the present invention, by combining the Patch Embedding layer with the state space model (SSM), first input the MRI image slices into the Patch Embedding layer, and by dividing the image into small blocks and mapping them into a high-dimensional space, generate a serialized representation of the image; subsequently, with the help of a four-layer feature extraction module based on the state space model (SSM), extract deep spatial features from the image; this method can effectively capture the local and global information of the tumor ROI region and simulate the temporal dependence relationship, improving the modeling ability of the long-term recovery trend.

[0015] Further, extracting the semantic features of the descriptive text of the patient's status specifically includes:

[0016] Input the descriptive text of the patient's status into the text feature extraction tool of the large language model to generate the semantic features of the descriptive text of the patient's status.

[0017] The present invention introduces a large language model (such as ChatGPT), combines its clinical data, and performs in-depth semantic analysis on the clinical data of patients (such as medical history, laboratory indicators, etc.) to generate high-quality clinical feature Embeddings. This method breaks through the limitations of traditional statistical methods, can extract personalized information of patients from complex text data, and provides more accurate input features for the identification of tumors.

[0018] Further, based on the bidirectional attention mechanism, the image features and the semantic features are fused, specifically including:

[0019] Transform the image features to generate a first query vector, a first key vector, and a first value vector; transform the semantic features to generate a second query vector, a second key vector, and a second value vector;

[0020] Perform cross-attention calculation on the first query vector, the second key vector, and the second value vector to generate new image features; perform cross-attention calculation on the second query vector, the first key vector, and the first value vector to generate new semantic features;

[0021] Concatenate the new image features and the new semantic features to obtain the fused features.

[0022] The present invention designs a cross-modal bidirectional attention mechanism. By transforming MRI image features and clinical data semantic features into query (Q), key (K), and value (V) vectors respectively, the bidirectional attention mechanism is applied for feature interaction. This way can not only strengthen the complementarity between images and clinical data, but also achieve more accurate feature fusion, improving the accuracy and robustness of the prediction model.

[0023] In a second aspect, the present invention provides a high-grade glioma and solitary brain metastasis identification system based on multi-modal data fusion, including:

[0024] A data acquisition unit for acquiring the MRI images and clinical data of patients;

[0025] An image feature extraction unit for extracting transverse slices from the MRI images and extracting the image features of the transverse slices;

[0026] A semantic feature extraction unit for organizing the clinical data into a patient status descriptive text and extracting the semantic features of the patient status descriptive text;

[0027] A fusion and detection unit for fusing the image features and the semantic features based on the bidirectional attention mechanism and generating an identification result based on the fused features.

[0028] Further, the image feature extraction unit includes:

[0029] An embedding representation subunit for generating a serialized representation of the transverse slice using a patch embedding layer;

[0030] A feature extraction subunit for inputting the serialized representation of the transverse slice into a feature extraction module based on a state space model to generate image features of the transverse slice; wherein, the feature extraction module based on the state space model includes a plurality of stacked SSM modules; the SSM module includes a first branch and a second branch arranged in parallel, and the outputs of the first branch and the second branch are concatenated as the final output; the first branch includes a layer normalization, a linear layer, a depth convolution layer, a state space layer, and a linear layer connected in sequence, and the output of the layer normalization is added to the output of the state space layer through a skip connection and used as the input of the subsequent linear layer; the second branch includes two 3×3 convolution layers and a point convolution layer connected in sequence.

[0031] Further, the semantic feature extraction unit is specifically configured to input the patient status descriptive text into a text feature extraction tool of a large language model to generate semantic features of the patient status descriptive text.

[0032] Further, the fusion and detection unit includes:

[0033] A feature transformation subunit for transforming the image features to generate a first query vector, a first key vector, and a first value vector; transforming the semantic features to generate a second query vector, a second key vector, and a second value vector;

[0034] A cross-attention subunit for performing cross-attention calculations on the first query vector, the second key vector, and the second value vector to generate new image features; performing cross-attention calculations on the second query vector, the first key vector, and the first value vector to generate new semantic features;

[0035] A concatenation subunit for concatenating the new image features and the new semantic features to obtain fused features.

[0036] In a third aspect, the present invention provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the program, the method described in the first aspect is implemented.

[0037] In a fourth aspect, the present invention provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0038] The beneficial effects of the present invention are:

[0039] (1) Improve prediction accuracy and comprehensiveness: Through the multi-modal feature extraction and fusion mechanism, the present invention effectively integrates MRI image features and patient clinical data, overcoming the limitations of traditional single-modal methods. Compared with traditional prediction methods that rely solely on image or clinical data, the present invention can make full use of the spatial features of image data and the semantic information of clinical data to comprehensively capture the patient's recovery process, thereby significantly improving the accuracy and comprehensiveness of tumor identification.

[0040] (2) Enhance the ability to capture details: By introducing a multi-layer state space model and deep convolutional layers, the present invention can capture multi-level and multi-scale detailed information during the image feature extraction process. This enables the model to not only focus on the overall structure of the MRI image but also deeply analyze the subtle changes in the image. Especially for complex ROI regions, it can more accurately identify and extract relevant features, improving the sensitivity and accuracy of the model.

[0041] (3) Improve the generalization ability and practicality of the model: By designing feature fusion based on a bidirectional attention mechanism, the present invention can adaptively fuse key information in MRI images and clinical data. Since this method can process data from different sources (such as MRI images and clinical data), the model not only performs well on the training dataset but also maintains high accuracy on new and unseen patient data. This strong generalization ability ensures that the present invention can be widely applied in different clinical environments, has higher practical clinical value, and can support the accurate diagnosis of different patient groups and the formulation of personalized treatment plans.

[0042] In summary, the present invention can effectively improve the accuracy, practicality of tumor identification, and the generalization ability of the model, providing strong technical support for clinical diagnosis and treatment, and having broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a schematic flow chart of a method for differentiating high-grade gliomas and solitary brain metastases based on multi-modal data fusion provided by an embodiment of the present invention;

[0044] Figure 2 It is a four-layer feature extraction module designed based on a state space model (SSM) provided by an embodiment of the present invention;

[0045] Figure 3 It is the multi-modal data feature fusion process provided by an embodiment of the present invention;

[0046] Figure 4 It is a schematic structural diagram of a system for differentiating high-grade gliomas and solitary brain metastases based on multi-modal data fusion provided by an embodiment of the present invention;

[0047] Figure 5 The structural block diagram of an electronic device provided by an embodiment of the present invention. Specific implementation manners

[0048] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0049] To address the limitations of existing methods in multi-modal data fusion and complex feature extraction, there is an urgent need for a new multi-modal prediction model that can comprehensively consider MRI images and patients' personalized clinical data, combine the advantages of deep learning and large language models, better capture the complex relationships between tumor features, improve the discrimination accuracy of HGG and SBM, and thus provide strong support for formulating personalized treatment plans and patient management strategies.

[0050] The objective of the present invention is to provide a method and system for discriminating HGG and SBM by realizing multi-modal data fusion based on a large language model and a state space model, aiming to effectively fuse MRI image data with patients' personalized clinical data, improve the accurate discrimination ability of tumor types, thereby providing more accurate diagnostic basis for doctors, optimizing clinical decisions, and enhancing the treatment effect of patients.

[0051] As Figure 1 shown, an embodiment of the present invention provides a method for discriminating high-grade glioma and solitary brain metastasis based on multi-modal data fusion, including a training stage and a testing stage. Among them, the training stage includes the following steps:

[0052] S101: Obtain MRI images and clinical data with known class labels;

[0053] Specifically, this embodiment uses a case private dataset from a certain medical center, including MRI images and clinical data of 101 HGG patients and 82 SBM patients. These datasets cover patients' basic information, past medical history, treatment process, and laboratory test indicators. These clinical features have been widely verified in multiple studies and are closely related to the treatment effect and prognosis of tumors, and have important reference value for disease assessment. It should be noted that the basic information of the patients here has been data desensitized and will not violate the privacy of the patients.

[0054] Furthermore, in this embodiment, the original image data and clinical data are also preprocessed. When processing clinical data, the interpolation method is used to complete the missing values to ensure the integrity and consistency of the data. For the preprocessing of MRI images, in this embodiment, axial slices are directly extracted from the original MRI images to provide standardized key information for subsequent analysis and model training.

[0055] S102: Extract axial slices from the MRI images and extract the image features of the axial slices;

[0056] Specifically, extracting axial slices has the following several significant advantages: First, axial slices usually contain the core structures and lesion areas of the brain and can accurately reflect the scope and severity of tumors or other lesions. Second, through standardized extraction methods, the image dimension can be effectively reduced, thereby improving the training efficiency and accelerating the convergence process of the model. At the same time, this method ensures the consistency of the input data among different patients, reduces the data heterogeneity, and enhances the generalization ability of the model on different data sets. Finally, focusing on the key slices helps to reduce the unnecessary computational burden, reduce noise and redundant information, thereby further improving the accuracy of prognosis prediction.

[0057] All MRI images are subjected to affine transformation, and the image size is adjusted to 256×256 to unify the standardized input size. By reducing the image resolution, the redundant information in the high-resolution images is reduced, thereby accelerating the training process of the model, reducing the computational burden, and improving the efficiency of the overall model at the same time.

[0058] S103: Organize the clinical data into a patient status descriptive text and extract the semantic features of the patient status descriptive text to provide key information about the patient's condition, past medical history, and treatment process;

[0059] Specifically, when differentiating HGG and SBM, extracting the clinical features of the patient is crucial for the accuracy of the model. By introducing the patient's individual information, the model can better capture the differences among different patients, thereby improving the prediction accuracy and personalization.

[0060] This embodiment combines a text feature extraction tool to make full use of the patient's clinical data to generate meaningful semantic features, helping to improve the prediction ability of the model. Specifically, first, various clinical indicators of the patient are constructed into a text that details the patient's status (i.e., one patient corresponds to one descriptive text of the patient's status). Then, using the text feature extraction tool provided by ChatGPT, semantic parsing is performed on these descriptive texts to generate corresponding feature Embeddings. ChatGPT's powerful natural language processing capabilities ensure that the extracted Embeddings can effectively capture the key information in the text, thereby providing a deep understanding of the patient's clinical status.

[0061] S104: Based on the bidirectional attention mechanism, fuse the image features and the semantic features, and generate a discrimination result based on the fused features.

[0062] Specifically, traditional medical prediction models usually only focus on single-modal data, such as MRI images or structured clinical data. It is difficult for the prior art to fully fuse image data and clinical data, especially the correlation between image data and unstructured text data has not been effectively mined. Due to the lack of an effective fusion mechanism, the prediction effect of the model is limited and it cannot make full use of the multi-dimensional information provided by various types of data. The present invention designs a deep fusion of MRI image features and the patient's personalized clinical data (including text information processed by large language models) based on the bidirectional attention mechanism, which can make full use of the synergy between image data and clinical data, mine their potential relationships, and thus improve the accuracy and reliability of prediction.

[0063] After the above steps, a trained tumor discrimination model can be obtained. In the test phase, the MRI images and clinical data of the patient to be discriminated are acquired, and according to the processing process in the training phase, the corresponding transverse slices and text are obtained and input into the trained tumor discrimination model, thereby obtaining a prediction result.

[0064] The present invention focuses on extracting effective features from MRI images for accurate discrimination of HGG and SBM. In one embodiment, in order to extract representative image features, this embodiment adopts a feature extraction process based on MRI transverse slices.

[0065] Specifically, each patient's coronal MRI slice is first input into a patch embedding layer, which divides the two-dimensional image slice into several small patches and maps these patches into a high-dimensional vector space to generate a serialized representation of the image. Next, these serialized image representations are passed to a feature extraction module designed based on a state space model (SSM). This feature extraction module consists of four stacked SSM modules to capture the spatial and temporal features in the image and analyze the dynamic changes of the tumor area. The SSM module is as follows: Figure 2 As shown in the figure. Each SSM module includes a first branch and a second branch in parallel. The output of the first branch and the output of the second branch are concatenated as the final output. The first branch includes a layer normalization layer, a linear layer, a depthwise convolution layer, a state-space layer, and a linear layer connected in sequence. The output of the layer normalization layer is added to the output of the state-space layer via a skip connection and then used as the input of the subsequent linear layer. The second branch includes two 3×3 convolutional layers and a point convolutional layer connected in sequence.

[0066] The layer normalization (LN) layer stabilizes network training and accelerates model convergence. The linear layer maps high-dimensional vectors to a specific feature space. The deep convolutional layer expands the receptive field, effectively capturing both local and global information in the image. The state space layer simulates temporal dependencies, thereby improving the ability to model long-term patient recovery trends. The point convolutional layer uses a 1×1 kernel.

[0067] It should be noted that in the feature extraction module designed in this embodiment, the second branch uses two 3×3 convolutional layers specifically for extracting fine-grained features, so that the detailed information in the MRI image can be captured more accurately. At the same time, in the feature extraction process, the present invention adopts the Skip Connection technology to ensure that the feature information of each layer is retained and the extraction results of the previous layer can be fully utilized. This design not only enhances the efficiency of feature transmission, but also avoids the problem of information loss. At the same time, the SSM modules are connected through downsampling operations, which not only reduces the computational overhead, but also further refines the high-level features of the image.

[0068] Most existing fusion methods are unable to simultaneously process information from different modalities (such as MRI images and text data). In particular, how to deeply fuse image data and text data at the semantic level to provide more accurate predictions remains a difficulty. In order to solve this problem, in the processing of MRI image data, the present invention adopts the MedMamba network based on the state-space model, which can more accurately extract key features in the image and combine these features with clinical data to improve the accuracy of image analysis. This method can better extract potential key information when processing complex medical imaging data.

[0069] Multi-modal learning, especially the research combining imaging data and unstructured clinical data, is still in its infancy, and the accuracy and reliability of related models still need to be further improved. In addition, with the development of large language models (LLMs) such as ChatGPT, which can effectively process and understand unstructured text data and provide more accurate semantic information, this provides new possibilities for the intelligent processing of clinical data. However, how to combine the semantic features extracted by large language models with medical imaging data remains a challenge faced by current technologies. The effective fusion of multi-modal features is crucial for accurately differentiating HGG and SBM. In one embodiment, in order to make full use of the complementarity between MRI image features and clinical data features, this embodiment designs a feature fusion module based on a bidirectional attention mechanism, as Figure 3 shown, aiming to improve the prediction performance of the model through cross-modal information interaction.

[0070] First, the image features extracted from MRI images and the semantic features extracted from clinical data are respectively converted into three new vectors. Among them, the image features are converted into query vector Q ct , key K ct and value V ct vectors, while the semantic features are converted into query vector Q cd , key K cd and value V cd vectors. The generation of these vectors is to enable the execution of the attention mechanism between multi-modal features, thereby capturing their interaction relationships.

[0071] Next, the Q ct of the MRI image features is subjected to attention calculation with the K cd and V cd of the clinical data semantic features. This process enables the MRI image features to be weighted according to the relevance of the clinical data semantic features, thereby obtaining a new MRI image feature containing clinical information. Similarly, the Q cd of the clinical data semantic features is subjected to attention calculation with the K ct and V ct of the MRI image features. This process enables the clinical data semantic features to also incorporate the key information of the MRI image features, generating a new clinical data semantic feature containing MRI information. The above process is expressed by the following formula:

[0072]

[0073]

[0074] Through this bidirectional attention mechanism, two new features are obtained. They not only retain the original information of their respective modalities but also introduce information from the other modality through interaction. This fusion strategy enables the model to consider the potential associations between imaging features and clinical features simultaneously during prediction, providing richer context information for the final decision-making.

[0075] Finally, the present invention concatenates (concat) these two newly generated features to obtain the fused features, which are input into a linear layer. Through further feature fusion and dimensionality reduction processing, the final prediction result is generated.

[0076] The present invention realizes more accurate preoperative differential diagnosis of HGG and SBM by deeply fusing MRI images and clinical data. Compared with traditional statistical models, the method of the present invention can consider the individual differences of patients, perform accurate differentiation of HGG and SBM according to the individual conditions of patients, provide more personalized diagnostic support for doctors, and thus improve the accuracy and effectiveness of clinical decision-making.

[0077] Since existing methods cannot effectively combine the personalized clinical data and imaging data of patients, the accuracy and reliability of prediction are often low. In practical applications, doctors may have difficulty making appropriate treatment decisions based on the prediction results, thus affecting the treatment effect and prognosis of patients. The present invention significantly improves the accuracy and reliability in the differentiation of HGG and SBM through the deep fusion of multi-modal data and the extraction of semantic features of large language models. Compared with the prior art, the present invention can provide more accurate tumor type differentiation results, help doctors make more scientific decisions in clinical practice, and optimize treatment plans and management strategies.

[0078] Based on the same inventive concept, as Figure 4 shown, an embodiment of the present invention further provides a system for differentiating high-grade glioma and single brain metastasis based on multi-modal data fusion, including a data acquisition unit, an imaging feature extraction unit, a semantic feature extraction unit, and a fusion and detection unit.

[0079] Among them, the data acquisition unit is used to acquire the MRI images and clinical data of patients; the imaging feature extraction unit is used to extract transverse slices from the MRI images and extract the imaging features of the transverse slices; the semantic feature extraction unit is used to organize the clinical data into a patient status descriptive text and extract the semantic features of the patient status descriptive text; the fusion and detection unit is used to fuse the imaging features and the semantic features based on the bidirectional attention mechanism and generate a differentiation result based on the fused features.

[0080] It should be noted that the differentiation system provided in the embodiment of the present invention is to implement the above method, and its functions can be specifically referred to the above method embodiments and will not be elaborated here.

[0081] Figure 5 An entity structure schematic diagram of an electronic device is exemplified. As Figure 5 shown, the electronic device may include: a processor 501, a communications interface 502, a memory 503, and a communication bus 504. Among them, the processor 501, the communications interface 502, and the memory 503 complete mutual communication through the communication bus 504. The processor 501 may call logical instructions in the memory 503 to execute a method for differentiating high-grade glioma and solitary brain metastasis based on multimodal data fusion. The method includes: acquiring the MRI images and clinical data of a patient; extracting transverse slices from the MRI images and extracting the image features of the transverse slices; organizing the clinical data into a descriptive text of the patient's state and extracting the semantic features of the descriptive text of the patient's state; fusing the image features and the semantic features based on a bidirectional attention mechanism, and generating a differentiation result based on the fused features.

[0082] In addition, when the logical instructions in the above-mentioned memory 503 are implemented in the form of software functional units and sold or used as independent products, they may be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, may be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs, etc., which can store program codes.

[0083] The embodiment of the present invention also provides a computer program product. The computer program product includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the method for differentiating high-grade glioma and solitary brain metastasis based on multimodal data fusion provided by each of the above method embodiments.

[0084] The embodiment of the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the method for differentiating high-grade glioma and solitary brain metastasis based on multimodal data fusion provided by each of the above method embodiments.

[0085] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of each embodiment of the present invention.

Claims

1. An advanced glioma and solitary brain metastasis differentiation method based on multimodal data fusion, characterized in that, including: Obtaining the MRI images and clinical data of the patient; Extracting cross-sectional slices from the MRI images and extracting the image features of the cross-sectional slices; Organizing the clinical data into a descriptive text of the patient's status and extracting the semantic features of the descriptive text of the patient's status; Fusing the image features and the semantic features based on a bidirectional attention mechanism and generating a discrimination result based on the fused features.

2. The method for differentiating high-grade glioma and solitary brain metastasis based on multi-modal data fusion according to claim 1, characterized in that, Extracting the image features of the cross-sectional slices specifically includes: Generating a serialized representation of the cross-sectional slices using a patch embedding layer; Inputting the serialized representation of the cross-sectional slices into a feature extraction module based on a state space model to generate the image features of the cross-sectional slices; wherein, the feature extraction module based on the state space model includes multiple stacked SSM modules; the SSM module includes a first branch and a second branch in parallel, and the outputs of the first branch and the second branch are concatenated as the final output; the first branch includes a layer normalization, a linear layer, a depth convolutional layer, a state space layer, and a linear layer connected in sequence, and the output of the layer normalization is added to the output of the state space layer through a skip connection and used as the input of the subsequent linear layer; the second branch includes two 3×3 convolutional layers and a point convolutional layer connected in sequence.

3. The method for differentiating high-grade glioma and solitary brain metastasis based on multimodal data fusion according to claim 1, wherein, Extracting the semantic features of the descriptive text of the patient's status specifically includes: Inputting the descriptive text of the patient's status into a text feature extraction tool of a large language model to generate the semantic features of the descriptive text of the patient's status.

4. The method for differentiating high-grade glioma and solitary brain metastasis based on multimodal data fusion according to claim 1, wherein Fusing the image features and the semantic features based on a bidirectional attention mechanism specifically includes: Converting the image features to generate a first query vector, a first key vector, and a first value vector; converting the semantic features to generate a second query vector, a second key vector, and a second value vector; Performing cross-attention calculation on the first query vector, the second key vector, and the second value vector to generate new image features; performing cross-attention calculation on the second query vector, the first key vector, and the first value vector to generate new semantic features; Concatenating the new image features and the new semantic features to obtain the fused features.

5. An advanced glioma and solitary brain metastasis differential diagnosis system based on multimodal data fusion, characterized in that, including: A data acquisition unit for obtaining the MRI images and clinical data of the patient; An image feature extraction unit for extracting cross-sectional slices from the MRI images and extracting the image features of the cross-sectional slices; A semantic feature extraction unit for organizing the clinical data into a descriptive text of the patient's status and extracting the semantic features of the descriptive text of the patient's status; A fusion and detection unit for fusing the image features and the semantic features based on a bidirectional attention mechanism and generating a discrimination result based on the fused features.

6. The high-grade glioma and solitary brain metastasis differential diagnosis system based on multi-modal data fusion according to claim 5, characterized in that, The image feature extraction unit includes: An embedded representation subunit for generating a serialized representation of the cross-sectional slices using a patch embedding layer; A feature extraction sub-unit, configured to input the serialized representation of the cross-sectional slice into a feature extraction module based on a state space model to generate image features of the cross-sectional slice; wherein, the feature extraction module based on the state space model includes a plurality of stacked SSM modules; the SSM module includes a first branch and a second branch arranged in parallel, and the outputs of the first branch and the second branch are concatenated as the final output; the first branch includes a layer normalization, a linear layer, a depth convolution layer, a state space layer, and a linear layer connected in sequence, and the output of the layer normalization is added to the output of the state space layer through a skip connection and used as the input of the subsequent linear layer; the second branch includes two 3×3 convolution layers and a point convolution layer connected in sequence.

7. The high-grade glioma and solitary brain metastasis differential diagnosis system based on multimodal data fusion according to claim 5, characterized in that The semantic feature extraction unit is specifically configured to input the patient state descriptive text into the text feature extraction tool of the large language model to generate semantic features of the patient state descriptive text.

8. The high-grade glioma and solitary brain metastasis discrimination system based on multi-modal data fusion according to claim 5, characterized in that, The fusion and detection unit includes: A feature transformation sub-unit, configured to transform the image features to generate a first query vector, a first key vector, and a first value vector; transform the semantic features to generate a second query vector, a second key vector, and a second value vector; A cross-attention sub-unit, configured to perform cross-attention calculation on the first query vector, the second key vector, and the second value vector to generate new image features; perform cross-attention calculation on the second query vector, the first key vector, and the first value vector to generate new semantic features; A concatenation sub-unit, configured to concatenate the new image features and the new semantic features to obtain fused features.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method according to any one of claims 1 to 4.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method according to any one of claims 1 to 4.

Images