A brain tumor molecular phenotype classification method and system based on multi-modal magnetic resonance

Abstract

This invention discloses a method and system for molecular phenotypic classification of brain tumors based on multimodal magnetic resonance imaging (MRI), relating to the fields of medical image analysis and artificial intelligence. MRI images from each modality are input into a feature extraction network to obtain multi-level features. These features are then aligned using a feature alignment module to unify spatial dimensions and channel dimensions. The aligned multi-level features are then input into a multi-level feature fusion module within a modality, enabling reversible interaction between shallow detail features and deep semantic features within the same modality, resulting in fused features for each modality. These fused features are then input into a global modeling network to obtain global feature vectors for each modality. These vectors are then input into a multi-view feature fusion module between modalities to complete cross-modal nonlinear interaction and information redistribution, yielding the final fused feature vector. The classification head outputs the molecular phenotypic prediction results, achieving automated prediction of glioma molecular phenotypes. This invention achieves efficient fusion of multi-scale and multimodal information while maintaining a relatively compact network structure.

Description

Technical Field

[0001] This invention relates to the fields of medical image analysis and artificial intelligence technology, and in particular to a method and system for molecular phenotypic classification of brain tumors based on multimodal magnetic resonance imaging. Background Technology

[0002] The statements in this section are merely background information relating to this disclosure and do not necessarily constitute prior art.

[0003] Brain tumors (especially gliomas) are characterized by high invasiveness and significant heterogeneity. With the development of molecular pathology, molecular markers such as IDH mutations, 1p / 19q co-deletion, and MGMT promoter methylation have been incorporated into the WHO classification of central nervous system tumors, playing a crucial role in prognostic assessment and treatment decisions. However, current molecular phenotypic testing mainly relies on postoperative or biopsy tissue, which presents challenges such as high invasiveness, limited sampling, and difficulty in dynamic follow-up.

[0004] MRI can provide information on tumor structure, edema, and blood-brain barrier disruption under non-invasive conditions. Commonly used multimodal sequences include T1, T2, T1ce, and FLAIR, and each modality is complementary in histological characterization. Therefore, radiomics and deep learning methods based on multimodal MRI are widely used for non-invasive prediction of molecular phenotypes.

[0005] Existing methods include schemes combining artificial radiomics features with traditional machine learning, and end-to-end deep learning schemes based on models such as CNN and Transformer (2D slice or 3D volumetric data modeling). In multimodal fusion, common strategies include input-level channel stacking, feature-level stitching / weighted / attention fusion, and decision-level ensemble. However, these methods still suffer from problems such as difficulty in explicitly modeling intermodal dependencies, insufficient utilization of multi-scale correlations due to a single fusion level, large model parameter count in 3D multimodal modeling, and susceptibility to overfitting in small-sample medical images.

[0006] Furthermore, 3D multimodal MRI data has a large voxel size and low signal-to-noise ratio. Some methods reduce computational cost by downsampling, cropping, or simplifying modalities / viewpoints, which may result in loss of spatial structure and cross-modal details. Other methods introduce complex attention or multi-stage structures to enhance expression, but stability and interpretability still have room for improvement under small sample conditions. Overall, how to more fully mine multimodal and multi-scale complementary information under controllable complexity to achieve accurate and robust prediction of glioma molecular phenotypes remains an urgent technical problem to be solved. Summary of the Invention

[0007] To address the issues of insufficient utilization of modal complementarity information, inadequate hierarchical multiscale fusion, and overfitting in small samples in multimodal MRI glioma molecular phenotype prediction, this invention provides a method and system for classifying brain tumor molecular phenotypes based on multimodal magnetic resonance imaging. This method performs feature extraction, feature fusion, and classification prediction on multimodal three-dimensional magnetic resonance images, enabling automated prediction of glioma molecular phenotypes (such as MGMT promoter methylation status, IDH-1p / 19q co-deletion status, etc.).

[0008] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: In a first aspect, the present invention provides a method for molecular phenotypic classification of brain tumors based on multimodal magnetic resonance imaging, comprising: Acquire multimodal three-dimensional magnetic resonance images, preprocess them, and obtain preprocessed multimodal three-dimensional magnetic resonance images; The preprocessed multimodal three-dimensional magnetic resonance images are input into a molecular phenotype classification model for processing to obtain molecular phenotype prediction results for brain tumors. The molecular phenotypic classification model includes multiple parallel modality-specific branches, a multi-view feature fusion module between modalities, and a classification head. Its construction process includes: The preprocessed multimodal 3D magnetic resonance images are input into specific branches of the corresponding modes for mode-specific feature extraction and fusion to obtain global feature vectors for each mode; The global feature vectors of each modality are input into the intermodal multi-view feature fusion module, and cross-modal nonlinear interaction and information redistribution are performed based on a two-layer affine coupling layer to obtain the fused feature vector. The fused feature vector is input into the classification head for prediction to obtain the molecular phenotype prediction result of brain tumor.

[0009] In a further technical solution, the modality-specific branch sequentially includes a feature extraction network, a feature alignment module, an intramodal multi-level feature fusion module, and a global modeling network.

[0010] A further technical solution is that the construction process of the modality-specific branch includes: The feature extraction network performs preliminary feature extraction on the three-dimensional magnetic resonance image to obtain multi-level features; the multi-level features are then input into the feature alignment module to align them in terms of channel number and spatial resolution to obtain aligned multi-level features. The shallow features in the aligned multi-level features are concatenated to obtain shallow concatenated features, and the deep features are concatenated to obtain deep concatenated features. The shallow and deep concatenated features are input into the intramodal multi-level feature fusion module, and the two are fused and interacted using an improved affine coupling layer to obtain comprehensive features. The comprehensive features are flattened and linearly mapped before being input into a global modeling network for global modeling and feature enhancement. Modal-level global feature vectors are constructed based on a multi-head self-attention mechanism.

[0011] In a further technical solution, the affine coupling layer in the intermodal multi-view feature fusion module is stacked in a series mode and a parallel mode. The series mode performs sequentially guided cross-modal interaction, while the parallel mode performs simultaneous mutual guidance cross-modal interaction.

[0012] A further technical solution is proposed: the transformation formula for the two-layer affine coupling layer in the series mode is as follows:

[0013]

[0014]

[0015]

[0016]

[0017]

[0018] in, This is the first set of global feature vectors. This is the second set of global feature vectors. For Hadama accumulation, This is the scaling function in the first affine coupling layer. This is the translation transformation function in the first affine coupling layer. This is the scaling function in the second affine coupling layer. This is the translation transformation function in the second affine coupling layer. To fuse feature vectors.

[0019] A further technical solution, the transformation formula for the parallel mode's two-layer affine coupling layer is:

[0020]

[0021]

[0022] in, This is the first set of global feature vectors. This is the second set of global feature vectors. For Hadama accumulation, This is the scaling function in the first branch affine coupling layer. Let be the translation transformation function in the first branch affine coupling layer. This is the scaling function in the second branch affine coupling layer. This is the translation transformation function in the second branch affine coupling layer.

[0023] A further technical solution involves grouping the global feature vectors of each modality according to a variety of preset grouping methods before fusing them, and then concatenating the global feature vectors within the same group as input.

[0024] Secondly, the present invention provides a molecular phenotypic classification system for brain tumors based on multimodal magnetic resonance imaging, comprising: The data acquisition module is configured to: acquire multimodal three-dimensional magnetic resonance images, preprocess them, and obtain preprocessed multimodal three-dimensional magnetic resonance images; The model classification module is configured to input the preprocessed multimodal three-dimensional magnetic resonance images into the molecular phenotype classification model for processing, and obtain the molecular phenotype prediction results of brain tumors. The molecular phenotypic classification model includes multiple parallel modality-specific branches, a multi-view feature fusion module between modalities, and a classification head. Its construction process includes: The preprocessed multimodal 3D magnetic resonance images are input into specific branches of the corresponding modes for mode-specific feature extraction and fusion to obtain global feature vectors for each mode; The global feature vectors of each modality are input into the intermodal multi-view feature fusion module, and cross-modal nonlinear interaction and information redistribution are performed based on a two-layer affine coupling layer to obtain the fused feature vector. The fused feature vector is input into the classification head for prediction to obtain the molecular phenotype prediction result of brain tumor.

[0025] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the method for molecular phenotypic classification of brain tumors based on multimodal magnetic resonance as described in the first aspect.

[0026] Fourthly, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the method for molecular phenotypic classification of brain tumors based on multimodal magnetic resonance as described in the first aspect.

[0027] The above one or more technical solutions have the following beneficial effects: This invention acquires one or more three-dimensional MRI images (T1, T2, T1ce, FLAIR) and corresponding molecular phenotypic labels from the subject, performs registration and normalization preprocessing, and inputs each modality into a feature extraction network to obtain multi-level features. These features are then aligned using a feature alignment module to unify spatial dimensions and channel dimensions. The aligned multi-level features are input into a modal-level feature fusion module to achieve reversible interaction between shallow detail features and deep semantic features within the same modality, resulting in fused features for each modality. These fused features are then input into a global modeling network to obtain global feature vectors for each modality. These vectors are then input into a multi-view feature fusion module between modalities, where cross-modal nonlinear interaction and information redistribution are performed based on serial or parallel affine coupling layers, yielding the final fused feature vector. Finally, a classification sub-network outputs the molecular phenotypic prediction results, enabling automated prediction of glioma molecular phenotypes.

[0028] This invention achieves efficient fusion of multi-scale and multi-modal information while maintaining a relatively compact network structure. It can improve the accuracy and robustness of molecular phenotypic predictions such as MGMT promoter methylation, IDH mutation, and 1p / 19q co-deletion, and has good prospects for clinical application.

[0029] This invention proposes a two-stage reversible fusion (intramodal multi-level feature fusion and intermodal multi-perspective feature fusion), which takes into account both local structural details and global discriminative semantics under the premise of controllable complexity, and explores the complementary and guiding relationships between different modalities; moreover, the reversible coupling mechanism helps to stabilize information interaction and feature redistribution, reduce parameter redundancy and overfitting risk, and improve robustness and generalization ability under small sample conditions.

[0030] The method of this invention can be trained end-to-end and directly inferred, and is applicable to tasks such as MGMT methylation state prediction and IDH-1p / 19q joint state prediction.

[0031] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0032] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0033] Figure 1 This is a schematic diagram of the overall structure of a molecular phenotypic classification method for brain tumors based on multimodal magnetic resonance according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the internal structure of the intramodal multi-level feature fusion module and the intermodal multi-view feature fusion module in an embodiment of the present invention. Detailed Implementation

[0034] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0035] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0036] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0037] Example 1 like Figure 1 As shown in the figure, this embodiment discloses a molecular phenotypic classification method for brain tumors based on multimodal magnetic resonance imaging. The method includes the following steps: S1: Acquire multimodal 3D magnetic resonance images, preprocess them, and obtain preprocessed multimodal 3D magnetic resonance images; In this embodiment, multimodal three-dimensional magnetic resonance images and their molecular phenotypic labels are acquired, and the images are processed by tumor region of interest extraction, cropping, resampling, and intensity normalization.

[0038] Acquire multimodal three-dimensional magnetic resonance images, including T1-weighted imaging, T2-weighted imaging, T1-weighted contrast-enhanced imaging (T1ce), and fluid attenuation inversion recovery sequence (FLAIR). T1-NIFTI, T2-NIFTI, T1ce-NIFTI, and FLAIR-NIFTI are brain magnetic resonance imaging (MRI) sequence data stored in NIFTI format.

[0039] The input MRI is preprocessed, mainly including region of interest (ROI) extraction, background voxel cropping, and volume data resampling to a fixed size. There are four input modalities. The result obtained after this preprocessing step for each mode is denoted as . .

[0040] S2: Input the preprocessed multimodal 3D magnetic resonance images into the molecular phenotype classification model for processing to obtain the molecular phenotype prediction results of brain tumors.

[0041] In this embodiment, the molecular phenotype classification model includes multiple parallel modality-specific branches, an intermodality multi-view feature fusion module, and a classification head. Each modality-specific branch includes, in sequence, a feature extraction network, a feature alignment module, an intramodality multi-level feature fusion module, and a global modeling network.

[0042] S201: The preprocessed multimodal 3D magnetic resonance images are input into the corresponding modal specific branches, and the feature extraction network performs preliminary feature extraction on the 3D magnetic resonance images to obtain multi-level features; The feature extraction network in each branch uses 3D-ResNet18 for initial feature extraction, resulting in multi-level features. Specifically, will The features are sequentially fed into the corresponding branches of the 3D-ResNet18 for initial feature extraction. Each 3D-ResNet18 consists of an initial convolutional layer, a pooling layer, and several residual blocks. These can be divided into four stages, from Layer 1 to Layer 4. The feature maps output by each stage, with different spatial resolutions and semantic levels, are denoted as... , For hierarchical indexes, where Generally speaking, shallower features (e.g.) , It emphasizes fine-grained information such as local edges and textures, while deeper features (e.g.) are less important. , It has a larger effective receptive field and is more inclined to encode high-level semantic features related to the target.

[0043] S202: Multi-level feature input feature alignment module, which aligns the multi-level features in terms of channel number and spatial resolution to obtain aligned multi-level features; the shallow features in the aligned multi-level features are concatenated to obtain shallow concatenated features, and the deep features are concatenated to obtain deep concatenated features. The shallow and deep concatenated features are input into the intramodal multi-level feature fusion module, and an improved affine coupling layer is used to fuse and interact the two to obtain comprehensive features; because The number of channels is inconsistent with the spatial resolution, so a feature alignment module (FeatureAligner unit) is introduced into the framework. For channel number dimensionality alignment, FeatureAligner uses a 1×1×1 three-dimensional convolutional kernel to remap the channels of each feature layer, unifying the channel number to a uniform dimension. In terms of spatial resolution alignment, FeatureAligner aligns four-level feature maps for each modality. Spatial resampling is performed. For deep features with small spatial dimensions, trilinear interpolation is used to upsample them to the target size. For shallow features with large spatial dimensions, 3D average pooling or convolution with a stride greater than 1 is used for downsampling. Finally, all four levels of features are adjusted to a uniform spatial size. The final result is an aligned multi-level feature set. .

[0044] Next, we will analyze the multi-level features within each modality. Feature-level fusion is performed to achieve reversible multi-scale fusion of features at different levels within the same modality. The module used is INN Early FeatureFusion, an intra-modal multi-level feature fusion module designed with a single-layer improved affine coupling layer structure. Its purpose is to guide and interact between shallow features (emphasizing local edges, texture, and other details) and deep features (preferring to encode high-level semantic features related to the target) within the same modality, ultimately yielding a fused feature map.

[0045] Specifically, firstly, the output of FeatureAligner... and The fused features are obtained by stitching along the channel dimension. Similarly, and The fused features are obtained by stitching along the channel dimension. Then and It can be considered as two parts of the input to the INN Early Feature Fusion module.

[0046] The internal structure of the INN Early Feature Fusion module is designed based on an Invertible Neural Network (INN). It employs a modified Affine Coupling Layer (ACL) to achieve the fusion and interaction of shallow and deep features. Finally, the two outputs of the ACL are concatenated to obtain the early fused features. The specific transformation formula is as follows:

[0047]

[0048]

[0049] in, and These are the output characteristics of the two branches of the affine coupling layer (which can also be regarded as intermediate outputs in the coupling transformation process). For the first Shallow splicing features of each modality For the first Deep splicing features of each modality, , , These are feature transformation subnetworks with identical structures but whose parameters are not shared (corresponding to the scaling / translation term or residual compensation term in affine coupling transformation). For Hadama accumulation, As a comprehensive feature, This is a splicing operation. Among them, the feature transformation subnetwork... It is a three-dimensional bottleneck residual block (BRB) unit.

[0050] The improved affine coupling layer differs from the classic single-layer ACL in that "part of the input directly serves as the corresponding output, while the other part undergoes affine transformation under conditions." The main difference lies in... Additional elements were introduced in the construction. This improvement allows deeper features to modulate shallower features in reverse, enabling bidirectional interaction between shallow and deep features using only one ACL layer. The improved affine coupling layer design considers that INN Early Feature Fusion is located at the front end of the overall classification framework. Using a classic ACL structure would require stacking multiple layers and involving swapping operations to achieve mutual guidance between the two feature sets, significantly increasing model complexity at the network front end and often introducing overfitting risks. The improved affine coupling layer (INN feature interaction mechanism) is introduced into multi-level feature fusion scenarios within the same MRI modality to achieve effective interaction between features from different levels of the feature extraction network. Together with the subsequent INN LateFeature Fusion module, it constitutes a staged, reversible fusion method for multimodal MRI brain tumor molecular phenotypic classification tasks.

[0051] S203: After flattening and linearly mapping the comprehensive features, they are input into the global modeling network for global modeling and feature enhancement. Modal-level global feature vectors are constructed based on the multi-head self-attention mechanism. The introduction of the FeatureProjector module will Rearrange and flatten it in spatial dimensions, and convert it into length. The token sequence, and the dimension of each token is determined by... Projected onto a fixed embedding dimension This process can be represented as Then... Send in Encoding is performed using stacked TransformerEncoder modules, and the multi-head self-attention (MHSA) mechanism within them is used to construct modality-level global feature vectors. The global feature vector is a high-dimensional discriminative feature vector.

[0052] S204: Input the global feature vectors of each modality into the intermodal multi-view feature fusion module, and perform cross-modal nonlinear interaction and information redistribution based on a two-layer affine coupling layer to obtain the fused feature vector; Next, we will analyze the global features of all modalities. For feature-level fusion, the intermodal multi-view feature fusion module INN Late Feature Fusion is used. This module is also designed based on the structure of invertible neural networks (INN). It adopts a stacked structure of two classic affine coupling layers, and the two layers include the exchange of corresponding positions between the two inputs / outputs. The purpose is to achieve guidance and interaction between high-level feature vectors across modalities. By fusing the multi-view features provided by different modalities, the fused feature vector is finally obtained.

[0053] Before fusing the global feature vectors of each modality, the global feature vectors of each modality are grouped according to several preset grouping methods. The grouped features are then concatenated to form two input parts to adapt to the stacked affine coupling layer structure of serial or parallel modes to achieve cross-modal feature interaction fusion. This is used to compare the differences in the impact of cross-modal guidance and interaction effects under different modal groupings on the downstream molecular phenotypic classification results. Specifically, since the INN core structure ACL is a dual-input dual-output structure, and there are four modal global feature vectors as inputs, it is first necessary to group the four modalities pairwise, resulting in a total of Modal combination methods, for example, and Divide into groups, and Divide into another group. After completing the modal grouping, concatenate the global feature vectors within the same group to obtain:

[0054] in, This is the first set of global feature vectors. This is the second set of global feature vectors. It will be used as the input for two parts of a single-layer ACL.

[0055] Regarding the stacking method, this invention first adopts the classic cascading form of ACLs, stacking two ACLs in series. The output of the first layer is swapped before being sent to the second layer to achieve sequential guided cross-modal transformation, such as... Figure 2 As shown in the Sequential ACL, the specific transformation formula of the Sequential ACL is as follows:

[0056]

[0057]

[0058]

[0059]

[0060]

[0061] in, This refers to the scaling function in the first-level ACL. This refers to the translation transformation function in the first-level ACL. This refers to the scaling function in the second-level ACL. This refers to the translation transformation function in the second-level ACL. To fuse feature vectors.

[0062] The core feature of the Sequential ACL is that it uses a sequential mechanism of "reference → guided → exchange → re-reference" to allow two sets of modes to take turns acting as conditional information / guided parts, thereby gradually completing the redistribution and fusion of cross-modal information.

[0063] In addition to the classic series structure, this invention also introduces a novel parallel ACL stacking method, such as... Figure 2 As shown in the Parallel ACL, in Parallel ACL mode, INN Late Feature Fusion still uses ( As input to each ACL layer, but unlike Sequential ACL, the two ACL paths here are parallel. Each path uses the other's modality group as a reference to transform its own modality group (acting as the guided one), thus forming a fusion mode of "simultaneous mutual guidance." This fusion mode has the advantage of Sequential ACL in that the two modal groups are mutually conditional and updated synchronously within the same layer, resulting in more symmetrical interaction. Since the modality grouping of the two paths no longer has a sequential order, the effective modality grouping method is... The specific transformation formula for Parallel ACL is as follows:

[0064]

[0065]

[0066] in, This refers to the scaling function in the first branch of ACL. This refers to the translation transformation function in the first branch of ACL. This refers to the scaling function in the second branch of ACL. This is the translation transformation function in the second branch of ACL.

[0067] The affine coupling layers in the intermodal multi-view feature fusion module are stacked in two modes: serial and parallel. The serial mode enables sequential guided cross-modal interaction, while the parallel mode enables simultaneous mutual guidance cross-modal interaction. Regardless of whether the Sequential ACL mode or the Parallel ACL mode is used, a unified fused feature vector can be obtained in the end. .

[0068] In summary, the intermodal multi-view feature fusion module INN Late Feature Fusion adopts a two-layer ACL stacked design. It's important to note that the ACL used here is a classic affine coupling layer structure, not the "improved affine coupling layer" used in the aforementioned INN Early Feature Fusion module for intramodal multi-level feature fusion. Therefore, the innovation of the INN Late Feature Fusion module lies not in redefining the basic transformation formula of the ACL, but in the structural design of its stacking method, interaction path, and input organization method tailored to the multimodal feature fusion task.

[0069] Specifically, for the sequential ACL stacking method, the basic idea of ​​"two-layer ACL cascading" originates from the classic normalized flow model. However, this invention introduces an additional swap operation between the two ACL layers, where the outputs of the first layer are swapped and used as the input of the second layer. This is a structural improvement proposed to adapt to practical problems. Compared to simple sequential stacking, this design avoids the problem of one feature always unidirectionally guiding and modulating another feature, allowing the two features to modulate each other sequentially in the two-layer transformation, thereby achieving more comprehensive cross-modal interaction.

[0070] The parallel ACL stacking method represents a fundamental innovation in the structural design of this invention. Considering that the serial method requires an exchange operation to allow the two features to guide each other sequentially, this invention further designs a parallel interaction mode, enabling the two features to achieve a fusion mechanism of "simultaneous mutual guidance" at the same level. Corresponding to the classic ACL transformation formula, this parallel method can be understood as two sets of features taking each other as conditions, retaining only the transformation part used for conditional modulation in the classic ACL, instead of adopting the identity transfer form where "a part of the input directly equals the corresponding output," thus forming a parallel fusion structure suitable for bidirectional synchronous interaction between modalities.

[0071] Furthermore, since the fusion object of this module is the global feature vectors of four modalities, and ACL itself is a dual-input dual-output structure, this invention further designs different modal grouping methods to construct combinations of different modalities interacting through ACL. This design is not a modification of the underlying formula of ACL, but rather a method design for organizing the application of the dual-input coupling mechanism in a four-modal scenario to meet the actual needs of multimodal fusion. It can be used to characterize and compare the impact of different modal pairing interactions on the final classification task.

[0072] Therefore, from an overall perspective, the macro-level innovation of the INN Late Feature Fusion module does not lie in proposing a new ACL basic unit, but in combining the core structure of INN (i.e., the affine coupling layer) with the multimodal feature fusion task. It designs serial, parallel, and modal grouping mechanisms around the requirements of cross-modal feature interaction, thus forming a new feature-level fusion algorithm. Among these, the exchange operation, parallel interaction method, and modal grouping strategy can be regarded as detailed innovations at the implementation level of this module.

[0073] S205: Input the fused feature vector into the classification head for prediction to obtain the molecular phenotype prediction results of brain tumors.

[0074] During the classification stage, The data is fed into the relevant classification module for molecular phenotypic classification and prediction of brain tumors. Specifically, it will... Inputting a classification head consisting of two fully connected (FC) layers yields a logits vector. The predicted probability distribution is obtained by Softmax normalization. ,Right now .

[0075] In this invention, cross-entropy loss is used to measure the difference between the predicted result and the true label. Let the true label after one-hot encoding be... Therefore, for a single sample, the form of its cross-entropy loss is:

[0076] in, For cross-entropy loss, The number of categories for the classification task. For the first Classification task, For real labels, To predict the probability distribution.

[0077] During training, the model will The model parameters are backpropagated using gradients as the optimization objective, and stochastic gradient descent-like optimization algorithms are used to iteratively update the network parameters, thereby gradually improving the model's discriminative performance on molecular phenotypic classification tasks. During the inference phase, only the forward propagation process is executed and the prediction results are output; there is no need to calculate the loss and gradients.

[0078] To verify the effectiveness of the method of the present invention, the following experimental results are provided: The datasets used in the experiment included BraTS2021 and UCSF-PDGM. BraTS2021 is an internationally widely used multimodal MRI benchmark dataset for gliomas, consisting of preoperative scans of real glioma patients collected from multiple medical institutions. The UCSF-PDGM dataset, constructed by the University of California, San Francisco, collected glioma cases with complete molecular pathological annotations, also derived from image data of real clinical patients. In the experiment, classification accuracy (Accuracy, denoted as Acc) and AUC (Area Under the Receiver Operating Characteristic Curve) were used as evaluation metrics to comprehensively measure the model's classification performance and its discriminative ability at different thresholds. Table 1 presents the detailed experimental results of the proposed method and the comparative methods on various tasks. All data in the table are averages obtained under five-fold cross-validation conditions.

[0079] Table 1. Classification results of the proposed method on two public datasets.

[0080] Table 1 presents the complete experimental results of the proposed model on three-class and two-class classification tasks. As mentioned above, in the INN-based Late Feature Fusion module, the input features need to be divided into two groups to adapt to the dual-input structure of the affine coupling layer. Therefore, the features of the four modalities need to be grouped in pairs first, and then fed into the ACL in serial or parallel stacking. A total of nine different modality grouping strategies were tested, and the last row of Table 1 shows the average results of the nine modality grouping strategies.

[0081] Example 2 This embodiment discloses a molecular phenotypic classification system for brain tumors based on multimodal magnetic resonance imaging, including: The data acquisition module is configured to: acquire multimodal three-dimensional magnetic resonance images, preprocess them, and obtain preprocessed multimodal three-dimensional magnetic resonance images; The model classification module is configured to input the preprocessed multimodal 3D magnetic resonance images into the molecular phenotype classification model for processing, and obtain classification prediction results. The molecular phenotypic classification model includes multiple parallel modality-specific branches, a multi-view feature fusion module between modalities, and a classification head. Its construction process includes: The preprocessed multimodal 3D magnetic resonance images are input into specific branches of the corresponding modes for mode-specific feature extraction and fusion to obtain global feature vectors for each mode; The global feature vectors of each modality are input into the intermodal multi-view feature fusion module, and cross-modal nonlinear interaction and information redistribution are performed based on a two-layer affine coupling layer to obtain the fused feature vector. The fused feature vector is input into the classification head for prediction to obtain the classification prediction result.

[0082] Example 3 The purpose of this embodiment is to provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method of Embodiment 1.

[0083] Example 4 The purpose of this embodiment is to provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the method of Embodiment 1.

[0084] The steps and methods involved in the apparatuses of Embodiments 3 and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0085] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0086] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0087] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for molecular phenotypic classification of brain tumors based on multimodal magnetic resonance imaging, characterized in that, include: Acquire multimodal three-dimensional magnetic resonance images, preprocess them, and obtain preprocessed multimodal three-dimensional magnetic resonance images; The preprocessed multimodal three-dimensional magnetic resonance images are input into a molecular phenotype classification model for processing to obtain molecular phenotype prediction results for brain tumors. The molecular phenotypic classification model includes multiple parallel modality-specific branches, a multi-view feature fusion module between modalities, and a classification head. Its construction process includes: The preprocessed multimodal 3D magnetic resonance images are input into specific branches of the corresponding modes for mode-specific feature extraction and fusion to obtain global feature vectors for each mode; The global feature vectors of each modality are input into the intermodal multi-view feature fusion module, and cross-modal nonlinear interaction and information redistribution are performed based on a two-layer affine coupling layer to obtain the fused feature vector. The fused feature vector is input into the classification head for prediction to obtain the molecular phenotype prediction result of brain tumor.

2. The method for molecular phenotypic classification of brain tumors based on multimodal magnetic resonance imaging as described in claim 1, characterized in that, The modality-specific branch includes, in sequence, a feature extraction network, a feature alignment module, an intramodal multi-level feature fusion module, and a global modeling network.

3. The method for molecular phenotypic classification of brain tumors based on multimodal magnetic resonance imaging as described in claim 2, characterized in that, The process of constructing the modality-specific branch includes: The feature extraction network performs preliminary feature extraction on the three-dimensional magnetic resonance image to obtain multi-level features; the multi-level features are then input into the feature alignment module to align them in terms of channel number and spatial resolution to obtain aligned multi-level features. The shallow features in the aligned multi-level features are concatenated to obtain shallow concatenated features, and the deep features are concatenated to obtain deep concatenated features. The shallow and deep concatenated features are input into the intramodal multi-level feature fusion module, and the two are fused and interacted using an improved affine coupling layer to obtain comprehensive features. The comprehensive features are flattened and linearly mapped before being input into a global modeling network for global modeling and feature enhancement. Modal-level global feature vectors are constructed based on a multi-head self-attention mechanism.

4. The method for molecular phenotypic classification of brain tumors based on multimodal magnetic resonance imaging as described in claim 1, characterized in that, The affine coupling layer in the intermodal multi-view feature fusion module is stacked in two ways: a serial mode and a parallel mode. The serial mode performs sequentially guided cross-modal interaction, while the parallel mode performs simultaneous, mutually guided cross-modal interaction.

5. The method for molecular phenotypic classification of brain tumors based on multimodal magnetic resonance imaging as described in claim 4, characterized in that, The transformation formula for the two-layer affine coupling layer in the cascaded mode is: in, This is the first set of global feature vectors. This is the second set of global feature vectors. For Hadama accumulation, This is the scaling function in the first affine coupling layer. This is the translation transformation function in the first affine coupling layer. This is the scaling function in the second affine coupling layer. This is the translation transformation function in the second affine coupling layer. To fuse feature vectors.

6. The method for molecular phenotypic classification of brain tumors based on multimodal magnetic resonance imaging as described in claim 4, characterized in that, The transformation formula for the two-layer affine coupling layer in the parallel mode is: in, This is the first set of global feature vectors. This is the second set of global feature vectors. For Hadama accumulation, This is the scaling function in the first branch affine coupling layer. Let be the translation transformation function in the first branch affine coupling layer. This is the scaling function in the second branch affine coupling layer. This is the translation transformation function in the second branch affine coupling layer.

7. The method for molecular phenotypic classification of brain tumors based on multimodal magnetic resonance imaging as described in claim 1, characterized in that, Before fusing the global feature vectors of each modality, the global feature vectors of each modality are grouped according to a variety of preset grouping methods, and the global feature vectors within the same group are concatenated as input.

8. A molecular phenotypic classification system for brain tumors based on multimodal magnetic resonance imaging, characterized in that, include: The data acquisition module is configured to: acquire multimodal three-dimensional magnetic resonance images, preprocess them, and obtain preprocessed multimodal three-dimensional magnetic resonance images; The model classification module is configured to input the preprocessed multimodal three-dimensional magnetic resonance images into the molecular phenotype classification model for processing, and obtain the molecular phenotype prediction results of brain tumors. The molecular phenotypic classification model includes multiple parallel modality-specific branches, a multi-view feature fusion module between modalities, and a classification head. Its construction process includes: The preprocessed multimodal 3D magnetic resonance images are input into specific branches of the corresponding modes for mode-specific feature extraction and fusion to obtain global feature vectors for each mode; The global feature vectors of each modality are input into the intermodal multi-view feature fusion module, and cross-modal nonlinear interaction and information redistribution are performed based on a two-layer affine coupling layer to obtain the fused feature vector. The fused feature vector is input into the classification head for prediction to obtain the molecular phenotype prediction result of brain tumor.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps in a method for molecular phenotypic classification of brain tumors based on multimodal magnetic resonance as described in any one of claims 1-7.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the method for molecular phenotypic classification of brain tumors based on multimodal magnetic resonance as described in any one of claims 1-7.

Images