Multi-sequence brain MRI (Magnetic Resonance Imaging) anomaly detection method under arbitrary mode loss

Abstract

The invention discloses a multi-sequence brain MRI (Magnetic Resonance Imaging) anomaly detection method under any mode loss. The method comprises the following steps: constructing an AnyAD neural network model comprising a dual-path pre-training encoder, a feature distribution alignment module, a bottleneck layer, an internal normal prototype extraction module and a prototype guide decoder; inputting a multi-sequence brain MRI image containing a full-modal image and a missing-modal image into a dual-path pre-training encoder to generate a multi-scale encoder feature; the feature distribution alignment module is used for constraining the features of the encoder with the missing mode to the features of the encoder with the complete mode; using an intrinsic normal prototype extraction module to extract an intrinsic normal prototype according to the encoder features; the encoder features are fused through a bottleneck layer and then input into a prototype guide decoder together with an internal normal prototype for feature reconstruction, and decoder features are generated; the encoder features are compared with the decoder features, a reconstruction error is calculated, and an anomaly score map for anomaly detection is generated.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence in medical imaging, and in particular to a method for detecting abnormalities in multi-sequence brain MRI under arbitrary modality loss. Background Technology

[0002] Multi-sequence brain MRI (including T1, T2, FLAIR, T1Gd, etc.) is the core imaging method for diagnosing brain tumors. Different sequences carry complementary diagnostic information, and clinical diagnosis usually requires the integration of multi-sequence information to make a reliable judgment.

[0003] Existing methods for detecting abnormalities in brain MRI mainly fall into two categories: single-class and multi-class anomaly detection. Single-class anomaly detection trains an independent model for each modality combination, which can be optimized for specific inputs, but requires storing complete model parameters for each combination, resulting in high storage and maintenance costs. Multi-class anomaly detection uses a unified network architecture to detect multiple modalities simultaneously, requiring only one set of shared weights, significantly reducing storage requirements. However, most methods assume that the input modalities are fixed, requiring the use of the same modality combinations during training and inference. When encountering unseen modality combinations, existing methods typically require retraining or generating pseudo-images of missing modalities through modality interpolation techniques, introducing additional uncertainty. Furthermore, existing methods are quite sensitive to the information content of the input modalities, with detection performance significantly degrading under single-modality or low-information combinations.

[0004] The existing technology has the following core defects: (1) It cannot adapt to arbitrary modal combination inputs, lacks a unified framework to work stably under different modal combinations, and requires separate training or adjustment of the model for each combination; (2) It has poor robustness to low-information modalities, and the detection performance drops significantly and the false positive rate increases under single modalities or combinations with insufficient information; (3) It has insufficient generalization ability across datasets and institutions, and is sensitive to distribution shifts caused by different acquisition conditions and equipment differences. In real clinical scenarios, random modal loss is common due to factors such as patient contraindications, equipment failures, and scanning interruptions. Existing methods are difficult to meet the multiple challenges of incomplete modalities and heterogeneous data, which limits clinical scalability and practicality. Summary of the Invention

[0005] In view of the above-mentioned deficiencies of the prior art, the present invention provides a method for detecting abnormalities in multi-sequence brain MRI under arbitrary modality loss. It addresses the common problems of modality loss and scarcity of abnormal samples in multi-sequence brain MRI scenarios. Under the condition of training only with normal samples, it realizes the detection and localization of abnormalities in brain MRI images with arbitrary modality combinations through feature distribution alignment and prototype-guided reconstruction. It eliminates the need to train models separately for each modality combination, significantly reduces storage and deployment costs, and improves clinical applicability and scalability.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A method for detecting abnormalities in multi-sequence brain MRI under arbitrary modality loss includes the following steps:

[0008] S1. Acquire and preprocess multi-sequence brain MRI images to construct a dataset;

[0009] S2. Construct an AnyAD neural network model comprising a dual-path pre-trained encoder, a feature distribution alignment module, a bottleneck layer, an intrinsic normal prototype extraction module, and a prototype-guided decoder. Input multi-sequence brain MRI images containing full-modality and missing-modality images into the dual-path pre-trained encoder to generate multi-scale encoder features. The feature distribution alignment module is used to constrain the encoder features of the missing modality to the encoder features of the full-modality. The intrinsic normal prototype extraction module extracts the intrinsic normal prototype based on the encoder features. The encoder features are fused by the bottleneck layer and then input together with the intrinsic normal prototype into the prototype-guided decoder for feature reconstruction to generate decoder features. Compare the encoder features and decoder features, calculate the reconstruction error, and generate an anomaly scoring map for anomaly detection.

[0010] S3. Train the model using the training set in the dataset;

[0011] S4. Test and evaluate the trained AnyAD neural network model.

[0012] Preferably, the preprocessing in S1 includes:

[0013] Three-dimensional multi-sequence brain MRI images containing T1, T2, FLAIR, and T1Gd sequences are sliced ​​axially into two-dimensional image slices. Each two-dimensional image slice is labeled as a normal sample without lesions and an abnormal sample with lesions according to the corresponding segmentation mask. Normal samples are divided into training and testing sets proportionally. Abnormal samples are used only for testing and evaluation. z-score standardization is performed on all two-dimensional image slices and their sizes are uniformly adjusted. Modal combinations with missing modalities are defined.

[0014] Preferably, S2 includes constructing a dual-path pre-trained encoder:

[0015] Multi-sequence brain MRI images are divided into several image blocks according to a preset block size; each image block is encoded into an image block label sequence through an embedding layer, and positional encoding is added; after multi-layer Transformer encoding, a multi-layer feature map is output.

[0016] The multi-layer feature map is divided into a deep feature group and a shallow feature group according to the preset feature hierarchy. Feature fusion operation is performed on the deep feature group and the shallow feature group respectively to generate multi-scale encoder features, including a first encoding fusion feature for capturing shallow texture and local structural information and a second encoding fusion feature that focuses on deep semantics and global pattern representation.

[0017] Preferably, S2 includes a feature distribution alignment module:

[0018] Extract the first and second coding fusion features of the full-modal image, calculate the mean and variance of each channel and cache them as a reference distribution;

[0019] Randomly occlude some modalities to generate missing modality images; extract the first and second coding fusion features of the missing modality images, and calculate the mean and variance of each channel;

[0020] The distribution of missing modal features to complete modal features is constrained by minimizing the following distribution alignment loss:

[0021]

[0022] Where, μ curr ,μ full These represent the mean values ​​of the second encoded fusion features of the missing modality image and the full modality image, respectively. The variances of the second encoded fusion features represent the variances of the missing modality image and the full modality image, respectively.

[0023] Preferably, in the bottleneck layer, the first and second encoded fusion features are added element-wise to generate a fusion feature map; the bottleneck features are generated by up-projection and down-compression of the channel dimension of the fusion feature map through two fully connected layers.

[0024] Preferably, S2 includes a module for constructing an intrinsic normal prototype extraction module:

[0025] Define a random prototype vector and initialize it using a normal distribution;

[0026] The random prototype vector and the second encoded fusion feature are used to generate queries, keys, and values ​​through linear mapping, respectively:

[0027] The attention output is calculated using the standard scaled dot product attention mechanism and then added to the random prototype to generate the attention prototype.

[0028] The attention prototype is further updated by two layers of feedforward neural network to generate the intrinsic normal prototype.

[0029] Preferably, the prototype consistency loss L is constructed by minimizing the cosine distance between each image feature and its intrinsic normal prototype. con :

[0030]

[0031] Among them, Q En1 (i) is the feature vector at the i-th spatial location of the second encoded fusion feature En1, INPs n For the nth intrinsic normal prototype, the cosine distance d i The minimum cosine distance between the representative feature and the prototype, d i ∈[0,2].

[0032] Preferably, S2 includes constructing a prototype bootloader decoder with multiple prototype bootloader decoding blocks:

[0033] Using bottleneck features as initial input, in each prototype-guided decoding block, the upsampled decoding features are used as queries, and the inherent normal prototypes are used as keys and values. Feature reconstruction is performed through an attention mechanism.

[0034] After layer-by-layer decoding and upsampling to the original spatial resolution, decoder features are generated, including a first decoding fusion feature corresponding to the first coding fusion feature and a second decoding fusion feature corresponding to the second coding fusion feature.

[0035] As a preferred option, S3 includes: constructing the total loss function:

[0036]

[0037] Where: L rec L represents the adaptive reconstruction loss, and L represents the cosine distance between the output features of the dual-path pre-trained encoder and the reconstructed features of the prototype-guided decoder. con For prototype consistency loss, L dist Let λ1 and λ2 be the distribution alignment loss, and let λ1 and λ2 be the weight hyperparameters.

[0038] Compared with the prior art, the beneficial effects of the present invention are reflected in:

[0039] 1. Unlike traditional techniques that require training independent models for each modality or combination of modalities and usually assume that the input modality is fixed, this invention adopts a unified anomaly detection technique of "feature distribution alignment + prototype-guided reconstruction + random modality masking training". This enables the detection and localization of anomalies in various modalities by supporting brain MRI inputs with any combination of modalities, under the premise of using only normal samples for a single training.

[0040] 2. Unlike traditional technologies that are highly sensitive to the amount of input modal information and data distribution, and whose performance degrades significantly when key modalities are missing or when applied across institutions, this invention adopts a robust generalization technology that combines "intrinsic normal prototype representation with feature distribution alignment constraints". This enables the model to maintain relatively stable anomaly detection and localization capabilities in complex scenarios such as single modality, low-information modality combinations, and cross-dataset and cross-institutional migrations. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the overall structure of the AnyAD neural network in Embodiment 1 of the present invention;

[0042] Figure 2 This is a schematic diagram of the prototype guide decoder structure of Embodiment 1 of the present invention;

[0043] Figure 3 This is a visualization diagram of the anomaly detection results of each model under different modality combinations on the BraTs2018 dataset in Embodiment 1 of the present invention. Detailed Implementation

[0044] To make the technical means, inventive features, objectives, and effects of the invention readily understandable, the invention is further described below with reference to specific illustrations. However, the invention is not limited to the embodiments described below.

[0045] It should be noted that the structures, proportions, sizes, etc., illustrated in the accompanying drawings of this specification are only used to complement the content disclosed in the specification for those skilled in the art to understand and read, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0046] Unlike traditional techniques that require training independent models for each modality or modality combination, have fixed input modalities, and struggle to handle missing random modalities, this invention employs a unified anomaly detection technology solution (the main invention point) of "feature distribution alignment + prototype-guided reconstruction + random modality masking training." This enables the detection and localization of brain MRI anomalies with any combination of modalities (single-modality, bimodality, trimodality, etc., a total of 7 combinations) under the premise of training only with normal samples in a single session. This achieves true "single-training, arbitrary modality inference" and significantly reduces the storage and deployment costs of the model in multimodal clinical environments. Experiments show that on the BraTS2018 dataset, the method of this invention achieves an average AUROC (image-level / pixel-level) of 0.9473 / 0.9922 under 7 modality combinations, which is 0.97 / 0.33 percentage points higher than the second-best method INP-Former, respectively. Moreover, it only requires one set of shared model weights, and the storage efficiency is about 7 times higher than the traditional single-class method. This proves that the present invention has significant performance and efficiency advantages in multimodal and arbitrary modality missing scenarios, and has outstanding practicality and inventiveness.

[0047] Unlike traditional techniques that rely directly on the amount of input modal information and suffer significant performance degradation when key modalities are missing or when migrating across datasets, this invention employs a robust generalization technique (a secondary inventive point) of "Intrinsic Normal Prototypes (INPs) Representation + Feature Distribution Alignment Constraints." This allows the model to maintain stable anomaly detection and localization performance even in low-information modalities, zero-sample sequences, and cross-dataset scenarios. Specifically, this invention learns shared intrinsic normal prototypes across samples from normal samples and performs statistical distribution alignment between missing modal branches and full-modal branches in the feature space. This guides the decoder to reconstruct only normal anatomical patterns, thereby amplifying the reconstruction error of anomaly regions when key modalities are missing. In low-information modalities (such as single-modal T1 and T2), the image-level AP of AnyAD achieved by this invention is 0.9429, an improvement of 27.09 percentage points compared to the traditional RD4AD method; the pixel-level F1 score is 0.6715, an improvement of 22.76 percentage points compared to skipTS. In terms of cross-dataset generalization, in zero-shot scenarios when migrating from BraTS2018 to the Pretreat-MetsToBrain-Masks dataset, the image-level / pixel-level AUROC still reaches 0.8606 / 0.9838, with a performance decrease of only 2.68 / 0.46 percentage points, far superior to the significant degradation of traditional methods. In the anomaly localization task, the pixel-level AP reaches 0.7883, an improvement of 5.87 percentage points compared to ReContrast. These results demonstrate that the present invention exhibits significantly better robustness and generalization ability than existing technologies in complex clinical scenarios such as low-information modalities, cross-device / cross-institutional scenarios, and unseen modal sequences, further reflecting the inventiveness and outstanding progress of the present invention.

[0048] Example 1:

[0049] like Figure 1 The method for detecting abnormalities in multi-sequence brain MRI under arbitrary modality loss includes the following steps:

[0050] S1. Acquire and preprocess multi-sequence brain MRI images to construct a dataset;

[0051] The preprocessing process includes:

[0052] Data slicing: Three-dimensional MRI data containing four modalities—T1 (T1-weighted), T2 (T2-weighted), FLAIR (Fluid-Attenuated Inversion Recovery), and T1Gd (T1-weighted with Gadolinium Contrast)—were sliced ​​axially into two-dimensional images. Layers with good signal quality were selected for systematic sampling, avoiding artifact layers at the head and tail. T1 sequences clearly display the anatomical structures of the brain, providing a basic reference for grayscale contrast between normal and abnormal tissues. T2 and FLAIR sequences are highly sensitive to edema, infiltration, and cerebrospinal fluid signals, highlighting edematous areas surrounding the tumor and non-enhanced tumor parenchyma. Contrast-enhanced T1Gd sequences, through the use of contrast agents, can significantly show blood-brain barrier disruption and enhanced areas of the tumor, possessing significant clinical value for glioma grading, postoperative recurrence, and metastasis assessment.

[0053] Normal / Abnormal Sample Labeling: Slices are labeled as normal or abnormal samples based on the corresponding segmentation mask. The mask for normal samples is all zeros, while the mask for abnormal samples contains non-zero pixels.

[0054] Training / Test Split: Normal image slices without lesions are divided into training and test sets in an 8:2 ratio; abnormal image slices with lesions and their segmentation masks are used only for test evaluation to ensure that only normal samples are used during the training phase.

[0055] Image normalization: Perform z-score normalization on all slices and adjust their size uniformly.

[0056] Modality Combination Definition: Seven modality combinations are defined to cover common modality loss scenarios in clinical practice:

[0057] Combo 1: FLAIR

[0058] Combo 2: T1

[0059] Combo 3: T2

[0060] Combo 4: T1 + FLAIR

[0061] Combo 5: T2 + FLAIR

[0062] Combo 6: T1 + T2

[0063] Combo 7: T1+T2+FLAIR

[0064] S2. Construct the AnyAD neural network model

[0065] The AnyAD model adopts a unified encoder-decoder architecture, and its core components include: a dual-path pre-trained encoder, a feature distribution alignment module, a bottleneck layer, an Intrinsic Normal Prototype Extractor (INPs Extractor), and an Intrinsic Prototype Guided Decoder (INP-guided Decoder).

[0066] S21. Construct a dual-path pre-trained encoder

[0067] The dual-path pre-trained encoder takes as input multiple sequences of brain MRI images with arbitrary modal combinations and outputs multi-scale encoded fusion features En0 and En1, which are used for anomaly comparison and subsequent reconstruction.

[0068] The encoder is based on a visual Transformer (ViT_base) architecture, loads DINOv2 pre-trained weights, and performs multi-scale feature extraction on the input multi-sequence MRI images. The specific process is as follows:

[0069] Image block segmentation and embedding: The input image is divided into image blocks according to a preset block size; each image block is encoded into an image block label sequence through a linear embedding layer, and positional encoding is added.

[0070] Multi-layer Transformer encoding: After processing by 16 layers of Transformer encoding blocks, multi-layer feature maps are output, with each feature map having a size of B×N×D.

[0071] Dual-path feature fusion: The 8-layer feature map is divided into two groups according to depth:

[0072] Shallow feature group F shallow Select features from layers 2 to 5, focusing on shallow textures and local structures;

[0073] Deep feature group F deep We selected features from layers 6 to 9, focusing on deep semantics and global patterns.

[0074] Generate fused features: Upsample the two sets of features respectively (restore to the original spatial resolution H×W through bilinear interpolation) and sum them element-wise to generate:

[0075] The first encoded fusion feature En0 is obtained by fusing shallow feature groups. Its size is B×768×32×32. It is used for subsequent anomaly comparison with the decoder output.

[0076] The second encoded fusion feature En1 is obtained by fusing deep feature groups. Its size is B×768×32×32. It is input to the bottleneck layer and used as the input feature for the extraction of the intrinsic normal prototype.

[0077] S22. Constructing a feature distribution alignment module

[0078] The input is the second encoded fusion feature En1 corresponding to the full-modal branch and the missing modal branch. full and En1 miss The output is the distribution alignment loss L used to constrain the distribution of missing modal features to the distribution of complete modal features. dist 】

[0079] This module aligns missing modal features to the full modal feature space through statistical matching, specifically including three steps:

[0080] S221. Pre-calculate the full-modal reference distribution: Before training begins, perform forward propagation on the full-modal input image, extract En0 and En1 features, and calculate the mean of each channel.

[0081]

[0082] Where, μ c , These represent the mean and variance of the channel, respectively. It is a feature map tensor of dimension (B, C, H, W), where B is the batch size (number of samples), H and W are the height and width of the feature map, and BHW is the normalization factor. Let μ be the variance of the c-th channel. The mean μ of the second encoded feature En1 of the full modal path is then used for fusion. full The variance of the second encoding fusion feature En1 of the full modal path Stored as a fixed reference distribution.

[0083] In this embodiment, the full-modal image includes complete input in three modalities: T1, T2, and FLAIR. The network structure in this embodiment also uses these three channels as input. T1ce modality usually requires the use of contrast agents for imaging, which may have some impact on the patient's health. Therefore, this embodiment does not include the T1ce modality, but focuses on the FLAIR, T1, and T2 modalities.

[0084] S222, Random Modal Masking During Training: Randomly mask some modalities in the training data to generate missing modal input branches. Fill the channels of the missing modalities with zero values.

[0085] S223. Calculate the distribution alignment loss: Extract features En0 and En1 from the missing modality input, calculate the mean and variance of the current branch in the channel dimension, and define the distribution alignment loss as:

[0086]

[0087] Where MSE(·) represents the mean squared error loss function, μ curr ,μ full The mean of the second encoded fusion feature En1, representing the missing modal path and the full modal path, respectively. The variances of the second encoded fusion features En1 represent the variances of the missing modal path and the full modal path, respectively. This loss minimizes the statistical difference between the current missing modal feature and the full modal reference distribution through MSE, enabling the model to achieve indirect feature completion at the feature space level.

[0088] S23, Constructing the bottleneck layer

[0089] The input consists of the first encoded fusion feature En0 and the second encoded fusion feature En1, both output from a dual-path pre-trained encoder; the output is the high-dimensional fusion bottleneck feature F. bottleneck It serves as a unified input for prototype extraction and decoding reconstruction.

[0090] The bottleneck layer performs high-dimensional fusion of En0 and En1 to enhance the separability and expressive power of features:

[0091] Feature concatenation and addition: First, perform element-wise addition on En0 and En1:

[0092]

[0093] Among them, F sum This represents the fused feature map obtained by element-wise addition of En0 and En1. This represents element-wise addition. Channel expansion and compression: Upward projection and downward compression of the channel dimension are achieved through two fully connected layers.

[0094] F high =Linear 768→3072 (F sum ),F bottleneck =Linear 3072→768 (F high ),

[0095] Among them, F high This represents a higher-level feature upgrade from 768 channels to 3072 channels. This step is to separate features that are not easily distinguishable in low dimensions; F bottleneckThis represents the bottleneck characteristics after restoring from channel 3072 to channel 768. The purpose of this step is to reduce the amount of subsequent calculations and to unify the dimensions of the data before and after.

[0096] Output bottleneck characteristics Used for subsequent decoding.

[0097] S24. Construct the Intrinsic Normal Prototype Extraction Module (INPs Extractor)

[0098] The input is the second encoded fusion feature En1 corresponding to a single normal image; the output is a set of intrinsic normal prototypes (INPs) shared across samples, used to characterize typical patterns of normal brain anatomy.

[0099] This module adaptively extracts several sets of intrinsic normal prototype vectors (INPs) representing normal anatomical structures from a single normal image, as follows:

[0100] Initialize random prototypes: Define a set of learnable random prototype vectors and initialize them using a normal distribution.

[0101] Cross-attention extraction: The random prototype P0 and the second encoded fusion feature En1 are used to generate query, key, and value respectively through linear mapping:

[0102] Q P =P0W Q ,K En1 =Flatten(En1)W K V En1 =Flatten(En1)W V

[0103] Flatten(·) flattens the spatial dimensions to (B×H×W)×D. Q is a learnable weight matrix. P This means that the product of a randomly initialized prototype and a weight matrix is ​​used as the query for subsequent attention calculations; similarly, K En1 and V En1 These represent multiplying the flattened second encoded fusion feature En1 with the corresponding weight matrix, and using the result as the key and value in the attention calculation.

[0104] Calculate the attention prototype: Calculate the attention output using the standard scaled dot product attention mechanism and add it to the initial prototype.

[0105]

[0106] Among them, Attn(Q P ,K En1 V En1) indicates that attention is calculated for Q, K, and V.

[0107] Feedforward Network Update: The attention prototype P′ is further updated via a two-layer feedforward neural network (FFN), outputting the final set of intrinsic normal prototypes INPs.

[0108] INPs=FFN(P′)+P′,FFN(x)=ReLU(xW1+b1)W2+b2

[0109] Where W1 and W2 are learnable weights, b1 and b2 are biases, and FFN(·) represents forward propagation.

[0110] Prototype Consistency Loss: To ensure that INPs consistently represent normal features during training (rather than capturing generic or anomalous features), a prototype consistency loss L is introduced. con This is achieved by minimizing the cosine distance between each image feature and its nearest prototype:

[0111]

[0112] Q En1 (i) is the feature vector at the i-th spatial location of the second encoded fusion feature En1, INPs n For the nth prototype, the cosine distance d i ∈[0,2],d i The minimum cosine distance between the representative feature and the prototype is used to ensure that the extracted prototype always has the closest normal features.

[0113] S25, Building the Prototype Bootstrap Decoder

[0114] The input is the bottleneck feature F bottleneck And the intrinsic normal prototype set INPs; the outputs are the first and second decoding fusion features De0 and De1, which are aligned with the first coding fusion feature En0 and the second coding fusion feature En1, respectively, to calculate the reconstruction error and generate anomaly score maps.

[0115] like Figure 2 As shown, in this embodiment, the decoder consists of four prototype-guided decoding blocks. During the reconstruction process, it relies solely on the inherent normal prototype INPs to reconstruct the input features. The process is as follows:

[0116] Prototype-guided attention decoding: In each decoding block, the upsampled decoding features are... As a query, INPs are used as keys and values, and features are reconstructed through an attention mechanism:

[0117]

[0118] in, This represents the decoding feature of the previous layer, l∈{1,2,3,4}. This is the initial input; This represents decoding the feature F from the previous layer. l-1 D With weight matrix W l Q The result of the product is used as the query result; similarly, K l INPs and V l INPs These represent multiplying the intrinsic normal prototype INPs with the corresponding weight matrix, and using the results as keys and values ​​in the attention calculation; Represent The corresponding learnable weight matrix.

[0119] Output decoding characteristics: After 4 layers of decoding and upsampling to the original spatial resolution, the output is:

[0120] The first decoded fusion feature De0 corresponds to En0 and has a size of B×768×32×32.

[0121] The second decoding fusion feature De1 corresponds to En1 and has a size of B×768×32×32.

[0122] Reconstructing only the normal pattern: Since the decoder only relies on normal prototype INPs for reconstruction, the decoder only reconstructs the normal anatomical pattern, which causes a large error in the reconstruction of abnormal regions. This amplifies the reconstruction error of abnormal regions in the inference stage to achieve anomaly detection and localization.

[0123] S3, Model Training

[0124] Implemented on an NVIDIA GeForce RTX 4090 GPU using the PyTorch 2.0 framework, employing the following training strategy:

[0125] S31, Random Modal Masking Training

[0126] For each training batch, a mode combination is randomly selected (uniformly sampled from Combo 1 to 7), and the input channels of the missing modes are filled with zero values ​​to generate the missing mode input branch. At the same time, a full mode input branch is reserved for calculating the reference distribution.

[0127] S32. Joint Loss Function Optimization

[0128] Total loss is defined as the weighted sum of three terms:

[0129]

[0130] in:

[0131] Adaptive reconstruction loss L rec Calculate the cosine distance between the encoder output features and the decoder reconstructed features, and then perform adaptive weighting by combining the hard sample weights;

[0132] Prototype consistency loss L con : Ensure that the intrinsic normal prototype always represents normality;

[0133] Distribution alignment loss L dist The statistical distribution of the features of the mode with missing constraints approximates the full-mode reference distribution.

[0134] Weight hyperparameters: λ1 = 0.2, λ2 = 0.2.

[0135] S33, Training Hyperparameter Settings

[0136] Optimizer: Adam optimizer, learning rate initialized to lr = 1 × 10⁻⁶ -4 It decays to 0.5 every 50 rounds, eventually dropping to 1×10 -6 ;

[0137] Batch size: B = 16;

[0138] Number of training rounds: 200 rounds;

[0139] Data augmentation: random horizontal flip (probability 0.5), random rotation (±10°±10°), random brightness adjustment (±10%±10%);

[0140] S4, Model Testing and Performance Evaluation

[0141] The model's performance under seven modal combinations was comprehensively evaluated using seven metrics:

[0142] S41, Evaluation Indicators

[0143] Image-level metrics:

[0144] (1) Image-level AUROC: measures the model’s ability to distinguish between normal and abnormal images. It is obtained by calculating the abnormality score of each image (taking the maximum or mean value of the reconstruction error map) and plotting the ROC curve. The value range is [0,1]. The closer it is to 1, the better the classification performance.

[0145] (2) Image-level AP (Average Precision): Calculates the area under the image-level Precision-Recall curve, focuses more on the detection accuracy of anomalous samples, and is more sensitive to imbalanced datasets;

[0146] (3) Image-level F1: Taking into account the harmonic mean of precision and recall, the F1 score is maximized by selecting the optimal threshold, which is suitable for scenarios that need to balance false positives and false negatives.

[0147] Pixel-level metrics:

[0148] (4) Pixel-level AUROC: measures the model’s ability to locate abnormal regions at the pixel level. Each pixel of the reconstruction error map is compared with the real segmentation mask to evaluate the model’s ability to accurately locate the lesion boundary.

[0149] (5) Pixel-level AP: Calculate the area under the pixel-level Precision-Recall curve, pay more attention to the detection accuracy of positive samples (abnormal pixels), and are more sensitive to the detection of small lesions;

[0150] (6) Pixel-level F1: The harmonic mean of precision and recall is calculated at the pixel level. The F1 score is maximized by adaptive threshold selection, reflecting the overall performance of lesion segmentation.

[0151] (7) Pixel-level AUPRO (Area Under Per-Region Overlap curve): Calculate the PRO curve for each abnormal connectivity region (by the region overlap rate under different thresholds) and take the area under the curve, which better meets the clinical need to focus on independent lesions and is more robust to the detection of multiple small lesions.

[0152] S42, Test Results

[0153] 1. Testing within the distribution domain (BraTS2018 training, BraTS2018 testing)

[0154] like Figure 3 As shown, in scenarios where the training and test sets come from the same data source, the average performance of the model across the seven modality combinations is:

[0155]

[0156] The results show that the model has excellent detection and localization performance on identically distributed data, with image-level and pixel-level AUROC exceeding 0.94 and 0.99, respectively, proving that the model can effectively learn normal brain anatomical structure patterns and accurately identify abnormal regions through reconstruction errors.

[0157] 2. Out-of-domain testing (BraTS2018 training, Pretreat-MetsToBrain-Masks testing)

[0158] In scenarios where the training and test sets come from different data sources and disease types (migrating from a glioma dataset to a brain metastasis dataset), the model performance is as follows:

[0159]

[0160] Despite significant differences in disease types, imaging devices, and image features between the test data and the training data, the model maintained high detection performance, achieving an image-level AUROC of 0.86 and a pixel-level AUROC of 0.98. This demonstrates that the model has strong generalization ability to unseen data distributions and that the learned intrinsic normal prototypes have cross-dataset universality.

[0161] 3. Zero-shot anomaly detection (BraTS2018 training, T1ce sequence testing)

[0162] In scenarios where T1ce (contrast-enhanced T1) sequences were not encountered during training and the modality was only used during testing, the model performance was as follows:

[0163]

[0164] The results show that even for completely unseen MRI sequence types, the model can still achieve effective anomaly detection through the general visual features extracted by the DINOv2 pre-trained encoder and the inherent normal prototype mechanism. The image-level AUROC reaches 0.86 and the pixel-level AUROC reaches 0.97, which verifies the model's zero-shot learning ability and cross-modal generalization ability.

[0165] (d3) Ablation test

[0166] To verify the contribution of each key module to the model performance, an ablation study was conducted on the BraTS2018 dataset, focusing on evaluating the distribution alignment loss L. dist The influence of the location of action and the hyperparameter λ2.

[0167] 1. Ablation experiments of distributed alignment loss Ldist.

[0168] Results on the BraTS2018 dataset (Combo1-FLAIR unimodal):

[0169]

[0170] Experimental conclusion:

[0171] After removing the distribution alignment loss, BraTS2018's out-of-domain generalization performance significantly decreased: image-level AUROC dropped from 0.9472 to 0.8529 (a decrease of 9.43 percentage points), and pixel-level AP dropped from 0.7898 to 0.5304.

[0172] When L distWhen applied to higher-level features En1, the performance is better than that of En0, which verifies that distribution alignment in higher-level feature spaces is more conducive to cross-modal consistency learning.

[0173] The distribution alignment loss L was verified. dist It is a key mechanism for achieving cross-domain robustness and zero-shot generalization.

[0174] 1. Experiment on the rationality of hyperparameter λ2

[0175]

[0176] Experimental conclusion:

[0177] When λ² = 0.2, the model achieves optimal performance on both datasets;

[0178] When λ2 is too small, the distribution alignment constraint is insufficient, resulting in a slight decrease in performance;

[0179] When λ2 is too large, the performance drops sharply: the pixel-level AP on BraTS2018 drops from 0.7898 to 0.5566, indicating that overemphasizing distribution alignment will suppress the feature representation ability.

[0180] S5, Clinical Application Deployment

[0181] After integrating the trained AnyAD model into the clinical auxiliary diagnostic system, and inputting MRI sequences of any modality combination (at least one of T1 / T2 / FLAIR / T1Gd), the system automatically outputs the following results:

[0182] Pixel-level reconstruction error map: The size is H×W, and each pixel value represents the degree of abnormality at that location (range [0,1));

[0183] Anomaly scoring map: The reconstruction error map is normalized and noise is reduced by Gaussian smoothing;

[0184] Binarized anomaly mask: Generates a binary mask based on a preset adaptive threshold to mark suspicious anomaly areas.

Claims

1. A method for multi-sequence brain MRI anomaly detection under arbitrary modality absence, characterized in that, The method comprises the following steps: S1, acquiring and preprocessing multi-sequence brain MRI images, and constructing a data set; S2, constructing an AnyAD neural network model comprising a double-path pre-training encoder, a feature distribution alignment module, a bottleneck layer, an intrinsic normal prototype extraction module, and a prototype guided decoder; inputting the multi-sequence brain MRI images comprising full-modal images and missing-modal images into the double-path pre-training encoder to generate multi-scale encoder features; the feature distribution alignment module is used to constrain the encoder features of the missing modal to align with the encoder features of the full modal; the intrinsic normal prototype extraction module is used to extract an intrinsic normal prototype according to the encoder features; the encoder features are fused through the bottleneck layer and are input into the prototype guided decoder together with the intrinsic normal prototype for feature reconstruction to generate decoder features; comparing the encoder features with the decoder features, calculating reconstruction errors, and generating an abnormal score map; S3, using a training set in the data set for model training; S4, testing and evaluating the AnyAD neural network model after training.

2. The method for multi-sequence brain MRI anomaly detection under arbitrary modality absence according to claim 1, characterized in that, The preprocessing in S1 comprises: three-dimensional multi-sequence brain MRI images containing T1 sequences, T2 sequences, FLAIR sequences, and T1Gd sequences are sliced into two-dimensional image slices along the axial direction; according to the corresponding segmentation mask, each two-dimensional image slice is labeled as a normal sample without lesions and an abnormal sample with lesions; the normal samples are divided into a training set and a test set in proportion; the abnormal samples are only used for test evaluation; z-score standardization is performed on all two-dimensional image slices, and the size is uniformly adjusted; and a modal combination with missing modal is defined.

3. The method of claim 1, wherein, S2 comprises constructing a double-path pre-training encoder: the multi-sequence brain MRI images are divided into image blocks according to a preset block size; each image block is encoded into an image block token sequence through an embedding layer, and position encoding is added; after multi-layer Transformer coding, multi-layer feature maps are outputted; the multi-layer feature maps are divided into a deep feature group and a shallow feature group according to a preset feature level; feature fusion operations are respectively performed on the deep feature group and the shallow feature group to generate multi-scale encoder features, including first encoding fusion features for capturing shallow texture and local structure information and second encoding fusion features focusing on deep semantic and global pattern representation.

4. The method of claim 3, wherein, S2 comprises constructing a feature distribution alignment module: the first encoding fusion features and the second encoding fusion features of the full-modal images are extracted, and the mean and variance of each channel are calculated and buffered as a reference distribution; missing-modal images are generated by randomly masking part of the modal; the first encoding fusion features and the second encoding fusion features of the missing-modal images are extracted, and the mean and variance of each channel are calculated; by minimizing the following distribution alignment loss, the missing-modal features are constrained to align with the full-modal feature distribution: wherein, μ curr , μ full respectively represent the second encoding fusion feature mean of the missing modality image and the second encoding fusion feature mean of the full modality image, respectively represent the second encoding fusion feature variance of the missing modality image and the second encoding fusion feature variance of the full modality image.

5. The method of claim 3, wherein, in the bottleneck layer, element-wise addition is performed on the first encoding fusion features and the second encoding fusion features to generate a fusion feature map; the fusion feature map channel dimension is up-projected and down-compressed through two fully connected layers to generate a bottleneck feature.

6. The method of claim 3, wherein, S2 comprises constructing the inherent normal prototype extraction module: define a random prototype vector and initialize it by normal distribution; fuse the random prototype vector and the second encoding fusion feature respectively through linear mapping to generate query, key and value; calculate attention output through standard scaling dot product attention mechanism, and add the random prototype vector to generate attention prototype; the attention prototype is further updated by two-layer feedforward neural network to generate inherent normal prototype.

7. The method of claim 6, wherein, A prototype consistency loss L is constructed by minimizing the cosine distance between each of the second encoded fusion features and its said intrinsic normal prototype con : where Q En1 (i) is the feature vector of the i-th spatial location of the second encoded fusion feature En1, INPs n is the n-th intrinsic normal prototype, the cosine distance d i represents the minimum cosine distance of the feature to the prototype, d i ∈ [0, 2].

8. The method of claim 5, wherein, S2 comprises constructing the prototype guided decoder with multiple prototype guided decoding blocks: take the bottleneck feature as initial input, in each of the prototype guided decoding blocks, take the up-sampled decoding feature as query, the inherent normal prototype as key and value, and reconstruct the feature through attention mechanism; after decoding layer by layer and up-sampling to the original spatial resolution, generate the decoder feature, including first decoding fusion feature corresponding to the first encoding fusion feature, and second decoding fusion feature corresponding to the second encoding fusion feature.

9. The method of claim 7, wherein, S3 comprises constructing a total loss function: wherein: L rec is an adaptive reconstruction loss, L con is a prototype consistency loss, L dist is a distribution alignment loss, λ1, λ2 are weight hyperparameters.

Images