Multi-mode brain anomaly detection method and system based on machine learning

Abstract

The invention relates to the technical field of biomedical engineering, in particular to a multi-mode brain anomaly detection method and system based on machine learning. The method comprises the following steps: acquiring brain medical image data of different modalities, and realizing spatial registration and alignment through a multi-modal registration algorithm based on mutual information; a multi-branch feature extraction model including a convolutional neural network, a converter and a state space model is utilized to perform feature embedding on the original image of each modal; performing frequency decoupling on the features of each mode through adaptive approximate wavelet transform, and decomposing the features into high-frequency detail information and low-frequency global information; a frequency band fusion strategy based on an attention mechanism is implemented on high and low frequency features of different modal images, and fused frequency sub-band features are input into a space-frequency Mama module. Through the adaptive frequency domain decomposition and cross-modal fusion mechanism, the multi-modal brain image information is effectively integrated, and the accuracy and robustness of brain anomaly detection are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the interdisciplinary field of medical image analysis and artificial intelligence, and in particular to a multimodal brain anomaly detection method and system based on machine learning. Background Art

[0002] In clinical diagnosis and treatment, thanks to technological advancements and medical developments, personalized medicine is moving toward greater efficiency and precision, promising improved healthcare quality. Advances in medical imaging equipment are also improving the quality of tissue imaging data, enabling more detailed visualization of tissue and lesion structures. Imaging technologies such as MRI, PET, CT, SPECT, and X-ray each have their own unique characteristics and are suitable for different diagnostic needs. Each imaging modality has its own advantages and limitations: MRI provides excellent soft tissue contrast and structural detail, CT clearly reveals bone structure and calcified lesions, PET reflects brain metabolic activity, and SPECT assesses cerebral blood perfusion. By examining medical images and extracting and analyzing characteristic regions, diagnostic efficiency can be effectively improved, which is of great significance for emergency care, surgical preparation, and medical research. These advanced medical imaging technologies provide anatomical images of organs that objectively reflect tissue structure and pathological changes, providing crucial support for physicians from localization to treatment guidance.

[0003] Improvements in medical imaging quality have driven the development of medical image processing methods. Currently, medical image processing primarily extracts and analyzes lesion features to obtain disease data. However, relying solely on medical image feature extraction to analyze lesions and disease states has limited accuracy and fails to capture dynamic changes in patient tissues: single-modality imaging techniques have inherent limitations. Relying solely on a single imaging modality often fails to fully assess the complexity of brain lesions and can easily miss important diagnostic information. For example, while MRI provides excellent soft tissue contrast, it is limited in detecting early metabolic abnormalities. While PET can reveal metabolic activity, its spatial resolution is relatively low. Multimodal data can provide information on brain structure and function from different angles and levels, which is crucial for accurate diagnosis. For example, MRI can reveal anatomical structures, fMRI reflects brain activity, and EEG records brainwave signals. Traditional brain abnormality detection mostly relies on single-modality imaging or manual feature extraction, and has the following major shortcomings: Single-modality limitations: lack of multi-dimensional comprehensive judgment, prone to missed or misjudgment of lesions; limited feature extraction: traditional reliance on manual feature extraction and simple models, severe information loss, and difficulty in capturing complex and subtle abnormal patterns; poor model generalization ability: traditional algorithms have weak adaptability and insufficient robustness for complex samples, multi-source noise and individual differences; insufficient real-time and efficiency: large amounts of data are slow to analyze, making it difficult to support high-concurrency clinical applications, and are heavily dependent on doctors and have a heavy workload; existing multimodal fusion difficulties: heterogeneous data (such as MRI and EEG) have different spatial and frequency characteristics, and fusion technology is immature, resulting in the loss of important information.

[0004] Therefore, while multimodal fusion methods based on deep learning (such as convolutional neural networks (CNNs), Transformers, and Mamba) have achieved numerous innovations, they also face technical bottlenecks such as complex model structures, loss of frequency detail, and difficulty in coordinating different modalities. Currently, there is an urgent need for an efficient, automatic, scalable, and robust multimodal brain abnormality detection method to achieve early, accurate, and intelligent identification of brain diseases. Summary of the Invention

[0005] This invention aims to achieve high-precision and interpretable abnormality identification by adaptively fusing multiple brain images and signals to automatically extract multi-scale and multi-frequency features. Through adaptive frequency domain decomposition and space-frequency fusion mechanisms, it effectively integrates multimodal brain image information, improving the accuracy of brain abnormality detection and addressing challenges faced by existing methods, such as insufficient fusion, severe feature loss, and poor real-time and robustness.

[0006] The technical solution of the present invention is a multimodal brain abnormality detection method based on machine learning, characterized by comprising:

[0007] S1. Acquire brain medical image data of different modalities and achieve spatial registration and alignment through a multimodal registration algorithm based on mutual information;

[0008] S2. Using a multi-branch feature extraction model consisting of a convolutional neural network, a transformer, and a state-space model, feature embedding is performed on the original images of each modality.

[0009] S3, frequency decoupling the features of each mode through adaptive approximate wavelet transform, decomposing the features into high-frequency detail information and low-frequency global information;

[0010] S4, implement the frequency band fusion strategy based on the attention mechanism for the high and low frequency features of different modal images respectively;

[0011] S5. Input the fused frequency sub-band features into the space-frequency Mamba module, and use 2D state space dual modeling combining spatial perception and frequency filtering to achieve shallow and deep feature fusion;

[0012] S6, reconstructing the fused multi-frequency features into a fused image through adaptive approximate inverse wavelet transform, and extracting abnormal features again through the space-frequency Mamba module;

[0013] S7. Input the fused multimodal features into a machine learning model including classification / regression to achieve brain abnormality detection, segmentation, labeling, or characterization.

[0014] Preferably, the specific process of step S1 is as follows:

[0015] S1-1: Create an image dataset to collect high-resolution images of medical devices;

[0016] S1-2: Divide high-resolution images and low-resolution images into training sets and test sets;

[0017] S1-3: Label the high-resolution images to identify the locations of the disease, and then divide the images into training and test sets according to a certain ratio.

[0018] In step S1-1, the specific process of acquiring a high-resolution image of the medical device is as follows:

[0019] Brain medical image data of different modalities include MRI, PET, CT, and SPECT, and can be expanded to other formats such as EEG and fMRI according to actual clinical needs. The spatial registration process is a multimodal registration algorithm based on mutual information. The registration algorithm includes a pre-registration stage and a fine registration stage. The pre-registration stage uses a rigid transformation to calculate the mutual information between images of different modalities as a similarity measure to find the optimal translation and rotation parameters; the fine registration stage uses a non-rigid transformation to fine-tune local anatomical structure differences through a B-spline free deformation model to ensure that the images of each modality are aligned in the same anatomical space coordinate system. The pre-processing process includes:

[0020] S1-1-1: The image resolution is unified to 512×512 pixels, and the pixel intensity is normalized to the range [0,1].

[0021] S1-1-2: Elastix's affine-elastic two-step method was used, with a maximum number of iterations of 1000 and mutual information as the optimization metric;

[0022] S1-1-3: Low-contrast images such as CT, PET, and SPECT should be median filtered / Gaussian denoised before registration, with a kernel size of 3×3×3 being preferred;

[0023] S1-1-4: Coordinate system standardization (such as MNI space) is used between multiple modalities to ensure spatial overlap;

[0024] S1-1-5: The data format is unified into NIfTI or DICOM, and a structured data set is generated for subsequent modeling.

[0025] Preferably, the specific process of step S2 is as follows:

[0026] S2-1: Feature embedding is trained end-to-end using a deep neural network. The network structure includes a convolutional neural network (CNN), a transformer, and a state-space model (SSM).

[0027] S2-2: The multi-branch feature extraction model includes an Octave convolutional network structure for brain image processing. Its high-frequency branches process spatial detail features, and the low-frequency branches extract contextual features of the lesion area.

[0028] S2-3: The multi-branch feature extraction model includes a 3D ResNeXt-based MRI sequence processing module that uses grouped convolution to capture the three-dimensional features of brain tissue;

[0029] S2-4: The multi-branch feature extraction model includes a short-time Fourier transform layer for EEG signals and a gated spatiotemporal convolution module;

[0030] S2-5: Use the publicly available labeled medical image data from Harvard University for training and testing.

[0031] Preferably, the specific process of multi-branch feature extraction in step S2-2 is:

[0032] S2-2-1: A separate "branch network" is established for each modality, with an input layer size of 224×224 pixels × 1 channel (for grayscale images);

[0033] S2-2-2: Each branch consists of three convolutional layers (the convolution kernel sizes are 7×7, 5×5, and 3×3, with strides of 2, 1, and 1, respectively), with BatchNorm and ReLU inserted in the middle, and the number of channels increasing to 32, 64, and 128;

[0034] S2-2-3: Each convolution layer is followed by a 2×2 max pooling layer with a pooling stride of 2;

[0035] S2-2-4: Each branch output is connected to the multimodal feature concatenation layer to obtain a preliminary embedding vector;

[0036] S2-2-5: The network backend is set up with 1-2 fully connected layers (number of nodes 512 to 128), and the activation function is LeakyReLU;

[0037] S2-2-6: Use publicly labeled brain medical images from Harvard University as the training set, with a split ratio of training / validation / test = 7:2:1;

[0038] S2-2-7: The optimization algorithm is Adam, the initial learning rate is 0.001, the batch size is 16, and the training is 100 rounds;

[0039] S2-2-8: Performance criteria are the accuracy of multimodal feature extraction (≥90%), AUC, F1, etc. If the model does not meet the standards, it will automatically return to network adjustment.

[0040] Preferably, the specific process of performing the experiment using the public medical image data in step S2-5 is as follows:

[0041] The Harvard University medical image dataset used includes 160 sets of CT-MRI paired images, 245 sets of PET-MRI paired images, and 333 sets of SPECT-MRI paired images, totaling 738 sets of training samples and 72 sets of test samples. The training process uses the Adam optimizer with an initial learning rate of 1×10 -4 The batch size is 16, the number of training rounds is 200, and the verification performance parameters include peak signal-to-noise ratio PSNR ≥ 28dB, structural similarity SSIM ≥ 0.85, and mutual information MI ≥ 2.0.

[0042] Preferably, the specific process of step S3 is as follows:

[0043] Adaptive approximate wavelet transform (AdaWAT) is used to perform multi-scale frequency decoupling on the features of each mode, decomposing the features into high-frequency detail information and low-frequency global information; four learnable convolution kernels K are defined LL , K LH , K HL , K HH , corresponding to low-low frequency, low-high frequency, high-low frequency, and high-high frequency components, respectively. u0 and u1 are the adaptively learned low-frequency and high-frequency filter vectors (what does the superscript T represent?). A 3×3 dilated convolution with a dilation rate of 3 is applied to the low-frequency component for low-pass filtering enhancement, while a 3×3 dilated convolution with a dilation rate of 1 is applied to the high-frequency component for high-pass filtering enhancement, achieving adaptive separation and enhancement of different frequency components. The wavelet transform coefficients are adaptively fine-tuned end-to-end using loss feedback to maximize frequency band energy separation. The output is multiple sets of high- and low-frequency features for each modality, which are used for subsequent segmented fusion.

[0044] Preferably, the specific process of step S4 is as follows:

[0045] S4-1: Adaptive fusion is performed on a frequency band basis. Fusion weights are obtained through adaptive learning of a neural network and dynamically adjusted based on the importance, confidence, and feature sparsity of each modality to enhance complementary information between modalities.

[0046] S4-2: Feature enhancement and noise suppression are performed on the high and low frequency fusion results respectively.

[0047] Preferably, the specific process of multi-band adaptive fusion in step S4-1 is as follows:

[0048] S4-1-1: The frequency band fusion strategy adopts a learning-based weight allocation mechanism and a weighted average method for low-frequency component fusion: LL fuse =α·LL MRI +β·LL CT +γ·LL CT +δ·LL SPECT , where LL MRI LL CT LL SPECT They represent the low-frequency components (LL subbands) extracted from the corresponding modality (MRI, CT, SPECT) images after low-frequency filtering, decomposition, and wavelet transform, containing the main structural features and global grayscale information of the modality. α, β, γ, and δ are weight parameters learned through the attention mechanism, satisfying α + β + γ + δ = 1;

[0049] S4-1-2: Use the maximum selection strategy for high-frequency component fusion: {LH,HL,HH} fuse =max({LH,HL,HH}MRI ,{LH,HL,HH} PET ,{LH,HL,HH} CT ,{LH,HL,HH} SPECT ), retaining the detailed information of each mode.

[0050] Preferably, the specific process of processing the fusion result of step S4-2 is as follows:

[0051] S4-2-1: High-frequency components are weightedly fused based on the channel attention mechanism (Squeeze-and-Excitation module) to improve the ability to capture subtle anomalies / edges;

[0052] S4-2-2: The low-frequency components use element-wise sum combined with spatial self-attention (Non-localBlock) to effectively integrate global structural features;

[0053] S4-2-3: In the fusion process, learnable weight coefficients are introduced for different modalities, the weights are initialized and averaged, and automatically tuned during training;

[0054] S4-2-4: Constrain the consistency of feature distribution after fusion through feature alignment loss (MMD / HSIC);

[0055] S4-2-5: The fusion operation is performed in batches, supports dynamic modal number input, and improves model adaptability.

[0056] Preferably, the specific process of step S5 is as follows:

[0057] S5-1: The spatial-frequency Mamba module includes a spatial perception branch and a frequency filtering branch, which respectively perform spatial convolution and frequency domain filtering on the input features. The outputs of the two are fused through adaptive gating (FFN layer, Gating) to achieve dynamic integration of cross-domain features.

[0058] S5-2: Adopting an asymmetric bilinear fusion strategy to establish an explicit mapping relationship for feature matching between modalities;

[0059] S5-3: Perform inter-modal feature similarity measurement in the spatial dimension, and the similarity matrix is ​​calculated as:

[0060]

[0061] where φ and ψ are linear projection layers, and Feature map elements from different modalities; is the kth eigenvector of mode B; T is the transposition operation to make the inner product valid; k is used to traverse all the feature points of mode B for normalization (Softmax).

[0062] Preferably, the specific process of shallow spatial-frequency feature fusion in step S5-1 is:

[0063] S5-1-1: The spatial-frequency Mamba module consists of a two-stream structure: one is a spatial convolution block (3×3 convolution + layernorm), and the other is a frequency convolution block (1×1 convolution + frequency gating unit), and the outputs of the two are cascaded;

[0064] S5-1-2: The spatial-frequency Mamba module includes N1=2 shallow Mamba blocks and N2=4 deep Mamba blocks. The hidden dimension of each Mamba block is C=64, and it includes an adaptive domain state space dual (AdaD-SSD) module. The AdaD-SSD module includes a spatial perception branch S(·) and a frequency filtering branch T(·); the spatial perception branch uses a 3×3 spatial convolution operation and a SiLU activation function to enhance the spatial domain features: L S =σ(Wse(L′)); The frequency filtering branch is transformed into the frequency domain through FFT, and the adaptive filter is applied for frequency selection:

[0065] L T =IFFT(FFT(L′)⊙||FFT(L′)|| 2 >λ), where λ is a learnable frequency threshold parameter.

[0066] Preferably, the specific process of the bilinear fusion strategy in step S5-2 is:

[0067] S5-2-1: 2D-SSD (State-Space Duality) is introduced in the state-space modeling part. Four state units (256 channels per unit) are set up, and the state update adopts a gated recursive mechanism (similar to LSTM but with a learnable frequency kernel as the weighting parameter).

[0068] S5-2-2: Shallow spatial-frequency Mamba is preferably set to 2 to 3 layers and residual connection;

[0069] S5-2-3: The deep fusion module is a cross-layer Transformer Attention module with 512 hidden units and 8 heads.

[0070] S5-2-4: The fused feature tensor is used to express abnormal patterns, and the channels are merged into 256 dimensions.

[0071] Preferably, the specific process of step S6 is as follows:

[0072] S6-1: The fused multi-frequency features are restored to the fused image or feature map of the original resolution through adaptive approximate inverse wavelet transform (AdaIWAT);

[0073] S6-2: The inverse wavelet kernel parameters are trained synchronously with the aforementioned wavelet transform parameters to enhance frequency domain reversibility and information preservation;

[0074] S6-3: The reconstructed features are input into the space-frequency Mamba module again (with the same configuration as S5) to deeply extract and fuse the abnormal characterization features;

[0075] S6-4: The output feature dimension and space is 224×224×128, which has high expressiveness and discriminative ability.

[0076] Preferably, the specific process of the adaptive approximate inverse wavelet transform in step S6-1 is:

[0077] Adaptive Approximate Inverse Wavelet Transform (AdaIWAT) uses transposed convolution to achieve feature reconstruction, uses the inverse transform kernel corresponding to AdaWAT to convert the frequency domain to the spatial domain, and uses jump connections to maintain detail information during the reconstruction process: Frecon = AdaIWAT (LL fused ,{LH,HL,HH} fused )+Residual, where Residual is the residual feature of the jump connection; the deep fusion module adopts the U-Net structure of the residual connection. The encoder contains 4 layers of downsampling convolution and the decoder contains 4 layers of upsampling transposed convolution. The number of feature channels in each layer is 64, 128, 256, and 512 respectively.

[0078] Preferably, the specific process of step S7 is as follows:

[0079] S7-1: The training loss function of the anomaly detection branch includes structural similarity (SSIM) loss, texture loss, and intensity loss. It includes a dynamic multi-receptive field mechanism, adaptively adjusts the feature receptive range through deformable convolution kernels, and adopts a joint supervision loss function to simultaneously ensure the structural fidelity, detail expression, and accurate reflection of global brightness information of the fused image:

[0080]

[0081] This includes classification loss Regression loss and feature consistency loss

[0082] S7-2: Design different detection / segmentation heads based on specific abnormality categories (e.g., tumor, infarct, abnormal activity area, etc.), and perform end-to-end joint optimization of abnormality detection and multimodal fusion models;

[0083] S7-3: This model can be widely used in abnormality detection, segmentation, lesion localization, auxiliary diagnosis, and analysis tasks of multimodal brain medical images. A clinical decision support module is introduced in the output stage to integrate patients' historical diagnosis and treatment data to generate risk prediction reports. The module builds a disease evolution model based on a graph neural network and uses the adjacency matrix A to describe the association between symptoms:

[0084] A ij =σ(MLP([h i ||h j ]))

[0085] where h i 、h j is the node feature, and σ is the Sigmoid activation function.

[0086] Preferably, the specific process of the abnormality detection branch in step S7-1 is:

[0087] The machine learning model adopts a multi-task learning framework, including anomaly detection branch, lesion segmentation branch and severity assessment branch. The anomaly detection branch adopts a 3-layer fully connected network with 256, 128 and 2 nodes respectively, and uses the softmax activation function to output normal / abnormal probabilities. The lesion segmentation branch adopts a semantic segmentation network to output pixel-level lesion area masks. The severity assessment branch adopts a regression network to output a continuous value between 0 and 1 to represent the severity of the lesion. The loss function adopts a weighted combination of multi-task losses: L total =λ1·L classification +λ2·L segmentation +λ3·L regression , where λ1=1.0, λ2=2.0, and λ3=1.5.

[0088] Preferably, the specific process of abnormality detection optimization in step S7-2 is:

[0089] S7-2-1: The classification / segmentation framework can use Vision Transformer (ViT) or SE-ResNet with Class Token, outputting abnormal / normal / multi-classification category labels and supporting multi-label discrimination;

[0090] S7-2-2: The segmentation network is preferably a U-Net structure with 128 channels, a resolution consistent with the input, and a BCE-Dice hybrid loss.

[0091] S7-2-3: supports regression tasks (such as anomaly probability classification). The regression layer uses a three-layer fully connected layer with the number of nodes in the order of 128 → 32 → 1, and the activation function is Sigmoid / Softmax.

[0092] S7-2-4: Outputs include abnormality location maps, probability heat maps, and lesion quantitative assessment data, and can be connected to the electronic medical record system.

[0093] S7-2-5: Support interpretability modules (such as Grad-CAM) to assist in weight visualization to facilitate doctors' decision review.

[0094] Compared with the prior art, this application has at least one of the following beneficial technical effects:

[0095] 1. This invention uses an adaptive approximate wavelet transform to decompose the features of different modalities into high-frequency details and low-frequency global information. It then implements a differentiated fusion strategy based on frequency band characteristics, effectively integrating multidimensional information such as structure, metabolism, and function of each modality to overcome the limitations of a single modality. At the same time, the adaptive approximate wavelet transform is used to dynamically adjust the wavelet transform kernel function to achieve adaptive separation of high-frequency details and low-frequency global information. Combined with dilated convolution to enhance the characteristics of different frequency bands, this method improves the ability to capture both subtle abnormalities (such as early-stage tumors) and global lesions (such as large-area infarctions).

[0096] 2. Support multi-modal data input such as MRI, PET, CT, SPECT, etc. Through spatial registration and alignment and multi-branch feature extraction network, it solves the differences in spatial resolution and imaging principles between different modalities and avoids information loss.

[0097] 3. The spatial-frequency Mamba module and 2D state space dual modeling are introduced. Through the dynamic gating fusion of spatial convolution and frequency filtering, the spatial correlation and frequency characteristics of abnormal areas are captured simultaneously at the global and local scales, enhancing the model's comprehensive perception of complex lesions.

[0098] 4. Through end-to-end training and joint supervision loss functions (including SSIM, texture, intensity loss and classification / regression loss), the structural fidelity and abnormal feature expression of the fused image are ensured, and the accuracy of abnormality detection such as tumors and infarctions is improved.

[0099] In summary, the adaptive frequency domain decomposition and dynamic weight fusion mechanism of this invention reduces the impact of multi-source noise and individual differences, and the model demonstrates high robustness in complex samples. A clinical decision support system built on a graph neural network, integrating multi-task modules such as anomaly detection, segmentation, lesion localization, and severity assessment, can generate risk prediction reports based on historical patient data, providing comprehensive diagnostic support for doctors. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] Figure 1 Schematic diagram of the process of the multimodal brain abnormality detection method based on machine learning of the present invention.

[0101] Figure 2 Schematic diagram of the medical image frequency decoupling according to the present invention.

[0102] Figure 3 Schematic diagram of 2D state space dual modeling of the present invention.

[0103] Figure 4 Schematic diagram of the reconstruction of multi-frequency features of medical images according to the present invention.

[0104] Figure 5 Schematic diagram of the process of the multimodal brain abnormality detection system based on machine learning of the present invention.

[0105] Figure 6 This is a system block diagram of the present invention. DETAILED DESCRIPTION

[0106] The technical solutions of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the disclosure claimed for protection, but merely represents selected embodiments of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0107] The present invention Figure 5 and Figure 6 As shown in the figure, a multimodal brain anomaly detection system based on machine learning is proposed for performing a multimodal brain anomaly detection method. The multimodal brain anomaly detection system includes a data acquisition module, a multi-branch feature extraction module, a frequency domain processing module, a fusion module, and a detection module. The data acquisition module is used to collect and align brain image data from medical devices of different modalities; the multi-branch feature extraction module contains a convolutional neural network, a transformer, and a state-space model to embed features of the original images of each modality; the frequency domain processing module is used for adaptive approximate wavelet transform decomposition and adaptive approximate inverse wavelet transform reconstruction; the fusion module integrates the space-frequency Mamba module and 2D state space dual modeling to achieve shallow and deep feature fusion; and the detection module performs brain anomaly detection, segmentation, labeling, or characterization based on a machine learning model.

[0108] Please refer to the attached Figure 1 As shown, the multimodal brain abnormality detection method includes the following steps:

[0109] S1: Acquire brain medical image data of different modalities, including but not limited to MRI, PET, CT, and SPECT, and align the images of each modality through spatial registration;

[0110] S2: Create a feature extraction model with a multi-branch image structure and embed features into the original image of each modality to obtain a preliminary feature representation. This feature extraction model is trained using publicly available labeled medical image data from Harvard University. The spatial-frequency domain feature extraction model for each modality is validated using a validation set to determine whether the specified performance parameters meet the specified requirements.

[0111] S3: Adaptive approximate wavelet transform (AdaWAT) is used to perform frequency decoupling on the features of each mode, decomposing the features into high-frequency detail information and low-frequency global information;

[0112] S4: Implement frequency band fusion strategies for high-frequency and low-frequency components of images of different modalities respectively;

[0113] S5: The fused frequency subband features are input into the space-frequency Mamba module, and 2D state-space dual modeling (AdaD-SSD) combining spatial perception and frequency filtering is used to achieve shallow space-frequency feature fusion. The expressiveness and robustness of the fusion results are further improved through deep fusion.

[0114] S6: Adaptive approximate inverse wavelet transform (AdaIWAT) is used to reconstruct the fused multi-frequency features into a fused image, which is then further deeply fused and anomaly-related features are extracted through the space-frequency Mamba module;

[0115] S7: Input the fused multimodal features into a machine learning model for brain abnormality detection, segmentation, labeling, or characterization.

[0116] As an embodiment, step S1 further includes:

[0117] The MRI T1-weighted images and PET metabolic images of brain tumor patients were used as input data. The MRI image resolution was 512 × 512 × 256 voxels, and the voxel size was 0.5 × 0.5 × 1.0 mm. 3 PET image resolution is 128 × 128 × 63 voxels, and voxel size is 2.0 × 2.0 × 2.4 mm 3 .

[0118] The spatial registration process adopts a registration algorithm based on normalized mutual information (NMI). The registration process is divided into two stages: the pre-registration stage uses a rigid transformation (containing 6 degrees of freedom: 3 translation parameters and 3 rotation parameters) and maximizes the normalized mutual information between the two images through the Powell optimization algorithm. The number of iterations is set to 200 and the convergence threshold is 1×10-6; the fine registration stage uses a non-rigid transformation based on B-splines, the control point grid spacing is 20 mm, the optimization algorithm adopts L-BFGS-B, and the maximum number of iterations is 500.

[0119] As an embodiment, step S2 further includes:

[0120] The multi-branch feature extraction model consists of two modality-specific branches and one shared feature branch. The MRI branch includes three convolutional layers: the first layer has a kernel size of 7×7×7, 64 output channels, a stride of 2×2×2, and 3×3×3 padding; the second layer has a kernel size of 5×5×5, 128 output channels, a stride of 1×1×1, and 2×2×2 padding; and the third layer has a kernel size of 3×3×3, 256 output channels, a stride of 1×1×1, and 1×1×1 padding. Each convolutional layer is followed by a BatchNorm3d, ReLU activation function, and a 3×3×3 max pooling layer with a pooling stride of 2×2×2.

[0121] The PET branch uses the same network structure, but adds a 3×3×3 dilated convolution (with a dilation rate of 2) before the first convolution layer to accommodate the low resolution of PET images. The shared branch contains two layers of depthwise separable convolution: a depthwise convolution kernel of size 3×3×3, a pointwise convolution kernel of size 1×1×1, and 256 output channels.

[0122] As an embodiment, step S3 further includes:

[0123] Adaptive Approximate Wavelet Transform (AdaWAT) achieves frequency decomposition through learnable wavelet filters, such as Figure 2 As shown. Define four 3×3×3 learnable convolution kernels: K LL Used to extract low-low frequency components, corresponding to global structural information; K LH Used to extract low-high frequency components, corresponding to horizontal edges; K HL Used to extract high-low frequency components, corresponding to vertical edges; K HH Used to extract high-frequency components, corresponding to diagonal textures.

[0124] In the specific implementation, the low-frequency filter vector u0 is initialized to [0.7071, 0.7071], and the high-frequency filter vector u1 is initialized to [0.7071, -0.7071], and is adaptively updated during the training process by gradient descent. m1 and LL m2 A 3×3×3 dilated convolution with a dilation rate of 3 is used for low-pass filtering enhancement, and the convolution weights are initialized using Xavier initialization; for high-frequency components {LH m1 ,HL m1 ,HH m1} and {LH m2 ,HL m2 ,HH m2} Apply a 3×3×3 standard convolution with a dilation rate of 1 for high-pass filtering enhancement.

[0125] As an embodiment, step S4 further includes:

[0126] The frequency band fusion uses the attention mechanism to dynamically assign weights. For low-frequency component fusion, an attention weight calculation module is constructed: First, the LL m1 and LL m2 They are compressed into 1×1×C feature vectors respectively, and then the attention weights αMRI and αPET are calculated through a two-layer fully connected network (the number of hidden layer neurons is C / 4, and the activation function is ReLU), satisfying αMRI+αPET=1.

[0127] The low-frequency fusion result is: LL fuse =αMRI×LL m1 +αPET×LL m2 .

[0128] For high-frequency component fusion, the channel-level maximum selection strategy is adopted, where max is the element-by-element maximum operation:

[0129] LH fuse =max(LH m1 ,LH m2 );

[0130] HL fuse =max(HL m1 ,HL m2 );

[0131] HH fuse =max(HH m1 ,HH m2 );

[0132] Among them, this strategy can retain the important detail features of each mode and avoid excessive smoothing of high-frequency information.

[0133] As an embodiment, step S5 further includes:

[0134] The space-frequency Mamba module contains N1=2 shallow blocks and N2=4 deep blocks, and the hidden dimension of each block is C=64.

[0135] like Figure 3 As shown in Figure 2, the specific implementation of 2D state space dual modeling using a combination of spatial perception and frequency filtering is as follows:

[0136] Input feature L in ∈R (H×W×64) First, we get L′∈R through the linear projection layer H×W×(C′+2G·d). , where the expansion factor C′=3, the state dimension d=16, and the number of groups G=8.

[0137] Implementation of spatial perception branch S(·): using 3×3 depth-wise separable convolution W SE , including depthwise convolution (convolution kernel 3×3×1, the number of groups is equal to the number of input channels) and pointwise convolution (convolution kernel 1×1×1), followed by SiLU activation function: L S =SiLU(W SE (L′)).

[0138] Implementation of the frequency filtering branch T(·): First, transform L' into the frequency domain through 2D FFT to obtain Λ = FFT 2D (L′), then calculate the frequency domain amplitude spectrum |Λ|, and perform adaptive filtering through the learnable parameter λ (initial value is 0.1): L T =IFFT 2D (Λ⊙(|Λ| 2 >λ)), where ⊙ represents the Hadamard product.

[0139] The 2D state space dual mechanism divides the image into 8×8 subregions, each of which maintains an independent hidden state H ij,t ∈R 16 , the state update equation is:

[0140] H ij,t =A t ⊙H ij,t -1+B t ⊙X ij,t

[0141] Y ij,t =C t T ⊙H ij,t

[0142] Among them A t 、B t 、C t is a learnable state transfer matrix, which is obtained by fusion feature L through a linear layer. S ⊕L T Separated from ij,t It is the spatial-frequency feature input of the (i, j)th sub-region at the tth moment. It is the fused feature assigned to each sub-region after being processed by the spatial perception convolution branch and the frequency filtering branch. It is used to describe the comprehensive local state of the current region and serves as the input for updating the state space model.

[0143] As an implementation method, Figure 4 , step S6 further comprises:

[0144] The AdaIWAT uses transposed convolution to achieve feature reconstruction. First, the fused frequency components {LLfuse ,LH fuse ,HL fuse ,HH fuse}Processed by 4 parallel transposed convolution branches:

[0145] Low-frequency reconstruction branch: 3×3×3 transposed convolution, stride 2×2×2, 64 output channels;

[0146] Three high-frequency reconstruction branches: 3×3×3 transposed convolution, stride 2×2×2, 64 output channels each.

[0147] The reconstruction process uses skip connections to preserve detail information:

[0148] Frecon=AdaIWAT(LL fuse ,{LH fuse ,HL fuse ,HH fuse})+Fres, where Fres is

[0149] Residual features from the corresponding layer of the encoder.

[0150] The deep fusion module adopts an improved U-Net structure. The encoder consists of 4 layers: the input is 64→128→256→512 channels, each layer contains two 3×3×3 convolutions, BatchNorm and ReLU; the decoder adopts a symmetrical structure and gradually restores the resolution through transposed convolution and jump connection.

[0151] As an implementation method, step S7 further includes:

[0152] Anomaly detection branch: A 3-layer fully connected network is used with the following structure: FC(512→256)→Dropout(0.2)→ReLU→FC(256→128)→Dropout(0.2)→ReLU→

[0153] FC(128→2)→Softmax, output normal / abnormal probability.

[0154] Lesion segmentation branch: A 3D U-Net structure is used. The encoder output features are reduced to the number of segmentation categories (background, tumor core, tumor enhancement area, tumor edema) through 1×1×1 convolution. The loss function uses a combination of Dice Loss and cross entropy loss.

[0155] Severity evaluation branch: Use the regression network FC(512→256)→ReLU→FC(256→128)→ReLU→FC(128→1)→Sigmoid to output a continuous value between 0 and 1.

[0156] Total loss function: L total=1.0×L classification +2.0×L segmentation +1.5×L regression +10×L ssim +20×L text +20×L int . Where L classification represents the classification loss, which measures the classification accuracy of brain abnormalities; L segmentation represents the segmentation loss, which measures the pixel-level segmentation accuracy of the brain lesion area; L regression Represents regression loss, which is the numerical prediction of indicators such as the severity of the lesion; L ssim Represents the structural similarity loss, ensuring that the structural information of the fusion output is highly consistent with the real medical image; L text represents the texture loss, which constrains the consistency of the fused image with the original single-modal image at the texture detail level; L int Represents the intensity loss, ensuring that the overall intensity distribution of the fused image is consistent with that of the original image. The weights of each loss term can be set through experimental tuning, allowing for joint multi-task optimization to ensure overall detection performance.

[0157] The network design supports any number and sequence of modal inputs and features dynamic weight assignment. An integrated interpretability module enables spatial localization of abnormality recognition results and feature heatmap interpretation, facilitating physician interpretation and secondary screening. Combining structural similarity (SSIM) loss, texture / detail loss, and intensity loss improves the model's sensitivity to various abnormality types and subtle changes, meeting both quantitative and qualitative diagnostic requirements.

[0158] The above specific embodiments are merely several optional embodiments of the present invention. Based on the technical solutions of the present invention and the relevant inspirations of the above embodiments, those skilled in the art may make various alternative improvements and combinations to the above specific embodiments.

Claims

1. A multimodal brain abnormality detection method based on machine learning, characterized in that: include: S1. Acquire brain medical image data of different modalities and achieve spatial registration and alignment through a multimodal registration algorithm based on mutual information; S2. Using a multi-branch feature extraction model consisting of a convolutional neural network, a transformer, and a state-space model, feature embedding is performed on the original images of each modality. S3, frequency decoupling the features of each mode through adaptive approximate wavelet transform, decomposing the features into high-frequency detail information and low-frequency global information; S4, implement the frequency band fusion strategy based on the attention mechanism for the high and low frequency features of different modal images respectively; S5. Input the fused frequency sub-band features into the space-frequency Mamba module, and use 2D state space dual modeling combining spatial perception and frequency filtering to achieve shallow and deep feature fusion; S6, reconstructing the fused multi-frequency features into a fused image through adaptive approximate inverse wavelet transform, and extracting abnormal features again through the space-frequency Mamba module; S7. Input the fused multimodal features into a machine learning model including classification / regression to achieve brain abnormality detection, segmentation, labeling, or characterization.

2. The multimodal brain abnormality detection method based on machine learning according to claim 1, characterized in that: In S1, the spatial registration and alignment includes: In the pre-registration stage, a rigid transformation is used to find the optimal translation and rotation parameters by calculating the mutual information between images of different modalities as a similarity measure; In the fine registration stage, non-rigid transformation is used to fine-tune the local anatomical structure differences through the B-spline free deformation model, and the images of each modality are aligned to the same anatomical space coordinate system.

3. The multimodal brain abnormality detection method based on machine learning according to claim 1, characterized in that: In S2, the multi-branch feature extraction model includes: The Octave convolutional network structure used to process brain images has high-frequency branches that process spatial detail features, while low-frequency branches extract contextual features of the lesion area; 3DResNeXt-based MRI sequence processing module, which uses grouped convolution to capture the three-dimensional features of brain tissue; An EEG signal processing module containing a short-time Fourier transform layer.

4. The multimodal brain abnormality detection method based on machine learning according to claim 1, characterized in that: In S3, the adaptive approximate wavelet transform applies different dilated convolutions to the high-frequency and low-frequency components respectively through grouped convolution and adaptive adjustment of a learnable kernel function.

5. The multimodal brain abnormality detection method based on machine learning according to claim 1, characterized in that: In S4, the frequency band fusion strategy includes: The low frequency component adopts the weighted average method: LL fuse =α·LL MRI +β·LL CT +γ·LL CT +δ·LL SPECT , where LL MRI LL CT LL SPECT They represent the low-frequency components extracted from the corresponding modal images after low-frequency filtering, decomposition, and wavelet transform; α, β, γ, and δ are weight parameters learned through the attention mechanism, satisfying α+β+γ+δ=1; The high-frequency component adopts the maximum value selection strategy to retain the detailed information of each mode.

6. The multimodal brain abnormality detection method based on machine learning according to claim 1, characterized in that: In the S5, the space-frequency Mamba module includes: The spatial perception branch uses 3×3 convolution and SiLU activation function; Frequency filtering branch, which implements frequency domain filtering through FFT transformation and learnable threshold λ; Adaptive gated fusion unit, through the similarity matrix Integrate cross-domain features, where φ and ψ are linear projection layers, and Feature map elements from different modalities; is the kth eigenvector of mode B; T is the transpose operation to make the inner product valid; k is used to traverse all the characteristic points of mode B for normalization.

7. The multimodal brain abnormality detection method based on machine learning according to claim 1, characterized in that: In S6, the space-frequency Mamba module captures the spatial and frequency correlation of abnormal regions at both global and local scales through two-dimensional state space modeling, including: Restoring the fused multi-frequency features to a fused image or feature map of original resolution through inverse transformation; The inverse wavelet kernel parameters and wavelet transform parameters are trained synchronously and inversely; The reconstructed features are input into the space-frequency Mamba module again to deeply extract and fuse the abnormal characterization features; Output feature dimension and space.

8. The multimodal brain abnormality detection method based on machine learning according to claim 1, characterized in that: In the S7, the machine learning model adopts a multi-task learning framework, including anomaly detection branch, lesion segmentation branch and severity assessment branch, wherein the anomaly detection branch adopts a 3-layer fully connected network with 256, 128 and 2 nodes respectively, and uses a softmax activation function to output normal / abnormal probabilities; the lesion segmentation branch adopts a semantic segmentation network to output pixel-level lesion area masks; the severity assessment branch adopts a regression network to output a continuous value between 0 and 1 to represent the severity of the lesion.

9. The multimodal brain abnormality detection method based on machine learning according to claim 8, characterized in that: The training loss function of the anomaly detection branch includes structural similarity loss, texture loss, and intensity loss, and includes a dynamic multi-receptive field mechanism. The feature receptive range is adaptively adjusted through the deformable convolution kernel, and the joint supervision loss function is expressed as: in is the classification loss, is the regression loss, is the feature consistency loss.

10. A multimodal brain anomaly detection system based on machine learning, characterized in that: For executing the multimodal brain abnormality detection method according to any one of claims 1 to 9, the multimodal brain abnormality detection system comprises: Data acquisition module, used to collect and register brain medical image data of different modalities; A multi-branch feature extraction module, which includes a convolutional neural network, a transformer, and a state-space model, to embed features of the original images of each modality; Frequency domain processing module, used for adaptive approximate wavelet transform decomposition and adaptive approximate inverse wavelet transform reconstruction; Fusion module, integrating space-frequency Mamba module and 2D state space dual modeling to achieve shallow and deep feature fusion; The detection module performs brain abnormality detection, segmentation, labeling, or characterization based on machine learning models.

Images