A method and system for aligning magnetoencephalography and magnetic resonance images
By extracting features from MEG and MRI using a dual-stream neural network model and calculating rigid body transformation parameters, the error problem in MEG-MRI image alignment was solved, achieving efficient and accurate cross-modal alignment.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANGHAI TONGJI HOSPITAL
- Filing Date
- 2026-06-08
- Publication Date
- 2026-07-03
AI Technical Summary
Existing manual matching methods based on reference points are prone to errors in MEG-MRI image alignment, and single-modal registration methods are difficult to effectively integrate the features of sparse point clouds and dense voxels, resulting in inaccurate alignment.
A pre-trained dual-stream neural network model is used to extract features from MEG and MRI using PointNet++ and 3D-ResNet-18, and rigid body transformation parameters are calculated using a multilayer perceptron to achieve automatic alignment of cross-modal features.
It improves the accuracy and robustness of image alignment, reduces human error, and simplifies the alignment process.
Smart Images
Figure CN122335931A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing, and more particularly to a method and system for aligning magnetoencephalogram (MEG) and magnetic resonance imaging (MRI) images based on cross-modal contrastive learning. Background Technology
[0002] With the deepening of brain science research and the increasing demand for precise localization in clinical neurosurgery, fusing high temporal resolution magnetoencephalography (MEG) with high spatial resolution magnetic resonance imaging (MRI) has become an important step in brain functional source localization and preoperative assessment of epileptic foci. Before performing source localization analysis, it is essential to establish the relative spatial relationship between the MEG sensor array and the subject's brain anatomy, a process known as "co-registration."
[0003] Currently, the most commonly used MEG-MRI alignment method is the manual matching method based on reference points. This involves placing coils or markers at the root of the nose and the points in front of both ears before MEG acquisition, recording these points, along with scalp surface contour points, using a digitizer. Then, identical anatomical landmarks are manually identified in the MRI images, and the rigid body transformation matrix is calculated using the least squares method. Intuitively, this method matches points in physical space with points in image space. Reference point-based alignment is simple in principle and widely integrated into commercial software, but it has several drawbacks:
[0004] If the subject moves during the scan, or if there are subjective errors in the locations of anatomical points manually marked on the MRI image (e.g., the root of the nose is difficult to locate precisely on the image), this method can lead to alignment errors of several millimeters or even centimeters, severely affecting the accuracy of subsequent brain signal source localization. Furthermore, while the iterative nearest-point algorithm based on scalp contours can assist in correction, it is prone to getting trapped in local optima and is sensitive to initial position.
[0005] Compared to single-modal registration, MEG data (sparse point clouds) and MRI data (dense voxels) belong to completely different data modalities with vastly different feature representations, making direct grayscale or geometric feature matching extremely difficult. However, since both describe the head geometry of the same subject, potential anatomical consistency exists. Therefore, how to utilize deep learning techniques to mine common anatomical features across modalities and achieve automatic and robust background alignment is the key problem that this invention aims to solve. Summary of the Invention
[0006] The purpose of this invention is to provide a method and system for aligning magnetoencephalogram (MEG) and magnetic resonance imaging (MRI) images based on cross-modal contrastive learning, so as to solve the problems mentioned in the background art.
[0007] To achieve the above-mentioned objective, one aspect of the present invention provides a method for aligning magnetoencephalography (MEG) and magnetic resonance imaging (MRI) images, comprising the following steps:
[0008] Step S1: Using a pre-trained dual-stream neural network model, define the MEG encoder and MRI encoder respectively to extract one-dimensional time series features and three-dimensional spatial structure features.
[0009] Step S2: Use the pre-trained encoder from step S1 to extract and project features from the original MEG data and MRI images, so that they enter the same common feature subspace.
[0010] Step S3: Concatenate the mapped feature vectors to obtain the global fusion feature for cross-modal alignment;
[0011] Step S4: Construct a multilayer perceptron for calculating rigid body transformation parameters through global fusion features, and use the global fusion features as input to the multilayer perceptron to predict rotation quaternions and translation vectors.
[0012] Step S5: Construct a rigid body transformation matrix using the rigid body transformation parameters to transform the MEG sensor position coordinates into the MRI coordinate space to obtain aligned fused data.
[0013] Furthermore, step S1 includes the following steps:
[0014] Step S101: MEG sensor location data, scalp surface point cloud data and corresponding MRI T1-weighted structural images collected from the publicly available HCP (Human Connectome Project) dataset are used as training set and validation set, and the training set and validation set are divided in an 8:2 ratio.
[0015] Step S102: Define a cross-modal contrastive network model that spatially constrains the position of MEG sensors based on MRI anatomical structures. The model includes a PointNet++ that processes sparse point cloud data as an MEG encoder and a 3D-ResNet that processes three-dimensional voxel data as an MRI encoder. Extract their respective deep semantic features and construct a contrastive loss function using the feature similarity of paired samples and the feature difference of unpaired samples.
[0016] Step S103: Pre-train the model using the training set and validate it on the validation set, then save the training model parameters obtained when the minimum alignment error is achieved on the validation set.
[0017] Furthermore, step S102 includes the following steps:
[0018] Step S121: Extract features from MEG point cloud and MRI image using PointNet++ and 3D-ResNet-18 respectively;
[0019] Step S122: Using a multilayer perceptron as a projection head, the features output by the two networks in step S121 are mapped onto the unit hypersphere to ensure the consistency of feature distribution.
[0020] Step S123: The projected features are used as input to the contrastive learning module to shorten the distance between positive sample pairs and widen the distance between negative sample pairs; the loss function used is the InfoNCE loss function, with the following formula:
[0021] ,
[0022] in: Batch size; The cosine similarity function; Temperature coefficient; and The first MEG and MRI projection features of individual samples.
[0023] Furthermore, step S121 is implemented as follows:
[0024] The PointNet++ network first performs set abstraction on the point cloud, and then includes three feature propagation layers; the 3D-ResNet-18 network includes four residual blocks, each of which includes four three-dimensional convolutional layers; then MEG data and MRI images are respectively input into the two networks to obtain their respective feature vectors;
[0025] Furthermore, step S2 is implemented as follows:
[0026] The optimal model parameters are loaded onto a cross-modal feature extraction network. MEG scalp surface point data acquired by a digitizer and MRI head scan images of the subjects are used as model inputs. The model is passed through a MEG encoder and an MRI encoder, respectively, and the output is a modal feature vector with two consistent dimensions.
[0027] Furthermore, step S3 includes the following steps:
[0028] Step S301, define the MEG feature vector as MRI feature vector is ,in For feature dimensions;
[0029] Step S302, Define For dimension size The global fusion feature vector is obtained by fusing features from two modalities through feature concatenation or Hadamard product, resulting in a global fusion feature vector for regression prediction.
[0030] Furthermore, step S4 includes the following steps:
[0031] Step S401: Construct a multilayer perceptron for calculating spatial transformation parameters through global fusion features. The multilayer perceptron consists of three fully connected layers. The seven values output by the last layer are the predicted spatial transformation parameters, including four quaternion parameters and three translation parameters.
[0032] Step S402: Construct a self-supervised training set by randomly generating rigid body transformation matrices. The training set includes MEG data with random transformation applied, original MRI data, and transformation parameters as ground truth. Use Geodesic Loss as the loss function to train a multilayer perceptron on the training set.
[0033] Step S403: Use the global fusion feature vector from step S3 as input to the multilayer perceptron to obtain the rotation quaternion. Translation vector .
[0034] Furthermore, step S5 includes the following steps:
[0035] Step S501, let the coordinates of a point in the MEG coordinate system be... Its corresponding coordinates in the MRI coordinate system It can be obtained from the following formula:
[0036] ,
[0037] in: Quaternions obtained from prediction Transformed Rotation matrix, The predicted translation vector;
[0038] Step S502: Transform all MEG sensor location points using the above formula to complete the alignment with the MRI image.
[0039] Another aspect of the present invention provides a system for aligning magnetoencephalogram (MEG) and magnetic resonance imaging (MRI) images, comprising a pre-training module, a feature extraction module, a stitching module, a parameter prediction module, and a registration module, wherein:
[0040] The pre-training module uses a pre-trained dual-stream neural network model, defining an MEG encoder and an MRI encoder to extract one-dimensional time series features and three-dimensional spatial structure features.
[0041] The feature extraction module uses the pre-trained encoder in the pre-training module to extract and project features from the raw MEG data and MRI images, so that they enter the same common feature subspace.
[0042] The concatenation module is used to concatenate the mapped feature vectors as global fusion features for cross-modal alignment;
[0043] The parameter prediction module is used to construct a multilayer perceptron for calculating rigid body transformation parameters through global fusion features. The global fusion features are used as input to the multilayer perceptron to predict rotation quaternions and translation vectors.
[0044] The registration module uses the rigid body transformation parameters to construct a rigid body transformation matrix, transforming the MEG sensor position coordinates into the MRI coordinate space to obtain aligned fused data.
[0045] Compared with existing technologies, this system and method have the following advantages:
[0046] 1) The method is simple and the registration accuracy is high.
[0047] 2) The transformation parameters required for registration can be obtained based on the deep learning statistical properties of cross-modal features, eliminating the need for manual labeling of reference points and providing great convenience.
[0048] 3) It greatly reduces the impact of human operation errors and initial position on the registration results, and better solves the spatial alignment problem in MEG and MRI multimodal fusion analysis. Attached Figure Description
[0049] Figure 1 This is a flowchart of a method for aligning brain magnetic resonance and magnetic resonance images.
[0050] Figure 2 This is a schematic diagram illustrating the principle of a method for aligning brain magnetic resonance and magnetic resonance images. Detailed Implementation
[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0052] like Figure 1 Figures 2 and 3 show the flowchart and schematic diagram of the first embodiment: a method for aligning magnetoencephalogram (MEG) and magnetic resonance imaging (MRI) images. This invention includes the following steps:
[0053] Step S1: Using a pre-trained dual-stream neural network model, define the MEG encoder and MRI encoder respectively to extract one-dimensional time series features and three-dimensional spatial structure features.
[0054] Step S101: MEG sensor location data, scalp surface point cloud data and corresponding MRI T1-weighted structural images obtained from the HCP dataset are used as training set and validation set, with the ratio of training set to validation set being 8:2.
[0055] Step S102 defines a cross-modal contrastive network model guided by MRI anatomy and capable of spatially constraining the location of MEG sensors. This model extracts features from sparse point clouds and 3D images using MEG encoders and MRI encoders, respectively. These features are then projected and input into a contrastive learning module to optimize the feature space distribution. A loss function is constructed using paired samples. Specifically, this includes the following sub-steps:
[0056] Step S121: Features are extracted from MEG point clouds and MRI images using PointNet++ and 3D-ResNet-18, respectively. The PointNet++ network first performs set abstraction on the point cloud, followed by three feature propagation layers. The 3D-ResNet-18 network consists of four residual blocks. The first residual block contains four 3D convolutional layers, the second residual block contains four 3D convolutional layers, the third residual block contains four 3D convolutional layers, and the last residual block contains four 3D convolutional layers. The MEG data and MRI images are input into the two networks respectively to obtain their respective feature vectors.
[0057] Step S122: The feature vector obtained in step S121 is mapped onto the unit hypersphere using a multilayer perceptron as the projection head.
[0058] In step S123, the projected features generated in step S122 are used as input to the contrastive learning module to bring the positive sample pairs closer and push the negative sample pairs further apart.
[0059] Step S124, the above operation aims to maximize the mutual information between modalities. The loss function used is the InfoNCE loss function expressed by the following equation b:
[0060]
[0061] in: Batch size; The cosine similarity function; Temperature coefficient; and The first MEG and MRI projection features of individual samples.
[0062] Step S103: Pre-train the model based on the training set and validate it on the validation set, and save the training model parameters when the minimum alignment error is obtained on the validation set.
[0063] Step S2: Use the pre-trained encoder from step S1 to extract and project features from the original MEG data and MRI images, so that they enter the same common feature subspace.
[0064] The pre-trained encoder described above is used to extract features from the original data. The specific steps are as follows:
[0065] Load the feature extraction network with the optimal model parameters obtained above, take the MEG data and MRI images collected by the digitizer as the model input, and output the result, which is the modal feature vector with consistent two dimensions.
[0066] Step S3: Concatenate the mapped feature vectors to obtain the global fusion feature for cross-modal alignment. Global similarity statistics are performed by fusing the mapped feature vectors according to the following steps, and the resulting fused feature is used as the global fusion feature for cross-modal alignment. Specifically, the steps include:
[0067] Step S301, define the MEG feature vector as MRI feature vector is .
[0068] Step S302, Define For dimension size The global statistical feature vector can be fused by feature concatenation, thus obtaining a global fused feature vector for regression prediction.
[0069] Step S4: Construct a multilayer perceptron for calculating rigid body transformation parameters through global fusion features. Use the global fusion features as input to the multilayer perceptron to predict rotation quaternions and translation vectors, and calculate spatial transformation parameters.
[0070] Construct a multilayer perceptron for calculating rigid body transformation parameters using global features. Utilize the aforementioned global statistical features as input to the multilayer perceptron to predict rotation quaternions and translation vectors. The specific steps include the following:
[0071] Step S401: Construct a multilayer perceptron for calculating spatial transformation parameters using global features. The multilayer perceptron consists of three fully connected layers, with the first fully connected layer having an input dimension of... The first layer has an output dimension of 1024, the second fully connected layer has an input dimension of 1024, and the last layer outputs 7 values, which are the predicted spatial transformation parameters.
[0072] Step S402: A self-supervised training set is constructed by randomly generating rigid body transformation matrices. The training set includes MEG data with random transformations applied, original MRI data, and transformation parameters as ground truth. The geodesic distance loss is used as the loss function, and the multilayer perceptron is trained on the training set.
[0073] Step S403: Use the global statistical feature vector obtained in step S3 as input to the multilayer perceptron to obtain the rotation quaternion. Translation vector .
[0074] Step S5: Construct a rigid body transformation matrix using the rigid body transformation parameters to transform the MEG sensor position coordinates into the MRI coordinate space to obtain aligned fused data. Perform spatial alignment to obtain aligned MEG point cloud coordinates.
[0075] Using the transformation parameters obtained above, a rigid body transformation matrix is constructed. The MEG sensor position coordinates are then transformed into the MRI coordinate space to obtain the aligned fused data. The specific steps include the following:
[0076] Step S501, let the coordinates of a point in the MEG coordinate system be... Its corresponding coordinates in the MRI coordinate system It can be obtained from the following formula:
[0077] ,
[0078] in: Quaternions obtained from prediction Transformed Rotation matrix, The predicted translation vector;
[0079] Step S502: Transform all MEG sensor location points using the above formula to complete the alignment with the MRI image.
[0080] The second embodiment of the present invention is a system for aligning magnetoencephalogram (MEG) and magnetic resonance imaging (MRI) images, comprising a pre-training module, a feature extraction module, a stitching module, a parameter prediction module, and a registration module, wherein:
[0081] The pre-training module uses a pre-trained dual-stream neural network model, defining an MEG encoder and an MRI encoder to extract one-dimensional time series features and three-dimensional spatial structure features.
[0082] The feature extraction module uses the pre-trained encoder from step S1 to extract and project features from the original MEG data and MRI images, so that they enter the same common feature subspace.
[0083] The concatenation module is used to concatenate the mapped feature vectors as global fusion features for cross-modal alignment;
[0084] The parameter prediction module is used to construct a multilayer perceptron for calculating rigid body transformation parameters through global fusion features. The global fusion features are used as input to the multilayer perceptron to predict rotation quaternions and translation vectors.
[0085] The registration module uses the rigid body transformation parameters to construct a rigid body transformation matrix, transforming the MEG sensor position coordinates into the MRI coordinate space to obtain aligned fused data.
[0086] This invention calculates the transformation parameters required for registration based on the statistical properties of cross-modal features obtained through deep learning, eliminating the need for manually marked reference points and offering significant convenience. Alignment based on contrastive learning features improves the accuracy and robustness of rigid body registration compared to previous methods using manual reference points.
[0087] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A method for aligning brain magnetic resonance (BMRI) and magnetic resonance imaging (MRI) images, characterized in that, Includes the following steps: Step S1: Using a pre-trained dual-stream neural network model, define the MEG encoder and MRI encoder respectively to extract one-dimensional time series features and three-dimensional spatial structure features. Step S2: Use the pre-trained encoder from step S1 to extract and project features from the original MEG data and MRI images, so that they enter the same common feature subspace. Step S3: Concatenate the mapped feature vectors to form a global fusion feature for cross-modal alignment; Step S4: Construct a multilayer perceptron for calculating rigid body transformation parameters through global fusion features, and use the global fusion features as input to the multilayer perceptron to predict rotation quaternions and translation vectors. Step S5: Construct a rigid body transformation matrix using the rigid body transformation parameters to transform the MEG sensor position coordinates into the MRI coordinate space to obtain aligned fused data.
2. The method for aligning brain magnetic resonance and magnetic resonance images according to claim 1, characterized in that, Step S1 includes the following steps: Step S101: MEG sensor location data, scalp surface point cloud data and corresponding MRI T1 weighted structural images collected from the publicly available HCP dataset are used as training set and validation set, and the training set and validation set are divided in an 8:2 ratio. Step S102: Define a cross-modal contrastive network model that spatially constrains the position of MEG sensors based on MRI anatomical structures. The model includes a PointNet++ that processes sparse point cloud data as an MEG encoder and a 3D-ResNet that processes three-dimensional voxel data as an MRI encoder. Extract their respective deep semantic features and construct a contrastive loss function using the feature similarity of paired samples and the feature difference of unpaired samples. Step S103: Pre-train the model using the training set and validate it on the validation set, then save the training model parameters obtained when the minimum alignment error is achieved on the validation set.
3. The method for aligning brain magnetic resonance and magnetic resonance images according to claim 2, characterized in that, Step S102 includes the following steps: Step S121: Extract features from MEG point cloud and MRI image using PointNet++ and 3D-ResNet-18 respectively; Step S122: Using a multilayer perceptron as a projection head, the features output by the two networks in step S121 are mapped onto the unit hypersphere to ensure the consistency of feature distribution. Step S123: The projected features from step S122 are used as input to the contrastive learning module to shorten the distance between positive sample pairs and widen the distance between negative sample pairs; the loss function used is the InfoNCEloss function, with the following formula: , in: Batch size; The cosine similarity function; Temperature coefficient; and The first MEG and MRI projection features of individual samples.
4. The method for aligning brain magnetic resonance and magnetic resonance images according to claim 3, characterized in that, The implementation method for step S121 is as follows: The PointNet++ network first performs set abstraction on the point cloud, and then includes three feature propagation layers; the 3D-ResNet-18 network includes four residual blocks, each of which includes four three-dimensional convolutional layers; then MEG data and MRI images are respectively input into the two networks to obtain their respective feature vectors.
5. The method for aligning brain magnetic resonance and magnetic resonance images according to claim 1, characterized in that, The implementation method for step S2 is as follows: The optimal model parameters are loaded onto a cross-modal feature extraction network. MEG scalp surface point data acquired by a digitizer and MRI head scan images of the subjects are used as model inputs. The model is passed through a MEG encoder and an MRI encoder, respectively, and the output is a modal feature vector with two consistent dimensions.
6. The method for aligning brain magnetic resonance and magnetic resonance images according to claim 1, characterized in that, Step S3 includes the following steps: Step S301, define the MEG feature vector as MRI feature vector is ,in For feature dimensions; Step S302, Define For dimension size The global fusion feature vector is obtained by fusing features from two modalities through feature concatenation or Hadamard product, resulting in a global fusion feature vector for regression prediction.
7. The method for aligning brain magnetic resonance and magnetic resonance images according to claim 1, characterized in that, Step S4 includes the following steps: Step S401: Construct a multilayer perceptron for calculating spatial transformation parameters through global fusion features. The multilayer perceptron consists of three fully connected layers. The seven values output by the last layer are the predicted spatial transformation parameters, including four quaternion parameters and three translation parameters. Step S402: Construct a self-supervised training set by randomly generating rigid body transformation matrices. The training set includes MEG data with random transformation applied, original MRI data, and transformation parameters as ground truth. Use GeodesicLoss as the loss function to train a multilayer perceptron on the training set. Step S403: Use the global fusion feature vector from step S3 as input to the multilayer perceptron to obtain the rotation quaternion. Translation vector .
8. The method for aligning brain magnetic resonance and magnetic resonance images according to claim 1, characterized in that, Step S5 includes the following steps: Step S501, let the coordinates of a point in the MEG coordinate system be... Its corresponding coordinates in the MRI coordinate system It can be obtained from the following formula: ， in: Quaternions obtained from prediction Transformed Rotation matrix, The predicted translation vector; Step S502, use calculation The formula transforms all MEG sensor location points to align them with the MRI image.
9. A system for aligning brain magnetic resonance and magnetic resonance images, characterized in that, It includes a pre-training module, a feature extraction module, a concatenation module, a parameter prediction module, and a registration module, among which: The pre-training module uses a pre-trained dual-stream neural network model, defining an MEG encoder and an MRI encoder to extract one-dimensional time series features and three-dimensional spatial structure features. The feature extraction module uses the encoder pre-trained by the pre-training module to extract and project features from the raw MEG data and MRI images, so that they enter the same common feature subspace. The concatenation module is used to concatenate the mapped feature vectors as global fusion features for cross-modal alignment; The parameter prediction module is used to construct a multilayer perceptron for calculating rigid body transformation parameters through global fusion features. The global fusion features are used as input to the multilayer perceptron to predict rotation quaternions and translation vectors. The registration module uses the rigid body transformation parameters to construct a rigid body transformation matrix, transforming the MEG sensor position coordinates into the MRI coordinate space to obtain aligned fused data.