Depth domain adaptation-based fMRI data consistency correction method and device
Through deep learning domain adaptation technology, the 7T fMRI data is processed using the autoencoder network and the moment matching algorithm, which solves the problem of cross-day data consistency, improves data consistency, and provides high-quality data support for subsequent model construction.
Patent Information
- Application Number
- CN202410991914.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-07-23
AI Technical Summary
The data consistency of the 7T fMRI visual data set after multiple batches across days is not high, which limits the size and quality of the data set.
Using domain adaptation technology based on deep learning, the fMRI data collected multiple times across days is regarded as data domains of different sources through the autoencoder network and moment matching algorithm, and the intermediate feature information is extracted and correction data is reconstructed to reduce the number of different data domains. Moment spacing enables data consistency correction.
The overall consistency of fMRI data collected across days is improved, and a high-quality data foundation is provided for the subsequent construction of interpretable coding models.
Smart Images

Figure CN118981471A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of fMRI visual function data processing, and in particular to a method and device for fMRI data consistency correction based on depth domain adaptation. Background Art
[0002] As observation data of neural activity in the visual cortex of the brain, fMRI datasets are the basic conditions for studying visual information encoding. Their scale and quality directly affect the construction and analysis quality of visual encoding models. Figure 1 As shown in the figure, a natural image is presented to the subject, and the visual cortex of the brain is stimulated and activated. The BOLD signal related to neural activity can be recorded through fMRI. Through a series of preprocessing operations such as three-dimensional reconstruction, registration, and head motion correction, the response value of each voxel of interest in the three-dimensional space of the visual cortex is finally obtained. A large number of voxel responses in each visual cortical brain area constitute a voxel response vector, which is the fMRI data corresponding to the image stimulus. In this way, images are continuously presented to the subject from the natural image data set, so as to construct an fMRI data set corresponding to the natural image stimulation data set. Each sample pair includes an image and a voxel response vector of the visual cortical area of interest.
[0003] During the data collection process, images are played to subjects in a random order using stimulus presentation software such as E-Prime, according to pre-set presentation time, presentation frequency, rest intervals, etc. Usually, one image presentation and data collection is called a trial. For example, the Gallant team used a 200-ms frequency to alternately present images and gray backgrounds for 1 second, and then presented a gray background for 3 seconds as a break, for a total of 4 seconds. The continuous presentation of multiple image stimuli is usually considered a scan (run), and multiple scans of a subject after entering the MRI machine once constitute a subject's experiment (session). Usually, the same image is presented to subjects multiple times, and the corresponding data is collected, and the quality of fMRI data is improved through preprocessing. In the data collection of the Gallant team, each subject had a total of 5 sessions, each session included 5 runs for model training (70 different images were presented each time, each was presented twice, and it took a total of 11 minutes) and 2 runs for model verification (12 different images were presented each time, each was presented 13 times, and it took a total of 12 minutes). Without considering the time for resting state scanning and visual area positioning, the duration of a single session of the subject exceeded 1 hour. However, the time limit of the subjects in the MRI equipment made it difficult to complete large-scale fMRI data collection in a short period of time. The subjects were required to conduct multiple experiments lasting for many days. At the same time, in order to ensure data stability, it was usually necessary to repeatedly present the same image for the subjects to watch. These realities limited the sample size of the fMRI data set.
[0004] In 2008, the Gallant team released an fMRI dataset stimulated by grayscale natural images, which collected data on the subjects' V1, V2, V3, V4, and LO visual interest areas, with a sample size of 1870. In 2017, Horikawa et al. released an fMRI dataset stimulated by natural images, which collected data on the subjects' V1, V2, V3, V4, LOC, FFA, and PPA visual interest areas, including 200 categories of natural images, with a sample size of 1250. In 2019, Chang et al. released the BOLD-5000fMRI dataset, including data from 1 primary visual area and 4 advanced visual areas (PPA, RSC, OPA, and LOC), with a sample size of nearly 5000. In the early stage, the sample size of these mainstream fMRI datasets was in the thousands, which was quite different from the sample size of publicly available visual datasets in the field of artificial intelligence (such as ImageNet, Microsoft COCO, etc.), which limited the performance of deep networks in the field of visual encoding.
[0005] In December 2021, the Kay research team at the University of Minnesota publicly released the world's first 7T large-scale fMRI visual dataset in the international top journal Nature Neuroscience. The team evaluated that it has built a bridge between cognitive neuroscience and artificial intelligence. Regarding the image stimuli used in the experiment, nearly 73,000 natural scene images were selected from the richly annotated Microsoft COCO image dataset. The experiment selected 8 subjects to watch 9,000 different images respectively. In addition, different subjects also watched 1,000 identical images, and each subject watched a total of 10,000 images. The sample size of the entire dataset reached 10,000, which is an order of magnitude higher than the previous fMRI dataset. It took the team a year to complete it. Among them, ultra-high field (7T) functional magnetic resonance imaging was used, and each image was repeatedly presented to the subject 3 times, which can produce a relatively robust response signal. Regarding the data collection experimental design, each subject conducted 40 experiments (sessions), each session included 12 scans (runs), each run lasted 5 minutes, including more than 60 4-second stimulation trials (trails). Each session contained 750 trails in total. Because each image was repeated 3 times, each subject completed the presentation of 40×750 / 3=10,000 images in total. At the same time, in order to ensure the quality of data collection, during the viewing process of the subjects, the subjects were required to answer whether they had seen this image before, so as to maintain the experimental attention.
[0006] However, the data consistency of the 7T fMRI visual dataset after being collected and processed in multiple batches across days is not high. Summary of the invention
[0007] In order to solve the problem of low consistency of fMRI data collected and processed in multiple batches across days (such as 7T fMRI visual dataset), the present invention proposes an fMRI data consistency correction method and device based on deep domain adaptation. By treating the data collected multiple times across days as data domains from different sources, the deep learning domain adaptation technology is innovatively adopted to improve the overall consistency of fMRI data collected across days.
[0008] In order to achieve the above object, the present invention adopts the following technical solutions:
[0009] The present invention proposes a method for fMRI data consistency correction based on depth domain adaptation, comprising:
[0010] The fMRI voxel response data of brain regions collected multiple times across days are input into the autoencoder network to extract intermediate feature information;
[0011] According to the intermediate feature information extracted by the autoencoder network, the fMRI voxel response data are reconstructed and corrected. During the data reconstruction process, the moment matching algorithm is introduced to use the fMRI voxel response data of the brain region collected multiple times across days as data domains from different sources, thereby narrowing the moment spacing between different data domains in the intermediate feature space extracted by the autoencoder network, realizing domain-independent representation of fMRI data, and completing consistency correction of fMRI data.
[0012] Furthermore, the distribution distances between different data domains are calculated as follows:
[0013]
[0014] Where D i and D j Indicates different data domains, MD(D i ,D j ) represents the distribution distance between different data domains, E represents the encoder, and It represents the intermediate feature information of different data domains extracted by the autoencoder network, k=1 represents the first-order moment, and k=2 represents the second-order moment.
[0015] Furthermore, in the moment matching algorithm, the distribution distance within the N batches of fMRI data collected and processed is comprehensively measured by weighted summing of domain distances between each two in multiple domains.
[0016] Furthermore, in the moment matching algorithm, the following optimization objective function is adopted:
[0017]
[0018] Where L i represents the reconstruction error loss function, Xi represents the preprocessed fMRI sample data, X i ' indicates X i The corresponding reconstructed data, N represents the total acquisition batches, D in MD(D) represents the decoder, D represents N batches of fMRI voxel response datasets collected across days, and λ represents the weighting coefficient of the moment matching term.
[0019] Another aspect of the present invention provides an fMRI data consistency correction device based on depth domain adaptation, comprising:
[0020] The feature extraction module is used to input the fMRI voxel response data of the brain region collected multiple times across days into the autoencoder network to extract the intermediate feature information;
[0021] The domain-adaptive moment matching module is used to reconstruct and correct the fMRI voxel response data based on the intermediate feature information extracted by the autoencoder network. During the data reconstruction process, the moment matching algorithm is introduced to use the fMRI voxel response data of the brain region collected multiple times across days as data domains from different sources, thereby narrowing the moment spacing between different data domains in the intermediate feature space extracted by the autoencoder network, realizing domain-independent representation of fMRI data, and completing consistency correction of fMRI data.
[0022] Furthermore, the distribution distances between different data domains are calculated as follows:
[0023]
[0024] Where D i and D j Indicates different data domains, MD(D i ,D j ) represents the distribution distance between different data domains, E represents the encoder, and It represents the intermediate feature information of different data domains extracted by the autoencoder network, k=1 represents the first-order moment, and k=2 represents the second-order moment.
[0025] Furthermore, in the moment matching algorithm, the distribution distance within the N batches of fMRI data collected and processed is comprehensively measured by weighted summing of domain distances between each two in multiple domains.
[0026] Furthermore, in the moment matching algorithm, the following optimization objective function is adopted:
[0027]
[0028] Where L i represents the reconstruction error loss function, X i represents the preprocessed fMRI sample data, X i' indicates X i The corresponding reconstructed data, N represents the total acquisition batches, D in MD(D) represents the decoder, D represents N batches of fMRI voxel response datasets collected across days, and λ represents the weighting coefficient of the moment matching term.
[0029] Compared with the prior art, the present invention has the following beneficial effects:
[0030] The present invention innovatively adopts deep learning domain adaptation technology, which improves the overall consistency of fMRI data collected across days by treating the fMRI data collected multiple times across days as data domains from different sources, and provides a high-quality data foundation for the subsequent construction of an interpretable coding model. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 A schematic diagram of the fMRI dataset production process and database construction in the background technology;
[0032] Figure 2 This is one of the flow charts of a method for fMRI data consistency correction based on depth domain adaptation according to an embodiment of the present invention;
[0033] Figure 3 This is a schematic diagram of the autoencoder network architecture according to an embodiment of the present invention;
[0034] Figure 4 This is a second flowchart of a method for fMRI data consistency correction based on domain adaptation according to an embodiment of the present invention;
[0035] Figure 5 The present invention is a schematic diagram of the structure of an fMRI data consistency correction device based on depth domain adaptation according to an embodiment of the present invention. DETAILED DESCRIPTION
[0036] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments:
[0037] like Figure 2 As shown, a method for fMRI data consistency correction based on deep domain adaptation includes:
[0038] S101: Input the brain region fMRI voxel response data collected multiple times across days into the autoencoder network to extract key intermediate feature information:
[0039] Specifically, the input of the present invention is the (specific) brain region fMRI voxel response data X collected multiple times across days (specifically, the 7T fMRI preprocessed data provided by the Kay team can be used), and the output is the reconstructed and corrected fMRI data X', which has the same dimension and similar values, so the deep domain adaptation technology is designed by the autoencoder network. Figure 3As shown in Figure 2, the autoencoder network mainly includes an encoder network and a decoder network. For N batches of fMRI voxel response datasets collected across days: D = {D1, D2, ..., D N}, define X1,X2,…,X N is N batches of fMRI data samples collected and preprocessed across days, and the encoder extracts the data of each sample X i The key intermediate feature Z i '.
[0040] S102: Reconstruct the corrected fMRI voxel response data based on the intermediate feature information extracted by the autoencoder network. In the data reconstruction process, introduce the moment matching algorithm, use the brain region fMRI voxel response data collected multiple times across days as data domains from different sources, reduce the moment spacing between different data domains in the intermediate feature space extracted by the autoencoder network, realize the domain-independent representation of fMRI data, and complete the consistency correction of fMRI data. Including:
[0041] The fMRI data X is reconstructed again through the decoder i ', during the training process of the autoencoder network, the reconstruction error is required to be as small as possible, that is, the voxel response value before and after reconstruction is not allowed to change too much. The loss function is shown in formula (1).
[0042]
[0043] Where L i represents the reconstruction error loss function, E represents the encoder, and D represents the decoder.
[0044] Specifically, the input fMRI voxel response data after preprocessing across multiple days consists of domain-related features and domain-independent features. The domain-related features make the multi-domain data show inconsistency. In the data reconstruction process, in order to reduce the impact of low consistency caused by multiple acquisitions across days (different data domains), the encoder network is required to extract domain-independent features related to the visual cortex activated by image stimulation when extracting input data features, and suppress the expression of domain-related features in the data, that is, feature extraction is required to perform domain-independent representation.
[0045] like Figure 4 As shown in the figure, through optimization, the encoder will try to remove the domain-related feature information in the data during encoding, so that the amount of domain-related feature information collected multiple times across days in the reconstructed fMRI data will be reduced, thereby improving the consistency of fMRI data collected multiple times across days, and laying a good data foundation for the subsequent construction and analysis of interpretable coding models.
[0046] In order to make the encoder network realize domain-independent representation during the encoding process, it is required that the intermediate features of data in different domains belong to the same domain. First, the domain distance between different features is characterized. Data moment is an effective way to measure the distribution distance between different data. As shown in formula (2), the first-order moment and second-order moment of feature distribution are used to measure the distribution distance MD between different domains.
[0047]
[0048] Where k=1 represents the first-order moment, and k=2 represents the second-order moment.
[0049] Furthermore, as shown in formula (3), the distribution distance within the N batches of fMRI data collected and processed is comprehensively measured by weighted summing of the domain distances between each two in multiple domains.
[0050]
[0051] Where C represents the number of combinations, and D represents N batches of fMRI voxel response datasets collected across days.
[0052] Therefore, in the process of autoencoder network optimization, a domain adaptation regularization term, namely the moment matching regularization term, is introduced to minimize the distribution distance between different data domains while minimizing the reconstruction error. Therefore, the entire optimization objective function is shown in formula (4).
[0053]
[0054] Where λ represents the weighting coefficient of the moment matching term.
[0055] Based on the above embodiments, Figure 5 As shown, the present invention also proposes an fMRI data consistency correction device based on depth domain adaptation, comprising:
[0056] The feature extraction module is used to input the fMRI voxel response data of the brain region collected multiple times across days into the autoencoder network to extract the intermediate feature information;
[0057] The domain-adaptive moment matching module is used to reconstruct and correct the fMRI voxel response data based on the intermediate feature information extracted by the autoencoder network. During the data reconstruction process, the moment matching algorithm is introduced to use the fMRI voxel response data of the brain region collected multiple times across days as data domains from different sources, thereby narrowing the moment spacing between different data domains in the intermediate feature space extracted by the autoencoder network, realizing domain-independent representation of fMRI data, and completing consistency correction of fMRI data.
[0058] Furthermore, the distribution distances between different data domains are calculated as follows:
[0059]
[0060] Where D i and D j Indicates different data domains, MD(D i ,D j ) represents the distribution distance between different data domains, E represents the encoder, and It represents the intermediate feature information of different data domains extracted by the autoencoder network, k=1 represents the first-order moment, and k=2 represents the second-order moment.
[0061] Furthermore, in the moment matching algorithm, the distribution distance within the N batches of fMRI data collected and processed is comprehensively measured by weighted summing of domain distances between each two in multiple domains.
[0062] Furthermore, in the moment matching algorithm, the following optimization objective function is adopted:
[0063]
[0064] Where L i represents the reconstruction error loss function, X i represents the preprocessed fMRI sample data, X i ' indicates X i The corresponding reconstructed data, N represents the total acquisition batches, D in MD(D) represents the decoder, D represents N batches of fMRI voxel response datasets collected across days, and λ represents the weighting coefficient of the moment matching term.
[0065] In summary, the present invention innovatively adopts deep learning domain adaptation technology, which improves the overall consistency of fMRI data collected across days by treating the fMRI data collected multiple times across days as data domains from different sources, and provides a high-quality data foundation for the subsequent construction of an interpretable coding model.
[0066] The above is only a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.
Claims
1. A method for fMRI data consistency correction based on deep domain adaptation, characterized in that: include: The fMRI voxel response data of brain regions collected multiple times across days are input into the autoencoder network to extract intermediate feature information; According to the intermediate feature information extracted by the autoencoder network, the fMRI voxel response data are reconstructed and corrected. During the data reconstruction process, the moment matching algorithm is introduced to use the fMRI voxel response data of the brain region collected multiple times across days as data domains from different sources, thereby narrowing the moment distance between different data domains in the intermediate feature space extracted by the autoencoder network, realizing domain-independent representation of fMRI data, and completing consistency correction of fMRI data.
2. The fMRI data consistency correction method based on depth domain adaptation according to claim 1, characterized in that: The distribution distance between different data domains is calculated as follows: Where D i and D j Indicates different data domains, MD(D i ,D j ) represents the distribution distance between different data domains, E represents the encoder, and It represents the intermediate feature information of different data domains extracted by the autoencoder network, k=1 represents the first-order moment, and k=2 represents the second-order moment.
3. The fMRI data consistency correction method based on depth domain adaptation according to claim 2, characterized in that: In the moment matching algorithm, the distribution distance within N batches of fMRI data collected and processed is comprehensively measured by weighted summing of domain distances between two domains in multiple domains.
4. The fMRI data consistency correction method based on depth domain adaptation according to claim 3 is characterized in that: In the moment matching algorithm, the following optimization objective function is adopted: Where L i represents the reconstruction error, X i represents the preprocessed fMRI sample data, X i ' indicates X i The corresponding reconstructed data, N represents the total acquisition batches, D in MD(D) represents the decoder, D represents N batches of fMRI voxel response datasets collected across days, and λ represents the weighting coefficient of the moment matching term.
5. An fMRI data consistency correction device based on depth domain adaptation, characterized in that: include: The feature extraction module is used to input the fMRI voxel response data of the brain region collected multiple times across days into the autoencoder network to extract the intermediate feature information; The domain-adaptive moment matching module is used to reconstruct and correct the fMRI voxel response data based on the intermediate feature information extracted by the autoencoder network. During the data reconstruction process, the moment matching algorithm is introduced to use the fMRI voxel response data of the brain region collected multiple times across days as data domains from different sources, thereby narrowing the moment spacing between different data domains in the intermediate feature space extracted by the autoencoder network, realizing domain-independent representation of fMRI data, and completing consistency correction of fMRI data.
6. The fMRI data consistency correction device based on depth domain adaptation according to claim 5, characterized in that: The distribution distance between different data domains is calculated as follows: Where D i and D j Indicates different data domains, MD(D i ,D j ) represents the distribution distance between different data domains, E represents the encoder, and It represents the intermediate feature information of different data domains extracted by the autoencoder network, k=1 represents the first-order moment, and k=2 represents the second-order moment.
7. The fMRI data consistency correction device based on depth domain adaptation according to claim 6, characterized in that: In the moment matching algorithm, the distribution distance within N batches of fMRI data collected and processed is comprehensively measured by weighted summing of domain distances between two domains in multiple domains.
8. The fMRI data consistency correction device based on depth domain adaptation according to claim 7, characterized in that: In the moment matching algorithm, the following optimization objective function is adopted: Where L i represents the reconstruction error, X i represents the preprocessed fMRI sample data, X i ' indicates X i The corresponding reconstructed data, N represents the total acquisition batches, D in MD(D) represents the decoder, D represents N batches of fMRI voxel response datasets collected across days, and λ represents the weighting coefficient of the moment matching term.
Citation Information
Patent Citations
An fMRI visual coding model construction method based on transfer learning
CN109816630A
Small sample-based fMRI data training method and system
CN117351286A
Controller of legged mobile robot
US20060173578A1
