FMRI analysis method and system based on Transform network in combination with frequency domain analysis

Abstract

The invention discloses an fMRI analysis method and system based on a Transform network in combination with frequency domain analysis. The method comprises the steps of obtaining resting state fMRI data; preprocessing the resting-state fMRI data to obtain a time sequence; the method comprises the following steps: adding a frequency domain analysis module in front of a Transform block to obtain a Transform network combined with frequency domain analysis; the method comprises the following steps of: inputting a time sequence into a Transform network combined with frequency domain analysis, extracting attention weights on time and space dimensions through a Self-Attention layer, and generating a corresponding Attention Map; and displaying the attention degrees of the model on different brain regions and time nodes by taking the brain regions and the time nodes as two-dimensional coordinates of the generated Attention Map, and converting the Attention Map into a heat map.

Description

Technical Field

[0001] The present invention relates to the field of image and signal processing, and particularly to an fMRI analysis method and system based on a Transformer network combined with frequency domain analysis. Background Art

[0002] Currently, for the research on BOLD time series, the mainstream methods still focus on directly analyzing the original time-domain data, such as exploring brain activity patterns by calculating correlation coefficients or constructing functional connectivity networks. However, this method ignores the rich information of the BOLD signal in the frequency domain, especially its frequency components and periodic activities, which are crucial for a deep understanding of the dynamic functions of the brain. Another drawback of the existing technology is that it is often difficult to capture the potential non-linear dynamic characteristics of the BOLD signal through time-domain analysis. In addition, although some studies have begun to attempt to use frequency domain analysis to reveal the periodic activities of the BOLD signal, this method is usually limited by the data sampling rate and noise level, resulting in instability and difficulty in interpreting the results. Therefore, how to effectively combine time-domain and frequency domain analysis to comprehensively reveal the complex information in the BOLD time series is an important challenge faced by current research. Summary of the Invention

[0003] To solve the above technical problems, the present invention provides an fMRI analysis method and system based on a Transformer network combined with frequency domain analysis, which adds a frequency domain processing module and uses rFFT to first transform the original time-domain data to the complex frequency domain for frequency interpolation. Then, the interpolated frequency data is mapped back to the time domain to generate an extended segment ready for supervision. Visualization graphs are generated in the self-attention mechanism of the Transformer to improve the interpretability of the model, and the transparency of model decision-making is enhanced by generating Attention Map.

[0004] The present invention provides an fMRI analysis method based on a Transformer network combined with frequency domain analysis, and the method includes:

[0005] Obtain resting-state fMRI data;

[0006] Preprocess the resting-state fMRI data to obtain BOLD time series;

[0007] Construct a Transformer network structure, and add a frequency domain analysis module before the Transformer block to obtain a Transformer network combined with frequency domain analysis;

[0008] Input the extracted BOLD time series into a Transformer network combined with frequency domain analysis. Extract the attention weights in the time and spatial dimensions through the Self-Attention layer to generate the corresponding Attention Map;

[0009] Take the generated Attention Map with brain regions and time nodes as two-dimensional coordinates to show the degree of attention of the model on different brain regions and time nodes, and convert the Attention Map into a heat map.

[0010] Preferably, preprocess the resting-state fMRI data to obtain the BOLD time series, including:

[0011] Convert the subject data with the suffix.dcm to the.nii format;

[0012] Remove or correct the head motion in the fMRI data;

[0013] Perform temporal correction on the fMRI data after removing or correcting the head motion;

[0014] Align the temporally corrected fMRI data with the standard brain template;

[0015] Perform spatial smoothing on the aligned fMRI data using a Gaussian kernel function;

[0016] Use the Automatic Anatomical Labeling template AAL to divide the brain into 116 regions to define the brain region labels and generate the BOLD time series.

[0017] Preferably, the Transformer network structure consists of an input embedding layer, an encoder layer, and a classification layer; the encoder layer is stacked by 4 Blocks. Each Block first performs layer normalization, then captures global information through the self-attention mechanism, performs layer normalization again, and finally processes it through a multi-layer perceptron.

[0018] Preferably, add a frequency domain analysis module before the Transformer block, including:

[0019] Normalize the input BOLD time series;

[0020] Use the real fast Fourier transform to convert the input BOLD time series to the frequency domain;

[0021] Apply a low-pass filter in the frequency domain to remove the high-frequency components above a specific cut-off frequency;

[0022] Use a complex-valued linear layer to interpolate the frequency representation to generate an extended time series segment in the frequency domain;

[0023] Convert the interpolated frequency representation back to the time domain through inverse fast Fourier transform;

[0024] Apply invertible instance normalization to restore the BOLD time series before normalization;

[0025] Among them, the frequency-domain data enters the low-pass filtering stage after normalization, and the frequency components higher than the cut-off frequency are removed according to the set cut-off frequency to compress and retain the low-frequency information that meets the preset requirements;

[0026] Using a complex-valued linear layer to interpolate the frequency representation includes:

[0027]

[0028] In the formula, W is the weight of the complex-valued linear layer, is the normalized frequency representation, with a length half of the original time series, is the interpolated frequency representation.

[0029] Preferably, the generation formula of the Attention Map is:

[0030]

[0031] In the formula, Q is the query matrix, K is the key matrix, d k is the dimension of the key vector, QK T Calculate the inner product of Q and K, representing the correlation between the current sequence element and other sequence elements.

[0032] Preferably, the formula for converting the Attention Map into a heat map is:

[0033] Heatmap(i, j) = f(Attention Map(i, j))

[0034] In the formula, f(x) is the color mapping function, i is the time node, and j is the brain region.

[0035] The present invention also provides an fMRI analysis system based on a Transformer network combined with frequency-domain analysis. The system is used to implement any one of the above methods. The system includes: an acquisition module, a preprocessing module, a construction module, an extraction module, and a conversion module;

[0036] The acquisition module is used to acquire resting-state fMRI data;

[0037] The preprocessing module is used to preprocess the resting-state fMRI data to obtain the BOLD time series;

[0038] The building block is used to construct a Transformer network structure, and a frequency domain analysis module is added before the Transformer block to obtain a Transformer network combined with frequency domain analysis;

[0039] The extraction module is used to input the extracted BOLD time series into the Transformer network combined with frequency domain analysis, and extract the attention weights in the time and space dimensions through the Self-Attention layer to generate the corresponding Attention Map;

[0040] The conversion module is used to display the attention degree of the model in different brain regions and time nodes with the brain regions and time nodes as two-dimensional coordinates for the generated Attention Map, and convert the Attention Map into a heat map.

[0041] Compared with the prior art, the beneficial effects of the present invention are:

[0042] By introducing a frequency domain processing module, the present invention uses rFFT to convert the original time domain data to the complex frequency domain for frequency interpolation, and maps the interpolated frequency data back to the time domain, thereby effectively capturing the frequency components and periodic activities in the BOLD time series data. The present invention proposes to use the self-attention mechanism to generate a visual heat map to provide reference information for subsequent doctor diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the present invention, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0044] Figure 1 It is a flowchart of an fMRI analysis method based on a Transformer network combined with frequency domain analysis according to an embodiment of the present invention;

[0045] Figure 2 It is a flowchart of preprocessing the subject image according to an embodiment of the present invention;

[0046] Figure 3 It is a flowchart of the frequency domain analysis module according to an embodiment of the present invention;

[0047] Figure 4 It is an operation diagram of the frequency domain analysis module according to an embodiment of the present invention;

[0048] Figure 5 It is an operation diagram of the Transformer Block module according to an embodiment of the present invention. Detailed implementation manners

[0049] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0050] It should be noted that unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should have the ordinary meaning understood by those of ordinary skill in the art belonging to the field of the present disclosure. The "first", "second" and similar terms used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. The terms such as "include" or "comprise" mean that the elements or objects appearing before this term cover the elements or objects listed after this term and their equivalents, without excluding other elements or objects. The terms such as "connect" or "couple" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The terms such as "upper", "lower", "left", "right" are only used to represent relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0051] To make the above objects, features and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific implementation manners.

[0052] Embodiment 1

[0053] As Figure 1 shown, this embodiment provides an AD classification method based on the combination of the Transformer network and frequency domain analysis, including:

[0054] Step S1: Obtain the publicly available resting-state fMRI data from the ADNI database.

[0055] Step S2: Use the DPABI toolbox on the Matlab platform to preprocess the 4D fMRI to obtain a BOLD time series of 140*116.

[0056] Specifically, preprocess the images of 62 AD patients and 86 normal subjects downloaded from the ADNI database. As Figure 2 shown, the specific preprocessing process is as follows:

[0057] Step S21: Convert the subject data with the suffix.dcm to the.nii format.

[0058] Step S22: Remove or correct head motion in fMRI data to eliminate the impact of head motion on the data;

[0059] Step S23: Perform temporal correction to eliminate the differences caused by the acquisition times of different slices;

[0060] Step S24: Align the fMRI images with a standard brain template for subsequent data analysis and comparison;

[0061] Step S25: Perform spatial smoothing using a Gaussian kernel function to eliminate noise;

[0062] Step S26: Use the Automated Anatomical Labeling template AAL to divide the brain into 116 regions to define brain region labels and generate a 140 * 116 time series.

[0063] Step S3: Construct a Transformer network structure, and use the 140 * 116 BOLD time series extracted from the preprocessing as the input of the Transformer network.

[0064] Specifically, the Transformer network structure consists of an input embedding layer, an encoder layer, and a classification layer. The encoder layer is stacked by 4 Blocks. Each Block first performs layer normalization to stabilize the training process and accelerate convergence. Then, the self-attention mechanism is used to capture the global information of the input data. After layer normalization again, the Block further processes the data through a multi-layer perceptron.

[0065] Step S4: Add a frequency domain analysis module before the Transformer block. The frequency domain analysis module is as Figure 3 、 Figure 4 shown. The specific steps are as follows:

[0066] Step S41: Normalize the input time series so that its mean is zero to eliminate the offset in the data.

[0067] Step S42: Use the real fast Fourier transform to convert the input time series data to the frequency domain;

[0068] Step S43: Apply a low-pass filter in the frequency domain to remove high-frequency components above a specific cut-off frequency;

[0069] Specifically, after the frequency domain data is normalized, it enters the low-pass filtering stage. According to the set cut-off frequency, the frequency components above the cut-off frequency are removed to compress and retain important low-frequency information at the same time.

[0070] Step S44: Use a complex-valued linear layer to interpolate the frequency representation to generate an extended time series segment in the frequency domain;

[0071] Specifically, after removing high-frequency information, a complex-valued linear layer is used to interpolate the frequency representation to scale and shift the amplitude and phase. The specific interpolation formula is as follows:

[0072]

[0073] where W is the weight of the complex-valued linear layer, is the normalized frequency representation, with a length equal to half of the original time series, is the interpolated frequency representation.

[0074] To control the output length of the model, the present invention defines an interpolation rate η. The frequency interpolation operates on the normalized complex frequency representation, and the defined formula is as follows:

[0075]

[0076] where L O is the output length, L i is the input length, and η freq is the frequency domain interpolation rate.

[0077] Step S45: Convert the interpolated frequency representation back to the time domain through the inverse fast Fourier transform;

[0078] Step S46: Apply invertible instance normalization to restore the time series before normalization;

[0079] Step S5: Input the extracted 140*116 BOLD time series into the Transformer network combined with frequency domain analysis, and extract the attention weights in the time and space dimensions through the Self-Attention layer to generate the corresponding Attention Map, as Figure 5 shown.

[0080] Specifically, the generation formula of the Attention Map is:

[0081]

[0082] where Q is the query matrix, K is the key matrix, and d k is the dimension of the key vector. QK T calculates the inner product of Q and K, representing the correlation between the current sequence element and other sequence elements.

[0083] Step S6: The generated Attention Map uses the brain regions and time nodes as two-dimensional coordinates to show the attention degree of the model in different brain regions and time nodes, and converts the Attention Map into a heat map.

[0084] Specifically, the formula for converting the Attention Map into a heatmap is as follows:

[0085] Heatmap(i, j) = f(Attention Map(i, j))

[0086] Wherein, f(x) is a color mapping function, i is a time node, and j is a brain region.

[0087] The technical solution of the present invention

[0088] By introducing a frequency domain processing module, the present invention uses rFFT to convert the original time domain data to the complex frequency domain for frequency interpolation, and maps the interpolated frequency data back to the time domain, thereby effectively capturing the frequency components and periodic activities in the BOLD time series data. The present invention proposes to use the self-attention mechanism to generate a visual heatmap to provide reference information for subsequent doctor diagnosis.

[0089] It should be noted that the method of the embodiments of the present disclosure can be executed by a single device, such as a computer or a server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In such a distributed scenario, one of the multiple devices can only execute one or more steps of the method of the embodiments of the present disclosure, and these multiple devices will interact with each other to complete the described method.

[0090] It should be noted that some embodiments of the present disclosure are described above. Other embodiments are within the scope of the appended claims. In some cases, it should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention. The actions or steps recited in the claims can be executed in a different order than in the above embodiments and still achieve the desired results. Additionally, the processes depicted in the figures do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0091] Embodiment 2

[0092] Based on the same inventive concept, corresponding to the method of any of the above embodiments, the present invention further provides an fMRI analysis system based on a Transformer network combined with frequency domain analysis. The system is used to implement the method of any one of the above, and the system includes: an acquisition module, a preprocessing module, a construction module, an extraction module, and a conversion module;

[0093] The acquisition module is used to acquire resting state fMRI data;

[0094] The preprocessing module is used to preprocess the resting-state fMRI data to obtain the BOLD time series;

[0095] The construction module is used to construct a Transformer network structure, add a frequency-domain analysis module before the Transformer block, and obtain a Transformer network combined with frequency-domain analysis;

[0096] The extraction module is used to input the extracted BOLD time series into the Transformer network combined with frequency-domain analysis, extract the attention weights in the time and space dimensions through the Self-Attention layer, and generate the corresponding AttentionMap;

[0097] The conversion module is used to use the brain region and time node as two-dimensional coordinates for the generated Attention Map to display the attention degree of the model on different brain regions and time nodes, and convert the Attention Map into a heat map.

[0098] In this embodiment, preprocessing the resting-state fMRI data to obtain the BOLD time series includes:

[0099] Convert the subject data with the suffix.dcm to the.nii format;

[0100] Remove or correct the head movement in the fMRI data;

[0101] Perform temporal correction on the fMRI data after removing or correcting the head movement;

[0102] Align the temporally corrected fMRI data with the standard brain template;

[0103] Perform spatial smoothing on the aligned fMRI data using a Gaussian kernel function;

[0104] Use the Automatic Anatomical Labeling template AAL to divide the brain into 116 regions to define brain region labels and generate the BOLD time series.

[0105] In this embodiment, the Transformer network structure consists of an input embedding layer, an encoder layer, and a classification layer; the decoder layer is stacked by 4 Blocks. Each Block first performs layer normalization, then captures global information through the self-attention mechanism, performs layer normalization again, and finally processes through a multi-layer perceptron.

[0106] In this embodiment, adding a frequency-domain analysis module before the Transformer block includes:

[0107] Normalize the input BOLD time series;

[0108] Use the real fast Fourier transform to convert the input BOLD time series to the frequency domain;

[0109] Apply a low-pass filter in the frequency domain to remove high-frequency components above a specific cut-off frequency;

[0110] Use a complex-valued linear layer to interpolate the frequency representation to generate an extended time series segment in the frequency domain;

[0111] Convert the interpolated frequency representation back to the time domain through the inverse fast Fourier transform;

[0112] Apply invertible instance normalization to restore the BOLD time series before normalization;

[0113] Among them, the frequency domain data enters the low-pass filtering stage after normalization, and the frequency components above the cut-off frequency are removed according to the set cut-off frequency to compress while retaining the low-frequency information that meets the preset requirements;

[0114] Using a complex-valued linear layer to interpolate the frequency representation includes:

[0115]

[0116] Where W is the weight of the complex-valued linear layer, is the normalized frequency representation, with a length half of the original time series, is the interpolated frequency representation.

[0117] In this embodiment, the generation formula of the Attention Map is:

[0118]

[0119] Where Q is the query matrix, K is the key matrix, d k is the dimension of the key vector, QK T Calculate the inner product of Q and K, representing the correlation between the current sequence element and other sequence elements.

[0120] In this embodiment, the formula for converting the Attention Map to a heat map is:

[0121] Heatmap(i, j) = f(Attention Map(i, j))

[0122] Where f(x) is the color mapping function, i is the time node, and j is the brain region.

[0123] The system of the above embodiments is used to implement the corresponding fMRI analysis method based on the combination of Transformer network and frequency domain analysis in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be elaborated here.

[0124] It should be noted that the above fMRI analysis system based on the combination of Transformer network and frequency domain analysis is embodied in the form of functional units. The term "module" here can be implemented in the form of software and / or hardware, and no specific limitation is made thereto.

[0125] For example, the "module" can be a software program, a hardware circuit, or a combination of both that implements the above functions. The hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, a processor (such as a shared processor, a dedicated processor, or a group of processors, etc.) for executing one or more software or firmware programs, and a memory, a merged logic circuit, and / or other suitable components that support the described functions.

[0126] The embodiments of the present invention are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omission, modification, equivalent substitution, improvement, etc. made within the spirit and principle of the embodiments of the present disclosure shall be included within the protection scope of the present disclosure.

Claims

1. An fMRI analysis method based on Transformer network combined with frequency domain analysis, characterized in that: The method comprises: Acquire resting-state fMRI data; The resting-state fMRI data were preprocessed to obtain the BOLD time series; Construct the Transformer network structure and add the frequency domain analysis module before the Transformer block to obtain the Transformer network combined with frequency domain analysis; The extracted BOLD time series is input into the Transformer network combined with frequency domain analysis, and the attention weights in time and space dimensions are extracted through the Self-Attention layer to generate the corresponding Attention Map; The generated Attention Map uses brain regions and time nodes as two-dimensional coordinates to show the degree of attention of the model on different brain regions and time nodes, and converts the Attention Map into a heat map.

2. The method according to claim 1, characterized in that The resting state fMRI data is preprocessed to obtain the BOLD time series including: Convert the test data with the suffix .dcm to the suffix .nii format; Remove or correct head motion in fMRI data; Temporal correction of fMRI data to remove or correct for head motion; Align the time-corrected fMRI data to a standard brain template; The aligned fMRI data were spatially smoothed using a Gaussian kernel function; The brain was divided into 116 regions using the automated anatomical labeling template AAL to define brain region labels and generate BOLD time series.

3. The method according to claim 1, characterized in that The Transformer network structure consists of an input embedding layer, an encoder layer, and a classification layer; the encoder layer is composed of 4 stacked blocks, each of which first performs layer normalization, then captures global information through a self-attention mechanism, performs layer normalization again, and finally processes it through a multi-layer perceptron.

4. The method according to claim 3, characterized in that Adding frequency domain analysis module before Transformer block includes: Normalize the input BOLD time series; The input BOLD time series was converted to the frequency domain using real fast Fourier transform; Apply a low-pass filter in the frequency domain to remove high-frequency components above a certain cutoff frequency; The frequency representation is interpolated using a complex-valued linear layer to generate extended time series segments in the frequency domain; The interpolated frequency representation is converted back to the time domain by inverse fast Fourier transform; Apply reversible instance normalization to restore the BOLD time series before normalization; Among them, the frequency domain data enters the low-pass filtering stage after normalization, and the frequency components higher than the cutoff frequency are removed according to the set cutoff frequency to compress and retain the low-frequency information that meets the preset requirements; Interpolating the frequency representation using a complex-valued linear layer involves: Where W is the weight of the complex-valued linear layer, is the normalized frequency representation, half the length of the original time series, is the frequency representation after interpolation.

5. The method according to claim 1, characterized in that The generation formula of Attention Map is: Where Q is the query matrix, K is the key matrix, and d k is the dimension of the key vector, QK T Calculate the inner product of Q and K, which represents the correlation between the current sequence element and other sequence elements.

6. The method according to claim 5, characterized in that The formula for converting Attention Map into heat map is: Heatmap(i,j)=f(Attention Map(i,j)) Where f(x) is the color mapping function, i is the time node, and j is the brain region.

7. An fMRI analysis system based on Transformer network combined with frequency domain analysis, the system is used to implement the method according to any one of claims 1 to 6, characterized in that: The system comprises: an acquisition module, a preprocessing module, a construction module, an extraction module and a conversion module; The acquisition module is used to acquire resting-state fMRI data; The preprocessing module is used to preprocess the resting state fMRI data to obtain the BOLD time series; The construction module is used to construct a Transformer network structure, add a frequency domain analysis module before the Transformer block, and obtain a Transformer network combined with frequency domain analysis; The extraction module is used to input the extracted BOLD time series into the Transformer network combined with frequency domain analysis, extract the attention weights in the time and space dimensions through the Self-Attention layer, and generate the corresponding AttentionMap; The conversion module is used to use the generated Attention Map as two-dimensional coordinates of brain regions and time nodes to display the degree of attention of the model on different brain regions and time nodes, and convert the Attention Map into a heat map.

Images