Functional magnetic resonance image decoding method based on explicit brain region interaction modeling

Abstract

The application discloses a functional magnetic resonance image decoding method based on explicit brain region interaction modeling. First, an encoder network is trained in a supervised manner through image-magnetic resonance signals, then a decoder network is trained in a supervised manner through magnetic resonance-image signals, and meanwhile, a decoding-encoding structure and an encoding-decoding structure formed by stacking the encoder and the decoder network are trained in a self-supervised manner using the magnetic resonance signals and the image signals, then an information interaction network containing brain region and inter-brain region information interaction modeling is trained in a combination of the supervised and unsupervised manners, finally, the training is completed using the images and the magnetic resonance signals provided by a data set, the magnetic resonance signals to be decoded are input into the information interaction network and the decoder network to obtain a final image decoding reconstruction result. The method can effectively perform image decoding reconstruction based on functional magnetic resonance signals, and provides an effective image decoding reconstruction model for various visual function exploration research based on magnetic resonance signals.

Description

Technical Field

[0001] This invention belongs to the fields of medical image processing technology and medical data processing technology, specifically relating to a functional magnetic resonance imaging decoding method based on explicit brain region interaction modeling. Background Technology

[0002] Visual neural information decoding explores the relationship between external image stimuli and their corresponding brain neural signals by acquiring visual image stimuli. Classified by task type, visual neural information decoding can be mainly divided into three parts: semantic classification, image recognition, and image reconstruction. Among these, stimulus-image decoding and reconstruction is the most valuable yet most challenging problem. During the viewing of natural image visual stimuli, functional magnetic resonance imaging (fMRI) signals representing brain activity are recorded. Stimulus-image reconstruction requires reconstructing the stimulus image from the given fMRI signals. Developing stimulus-image reconstruction models is beneficial for exploring the brain's information processing mechanisms and can, in turn, guide research and applications of brain-like computing, thus promoting the development of machine intelligence. With the rapid development of deep neural networks, current work has used multilayer perceptron networks and convolutional neural networks to extract features from fMRI signals and combine them with image generation models to achieve stimulus-image decoding and reconstruction.

[0003] For models that reconstruct natural stimulus images based on functional magnetic resonance imaging (fMRI) signal data, the paper "Beliy R, Gaziv G, Hoogi A, et al. From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI. Advances in Neural Information Processing Systems, 2019, 32" proposed a self-supervised image reconstruction model. This model can be trained using unlabeled fMRI signal data and natural image data respectively, which expands the amount of data available for model training to a certain extent and obtains better image decoding and reconstruction results. However, this method ignores the intra-regional and inter-regional information interactions that are widely present in various regions of the brain. When the amount of paired fMRI signal and image data is limited, there is still room for improvement in the quality of stimulus image decoding and reconstruction results. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a functional magnetic resonance imaging (fMRI) image decoding method based on explicit brain region interaction modeling. By considering the inherent information interaction nature within the brain when processing medical MRI data through information interaction networks within and between brain regions, this method can extract features from MRI data more effectively. Furthermore, by using paired and unpaired training label data, the decoded and reconstructed image can implicitly learn the data distribution of the target image and the data distribution of the natural image, thereby improving the quality of the decoded and reconstructed image.

[0005] The technical solution adopted in this invention is: a functional magnetic resonance imaging decoding method based on explicit brain region interaction modeling, the specific steps of which are as follows:

[0006] S1. Use a basic neural network to extract fixed features from the input image signal, and then use the feature extraction module and feature mapping module to perform data transformation to obtain the estimation result of the magnetic resonance signal corresponding to the image. Train the encoder network in a supervised manner.

[0007] S2. The feature mapping module is used to perform data transformation on the input magnetic resonance signal, and then the image generation module generates the image signal estimation result corresponding to the magnetic resonance signal. The decoder network is trained in a supervised manner.

[0008] S3. Using a stacked decoder network and encoder network decoder-encoder structure, the input magnetic resonance signal is reconstructed, and the decoder network is trained in a self-supervised manner.

[0009] S4. Using a stacked encoder-decoder network and decoder network encoder-decoder network encoder-decoder structure, reconstruct the input image signal, and train the decoder network in a self-supervised manner;

[0010] S5. Use an information interaction network that includes information interaction within and between brain regions to explicitly model brain region interactions of the input magnetic resonance signal. The obtained interactive magnetic resonance signal is input into the decoder to obtain the estimated result of the decoded and reconstructed image. The information interaction network is trained by using paired and unpaired labeled image data and by combining supervised and unsupervised methods.

[0011] S6. Use the image signal and magnetic resonance signal provided by the dataset to complete the training of steps S1-S5, and then input the magnetic resonance signal to be decoded into the information interaction network and decoder network to obtain the final image decoding and reconstruction result.

[0012] Furthermore, step S1 is specifically as follows:

[0013] Using a basic neural network to extract fixed features from the input image signal, the expression is as follows:

[0014] F image-fixed =Netbackbone (I input (1)

[0015] Among them, F image-fixed Represents the extracted fixed image features, Net backbone I represents a fixed pre-trained base neural network. input This represents the input image signal.

[0016] Then, the feature extraction module and the feature mapping module are used to perform data transformation to obtain the estimated result of the magnetic resonance signal corresponding to the image, as shown in the following expression:

[0017] F image-dynamic =Net extraction (F image-fixed (2)

[0018] D estimated =Net mapping-image-to-fmri (F image-dynamic (3)

[0019] Among them, F image-dynamic Represents the extracted dynamic image features, Net extraction D represents a trainable feature extraction module. estimated Net represents the magnetic resonance signal estimation result. mapping-image-to-fmri This represents the feature mapping module from image to magnetic resonance signal.

[0020] Finally, the encoder network is trained using a supervised method:

[0021] Encoder = Compose(Net) backbone Net extraction Net mapping-image-to-fmri (4)

[0022] Encoder * =Train supervised (Encoder, D) estimated D groundtruth Loss stage1 (5)

[0023] Here, Encoder represents the entire encoder network, and Compose represents the concatenation and combination operation of the various network modules. * The term "Train" represents the optimal encoder network obtained after training convergence. supervised D represents the supervised training method. groundtruth Loss represents the magnetic resonance signal label data provided during supervised training. stage1This represents the loss function used during the supervised training phase of the encoder network, including the mean squared error loss function and the cosine similarity error loss function.

[0024] Furthermore, step S2 is specifically as follows:

[0025] The feature mapping module is used to perform data transformation on the input magnetic resonance signal, as shown in the following expression:

[0026] F image =Net mapping-fmri-t。-image (D input (6)

[0027] Among them, F image Net represents the transformed image features. mapping-fmri-to-image D represents the feature mapping module from magnetic resonance signals to image features. input This indicates the input magnetic resonance signal.

[0028] The image generation module then generates the image signal estimation result corresponding to the magnetic resonance signal, and the decoder network is trained in a supervised manner, as follows:

[0029] I estimated =Net image-generation (F image (7)

[0030] Among them, Net image-generation I represents the image generation module. estimated This represents the image signal estimation result.

[0031] Decoder = Compose(Net) mapping-fmri-to-image Net image-generation (8)

[0032]

[0033] Here, Decoder represents the entire decoder network. I represents the optimal decoder network obtained by convergence during the supervised training phase of the decoder network. groundtruth Loss represents the image label data provided during supervised training. stage2 This indicates the loss function used during the supervised training phase of the decoder network, including the mean absolute error loss function, the feature similarity error loss function, and the smoothing regularization loss function.

[0034] Furthermore, step S3 is specifically as follows:

[0035] The input magnetic resonance signal is reconstructed using a decoder-encoder structure with a stacked decoder network and encoder network, as shown in the following expression:

[0036] DEStack=Compose(Decoder,Encoder * (10)

[0037] D estimated =DEStack(D input (11)

[0038] DEStack represents a decoder-encoder structure consisting of stacked decoder and encoder networks.

[0039] Then, the decoder network is trained using a self-supervised approach:

[0040]

[0041] in, This represents the optimal decoder network obtained by the self-supervised training phase of the decoder network based on magnetic resonance signals. self-supervised This indicates a self-supervised training method, Loss stage3 This represents the loss function used in the self-supervised training phase of the decoder network based on magnetic resonance signals, including the mean squared error loss function and the cosine similarity error loss function.

[0042] Furthermore, step S4 is specifically as follows:

[0043] The input image signal is reconstructed using an encoder-decoder structure with stacked encoder and decoder networks, as shown in the following expression:

[0044] EDStack = Compose(Encoder*) ， Decoder) (13)

[0045] I estimated =EDStack(I input (14)

[0046] EDStack represents an encoder-decoder structure consisting of stacked encoder and decoder networks.

[0047] Then, the decoder network is trained using a self-supervised approach:

[0048]

[0049] in, Loss represents the optimal decoder network obtained by the self-supervised training phase of the image signal-based decoder network. stage4This represents the loss function used in the self-supervised training phase of the image signal-based decoder network, including the mean absolute error loss function, the feature similarity error loss function, and the smoothing regularization loss function.

[0050] Furthermore, step S5 is specifically as follows:

[0051] Explicit brain region interaction modeling is performed on the input magnetic resonance imaging (MRI) signal using a network of information interaction within and between brain regions. The resulting interactive MRI signal is then input into a decoder to obtain the estimated decoded and reconstructed image, expressed as follows:

[0052]

[0053]

[0054]

[0055] D inter =Net interaction-between-roi (D intra (19)

[0056] I estimated =Decoder(D inter (20)

[0057] Where Split represents the vector partitioning operation. Representing different brain regions' magnetic resonance imaging (MRI) signals, i = {V1, V2, V3, V4, LOC, FFA, PPA}, representing various brain regions, Net interaction-in-roi This represents the information exchange network within brain regions. This represents the magnetic resonance signal after interaction within brain regions; Merge represents the vector merging operation; D intra Net represents the post-interaction magnetic resonance imaging (MRI) signals within the merged brain regions. interaction-between-roi D represents the information exchange network between brain regions. inter This indicates the magnetic resonance signal after interaction between brain regions.

[0058] Then, using paired and unpaired labeled image data, and combining supervised and unsupervised methods, an information interaction network is trained, as shown in the following expression:

[0059] InteractionNet = Compose(Net) interaction-in-roi Net interaction-between-roi ) (twenty one)

[0060]

[0061] InteractionNet represents an information interaction network composed of information interaction networks between brain regions. * I represents the optimal information interaction network obtained after training convergence. groundtruth-paired Loss represents the paired image label data provided during training. stage5-supervised This refers to the loss functions used in the supervised training phase of the information interaction network, including the mean absolute error loss function, the feature similarity error loss function, and the smoothing regularization loss function; Tratin unsupervised Indicates unsupervised training method, I groundtruth-unpaired Loss represents the unpaired image label data provided during training. stage5-unsupervised This refers to the loss function used in the unsupervised training phase of the information interaction network training process, including the adversarial loss function.

[0062] Furthermore, step S6 is specifically as follows:

[0063] The training steps S1-S5 are completed using the image and magnetic resonance signals provided in the dataset, ensuring that the training of each component of the provided network converges. The expression is as follows:

[0064] Encoder converge =Optimize(Encoder) * ) (twenty three)

[0065]

[0066] InteractionNet converge =Optimize(InteractionNet) * (25)

[0067] Among them, Encoder converge This represents the encoder network after training convergence, where Optimize represents the model optimization operation, and Decoder represents the encoder network after training convergence. converge This represents the decoder network after training convergence; JointOptimize represents the multi-objective joint model optimization operation; InteractionNet converge This represents the information exchange network after training convergence.

[0068] Then, the magnetic resonance signal to be decoded is input into the information interaction network and the decoder network to obtain the final image decoding and reconstruction result, as shown in the following expression:

[0069] I estimated-test =Decoder converge (InteractionNet converge (Dinput-test (26)

[0070] Among them, I estimated-test D represents the decoded and reconstructed image signal estimated during the testing phase. input-test This represents the magnetic resonance signal data to be decoded input during the testing phase.

[0071] The beneficial effects of this invention are as follows: The method of this invention first trains an encoder network in a supervised manner using image-magnetic resonance (MRI) signals, then trains a decoder network in a supervised manner using MRI-image signals. Simultaneously, it uses MRI signals and image signals to self-supervisedly train a decoder-encoder structure and an encoder-decoder structure formed by stacking the encoder and decoder networks. Then, it combines supervised and unsupervised training to create an information interaction network that models information interactions within and between brain regions. Finally, it uses images and MRI signals provided by the dataset to complete the training, inputting the MRI signal to be decoded into the information interaction network and decoder network to obtain the final image decoding and reconstruction result. This method of the invention considers the inherent information interaction nature within the brain when processing medical MRI data through information interaction networks within and between brain regions. The information interaction network can more effectively extract features from MRI data. Furthermore, through paired and unpaired training label data, the decoded and reconstructed image can implicitly learn the data distribution of the target image and the data distribution of the natural image, improving the quality of the decoded and reconstructed image. It can effectively perform image decoding and reconstruction based on functional MRI signals, providing an effective image decoding and reconstruction model for subsequent research on visual function exploration and brain-computer interface applications based on MRI signals. Attached Figure Description

[0072] Figure 1 This is a flowchart of a functional magnetic resonance imaging decoding method based on explicit brain region interaction modeling according to the present invention.

[0073] Figure 2 The following is a flowchart of the model at each stage in the embodiments of the present invention.

[0074] Figure 3 This is a network structure diagram of the encoder network provided in an embodiment of the present invention.

[0075] Figure 4 This is a network structure diagram of the decoder network provided in an embodiment of the present invention.

[0076] Figure 5 This is a network structure diagram of the information interaction network within the brain region provided in an embodiment of the present invention.

[0077] Figure 6 This is a network structure diagram of the information interaction network between brain regions provided in an embodiment of the present invention.

[0078] Figure 7 This is a network structure diagram of the image signal discrimination network provided in an embodiment of the present invention.

[0079] Figure 8 This is an image decoding result diagram from an embodiment of the present invention. Detailed Implementation

[0080] The method of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0081] like Figure 1 The flowchart of a functional magnetic resonance imaging decoding method based on explicit brain region interaction modeling of the present invention is shown below. The specific steps are as follows:

[0082] S1. Use a basic neural network to extract fixed features from the input image signal, and then use the feature extraction module and feature mapping module to perform data transformation to obtain the estimation result of the magnetic resonance signal corresponding to the image. Train the encoder network in a supervised manner.

[0083] S2. The feature mapping module is used to perform data transformation on the input magnetic resonance signal, and then the image generation module generates the image signal estimation result corresponding to the magnetic resonance signal. The decoder network is trained in a supervised manner.

[0084] S3. Using a stacked decoder network and encoder network decoder-encoder structure, the input magnetic resonance signal is reconstructed, and the decoder network is trained in a self-supervised manner.

[0085] S4. Using a stacked encoder-decoder network and decoder network encoder-decoder network encoder-decoder structure, reconstruct the input image signal, and train the decoder network in a self-supervised manner;

[0086] S5. Use an information interaction network that includes information interaction within and between brain regions to explicitly model brain region interactions of the input magnetic resonance signal. The obtained interactive magnetic resonance signal is input into the decoder to obtain the estimated result of the decoded and reconstructed image. The information interaction network is trained by using paired and unpaired labeled image data and by combining supervised and unsupervised methods.

[0087] S6. Use the image signal and magnetic resonance signal provided by the dataset to complete the training of steps S1-S5 (to make the training of each component of the provided network converge), and then input the magnetic resonance signal to be decoded into the information interaction network and the decoder network to obtain the final image decoding and reconstruction result.

[0088] in, Figure 2 This is a flowchart illustrating the specific process of each stage of the model in this embodiment of the invention.

[0089] In this embodiment, step S1 is specifically as follows:

[0090] Using a basic neural network to extract fixed features from the input image signal, the expression is as follows:

[0091] F image-fixed =Net backbone (I input (1)

[0092] Among them, F image-fixed Represents the extracted fixed image features, Net backbone I represents a fixed pre-trained base neural network. input This represents the input image signal. The pre-trained basic neural network structure is as follows: Figure 3 As shown.

[0093] Then, the feature extraction module and the feature mapping module are used to perform data transformation to obtain the estimated result of the magnetic resonance signal corresponding to the image, as shown in the following expression:

[0094] F image-dynamic =Ne textraction (F image-fixed (2)

[0095] D estimated =Net mapping-image-to-fmri (F image-dynamic (3)

[0096] Among them, F image-dynamic Represents the extracted dynamic image features, Net extraction D represents a trainable feature extraction module. estimated Net represents the magnetic resonance signal estimation result. mapping-image-to-fmri This represents the feature mapping module from image to magnetic resonance signal. The network structure of the feature extraction module and the feature mapping module is as follows: Figure 3 As shown.

[0097] Finally, the encoder network is trained using a supervised method:

[0098] Encoder = Compose(Net) backbone Net extraction Net mapping-image-to-fmri (4)

[0099] Encoder * =Train supervised (Emcoder, D) estimated D groundtruth Loss stage1 (5)

[0100] Here, Encoder represents the entire encoder network, and Compose represents the concatenation and combination operation of the various network modules.* The term "Train" represents the optimal encoder network obtained after training convergence. supervised D represents the supervised training method. groundtruth Loss represents the magnetic resonance signal label data provided during supervised training. stage1 This represents the loss function used during the supervised training phase of the encoder network. The overall encoder structure is as follows: Figure 3 As shown.

[0101] In this embodiment, step S1 uses an input image with image size [112, 112, 3], where the first two dimensions represent the image width and height, and the last dimension represents the number of image channels. After feature extraction by the pre-trained basic neural network, the dimensions change to [56, 56, 96]. During the feature extraction module, the dimensions change to [28, 28, 32] and [14, 14, 32]. After the flattening operation of the feature mapping module, the dimensions change to

[6272] . The feature mapping module outputs a dimension of

[7280] .

[0102] In this embodiment, the loss functions used in step S1 include the mean squared error loss function and the cosine similarity error loss function, specifically the Loss function. stage1 =Loss D-MSE +Loss D-CosSim .

[0103] in, Let N represent the mean squared error loss function, and let N represent the number of elements in the magnetic resonance signal vector. This represents the i-th element in the vector of magnetic resonance signal estimation results. This represents the i-th element in the magnetic resonance signal label vector, where i = 1, 2, ..., N. This represents the cosine similarity error loss function.

[0104] In this embodiment, step S2 is specifically as follows:

[0105] The feature mapping module is used to perform data transformation on the input magnetic resonance signal, as shown in the following expression:

[0106] F image =Net mapping-fmri-to-image (D input (6)

[0107] Among them, F image Net represents the transformed image features. mapping-fmri-to-image D represents the feature mapping module from magnetic resonance signals to image features. input This represents the input magnetic resonance signal. The feature mapping module network structure is as follows: Figure 4 As shown.

[0108] The image generation module then generates the image signal estimation result corresponding to the magnetic resonance signal, and the decoder network is trained in a supervised manner, as follows:

[0109] I estimated =Net image-generation ( F i mage (7)

[0110] Among them, Net image-generation I represents the image generation module. estimated This represents the image signal estimation result. The image generation module, such as... Figure 4 As shown.

[0111] Decoder = Compose(Net) mapping-fmri-to-image Net image-generation (8)

[0112]

[0113] Here, Decoder represents the entire decoder network. I represents the optimal decoder network obtained by convergence during the supervised training phase of the decoder network. groundtruth Loss represents the image label data provided during supervised training. stage2 This represents the loss function used during the supervised training phase of the decoder network. The overall decoder structure is as follows: Figure 4 As shown.

[0114] In this embodiment, step S2 uses an input magnetic resonance signal with a size of

[7280] . This dimension represents the number of brain voxels contained in the magnetic resonance signal. After passing through the fully connected layer of the feature mapping module, the dimension is transformed to

[12544] , and after subsequent size adjustment operations, it becomes [14, 14, 64]. During the process of passing through the image generation module, the dimension changes to [28, 28, 64], [56, 56, 64], [112, 112, 64] and [112, 112, 3].

[0115] In this embodiment, the loss functions used in step S2 include the mean absolute error loss function, the feature similarity error loss function, and the smoothing regularization loss function, specifically the formula Loss. stage2 =Loss I MAE +Loss I Feature +Loss I Smooth .

[0116] in, Let H represent the mean absolute error loss function, H represent the height of the image signal matrix, and W represent the width of the image signal matrix. This represents the element at spatial location (i, j) in the image signal estimation result matrix. This represents the element at spatial position (i, j) in the image signal label matrix, where i = 1, 2, ..., H, j = 1, 2, ..., W. VGG19 represents the feature similarity error loss function; VGG19 represents the pre-trained feature extraction network. This represents the smoothing regularization loss function.

[0117] In this embodiment, step S3 is specifically as follows:

[0118] The input magnetic resonance signal is reconstructed using a decoder-encoder structure with a stacked decoder network and encoder network, as shown in the following expression:

[0119] DEStack=Compose(Decoder,Encoder * (10)

[0120] D estimated =DEStack(D input (11)

[0121] DEStack represents a decoder-encoder structure consisting of stacked decoder and encoder networks.

[0122] Then, the decoder network is trained using a self-supervised approach:

[0123]

[0124] in, This represents the optimal decoder network obtained by the self-supervised training phase of the decoder network based on magnetic resonance signals. self-supervised This indicates a self-supervised training method, Loss stage3 This represents the loss function used during the self-supervised training phase of the decoder network based on magnetic resonance signals.

[0125] In this embodiment, step S3 uses an input magnetic resonance signal with a size of

[7280] . This dimension represents the number of brain voxels contained in the magnetic resonance signal. After the decoding-encoding structure, the dimension remains unchanged and is

[7280] .

[0126] In this embodiment, the loss functions used in step S3 include the mean squared error loss function and the cosine similarity error loss function, specifically, the formula is: Loss stage3 =Loss D-MSE +Loss D-CosSim .

[0127] In this embodiment, step S4 is specifically as follows:

[0128] The input image signal is reconstructed using an encoder-decoder structure with stacked encoder and decoder networks, as shown in the following expression:

[0129] EDStack = Compose(Encoder) * (Decoder) (13)

[0130] I estimated =EDStack(I input (14)

[0131] EDStack represents an encoder-decoder structure consisting of stacked encoder and decoder networks.

[0132] Then, the decoder network is trained using a self-supervised approach:

[0133]

[0134] in, Loss represents the optimal decoder network obtained by the self-supervised training phase of the image signal-based decoder network. stage4 This represents the loss function used during the self-supervised training phase of the image signal-based decoder network.

[0135] In this embodiment, step S4 uses an input image with an image size of [112, 112, 3]. The first two dimensions represent the width and height of the image, and the last dimension represents the number of image channels. After the encoding-decoding structure, the dimensions remain unchanged and are [112, 112, 3].

[0136] In this embodiment, the loss functions used in step S4 include the mean absolute error loss function, the feature similarity error loss function, and the smoothing regularization loss function, specifically the formula Loss. stage4 =Loss I MAE +Loss I Feature +Loss I Smooth .

[0137] In this embodiment, step S5 is specifically as follows:

[0138] Explicit brain region interaction modeling is performed on the input magnetic resonance imaging (MRI) signal using a network of information interaction within and between brain regions. The resulting interactive MRI signal is then input into a decoder to obtain the estimated decoded and reconstructed image, expressed as follows:

[0139]

[0140]

[0141]

[0142] D inter =Net interaction-between-roi (D intra (19)

[0143] I estimated =Decoder(D inter (20)

[0144] Where Split represents the vector partitioning operation. Representing different brain regions' magnetic resonance imaging (MRI) signals, i = {V1, V2, V3, V4, LOC, FFA, PPA}, representing various brain regions, Net interaction-in-roi This represents the information exchange network within brain regions. This represents the magnetic resonance signal after interaction within brain regions; Merge represents the vector merging operation; D intra Net represents the post-interaction magnetic resonance imaging (MRI) signals within the merged brain regions. interaction-between-roi D represents the information exchange network between brain regions. inter This represents the magnetic resonance imaging (MRI) signal after interaction between brain regions. The information exchange networks within brain regions and between brain regions are as follows: Figure 5 and Figure 6 As shown.

[0145] Then, using paired and unpaired labeled image data, and combining supervised and unsupervised methods, an information interaction network is trained, as shown in the following expression:

[0146] InteractionNet = Compose(Net) interaction-in-roi Net interaction-between-roi ) (twenty one)

[0147]

[0148] InteractionNet represents an information interaction network composed of information interaction networks between brain regions. * I represents the optimal information interaction network obtained after training convergence. groundtruth-paired Loss represents the paired image label data provided during training. stage5-supervised This represents the loss function used in the supervised training phase of the information interaction network. unsupervised Indicates unsupervised training method, I groundtruth-unpaired Loss represents the unpaired image label data provided during training. stage5-unsupervised This represents the loss function used in the unsupervised training phase of the information interaction network training process.

[0149] In this embodiment, step S5 uses an input magnetic resonance signal with an input size of

[7280] . This dimension represents the number of brain voxels contained in the magnetic resonance signal. After passing through the brain region interaction network, the dimension remains

[7280] . After passing through the brain region interaction network, the dimension remains

[7280] . After passing through the decoder network, the dimension changes to [112, 112, 3]. The paired image labels and unpaired image labels used in the model training process both have a size of [112, 112, 3]. Both the output image and the label image will pass through a discriminant network. The network structure of the discriminant network is as follows: Figure 7 As shown, the output probability map dimension becomes [14, 14, 1].

[0150] In this embodiment, the supervised loss function used in step S5 includes the mean absolute error loss function, the feature similarity error loss function, and the smoothing regularization loss function. The specific formula is Loss. stage5-supervised =Loss I MAE +Loss I Feature +Loss I Smooth .

[0151] In this embodiment, the unsupervised partial loss function used in step S5 includes an adversarial loss function, specifically the formula Loss. stage5-unsupervised =Loss I ADV-EN +Loss I ADV-D sc .

[0152] in, This represents the generator adversarial loss function. Discriminator represents the adversarial loss function. (i，j) Let represent the element at spatial position (i, j) in the probability estimation matrix obtained after the image signal passes through the discriminator network, where i = 1, 2, ..., H, j = 1, 2, ..., W.

[0153] In this embodiment, step S6 is specifically as follows:

[0154] The training steps S1-S5 are completed using the image and magnetic resonance signals provided in the dataset, ensuring that the training of each component of the provided network converges. The expression is as follows:

[0155] Encoder converge =Optimize(Encoder) * ) (twenty three)

[0156]

[0157] InteractionNetconverge =Optimize(InteractionNet) * (25)

[0158] Among them, Encoder converge This represents the encoder network after training convergence, where Optimize represents the model optimization operation, and Decoder represents the encoder network after training convergence. converge This represents the decoder network after training convergence. JointOptimize indicates the multi-objective joint model optimization operation (effectively indicating that steps S2-S4 are performed simultaneously). InteractionNet converge This represents the information exchange network after training convergence.

[0159] Then, the magnetic resonance signal to be decoded is input into the information interaction network and the decoder network to obtain the final image decoding and reconstruction result, as shown in the following expression:

[0160] I estimated-test =Decoder converge (InteractionNet converge (D input-test (26)

[0161] Among them, I estimated-test D represents the decoded and reconstructed image signal estimated during the testing phase. input-test This represents the magnetic resonance signal data to be decoded input during the testing phase.

[0162] In this embodiment, step S6 uses a test input magnetic resonance signal with a size of

[7280] . This dimension represents the number of brain voxels contained in the magnetic resonance signal. After passing through the information interaction network, the dimension remains unchanged at

[7280] . After passing through the decoder network, the output image size is [112, 112, 3]. A partial image decoding and reconstruction example image obtained in this embodiment is shown below. Figure 8 As shown.

[0163] In summary, the method of this invention takes into account the inherent information interaction nature within the brain when processing medical magnetic resonance imaging (MRI) data by utilizing information interaction networks within and between brain regions. Through these information interaction networks, features can be extracted more effectively from MRI data. Furthermore, by using paired and unpaired training label data, the decoded and reconstructed images can implicitly learn the data distribution of the target image and the data distribution of the natural image, improving the quality of the decoded and reconstructed images. This method effectively performs image decoding and reconstruction based on functional magnetic resonance imaging (fMRI) signals, providing an effective image decoding and reconstruction model for subsequent research on visual functions based on MRI signals and for brain-computer interface applications.

[0164] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A functional magnetic resonance imaging decoding method based on explicit brain region interaction modeling, the specific steps of which are as follows: S1. Use a basic neural network to extract fixed features from the input image signal, and then use the feature extraction module and feature mapping module to perform data transformation to obtain the estimation result of the magnetic resonance signal corresponding to the image. Train the encoder network in a supervised manner. S2. The feature mapping module is used to perform data transformation on the input magnetic resonance signal, and then the image generation module generates the image signal estimation result corresponding to the magnetic resonance signal. The decoder network is trained in a supervised manner. S3. Using a stacked decoder network and encoder network decoder-encoder structure, the input magnetic resonance signal is reconstructed, and the decoder network is trained in a self-supervised manner. S4. Using a stacked encoder-decoder network and decoder network encoder-decoder network encoder-decoder structure, reconstruct the input image signal, and train the decoder network in a self-supervised manner; S5. Use an information interaction network that includes information interaction within and between brain regions to explicitly model brain region interactions of the input magnetic resonance signal. The obtained interactive magnetic resonance signal is input into the decoder to obtain the estimated result of the decoded and reconstructed image. The information interaction network is trained by using paired and unpaired labeled image data and by combining supervised and unsupervised methods. Explicit brain region interaction modeling is performed on the input magnetic resonance imaging (MRI) signal using a network of information interaction within and between brain regions. The resulting interactive MRI signal is then input into a decoder to obtain the estimated decoded and reconstructed image, expressed as follows: ； ； ； ； ； in, This represents the vector partitioning operation. Indicates the input magnetic resonance signal. This represents the magnetic resonance signals from different brain regions. This represents the various regions within the brain. This represents the information exchange network within brain regions. This indicates the magnetic resonance signal after it has interacted with other brain regions. This represents a vector merging operation. This indicates the interaction of magnetic resonance imaging (MRI) signals within the merged brain regions. This represents a network of information exchange between brain regions. This represents the image signal estimation result. Represents the entire decoder network. This indicates the magnetic resonance signal after interaction between brain regions; Then, using paired and unpaired labeled image data, and combining supervised and unsupervised methods, an information interaction network is trained, as shown in the following expression: ； ； in, This refers to an information exchange network composed of information exchange networks between brain regions. This represents the optimal information interaction network obtained after training convergence. Indicates the method of supervised training. This refers to the paired image label data provided during the training process. This refers to the loss function used in the supervised training phase of the information interaction network training stage, including the mean absolute error loss function, the feature similarity error loss function, and the smoothing regularization loss function. This indicates unsupervised training methods. This refers to the unpaired image label data provided during the training process. This refers to the loss function used in the unsupervised training phase of the information interaction network training process, including the adversarial loss function. S6. Use the image signal and magnetic resonance signal provided by the dataset to complete the training of steps S1-S5, and then input the magnetic resonance signal to be decoded into the information interaction network and decoder network to obtain the final image decoding and reconstruction result.

2. The functional magnetic resonance imaging decoding method based on explicit brain region interaction modeling according to claim 1, characterized in that, The specific steps of S1 are as follows: Using a basic neural network to extract fixed features from the input image signal, the expression is as follows: ； in, This represents the extracted fixed image features. This represents a fixed pre-trained base neural network. Indicates the input image signal; Then, the feature extraction module and the feature mapping module are used to perform data transformation to obtain the estimated result of the magnetic resonance signal corresponding to the image, as shown in the following expression: ； ； in, This represents the extracted dynamic image features. This represents a trainable feature extraction module. This represents the result of the magnetic resonance signal estimation. This module represents the feature mapping from image to magnetic resonance signal. Finally, the encoder network is trained using a supervised method: ； ； in, Represents the entire encoder network. This indicates the operation of splicing and combining various network modules. This represents the optimal encoder network obtained after training convergence. Indicates the method of supervised training. This refers to the magnetic resonance signal label data provided during supervised training. This represents the loss function used during the supervised training phase of the encoder network, including the mean squared error loss function and the cosine similarity error loss function.

3. The functional magnetic resonance imaging decoding method based on explicit brain region interaction modeling according to claim 1, characterized in that, Step S2 is as follows: The feature mapping module is used to perform data transformation on the input magnetic resonance signal, as shown in the following expression: ； in, Indicates the features of the converted image. This represents the feature mapping module from magnetic resonance signals to image features. Indicates the input magnetic resonance signal; The image generation module then generates the image signal estimation result corresponding to the magnetic resonance signal, and the decoder network is trained in a supervised manner, as follows: ； in, This represents the image generation module. This represents the image signal estimation result; ； ； in, Represents the entire decoder network. This represents the optimal decoder network obtained by convergence during the supervised training phase of the decoder network. This refers to the image label data provided during supervised training. This indicates the loss function used during the supervised training phase of the decoder network, including the mean absolute error loss function, the feature similarity error loss function, and the smoothing regularization loss function.

4. The functional magnetic resonance imaging decoding method based on explicit brain region interaction modeling according to claim 1, characterized in that, Step S3 is as follows: The input magnetic resonance signal is reconstructed using a decoder-encoder structure with a stacked decoder network and encoder network, as shown in the following expression: ； ； in, This represents the decoder-encoder structure consisting of a stacked decoder network and an encoder network; Then, the decoder network is trained using a self-supervised approach: ； in, This represents the optimal decoder network obtained by the self-supervised training phase of the decoder network based on magnetic resonance signals. This indicates a self-supervised training method. This represents the loss function used in the self-supervised training phase of the decoder network based on magnetic resonance signals, including the mean squared error loss function and the cosine similarity error loss function.

5. The functional magnetic resonance imaging decoding method based on explicit brain region interaction modeling according to claim 1, characterized in that, Step S4 is as follows: The input image signal is reconstructed using an encoder-decoder structure with stacked encoder and decoder networks, as shown in the following expression: ； ； in, This represents the encoder-decoder structure consisting of a stacked encoder network and a decoder network; Then, the decoder network is trained using a self-supervised approach: ； in, This represents the optimal decoder network obtained by the self-supervised training phase of the image signal-based decoder network. This represents the loss function used in the self-supervised training phase of the image signal-based decoder network, including the mean absolute error loss function, the feature similarity error loss function, and the smoothing regularization loss function.

6. The functional magnetic resonance imaging decoding method based on explicit brain region interaction modeling according to claim 1, characterized in that, Step S6 is as follows: The training steps S1-S5 are completed using the image and magnetic resonance signals provided in the dataset, ensuring that the training of each component of the provided network converges. The expression is as follows: ； ； ； in, This represents the encoder network after training has converged. This indicates model optimization operations. This represents the decoder network after training has converged. This represents the optimization operation of a multi-objective joint model. This represents the information exchange network after training convergence; Then, the magnetic resonance signal to be decoded is input into the information interaction network and the decoder network to obtain the final image decoding and reconstruction result, as shown in the following expression: ； in, This represents the decoded and reconstructed image signal estimated during the testing phase. This represents the magnetic resonance signal data to be decoded input during the testing phase.

Images