A multimodal 3D brain image fusion method based on unsupervised domain adaptive method

Abstract

This invention relates to the field of medical image processing technology, and more particularly to a multimodal 3D brain image fusion method based on unsupervised domain adaptation. The method includes: acquiring source domain and target domain datasets; performing image preprocessing using contrast-limited adaptive histogram equalization; constructing a style transfer model, including a generative adversarial network architecture; inputting source and target domain images into the style transfer model for training, transforming the source domain image into a new target domain image through the trained style transfer model; constructing an image fusion model for fusing multimodal 3D brain images; and inputting the new target domain image, the corresponding MRI images of the source domain images, and the fluorescence images to be fused into the image fusion model for fusion to obtain multimodal 3D brain images. The advantages are: based on a domain adaptation method, it achieves efficient multimodal macaque brain image fusion with high structural similarity without using cell-based imaging labels.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and in particular to a multimodal three-dimensional brain image fusion method based on unsupervised domain adaptation. Background Technology

[0002] With the continuous development and improvement of optical microscopy section tomography (MOST) technology, it has become possible to collect massive amounts of three-dimensional macaque brain optical imaging data. For example, fMOST can acquire a large amount of high spatial resolution (0.325*0.325*1μm) data in multiple channels. 3 The vast amount of images available provides numerous cutting-edge research opportunities, but also presents many challenges. Among these challenges, the fusion or registration of multimodal brain images has become one of the most attention-grabbing and long-standing problems, as it is a key technology widely used in medical image processing. However, researchers have not yet proposed a highly effective solution for the fusion of high spatial resolution multimodal brain images. Manual brain image registration and fusion is not only time-consuming and labor-intensive, but the quality is sometimes unsatisfactory.

[0003] In recent years, convolutional neural networks have demonstrated effectiveness in several biomedical image fusion tasks, such as TransMorph and HyperMorph. The advantage of deep learning-based image fusion is its ability to automatically extract image features from a large amount of paired supervised input, thus completing the image fusion process. However, this type of supervised labeled data is often obtained manually, requiring significant time and effort, especially with high spatial resolution data. Furthermore, due to the different data distributions (domain shift) between different modal images (different optical imaging channels), the fusion results often fail to meet requirements. In the field of microscopic optical imaging, domain shift is caused by differences in imaging optical conditions and sensor acquisition variations. Therefore, no effective solution has yet been proposed for the fusion problem based on high-resolution multimodal brain images. Summary of the Invention

[0004] To address the aforementioned problems, this invention provides a multimodal 3D brain image fusion method based on unsupervised domain adaptation.

[0005] The primary objective of this invention is to provide a multimodal 3D brain image fusion method based on unsupervised domain adaptation, comprising the following steps:

[0006] S1. Obtain source domain dataset and target domain dataset: Collect the source domain dataset yourself, and download the target domain dataset from publicly available stereotactic rhesus monkey brain atlas data; the source domain dataset includes source domain images, and the target domain dataset includes target domain images.

[0007] S2. Image preprocessing: The source domain image is mapped to the intensity level of the target domain image using the contrast-limited adaptive histogram equalization method; the skull of the macaque brain image after intensity correction is removed using the FSL tool, and the data after skull removal is used as the input of the style transfer model.

[0008] S3. Construct a style transfer model, wherein the style transfer model includes a generative adversarial network architecture;

[0009] S4. Generate a new target domain image: Input the source domain image and the target domain image into the style transfer model for training, and use the trained style transfer model to transform the source domain image into the target domain to generate a new target domain image.

[0010] S5. Construct an image fusion model for the fusion of multimodal 3D brain images;

[0011] S6. Multimodal 3D brain image fusion: The new target domain image, the MRI imaging corresponding to the source domain image, and the fluorescence imaging to be fused are input into the image fusion model for fusion to obtain a multimodal 3D brain image.

[0012] Preferably, the generative adversarial network architecture includes block resampling, a generator, and a discriminator; the generator and the discriminator each include a first-layer 3D convolutional block, a second-layer 3D convolutional block, a third-layer 3D convolutional block, and two pooling layers; the pooling layers are located in the middle of the different layers of 3D convolutional blocks, and output 128, 256, and 512 convolutional features from the first-layer 3D convolutional block, the second-layer 3D convolutional block, and the third-layer 3D convolutional block, respectively.

[0013] Preferably, the source domain dataset includes cell architecture imaging, fluorescence imaging, and corresponding MRI imaging. The source domain dataset is acquired using fluorescence microscopy optical sectioning tomography, wherein the channel stained with propidium iodide is for cell architecture imaging, the channel stained with immunohistochemistry is for fluorescence imaging, and the corresponding MRI imaging has a resolution of 500*500*500μm. 3 The cell construction imaging and the fluorescence imaging are acquired in parallel in two channels;

[0014] The target domain image is the average standard template of the neuromorphic tracing standard atlas NMT of the macaque brain, which includes the labeled atlas D99 covering the cerebral cortex region and the labeled atlas SARM containing subcortical nuclei and part of the cortical region.

[0015] Preferably, step S3 includes the following sub-steps:

[0016] S301, Image Augmentation: Perform block resampling, divide the source domain dataset into three dimensions, resample each block after division, and use an overlap smoothing strategy near the block boundaries.

[0017] S302. Generator extracts image features: The generator transforms the source domain image to the target domain by calculating tensors to generate a new target domain image;

[0018] S303. Discriminator distinguishes data: The discriminator adjusts its parameters using the stochastic gradient descent algorithm to correctly distinguish between real and generated data. It uses the ReLU function as the activation function for convolution and the binary cross-entropy (BCE) function as the loss function.

[0019] Preferably, the generator includes an encoder, a decoder, and a skip connection layer; the encoder includes a downsampling pooling layer, the decoder includes an upsampling deconvolution layer, and the skip connection layer is used for direct connections between different layers;

[0020] Step S302 specifically includes: inputting the cell construction imaging from the source domain dataset into the encoder; performing matrix calculation on the patch resampled in blocks; encoding to obtain the feature matrix of the source domain data; obtaining the features in the feature map; and outputting a 128*128*128 image; then optimizing downsampling through a downsampling pooling layer to extract the features of the input image and outputting a high-dimensional feature map of 256*256*256; and the decoder restoring the extracted image features to the original resolution.

[0021] Preferably, in step S303, if the discriminator output value is greater than the threshold, the discriminator determines the data sample as real data, i.e., logically true; if the output value is less than the threshold, the discriminator determines the data sample as generated data, i.e., logically false.

[0022] Preferably, step S4 specifically includes the following sub-steps:

[0023] S401. The preprocessed cell architecture imaging and NMT averaged standard template are resampled in blocks, with each block having a scale of 128*128*128 and a step size of 10.

[0024] S402. Set the total number of iterations in the training process to 1000. Save the current model parameters every 100 rounds to obtain 10 sets of style transfer model parameters.

[0025] S403. Input the source domain data as new inputs into the style transfer model and calculate the scores using the following formula:

[0026]

[0027] In the formula, x s To image the cell construction in the source domain dataset, y s To construct the corresponding MRI image for source domain cells, where i is the current training iteration number, and G... S→T∪ represents a positive generator, ∪ represents the union operation, and ∩ represents the intersection operation.

[0028] S404. Select the group with the highest corresponding score as the final network parameter selection for the style transfer model, and convert it to the target domain to generate a new target domain image.

[0029] Preferably, the image fusion model in step S5 includes an affine registration module and a resampling weighting module;

[0030] The affine registration module initializes the affine transformation matrix randomly, uses mean squared error (MSE) as a similarity measure, performs linear transformation on cell architecture imaging through matrix multiplication, and outputs the affine transformation matrix, i.e. the deformation field, after the calculated value of the similarity measure is less than a set threshold and the convergence condition is met, so as to guide the alignment of the style-transferred source domain image with the individual MRI in spatial location.

[0031] The resampling weighting module performs a weighted average of the aligned cross-modal 3D imaging data based on the original data resolution.

[0032] Preferably, the method for multimodal three-dimensional brain image fusion in step S6 specifically includes:

[0033] S601. The new target domain image and the corresponding source domain image's MRI imaging are input into the image fusion model for fusion. The new target domain image is the cell architecture imaging of the first channel in the source domain dataset, generated by a style transfer model, which corresponds to the intensity distribution of individual MRI. The deformation field calculated in step S5 is used to perform a linear transformation of the new target domain image by matrix multiplication, and then fused with the MRI imaging in the corresponding source domain dataset.

[0034] S602. Input the fluorescence image to be fused into the image fusion model to finally obtain the fused multimodal macaque brain image.

[0035] The second objective of this invention is to provide a multimodal 3D brain image fusion system based on unsupervised domain adaptation, wherein the system is used to execute the aforementioned multimodal 3D brain image fusion method based on unsupervised domain adaptation; specifically including:

[0036] The image preprocessing module is used to map the intensity levels of the acquired source domain dataset to the target domain dataset, and then perform intensity correction processing on the image.

[0037] A style transfer model construction module is used to construct a style transfer model, which includes a generative adversarial network architecture, comprising block resampling, a generator, and a discriminator.

[0038] The transfer learning module is used to input source domain images and target domain images into the style transfer model for training, and generate new target domain images.

[0039] An image fusion model construction module is used to construct an image fusion model, which includes an affine registration module and a resampling weighting module;

[0040] The image fusion module is used to input the new target domain image, the MRI imaging corresponding to the source domain image, and the fluorescence imaging to be fused into the image fusion model for fusion to obtain multimodal three-dimensional brain images.

[0041] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0042] The present invention provides a complete multimodal macaque brain image fusion framework, which includes style transfer model training, model parameter selection in a bidirectional image converter, and cross-modal source domain-new target domain fusion. The input macaque brain data is segmented and resampled, and a three-dimensional convolutional kernel is used in the generative adversarial network model to achieve domain adaptive transfer in high-dimensional space. Based on the domain adaptive method, efficient multimodal macaque brain imaging fusion with high structural similarity is achieved without using cell-based imaging labels. Attached Figure Description

[0043] Figure 1 This is a flowchart illustrating a multimodal three-dimensional brain image fusion method based on unsupervised domain adaptation provided in an embodiment of the present invention.

[0044] Figure 2 This is a schematic diagram of a generative adversarial network architecture provided according to an embodiment of the present invention.

[0045] Figure 3 This is a schematic diagram of the results of the style transfer model provided in the embodiments of the present invention; (A) coronal plane; (B) sagittal plane; (C) horizontal plane.

[0046] Figure 4 This is a schematic diagram of the result of multimodal macaque brain image fusion according to an embodiment of the present invention; (A) coronal plane; (B) sagittal plane; (C) horizontal plane. Detailed Implementation

[0047] In the following description, embodiments of the invention will be described with reference to the accompanying drawings. In the description below, the same modules are denoted by the same reference numerals. Where the same reference numerals are used, their names and functions are also the same. Therefore, their detailed description will not be repeated.

[0048] The purpose of this invention is to provide a multimodal three-dimensional brain image fusion method based on unsupervised domain adaptation, which achieves high structural similarity and efficient multimodal macaque brain image fusion without using cell-based imaging labels.

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not constitute a limitation thereof.

[0050] Figure 1 The flowchart illustrates a multimodal 3D brain image fusion method based on unsupervised domain adaptation provided by this invention. Figure 1 As shown, the present invention provides a multimodal 3D brain image fusion method based on unsupervised domain adaptation, which specifically includes the following steps:

[0051] S1. The source domain dataset was acquired independently using the fMOST imaging system, and the target domain dataset was downloaded from publicly available stereotactic rhesus monkey brain atlas data. The source domain dataset includes PI (cell architecture) imaging, fluorescence imaging, and corresponding MRI images (all source domain images). The source domain dataset was acquired using fMOST (fluorescence microscopy optical sectioning tomography), where the propidium iodide staining channel is for cell architecture imaging, the immunohistochemistry channel is for fluorescence imaging, and the resolution of the corresponding MRI data is 500*500*500μm. 3 The target domain dataset includes target domain images, which are the average standard templates of the neuromorphic tracing standard atlas of the macaque brain, including D99 and SARM. D99 is a labeled atlas covering the cerebral cortex region, and SARM is a labeled atlas containing subcortical nuclei and part of the cortical region.

[0052] S2. Image Preprocessing: The average standard template of the neuromorphic tracing atlas of the macaque brain (NMT) was selected as the reference image. The contrast-limited adaptive histogram equalization method was used to map the intensity level of the acquired source domain image to the standard reference image. The purpose of using the contrast-limited histogram equalization method is to enhance the image contrast while preserving the local features of the image. The BET (Brain Extraction Tool) tool in FSL was used to remove the skull from the intensity-corrected macaque brain image. The data after removing the skull was used as the input of the style transfer model.

[0053] S3. Constructing a style transfer model: The style transfer model includes a generative adversarial network architecture (such as...). Figure 2 (as shown); specifically including:

[0054] A generative adversarial network (GAN) architecture is constructed for training a style transfer model. The GAN architecture specifically includes patch resampling, a generator, and a discriminator for mutual conversion between source and target domain images. Both the generator and discriminator contain a first-layer 3D convolutional block, a second-layer 3D convolutional block, a third-layer 3D convolutional block, and two pooling layers. The pooling layers are located between the different layers of 3D convolutional blocks, outputting 128, 256, and 512 convolutional features from the first, second, and third layers, respectively. The weight parameters of the generator and discriminator are randomly initialized, using adaptive moment estimation as the optimizer, with an initial learning rate of 0.0002.

[0055] S301. Image Augmentation: Block resampling is performed. Specifically, the source domain dataset is divided into three dimensions using the resize operation in ANTs to reduce computational and storage complexity. Simultaneously, each block is resampled and adjusted to the target resolution of 250*250*250μm. 3 An overlap smoothing strategy is used near the boundaries of the blocks;

[0056] S302. The generator transforms the source domain image to the target domain to generate a new target domain image, which is achieved by calculating the tensor methods Conv3D and Pool3D. The generator includes two modules: an encoder and a decoder, as well as a skip connection layer. The encoder module includes a downsampling pooling layer, and the decoder module includes an upsampling deconvolution layer. Specifically, the cell construction image in the source domain dataset is first input into the encoder. The patch resampled in blocks is matrix-calculated and encoded to obtain the feature matrix of the source domain data, resulting in the features obtained in the feature map, outputting a 128*128*128 image. Then, the downsampling pooling layer optimizes the downsampling and extracts the features of the input image, outputting a high-dimensional feature map of 256*256*256. The skip connection layer is used for direct connection between different layers. The decoder module includes upsampling deconvolution layers corresponding to the three layers of the encoder to restore the extracted image features to the original resolution.

[0057] The downsampling pooling layer has a 5*5*5 average pooling kernel, and the input and output are 3*3*3 fully convolutions. Average pooling is used to guide the deconvolution.

[0058] S303. Distinguishing between real and generated data using a discriminator: The discriminator is a convolutional neural network, employing the ReLU function as the activation function for all convolutional layers and binary cross-entropy (BCE) as the loss function. During training, the discriminator adjusts its parameters using a stochastic gradient descent algorithm to correctly distinguish between real and generated data. If the discriminator's output value is greater than a threshold, the discriminator classifies the data sample as real data (logically true); if the output value is less than the threshold, the discriminator classifies the data sample as generated data (logically false).

[0059] The structure-consistent cyclic loss function is used during the training of the style transfer model. Its expression is as follows:

[0060]

[0061] In the formula, Let F represent the mathematical expectation function. S (x s ) is x s Input to model F S The output result obtained in F T (G S→T (x s )) is G S→T (x s Input to model F T The output result obtained in F T (x t ) is x t Input to model F T The output result obtained in F S (G T→S (x t )) is G T→S (x t Input to model F S The output obtained in For F S (x s ) and F T (G S→T (x s The similarity between )) For F T (x t ) and F S (G T→S (x t The similarity between ))

[0062] S4. Input the source domain image and the target domain image into the style transfer model for training. Transform the source domain image into the target domain using the trained style transfer model to generate a new target domain image. This includes the following sub-steps:

[0063] S401. The preprocessed cell architecture imaging and NMT averaged standard template are resampled in blocks, with each block having a scale of 128*128*128 and a step size of 10.

[0064] S402. Set the total number of iterations in the training process to 1000. Save the current model parameters every 100 rounds to obtain 10 sets of style transfer model parameters.

[0065] S403. Input the source domain data as new inputs into the style transfer model and calculate the scores using the following formula:

[0066]

[0067] In the formula, x s To image the cell construction in the source domain dataset, y s To construct the corresponding MRI image for source domain cells, where i is the current training iteration number, and G... S→T ∪ represents a positive generator, ∪ represents the union operation, and ∩ represents the intersection operation.

[0068] S404. Select the group with the highest corresponding score as the final network parameter selection for the style transfer model, and convert it to the target domain to generate a new target domain image.

[0069] Figure 3 The results of the style transfer model are shown.

[0070] S5. Construct an image fusion model, including affine registration and resampling weighting modules, for fusing multimodal 3D macaque brain images. Specifically, affine registration is used to merge image information from different imaging modalities for accurate identification and localization. The affine registration method includes translation, rotation, scaling, and shearing operations. Specifically, it is implemented by randomly initializing the affine transformation matrix, using mean squared error (MSE) as a similarity measure, and performing a linear transformation on cell architecture imaging through matrix multiplication. Once the calculated similarity measure value is less than a set threshold, i.e., the convergence condition is met, the affine transformation matrix is ​​output to guide the spatial alignment of the style-transferred source domain image with the individual MRI. The resampling weighting calculates a weighted average of the aligned cross-modal 3D imaging data based on the original data resolution, where the cell architecture imaging and fluorescence imaging data in the source domain are both 0.325*0.325*1μm. 3 The corresponding MRI resolution is 500*500*500μm 3 .

[0071] S6. Multimodal 3D brain image fusion, including:

[0072] S601. The new target domain image and the corresponding source domain image's MRI imaging are input into the image fusion model for fusion; the new target domain image is the cell architecture imaging of the first channel in the source domain dataset generated by a style transfer model, corresponding to the intensity distribution of individual MRI; the deformation field calculated by S5 is used to perform a linear transformation of the new target domain image by matrix multiplication, and then fused with the MRI imaging in the corresponding source domain dataset; the deformation field is an affine transformation matrix that satisfies the convergence condition.

[0073] S602. The fluorescence imaging to be fused is input into the image fusion model to finally obtain the fused multimodal macaque brain image; the fluorescence imaging is the second channel acquired in parallel with cell architecture imaging under the fMOST imaging system; the fluorescence imaging is processed according to the cell architecture imaging process in the source domain dataset to obtain a fused multimodal macaque brain image covering the spatial location information of stereotactic monkey brain atlas, cell architecture of propidium iodide imaging, neuronal projection intensity of fluorescence imaging, and MRI brain tissue structure.

[0074] Figure 4 The results of multimodal macaque brain image fusion are shown; by Figure 4 It can be seen that the fusion results obtained by using the fusion method of the present invention can greatly improve the efficiency and structural similarity of generation fusion in cross-modal high-dimensional macaque brain data fusion tasks.

[0075] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0076] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A multimodal 3D brain image fusion method based on unsupervised domain adaptation, characterized in that, Includes the following steps: S1. Obtain source domain dataset and target domain dataset: Collect the source domain dataset yourself, and download the target domain dataset from publicly available stereotactic rhesus monkey brain atlas data; the source domain dataset includes source domain images, and the target domain dataset includes target domain images; the source domain dataset includes cell architecture imaging, fluorescence imaging, and corresponding MRI imaging; the target domain images are the average standard template of the neuromorphic tracing standard atlas of rhesus monkey brain; S2. Image preprocessing: The cell architecture image is mapped to the intensity level of the target domain image using the contrast-limited adaptive histogram equalization method; after intensity correction, the cell architecture image is removed using the FSL tool, and the data after removing the skull is used as the input for the style transfer model training process. S3. Construct a style transfer model, wherein the style transfer model includes a generative adversarial network architecture; S4. Generate a new target domain image: Input the preprocessed cell architecture image and the target domain image into the style transfer model for training. After training, the cell architecture image is converted to the target domain through the trained style transfer model to generate a new target domain image. S5. Construct an image fusion model for the fusion of multimodal 3D brain images; S6. Multimodal 3D brain image fusion: The new target domain image, the MRI imaging corresponding to the source domain image, and the fluorescence imaging to be fused are input into the image fusion model for fusion to obtain a multimodal 3D brain image.

2. The multimodal three-dimensional brain image fusion method based on unsupervised domain adaptation according to claim 1, characterized in that: The generative adversarial network architecture includes block resampling, a generator, and a discriminator; the generator and the discriminator each include a first-layer 3D convolutional block, a second-layer 3D convolutional block, a third-layer 3D convolutional block, and two pooling layers; the pooling layers are located in the middle of the different layers of 3D convolutional blocks, and output 128, 256, and 512 convolutional features from the first-layer 3D convolutional block, the second-layer 3D convolutional block, and the third-layer 3D convolutional block, respectively.

3. The multimodal three-dimensional brain image fusion method based on unsupervised domain adaptation according to claim 2, characterized in that: The source domain dataset was acquired using fluorescence microscopy optical sectioning tomography, with the propidium iodide-stained channels used for cell architecture imaging and the immunohistochemical channels used for fluorescence imaging. The corresponding MRI imaging resolution was 500 μm. 500μm 500 μm; the cell construction imaging and the fluorescence imaging are acquired in parallel in two channels; The target domain images include the labeled atlas D99 covering the cerebral cortex region, and the labeled atlas SARM containing subcortical nuclei and parts of the cortical region.

4. The multimodal three-dimensional brain image fusion method based on unsupervised domain adaptation according to claim 3, characterized in that, Step S3 includes the following sub-steps: S301, Image Augmentation: Perform block resampling, divide the source domain dataset into three dimensions, resample each block after division, and use an overlap smoothing strategy near the block boundaries. S302, Generator extracts image features: The generator transforms the source domain image to the target domain by computing tensors to generate a new target domain image; S303. Discriminator distinguishes data: The discriminator adjusts its parameters using the stochastic gradient descent algorithm to correctly distinguish between real and generated data. It uses the ReLU function as the activation function for convolution and the binary cross-entropy (BCE) function as the loss function.

5. The multimodal three-dimensional brain image fusion method based on unsupervised domain adaptation according to claim 4, characterized in that: The generator includes an encoder, a decoder, and a skip connection layer; the encoder includes a downsampling pooling layer, and the decoder includes an upsampling deconvolution layer; The skip connection layer is used for direct connection between different layers; Step S302 specifically includes: first inputting the cell architecture images from the source domain dataset into the encoder; performing matrix calculations on the resampled patches; encoding to obtain the feature matrix of the source domain data; obtaining the features from the feature map; and outputting a size of 128. 128 The feature map is 128; after downsampling through a downsampling pooling layer, the features of the input image are extracted, and the output size is 256. 256 256 high-dimensional feature maps; The decoder restores the extracted image features to their original resolution.

6. The multimodal three-dimensional brain image fusion method based on unsupervised domain adaptation according to claim 5, characterized in that: If the discriminator output value is greater than the threshold in step S303, the discriminator will determine the data sample as real data, i.e., logically true. If the output value is less than the threshold, the discriminator will classify the data sample as generated data, i.e., logically false.

7. The multimodal three-dimensional brain image fusion method based on unsupervised domain adaptation according to any one of claims 1-6, characterized in that, Step S4 specifically includes the following sub-steps: S401. The preprocessed cell architecture imaging and NMT averaged standard template are resampled in blocks, with each block having a scale of 128. 128 128, step size is 10; S402. Set the total number of iterations in the training process to 1000. Save the current model parameters every 100 rounds to obtain 10 sets of style transfer model parameters. S403. Input the source domain data as new inputs into the style transfer model and calculate the scores using the following formula: In the formula, x s To construct images of cells in the source domain dataset, y s To construct the corresponding MRI imaging for source domain cells, This represents the current training iteration number. For the union operation, To perform the intersection operation; S404. Select the group with the highest corresponding score as the final network parameter selection for the style transfer model, and convert it to the target domain to generate a new target domain image.

8. The multimodal three-dimensional brain image fusion method based on unsupervised domain adaptation according to claim 7, characterized in that: The image fusion model in step S5 includes an affine registration module and a resampling weighting module; The affine registration module initializes the affine transformation matrix randomly, uses mean squared error (MSE) as a similarity measure, performs linear transformation on cell architecture imaging through matrix multiplication, and outputs the affine transformation matrix, i.e. the deformation field, after the calculated value of the similarity measure is less than a set threshold and the convergence condition is met, so as to guide the alignment of the style-transferred source domain image with the individual MRI in spatial location. The resampling weighting module performs a weighted average of the aligned cross-modal 3D imaging data based on the original data resolution.

9. The multimodal three-dimensional brain image fusion method based on unsupervised domain adaptation according to claim 8, characterized in that, The method for multimodal three-dimensional brain image fusion in step S6 specifically includes: S601. The new target domain image and the corresponding source domain image's MRI imaging are input into the image fusion model for fusion. The new target domain image is the cell architecture imaging of the first channel in the source domain dataset, generated by a style transfer model, which corresponds to the intensity distribution of individual MRI. The deformation field calculated in step S5 is used to perform a linear transformation of the new target domain image by matrix multiplication, and then fused with the MRI imaging in the corresponding source domain dataset. S602. Input the fluorescence image to be fused into the image fusion model to finally obtain the fused multimodal macaque brain image.

10. A multimodal 3D brain image fusion system based on unsupervised domain adaptation, characterized in that, The system is used to execute the unsupervised domain-adaptive multimodal three-dimensional brain image fusion method according to claim 1; specifically including: The image preprocessing module is used to map the acquired cell architecture images to the intensity levels of the target domain dataset, and then process the images through intensity correction. A style transfer model construction module is used to construct a style transfer model, which includes a generative adversarial network architecture, comprising block resampling, a generator, and a discriminator. The transfer learning module is used to input cell architecture imaging and target domain images into the style transfer model for training. After training, the cell architecture imaging is converted to the target domain through the trained style transfer model to generate a new target domain image. An image fusion model construction module is used to construct an image fusion model, which includes an affine registration module and a resampling weighting module; The image fusion module is used to input the new target domain image, the MRI imaging corresponding to the source domain image, and the fluorescence imaging to be fused into the image fusion model for fusion to obtain multimodal three-dimensional brain images.

Images