Brain slice three-dimensional reconstruction and cross-modal registration method based on deep learning

Abstract

The invention relates to a brain slice three-dimensional reconstruction and cross-modal registration method based on deep learning, and the method comprises the following steps: S1, carrying out the photographing of the surface of a brain tissue block before the execution of a slicing operation, obtaining a block image, and converting a 2D block image into a 3D block image; s2, collecting a fluorescence slice image, and migrating the mode of the fluorescence slice image into a block surface mode; s3, performing registration on the 3D block surface image and an MRI image acquired in a living body state; and S4, carrying out registration on the fluorescence slice image after modal migration and the registered 3D block surface image. According to the method, not only can the registration precision be improved, but also the high automation of the process from image preprocessing, registration, map mapping to subsequent neuron mapping reconstruction is realized, and the labor cost is remarkably reduced.

Description

Technical Field

[0001] The present invention relates to the field of biomedical technology, and in particular to a method for three-dimensional reconstruction and cross-modal registration of brain slices based on deep learning. Background Art

[0002] With the rapid development of neuroscience and biological imaging technologies, whole-brain mapping and connectivity information at mesoscopic resolution have become key tools for in-depth exploration of brain structure and function. Currently, a large number of single-cell images of cytoarchitecture, spatial transcription, and neural projections are being generated. These ultra-high-resolution imaging data urgently require analytical tools to enable comparative analysis using atlases in standard spaces. Consequently, this places higher demands on 3D slice reconstruction, multimodal registration, and brain region segmentation.

[0003] In recent years, various mesoscopic image registration techniques have been successfully applied to mouse brain research. These multimodal registration methods, primarily based on point cloud registration and morphological feature registration, have achieved remarkable results in the mouse brain. However, the monkey brain faces challenges in accurately registering the cerebral cortex and some internal structures due to its more complex brain structure, significant individual variability, and significant deformation during brain slice preparation. This further impacts the precision of brain region localization and, consequently, the accuracy of subsequent analysis results. Furthermore, deep learning techniques have shown promise in multimodal registration, particularly by using style transfer techniques to generate images with intensity distributions similar to those of the target image to enhance registration accuracy. However, most deep learning models rely on 2D images, requiring 3D images to be sliced ​​into 2D images for training. This can lead to a loss of 3D spatial structural information, compromising the accuracy of the generated 2D images. Furthermore, 3D images reconstructed from these 2D images are often accompanied by artifacts and irrelevant background signals, further reducing registration accuracy. Some methods also employ manual correction to correct registration errors. However, for high-resolution images, manual correction requires a high time cost and requires the operator to have certain anatomical knowledge.

[0004] Currently, relatively few studies have focused on cross-scale and cross-modal registration of monkey brains, primarily on marmosets. This is primarily due to the numerous challenges encountered in converting 2D slices to 3D reconstruction. The monkey brain undergoes significant deformation during ex vivo processing, which complicates the alignment of 2D slices. While existing methods have achieved some progress in cross-scale and cross-modal registration of marmoset brains, achieving high-precision 3D reconstruction and registration remains a significant challenge when applying these techniques to the larger and more complex macaque brain. Summary of the Invention

[0005] The purpose of the present invention is to provide a deep learning-based three-dimensional reconstruction and cross-modal registration method for brain slices, which can automatically achieve high-precision three-dimensional reconstruction and registration for the macaque brain.

[0006] In order to achieve this object, the present invention provides the following technical solutions:

[0007] In a first aspect, an embodiment of the present invention provides a method for three-dimensional reconstruction and cross-modal registration of brain slices based on deep learning, comprising the following steps:

[0008] S1, before performing the slicing operation, take a picture of the surface of the brain tissue block to obtain a block surface image, and convert the 2D block surface image into a 3D block surface image;

[0009] S2, acquiring a fluorescent slice image, and migrating the modality of the fluorescent slice image to a block surface modality;

[0010] S3, registering the 3D block image with an in vivo MRI image;

[0011] S4, registering the fluorescence slice image after modality transfer with the registered 3D block surface image.

[0012] In S1, the process of converting the 2D block surface image into the 3D block surface image is as follows:

[0013] S11, resizing the collected original block surface image;

[0014] S12, the resized 2D block image is input into the SAM2 deep learning model to segment and extract brain tissue;

[0015] S13, for the block surface image processed by S12, calculating the mean square error (MSE) between adjacent block surface images;

[0016] S14, filter out the block images whose MSE value is greater than the sum of the overall mean and twice the standard deviation;

[0017] S15, performing centroid alignment and affine transformation on the block surface images filtered out in step S14;

[0018] S16, stacking the block surface images processed in step S15, performing 3D reconstruction, and obtaining a 3D block surface image.

[0019] In S2, the original fluorescence slice image is first downsampled to 500*500 pixels, and then denoising and smoothing are performed. Finally, a 2D CycleGan deep learning model is used to perform modality migration on the denoised fluorescence slice image to generate a synthetic block surface modality from the fluorescence slice image.

[0020] In S3, the 3D block image is first modally migrated to a synthetic MRI modality; the 3D block image processed by modal migration is then aligned with the individual MRI image collected in a living state to obtain a first deformation field; the 3D block image processed by modal migration and aligned with the individual MRI is then adjusted to the MRI standard space through a nonlinear registration method of symmetric differential homeomorphism to obtain a second deformation field.

[0021] In S4, the registration process of the fluorescence slice image after modality migration and the registered 3D block surface image includes the following steps:

[0022] S41, the cerebellum is removed from the 3D block image through the cerebellum mask;

[0023] S42, aligning the synthesized fluorescence slice image of the block face modality with the corresponding block face image through affine transformation;

[0024] S43, performing tissue segmentation on the fluorescent slice images and corresponding block surface images of all slices to obtain various tissue regions;

[0025] S44, registering the fluorescent slice image and the block surface image based on a tissue region alignment method to obtain registered fluorescent slice images and block surface images respectively;

[0026] S45 , reconstructing all registered fluorescence slice images to obtain a 3D fluorescence volume image, and nonlinearly registering the 3D fluorescence volume image to the 3D block surface.

[0027] In a second aspect, the present invention provides a computer program product comprising computer-readable instructions, characterized in that when the computer-readable instructions are executed by a processor, the computer-readable instructions implement the steps of the deep learning-based brain slice three-dimensional reconstruction and cross-modal registration method of the present invention.

[0028] In a third aspect, the present invention provides a computer-readable storage medium comprising computer-readable instructions, characterized in that when the computer-readable instructions are executed by a processor, the steps in the deep learning-based brain slice three-dimensional reconstruction and cross-modal registration method of the present invention are implemented.

[0029] In a fourth aspect, the present invention provides an electronic device comprising: a memory storing program instructions; a processor connected to the memory, executing the program instructions in the memory, and implementing the steps of the deep learning-based brain slice three-dimensional reconstruction and cross-modal registration method of the present invention.

[0030] Compared with the existing technology, the present invention constructs an automated cross-modal, cross-scale 3D reconstruction and registration method for monkey brain microscopic optical images, mainly targeting autofluorescence imaging based on manual histological sections. This workflow not only includes an accurate whole-brain cross-modal and cross-scale registration method, but also integrates multiple robust preprocessing steps, such as 2D optical section reconstruction, 2D block brain segmentation, and neuronal mapping reconstruction in a standard space. Unlike the conventional direct registration of individual brain images with a standard template, this method uses MRI collected from living individuals as a bridge to address the effects of ex vivo deformation, and adopts a 3D style transfer model to address the differences between modalities, thereby achieving accurate cross-modal and cross-scale monkey brain registration. In addition, a more flexible and robust 3D reconstruction method is designed for 2D microscopic optical images of histological sections, achieving high-precision 3D reconstruction. The process from image preprocessing, registration, atlas mapping to subsequent neuronal mapping reconstruction is highly automated, significantly reducing labor costs.

[0031] For other advantages of the present invention, please refer to the relevant description in the embodiment section. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0033] Figure 1 The figure is a flowchart of a method for three-dimensional reconstruction and cross-modal registration of brain slices based on deep learning as an example in an embodiment of the present invention.

[0034] Figure 2 Schematic diagram of brain tissue segmentation extracted for the SAM2 deep learning model.

[0035] Figure 3 Schematic diagram of the mean square error (MSE) between adjacent block images for example.

[0036] Figure 4a 、 Figure 4b Schematic diagrams showing the comparison of block surface images before and after alignment.

[0037] Figure 5 Schematic diagram of the reconstructed 3D block surface image for example.

[0038] Figure 6 Schematic diagram of the process for generating synthetic block-face modality maps for fluorescence slice images.

[0039] Figure 7a 、 Figure 7bThese are comparison diagrams of the segmentation results of the fluorescent slice image and the block surface image.

[0040] Figure 8 Schematic diagram of the slice registration process based on tissue segmentation.

[0041] Figure 9a 、 Figure 9b These are the registration results of the fluorescent slice image and the block surface image respectively.

[0042] Figure 10 This is a diagram of the composition framework of electronic products. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0044] See also Figure 1 In this embodiment, a method for three-dimensional reconstruction and cross-modal registration of brain slices based on deep learning is provided, comprising the following steps:

[0045] S1, before performing the slicing operation, take a picture of the surface of the brain tissue block to obtain a block surface image, and convert it from a 2D image to a 3D image.

[0046] After paraffin embedding and before tissue sectioning, the surface of the brain tissue block is photographed and recorded to generate blockface images. These images preserve the morphology of the tissue before sectioning and are used to align subsequent histological sections with the actual tissue block before sectioning, ensuring the geometric accuracy of 3D reconstruction. Because sectioning is performed immediately after the photograph is taken, the blockface images collected before sectioning are the images of the slices obtained after sectioning.

[0047] You can continue to read Figure 1 , the process of converting 2D block surface image into 3D block surface image is as follows:

[0048] S11, resizing the acquired original block surface image so that the resized image size meets the input requirement of the subsequent segmentation model.

[0049] In this embodiment, the segmentation model uses the SAM2 deep learning model, so the original block image needs to be downsampled to a resolution of 1024*1024. The resolution mainly depends on the size of the input image of the SAM2 deep learning model used.

[0050] S12, the resized 2D block surface image is input into the SAM2 deep learning model to segment and extract the brain tissue to remove the background signal in the block surface image.

[0051] The block surface image contains a lot of background information. In order to deal with this background, the SAM2 (SegmentAnything 2) model is used here to segment the continuous block surface images. Figure 2 The SAM2 video segmentation model is applied to continuous block face image processing, and cross-slice information transfer is achieved through the memory attention mechanism and memory bank, which overcomes the processing limitations of traditional segmentation methods on continuous slices.

[0052] The SAM2 model consists of five modules: an image encoder, a cue encoder, a mask decoder, a memory attention module, and a memory bank. The memory attention module associates the features of the current frame with the features of previous frames, predictions, and any new cues. The memory bank maintains prediction information for adjacent frames, including previous target predictions and stored cues. Because the acquired block images are continuous and adjacent slices are similar in size and structure, these consecutive images are treated as frames of a continuous video for segmentation prediction.

[0053] SAM2 is initialized based on the pre-trained model and fine-tuned accordingly (the process of using the pre-trained model (a model trained on a large-scale dataset) as a starting point to adjust parameters on the new target task dataset to adapt it to the new task). Given that the data consists of continuous static images, a training strategy is designed (alternating training of video images and static images) to train by randomly sampling the block face dataset of macaque subjects. For each subject's dataset, an alternating training strategy is adopted to randomly extract images and videos for training. In addition, it is ensured that the image set extracted from the static images is continuous when used for video training. The slices are also sorted in descending order according to their size. The model includes four loss functions: focal loss and Dice loss for mask prediction, mean squared error (MSE) loss for intersection over union (IoU) prediction, and cross entropy loss for target prediction. The total loss formula is as follows:

[0054]

[0055] in,

[0056] α t (y) = α × y + (1 - α) × (1 - y);

[0057]

[0058] in Represents the prediction result of the model, and y represents the true label of the mask image. t Represents the predicted probability. γ is the focus parameter used to adjust the weights of easy-to-classify samples and difficult-to-classify samples. t As a balancing factor for positive and negative samples. N represents the number of voxels in the image, s represents the predicted target score, represents the true label of the numerical score. λ1, λ2, λ3, and λ4 are weight coefficients. In this embodiment, the weight coefficients are set to 400:1:1:1.

[0059] S13, for the block surface images processed by S12, calculating the mean square error (MSE) between adjacent block surface images.

[0060] Since there is platform or tissue movement during the acquisition of block images, and the macaque brain is divided into two halves along the coronal plane, there is an offset in the block images, which prevents direct 3D reconstruction. Therefore, in order to achieve 3D reconstruction of block images, it is necessary to first calculate the mean square error (MSE) between the block images (essentially slices) to identify the specific location of the offset. For example, the mean square error MSE between adjacent block images is as follows: Figure 3 shown.

[0061] S14, filter out the block face images whose MSE values ​​are greater than the sum of the overall mean and twice the standard deviation. The overall mean refers to the mean square error of all block face images.

[0062] The mean squared error (MSE) between adjacent block images should maintain a relatively stable value. Specifically, if the block images are not misaligned, the MSE value should not show significant jumps. Large outliers in the MSE value indicate a significant positional offset between the current block image and the adjacent block image, so these outliers should be screened out to determine the offset between the block images.

[0063] By calculating the mean square error between adjacent block surface images, the slice position where the offset occurs can be automatically determined based on the mean square error value.

[0064] S15, performing centroid alignment and affine transformation on the offset block surface images, that is, performing centroid alignment and affine transformation on the block surface images filtered out in step S14.

[0065] The centroid of the patch image is calculated by taking a weighted average of the voxel positions, using the voxel intensity as a weighting factor. Specifically, for each pixel, its actual coordinates are multiplied by the intensity value, then all pixels are summed up and finally divided by the sum of all intensities to obtain the overall center point of the image.

[0066] The first offset slice is aligned with the adjacent slice using the calculated centroid coordinates and the translation distances in both directions are determined. This translation distance is then applied to the alignment of subsequent offset slices. Figure 4a 、 Figure 4b shown.

[0067] Since the offset block image undergoes an affine transformation relative to the lens, affine correction is still required for the offset slices after centroid alignment. Specifically, the first offset block image and adjacent slices are affine transformed, and the calculated affine matrix is ​​then applied to subsequent offset slices to achieve precise alignment of all slices.

[0068] The method based on centroid alignment and affine correction is used to effectively overcome the platform movement problem that traditional rigid registration methods cannot handle.

[0069] It should be noted that the slices without position shift directly proceed to step S16 to wait for stack reconstruction, and step S15 is only executed for the slices with position shift.

[0070] S16, stacking the block surface images processed in step S15 and performing 3D reconstruction to obtain a 3D block surface image. The reconstructed 3D block surface image is as follows: Figure 5 shown.

[0071] S2, acquire fluorescence slice images and transfer their modality to block-face modality.

[0072] Before slicing, a camera is used to capture the top layer of fixed brain tissue to obtain a block image. The top layer is then cut to a certain thickness to obtain brain tissue slices. The brain tissue slices are then stained and subjected to fluorescence imaging to obtain fluorescent slice images.

[0073] You can continue to read Figure 1 After acquiring the fluorescence slice image, the original fluorescence slice image can be downsampled to 500*500 pixels, and then denoised and smoothed. Finally, the 2D CycleGan deep learning model is used to perform modal migration on the denoised fluorescence slice image, so that the fluorescence slice image generates a synthetic block surface modality, such as Figure 6 shown.

[0074] S3, registering the 3D block image with the MRI image acquired in vivo.

[0075] For details, please refer to Figure 1First, the 3D block images are modally transferred using the 3D CycleGAN model, transferring the block style to a synthetic MRI modality. The modality-transferred 3D block images are then registered with structural MRI images acquired in vivo (T1w in this case). In this process, a symmetric diffeomorphic nonlinear registration technique based on mutual information is employed to achieve precise image alignment and simultaneously obtain a first deformation field. The modality-transferred 3D block images, which have been registered with the individual MRI, are then fine-tuned to the standard MRI space using a symmetric diffeomorphic nonlinear registration method to obtain a second deformation field. Using these two inverse deformation fields, the atlas in the standard space is inversely mapped onto the individual fluorescence slice images.

[0076] With the help of a deep learning model of style transfer, the 3D block image modality is transferred to the MRI modality, maintaining the three-dimensional spatial continuity, avoiding the artifact problem of 2D slices, and solving the difficulties of traditional cross-modality registration.

[0077] It is easy to understand that, according to the logical relationship, step S3 is executed after step S1, but there is no difference in the execution order between step S2 and steps S1 and S3.

[0078] S4, registering the fluorescence slice image after modality transfer with the registered 3D block surface image.

[0079] You can continue to read Figure 1 The specific processing flow of this step is as follows:

[0080] S41, the cerebellum is removed from the 3D reconstructed block image using the cerebellum mask.

[0081] Considering that the cerebellum tissue in the fluorescent slice image is missing due to the tissue slicing process, in order to ensure that the homeomorphic condition is maintained with the three-dimensional reconstructed block surface image, the present invention removes the cerebellum part in the three-dimensional block surface image by aligning the three-dimensional reconstructed block surface image with the MRI standard template to obtain a 3D block surface image without the cerebellum.

[0082] S42, aligning the synthesized fluorescence slice image of the block face modality with the corresponding block face image through affine transformation.

[0083] In view of the large deformation generated during the cutting of fluorescent slices, the slices are first preliminarily aligned through affine transformation to correct their general outlines.

[0084] S43, applying a segmentation method based on a watershed algorithm to perform tissue segmentation on the fluorescent slice images of all slices and the corresponding block surface images to obtain various tissue regions.

[0085] Specifically, the fluorescence slice images and the corresponding block images were segmented, with special attention paid to areas showing significant deformation, such as the separation of left and right brain tissue and temporal lobe. The segmentation results of the fluorescence slice images and the block images are shown in Figure 2. Figure 7a 、 Figure 7b As shown in the figure, brain tissue is cut and the slices are very soft and easily deformed, so various deformations will occur. Figure 7a 、 Figure 7b This is one type of deformation. The left and right brains do not deform as a whole, but each produces different deformations independently. In this case, they need to be separated and corrected separately.

[0086] S44, registering the fluorescent slice image and the block surface image based on a tissue region alignment method.

[0087] Slice registration based on tissue segmentation Figure 8 As shown in Figure 1. On the one hand, based on the fluorescence slice image, the tissue region in the block image is aligned with the region in the corresponding fluorescence slice image to obtain a registered fluorescence slice image; after all the registered fluorescence slice images are reconstructed to obtain a 3D fluorescence volume image. On the other hand, based on the block image, the tissue region in the fluorescence slice image is aligned with the region in the corresponding block image to obtain a registered block image.

[0088] Registration based on fluorescent slice images can correct for slice deformation caused by cutting. Registration based on block images can correct for ex vivo deformation. This is because brain tissue deforms during excision from the skull, and tissue processing and embedding also produce various deformations. Reconstructing 3D block images can effectively correct for these deformations.

[0089] S45, nonlinearly registering the 3D reconstructed fluorescence volume image to the 3D block plane to better achieve three-dimensional correction of the fluorescence slice.

[0090] The three-dimensional structures of the registration results of the fluorescence slice image and the block surface image are as follows: Figure 9a 、 Figure 9b shown.

[0091] In this step S4, a registration technology based on tissue segmentation is designed to address the deformation caused by cutting. The fluorescent slice image and the corresponding block surface image are segmented. After the segmentation is completed, the fluorescent slice image of the synthesized block surface modality is used to align the tissue area in the fluorescent slice image to the area in the corresponding block surface image, thereby restoring the severely misplaced tissue structure in the fluorescent slice image to the correct anatomical position and ensuring the registration accuracy.

[0092] In this method, prior to tissue section imaging, a block image corresponding to each slice is first acquired to preserve the pre-section tissue morphology. This technique uses these block images acquired before tissue sectioning to extract brain tissue within the image using the SAM segmentation model. Brain tissue is then reconstructed using a 3D reconstruction algorithm specifically designed for block images. The reconstructed 3D block images are then modally transferred using the 3DCycleGAN deep learning model, synthesizing them into an MRI modality. Registration with in vivo MRI images allows for precise alignment of the 3D block images with a standard template.

[0093] For 2D fluorescence slice images, after completing whole-brain registration of 3D block images to standard space, 2D inter-slice registration techniques are further employed to achieve precise 2D registration of the fluorescence slice images with the corresponding block images. Given the significant non-uniform deformation characteristics of 2D fluorescence slices and modal differences from block images, this method innovatively integrates a 2D style transfer deep learning model with a brain tissue segmentation algorithm to successfully register 2D fluorescence slices to 3D block space, thereby achieving registration of fluorescence slice maps and mapping of cell body spatial positions.

[0094] like Figure 10 As shown, this embodiment also provides an electronic device, which may include a processor 41 and a memory 42, wherein the memory 42 is coupled to the processor 41. It is worth noting that this figure is exemplary, and other types of structures may be used to supplement or replace this structure to implement data extraction, report generation, communication or other functions.

[0095] like Figure 10 As shown, the electronic device may further include: an input unit 43, a display unit 44 and a power supply 45. It is worth noting that the electronic device does not necessarily have to include Figure 10 In addition, electronic devices may also include Figure 10 For components not shown, reference may be made to the prior art.

[0096] The processor 41 is sometimes also called a controller or an operation control, and may include a microprocessor or other processor devices and / or logic devices. The processor 41 receives inputs and controls the operations of various components of the electronic device.

[0097] The memory 42 may be, for example, one or more of a cache, flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory, or other suitable devices, and may store information such as configuration information of the processor 41 and instructions executed by the processor 41. The processor 41 may execute programs stored in the memory 42 to implement information storage or processing. In one embodiment, the memory 42 also includes a buffer memory to store intermediate information.

[0098] An embodiment of the present invention further provides a computer program product, comprising computer-readable instructions. When the computer-readable instructions are executed in an electronic device, the program product enables the electronic device to perform the operating steps included in the method of the present invention.

[0099] An embodiment of the present invention further provides a storage medium storing computer-readable instructions, wherein the computer-readable instructions enable an electronic device to execute the operation steps included in the method of the present invention.

[0100] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the composition and steps of each example according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0101] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0102] The above-described embodiments are merely specific implementations of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications, substitutions, and improvements within the technical scope disclosed in the present invention, and such modifications, substitutions, and improvements are intended to be encompassed within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A deep learning-based 3D reconstruction and cross-modal registration method for brain slices, characterized by: The following steps are involved: S1, before performing the slicing operation, take a picture of the surface of the brain tissue block to obtain a block surface image, and convert the 2D block surface image into a 3D block surface image; S2, acquiring a fluorescent slice image, and migrating the modality of the fluorescent slice image to a block surface modality; S3, registering the 3D block image with an in vivo MRI image; S4, registering the fluorescence slice image after modality transfer with the registered 3D block surface image.

2. The method for three-dimensional reconstruction and cross-modal registration of brain slices based on deep learning according to claim 1, characterized in that: In S1, the process of converting the 2D block surface image into the 3D block surface image is as follows: S11, resizing the collected original block surface image; S12, the resized 2D block image is input into the SAM2 deep learning model to segment and extract brain tissue; S13, for the block surface image processed by S12, calculating the mean square error (MSE) between adjacent block surface images; S14, filter out the block images whose MSE value is greater than the sum of the overall mean and twice the standard deviation; S15, performing centroid alignment and affine transformation on the block surface images filtered out in step S14; S16, stacking the block surface images processed in step S15, performing 3D reconstruction, and obtaining a 3D block surface image.

3. The method for three-dimensional reconstruction and cross-modal registration of brain slices based on deep learning according to claim 2, characterized in that: In S12, the total loss formula of the SAM2 deep learning model is as follows: L total =λ1L focal +λ2L dice +λ3L MSE +λ4L CE ; in, a t (y)=α×y+(1-α)×(1-y); represents the prediction result of the model, y represents the true label of the mask image, P t represents the predicted probability, γ is the focus parameter, α t As a balancing factor between positive and negative samples, N represents the number of voxels in the image, and s represents the predicted object score. represents the true label of the numerical score, λ1, λ2, λ3, and λ4 are weight coefficients, and the weight coefficients are set to 400:1:1:

1.

4. The method for three-dimensional reconstruction and cross-modal registration of brain slices based on deep learning according to claim 2, characterized in that: In S15, when performing centroid alignment, the centroid of the block face image is calculated by taking the weighted average of the voxel positions and using the voxel intensity as a weighting factor, and the first offset block face image is aligned with the adjacent block face image using the calculated centroid coordinates, and the translation distances in two directions are determined, and then the translation distances are applied to the alignment of subsequent offset block face images; After centroid alignment, the first offset block face image is affine transformed with the adjacent block face images, and then the calculated affine matrix is ​​applied to the subsequent offset block face images to achieve accurate alignment of all block face images.

5. The method for three-dimensional reconstruction and cross-modal registration of brain slices based on deep learning according to claim 1, characterized in that: In S2, the original fluorescence slice image is first downsampled to 500*500 pixels, and then denoising and smoothing are performed. Finally, a 2D CycleGan deep learning model is used to perform modality migration on the denoised fluorescence slice image to generate a synthetic block surface modality from the fluorescence slice image.

6. The method for three-dimensional reconstruction and cross-modal registration of brain slices based on deep learning according to claim 1, characterized in that: In S3, the 3D block image is first modally migrated to a synthetic MRI modality; the 3D block image processed by modal migration is then aligned with the individual MRI image collected in a living state to obtain a first deformation field; the 3D block image processed by modal migration and aligned with the individual MRI is then adjusted to the MRI standard space through a nonlinear registration method of symmetric differential homeomorphism to obtain a second deformation field.

7. The method for three-dimensional reconstruction and cross-modal registration of brain slices based on deep learning according to claim 1, characterized in that: In S4, the registration process of the fluorescence slice image after modality migration and the registered 3D block surface image includes the following steps: S41, the cerebellum is removed from the 3D block image through the cerebellum mask; S42, aligning the synthesized fluorescence slice image of the block face modality with the corresponding block face image through affine transformation; S43, performing tissue segmentation on the fluorescent slice images and corresponding block surface images of all slices to obtain various tissue regions; S44, registering the fluorescent slice image and the block surface image based on a tissue region alignment method to obtain registered fluorescent slice images and block surface images respectively; S45 , reconstructing all registered fluorescence slice images to obtain a 3D fluorescence volume image, and nonlinearly registering the 3D fluorescence volume image to the 3D block surface.

8. A computer program product comprising computer-readable instructions, characterized in that: When executed by a processor, the computer-readable instructions implement the steps in the deep learning-based three-dimensional reconstruction and cross-modal registration method of brain slices according to any one of claims 1 to 7.

9. A computer-readable storage medium comprising computer-readable instructions, characterized in that: When executed by a processor, the computer-readable instructions implement the steps in the deep learning-based three-dimensional reconstruction and cross-modal registration method of brain slices according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: Memory, which stores program instructions; A processor is connected to the memory and executes program instructions in the memory to implement the steps of the deep learning-based three-dimensional reconstruction and cross-modal registration method of brain slices according to any one of claims 1 to 7.

Images