A three-dimensional whole-brain neuron mapping method and device

Abstract

This invention relates to the field of image recognition technology and discloses a method and apparatus for mapping three-dimensional whole-brain neurons. The method includes: acquiring a three-dimensional brain image dataset; processing the three-dimensional brain image dataset to obtain a first image dataset; classifying the first image dataset to obtain k second image datasets; processing the second image data in the k second image datasets to obtain k second-label image datasets; processing the k second-label image datasets to obtain a neuron cell body image set; processing the coordinates of the neuron cell body image set to obtain a cell body coordinate set; and constructing a whole-brain neuron density distribution map based on the cell body coordinate set and the whole-brain neuron coordinate framework to obtain a whole-brain neuron density distribution map. This invention employs CNN, video Swing Transformer, and other deep learning models, utilizing the feature extraction capabilities of CNN and the ability of Swing Transformer to capture global spatial information to achieve large-scale, highly complex whole-brain imaging neuron mapping.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, specifically to a three-dimensional whole-brain neuron mapping method and apparatus. Background Technology

[0002] The three-dimensional whole-brain neuron detection and mapping technology aims to accurately identify and segment neurons in the entire brain structure on a three-dimensional scale, and then compare the detected neuron locations with existing brain atlases to complete functional mapping.

[0003] The current mainstream methods in this field include: detection algorithms based on cell body thresholds, detection algorithms based on convolutional neural networks, detection algorithms based on morphological features, and registration algorithms based on Fourier transform and mutual information.

[0004] Among them, the detection algorithm based on convolutional neural networks uses deep learning technology and is trained with a large amount of labeled data. It can automatically identify and separate neurons in three-dimensional images and performs well in handling complex neuron morphology and noise interference. However, its accuracy may encounter a bottleneck when facing dense cell scenes, and further innovation is needed to overcome this limitation. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes a three-dimensional whole-brain neuron mapping method and apparatus to solve the aforementioned technical problems.

[0006] Firstly, a three-dimensional whole-brain neuron mapping method is provided, including:

[0007] Obtain a 3D brain image dataset;

[0008] The three-dimensional brain image dataset is subjected to image enhancement processing to obtain a first image dataset;

[0009] The first image dataset is classified to obtain k second image datasets;

[0010] The second image data in each of the k second image datasets is labeled using the point clustering algorithm to obtain k second label image datasets;

[0011] The second-label image data in the k second-label image datasets are segmented to obtain a neuron cell body image set;

[0012] The coordinate labeling of the neuron cell body image set is processed to obtain the cell body coordinate set;

[0013] A whole-brain neuron density distribution map is constructed based on the cell body coordinate set and the whole-brain neuron coordinate framework.

[0014] Furthermore, the three-dimensional brain image dataset is subjected to image enhancement processing to obtain a first image dataset, including:

[0015] The three-dimensional image data in the three-dimensional brain image dataset are stitched together to obtain a three-dimensional whole brain image;

[0016] The three-dimensional whole-brain image is segmented to obtain k three-dimensional sub-block images;

[0017] The contrast of each of the k three-dimensional sub-block images is enhanced to obtain k contrast-adjusted three-dimensional sub-block images;

[0018] Median filtering and Gaussian filtering are used to denoise the k contrast-adjusted 3D sub-block image sets respectively to obtain k 3D denoised sub-block images;

[0019] The spatial resolution of each of the k 3D denoising sub-block images is adjusted using interpolation techniques to obtain the first image dataset.

[0020] Furthermore, the first image dataset is classified to obtain k second image datasets, including:

[0021] The k three-dimensional sub-block images in the first image dataset are transformed to obtain k two-dimensional image datasets;

[0022] The k two-dimensional image datasets are divided into k datasets according to a random partitioning algorithm to obtain k datasets;

[0023] Construct a classification network model based on k datasets.

[0024] The classification network model is optimized based on the area enclosed by the ROC curve and the coordinate axis to obtain k second image datasets.

[0025] Furthermore, the second image data in the k second image datasets are labeled using a point clustering algorithm to obtain k second-labeled image datasets, including:

[0026] The k second image data are segmented according to the super-voxel segmentation algorithm to obtain a preliminary segmentation model for the k second image data;

[0027] The label propagation algorithm is used to label the preliminary segmentation models of k second image data respectively to obtain k first-level pseudo-labels;

[0028] By using the similarity between hypervoxels, the k first-level pseudo-labels are updated to obtain the k second-level pseudo-labels.

[0029] The k second-level pseudo-labels are grouped according to the K-means clustering algorithm to obtain k third-level pseudo-labels;

[0030] Based on the weakly supervised training network framework, k third-level pseudo-labels are trained to obtain k second-label image datasets.

[0031] Furthermore, the second-label image data in the k second-label image datasets are segmented to obtain a neuron cell body image set, including:

[0032] Preliminary feature extraction is performed on the second labeled image dataset to obtain k primary image features;

[0033] Deep feature extraction is performed on each of the k primary image features to obtain k advanced image features;

[0034] Global feature interaction is performed on each of the k high-level image features to obtain k feature images;

[0035] The feature fusion module fuses the k feature images respectively to obtain k fused feature images;

[0036] The k fused feature images are transformed to obtain a neuron cell body image set. Further, the coordinate labeling of the neuron cell body image set is processed to obtain a cell body coordinate set, including:

[0037] Connectivity analysis is performed on the set of neuronal cell body images to obtain the centroids of k connected components;

[0038] The coordinates of the centroids of the k connected domains are calculated to obtain the coordinates of the cell bodies of the k neurons.

[0039] The coordinates of the k neuron cell bodies are integrated to obtain a cell body coordinate set.

[0040] Furthermore, a whole-brain neuron density distribution map is constructed based on the cell body coordinate set and the whole-brain neuron coordinate framework, resulting in a whole-brain neuron density distribution map, including:

[0041] The cell body coordinate set is mapped onto the whole brain neuron coordinate frame to obtain the mapped cell body coordinate set;

[0042] The mapped cell coordinate set is registered using an affine transformation algorithm to obtain the registered cell coordinate set.

[0043] The brain regions are divided according to the registered cell body coordinate set to obtain the neuron density of k brain regions;

[0044] The neuron density of each of the k brain regions was calculated to obtain the neuron density values ​​for each of the k brain regions.

[0045] The neuron density values ​​of k brain regions are visualized to obtain a whole-brain neuron density distribution map.

[0046] Secondly, a three-dimensional whole-brain neuron mapping device is provided, comprising:

[0047] The acquisition module is used to acquire a 3D brain image dataset;

[0048] An enhancement processing module is used to perform image enhancement processing on the three-dimensional brain image dataset to obtain a first image dataset;

[0049] The classification module is used to classify the first image dataset to obtain k second image datasets;

[0050] The annotation processing module is used to annotate the second image data in the k second image datasets according to the point clustering algorithm to obtain k second label image datasets;

[0051] The segmentation processing module is used to segment the second label image data in the k second label image datasets to obtain a neuron cell body image set;

[0052] The labeling processing module is used to process the coordinate labels of the neuron cell body image set to obtain the cell body coordinate set;

[0053] The module is used to construct a whole-brain neuron density distribution map based on the cell body coordinate set and the whole-brain neuron coordinate frame, thus obtaining the whole-brain neuron density distribution map.

[0054] Furthermore, a three-dimensional whole-brain neuron mapping device includes a processor and a memory storing program instructions, the processor being configured to execute a three-dimensional whole-brain neuron mapping method as described in any of the preceding claims when running the program instructions.

[0055] Thirdly, an electronic device is provided, including the three-dimensional whole-brain neuron mapping device described above.

[0056] The invention employing the above technical solution has the following advantages:

[0057] 1. This invention employs CNN, video Swing Transformer, and other related deep learning models. By leveraging the advantages of CNN in feature extraction and the ability of Swing Transformer to capture global spatial information, it achieves neuron mapping for large-scale, highly complex whole-brain imaging. Furthermore, it can significantly improve the accuracy of cell body segmentation when processing complex three-dimensional brain imaging data, especially in dense cell body regions, where the segmentation effect is even more significant.

[0058] 2. This invention effectively reduces the amount of irrelevant data blocks that need to be processed, lowering the computational burden on the segmentation module. Simultaneously, the lightweight design of the classification network reduces the risk of overfitting, enabling the model to maintain high efficiency during both training and inference phases. This optimized computational flow ensures rapid analysis at the whole-brain scale.

[0059] 3. This invention greatly reduces the amount of data labeling work required for running the segmentation network by using a weakly supervised network model, and significantly reduces the amount of work required to train segmentation data using this invention.

[0060] 4. The present invention can maintain stable detection performance under different imaging conditions (such as brightness fluctuations, uneven fluorescence intensity, etc.), can adapt to various complex environments, and improves the versatility of practical applications.

[0061] 5. This invention achieves fully automated operation from data preprocessing, classification, segmentation to final coordinate mapping, reducing the need for manual intervention. Researchers only need minimal setup to complete automated detection and analysis of neurons throughout the brain. This fully automated process significantly improves research efficiency, making large-scale neuroscience research possible.

[0062] 6. This invention provides an in-depth analysis of neuronal distribution patterns and neural network connectivity patterns within brain regions. This capability enables researchers to better understand the complex functions of the brain at both macroscopic and microscopic levels, providing solid data support for neurological disease research and brain-computer interface development. Attached Figure Description

[0063] To more clearly illustrate the specific embodiments of the present invention, the accompanying drawings used in the specific embodiments will be briefly described below. In all the drawings, the elements or parts are not necessarily drawn to scale.

[0064] Figure 1 This is a front view of a three-dimensional whole-brain neuron mapping method and device according to the present invention;

[0065] Figure 2 is a schematic diagram of whole-brain imaging data preprocessing in a three-dimensional whole-brain neuron mapping method of the present invention;

[0066] Figure 3 is a schematic diagram of the effect of the maximum value projection algorithm on the three-dimensional sub-block data in the three-dimensional whole brain neuron mapping method of the present invention.

[0067] Figure 4 This is a schematic diagram of the classification network structure in a three-dimensional whole-brain neuron mapping method of the present invention;

[0068] Figure 5 This is a schematic diagram of the cell body segmentation network structure in a three-dimensional whole-brain neuron mapping method of the present invention;

[0069] Figure 6 is a schematic diagram of the cell body segmentation effect in a three-dimensional whole-brain neuron mapping method of the present invention;

[0070] Figure 7 is a schematic diagram of obtaining the cell body position from the segmented region in a three-dimensional whole-brain neuron mapping method of the present invention;

[0071] Figure 8 This is a schematic diagram illustrating the registration and fitting of cell body positions with whole-brain neuron coordinates in a three-dimensional whole-brain neuron mapping method according to the present invention.

[0072] Figure 9 This is a flowchart of a three-dimensional whole-brain neuron mapping method according to the present invention;

[0073] Figure 10 This is a flowchart of a three-dimensional whole-brain neuron mapping device according to the present invention. Detailed Implementation

[0074] The embodiments of the technical solution of the present invention will now be described in detail with reference to the accompanying drawings. These embodiments are merely illustrative of the technical solution of the present invention and are therefore intended to limit the scope of protection of the present invention.

[0075] like Figures 1-10 As shown, a three-dimensional whole-brain neuron mapping method of the present invention includes:

[0076] Step S01: Obtain a three-dimensional brain image dataset;

[0077] Step S02: Perform image enhancement processing on the three-dimensional brain image dataset to obtain the first image dataset;

[0078] Step S03: Classify the first image dataset to obtain k second image datasets;

[0079] Step S04: Label the second image data in the k second image datasets according to the point clustering algorithm to obtain k second label image datasets;

[0080] Step S05: Segment the second label image data in the k second label image datasets to obtain a neuron cell body image set;

[0081] Step S06: Process the coordinate labeling of the neuron cell body image set to obtain the cell body coordinate set;

[0082] Step S07: Construct a whole-brain neuron density distribution map based on the cell body coordinate set and the whole-brain neuron coordinate frame to obtain the whole-brain neuron density distribution map.

[0083] In some embodiments, image enhancement processing is performed on the three-dimensional brain image dataset to obtain a first image dataset, including:

[0084] Three-dimensional image data from a three-dimensional brain image dataset are stitched together to obtain a three-dimensional whole-brain image.

[0085] The three-dimensional whole brain image is segmented to obtain k three-dimensional sub-block images;

[0086] The contrast of each of the k 3D sub-block images is enhanced to obtain k contrast-adjusted 3D sub-block images;

[0087] Median filtering and Gaussian filtering are used to denoise the k contrast-adjusted 3D sub-block image sets respectively, resulting in k denoised 3D sub-block images;

[0088] The spatial resolution of each of the k 3D denoised sub-block images is adjusted using interpolation techniques to obtain the first image dataset.

[0089] Specifically, this invention uses Python code to automatically preprocess whole-brain image data, improving data quality and maintaining consistency in format, size, pixels, and other information, reducing the possibility of errors during network input, and further improving network performance during training.

[0090] The preprocessing steps mainly include: 1. Image stitching. In this step, a feature-point-based stitching algorithm is used. This method first identifies significant feature points in each image segment using a keypoint detection algorithm. These feature points are corners or edges in the image, representing the most identifiable parts. After identifying the keypoints, a matching algorithm is used to pair corresponding feature points in different image segments, thereby determining the overlapping areas between images. After matching, a random sampling consensus algorithm is used to eliminate incorrect matches, and a transformation matrix is ​​calculated to ensure precise alignment of the image segments. Finally, image fusion technology is used to seamlessly stitch these aligned image segments together to generate a complete three-dimensional brain image.

[0091] 2. Sub-block partitioning. A complete 3D brain image far exceeds the learning capacity of a network. Therefore, it is necessary to partition the complete brain image into sub-blocks in the preprocessing step. The algorithm automatically divides the image into sub-blocks of a specified size according to the X-axis, Y-axis, and Z-axis numbering order. The numbering facilitates the statistical analysis of sub-block information.

[0092] 3. Brightness Adjustment. Histogram equalization was employed. This method redistributes the pixel values ​​of the image to make them more uniform across the brightness range. First, the gray-level histogram of the image is calculated, and the number of pixels at each gray level is counted. Then, the cumulative distribution function is used to map the original pixel values ​​to the new gray levels, thereby enhancing the image contrast.

[0093] 4. Noise Removal. A combination of median filtering and Gaussian filtering was used to remove noise. Median filtering is suitable for removing salt-and-pepper noise in images by replacing each pixel value with the median of its neighboring pixels, effectively eliminating isolated noise points without blurring edges. Gaussian filtering is suitable for removing Gaussian noise by performing a convolution operation with the image using a convolution kernel to achieve image smoothing. Combining the two methods can effectively remove various types of noise while preserving image details.

[0094] 5. Data Standardization. In the spatial resolution standardization process, cubic interpolation was used to adjust the image resolution. This method estimates the new pixel value using the values ​​of 16 neighboring pixels, thus scaling the image while preserving image details. The preprocessed image is shown in Figure 2, where Figure 2a shows the image after whole-brain stitching, and Figure 2b shows the original image segmented into sub-blocks. Figure 2c This represents the image after brightness adjustment, noise removal, and data standardization.

[0095] In some embodiments, the first image dataset is classified to obtain k second image datasets, including:

[0096] Transform the k three-dimensional sub-block images in the first image dataset to obtain a k two-dimensional image dataset;

[0097] The k two-dimensional image datasets are divided into k datasets using a random partitioning algorithm.

[0098] Construct a classification network model based on k datasets;

[0099] The classification network model is optimized based on the area enclosed by the ROC curve and the coordinate axis to obtain k second image datasets.

[0100] Specifically, after obtaining high-quality sub-block brain image data, it was found that existing networks still occupy a large amount of GPU memory when reading and learning features from these high-dimensional data sub-blocks, significantly increasing training time and server space. In order to enable rapid training and testing of the network, the high-dimensional original data sub-blocks were dimensionality-reduced using a convolution algorithm to obtain low-dimensional data that can be input into the network.

[0101] CNN-based classification networks can distinguish between networks containing and without neuron cell bodies. Data containing neuron cell bodies is selected and input into the network for training, while data without neuron cell bodies is not input, reducing the computational load of the segmentation network. The steps for using this network are as follows: 1. Use the maximum intensity projection method to quickly batch process 3D sub-block data into 2D image data to obtain 2D image data that can be input into the classification network; use the Maximum Intensity Projection (MIP) method to quickly convert the 3D sub-block data into 2D image data. The MIP method projects along the Z-axis of the 3D image data, using the maximum intensity value of each image on the Z-axis as the pixel value of the 2D image, thus preserving the neurons in each image on the Z-axis dimension on a single image. This method preserves the most salient features in the 3D data and is suitable for neuron cell body detection. Batch processing technology further accelerates this process, allowing multiple sub-blocks to be converted simultaneously, significantly improving processing efficiency. The image after MIP algorithm processing is shown in Figure 3, where Figure 3a represents a dense neuron cell body image, Figure 3b represents a sparse neuron cell body image, and Figure 3c represents an image containing interference. Figure 3d This represents a background image that does not contain neurons.

[0102] 2. A fast random partitioning algorithm is used to divide the dataset into training, validation, and test sets for the classification network, resulting in three randomly partitioned datasets. To ensure the fairness and effectiveness of the training, validation, and testing processes, a fast random partitioning algorithm is used to divide the dataset into training, validation, and test sets. This algorithm randomly selects data samples and distributes them to different subsets according to a predefined ratio (e.g., 4:1:1). This ensures that each subset is representative, thus avoiding overfitting or underfitting during the training and testing of the classification network.

[0103] 3. The dataset is input into a lightweight classification network backbone to obtain image features. The 2D image data of the pre-divided dataset is then input into a lightweight shallow classification network backbone. A key feature is the introduction of shortcut connections, allowing the network to learn image features more effectively. The shallow structure further reduces the network's complexity and computational cost, making it suitable for processing large-scale whole-brain imaging data. Network structures, such as... Figure 4 As shown in the diagram, the blocks and arrows represent different processing layers and connection methods. Blocks ① and ② represent convolutional layers with different amplitudes, both with a kernel size of 3×3. Convolutional layer ② has a downsampling step of 2. Arrow ③ represents a stack of average pooling, fully connected layers, and the Softmax function. Dashed arrow ④ indicates the direction of data flow, while dashed arrows ⑤ and ⑥ represent shortcut connections with or without 1×1 convolutional layers, respectively.

[0104] 4. Fully connected layers are used to process image features, increasing the network's processing dimensionality and improving the classification network's performance in this task, resulting in a classification network model. The extracted image features are then input into fully connected layers for processing. The role of fully connected layers is to further linearly combine the high-dimensional features extracted by the preceding convolutional layers, increasing the network's expressive power. In this step, linear layers are used to increase processing dimensionality, and the ReLU activation function is used to increase the network's non-linear expressive power, thereby improving the overall performance of the classification network in this task. Finally, the output classification results are used to determine whether each input image contains neuron cell bodies.

[0105] 5. During training, the algorithm automatically selects and saves the model with the highest AUC value, thus obtaining the optimal model in each training round. In the training of the classification network, AUC is used as a key metric for model evaluation. The higher the AUC value, the better the model's classification performance. During training, the algorithm automatically calculates the AUC value at the end of each training cycle and saves the model with the highest AUC value. This strategy ensures that the best model from each training round is saved for subsequent testing and practical applications.

[0106] 6. The best-performing model after ten rounds of training was used to classify two-dimensional projected images of the whole brain data, obtaining two-dimensional projected images containing neuronal cell bodies from the whole brain data. After multiple rounds of training, the algorithm selects the best-performing model for classifying the whole brain data. In this test, ten rounds of training were conducted, and the model with the highest AUC was selected from these ten rounds. Using this optimal model, all two-dimensional projected images were classified, and images containing neuronal cell bodies were identified and labeled.

[0107] 7. Referring to the names of the two-dimensional projected images corresponding to the results of the classification network, three-dimensional data sub-blocks containing neuronal cell bodies are obtained from the whole-brain data; the output of the classification network is used to identify which two-dimensional projected images contain neuronal cell bodies. By mapping the labels of these two-dimensional images to the original three-dimensional data sub-blocks, it can be determined which three-dimensional sub-blocks contain neuronal cell bodies. This step directly maps the output of the classification network back to the original data, generating an efficient and accurate three-dimensional neuronal cell body localization result.

[0108] The classification network algorithm of this invention was tested against existing classification networks on ten random datasets. Table 1 shows the test results for classifying images projected onto MIP sub-blocks of size 128×128×128. The performance of the classification network was evaluated using AUC (area under the average ROC curve over ten tests), ACC (average accuracy over ten tests), number of parameters, FLOPs (floating-point computations), and FPS (average number of frames displayed per second over ten tests). The classification networks tested included: Densenet121 (a 121-layer densely connected network), Resnet18 (an 18-layer residual network), and Vgg16 (a 16-layer visual geometry group network). The testing process was as follows: 180 MIP images from four different mice were selected to form the initial dataset. Positive samples were manually selected from the dataset, covering various cases of different sibling sub-blocks, such as different densities, brightness levels, and distribution locations. Negative samples were randomly selected from the background images. The initial dataset was randomly divided into three subsets: a training dataset (120 images), a validation dataset (30 images), and a test dataset (30 images). The initial dataset was randomly reassigned 10 times to verify the stability of the classification network. Each of the four methods was trained for 100 epochs. The model with the highest AUC during each training epoch was selected for testing. After loading the model, the corresponding test dataset was read to test various metrics, and the results are finally obtained in the table.

[0109] Table 1. Classification Network Test Results

[0110]

[0111] In some embodiments, the second image data in the k second image datasets are labeled according to a point clustering algorithm to obtain k second labeled image datasets, including:

[0112] The k second image data are segmented according to the super-voxel segmentation algorithm to obtain a preliminary segmentation model for the k second image data;

[0113] The label propagation algorithm is used to label the preliminary segmentation models of k second image data respectively, resulting in k first-level pseudo-labels;

[0114] By using the similarity between hypervoxels, the k first-level pseudo-labels are updated to obtain the k second-level pseudo-labels;

[0115] The k second-level pseudo-labels are grouped according to the K-means clustering algorithm to obtain k third-level pseudo-labels;

[0116] Based on the weakly supervised training network framework, k third-level pseudo-labels are trained to obtain k second-label image datasets.

[0117] Specifically, a weakly supervised 3D data network based on a point clustering algorithm is used to reduce the amount of manually labeled data required for 3D whole-brain segmentation training and testing. The steps of this method are as follows:

[0118] 1. A supervoxel segmentation algorithm based on the geometric similarity of 3D data is used to obtain a preliminary data segmentation model, reducing the computational burden of subsequent processing. The algorithm employs a supervoxel segmentation method, which performs preliminary segmentation based on the geometric similarity of 3D data. First, the input 3D data is preprocessed to extract feature points and surfaces. Using a geometric similarity metric, points are assigned to different supervoxels based on their spatial location, normal direction, and surface features. Points with high geometric similarity are aggregated into a single supervoxel to generate a preliminary segmentation model, reducing the complexity of subsequent calculations.

[0119] 2. Initial pseudo-labels are obtained using a label propagation algorithm based on the supervoxel segmentation model;

[0120] 3. A graph model was constructed based on 3D geometric hypervoxels, where each hypervoxel is considered a node in the graph, and the edges between nodes represent the similarity between two hypervoxels. The predicted probability distribution of the hypervoxels was obtained by calculating all points within each hypervoxel.

[0121]

[0122] Among them, y i p represents the initial pseudo-label of the i-th superprime point. i c represents the coordinates of each supervoxel. i Indicates the color of each supervoxel. It corresponds to point p. i The basic true class, Θ, represents the output of the backbone network.

[0123] 4. Use a relational network to learn the similarity between supervoxels, and use a Transformer for iterative updating of pseudo-labels. Obtain the similarity measure of the relational network between supervoxels to optimize label propagation; after obtaining the predicted probability distribution of supervoxels, use the relational network, and the following formula optimizes the label propagation process:

[0124]

[0125] Where R is the output of the relational network, ψ p (j k ) represents supervoxel v j and v′ jSimilarity between them, ψ u (y j |V,Θ) represents the hypervoxel v j With label y j The negative logarithmic probability can be expressed as:

[0126] ψ u (y j |V,Θ)=-log P(y j |V,Θ)

[0127] Both low-level features and learned features are used to measure similarity, as shown in the following formula:

[0128]

[0129] Among them, Ι(y) j ,y j′ ) indicates if y j ,y j′ This part is 1 if it has a different predicted label, and 0 otherwise. These two items indicate that if hypervoxels with similar features are predicted to different categories, it will increase costs. j ,c j′ ,p j ,p j′ ,u j ,u j′ Indicates the corresponding supervoxel v j The normalized average color, average coordinates, and average features.

[0130] Furthermore, a Transformer network is used to learn the similarity between supervoxels. The network predicts the relationships between them by analyzing the features of the supervoxels. Based on the similarity metric output by the relationship network, the pseudo-labels are iteratively updated using the Transformer framework to gradually improve their quality. The specific formula is as follows:

[0131]

[0132] Where Q, K, and V represent the MLP (Multilayer Perceptron) layer in the Transformer network, respectively, and F... j Represents the feature vector. The Transformer network computes F... j With category prototype k c Similarity between them to aggregate category prototype k c The obtained features With F j The predicted results of the supervoxel semantic category are concatenated and used for iterative updates.

[0133] 5. The K-means clustering algorithm is used to further group the supervoxels, obtaining instance-level pseudo-labels; K-means clustering is then performed on the supervoxels to classify similar supervoxels as the same instance. The specific formula is as follows:

[0134]

[0135] This formula achieves clustering by minimizing the K-means, grouping n hypervoxels. Divided into K clusters The centroid of the k-th cluster is set to maximize the similarity of supervoxels within each cluster and maximize the difference between different clusters. Each clustering result corresponds to an instance label, resulting in more fine-grained instance segmentation pseudo-labels, which can be represented as:

[0136] y i =arg min k ||f i -μ k ||

[0137] Among them, f i Representing each supervoxel, y i This indicates the corresponding pseudo-tag.

[0138] 6. A weakly supervised network training framework is adopted, and label propagation and network training are performed iteratively to obtain a weakly supervised network model. The generated pseudo-labels are used to iteratively train the network, and the model's performance is gradually improved by continuously optimizing the pseudo-labels and network weights. A total of 200 epochs are trained, and the last model is used as the base model of the weakly supervised network for subsequent segmentation networks.

[0139] In some embodiments, the second-label image data in the k second-label image datasets are segmented to obtain a neuron cell body image set, including:

[0140] Preliminary feature extraction was performed on the second-labeled image dataset to obtain k primary image features;

[0141] Deep feature extraction is performed on k primary image features to obtain k high-level image features;

[0142] Global feature interaction is performed on k high-level image features respectively to obtain k feature images;

[0143] The feature fusion module fuses the k feature images separately to obtain k fused feature images;

[0144] The k fused feature images are transformed to obtain a set of neuron cell body images.

[0145] Specifically, a segmentation network based on the video Swin Transformer is used to separate the neuron cell body region from other parts of the image;

[0146] 1. Preliminary feature extraction of 3D whole-brain data sub-blocks yields primary features of the image to be segmented. The input 3D sub-blocks first pass through a convolutional layer, which uses multiple convolutional kernels to operate on the input data and extract primary spatial features. The output of the convolutional layer is processed by a Batch Normalization (BN) layer and a non-linear activation function (such as ReLU) to enhance the training effect of the network and accelerate convergence.

[0147] 2. A multi-layer convolutional encoder, combining pseudo-labels and instance data labels, is used for deep feature extraction to obtain high-level image features and local image features. After obtaining preliminary features, a multi-layer convolutional encoder is used to extract deeper features from the image, combining pseudo-labels and instance data labels. The encoder module consists of multiple convolutional layers, pooling layers, and skip connections. The segmentation network structure is shown in [link to network structure]. Figure 5 As shown in the figure, cubes and arrows with different numbers represent different processing layers in the segmentation network. Cube ① represents the convolutional encoder layer, and cube ② represents the convolutional decoder layer; arrow ③ represents the downsampling layer, and arrow ④ represents the upsampling layer. They are all convolutional layers with a stride of 2; arrow ⑤ represents a skip connection; the Transformer layer consists of L video Swing Transformer modules, where LN represents the normalization layer, W-MSA represents the window multi-head self-attention mechanism module, and SW-MSA represents the translation window multi-head self-attention mechanism module. This indicates a residual connection.

[0148] 3. By employing global feature interaction based on the video-based Swing Transformer, the network model can capture global features of the image, resulting in a more effective feature network. The Swing Transformer performs global feature interaction on the extracted features through W-MSA and SW-MSA. These self-attention mechanisms enable the network to capture global features of the image and learn deeper semantic information. The self-attention mechanism is the core of the Transformer network; for each location in the input feature map, its correlation with other locations is calculated using the following formula:

[0149] Q = XW Q

[0150] K = XW K

[0151] V = XW V

[0152] Where Q is the query matrix, K is the key matrix, V is the value matrix, X is the input feature map, and W... Q \W K \W V These are the corresponding linear transformation matrices. The formula for calculating the attention weights is as follows:

[0153]

[0154] Where, d k The dimension of the key vector is used to scale the dot product result. Then, attention weights are used to weight the matrix Z = A·V.

[0155] W-MSA and SW-MSA are implemented based on the above self-attention mechanism principle. The Swing Transformer divides the feature map into multiple non-overlapping windows, and calculates the self-attention score independently within each window:

[0156]

[0157] 4. Using an upsampling and skip connection feature fusion module, high resolution is restored from low resolution to obtain important spatial information and details. After obtaining global features, the network needs to restore these features to their original resolution to generate fine-grained segmentation results. This step is achieved through the upsampling and skip connection feature fusion module. The upsampling operation gradually restores the feature map to high resolution, while the skip connections combine the features from the encoder layer with those from the decoder layer, ensuring that important spatial information and details are preserved.

[0158] 5. Generate the final segmentation result using a convolutional layer. After all feature extraction and fusion are completed, the feature map is finally transformed into the final segmentation result through a convolutional layer. This convolutional layer uses a 1×1 convolutional kernel to map the high-dimensional features to the target class space, generating the class probability distribution for each pixel or voxel, thus determining the final segmentation result. A schematic diagram of the final segmentation result is shown in Figure 6.

[0159] The segmentation network of this invention is then tested against existing segmentation networks on a specific dataset. Table 2 shows the test results for segmenting a 512×512×512 three-dimensional data sub-block. The test metrics for the segmentation network include DSC (Dice Similarity Coefficient), Precision, Recall, and F1 score. The formulas for calculating these metrics are as follows:

[0160]

[0161] Among them, S g S represents the result of automatic segmentation using a segmentation network.t This indicates the result of manual labeling.

[0162]

[0163] Where, N GT N represents the number of GT (Ground Truth, label) neurons. TP N represents the number of true positive neurons. detected This indicates the number of neurons detected.

[0164] The segmentation networks tested included: SegNet (Image Segmentation Network), 3DU-Net (3D U-Net), Res-U-Net (Residual U-Net), and TransBTS (Transformer in 3D CNN for 3D Brain Tumor Segmentation). The testing process involved dividing the dataset into three subsets: a training dataset (22 sub-blocks), a validation dataset (6 sub-blocks), and a test dataset (10 sub-blocks). Dynamic random flipping and random centering were used to augment the training data. The segmentation networks were trained using the Adam optimizer with an initial learning rate of 0.0001. A learning rate decay strategy was employed, decreasing the learning rate by 10% every 10 epochs. The optimal model was obtained using an early stopping strategy, with each epoch containing 80 iterations. The minimum batch size was set to 1 during the segmentation network training phase. Dice loss was used to train the network, and the model with the highest DSC during training was selected as the test model. During the testing phase, the optimal model is loaded and the test dataset is read, ultimately yielding the results in the table.

[0165] Table 2. Test Results of the Segmentation Network

[0166]

[0167] In some embodiments, the coordinate labeling of the neuron cell body image set is processed to obtain a cell body coordinate set, including:

[0168] Connectivity analysis was performed on the neuron cell body image set to obtain the centroids of k connected regions;

[0169] The coordinates of the centroids of the k connected domains are calculated to obtain the coordinates of the cell bodies of the k neurons.

[0170] The coordinates of the cell bodies of k neurons are integrated to obtain the cell body coordinate set.

[0171] Specifically, based on connected component analysis, an algorithm is used to calculate the cell body centroid to determine the cell body location using the segmented result region. This algorithm receives the output of a segmentation network, which produces a 3D binary image where each voxel is labeled as belonging to a cell body or not. Connected component analysis is used to process the segmentation results. Connected component analysis groups adjacent voxels labeled as cell bodies into the same region (i.e., a connected region). To reduce noise and false detections, we use an algorithm to remove very small connected regions, deleting connected regions with an area or volume smaller than a certain threshold, thus retaining only sufficiently large connected regions. These large connected regions are more likely to be true cell bodies. For each remaining connected region, its centroid is calculated, where the centroid (x...)... c ,y c ,z c The calculation formula is:

[0172]

[0173] Where |R| is the number of voxels in the connected region, and (x,y,z) are the coordinates of each voxel in the region.

[0174] The calculated centroid coordinates will be saved in a specific format (e.g., .swc file) for subsequent analysis and processing. Figure 7 shows a schematic diagram of the centroids of the sub-blocks after algorithm processing, illustrating the centroid coordinates of neuronal cell bodies in the three-dimensional mouse whole-brain data demonstrated in this example.

[0175] In some embodiments, a whole-brain neuron density distribution map is constructed based on the cell body coordinate set and the whole-brain neuron coordinate frame to obtain the whole-brain neuron density distribution map, including:

[0176] The cell body coordinate set is mapped onto the whole brain neuron coordinate frame to obtain the mapped cell body coordinate set;

[0177] The mapped cell body coordinate set is registered using the affine transformation algorithm to obtain the registered cell body coordinate set.

[0178] The brain regions are divided based on the registered cell body coordinate set to obtain the neuron density of k brain regions;

[0179] The neuron density of each of the k brain regions was calculated to obtain the neuron density values ​​for each of the k brain regions.

[0180] The neuron density values ​​of k brain regions are visualized to obtain a whole-brain neuron density distribution map.

[0181] Specifically, a fast fitting algorithm is used to register and map the detected cell bodies to the existing whole-brain neuron coordinate frame. The obtained centroid coordinates are then mapped to the whole-brain neuron coordinate frame using a scaling factor. Scaling calculations are typically performed based on the resolution difference between the actual image and the coordinate system to accurately map the centroid coordinates to the standard coordinate system. An automatic neuron technology algorithm within each brain region is used to obtain neuron density and connection strength between different brain regions. For different brain regions or the entire brain region, an affine transformation algorithm is used to register the detected cell body positions to the existing standard coordinate system. This is achieved using the registration method in SimpleITK. Based on the mapped coordinate information, each centroid is assigned to a corresponding brain region, and centroids belonging to a certain brain region are included in the neuron count of that region. A schematic diagram of the whole-brain neuron cell body brain region mapping is shown below. Figure 8 As shown, different colors represent different brain regions. After obtaining the total number of neurons in each brain region, the neuron density of that brain region is calculated. The neuron density is usually expressed as the number of neurons per unit volume, which can be further standardized to the number of neurons per unit brain region volume. Finally, statistical methods based on neuron distribution are used to analyze the neuronal connectivity strength between different brain regions.

[0182] Neuron distribution density formula:

[0183]

[0184] in, It refers to the density of neurons in a brain region, N. cells It is the total number of neurons in the brain region, V region It refers to the volume of the brain region.

[0185] Connection strength formula:

[0186]

[0187] Where σ is the connection strength, d(P) i ,P j ) is the center of mass P i and P j The Euclidean distance between them.

[0188] In other embodiments, a three-dimensional whole-brain neuron mapping device is provided, comprising:

[0189] The acquisition module is used to acquire a 3D brain image dataset;

[0190] The enhancement processing module is used to perform image enhancement processing on the three-dimensional brain image dataset to obtain the first image dataset;

[0191] The classification module is used to classify the first image dataset to obtain k second image datasets;

[0192] The annotation processing module is used to annotate the second image data in the k second image datasets according to the point clustering algorithm, so as to obtain k second label image datasets;

[0193] The segmentation processing module is used to segment the second-label image data in the k second-label image dataset to obtain a neuron cell body image set;

[0194] The labeling module is used to label the coordinates of the neuron cell body image set to obtain the cell body coordinate set.

[0195] The module is used to construct a whole-brain neuron density distribution map based on the cell body coordinate set and the whole-brain neuron coordinate frame, thus obtaining the whole-brain neuron density distribution map.

[0196] In some embodiments, a three-dimensional whole-brain neuron mapping device includes a processor and a memory storing program instructions, the processor being configured to execute a three-dimensional whole-brain neuron mapping method as described above when the program instructions are executed.

[0197] In other embodiments, an electronic device (e.g., a mobile phone, a computer) is provided, including the aforementioned three-dimensional whole-brain neuron mapping device.

[0198] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.

Claims

1. A three-dimensional whole-brain neuron mapping method, characterized in that, include: Obtain a 3D brain image dataset; The three-dimensional brain image dataset is subjected to image enhancement processing to obtain a first image dataset; The first image dataset is classified to obtain k second image datasets; The second image data in each of the k second image datasets is labeled using the point clustering algorithm to obtain k second label image datasets; The second-label image data in the k second-label image datasets are segmented to obtain a neuron cell body image set; The coordinate labeling of the neuron cell body image set is processed to obtain the cell body coordinate set; A whole-brain neuron density distribution map is constructed based on the cell body coordinate set and the whole-brain neuron coordinate framework. The second image data in each of the k second image datasets is labeled using a point clustering algorithm to obtain k second-labeled image datasets, including: The k second image data are segmented according to the super-voxel segmentation algorithm to obtain a preliminary segmentation model for the k second image data; The label propagation algorithm is used to label the preliminary segmentation models of k second image data respectively to obtain k first-level pseudo-labels; By using the similarity between hypervoxels, the k first-level pseudo-labels are updated to obtain the k second-level pseudo-labels. The k second-level pseudo-labels are grouped according to the K-means clustering algorithm to obtain k third-level pseudo-labels; Based on the weakly supervised training network framework, k third-level pseudo-labels are trained to obtain k second-label image datasets. The second-label image data in the k second-label image datasets are segmented to obtain a neuron cell body image set, including: Preliminary feature extraction is performed on the second labeled image dataset to obtain k primary image features; Deep feature extraction is performed on each of the k primary image features to obtain k advanced image features; Global feature interaction is performed on each of the k high-level image features to obtain k feature images; The feature fusion module fuses the k feature images respectively to obtain k fused feature images; The k fused feature images are transformed to obtain a neuron cell body image set.

2. The three-dimensional whole-brain neuron mapping method according to claim 1, characterized in that, The three-dimensional brain image dataset is subjected to image enhancement processing to obtain a first image dataset, including: The three-dimensional image data in the three-dimensional brain image dataset are stitched together to obtain a three-dimensional whole brain image; The three-dimensional whole-brain image is segmented to obtain k three-dimensional sub-block images; The contrast of each of the k three-dimensional sub-block images is enhanced to obtain k contrast-adjusted three-dimensional sub-block images; Median filtering and Gaussian filtering are used to denoise the k contrast-adjusted 3D sub-block image sets respectively to obtain k 3D denoised sub-block images; The spatial resolution of each of the k 3D denoising sub-block images is adjusted using interpolation techniques to obtain the first image dataset.

3. The three-dimensional whole-brain neuron mapping method according to claim 2, characterized in that, The first image dataset is classified to obtain k second image datasets, including: The k three-dimensional sub-block images in the first image dataset are transformed to obtain k two-dimensional image datasets; The k two-dimensional image datasets are divided into k datasets according to a random partitioning algorithm to obtain k datasets; Construct a classification network model based on k datasets. The classification network model is optimized based on the area enclosed by the ROC curve and the coordinate axis to obtain k second image datasets.

4. The three-dimensional whole-brain neuron mapping method according to claim 1, characterized in that, The coordinate labeling of the neuron cell body image set is processed to obtain a cell body coordinate set, including: Connectivity analysis is performed on the set of neuronal cell body images to obtain the centroids of k connected components; The coordinates of the centroids of the k connected domains are calculated to obtain the coordinates of the cell bodies of the k neurons. The coordinates of the k neuron cell bodies are integrated to obtain a cell body coordinate set.

5. The three-dimensional whole-brain neuron mapping method according to claim 1, characterized in that, A whole-brain neuron density distribution map was constructed based on the cell body coordinate set and the whole-brain neuron coordinate frame, resulting in the whole-brain neuron density distribution map, including: The cell body coordinate set is mapped onto the whole brain neuron coordinate frame to obtain the mapped cell body coordinate set; The mapped cell coordinate set is registered using an affine transformation algorithm to obtain the registered cell coordinate set. The brain regions are divided according to the registered cell body coordinate set to obtain the neuron density of k brain regions; The neuron density of each of the k brain regions was calculated to obtain the neuron density values ​​for each of the k brain regions. The neuron density values ​​of k brain regions are visualized to obtain a whole-brain neuron density distribution map.

6. A three-dimensional whole-brain neuron mapping device, characterized in that, include: The acquisition module is used to acquire a 3D brain image dataset; An enhancement processing module is used to perform image enhancement processing on the three-dimensional brain image dataset to obtain a first image dataset; The classification module is used to classify the first image dataset to obtain k second image datasets; The first annotation processing module is used to annotate the second image data in the k second image datasets according to the point clustering algorithm to obtain k second label image datasets; The first segmentation processing module is used to segment the second label image data in the k second label image datasets to obtain a neuron cell body image set; The labeling processing module is used to process the coordinate labels of the neuron cell body image set to obtain the cell body coordinate set; The module is used to construct a whole-brain neuron density distribution map based on the cell body coordinate set and the whole-brain neuron coordinate frame, thus obtaining the whole-brain neuron density distribution map. The second segmentation processing module is used to segment the k second image data according to the super-voxel segmentation algorithm to obtain a preliminary segmentation model of the k second image data. The second annotation processing module is used to perform annotation processing on the preliminary segmentation models of k second image data respectively according to the label propagation algorithm to obtain k first-level pseudo-labels; The update module is used to update the k first-level pseudo-labels based on the similarity between supervoxels to obtain k second-level pseudo-labels; The grouping module is used to group the k second-level pseudo-labels according to the K-means clustering algorithm to obtain k third-level pseudo-labels; The training module is used to train k third-level pseudo-labels respectively according to the weakly supervised training network framework to obtain k second-label image datasets; The preliminary feature extraction module is used to perform preliminary feature extraction on the second labeled image dataset to obtain k primary image features; The deep feature extraction module is used to extract deep features from the k primary image features respectively to obtain k high-level image features; A global feature interaction module is used to perform global feature interaction on k of the high-level image features respectively to obtain k feature images; The fusion module is used to fuse the k feature images according to the feature fusion module to obtain k fused feature images; The conversion processing module is used to convert and process the k fused feature images respectively to obtain a neuron cell body image set.

7. A three-dimensional whole-brain neuron mapping device, comprising a processor and a memory storing program instructions, characterized in that, The processor is configured to execute a three-dimensional whole-brain neuron mapping method as described in any one of claims 1 to 5 when running the program instructions.

8. An electronic device, characterized in that, Includes a three-dimensional whole-brain neuron mapping device as described in claim 6 or 7.

Images