Three-dimensional whole-brain neuron mapping method and device

Abstract

The invention relates to the technical field of image recognition, and discloses a three-dimensional whole-brain neuron mapping method and device, and the method comprises the steps: obtaining a three-dimensional brain image data set; processing the three-dimensional brain image data set to obtain a first image data set; classifying the first image data set to obtain k second image data sets; processing second image data in the k second image data sets to obtain k second label image data sets; processing the k second label image data sets to obtain a neuron cell body image set; processing coordinates of the neuron cell body image set to obtain a cell body coordinate set; and according to the cell body coordinate set and the whole-brain neuron coordinate framework, constructing a whole-brain neuron density distribution diagram to obtain the whole-brain neuron density distribution diagram. According to the method, the CNN, the video Swin Transform and other deep learning models are adopted, the CNN feature extraction capability and the Swin Transform global space information capturing capability are utilized, and large-scale and high-complexity neuron mapping of whole-brain imaging is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a three-dimensional whole-brain neuron mapping method and device. Background Art

[0002] The three-dimensional whole-brain neuron detection and mapping technology aims to accurately identify and segment neurons in the entire brain structure on a three-dimensional scale, and compare the detected neuron positions with existing brain maps to complete functional mapping.

[0003] The current mainstream methods in this field include: detection algorithms based on cell body thresholds, detection algorithms based on convolutional neural networks, detection algorithms based on morphological features, and registration algorithms based on Fourier transform and mutual information.

[0004] Among them, the detection algorithm based on convolutional neural networks uses deep learning technology and a large amount of labeled data for training. It can automatically identify and separate neurons in three-dimensional images and performs well in dealing with complex neuronal morphology and noise interference. However, its accuracy may encounter bottlenecks when faced with dense cell body scenes, and further innovation is needed to break through this limitation. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention proposes a three-dimensional whole-brain neuron mapping method and device to solve the above-mentioned technical problems.

[0006] In a first aspect, a three-dimensional whole-brain neuron mapping method is provided, comprising:

[0007] Acquire a three-dimensional brain image dataset;

[0008] Performing image enhancement processing on the three-dimensional brain image data set to obtain a first image data set;

[0009] Performing classification processing on the first image data set to obtain k second image data sets;

[0010] performing labeling processing on the second image data in the k second image data sets respectively according to a point clustering algorithm to obtain k second labeled image data sets;

[0011] Segmenting the second labeled image data in the k second labeled image data sets to obtain a neuron cell body image set;

[0012] Marking the coordinates of the neuron cell body image set to obtain a cell body coordinate set;

[0013] The whole-brain neuron density distribution map is constructed according to the cell body coordinate set and the whole-brain neuron coordinate framework to obtain the whole-brain neuron density distribution map.

[0014] Further, performing image enhancement processing on the three-dimensional brain image dataset to obtain a first image dataset includes:

[0015] splicing the three-dimensional image data in the three-dimensional brain image data set to obtain a three-dimensional whole-brain image;

[0016] Segmenting the three-dimensional whole-brain image to obtain k three-dimensional sub-block images;

[0017] performing contrast enhancement processing on the k three-dimensional sub-block images respectively to obtain k three-dimensional sub-block images after contrast adjustment;

[0018] Using median filtering and Gaussian filtering, respectively perform noise reduction processing on the k contrast-adjusted three-dimensional sub-block image sets to obtain k three-dimensional noise-reduced sub-block images;

[0019] The spatial resolutions of the k three-dimensional denoised sub-block images are adjusted respectively according to the interpolation technology to obtain a first image data set.

[0020] Furthermore, the first image data set is classified to obtain k second image data sets, including:

[0021] Converting k three-dimensional sub-block images in the first image data set to obtain k two-dimensional image data sets;

[0022] Dividing the k two-dimensional image data sets respectively according to a random partitioning algorithm to obtain k data sets;

[0023] According to the k data sets, a classification network model is constructed;

[0024] The classification network model is optimized according to the area enclosed by the coordinate axis under the ROC curve to obtain k second image data sets.

[0025] Furthermore, the second image data in the k second image data sets are respectively labeled according to the point clustering algorithm to obtain k second labeled image data sets, including:

[0026] Perform segmentation processing on the k pieces of the second image data respectively according to a supervoxel segmentation algorithm to obtain preliminary segmentation models of the k pieces of the second image data;

[0027] According to the label propagation algorithm, the k preliminary segmentation models of the second image data are respectively labeled to obtain k first-level pseudo labels;

[0028] Based on the similarity between supervoxels, k first-level pseudo labels are updated respectively to obtain k second-level pseudo labels;

[0029] According to the K-means clustering algorithm, k second-level pseudo labels are grouped to obtain k third-level pseudo labels;

[0030] The k third-level pseudo labels are trained respectively according to the weakly supervised training network framework to obtain k second-label image data sets.

[0031] Furthermore, the second labeled image data in the k second labeled image data sets are segmented to obtain a neuron cell body image set, including:

[0032] Performing preliminary feature extraction on the second labeled image data set to obtain k primary image features;

[0033] Performing deep feature extraction on the k primary image features respectively to obtain k high-level image features;

[0034] Performing global feature interaction on the k high-level image features respectively to obtain k feature images;

[0035] According to the feature fusion module, the k feature images are fused respectively to obtain k fused feature images;

[0036] The k fused feature images are transformed and processed respectively to obtain a neuron cell body image set. Further, the coordinates of the neuron cell body image set are marked to obtain a cell body coordinate set, including:

[0037] Performing connected domain analysis on the neuron cell body image set to obtain the centroids of k connected domains;

[0038] Calculating the coordinates of the k centroids of the connected domains respectively to obtain the coordinates of k neuron cell bodies;

[0039] The coordinates of the k neuron cell bodies are integrated to obtain a cell body coordinate set.

[0040] Furthermore, a whole-brain neuron density distribution map is constructed based on the cell body coordinate set and the whole-brain neuron coordinate framework to obtain a whole-brain neuron density distribution map, including:

[0041] Mapping the cell body coordinate set to a whole-brain neuron coordinate frame to obtain a mapped cell body coordinate set;

[0042] Registering the mapped cell body coordinate set according to an affine transformation algorithm to obtain a registered cell body coordinate set;

[0043] Dividing the brain regions according to the registered cell body coordinate sets to obtain neuron densities of k brain regions;

[0044] Calculating the neuron density of the k brain regions respectively to obtain the neuron density values ​​of the k brain regions;

[0045] The neuron density values ​​of the k brain regions are visualized to obtain a neuron density distribution map of the whole brain.

[0046] In a second aspect, a three-dimensional whole-brain neuron mapping device is provided, comprising:

[0047] An acquisition module, used for acquiring a three-dimensional brain image data set;

[0048] an enhancement processing module, configured to perform image enhancement processing on the three-dimensional brain image data set to obtain a first image data set;

[0049] A classification module, used to classify the first image data set to obtain k second image data sets;

[0050] a labeling processing module, used for labeling the second image data in the k second image data sets respectively according to a point clustering algorithm to obtain k second labeled image data sets;

[0051] A segmentation processing module, used for performing segmentation processing on the second label image data in the k second label image data sets to obtain a neuron cell body image set;

[0052] A marking processing module, used for marking the coordinates of the neuron cell body image set to obtain a cell body coordinate set;

[0053] The construction module is used to construct a whole-brain neuron density distribution map according to the cell body coordinate set and the whole-brain neuron coordinate framework to obtain the whole-brain neuron density distribution map.

[0054] Furthermore, a three-dimensional whole-brain neuron mapping device includes a processor and a memory storing program instructions, and the processor is configured to execute a three-dimensional whole-brain neuron mapping method as described in any of the above items when running the program instructions.

[0055] In a third aspect, an electronic device is provided, comprising the three-dimensional whole-brain neuron mapping device described above.

[0056] The invention adopting the above technical solution has the following advantages:

[0057] 1. The present invention adopts CNN, video Swin Transformer and other related deep learning models, and utilizes the advantages of CNN in feature extraction and the ability of Swin Transformer in capturing global spatial information to achieve large-scale, highly complex neuron mapping of whole-brain imaging. In addition, when processing complex three-dimensional brain imaging data, it can significantly improve the accuracy of cell body segmentation, especially in dense cell body areas, where the segmentation effect is more significant.

[0058] 2. The present invention effectively reduces the irrelevant data blocks that need to be processed and reduces the computational burden of the segmentation module. At the same time, the classification network is designed to be relatively lightweight, reducing the risk of overfitting, so that the model can maintain efficient operation in both the training and reasoning stages. This optimized computational process ensures rapid analysis at the whole-brain scale.

[0059] 3. The present invention greatly reduces the workload of data labeling required for the operation of the segmentation network through a weakly supervised network model, and greatly reduces the workload of training segmentation data using the present invention.

[0060] 4. The present invention can maintain stable detection performance when facing different imaging conditions (such as brightness fluctuations, uneven fluorescence intensity, etc.), can adapt to various complex environments, and improves the breadth of practical applications.

[0061] 5. The present invention realizes fully automated operations from data preprocessing, classification, segmentation to final coordinate mapping, reducing the need for manual intervention. Researchers only need a small amount of settings to complete the automated detection and analysis of neurons in the whole brain. This fully automated process significantly improves research efficiency and makes large-scale neuroscience research possible.

[0062] 6. The present invention provides an in-depth analysis of the distribution patterns of neurons in brain regions and the connection patterns of neural networks. This function enables researchers to better understand the complex functions of the brain from both macro and micro levels, and provides solid data support for neurological disease research, brain-computer interface development, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the specific implementation of the present invention, the following will briefly introduce the drawings required for use in the specific implementation. In all the drawings, each element or part is not necessarily drawn according to the actual scale.

[0064] Figure 1 This is a front view of a three-dimensional whole-brain neuron mapping method and device of the present invention;

[0065] FIG2 is a schematic diagram of whole-brain imaging data preprocessing in a three-dimensional whole-brain neuron mapping method of the present invention;

[0066] FIG3 is a schematic diagram showing the effect of a maximum projection algorithm on three-dimensional sub-block data in a three-dimensional whole-brain neuron mapping method of the present invention;

[0067] Figure 4 A schematic diagram of a classification network structure in a three-dimensional whole-brain neuron mapping method of the present invention;

[0068] Figure 5 A schematic diagram of a cell body segmentation network structure in a three-dimensional whole-brain neuron mapping method of the present invention;

[0069] FIG6 is a schematic diagram of a cell body segmentation effect in a three-dimensional whole-brain neuron mapping method of the present invention;

[0070] FIG7 is a schematic diagram of obtaining the cell body position by segmenting the result area in a three-dimensional whole-brain neuron mapping method of the present invention;

[0071] Figure 8 A schematic diagram of registering and fitting the cell body position with the whole-brain neuron coordinates in a three-dimensional whole-brain neuron mapping method of the present invention;

[0072] Fig. 9 A flowchart of a three-dimensional whole-brain neuron mapping method of the present invention;

[0073] Fig.10 This is a flow chart of a three-dimensional whole-brain neuron mapping device of the present invention. DETAILED DESCRIPTION

[0074] The following embodiments of the technical solution of the present invention are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and are therefore only used as examples, and cannot be used to limit the protection scope of the present invention.

[0075] like Figures 1 to 10 As shown, a three-dimensional whole-brain neuron mapping method of the present invention comprises:

[0076] Step S01, obtaining a three-dimensional brain image data set;

[0077] Step S02, performing image enhancement processing on the three-dimensional brain image data set to obtain a first image data set;

[0078] Step S03: classify the first image data set to obtain k second image data sets;

[0079] Step S04: labeling the second image data in the k second image data sets respectively according to the point clustering algorithm to obtain k second labeled image data sets;

[0080] Step S05, segmenting the second label image data in the k second label image data sets to obtain a neuron cell body image set;

[0081] Step S06, marking the coordinates of the neuron cell body image set to obtain a cell body coordinate set;

[0082] Step S07: construct a whole-brain neuron density distribution map based on the cell body coordinate set and the whole-brain neuron coordinate framework to obtain a whole-brain neuron density distribution map.

[0083] In some embodiments, performing image enhancement processing on the three-dimensional brain image dataset to obtain a first image dataset includes:

[0084] splicing the three-dimensional image data in the three-dimensional brain image data set to obtain a three-dimensional whole-brain image;

[0085] Segment the three-dimensional whole brain image to obtain k three-dimensional sub-block images;

[0086] Performing contrast enhancement processing on the k three-dimensional sub-block images respectively to obtain k three-dimensional sub-block images after contrast adjustment;

[0087] Using median filtering and Gaussian filtering, denoising is performed on the k contrast-adjusted three-dimensional sub-block image sets to obtain k three-dimensional denoised sub-block images;

[0088] The spatial resolutions of the k three-dimensional denoised sub-block images are adjusted respectively according to the interpolation technology to obtain a first image data set.

[0089] Specifically, the present invention uses Python code to automatically preprocess whole-brain image data, improve data quality and maintain consistency in their format, size, pixels and other information, reduce the possibility of errors during network input, and further improve network performance during training.

[0090] The preprocessing steps mainly include: 1. Image stitching. In the image stitching step, a feature point-based stitching algorithm is used. This method first identifies significant feature points in each image segment through a key point detection algorithm. These feature points are corners or edges in the image and are the most recognizable parts of the image. After identifying the key points, a matching algorithm is used to pair the corresponding feature points in different image segments to determine the overlapping areas between the images. After the matching is completed, a random sampling consistency algorithm is used to eliminate false matches and calculate the transformation matrix so that the image segments can be accurately aligned. Finally, image fusion technology is used to seamlessly stitch these aligned image segments together to generate a complete three-dimensional brain image.

[0091] 2. Sub-block division. The complete three-dimensional brain image far exceeds the upper limit of what the network can accept for learning. Therefore, it is necessary to divide the complete brain image into sub-blocks in the preprocessing step, and use an algorithm to automatically divide it into sub-blocks of specified sizes in the order of the X-axis, Y-axis, and Z-axis numbers. The numbers facilitate the statistics of sub-block information.

[0092] 3. Brightness adjustment. The histogram equalization method is used. This method redistributes the pixel values ​​of the image to make it more uniform within the brightness range. First, the grayscale histogram of the image is calculated to count the number of pixels at each grayscale level. Then, the cumulative distribution function is used to map the original pixel values ​​to the new grayscale level, thereby enhancing the contrast of the image.

[0093] 4. Noise removal. A combination of median filtering and Gaussian filtering is used to remove noise. Median filtering is suitable for removing salt and pepper noise in images. It replaces each pixel value in the image with the median value of the pixel value in the neighborhood of the pixel, thereby effectively eliminating isolated noise points without blurring the edges; Gaussian filtering is suitable for removing Gaussian noise by performing a convolution operation with the image through a convolution kernel to achieve image smoothing. Combining the two can effectively remove various types of noise while retaining image details.

[0094] 5. Data normalization. In the process of spatial resolution normalization, cubic interpolation technology is used to adjust the resolution of the image. This method uses the values ​​of 16 neighboring pixels to estimate the new pixel value, thereby scaling the image while maintaining image details. The image after the preprocessing algorithm is shown in Figure 2, where Figure 2a shows the image after the whole brain is spliced, and Figure 2b shows the original image divided into sub-blocks. Figure 2c Represents an image after brightness adjustment, noise removal, and data normalization.

[0095] In some embodiments, the first image data set is classified to obtain k second image data sets, including:

[0096] Converting k three-dimensional sub-block images in the first image data set to obtain k two-dimensional image data sets;

[0097] According to the random partitioning algorithm, k two-dimensional image data sets are divided respectively to obtain k data sets;

[0098] Based on k data sets, a classification network model is constructed;

[0099] The classification network model is optimized according to the area enclosed by the coordinate axis under the ROC curve to obtain k second image data sets.

[0100] Specifically, after obtaining high-quality sub-block brain image data, it was found that the existing network reading and learning the features in these high-dimensional data sub-blocks still occupies a large amount of video memory, greatly increasing the training time and server space. In order to achieve fast training and testing of the network, the high-dimensional original data sub-blocks were reduced in dimension through the convolution algorithm to obtain low-dimensional data that can be input into the network.

[0101] The classification network based on CNN can distinguish the network containing neuron cell bodies from the network without neuron cell bodies, screen out the neuron data containing cell bodies and input them into the network for training and learning, and do not input the data without neuron cell bodies into the network, which reduces the computational complexity of the segmentation network. The steps of using the network are as follows: 1. Use the maximum projection method to quickly process the three-dimensional sub-block data into two-dimensional image data in batches to obtain two-dimensional image data that can be input into the classification network; use the maximum intensity projection (MIP) method to quickly convert the three-dimensional sub-block data into two-dimensional image data. The MIP method projects along the Z axis of the three-dimensional image data, and uses the maximum intensity value of each image on the Z axis as the pixel value of the two-dimensional image, which means that the neurons on each image on the Z axis dimension are retained in one image. This method retains the most significant features of the three-dimensional data and is suitable for neuron cell body detection. Batch processing technology further accelerates this process, allowing multiple sub-blocks to be converted simultaneously, significantly improving the processing efficiency. Image processed by the MIP algorithm. As shown in Figure 3, Figure 3a represents a dense neuron cell body image, Figure 3b represents a sparse neuron cell body image, and Figure 3c represents an image with impurity interference. Figure 3d Represents a background image without neurons.

[0102] 2. Use the fast random partitioning algorithm to partition the training set, validation set, and test set required by the classification network, and obtain three randomly partitioned data sets for the classification network; in order to ensure the fairness and effectiveness of the training, validation, and testing processes of the classification network, the fast random partitioning algorithm is used to divide the data set into training set, validation set, and test set. The algorithm randomly selects data samples and distributes them to different subsets according to a predefined ratio (for example, 4:1:1). In this way, each subset can be guaranteed to be representative, thereby avoiding overfitting or underfitting during the training and testing of the classification network.

[0103] 3. Input the data set into a lightweight classification network backbone to obtain image features; the two-dimensional image data of the divided data set is input into a lightweight shallow classification network backbone. The core feature is the introduction of shortcut connections, which allows the network to learn image features more effectively. The shallow structure further reduces the complexity and computational complexity of the network and is suitable for processing large-scale whole-brain imaging data. Network structure, such as Figure 4 As shown in the figure, the blocks and arrows represent different processing layers and connection methods. Blocks ① and ② represent convolutional layers with different step sizes, and their convolution kernel sizes are both 3×3. The number of downsampling steps of convolutional layer ② is 2. Arrow ③ represents the stacking of average pooling, fully connected layers, and Softmax functions. The dotted arrow ④ indicates the direction of data flow, while the dotted arrows ⑤ and ⑥ represent shortcut connections with or without 1×1 convolutional layers, respectively.

[0104] 4. Use the fully connected layer to process the image features, increase the network processing dimension, improve the performance of the classification network in this task, and obtain the classification network model; the extracted image features are then input into the fully connected layer for processing. The function of the fully connected layer is to further linearly combine the high-dimensional features extracted by the previous convolutional layer to increase the network's expressive power. In this step, the linear layer is used to increase the processing dimension, and the ReLU function activation function is used to increase the nonlinear expression ability of the network, thereby improving the overall performance of the classification network in this task. Finally, the output classification result is used to determine whether each input image contains a neuron cell body.

[0105] 5. During the training process, the algorithm automatically selects the model with the highest AUC value and saves it to obtain the optimal model in a round of training; during the training process of the classification network, AUC is used as the key indicator for model evaluation. The higher the AUC value, the better the classification performance of the model. During the training process, the algorithm automatically calculates the AUC value after each training cycle and saves the model with the highest AUC value. This strategy ensures that the best model can be saved in each round of training results for subsequent testing and practical applications.

[0106] 6. Use the best performing model after ten rounds of training to classify the 2D projection images of the whole brain data, and obtain the 2D projection images of the whole brain data containing neuronal cell bodies; after completing multiple rounds of training, the algorithm will select the best performing model to classify the whole brain data. Ten rounds of training were conducted in this test, and the model with the highest AUC was selected in these ten rounds. Use this best model to classify all 2D projection images, and identify and mark the images containing neuronal cell bodies.

[0107] 7. Refer to the names of the two-dimensional projection images corresponding to the results of the classification network to obtain the three-dimensional data sub-blocks containing neuronal cell bodies in the whole brain data; the results of the classification network output are used to identify which two-dimensional projection images contain neuronal cell bodies. By matching the labels of these two-dimensional images with the original three-dimensional data sub-blocks, it can be determined which three-dimensional sub-blocks contain neuronal cell bodies. This step directly maps the output of the classification network back to the original data, generating an efficient and accurate three-dimensional neuronal cell body positioning result.

[0108] The classification network algorithm of the present invention and the existing classification network are used to test ten random data sets. Table 1 shows the test results of classifying images after MIP projection of sub-blocks of size 128×128×128. The test performance of the classification network includes AUC (the average area under the ROC curve of ten tests), ACC (the average accuracy of ten tests), parameter quantity, FLOPs (floating point calculation quantity) and FPS (the average number of image frames displayed per second in ten tests). The tested classification networks include: Densenet121 (121-layer dense connection network: Dense Convolutional Network-121), Resnet18 (18-layer residual network: Residual Network-18), Vgg16 (16-layer visual geometry group network: Visual GeometryGroup-16). The test process is: 180 MIP images from four different mice are selected to form the initial data set. The positive samples in the data set are manually selected to cover various situations of different human sub-blocks, such as different densities, brightness levels and distribution positions. Negative samples are randomly selected from the background image. The initial dataset was randomly divided into three sub-datasets: training dataset (120 images), validation dataset (30 images), and test dataset (30 images). The initial dataset randomly repeated data allocation 10 times to verify the stability of the classification network. Each of the four methods was trained for 100 epochs. The model with the highest AUC in each round of training was selected for testing. After the test algorithm loaded the model, it read the corresponding test dataset to test each indicator, and finally obtained the results in the table.

[0109] Table 1 Classification network test results

[0110]

[0111] In some embodiments, the second image data in the k second image data sets are respectively annotated according to a point clustering algorithm to obtain k second labeled image data sets, including:

[0112] Perform segmentation processing on the k second image data respectively according to the supervoxel segmentation algorithm to obtain preliminary segmentation models of the k second image data;

[0113] According to the label propagation algorithm, the preliminary segmentation models of the k second image data are respectively labeled to obtain k first-level pseudo labels;

[0114] Through the similarity between supervoxels, the k first-level pseudo labels are updated respectively to obtain k second-level pseudo labels;

[0115] According to the K-means clustering algorithm, k second-level pseudo labels are grouped and processed to obtain k third-level pseudo labels;

[0116] According to the weakly supervised training network framework, k third-level pseudo labels are trained respectively to obtain k second-label image data sets.

[0117] Specifically, a weakly supervised three-dimensional data network based on a point clustering algorithm is used to reduce the amount of manually annotated data required for three-dimensional whole-brain segmentation training and testing. The steps of using this method are as follows:

[0118] 1. Use the supervoxel segmentation algorithm based on the geometric similarity of three-dimensional data to obtain a preliminary data segmentation model and reduce the amount of subsequent processing calculations; the supervoxel segmentation algorithm is used, which performs preliminary data segmentation based on the geometric similarity of three-dimensional data. First, the input three-dimensional data is preprocessed to extract feature points and faces. Using the geometric similarity measurement method, points are assigned to different supervoxels according to their spatial position, normal direction, and surface features. Points with high geometric similarity are aggregated into a supervoxel to generate a preliminary segmentation model, reducing the complexity of subsequent calculations.

[0119] 2. Use the label propagation algorithm based on the supervoxel segmentation model to obtain the initial pseudo-label;

[0120] 3. A graph model is constructed based on three-dimensional geometric supervoxels. Each supervoxel is regarded as a node in the graph. The edge between nodes represents the similarity between two supervoxels. The predicted probability distribution of the supervoxel is obtained by calculating all the points in each supervoxel:

[0121]

[0122] Among them, y i represents the initial pseudo label of the i-th superprime point, p i represents the coordinates of each supervoxel point, c i represents the color of each supervoxel, is the corresponding point p i The ground truth category of , Θ represents the output of the backbone network.

[0123] 4. Use the relational network to learn the similarity between supervoxels, and use Transformer to iteratively update pseudo labels. Obtain the similarity measure between supervoxels of the relational network to optimize label propagation; after obtaining the supervoxel prediction probability distribution, use the relational network and the following formula to optimize the label propagation process:

[0124]

[0125] Among them, R is the output of the relational network, ψ p (j k ) represents the supervoxel v j and v′ jThe similarity between u (y j |V,Θ) represents the supervoxel v j With label y j The negative log probability of . It can be expressed as:

[0126] ψ u (y j |V,Θ)=-log P(y j |V,Θ)

[0127] Both low-level features and learned features are used to measure similarity. The formula is as follows:

[0128]

[0129] Among them, Ι(y j ,y j′ ) means if y j ,y j′ This part is 1 if there are different predicted labels, otherwise it is 0. These two terms indicate that if supervoxels with similar features are predicted as different classes, the cost will be increased. j ,c j′ ,p j ,p j′ ,u j ,u j′ Denotes the corresponding supervoxel v j The normalized average color, average coordinates, and average features of .

[0130] In addition, the Transformer network is used to learn the similarity between supervoxels. The network predicts the relationship between them by analyzing the characteristics of supervoxels. Based on the similarity measurement output by the relationship network, the Transformer framework is used to iteratively update the pseudo-labels to gradually improve the quality of the pseudo-labels. The specific formula is as follows:

[0131]

[0132] Among them, Q\K\V represent the MLP (Multilayer Perceptron) layer in the Transformer network, F j Represents the feature vector. The Transformer network calculates F j With the category prototype k c The similarity between them is used to aggregate the category prototype k c , the obtained features With F j The prediction results of supervoxel semantic categories are concatenated for iterative updating.

[0133] 5. Use the K-means clustering algorithm to further group the supervoxels to obtain instance segmentation-level pseudo labels; perform K-means clustering on the supervoxels to classify similar supervoxels into the same instance. The specific formula is as follows:

[0134]

[0135] This formula achieves clustering by minimizing K-means, dividing n supervoxels into Divide into K clusters is the centroid of the kth cluster, so that the supervoxels within each cluster have the maximum similarity, while the differences between different clusters are the largest. Each clustering result corresponds to an instance label, and a more fine-grained instance segmentation pseudo label is obtained. The label can be expressed as:

[0136] y i = arg min k ||f i -μ k ||

[0137] Among them, f i represents each supervoxel, y i Indicates the corresponding pseudo label.

[0138] 6. Use the weakly supervised training network framework to iteratively perform label propagation and network training to obtain a weakly supervised network model. Use the generated pseudo-labels to iteratively train the network, and gradually improve the performance of the model by continuously optimizing the pseudo-labels and network weights. Train for a total of 200 epochs, and use the last model as the base model of the weakly supervised network for subsequent segmentation networks.

[0139] In some embodiments, segmenting the second labeled image data in the k second labeled image data sets to obtain a neuron cell body image set includes:

[0140] Perform preliminary feature extraction on the second labeled image data set to obtain k primary image features;

[0141] Perform deep feature extraction on the k primary image features respectively to obtain k high-level image features;

[0142] Perform global feature interactions on k high-level image features respectively to obtain k feature images;

[0143] According to the feature fusion module, k feature images are fused respectively to obtain k fused feature images;

[0144] The k fused feature images are transformed respectively to obtain a neuron cell body image set.

[0145] Specifically, a segmentation network based on the video Swin Transformer is used to separate the neuron cell body region from the rest of the image;

[0146] 1. Preliminary feature extraction of 3D whole-brain data sub-blocks to obtain primary features of the image to be segmented; the input 3D sub-blocks first pass through a convolutional layer, which uses multiple convolution kernels to operate on the input data to extract primary spatial features. The output of the convolutional layer is processed by a BN layer (Batch Normalization) and a nonlinear activation function (such as ReLU) to enhance the training effect of the network and accelerate convergence.

[0147] 2. Combine pseudo labels and instance data labels to extract deep features through multi-layer convolution encoders to obtain high-level image features and local image features; after obtaining preliminary features, combine pseudo labels and instance data labels to extract deep features of the image through multi-layer convolution encoders. The encoder module consists of multiple convolution layers, pooling layers, and skip connections. The segmentation network structure is shown in Figure 5 As shown in the figure, cubes and arrows with different numbers represent different processing layers in the segmentation network. Cube ① represents the convolutional encoder layer, and cube ② represents the convolutional decoder layer; arrow ③ represents the downsampling layer, and arrow ④ represents the upsampling layer, both of which are convolutional layers with a stride of 2; arrow ⑤ represents a skip connection; the Transformer layer consists of L video Swin Transformer modules, LN represents the normalization layer, W-MSA represents the window multi-head self-attention mechanism module, and SW-MSA represents the shift window multi-head self-attention mechanism module; represents a residual connection.

[0148] 3. Use the global feature interaction based on the video Swin Transformer to allow the network model to capture the global features of the image and obtain a better feature network; Swin Transformer performs global feature interaction on the extracted features through W-MSA and SW-MSA. These self-attention mechanisms enable the network to capture the global features of the image and learn deeper semantic information. The self-attention mechanism is the core of the Transformer network. For each position in the input feature map, its correlation with other positions is calculated. This correlation is calculated by the following formula:

[0149] Q=XW Q

[0150] K=XW K

[0151] V=XW V

[0152] Among them, Q is the query matrix, K is the key matrix, V is the value matrix, X is the input feature map, and W Q \W K \W V are the corresponding linear transformation matrices respectively. The calculation formula of attention weight is as follows:

[0153]

[0154] Among them, d k is the dimension of the key vector used to scale the dot product result. Then the attention weights are used to weight the value matrix Z = A·V

[0155] W-MSA and SW-MSA are implemented based on the above self-attention mechanism principle. Swin Transformer divides the feature map into multiple non-overlapping windows, and calculates the self-attention score independently in each window:

[0156]

[0157] 4. Use the feature fusion module of upsampling and skip connection to restore high resolution from low resolution and obtain important spatial information and details; after obtaining the global features, the network needs to restore these features to the original resolution in order to generate fine segmentation results. This step is achieved through the feature fusion module of upsampling and skip connection. The upsampling operation gradually restores the feature map to high resolution, while the skip connection combines the features of the encoder layer with the features of the decoder layer to ensure that important spatial information and details are preserved.

[0158] 5. Use the convolution layer to generate the final segmentation result. After all feature extraction and fusion are completed, a convolution layer is used to convert the feature map into the final segmentation result. The convolution layer uses a 1×1 convolution kernel to map the high-dimensional features to the target category space, generate the category probability distribution of each pixel or voxel, and then determine the final segmentation result. The schematic diagram of the final segmentation result is shown in Figure 6.

[0159] The segmentation network of the present invention and the existing segmentation network are used to test a specific data set. Table 2 shows the test results of segmenting a 3D data sub-block of 512×512×512 size. The test indicators of the segmentation network include DSC (DiceSimilarity Coefficient), Precision, Recall and F1 score. The calculation formulas of the above indicators are as follows:

[0160]

[0161] Among them, S g represents the result of automatic segmentation using the segmentation network, St Indicates the results of manual labeling.

[0162]

[0163] Among them, N GT Indicates the number of GT (Ground Truth, the actual label) neurons, N TP represents the number of true positive neurons, N detected Represents the number of neurons detected.

[0164] The tested segmentation networks include: SegNet (Image Segmentation Network), 3DU-Net (3D U-Net), Res-U-Net (Residual U-Net), TransBTS (Transformer in 3D CNN for 3D Brain Tumor Segmentation). The testing process is as follows: the dataset is divided into 3 sub-datasets: training dataset (22 sub-blocks), validation dataset (6 sub-blocks) and test dataset (10 sub-blocks). Dynamic random flipping and random center cropping are used to enhance the training data. The segmentation network is trained using the Adam optimizer with an initial learning rate of 0.0001. The learning rate decay strategy is adopted, and the learning rate decays by 10% every 10 epochs. The best model obtained by the early stopping strategy contains 80 iterations per epoch. The minimum batch size is set to 1 during the segmentation network training stage. Dice loss is used for training network tasks, and the model with the highest DSC in each segmentation network training process is selected as the test model. During the testing phase, the optimal models are loaded and the test data sets are read, and finally the results in the table are obtained.

[0165] Table 2 Segmentation network test results

[0166]

[0167] In some embodiments, the coordinate marking process of the neuron cell body image set is performed to obtain the cell body coordinate set, including:

[0168] Perform connected domain analysis on the neuron cell body image set to obtain the centroids of k connected domains;

[0169] Calculate the coordinates of the centroids of k connected domains respectively to obtain the coordinates of k neuron cell bodies;

[0170] The coordinates of the k neuron cell bodies are integrated to obtain a cell body coordinate set.

[0171] Specifically, based on the connected domain analysis, an algorithm is used to calculate the cell body centroid using the segmentation result area to determine the cell body position; this algorithm receives the output result of the segmentation network, and the segmentation result is a three-dimensional binary image in which each voxel is marked as belonging to the cell body or not. The segmentation result is processed using the connected domain analysis method. Connected domain analysis will classify adjacent voxels marked as cell bodies into the same region (i.e., connected domain). In order to reduce noise and false detection, we use an algorithm to remove very small connected domains and delete connected domains with an area or volume less than a specific threshold, so that only sufficiently large connected domains are retained. These large connected domains are more likely to be real cell bodies. For each remaining connected region, its centroid is calculated, the centroid (x c ,y c ,z c ) is calculated as:

[0172]

[0173] Where |R| is the number of voxels in the connected region, and (x, y, z) are the coordinates of each voxel in the region.

[0174] The calculated centroid coordinates will be saved in a specific format (such as a .swc file) for subsequent analysis and processing. The schematic diagram of the sub-block centroid after algorithm processing is shown in Figure 7, which shows the centroid coordinates of the neuron cell body of the three-dimensional mouse whole brain data demonstrated in this example.

[0175] In some embodiments, constructing a whole-brain neuron density distribution map based on the cell body coordinate set and the whole-brain neuron coordinate framework to obtain the whole-brain neuron density distribution map includes:

[0176] Mapping the cell body coordinate set to the whole-brain neuron coordinate frame to obtain a mapped cell body coordinate set;

[0177] The mapped cell body coordinate set is registered according to an affine transformation algorithm to obtain a registered cell body coordinate set;

[0178] The brain regions are divided according to the registered cell body coordinate sets to obtain the neuron density of k brain regions;

[0179] The neuron density of k brain regions is calculated respectively to obtain the neuron density value of k brain regions;

[0180] The neuron density values ​​of k brain regions are visualized to obtain the neuron density distribution map of the whole brain.

[0181] Specifically, a fast fitting algorithm is used to align and map the measured cell bodies with the existing whole-brain neuron coordinate frame; the obtained center of mass coordinates are used to map them to the whole-brain neuron coordinate frame through a scaling factor. It is usually necessary to perform a scaling calculation based on the resolution difference between the actual image and the coordinate system to accurately map the center of mass coordinates to the standard coordinate system. The neuron automatic technology algorithm in each brain region obtains the neuron density and connection strength between different brain regions; for different brain regions or the entire brain area, an affine transformation algorithm is used to align the detected cell body positions with the existing standard coordinate system. This is achieved through the alignment method in SimpleITK; based on the mapped coordinate information, each center of mass is assigned to the corresponding brain region, and the center of mass within a certain brain region is counted into the neuron count of that brain region. The schematic diagram of the whole-brain neuron cell body brain region mapping is shown in the figure. Figure 8 As shown in the figure, different colors represent different brain regions. After obtaining the total number of neurons in each brain region, the neuron density of the brain region is calculated. The calculation of neuron density is usually expressed as the number of neurons per unit volume, which can be further standardized as the number of neurons per unit volume of the brain region. Finally, a statistical method based on neuron distribution is used to analyze the strength of neuron connections between different brain regions.

[0182] Neuron distribution density formula:

[0183]

[0184] in, is the neuron density of the brain region, N cells is the total number of neurons in the brain region, V region is the volume of the brain region.

[0185] Connection strength formula:

[0186]

[0187] Where σ is the connection strength, d(P i ,P j ) is the center of mass P i and P j The Euclidean distance between .

[0188] In some other embodiments, a three-dimensional whole-brain neuron mapping device is provided, comprising:

[0189] An acquisition module, used for acquiring a three-dimensional brain image data set;

[0190] An enhancement processing module, used for performing image enhancement processing on the three-dimensional brain image data set to obtain a first image data set;

[0191] A classification module, used to classify the first image data set to obtain k second image data sets;

[0192] A labeling processing module, used for labeling the second image data in the k second image data sets according to the point clustering algorithm, to obtain k second labeled image data sets;

[0193] A segmentation processing module, used for performing segmentation processing on the second label image data in the k second label image data sets to obtain a neuron cell body image set;

[0194] A marking processing module is used for marking the coordinates of the neuron cell body image set to obtain a cell body coordinate set;

[0195] The construction module is used to construct a whole-brain neuron density distribution map according to the cell body coordinate set and the whole-brain neuron coordinate framework to obtain the whole-brain neuron density distribution map.

[0196] In some embodiments, a three-dimensional whole-brain neuron mapping device includes a processor and a memory storing program instructions, and the processor is configured to execute a three-dimensional whole-brain neuron mapping method as described in any of the above when running the program instructions.

[0197] In other embodiments, an electronic device (such as a mobile phone, a computer) is provided, comprising a three-dimensional whole-brain neuron mapping device as described above.

[0198] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

Claims

1. A three-dimensional whole-brain neuron mapping method, characterized in that: include: Acquire a three-dimensional brain image dataset; Performing image enhancement processing on the three-dimensional brain image data set to obtain a first image data set; Performing classification processing on the first image data set to obtain k second image data sets; performing labeling processing on the second image data in the k second image data sets respectively according to a point clustering algorithm to obtain k second labeled image data sets; Segmenting the second labeled image data in the k second labeled image data sets to obtain a neuron cell body image set; Marking the coordinates of the neuron cell body image set to obtain a cell body coordinate set; The whole-brain neuron density distribution map is constructed according to the cell body coordinate set and the whole-brain neuron coordinate framework to obtain the whole-brain neuron density distribution map.

2. A three-dimensional whole-brain neuron mapping method according to claim 1, characterized in that: Performing image enhancement processing on the three-dimensional brain image dataset to obtain a first image dataset includes: splicing the three-dimensional image data in the three-dimensional brain image data set to obtain a three-dimensional whole-brain image; Segmenting the three-dimensional whole-brain image to obtain k three-dimensional sub-block images; performing contrast enhancement processing on the k three-dimensional sub-block images respectively to obtain k three-dimensional sub-block images after contrast adjustment; Using median filtering and Gaussian filtering, respectively perform noise reduction processing on the k contrast-adjusted three-dimensional sub-block image sets to obtain k three-dimensional noise-reduced sub-block images; The spatial resolutions of the k three-dimensional denoised sub-block images are adjusted respectively according to the interpolation technology to obtain a first image data set.

3. A three-dimensional whole-brain neuron mapping method according to claim 2, characterized in that: The first image data set is classified to obtain k second image data sets, including: Converting k three-dimensional sub-block images in the first image data set to obtain k two-dimensional image data sets; Dividing the k two-dimensional image data sets respectively according to a random partitioning algorithm to obtain k data sets; According to the k data sets, a classification network model is constructed; The classification network model is optimized according to the area enclosed by the coordinate axis under the ROC curve to obtain k second image data sets.

4. A three-dimensional whole-brain neuron mapping method according to claim 1, characterized in that: The second image data in the k second image data sets are respectively labeled according to the point clustering algorithm to obtain k second labeled image data sets, including: Perform segmentation processing on the k pieces of the second image data respectively according to a supervoxel segmentation algorithm to obtain preliminary segmentation models of the k pieces of the second image data; According to the label propagation algorithm, the k preliminary segmentation models of the second image data are respectively labeled to obtain k first-level pseudo labels; Based on the similarity between supervoxels, k first-level pseudo labels are updated respectively to obtain k second-level pseudo labels; According to the K-means clustering algorithm, k second-level pseudo labels are grouped to obtain k third-level pseudo labels; The k third-level pseudo labels are trained respectively according to the weakly supervised training network framework to obtain k second-label image data sets.

5. A three-dimensional whole-brain neuron mapping method according to claim 1, characterized in that: The second labeled image data in the k second labeled image data sets are segmented to obtain a neuron cell body image set, including: Performing preliminary feature extraction on the second labeled image data set to obtain k primary image features; Performing deep feature extraction on the k primary image features respectively to obtain k high-level image features; Performing global feature interaction on the k high-level image features respectively to obtain k feature images; According to the feature fusion module, the k feature images are fused respectively to obtain k fused feature images; The k fused feature images are transformed respectively to obtain a neuron cell body image set.

6. A three-dimensional whole-brain neuron mapping method according to claim 1, characterized in that: The coordinates of the neuron cell body image set are marked to obtain a cell body coordinate set, including: Performing connected domain analysis on the neuron cell body image set to obtain the centroids of k connected domains; Calculating the coordinates of the k centroids of the connected domains respectively to obtain the coordinates of k neuron cell bodies; The coordinates of the k neuron cell bodies are integrated to obtain a cell body coordinate set.

7. A three-dimensional whole-brain neuron mapping method according to claim 1, characterized in that: The whole-brain neuron density distribution map is constructed according to the cell body coordinate set and the whole-brain neuron coordinate framework, and the whole-brain neuron density distribution map is obtained, including: Mapping the cell body coordinate set to a whole-brain neuron coordinate frame to obtain a mapped cell body coordinate set; Registering the mapped cell body coordinate set according to an affine transformation algorithm to obtain a registered cell body coordinate set; Dividing the brain regions according to the registered cell body coordinate sets to obtain neuron densities of k brain regions; Calculating the neuron density of the k brain regions respectively to obtain the neuron density values ​​of the k brain regions; The neuron density values ​​of the k brain regions are visualized to obtain a neuron density distribution map of the whole brain.

8. A three-dimensional whole-brain neuron mapping device, characterized in that: include: An acquisition module, used for acquiring a three-dimensional brain image data set; an enhancement processing module, configured to perform image enhancement processing on the three-dimensional brain image data set to obtain a first image data set; A classification module, used to classify the first image data set to obtain k second image data sets; a labeling processing module, used for labeling the second image data in the k second image data sets respectively according to a point clustering algorithm to obtain k second labeled image data sets; A segmentation processing module, used for performing segmentation processing on the second label image data in the k second label image data sets to obtain a neuron cell body image set; A marking processing module, used for marking the coordinates of the neuron cell body image set to obtain a cell body coordinate set; The construction module is used to construct a whole-brain neuron density distribution map according to the cell body coordinate set and the whole-brain neuron coordinate framework to obtain the whole-brain neuron density distribution map.

9. A three-dimensional whole-brain neuron mapping device, comprising a processor and a memory storing program instructions, characterized in that: The processor is configured to execute a three-dimensional whole-brain neuron mapping method as described in any one of claims 1 to 7 when running the program instructions.

10. An electronic device, characterized in that: It comprises a three-dimensional whole-brain neuron mapping device as described in claim 8 or 9.

Images