Neuron morphology recognition method based on task specific representation learning

Abstract

The invention relates to the technical field of computer vision, and discloses a neuron morphology recognition method based on task specific representation learning, and the implementation process of a learning model comprises the following steps: firstly projecting 3D neuron data into three 2D view images, and then taking the 2D view images as the input of different tasks; the method comprises the following steps of: firstly extracting the same task common features for different tasks from a 2D view image through a task-shared feature extractor; enhancing the task specific features by using a morphological perception regional feature enhancement module to obtain enhanced task specific features; and adaptively fusing the task common features and the enhanced task specific features together by using a fusion module guided by a global category, inputting the obtained feature descriptors into a linear classifier, and predicting to obtain the neuron category of the 3D neuron data. According to the method, morphological characterization of the cone cells is accurately extracted in a task-related mode, and a plurality of morphological analysis tasks are carried out at the same time.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a neuron morphology recognition method based on task-specific representation learning. Background Art

[0002] Pyramidal cells are the main type of excitatory neurons in the cerebral cortex, accounting for about two-thirds of all neurons in the mammalian cortex. These cells are distributed in various brain regions of mammals, including the cerebral cortex, hippocampus, and amygdala. Therefore, they are associated with a variety of complex cognitive activities of the brain. Analyzing and understanding the morphology of pyramidal cells can help to better understand the cognitive activities and functions of the brain. Most current methods describe the morphology of pyramidal cells in a task-independent manner, resulting in an inaccurate description of the morphology of pyramidal cells in a variety of analysis tasks, thereby reducing the performance of the method. Summary of the invention

[0003] To solve the above technical problems, the present invention provides a neuron morphology recognition method based on task-specific representation learning. In order to extract task-related features and improve the performance of multiple morphological analysis tasks, the present invention designs a learning model based on task-related features. In view of the sparsity of neuron data itself and the high consumption of directly training 3D data, the present invention adopts a multi-view projection method to project 3D pyramidal cells to a 2D plane, obtain multiple 2D images and connect these images along the channel direction. In addition, in order to extract discriminative morphological representations, the present invention introduces a feature enhancement module based on morphological perception areas to capture specific representations of different tasks. Finally, in order to generate a unique feature descriptor, the present invention uses a global category-guided fusion module to adaptively fuse the common features of the task and the task-specific features. In this way, the morphological features obtained by the present invention can describe the structure of pyramidal cells to the greatest extent and improve the performance of each recognition task.

[0004] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0005] A neuron morphology recognition method based on task-specific representation learning, wherein neuron data is input into a learning model and the category to which the neuron data belongs is output; the implementation process of the learning model includes the following steps:

[0006] Step 1: Data preprocessing: The 3D neuron data is first projected into three 2D view images, and then the 2D view images are used as input for different tasks;

[0007] Step 2, feature extraction: The 2D view image is firstly passed through a task-shared feature extractor to extract the same task-common features for different tasks; task-specific features are extracted for each task, and the task-specific features are enhanced using a morphology-aware regional feature enhancement module to obtain enhanced task-specific features;

[0008] Step 3: Feature fusion: Use the global category-guided fusion module to adaptively fuse the common features of the task and the enhanced task-specific features to form an instance-level feature descriptor.

[0009] Step 4: Classification: Input the feature descriptor into a linear classifier to predict the neuron category of the 3D neuron data.

[0010] Furthermore, step one specifically includes: orthogonally projecting the 3D neuron data onto three planes, and connecting the three acquired 2D view images together along the channels as inputs for different tasks.

[0011] Furthermore, in step 2, extracting task-specific features for each task and enhancing the task-specific features using a morphologically aware regional feature enhancement module to obtain enhanced task-specific features specifically includes:

[0012] Use the feature conversion layer to obtain task-specific features; the feature conversion layer consists of two convolutional layers and a batch normalization layer;

[0013] The morphologically aware regional feature enhancement module divides the task-specific features into 2n sub-feature maps along the channel, and further divides the 2n sub-feature maps into three groups. Then, an attention map generation module is used to calculate the attention map, which specifically includes: the attention map generation module first uses a maximum pooling layer and an average pooling layer to obtain global features, connects them along the channel, and then passes through a fully connected layer and a softmax activation function layer to obtain the attention map;

[0014] The attention map is assigned as weights to the task-specific features to obtain enhanced task-specific features.

[0015] Furthermore, in step 3, the fusion module using global category guidance adaptively fuses the common features of the tasks and the enhanced task-specific features to form an instance-level feature descriptor, which specifically includes:

[0016] First, global features are extracted by using average pooling operations on enhanced task-specific features and task-common features respectively;

[0017] The following operations are performed on the global features that enhance the task-specific features and the global features that enhance the task-common features:

[0018] The global features are passed through a classifier and a softmax activation function layer to generate a prediction output; the classifier contains two fully connected layers; the generated prediction output is used to obtain the weight of each sample, the weight is assigned to the corresponding sample, and the weight is added to the global features after batch normalization to obtain a descriptor;

[0019] The descriptors corresponding to the enhanced task-specific features and the descriptors corresponding to the task-common features are connected to obtain the instance-level feature descriptors.

[0020] Compared with the prior art, the beneficial technical effects of the present invention are:

[0021] 1. The present invention accurately extracts the morphological representation of pyramidal cells in a task-related manner and performs multiple morphological analysis tasks simultaneously.

[0022] 2. We introduce a morphology-aware regional feature enhancement module to capture task-specific salient features from morphology-aware regions, and a global category-guided fusion module to learn the importance of task-common and task-specific features and adaptively fuse them into a more discriminative descriptor.

[0023] 3. The present invention achieves good analytical performance in identifying the brain layer and species to which the neuron belongs. It is noteworthy that the classification accuracy of the present invention reaches 90.34% when identifying the species of pyramidal cells. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 Schematic diagram of pyramidal cells in different species and brain regions;

[0025] Figure 2 It is a schematic diagram of a framework for implementing the method of the present invention;

[0026] Figure 3 A schematic diagram of a morphology-aware regional feature enhancement module proposed in the present invention;

[0027] Figure 4 Schematic diagram of the global category-guided fusion module designed for the present invention. DETAILED DESCRIPTION

[0028] A preferred embodiment of the present invention is described in detail below with reference to the accompanying drawings.

[0029] The present invention designs a task-related feature learning method to extract subtle morphological differences between pyramidal cells and identify the types of pyramidal cells in a task-related manner. First, the common features between multiple tasks are extracted through a cross-task, shared feature extractor, and then task-specific features are generated for each task separately, and a morphology-aware regional feature enhancement module is used to refine the task-specific features. In addition, the present invention uses a global category-guided fusion module to adaptively fuse and enhance task-specific features and task-common features to generate unique feature descriptors. In addition, by designing two models, the present method can significantly improve the classification performance, providing a simple and effective tool for large-scale neuronal morphological analysis, neuropathological analysis, neuromorphic development research, and neuronal anatomy research.

[0030] 1. Specific architecture

[0031] There are many types of pyramidal cells, and pyramidal cells in different brain regions or species show morphological diversity, such as Figure 1 Analyzing multiple properties of pyramidal cells is helpful for better understanding the circuits and functions of the brain. The present invention aims to utilize the relationship between different properties of pyramidal cells to promote the extraction of morphological features, thereby improving the performance of each analysis task. Figure 2 As shown in FIG. 1 , the present invention includes steps such as data preprocessing, feature extraction, learning of specific representations, feature fusion and classification. It extracts morphological features in a task-related manner and simultaneously analyzes the species and brain region to which the neurons belong. Specifically, the present invention first projects the 3D neuron data to generate three 2D view images and fuses these images along the channel direction. Then, the present invention uses a task-shared feature extractor to extract the common features F of the task from these 2D images. c Next, the present invention extracts task-specific features for each task: the task-specific features of task s are F s , the task-specific feature of task l is F l , and use the morphologically aware Region Feature Enhancement Module (RFEM) to enhance the task-specific features to obtain enhanced task-specific features: F s The corresponding enhanced task-specific feature is F se , F l The corresponding enhanced task-specific feature is F le .

[0032] Secondly, the present invention uses the Global Category-guided Fusion Module (GCFM) to fuse F c and F se 、F cand F le , to generate a more discriminative feature descriptor f for pyramidal cells sd ,f ld Finally, the fused feature descriptors were input into a classifier to predict the species category or brain region of the pyramidal cells.

[0033] Due to the tree-like structure of neurons, most areas of the neuron image are background, and only a few pixel areas contain branches of the neuron tree. Therefore, the present invention introduces a morphologically aware regional feature enhancement module to effectively learn the morphological features of neurons from the morphologically aware area and eliminate the influence of the background. Specifically, the present invention first uses a feature transform layer (FTL) to obtain basic task-specific features (i.e., F s and F l ). This layer consists of two convolutional layers with a kernel size of 3×3 and a batch normalization layer. Then, the present invention uses a morphologically aware regional feature enhancement module to identify and focus on the salient features in the feature map. It has been shown that in CNN, the channels of features are associated with different regions or parts of the image. Therefore, the present invention performs morphologically aware regional feature enhancement based on channel attention. The morphologically aware regional feature enhancement module transforms the feature map F along the channel. s Divided into 2n sub-feature graphs Z = {Z1, Z2, ..., Z 2n}.like Figure 3 As shown in Figure 2, these sub-feature maps are further divided into three groups, namely Round down. Then, the morphologically aware regional feature enhancement module uses the Attention Map Generation Module (AMGM) to calculate the attention map. The Attention Map Generation Module first uses the maximum pooling layer and the average pooling layer to obtain the global features. Subsequently, their outputs are connected along the channel. Next, the present invention applies a fully connected layer followed by a softmax layer to calculate the attention map as shown below:

[0034] A i =softmax(fc([max(Z i );avg(Z i )]));

[0035] where i∈{L,R,M}, [·] is a concatenation operation. fc is a fully connected layer used to reduce the dimension of its output to the same size as F s Considering that the key morphological area of ​​the neuron tree is usually located in the center, the present invention emphasizes A M The information is used to generate the final attention map A t , as shown below:

[0036] A t =[A L ; A R ];

[0037] A t [Z(nn / 2),Z(nn / 2+1),…,Z(n+n / 2)]+=A M .

[0038] Finally, by adding the weight A t Assigned to the feature map F s To obtain the enhanced task-specific feature map F se :

[0039] F se =A t ⊙F s +BN(F s );

[0040] Where BN stands for batch normalization.

[0041] The present invention first uses a broadcast mechanism to expand A along the last two dimensions t Then with F s Multiply. The F obtained se The salient features of the morphological perception area can be captured and the task-specific features in the species analysis task can be enhanced. Similarly, the present invention can obtain the enhanced task-specific features F of the brain region analysis task. le .

[0042] In order to improve the classification performance of pyramidal cells, the present invention uses a fusion module guided by global categories to adaptively fuse the F c and task-specific features (F se or F le ), a unique morphological descriptor (F sd or F ld ). This module aims to amplify the impact of task-specific features while retaining the beneficial effects of task-common features on the final decision. For the species analysis task, in order to better utilize the task-common and task-specific features, the present invention first predicts F se and F c The weight W se and W c , which is then reflected as the final morphological descriptor. In order to make accurate category predictions, the present invention measures the importance of samples by using task-common and task-specific features to generate a discriminative descriptor. Specifically, the present invention first generates a discriminative descriptor by respectively se and F cThe average pooling operation is used to extract the global features (f se and f c ). Then, for f se , the present invention generates a prediction output through a classifier (G) and a softmax layer, denoted as P i (x) = softmax(G(x)). G contains two fully connected layers, and the outputs of these fully connected layers are 200 and the total number of classes related to the task (C se ).

[0043] Therefore, P i The length is equal to C se Next, the present invention uses the one-dimensional vector of P i The calculated entropy and task-specific features are used to obtain the weight of each sample (H i ):

[0044]

[0045] in is a function that converts the estimated entropy into a real number and measures the importance of the sample according to task-specific features.

[0046] Therefore, according to f se All calculated H i The set of composes the weight W assigned to all samples for species analysis. se . Note that the entropy of the predicted output is a probability across categories, representing the confidence of the input. A higher output confidence indicates a more accurate prediction of the sample. Therefore, the present invention attaches this importance to the descriptor f by assigning weights to the inputs. se1 superior:

[0047] f se1 =W se ⊙f se +BN(f se );

[0048] W se ={H i};

[0049] Similarly, the present invention can obtain the feature f from the common features of the tasks through the above operations. c1 Finally, the present invention connects f se1 and f c1 Get the final morphological descriptor F sd In addition, the present invention can obtain the fusion feature F for brain region analysis tasks. ld .

[0050] 2. Experimental results

[0051] 2.1 Experimental Setup

[0052] In order to verify the effectiveness of the present invention, the present invention randomly downloaded 5951 pyramidal cell PCs from the public dataset NeuroMorpho.org. These cells originate from the somatosensory neocortex and medial prefrontal neocortex of rats and mice, and are located in the L2 / 3, L4, L5, and L6 cerebral cortex. For the species analysis task, the dataset contains 2,630 rat cells and 3,321 mouse cells. For the brain region analysis task, the dataset has 2,350 L2 / 3 cells, 723 L4 cells, 2,261 L5 cells, and 617 L6 cells.

[0053] The present invention uses the PyTorch framework to implement the task-related feature-based learning model proposed in the present invention. The model uses ResNet50 pre-trained on ImageNet as a common feature extractor and is trained on a single 3090 GPU card. The present invention sets n in the morphologically aware regional feature enhancement module to 2, and the Adam optimizer optimizes the model of the present invention. The present invention sets the learning rate to 0.001, the batch size to a configuration of 16, and the number of training iterations to 80.

[0054] 2.2 Experimental Results

[0055] Experimental results show that the present invention can extract effective morphological representations for multiple morphological analysis tasks and accurately identify the species or brain regions of pyramidal cells, and its performance is better than other methods.

[0056] The present invention was compared with a task-independent analysis method. In addition, the present invention was also compared with a method that performs the analysis task in a single-task manner. This method does not consider task-common and task-specific features and is labeled as a baseline method. Compared with the baseline, the method of the present invention improved the classification accuracy by 5.12% in the species analysis task. In addition, when analyzing the brain region of pyramidal cells, the method of the present invention achieved a 5.55% performance improvement. In addition, compared with the task-related method, the method of the present invention also achieved a significant improvement in classification accuracy. This is because the method of the present invention considers both task-common features and task-specific features. These results show that the method of the present invention can extract subtle morphological differences between species or brain layers of pyramidal cells and can effectively characterize them in a task-related manner.

[0057] The learning model based on task-related features proposed in the present invention successfully identifies the species and brain regions of pyramidal cells, can effectively characterize the morphology of pyramidal cells, and can effectively assist in large-scale neuronal morphological analysis, neuropathological analysis, neurodevelopmental research, neuroanatomy research, etc., thereby promoting brain science research.

[0058] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention, and any reference numerals in the claims should not be regarded as limiting the claims involved.

[0059] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A neuron morphology recognition method based on task-specific representation learning, characterized in that: Input the neuron data into the learning model and output the category to which the neuron data belongs; the implementation process of the learning model includes the following steps: Step 1: Data preprocessing: Project the 3D neuron data into three 2D view images, and then use the 2D view images as input for different tasks; Step 2: Feature extraction: First, the 2D view image is passed through a shared feature extractor to extract the common features for different tasks; task-specific features are extracted for each task, and the task-specific features are enhanced using a morphology-aware regional feature enhancement module to obtain enhanced task-specific features; Step 3: Feature fusion: Use the global category-guided fusion module to adaptively fuse the common features of the task and the enhanced task-specific features to form an instance-level feature descriptor. Step 4: Classification: Input the feature descriptor into a linear classifier to predict the neuron category of the 3D neuron data.

2. The method for neuron morphology recognition based on task-specific representation learning according to claim 1, characterized in that: Step 1 specifically includes: orthogonally projecting the 3D neuron data onto three planes, and connecting the three acquired 2D view images along the channels as inputs for different tasks.

3. The neuron morphology recognition method based on task-specific representation learning according to claim 1, characterized in that: In step 2, extracting task-specific features for each task and enhancing the task-specific features using a morphology-aware regional feature enhancement module to obtain enhanced task-specific features specifically includes: Use the feature conversion layer to obtain task-specific features; the feature conversion layer consists of two convolutional layers and a batch normalization layer; The morphologically aware regional feature enhancement module divides the task-specific features into 2n sub-feature maps along the channel, and further divides the 2n sub-feature maps into three groups. Then, an attention map generation module is used to calculate the attention map, which specifically includes: the attention map generation module first uses a maximum pooling layer and an average pooling layer to obtain global features, connects them along the channel, and then passes through a fully connected layer and a softmax activation function layer to obtain the attention map; The attention map is assigned as weights to the task-specific features to obtain enhanced task-specific features.

4. The method for neuron morphology recognition based on task-specific representation learning according to claim 1, characterized in that: In step 3, the fusion module using global category guidance adaptively fuses the common features of the task and the enhanced task-specific features to form an instance-level feature descriptor, which specifically includes: First, global features are extracted by using average pooling operations on enhanced task-specific features and task-common features respectively; The following operations are performed on the global features that enhance the task-specific features and the global features that enhance the task-common features: The global features are passed through a classifier and a softmax activation function layer to generate a prediction output; the classifier contains two fully connected layers; the generated prediction output is used to obtain the weight of each sample, the weight is assigned to the corresponding sample, and the weight is added to the global features after batch normalization to obtain a descriptor; The descriptors corresponding to the enhanced task-specific features and the descriptors corresponding to the task-common features are connected to obtain the instance-level feature descriptors.

Images