Method for spine annotation and tracking of neuron fluorescence microscopy images

Abstract

The disclosure provides a dendritic spine annotation method and tracking method of a neuron fluorescence microscopic image, which can be applied to the technical field of image processing and the technical field of neurobiology. The annotation method comprises: determining a neuron region in the neuron fluorescence microscopic image according to a pixel value of a pixel point in the neuron fluorescence microscopic image; determining a candidate peak point located in the neuron region from the pixel point based on a local peak value of the pixel value, wherein the distance between adjacent local peak values is greater than a preset threshold; determining a target peak point matched with the dendritic spine in the neuron fluorescence microscopic image according to the pixel value change information of the candidate peak point; and annotating a dendritic spine region in the neuron fluorescence microscopic image based on the target peak point and using gradient change information of the pixel points in a preset search box to obtain a target annotation image.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of image processing and the technical field of neurobiology, and more particularly, to a dendritic spine labeling method and a tracking method for a neuron fluorescence microscopic image. BACKGROUND

[0002] Neurons communicate by forming synaptic sites, and in mammals, more than 90% of excitatory synapses are located on dendritic protrusions, known as dendritic spines. Changes in the geometry, size, density, and spatial distribution of dendritic spines affect the information integration of neurons, and therefore, dendritic spines play an important role in learning and memory and various neural information processing. Many neurological diseases are accompanied by abnormalities in dendritic spines, and therefore, light microscopy is often used to reconstruct the morphology of dendritic spines for quantitative analysis of dendritic spines.

[0003] In the process of implementing the present disclosure, the inventors found that at least the following problems exist in the related art: In the related art, the method for labeling dendritic spines in a neuron fluorescence microscopic image takes a long time, resulting in low labeling efficiency, and cannot track dendritic spines over time. SUMMARY

[0004] Therefore, the present disclosure provides a dendritic spine labeling method and a tracking method for a neuron fluorescence microscopic image.

[0005] One aspect of the present disclosure provides a dendritic spine labeling method for a neuron fluorescence microscopic image, comprising:

[0006] determining a neuron region in the neuron fluorescence microscopic image according to pixel values of pixel points in the neuron fluorescence microscopic image;

[0007] determining a candidate peak point located in the neuron region from the pixel points based on a local peak value of the pixel values, wherein the local peak value represents a maximum pixel value in a local region in the neuron fluorescence microscopic image, and a distance between adjacent local peak values is greater than a preset threshold;

[0008] determining a target peak point matching the dendritic spine in the neuron fluorescence microscopic image according to pixel value change information of the candidate peak point;

[0009] labeling a dendritic spine region in the neuron fluorescence microscopic image based on the target peak point and using gradient change information of pixel points in a preset search box to obtain a target labeling image.

[0010] According to an embodiment of the present disclosure, the determination of the neuron region in the neuron fluorescence microscopic image according to the pixel values of the pixel points in the neuron fluorescence microscopic image comprises:

[0011] According to the pixel value, the mean value and the standard deviation of the pixel points in the neuron fluorescence microscopic image are determined;

[0012] Based on the mean value and the standard deviation, the first threshold and the second threshold are determined, wherein the first threshold is the sum of the mean value and three times the standard deviation, and the second threshold is the sum of the mean value and the standard deviation;

[0013] Based on the first threshold and the second threshold, the initial neuron region in the neuron fluorescence microscopic image is determined;

[0014] The neuron region is determined in the initial neuron region by using the adaptive threshold.

[0015] According to an embodiment of the present disclosure, wherein the target peak point matched with the dendritic spine in the neuron fluorescence microscopic image is determined according to the pixel value of the candidate peak point, comprising:

[0016] According to the pixel value of the candidate peak point, the partial derivative of the pixel value is determined;

[0017] According to the partial derivative, the structure tensor matrix of the candidate peak point is determined;

[0018] Based on a plurality of first eigenvalues of the structure tensor matrix, the target peak point matched with the dendritic spine in the neuron fluorescence microscopic image is determined.

[0019] According to an embodiment of the present disclosure, wherein the target peak point matched with the dendritic spine in the neuron fluorescence microscopic image is determined according to the pixel value of the candidate peak point, comprising:

[0020] According to the pixel value of the candidate peak point, the second-order partial derivative of the pixel value is determined;

[0021] Based on the second-order partial derivative, the Hessian matrix of the candidate peak point is determined;

[0022] The absolute value processing is performed on a plurality of eigenvalues of the Hessian matrix to obtain a second eigenvalue;

[0023] Based on the second eigenvalue, the target peak point matched with the dendritic spine in the neuron fluorescence microscopic image is determined.

[0024] According to an embodiment of the present disclosure, wherein based on the target peak point, the dendritic spine region in the neuron fluorescence microscopic image is labeled by using a preset search box, comprising:

[0025] A plurality of pixel blocks in the neuron fluorescence microscopic image are determined by using the preset search box with the target peak point as the center;

[0026] For each pixel block in the plurality of pixel blocks, region growing is performed based on the target peak point to obtain an intermediate region;

[0027] According to the gray gradient direction of the intermediate region, the dendritic spine region in the neuron fluorescence microscopic image is labeled.

[0028] Another aspect of the present disclosure provides a dendritic spine tracking method of a neuron fluorescence microscopic image, comprising:

[0029] A plurality of neuron fluorescence microscopic images at different time points are acquired through continuous imaging.

[0030] The dendritic spine regions in the neuron fluorescence microscopic images at different time points are labeled by using a pre-trained dendritic spine labeling model to obtain a plurality of dendritic spine regions, wherein the dendritic spine labeling model is obtained by training a deep learning model using the target labeled images as training samples.

[0031] According to the swing range of the dendritic spine in the plurality of dendritic spine regions, a target tracking method is used to determine the motion trajectory of the dendritic spine, so as to track the dendritic spine in the neuron fluorescence microscopic image.

[0032] According to an embodiment of the present disclosure, the training method of the dendritic spine labeling model comprises:

[0033] The training samples are acquired, wherein the training samples include target labeled image training sub-samples and neuron fluorescence microscopic image training sub-samples.

[0034] The target labeled image training sub-samples are used as labels, and the neuron fluorescence microscopic image training sub-samples are used as inputs to train a deep learning model to obtain the dendritic spine labeling model.

[0035] According to an embodiment of the present disclosure, the target tracking method at least includes one of the following: a maximum value projection tracking method, a Kalman filter tracking method, and a Bayesian tracking method.

[0036] According to an embodiment of the present disclosure, in the case where the target tracking method includes the maximum value projection tracking method, according to the swing range of the dendritic spine in the plurality of dendritic spine regions, the target tracking method is used to determine the motion trajectory of the dendritic spine, so as to track the dendritic spine in the neuron fluorescence microscopic image, comprising:

[0037] Based on the maximum swing range of the dendritic spine in the plurality of dendritic spine regions, a maximum value projection result of the neuron fluorescence microscopic image in the time domain is determined.

[0038] According to the maximum value projection result of the neuron fluorescence microscopic image in the time domain, the motion trajectory of the dendritic spine is determined, so as to track the dendritic spine in the neuron fluorescence microscopic image.

[0039] According to an embodiment of the present disclosure, in a case where the target tracking method comprises a Kalman filter tracking method or a Bayesian tracking method, the motion trajectory of the dendritic spine is determined by using the target tracking method so as to track the dendritic spine in the neuron fluorescence microscopic image, comprising:

[0040] Based on the dendritic spine in the plurality of dendritic spine regions, the centroid position of the dendritic spine is determined;

[0041] According to the centroid position, the motion trajectory of the dendritic spine is determined so as to track the dendritic spine in the neuron fluorescence microscopic image.

[0042] According to an embodiment of the present disclosure, because the technical means of determining the target peak point matched with the dendritic spine by using the pixel value of different pixel points in the neuron fluorescence microscopic image, and labeling the dendritic spine region according to the target peak point is adopted, the dependence on the dendritic spine can be reduced without skeleton recognition of the dendritic spine, and manual recognition is not required, so that at least part of the technical problems that the method of labeling the dendritic spine in the neuron fluorescence microscopic image in the related art needs to spend a long time and leads to low labeling efficiency is overcome, and the technical effect of improving the labeling efficiency of the dendritic spine region in the neuron fluorescence microscopic image is achieved. BRIEF DESCRIPTION OF DRAWINGS

[0043] The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments of the present disclosure taken in conjunction with the accompanying drawings, in which:

[0044] Figure 1 A flowchart of a dendritic spine labeling method of a neuron fluorescence microscopic image according to an embodiment of the present disclosure is schematically shown;

[0045] Figure 2A A candidate peak point schematic diagram of a neuron fluorescence microscopic image according to an embodiment of the present disclosure is schematically shown;

[0046] Figure 2B A local peak schematic diagram of a neuron fluorescence microscopic image according to an embodiment of the present disclosure is schematically shown;

[0047] Figure 2C A candidate peak point schematic diagram of a neuron fluorescence microscopic image according to an embodiment of the present disclosure is schematically shown;

[0048] Figure 2D A schematic diagram of a target peak point of a neuron fluorescence microscopic image according to an embodiment of the present disclosure is schematically shown;

[0049] Figure 2E A schematic diagram of a target labeling image according to an embodiment of the present disclosure is schematically shown;

[0050] Figure 3AA schematic diagram of a first eigenvalue λ1 of a structure tensor matrix according to an embodiment of the present disclosure is shown schematically;

[0051] Figure 3B A schematic diagram of a first eigenvalue λ2 of a structure tensor matrix according to an embodiment of the present disclosure is shown schematically;

[0052] Figure 3C A schematic diagram of a first eigenvalue λ3 of a structure tensor matrix according to an embodiment of the present disclosure is shown schematically;

[0053] Figure 4A A schematic diagram of a second eigenvalue λ6 of a Hessian matrix according to an embodiment of the present disclosure is shown schematically;

[0054] Figure 4B A schematic diagram of a second eigenvalue λ7 of a Hessian matrix according to an embodiment of the present disclosure is shown schematically;

[0055] Figure 4C A schematic diagram of a second eigenvalue λ8 of a Hessian matrix according to an embodiment of the present disclosure is shown schematically;

[0056] Figure 5A A schematic diagram of region growing based on target peak points of a neuron fluorescence microscopic image according to an embodiment of the present disclosure is shown schematically;

[0057] Figure 5B A schematic diagram of radiation constraint of a neuron fluorescence microscopic image according to an embodiment of the present disclosure is shown schematically;

[0058] Figure 5C A schematic diagram of an intermediate of a neuron fluorescence microscopic image according to an embodiment of the present disclosure is shown schematically;

[0059] Figure 5D A schematic diagram of a dendritic spine region of a neuron fluorescence microscopic image according to an embodiment of the present disclosure is shown schematically;

[0060] Figure 6 A schematic diagram of a dendritic spine annotation model training process according to an embodiment of the present disclosure is shown schematically;

[0061] Figure 7A A schematic diagram of a dendritic spine annotation model annotating neuron morphology data according to an embodiment of the present disclosure is shown schematically;

[0062] Figure 7B A schematic diagram of a dendritic spine annotation model annotating dendritic spine data according to an embodiment of the present disclosure is shown schematically;

[0063] Figure 8 A flowchart of a dendritic spine tracking method of a neuron fluorescence microscopic image according to an embodiment of the present disclosure is shown schematically;

[0064] Figure 9A A schematic diagram of a plurality of neuron fluorescence microscopy images is shown schematically according to an embodiment of the present disclosure;

[0065] Figure 9B A schematic diagram of a dendritic spine region matching result in a neuron fluorescence microscopy image in a time domain is shown schematically. DETAILED DESCRIPTION

[0066] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It is to be understood, however, that the description is merely exemplary and is not intended to limit the scope of the present disclosure. In the following detailed description of the embodiments of the present disclosure, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. However, it would be apparent to those skilled in the art that the embodiments of the present disclosure can be practiced without these specific details. In other instances, well-known structures and methods are not described in detail in order to avoid obscuring the concepts of the present disclosure.

[0067] The terms used herein are merely used to describe specific embodiments and are not intended to limit the present disclosure. The terms "include", "comprise" and the like as used herein indicate the presence of the described features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0068] All terms used herein, including technical and scientific terms, have the meanings commonly understood by one of ordinary skill in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having meanings consistent with the context of the specification, and should not be interpreted in an idealized or overly formal manner.

[0069] In the case of using expressions similar to "at least one of A, B, and C, etc.", it should be generally interpreted as having the meaning of including at least one of the items listed. In the case of using expressions similar to "at least one of A, B, or C, etc.", it should be generally interpreted as having the meaning of including at least one of the items listed. For example, "a system having at least one of A, B, and C" should include but not be limited to a system having A alone, a system having B alone, a system having C alone, a system having both A and B, a system having both A and C, a system having both B and C, and / or a system having A, B, and C, etc. In the case of using expressions similar to "at least one of A, B, or C, etc.", it should be generally interpreted as having the meaning of including at least one of the items listed. For example, "a system having at least one of A, B, or C" should include but not be limited to a system having A alone, a system having B alone, a system having C alone, a system having both A and B, a system having both A and C, a system having both B and C, and / or a system having A, B, and C, etc.

[0070] In the technical solutions of the present disclosure, the acquisition, storage, and application of user personal information involved comply with relevant legal regulations, necessary security measures are taken, and do not violate public order and good customs.

[0071] In the technical solutions of the present disclosure, the authorization or consent of the user is obtained before the personal information of the user is acquired or collected.

[0072] In the related art, the main method for analyzing dendritic spines is to draw by manual method, but due to the small, dense and rich structure of the observed dendritic spines, the manual method requires considerable time and effort, and it is more difficult on three-dimensional images, and the edge is relatively rough and the precision is not high, thereby limiting the application in a large amount of imaging data analysis.

[0073] In the algorithm based on morphological recognition of dendritic spines, it generally depends on the segmentation of dendrites, and has a high requirement for the pixel resolution and signal-to-noise ratio of the neuron fluorescence microscopic image, and when the dendrites of the neuron fluorescence microscopic image are weak or the branches are complex, the dendritic spines cannot be well recognized. Supervised machine learning can improve the accuracy, but a large amount of neuron fluorescence microscopic image data accurately labeled by manual labor is needed for training, so that the preparation of the training set needs to consume a lot of time and effort, and at present it is not widely used. In addition, the current algorithm mostly processes single frame image and does not support time tracking.

[0074] Therefore, the embodiments of the present disclosure provide a dendritic spine labeling method of a neuron fluorescence microscopic image. The method comprises: determining a neuron region in the neuron fluorescence microscopic image according to pixel values of pixel points in the neuron fluorescence microscopic image; determining candidate peak points located in the neuron region from the pixel points based on local peak values of the pixel values, wherein the local peak value represents local region maximum pixel value change information in the neuron fluorescence microscopic image, and distances between adjacent local peak values are greater than a preset threshold; determining target peak points matched with the dendritic spine in the neuron fluorescence microscopic image according to the pixel values of the candidate peak points; and labeling a dendritic spine region in the neuron fluorescence microscopic image based on the target peak points and gradient change information of the pixel points labeled by a preset search box to obtain a target labeling image.

[0075] Figure 1 A flowchart of a dendritic spine labeling method of a neuron fluorescence microscopic image according to an embodiment of the present disclosure is schematically shown.

[0076] As shown in FIG. 1, the method comprises operations S101-S104. Figure 1

[0077] In operation S101, a neuron region in the neuron fluorescence microscopic image is determined according to pixel values of pixel points in the neuron fluorescence microscopic image.

[0078] According to the embodiments of the present disclosure, the neuron fluorescence microscopic image can comprise a neuron confocal image, and the neuron confocal image can be an image taken under a confocal microscope. ​

[0079] According to an embodiment of the present disclosure, the neuron fluorescence microscopic image can be two-dimensional or three-dimensional.

[0080] According to an embodiment of the present disclosure, since the brightness of the neuron region and the non-neuron region in the neuron fluorescence microscopic image is different, the pixel values of the neuron region and the non-neuron region are also different, so the neuron region can be determined by the pixel value of the pixel point in the neuron fluorescence microscopic image. As shown in FIG. 2, it should be noted that the embodiments of the present disclosure are all based on the processing of neuron fluorescence microscopic image data of the confocal image type, and other types of neuron fluorescence microscopic image can be improved by the conventional technical means of the person skilled in the art to realize the present disclosure.

[0081] Figure 2A The neuron region of the neuron fluorescence microscopic image according to an embodiment of the present disclosure is schematically shown.

[0082] As shown in FIG. 2, the white region is the neuron region. Figure 2A

[0083] In operation S102, based on the local peak value of the pixel value, the candidate peak point located in the neuron region is determined from the pixel point, wherein the local peak value represents the maximum pixel value of the local region in the neuron fluorescence microscopic image, and the distance between adjacent local peak values is greater than a preset threshold value.

[0084] According to an embodiment of the present disclosure, the neuron fluorescence microscopic image can be a two-dimensional or three-dimensional image, and the local maximum pixel value can be determined by using the idea of local peak value according to the pixel value of each pixel point in the neuron fluorescence microscopic image. The local maximum pixel value is taken as the local peak value, so as to obtain as many candidate peak points as possible, which cover the whole and are distributed on the neuron dendrite and dendritic spine.

[0085] According to an embodiment of the present disclosure, since the distance between the dendritic spines is close, the preset threshold value r is needed to be determined first, so that the distance between the two adjacent candidate peak points is greater than the preset threshold value r. The size of the preset threshold value r is related to the pixel interval of the neuron fluorescence microscopic image. A filtering core with a radius of 2*r+1 and a size of all 1 is used, and a maximum value filter is used to process the neuron fluorescence microscopic image I to obtain a processed first image I'. The pixel points in the neuron fluorescence microscopic image I greater than the first image I' are determined as the candidate peak points, as shown in FIG. 3.

[0086] Figure 2B The local peak value of the neuron fluorescence microscopic image according to an embodiment of the present disclosure is schematically shown.

[0087] Figure 2C ​A schematic diagram of a candidate peak point of a neuron fluorescence microscopic image is shown.

[0088] As shown in Figure 2B and 2C , there are multiple candidate peak points in the neuron fluorescence microscopic image, and the candidate peak points are all in the neuron region.

[0089] In operation S103, according to the pixel value change information of the candidate peak point, a target peak point in the neuron fluorescence microscopic image that matches the dendritic spine is determined.

[0090] According to an embodiment of the present disclosure, in the neuron fluorescence microscopic image, the dendritic spine and the dendrite differ in the shape features that are obvious in the image, that is, the dendrite is generally linear and the dendritic spine is point radiation, so the dendritic spine and the dendrite can be abstracted as a line and a point, and by constructing a point-line classifier, the classification of the candidate peak point is realized, and the candidate peak point is divided into the target peak point that matches the dendritic spine and other peak points that are not dendritic spines.

[0091] Figure 2D A schematic diagram of a target peak point of a neuron fluorescence microscopic image is shown.

[0092] As shown in Figure 2D , the target peak point falls in the dendritic spine position of the neuron.

[0093] In operation S104, based on the target peak point, the gradient change information of the pixel points in the preset search box is used to label the dendritic spine region in the neuron fluorescence microscopic image, and a target labeled image is obtained.

[0094] According to an embodiment of the present disclosure, since the dendritic spine is very dense and small in structure, the dendritic spine is often close and difficult to distinguish, and the brightness gradient direction of the adjacent dendritic spine at the edge of the two has the feature of pointing to the respective dendritic spine, so a preset search box can be set, and the dendritic spine region is labeled according to the image gray scale change feature of the target peak point in the preset search box, and a target labeled image is obtained.

[0095] Figure 2E A schematic diagram of a target labeled image is shown.

[0096] As shown in Figure 2E , the dendritic spine region in the neuron is labeled, and the adjacent dendritic spines can also be distinguished.

[0097] According to the embodiment of the present disclosure, because the technical means of determining the target peak point matched with the dendritic spine by using the pixel value of different pixels in the neuron fluorescence microscopic image, and labeling the dendritic spine region according to the target peak point is adopted, the dependence on the dendrite can be reduced, and manual identification is not required, so that the technical problems that the method of labeling the dendritic spine in the neuron fluorescence microscopic image in the related art needs to spend a long time, resulting in low labeling efficiency, and the dendritic spine cannot be tracked in time are at least partially overcome, and the technical effect of improving the labeling efficiency of the dendritic spine region in the neuron fluorescence microscopic image is achieved.

[0098] According to the embodiment of the present disclosure, wherein the neuron region in the neuron fluorescence microscopic image is determined according to the pixel value of the pixel point in the neuron fluorescence microscopic image, comprising:

[0099] According to the pixel value, the mean value and the standard deviation of the pixel point in the neuron fluorescence microscopic image are determined;

[0100] Based on the mean value and the standard deviation, the first threshold and the second threshold are determined, wherein the first threshold is the sum of the mean value and three times the standard deviation, and the second threshold is the sum of the mean value and the standard deviation;

[0101] Based on the first threshold and the second threshold, the initial neuron region in the neuron fluorescence microscopic image is determined;

[0102] The neuron region is determined in the initial neuron region by using the adaptive threshold.

[0103] According to the embodiment of the present disclosure, the mean value mean and the standard deviation sd of the pixel point in the neuron fluorescence microscopic image are calculated from the pixel value of all pixel points in the neuron fluorescence microscopic image, and the first threshold is mean+3*sd, and the second threshold is mean+sd. Specifically, a smaller first contour is obtained in the neuron fluorescence microscopic image by a larger first threshold, and an initial neuron region is obtained by expanding the smaller first contour by a smaller second threshold, that is, there is a pixel point with a higher pixel value in each initial neuron region.

[0104] According to the embodiment of the present disclosure, the second contour is determined in the neuron fluorescence microscopic image by the adaptive threshold, and the neuron region is obtained according to the part common to the second contour and the initial neuron region.

[0105] According to the embodiment of the present disclosure, the neuron region is obtained by combining the first threshold and the second threshold with the adaptive threshold, so that the background noise in the neuron fluorescence microscopic image can be better removed.

[0106] According to an embodiment of the present disclosure, the target peak point matched with the dendritic spine in the neuron fluorescence microscopic image is determined according to the pixel value of the candidate peak point, comprising:

[0107] The partial derivative of the pixel value is determined according to the pixel value of the candidate peak point.

[0108] The structure tensor matrix of the candidate peak point is determined according to the partial derivative.

[0109] The target peak point matched with the dendritic spine in the neuron fluorescence microscopic image is determined based on the plurality of first eigenvalues of the structure tensor matrix.

[0110] According to an embodiment of the present disclosure, the neuron fluorescence microscopic image can be three-dimensional or two-dimensional. Here, taking a three-dimensional image as an example, the partial derivatives of the neuron fluorescence microscopic image I in the x, y, and z dimensions are represented as I x , I y , and I z respectively, so as to obtain the characteristics of the change of the neuron fluorescence microscopic image. The partial derivatives are multiplied two by two to form the following structure tensor matrix S:

[0111]

[0112] According to an embodiment of the present disclosure, the three-dimensional neuron fluorescence microscopic image generates a 3x3 structure tensor matrix, and the two-dimensional neuron fluorescence microscopic image generates a 2x2 structure tensor matrix.

[0113] According to an embodiment of the present disclosure, the eigenvalues of the structure tensor matrix are calculated, which are denoted as first eigenvalues. The three-dimensional image can obtain three first eigenvalues λ1, λ2, and λ3, which are arranged in descending order as λ1, λ2, and λ3. The two-dimensional image can obtain two first eigenvalues λ4 and λ5, which are arranged in descending order as λ4 and λ5. Thus, the rate of change of the neuron fluorescence microscopic image in the orthogonal direction is obtained. Since the structure tensor matrix is a non-negative matrix, the first eigenvalues are not less than zero.

[0114] According to an embodiment of the present disclosure, the first eigenvalues λ1, λ2, and λ3 of the neuron fluorescence microscopic image are plotted as shown in FIG. 1. Figures 3A-3C

[0115] Figure 3A A schematic diagram of the first eigenvalue λ1 of the structure tensor matrix according to an embodiment of the present disclosure is shown schematically.

[0116] Figure 3B A schematic diagram of the first eigenvalue λ2 of the structure tensor matrix according to an embodiment of the present disclosure is shown schematically.

[0117] Figure 3C ​A schematic diagram of the first eigenvalue λ3 of the structure tensor matrix according to an embodiment of the present disclosure is shown schematically.

[0118] As shown in Figures 3A-3C It can be seen that the first eigenvalue λ1 and the first eigenvalue λ2 highlight more elongated dendrites, while the first eigenvalue λ3 highlights more rounded dendritic spines.

[0119] According to an embodiment of the present disclosure, the second image R highlighting the dendritic spine region is calculated from the target peak point in the neuron fluorescence microscopic image using formula (2) for a three-dimensional image and formula (3) for a two-dimensional image.

[0120] R = λ3 * λ3 / (λ1 - λ2 + ε) / (λ2 - λ3 + ε) (2)

[0121] R = λ5 / (λ4 - λ5 + ε) (3)

[0122] According to an embodiment of the present disclosure, the second image R is binarized using an adaptive threshold to obtain a mask mask, and the target peak point in the neuron fluorescence microscopic image that matches the dendritic spine is determined by screening the candidate peak point according to the mask mask:

[0123]

[0124] According to an embodiment of the present disclosure, wherein the target peak point in the neuron fluorescence microscopic image that matches the dendritic spine is determined according to the pixel value of the candidate peak point, comprising:

[0125] determining the second-order partial derivative of the pixel value according to the pixel value of the candidate peak point;

[0126] determining the Hessian matrix of the candidate peak point based on the second-order partial derivative;

[0127] performing absolute value processing on a plurality of eigenvalues of the Hessian matrix to obtain a second eigenvalue;

[0128] determining the target peak point in the neuron fluorescence microscopic image that matches the dendritic spine based on the second eigenvalue.

[0129] According to an embodiment of the present disclosure, the Hessian matrix (Hessian) is a second-order partial derivative of the neuron fluorescence microscopic image, thereby obtaining the change of the change of the image, and the Hessian matrix H is directly constructed from the second-order partial derivative result, representing the change of the change of the image. Like the structure tensor, a 3x3 structure tensor matrix is generated for each point in the three-dimensional image, and if it is a two-dimensional image, it is a 2x2 matrix:

[0130]

[0131] According to an embodiment of the present disclosure, the Hessian matrix is different from the structure tensor matrix in that the positive and negative characteristics of points with different brightness changes in the Hessian matrix are different. If it is a positive definite matrix, a local minimum value is at a critical point, if it is a negative definite matrix, a local maximum value is at a critical point, and if it is an indefinite matrix, no extreme value is at a critical point.

[0132] According to an embodiment of the present disclosure, if the neuron fluorescence microscopic image is black as background and white as foreground, the candidate peak point is a negative definite matrix, and the eigenvalue is a negative number. Therefore, the eigenvalue needs to be processed first: the positive eigenvalue is set to 0, and the negative eigenvalue is inverted, and then the eigenvalues are sorted from large to small, denoted as the second eigenvalue. Similarly, three-dimensional images can obtain three processed second eigenvalues, which are arranged in descending order: λ6, λ7, λ8; two-dimensional images can obtain two processed second eigenvalues, which are arranged in descending order: λ9, λ10. Due to the processing of zero setting and inversion, these second eigenvalues are not less than zero.

[0133] According to an embodiment of the present disclosure, the second eigenvalues λ6, λ7, λ8 of the neuron fluorescence microscopic image are plotted as shown in Figures 4A-4C .

[0134] Figure 4A A schematic diagram of the second eigenvalue λ6 of the Hessian matrix according to an embodiment of the present disclosure is schematically shown.

[0135] Figure 4B A schematic diagram of the second eigenvalue λ7 of the Hessian matrix according to an embodiment of the present disclosure is schematically shown.

[0136] Figure 4C A schematic diagram of the second eigenvalue λ8 of the Hessian matrix according to an embodiment of the present disclosure is schematically shown.

[0137] As shown in Figures 4A-4C , it can be seen that the second eigenvalue λ6 and the second eigenvalue λ7 highlight more elongated dendrites, and the second eigenvalue λ8 highlights more round dendritic spines.

[0138] According to an embodiment of the present disclosure, after obtaining the second eigenvalue, the target peak point can be obtained by using formula (2) and formula (3) and the mask method.

[0139] According to an embodiment of the present disclosure, the results of the structure tensor matrix and the Hessian matrix can be corrected by manual operation to make the results more accurate.

[0140] According to an embodiment of the present disclosure, the structure tensor matrix and the Hessian matrix are both obtained in a non-supervised manner, and a certain radius is extended outward from the target peak point to obtain dendritic spines and small images that do not belong to dendritic spines. The small image is a two-dimensional image, which can be obtained from an original two-dimensional image or a two-dimensional projection of a three-dimensional image. The skeleton recognition of the dendritic spine is not required, so that the dependence on the segmentation of the dendritic spine is reduced, the robustness is improved, and the scope of applicability is expanded.

[0141] According to an embodiment of the present disclosure, a simple point-line classifier can also be constructed by using a simple machine learning method. Specifically, the method can be a supervised support vector machine (SVM) method and a convolutional neural network (CNN) method. The SVM method flattens a two-dimensional neuron fluorescence microscopic image to one dimension, and then uses principal component analysis (PCA) for dimension reduction. Finally, the SVM classifier is used for binary classification. The CNN method performs two-layer convolution and pooling processing on a two-dimensional neuron fluorescence microscopic image, and finally flattens and fully connects the neuron fluorescence microscopic image to judge the category of the neuron fluorescence microscopic image.

[0142] According to an embodiment of the present disclosure, the four classifiers can all determine the target peak point, and can be selected according to convenience and actual effect. Specifically, the structure tensor matrix and the Hessian matrix can be used to process dendritic spine-specific marker data, and the SVM and the CNN can be used to process neuron morphology filling marker data.

[0143] According to an embodiment of the present disclosure, based on the target peak point, the dendritic spine region in the neuron fluorescence microscopic image is labeled by using a preset search box, and the method comprises the following steps:

[0144] A plurality of pixel blocks are determined in the neuron fluorescence microscopic image by using the preset search box with the target peak point as the center.

[0145] For each pixel block in the plurality of pixel blocks, region growing is performed based on the target peak point to obtain an intermediate region.

[0146] The dendritic spine region in the neuron fluorescence microscopic image is labeled according to the gray gradient direction of the intermediate region.

[0147] According to an embodiment of the present disclosure, since the dendritic spines are very dense and small in structure, the dendritic spines are often close to each other and difficult to distinguish. Therefore, the morphological geodesic contour line algorithm is used to segment and label the dendritic spine region by using the feature that the gradient flow of the neuron fluorescence microscopic image is directed to the brightest center.

[0148] According to an embodiment of the present disclosure, a plurality of pixel blocks are determined in the neuron fluorescence microscopic image by using a preset search box centered on the target peak point, without analyzing all positions in the neuron fluorescence microscopic image, thereby reducing the amount of calculation and improving the labeling efficiency.

[0149] According to an embodiment of the present disclosure, for each pixel block of the plurality of pixel blocks, region growing is performed from the target peak point as a starting point, such as Figure 5A .

[0150] Figure 5A A schematic diagram of region growing based on a target peak point of a neuron fluorescence microscopic image according to an embodiment of the present disclosure is shown.

[0151] As shown in Figure 5A , each time a layer of pixels is extended outward from the current region to obtain an intermediate region, then the image gradient at the boundary of the intermediate region and the gradient of the contour mask are calculated to obtain two gradient vectors. The two gradient vectors are converted into unit vectors e1 and e2, if the inner product of the two unit vectors is greater than 0.1 and belongs to the neuron region, that is, the vector points to the dendritic spine and is a neuron region, it is labeled as a dendritic spine region, otherwise it is not a dendritic spine region. If there is no new dendritic spine region or the number of iterations reaches, then stop. If the grown region is too large and exceeds the set maximum dendritic spine size, it is not considered as a dendritic spine.

[0152] According to an embodiment of the present disclosure, since the convex hull property of the shape is not considered in the process of boundary iteration of the morphological geodesic contour line, it will cause the extension through a certain elongated path to an undesired coverage area, which will incorrectly segment the dendritic region in the dendritic spine region labeling task. Therefore, the present disclosure adds a radioactive nucleus for constraint.

[0153] Figure 5B A schematic diagram of radioactive constraint of a neuron fluorescence microscopic image according to an embodiment of the present disclosure is shown.

[0154] According to an embodiment of the present disclosure, the radioactive nucleus is an ideal displacement matrix pointing from the outside to the center, and the radioactive nucleus maintains the same size as the preset search box and is used to constrain the shape to be radial. In data testing, the radioactive nucleus can greatly eliminate the problem of dendritic spine extending to the dendritic region during region growing. The iteration mode is the same as the geodesic contour, and the two are iterated at the same time.

[0155] Figure 5C A schematic diagram of a neuron fluorescence microscopic image according to an embodiment of the present disclosure is shown. Figure 5D A schematic diagram of a dendritic spine region of a neuron fluorescence microscopic image according to an embodiment of the present disclosure is shown.

[0156] As shown in Figure 5C andFigure 5D , the two dendritic spines are segmented, and in Figure 5C , the image gradient of the gradient flow of each pixel point in the middle region and the gradient of the contour mask are calculated to obtain two unit vectors, and the circle point is the starting point of the vector. When the two dendritic spines are close to each other, they are likely to be divided into one dendritic spine if a common segmentation method based on fluorescence intensity is used. However, there is a clear vector pointing to the center of each dendritic spine at the boundary of the dendritic spine, so the method used in the present disclosure, such as Figure 5D , can correctly segment and label the two dendritic spine regions, and the use of radioactive verification and shape expansion restriction can avoid false segmentation into dendritic regions.

[0157] According to an embodiment of the present disclosure, after obtaining the neuron fluorescence microscopic image with labeled dendritic spine regions, the neuron fluorescence microscopic image can be used for training of deep learning.

[0158] According to an embodiment of the present disclosure, after obtaining the target labeled image by using the dendritic spine labeling method of the neuron fluorescence microscopic image according to the embodiment of the present disclosure, the target labeled image can be used as a training sample to train a deep learning model to obtain a dendritic spine labeling model.

[0159] According to an embodiment of the present disclosure, the training method of the dendritic spine labeling model comprises:

[0160] obtaining training samples, wherein the training samples comprise target labeled image training sub-samples and neuron fluorescence microscopic image training sub-samples;

[0161] training the deep learning model by taking the target labeled image training sub-samples as labels and the neuron fluorescence microscopic image training sub-samples as inputs to obtain the dendritic spine labeling model.

[0162] According to an embodiment of the present disclosure, the deep training can use a two-dimensional UNet network, such as Figure 6 .

[0163] Figure 6 An illustrative dendritic spine labeling model training process diagram according to an embodiment of the present disclosure is shown.

[0164] As shown in Figure 6 , the training samples can include target labeled image training sub-samples and neuron fluorescence microscopic image training sub-samples, the target labeled image training sub-samples can be neuron fluorescence microscopic images labeled with dendritic spine regions, and the labeled neuron fluorescence microscopic images can be subjected to semantic label conversion and image enhancement. The deep learning model is trained by the training samples, and after a plurality of layers of convolution, activation and pooling, the model converges to obtain a trained dendritic spine labeling model.

[0165] According to an embodiment of the present disclosure, it should be noted that the present disclosure adopts a simple UNet structure to realize training on lightweight data, and other deep learning models can also be used.

[0166] According to an embodiment of the present disclosure, the dendritic spine region in the neuron fluorescence microscopic image is labeled by using a dendritic spine labeling model, such as Figure 7A and 7B .

[0167] Figure 7A A schematic diagram of the dendritic spine labeling model labeling neuron morphology data according to an embodiment of the present disclosure is shown schematically.

[0168] Figure 7B A schematic diagram of the dendritic spine labeling model labeling dendritic spine data according to an embodiment of the present disclosure is shown schematically.

[0169] As shown in Figure 7A and Figure 7B , it can be seen that the dendritic spine labeling model can effectively label the neuron morphology and the dendritic spine region. It should be noted that, for the convenience of display, the three-dimensional image shows the image result of one cutting plane.

[0170] Figure 8 A flowchart of a dendritic spine tracking method for a neuron fluorescence microscopic image according to an embodiment of the present disclosure is shown schematically.

[0171] As shown in Figure 8 , the method comprises operation S801 to operation S803.

[0172] In operation S801, a plurality of neuron fluorescence microscopic images at different time points are acquired by continuous imaging.

[0173] In operation S802, the dendritic spine region in each neuron fluorescence microscopic image at different time points is labeled by using a pre-trained dendritic spine labeling model, and a plurality of dendritic spine regions are obtained. The dendritic spine labeling model is obtained by using the target labeling image obtained by the neuron fluorescence microscopic image dendritic spine labeling method according to an embodiment of the present disclosure as a training sample.

[0174] In operation S803, according to the swing range of the dendritic spine in the plurality of dendritic spine regions, a target tracking method is used to determine the motion trajectory of the dendritic spine, so as to track the dendritic spine in the neuron fluorescence microscopic image.

[0175] According to an embodiment of the present disclosure, since the change of the dendritic spine is related to many neurological diseases, tracking the change of the dendritic spine is also crucial for the treatment of neurological diseases.

[0176] According to an embodiment of the present disclosure, a plurality of neuron fluorescence microscopic images at different time points can be obtained by continuous imaging in time sequence, and a plurality of neuron fluorescence microscopic images are obtained, such as Figure 9A .

[0177] Figure 9A A schematic diagram of a plurality of neuron fluorescence microscopic images according to an embodiment of the present disclosure is shown schematically.

[0178] As shown in Figure 9A , maximum projection t -mip The neuron fluorescence microscopic images at different time points t0, t1, t2, … are analyzed, and the dendritic spine regions are labeled.

[0179] According to an embodiment of the present disclosure, there are some errors in the labeling process of the dendritic spine region, such as not being recognized or being incorrectly segmented, and the errors of a single frame image are accumulated in a plurality of continuous frame images, which makes it more troublesome to correct. Therefore, for images with small dendritic spine swing, maximum projection can be considered in time to limit the segmentation range of each frame, and the problem of incorrect segmentation can be avoided to some extent.

[0180] According to an embodiment of the present disclosure, since the swing range of the dendritic spine in different neurons is different, for example, the swing range of the dendritic spine in an abnormal neuron can be larger, and the swing range of the dendritic spine in a normal neuron can be smaller, different tracking methods can be developed according to the dendritic spine with different swing ranges, so as to determine a target tracking method according to the swing range of the dendritic spine from a plurality of tracking methods, and determine the motion trajectory of the dendritic spine by using the tracking method, so as to track the dendritic spine in the neuron fluorescence microscopic image.

[0181] According to an embodiment of the present disclosure, the target tracking method at least includes one of the following: maximum projection tracking method, Kalman filter tracking method, and Bayesian tracking method.

[0182] According to an embodiment of the present disclosure, in the case where the target tracking method includes the maximum projection tracking method, the motion trajectory of the dendritic spine is determined by using the target tracking method according to the swing range of the dendritic spine in the plurality of dendritic spine regions, so as to track the dendritic spine in the neuron fluorescence microscopic image, comprising:

[0183] Based on the maximum swing range of the dendritic spine in the plurality of dendritic spine regions, a maximum projection result of the neuron fluorescence microscopic image in the time domain is determined;

[0184] The motion trajectory of the dendritic spine is determined according to the maximum projection result of the neuron fluorescence microscopic image in the time domain, so as to track the dendritic spine in the neuron fluorescence microscopic image.

[0185] According to an embodiment of the present disclosure, in the case of a small dendritic spine movement range, a maximum value projection tracking method can be used to determine the movement trajectory of the dendritic spine, the maximum swing range of each dendritic spine is obtained by maximum value projection of the image in the time domain, and a unique ID is assigned to the dendritic spine region. The dendritic spine region in each neuron fluorescence microscopic image in a plurality of neuron fluorescence microscopic images is labeled by using a dendritic spine labeling model, and the contour of each dendritic spine at different time points is obtained.

[0186] According to an embodiment of the present disclosure, in the case of a target tracking method including a Kalman filter tracking method or a Bayesian tracking method, the movement trajectory of the dendritic spine is determined by using the target tracking method, so as to track the dendritic spine in the neuron fluorescence microscopic image, comprising:

[0187] Based on the dendritic spine in the plurality of dendritic spine regions, the centroid position of the dendritic spine is determined;

[0188] According to the centroid position, the movement trajectory of the dendritic spine is determined, so as to track the dendritic spine in the neuron fluorescence microscopic image.

[0189] According to an embodiment of the present disclosure, when the dendritic spine movement range is large, the dendritic spines on different neuron fluorescence microscopic images will exist in space. The Kalman filter tracking method or the Bayesian tracking method can be used to determine the movement trajectory of the dendritic spine. Each neuron fluorescence microscopic image can be regarded as an independent image sample. After labeling the dendritic spine region, the centroid of the dendritic spine region is calculated as the position of the dendritic spine. Then, the Kalman filter tracking method or the Bayesian tracking method is used to construct a probability network using the spatial information, velocity information and appearance information of the object to link the trajectory, and consider the probability of target disappearance, new birth and false recognition. According to the position and area of the dendritic spine region and other information, the movement trajectory of the distribution point is calculated. According to the points distributed to the same trajectory, the same ID is assigned, so as to realize the tracking of the dendritic spine.

[0190] Figure 9B An example of matching the dendritic spine region in the neuron fluorescence microscopic image in the time domain is schematically shown.

[0191] As Figure 9B shown, the dendritic spines at different time points can be assigned the same ID to match the same dendritic spine region in a plurality of neuron fluorescence microscopic images, realize the tracking of the dendritic spine, and further realize the prediction of the disease according to the change of the dendritic spine.

[0192] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features recited in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not expressly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure may be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0193] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A dendritic spine annotation method of a neuron fluorescence microscopic image, comprising: determining a neuron region in the neuron fluorescence microscopic image according to pixel values of pixel points in the neuron fluorescence microscopic image, comprising: determining a mean value and a standard deviation of the pixel points in the neuron fluorescence microscopic image according to the pixel values; determining a first-level threshold and a second-level threshold based on the mean value and the standard deviation, wherein the first-level threshold is a sum of the mean value and three times the standard deviation, and the second-level threshold is a sum of the mean value and the standard deviation; determining an initial neuron region in the neuron fluorescence microscopic image based on the first-level threshold and the second-level threshold; and determining the neuron region in the initial neuron region by using an adaptive threshold; determining candidate peak points located in the neuron region from the pixel points based on local peak values of the pixel values, wherein the local peak values represent maximum pixel values of local regions in the neuron fluorescence microscopic image, and distances between adjacent local peak values are greater than a preset threshold; determining target peak points matching dendritic spines in the neuron fluorescence microscopic image according to pixel values of the candidate peak points, comprising: determining a partial derivative of the pixel values according to the pixel values of the candidate peak points; determining a structure tensor matrix of the candidate peak points according to the partial derivative; and determining target peak points matching dendritic spines in the neuron fluorescence microscopic image based on a plurality of first eigenvalues of the structure tensor matrix; annotating dendritic spine regions in the neuron fluorescence microscopic image based on the target peak points by using gradient change information of pixel points in a preset search box to obtain a target annotation image.

2. The method of claim 1, wherein, The determining target peak points matching dendritic spines in the neuron fluorescence microscopic image according to the pixel values of the candidate peak points further comprises: determining a second-order partial derivative of the pixel values according to the pixel values of the candidate peak points; determining a Hessian matrix of the candidate peak points based on the second-order partial derivative; performing absolute value processing on a plurality of eigenvalues of the Hessian matrix to obtain second eigenvalues; determining target peak points matching dendritic spines in the neuron fluorescence microscopic image based on the second eigenvalues.

3. The method of claim 1, wherein, The annotating dendritic spine regions in the neuron fluorescence microscopic image based on the target peak points by using a preset search box comprises: determining a plurality of pixel blocks in the neuron fluorescence microscopic image by using the preset search box with the target peak points as a center; for each pixel block of the plurality of pixel blocks, performing region growing based on the target peak points to obtain an intermediate region; annotating dendritic spine regions in the neuron fluorescence microscopic image according to a gray gradient direction of the intermediate region. 4.A dendritic spine tracking method of a neuron fluorescence microscopic image, comprising: acquiring neuron fluorescence microscopic images at a plurality of time points by continuous imaging; annotating, by using a pre-trained dendritic spine annotation model, dendritic spine regions in the neuron fluorescence microscopic images at each time point to obtain a plurality of dendritic spine regions, wherein the dendritic spine annotation model is obtained by training a target annotation image obtained by the dendritic spine annotation method of any one of claims 1-3 as a training sample; tracking, by using a target tracking method, a motion trajectory of the dendritic spine according to a swing range of the dendritic spine in the plurality of dendritic spine regions, so as to track the dendritic spine in the neuron fluorescence microscopic images.

5. The method of claim 4, wherein the training method of the dendritic spine annotation model comprises: obtaining training samples, wherein the training samples comprise target annotation image training sub-samples and neuron fluorescence microscopic image training sub-samples; training a deep learning model by using the target annotation image training sub-samples as labels and the neuron fluorescence microscopic image training sub-samples as inputs to obtain a dendritic spine annotation model.

6. The method of claim 4, wherein, The target tracking method comprises at least one of a maximum value projection tracking method, a Kalman filter tracking method, and a Bayesian tracking method.

7. The method of claim 6, wherein when the target tracking method comprises the maximum value projection tracking method, the tracking, by using a target tracking method, a motion trajectory of the dendritic spine according to a swing range of the dendritic spine in the plurality of dendritic spine regions, so as to track the dendritic spine in the neuron fluorescence microscopic images, comprises: determining a maximum value projection result of the neuron fluorescence microscopic images in a time domain based on a maximum swing range of the dendritic spine in the plurality of dendritic spine regions; tracking, by using the target tracking method, a motion trajectory of the dendritic spine according to the maximum value projection result of the neuron fluorescence microscopic images in the time domain, so as to track the dendritic spine in the neuron fluorescence microscopic images.

8. The method of claim 6, wherein when the target tracking method comprises the Kalman filter tracking method or the Bayesian tracking method, the tracking, by using a target tracking method, a motion trajectory of the dendritic spine according to a swing range of the dendritic spine in the plurality of dendritic spine regions, so as to track the dendritic spine in the neuron fluorescence microscopic images, comprises: determining a center of mass position of the dendritic spine based on the dendritic spine in the plurality of dendritic spine regions; tracking, by using the target tracking method, a motion trajectory of the dendritic spine according to the center of mass position, so as to track the dendritic spine in the neuron fluorescence microscopic images.

Images