Neural cell spectral detection method and system based on deep learning

Abstract

This invention provides a method and system for spectral detection of neural cells based on deep learning, belonging to the field of spectral analysis technology. The method includes: constructing and jointly training a spectral feature extraction network, a spatial context aggregation network, and an abundance prediction network; acquiring hyperspectral microscopic images of the neural cell sample to be detected and preprocessing them to obtain a hyperspectral data cube; inputting the hyperspectral data cube into the pre-trained spectral feature extraction network to output a deep spectral feature map for each pixel; inputting the deep spectral feature map into the pre-trained spatial context aggregation network to output a spatial-spectral joint feature map; and inputting the spatial-spectral joint feature map into the pre-trained abundance prediction network to output an abundance map of the neural cell sample to be detected, which serves as the abundance detection result for the neural cell sample. This invention solves the problem that traditional methods struggle to accurately separate and quantitatively analyze the abundance distribution of various subcellular components.

Description

Technical Field

[0001] This invention relates to the field of spectral analysis technology, specifically to a method and system for detecting neural cell spectra based on deep learning. Background Technology

[0002] Neuronal cell spectroscopy is based on hyperspectral imaging technology to acquire the spectral fingerprint information of nerve cells. By analyzing the differences in absorption or scattering at characteristic wavelengths, it enables label-free, non-invasive identification and quantitative detection of changes in cellular molecular composition and structure. Existing neuronal cell spectroscopy detection methods mainly rely on hyperspectral microscopy to acquire transmission or reflectance spectral data of cell samples, and use this as a basis to analyze the abundance of molecular composition and subcellular structures within the cell. This process typically involves three stages: First, using purely mathematical optimization methods such as linear unmixing models, independent component analysis, or nonnegative matrix factorization, endmember extraction and abundance inversion are performed on mixed pixels in the hyperspectral image; second, to compensate for the shortcomings of pure spectral information, some improved methods introduce spatial regularization constraints; finally, through comparison with a pre-set spectral library or statistical feature selection, qualitative identification and quantitative distribution detection of specific molecules in nerve cells are completed.

[0003] However, the aforementioned existing technologies still have significant drawbacks in practical applications: applying spatial information as an independent post-processing constraint to the spectral unmixing model results in insufficient fusion depth of spectral and spatial information; the guiding role of spatial structural information in spectral feature expression is limited, making it difficult to accurately capture the boundary and texture details of subcellular structures such as neural axons; the feature expression capability is highly dependent on the researcher's prior knowledge and parameter tuning, making it difficult to adaptively mine complex and deep nonlinear mixed features from high-dimensional spectral data cubes; for different neural cell subtypes with highly similar spectral features, such as the differentiation intermediates of neurons and glial cells, the ability to distinguish them is weak, i.e., the feature extraction capability is limited and overly reliant on manual presets; through multiple independent steps, small noise or estimation biases generated in the preceding steps will be amplified step by step in the subsequent process, and it is impossible to achieve fast end-to-end parallel reasoning, making it difficult to meet the needs of real-time label-free monitoring of the dynamic processes of living cells, i.e., multi-stage independent processing leads to error accumulation and low efficiency.

[0004] In summary, there is an urgent need for a new method that can overcome the limitations of the traditional methods, improve the accuracy, robustness, and real-time performance of neural cell spectral detection, and accurately separate and quantitatively analyze the abundance distribution of each subcellular component. Summary of the Invention

[0005] This application provides a method and system for the spectral detection of neural cells based on deep learning, which solves the problems of traditional methods such as single-dimensional information utilization, improper introduction of spatial features, fragmented multi-stage optimization, and difficulty in accurately separating and quantitatively analyzing the abundance distribution of each subcellular component.

[0006] In view of the above problems, this application provides a method and system for neural cell spectral detection based on deep learning.

[0007] Firstly, this application provides a deep learning-based neural cell spectral detection method, including:

[0008] A spectral feature extraction network, a spatial context aggregation network, and an abundance prediction network are constructed and jointly trained to obtain pre-trained spectral feature extraction network, pre-trained spatial context aggregation network, and pre-trained abundance prediction network.

[0009] A hyperspectral microscopic image of the nerve cell sample to be detected is acquired, and the hyperspectral microscopic image is preprocessed to obtain a normalized hyperspectral data cube.

[0010] The hyperspectral data cube is input into the pre-trained spectral feature extraction network, which outputs a deep spectral feature map for each pixel.

[0011] The deep spectral feature map is input into the pre-trained spatial context aggregation network, and the spatial-spectral joint feature map is output by aggregating the spectral features in the neighborhood of each pixel.

[0012] The spatial-spectral joint feature map is input into the pre-trained abundance prediction network, and the abundance map of the neural cell sample to be detected is output as the abundance detection result of the neural cell sample to be detected.

[0013] Secondly, this application provides a deep learning-based neural cell spectral detection system, including:

[0014] Joint pre-training module: Construct and jointly train the spectral feature extraction network, spatial context aggregation network, and abundance prediction network to obtain pre-trained spectral feature extraction network, pre-trained spatial context aggregation network, and pre-trained abundance prediction network;

[0015] Hyperspectral image acquisition and preprocessing module: acquires hyperspectral microscopic images of the nerve cell sample to be detected, preprocesses the hyperspectral microscopic images to obtain a normalized hyperspectral data cube;

[0016] Spectral feature extraction module: Inputs the hyperspectral data cube into the pre-trained spectral feature extraction network and outputs a deep spectral feature map for each pixel;

[0017] Spatial context aggregation module: Inputs the deep spectral feature map into the pre-trained spatial context aggregation network, and outputs a spatial-spectral joint feature map by aggregating the spectral features in the neighborhood of each pixel;

[0018] Abundance prediction output module: Input the spatial-spectral joint feature map into the pre-trained abundance prediction network, and output the abundance map of the neural cell sample to be detected as the abundance detection result of the neural cell sample to be detected.

[0019] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0020] The technical solution of this application, firstly, utilizes a multi-scale spectral feature extraction network to adaptively capture cross-scale spectral features such as broad peaks of lipid CH and narrow peaks of protein amide I using parallel one-dimensional convolutional branches, significantly improving the ability to distinguish nerve cell subtypes with similar spectral features but different biochemical properties; secondly, a spatial context aggregation network matches clustered and linear cell morphologies respectively through dilated convolution of the cell body branch and strip convolution of the axon branch, and achieves pixel-level dynamic fusion through adaptive spatial attention gates, effectively solving the boundary blurring and structural breakage problems caused by the fragmentation of spectral-spatial information in traditional methods, and the output spatial-spectral joint feature map has both cell body internal uniformity, axonal direction continuity and background noise suppression capabilities; finally, an abundance prediction network embeds a transformation matrix encoding the co-occurrence and mutual exclusion relationships of components in the prior relation constraint layer, so that the abundance prediction results strictly conform to the real biological composition logic of nerve cells, and at the same time, the joint loss function introduces component adaptive spatial regularization and endmember spectral smoothing regularization, further ensuring the spatial consistency of the abundance map and the physical interpretability of the endmember curves.

[0021] In summary, the technical solution of this application enables the output of subcellular abundance maps with high numerical accuracy, intact spatial structure, and smooth and realistic endmember spectra, requiring only a small number of labeled samples. Moreover, end-to-end inference only requires a single forward propagation to complete the mapping from the original hyperspectral data cube to the abundance map, balancing detection accuracy, computational efficiency, and biological rationality. This meets the application requirements for label-free, non-invasive, and real-time quantitative molecular imaging of nerve cells, thereby solving the problem that traditional methods are difficult to accurately separate and quantitatively analyze the abundance distribution of each subcellular component. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating the deep learning-based neural cell spectral detection method provided in the embodiments of this application.

[0023] Figure 2 This is a schematic diagram of the structure of the deep learning-based neural cell spectral detection system provided in the embodiments of this application.

[0024] The components represented by each number in the attached diagram are explained below:

[0025] The module includes a joint pre-training module 11, a hyperspectral image acquisition and preprocessing module 12, a spectral feature extraction module 13, a spatial context aggregation module 14, and an abundance prediction output module 15. Detailed Implementation

[0026] This application provides a method and system for neural cell spectral detection based on deep learning, which solves the problems of traditional methods such as single-dimensional information utilization, improper introduction of spatial features, and fragmented multi-stage optimization.

[0027] It should be noted that the terms "comprising" and "having" are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or modules that are not explicitly listed or that are inherent to these processes, methods, products, or devices.

[0028] Example 1, as Figure 1 As shown in the embodiments of this application, a neural cell spectral detection method based on deep learning is provided, the method comprising:

[0029] S100: Construct and jointly train the spectral feature extraction network, the spatial context aggregation network, and the abundance prediction network to obtain the pre-trained spectral feature extraction network, the pre-trained spatial context aggregation network, and the pre-trained abundance prediction network.

[0030] Step S100 in the method provided in this application embodiment includes:

[0031] Construct a multi-scale spectral feature extraction network, a spatial context aggregation network, and an abundance prediction network;

[0032] Using the reconstruction of hyperspectral data cubes as a self-supervised task, the multi-scale spectral feature extraction network and the spatial context aggregation network are pre-trained using unlabeled hyperspectral microscopic image samples to obtain an initialized multi-scale spectral feature extraction network and an initialized spatial context aggregation network.

[0033] The abundance prediction network is trained in a supervised manner using labeled hyperspectral microscopic image samples to obtain an initialized abundance prediction network. The labeled hyperspectral microscopic image samples are labeled with real abundance maps, and the real abundance maps contain the real abundance vector corresponding to each pixel.

[0034] Construct a joint loss function, which includes an abundance prediction error term, a spatial continuity regularization term, and a spectral smoothness regularization term;

[0035] The initialized multi-scale spectral feature extraction network, the initialized spatial context aggregation network, and the initialized abundance prediction network are treated as a whole and subjected to end-to-end supervised joint training using the joint loss function until the joint loss function converges, resulting in pre-trained spectral feature extraction network, pre-trained spatial context aggregation network, and pre-trained abundance prediction network.

[0036] In this embodiment, a multi-scale spectral feature extraction network and a spatial context aggregation network are constructed, including:

[0037] Multiple parallel convolutional branches are set up, each containing a one-dimensional convolutional layer. Different convolutional kernel sizes are assigned to the one-dimensional convolutional layers of different branches, wherein the convolutional kernel sizes correspond to the typical widths of different characteristic spectral peaks of nerve cells.

[0038] The outputs of all convolutional branches are concatenated along the channel dimension to obtain the concatenated multi-scale feature vector;

[0039] A 1×1 convolutional layer is set after the spliced ​​multi-scale feature vector to obtain a multi-scale spectral feature extraction network.

[0040] A dual-branch spatial context aggregation network is constructed, which includes a cell body processing branch and an axon processing branch;

[0041] One or more two-dimensional convolutional layers with small convolutional kernels are set in the cell body processing branch, and dilated convolution is introduced in the two-dimensional convolutional layers with small convolutional kernels to extract the clustered spatial features of neuronal cell bodies and output the cell body spatial feature map.

[0042] A two-dimensional convolutional layer with strip convolutional kernels is set in the axon processing branch. The strip convolutional kernels include horizontal strip convolutional kernels and vertical strip convolutional kernels. The output feature maps of the horizontal strip convolutional kernels and the vertical strip convolutional kernels are added together to obtain the axon spatial feature map.

[0043] An adaptive spatial attention gate is set in the dual-branch spatial context aggregation network. The adaptive spatial attention gate receives the cell body spatial feature map and the axon spatial feature map, calculates the fusion weight at each spatial location through a 1×1 convolutional layer, and weights and sums the cell body spatial feature map and the axon spatial feature map according to the fusion weight to obtain the spatial-spectral joint feature map.

[0044] Specifically, firstly, based on the characteristic that nerve cells have different features and their spectral peaks have different typical widths, multiple parallel convolutional branches are set up to extract the spectral feature peak widths of different nerve cells. Each convolutional branch contains a one-dimensional convolutional layer and is assigned different convolutional kernel sizes to correspond to the typical widths of the spectral feature peaks of different nerve cells.

[0045] For example, to distinguish the spectral peak widths of neurons from glial cells, since neuronal cell membranes are rich in cholesterol, the typical characteristic peak is 2930 cm⁻¹. −1 The mitochondria of glial cells are rich in cytochrome c, therefore their typical characteristic peak is 750 cm⁻¹. −1 Two parallel convolutional branches are set up, branch A and branch B. The one-dimensional convolutional layer in branch A has a kernel size of 31, covering a region of 2900 cm⁻¹. −1 -2950cm −1 This corresponds to the typical characteristic peaks of neuronal cells; the kernel size allocated in the one-dimensional convolutional layer of branch B is 5, covering a range of 720cm. −1 -780cm −1 This corresponds to the typical characteristic peaks of glial cells.

[0046] Furthermore, the output channel dimensions of all the above convolutional branches are concatenated, that is, the feature vectors output by each convolutional branch are concatenated to obtain a concatenated multi-scale feature vector. A 1×1 convolutional layer is then placed after the concatenated multi-scale feature vector. Without changing the spectral length, the multi-scale features are weighted and summed, and then filtered to finally output a multi-scale fused spectral feature map. Through the above process, a multi-scale spectral feature extraction network is obtained.

[0047] For example, continuing with the previous example, in addition to branches A and B, there is also branch C, where the kernel size in the one-dimensional convolutional layer is 15, corresponding to the typical characteristic peak of protein secondary structure such as amide I band at 1650. −1After convolution, three feature vectors with 32 channels are output. The outputs of all convolutional branches are concatenated along the channel dimension to obtain a concatenated multi-scale feature vector with 32+32+32=96 channels. This concatenated multi-scale feature vector is then input into a 1×1 convolutional layer to compress the number of channels to 32. The above process reveals that when channel 2 of branch A and channel 28 of branch B appear simultaneously, channel 10 of branch C is activated. This means that the feature vector extracted by branch C may be a linear combination of the feature vectors extracted by branches A and B. Therefore, the 1×1 convolutional layer retains the core features of branches A and B, while reducing the weight of channel 10 of branch C to 0. Through weighted calculation, when the input spectral length is 3000 bands, the output result is a 32×3000 matrix, representing the trend curves of 32 composite spectral features as a function of wavelength. This yields a multi-scale fused spectral feature map.

[0048] Furthermore, a dual-branch spatial context aggregation network consisting of a cell body processing branch and an axon processing branch is constructed. Since the neuron cell body is approximately circular, the cell body processing branch is composed of stacked two-dimensional convolutional layers with one or more small convolutional kernels. This achieves stronger nonlinear expression with fewer parameters. Dilated convolutions are introduced into the two-dimensional convolutional layers to exponentially expand the receptive field without increasing the number of parameters or computational cost. The cell body spatial feature map is output after dilated convolution processing. The neuron axon is long and tubular with variable orientation; therefore, the axon processing branch consists of horizontal and vertical strip convolutional kernels. The output axon spatial feature map is obtained by adding the feature maps output by the horizontal and vertical strip convolutional kernels.

[0049] For example, the cell body processing branch sets up two cascaded 3×3 small convolutional kernels and performs dilated convolution with a dilation rate of 2. Then, the 3×3 convolutional kernel is equivalent to a 7×7 receptive field, which can identify complete cell body clusters and output a cell body spatial feature map. The axon processing branch uses a 1×7 horizontal strip convolutional kernel and a vertical strip convolutional kernel. When the axon is identified in the horizontal direction, all 7 sampling points in the window are located in the high reflectivity axon region, effectively shielding the interference of blank background. The feature maps output by the horizontal strip convolutional kernel and the vertical strip convolutional kernel are added together to output an axon spatial feature map.

[0050] Furthermore, an adaptive spatial attention gate is set in the obtained dual-branch spatial context aggregation network to receive the obtained cell body spatial feature map and axon spatial feature map. Since there are many morphologically blurred or transitional regions in the neuronal microscopic images, fixed weight fusion cannot process them, so an adaptive spatial attention gate is used. After concatenating the cell body spatial feature map and the axon spatial feature map, a 1×1 convolutional layer is set to output two spatial weight maps. Then, the weight values ​​are normalized by the Softmax function, that is, the value range of the weight values ​​is reduced to [0, 1] to obtain the weights of the cell body spatial features and the axon spatial features. After weighted summation calculation by the adaptive spatial attention gate, the spatial-spectral joint feature map is obtained.

[0051] For example, the network processes information about the axon hill region of a neuron, which includes both the clustering characteristics of the cytoplasm and the extension characteristics of the axon. After processing by a dual-branch spatial context aggregation network, the cell body spatial feature map and the axon spatial feature map are obtained. Then, they are concatenated along the channel dimension and fed into a 1×1 convolutional layer. The convolutional layer automatically calculates a set of normalized fusion weights by analyzing the distribution of joint features in the neighborhood of the pixel. For example, it is determined that although the position has certain cell body clustering characteristics, the weight is 0.42, but because the longitudinal arrangement of microtubules extending in a single direction is detected, the axon branch is given a higher response, and the weight is 0.58. The adaptive spatial attention gate performs pixel-by-pixel weighted summation on the two branch feature maps according to these weights. The final output is a spatial-spectral joint feature map that presents a natural and smooth transition from cell body features to axon features in the axon hill region.

[0052] It should be noted that the above values ​​are for illustrative purposes only and do not constitute a limitation on the present invention.

[0053] In this embodiment, an abundance prediction network is constructed, including:

[0054] Construct a transformation matrix whose dimension is equal to the number of types of nerve cell components. Initialize the diagonal elements of the transformation matrix to 1, and initialize the off-diagonal elements of the transformation matrix according to the symbiotic or mutually exclusive relationship between nerve cell components: for two components with a symbiotic relationship, initialize the corresponding off-diagonal elements to positive numbers; for two components with a mutually exclusive relationship, initialize the corresponding off-diagonal elements to negative numbers.

[0055] An abundance prediction network is constructed, comprising a fully connected layer, a prior relation constraint layer, a softmax activation layer, and an abundance map reconstruction layer connected in sequence. The prior relation constraint layer contains the transformation matrix, the softmax activation layer is used to output the abundance vector of each pixel, and the abundance map reconstruction layer is used to arrange the abundance vectors of all pixels according to their pixel positions and output the abundance map.

[0056] Specifically, first, determine the number K of nerve cell components. For example, if the component types are set to four, then the number of component types K=4. Construct a transformation matrix of dimension K×K, where all diagonal elements in the matrix are initialized to 1, indicating that the abundance response benchmark of each component is unity gain. Based on prior knowledge of neuronal biology, identify component pairs with symbiotic relationships: initialize the off-diagonal elements corresponding to two components with symbiotic relationships to positive numbers, and initialize the off-diagonal elements corresponding to two components with mutually exclusive relationships to negative numbers.

[0057] For example, firstly, four types of nerve cell components are predefined: cell nucleus, cytoplasmic proteins, lipid myelin sheath, and mitochondria. A 4×4 transformation matrix is ​​constructed, and all diagonal elements are initialized to 1 to maintain the baseline response of the abundance of each component. Subsequently, based on the symbiotic relationship of mitochondria being enriched in the metabolically active cytoplasmic region, the off-diagonal elements corresponding to cytoplasmic proteins and mitochondria are initialized to +0.3. At the same time, based on the exclusive mutual exclusion relationship of the absence of lipid myelin sheath in the cell nucleus, the off-diagonal elements corresponding to the cell nucleus and lipid myelin sheath are initialized to -0.5. In this way, the biochemical coexistence and spatial exclusion rules of nerve cell components are explicitly encoded in the matrix.

[0058] Furthermore, a complete abundance prediction network is constructed, comprising four sub-layers: a fully connected layer, a priori constraint layer, a Softmax activation layer, and an abundance map reconstruction layer, which are connected sequentially. Specifically, the fully connected layer receives the spatial-spectral joint feature map and maps the high-dimensional feature vector of each pixel to the original abundance logical value; the transformation matrix obtained above is used as the priori constraint layer, multiplying the original abundance logical value with the transformation matrix to obtain a constraint vector with priori correction; the Softmax activation layer normalizes each component of the vector, mapping it to the interval [0, 1], ensuring that the sum of all components is 1, and outputs the normalized abundance vector for each pixel; the abundance vectors of all pixel positions are rearranged according to the spatial row and column coordinates of the original image, outputting an abundance map with the same spatial size as the input hyperspectral image.

[0059] For example, continuing with the previous example, when the network predicts the abundance of a certain pixel X, if the fully connected layer initially determines that pixel X has both high cytoplasmic protein signals and mitochondrial signals, the positive weights in the transformation matrix will further enhance the abundance response of the two components. However, when the network erroneously detects lipid myelin signals in the cell nucleus region, the negative weights in the transformation matrix will actively suppress the abundance value of the mutually exclusive component, thereby ensuring that the abundance vector output by the Softmax activation layer and the abundance map generated by the abundance map recombination layer strictly conform to the real biological composition logic of nerve cells in terms of spatial distribution.

[0060] It should be noted that the above values ​​are for illustrative purposes only and do not constitute a limitation on the present invention.

[0061] In this embodiment, the reconstruction of a hyperspectral data cube is taken as a self-supervised task. Unlabeled hyperspectral microscopic image samples are used to pre-train the multi-scale spectral feature extraction network and the spatial context aggregation network, resulting in an initialized multi-scale spectral feature extraction network and an initialized spatial context aggregation network, including:

[0062] Multiple unlabeled hyperspectral microscopic image samples were acquired. Each unlabeled hyperspectral microscopic image sample was preprocessed by denoising and normalization to obtain a normalized hyperspectral data cube corresponding to each unlabeled hyperspectral microscopic image sample.

[0063] Construct a decoder network and connect the decoder network to the spatial context aggregation network;

[0064] The normalized hyperspectral data cube is sequentially input into the multi-scale spectral feature extraction network and the spatial context aggregation network to obtain a spatial-spectral joint feature map.

[0065] The spatial-spectral joint feature map is input into the decoder network, and a reconstructed hyperspectral data cube is output.

[0066] Calculate the mean square error between the reconstructed hyperspectral data cube and the normalized hyperspectral data cube, and use the mean square error as a self-supervised loss function.

[0067] Based on the self-supervised loss function, the parameters of the multi-scale spectral feature extraction network, the spatial context aggregation network, and the decoder network are iteratively optimized using the backpropagation algorithm until the self-supervised loss function converges, resulting in the initialized multi-scale spectral feature extraction network and the initialized spatial context aggregation network.

[0068] Specifically, firstly, multiple unlabeled hyperspectral microscopic images of neural cells were collected. Each image was then preprocessed sequentially. The preprocessing operations included: suppressing random thermal noise and shot noise introduced by the detector using minimum noise separation transform or wavelet thresholding; normalizing the spectral reflectance values ​​of each band to their maximum or minimum values, mapping the spectral values ​​of all pixels to the [0, 1] interval, and eliminating intensity differences caused by uneven illumination during imaging. Finally, a normalized hyperspectral data cube corresponding to each unlabeled sample was obtained.

[0069] Furthermore, a decoder network symmetrical to the encoder structure is constructed. The decoder network typically consists of several deconvolutional layers, whose function is to recover the low-dimensional spatial-spectral joint feature map into a hyperspectral data cube with the same size as the original input. The decoder network is then concatenated with a spatial context aggregation network to form a complete encoder-decoder architecture.

[0070] Furthermore, the normalized hyperspectral data cube obtained after the above preprocessing is input into the multi-scale spectral feature extraction network and the spatial context aggregation network to obtain a spatial-spectral joint feature map. The obtained spatial-spectral joint feature map is then input into the decoder network to output the reconstructed hyperspectral data cube.

[0071] Furthermore, the mean square error between the reconstructed hyperspectral data cube and the original normalized hyperspectral data cube is calculated to measure the fidelity of the network's overall compression and restoration of the original hyperspectral data. The obtained mean square error result is used as a self-supervised loss function.

[0072] Finally, based on the self-supervised loss function obtained above, the parameters of the multi-scale spectral feature extraction network, spatial context aggregation network, and decoder network are iteratively optimized using the backpropagation algorithm until the self-supervised loss function converges, thus obtaining the initialized multi-scale spectral feature extraction network and the initialized spatial context aggregation network.

[0073] For example, during self-supervised pre-training of the multi-scale spectral feature extraction network and the spatial context aggregation network, 1000 unlabeled hyperspectral microscopic images of neural cells are first acquired. After noise reduction and band normalization, 1000 normalized hyperspectral data cubes of size 256×256×150 are obtained. Subsequently, a decoder network consisting of three deconvolution layers is temporarily attached after the spatial context aggregation network to progressively upsample and reconstruct the spatial-spectral joint feature map output from the encoder into 1000 reconstructed data cubes of size 256×256×150 with the same size as the input. The mean square error between the reconstructed image and the original image is calculated. For example, when a pixel in the original hyperspectral data is at 2930 cm⁻¹, the mean square error is calculated. −1When a band exhibits low reflectance due to myelin lipid absorption, a large squared error will occur if the reconstructed data output by the decoder incorrectly predicts high reflectance at that position in that band. This error signal is then propagated back layer by layer through backpropagation. The multi-scale spectral feature extraction network learns to extract the most discriminative absorption peak features from the spectral curve, while the spatial context aggregation network learns how to utilize the cluster continuity of cell bodies and the linear extension of axons to assist in reconstructing bands contaminated by noise or undersampled. After 300-500 iterations, the mean square error decreases from 0.15 to 0.0023 and tends to stabilize at 0.0023, indicating that the mean square error has converged. The decoder network is then removed, while the initialized multi-scale spectral feature extraction network and spatial context aggregation network are retained.

[0074] It should be noted that the above values ​​are for illustrative purposes only and do not constitute a limitation on the present invention.

[0075] In this embodiment, the joint loss function is constructed, including:

[0076] The mean square error between the abundance map output by the abundance prediction network and the actual abundance map is used as the abundance prediction error term.

[0077] The total variation loss of the abundance map is used as a spatial continuity regularization term, and different weights are assigned to the abundance channels corresponding to different neural cell components in the abundance map, wherein the channel weights corresponding to axonal components are higher than those corresponding to cell body components.

[0078] A learnable endmember spectral matrix is ​​constructed, and the second difference norm of the adjacent bands of the endmember spectral matrix is ​​used as the spectral smoothness regularization term. The dimension of the endmember spectral matrix is ​​equal to the number of types of neural cell components multiplied by the number of bands of the hyperspectral image.

[0079] The abundance prediction error term, the spatial continuity regularization term, and the spectral smoothness regularization term are added together to obtain the joint loss function.

[0080] Specifically, the mean square error between the abundance map output by the abundance prediction network and the actual abundance map is first calculated pixel by pixel and channel by channel, and used as the abundance prediction error term.

[0081] Furthermore, the total variation loss of the abundance map output by the abundance prediction network is calculated, where the total variation loss is defined as the mean of the sum of the absolute values ​​of the gradients of the component abundance channels in the horizontal and vertical directions. Different channel weights are assigned based on the morphological priors of different neural cell components. Axonal components, due to their elongated and continuous linear distribution, are extremely sensitive to spatial breaks and should be assigned higher weights, while cell body components, due to their clustered and discrete distribution with natural intervals between adjacent cell bodies, should be assigned lower weights. The sum of the weighted total variation losses of each channel is used as a spatial continuity regularization term.

[0082] Furthermore, a learnable endmember spectral matrix is ​​constructed, the dimension of which consists of the number of component species and the number of bands in the hyperspectral image. The second difference norm of the endmember spectral matrix along the spectral dimension, i.e. the column direction of the matrix, is calculated as a spectral smoothness regularization term.

[0083] Furthermore, the abundance prediction error term, spatial continuity regularization term, and spectral smoothness regularization term are weighted and summed to obtain the joint loss function. The spatial continuity regularization term and the spectral smoothness regularization term are each assigned a hyperparameter that balances the contribution of each loss term as a weight; those skilled in the art can set these parameters based on experience.

[0084] For example, firstly, the pixel-wise mean square error between the predicted abundance map output by the abundance prediction network and the true abundance map is calculated in the current training iteration, and the abundance prediction error term is 0.028.

[0085] Furthermore, the total variation loss of each component channel in the abundance map output by the abundance prediction network was calculated. The total variation loss of the nuclear channel was 0.5, with a weight of 0.0012; the total variation loss of the cytoplasmic protein channel was 0.5, with a weight of 0.0015; the total variation loss of the lipid myelin channel was 2.0, with a weight of 0.0008; and the total variation loss of the mitochondrial channel was 0.8, with a weight of 0.0010. The weighted summation yielded a spatial continuity regularization term of 0.00375, which is abbreviated as 0.003.

[0086] Furthermore, the second-order difference norm of adjacent bands is calculated along the spectral dimension of the learnable endmember spectral matrix. The squared values ​​of the second-order difference of the nuclear endmember curve, cytoplasmic protein endmember curve, cytoplasmic protein endmember curve, and lipid myelin endmember curve are 0.00012, 0.00009, 0.00015, and 0.00011, respectively. The total loss of the spectral smoothness regularization term is 0.001, which is obtained by summing the four terms and averaging them.

[0087] Furthermore, substituting the above results into the joint loss function calculation formula, and setting both the spatial regularization weight coefficient and the spectral regularization weight coefficient to 0.1, the joint loss function is obtained as 0.028 + 0.1 × 0.00375 + 0.1 × 0.001 ≈ 0.0285.

[0088] It should be noted that the above values ​​are for illustrative purposes only and do not constitute a limitation on the present invention.

[0089] In this embodiment, the initialized multi-scale spectral feature extraction network, the initialized spatial context aggregation network, and the initialized abundance prediction network are treated as a whole, and end-to-end supervised joint training is performed using the joint loss function until the joint loss function converges, resulting in a pre-trained spectral feature extraction network, a pre-trained spatial context aggregation network, and a pre-trained abundance prediction network, including:

[0090] The hyperspectral data cube of the labeled hyperspectral microscopic image sample is sequentially input into the initialized multi-scale spectral feature extraction network, the initialized spatial context aggregation network, and the initialized abundance prediction network. The abundance prediction error term between the abundance map output by the abundance prediction network and the true abundance map is calculated.

[0091] The spatial continuity regularization term is calculated based on the abundance map output by the abundance prediction network, and the second difference norm of the adjacent bands of the endmember spectral matrix is ​​calculated as the spectral smoothness regularization term.

[0092] The value of the joint loss function is obtained by adding the abundance prediction error term, the spatial continuity regularization term, and the spectral smoothness regularization term.

[0093] The gradient of the joint loss function with respect to all trainable parameters in the initialized multi-scale spectral feature extraction network, the initialized spatial context aggregation network, and the initialized abundance prediction network is calculated simultaneously using the backpropagation algorithm.

[0094] A gradient descent optimizer is used to synchronously update all trainable parameters of the initialized multi-scale spectral feature extraction network, the initialized spatial context aggregation network, and the initialized abundance prediction network according to the gradient.

[0095] Repeat the forward propagation, loss calculation, backpropagation, and parameter update steps until the joint loss function converges to obtain the pre-trained spectral feature extraction network, the pre-trained spatial context aggregation network, and the pre-trained abundance prediction network.

[0096] Specifically, firstly, the hyperspectral data cubes of labeled hyperspectral microscopic image samples are sequentially input into an initialized multi-scale spectral feature extraction network, an initialized spatial context aggregation network, and an initialized abundance prediction network. The abundance prediction error term between the abundance map output by the abundance prediction network and the true abundance map is calculated. Further, a spatial continuity regularization term is calculated based on the abundance map output by the abundance prediction network, and the second-order difference norm of adjacent bands of the endmember spectral matrix is ​​calculated as a spectral smoothness regularization term.

[0097] Furthermore, after obtaining the abundance prediction error term, spatial continuity regularization term, and spectral smoothness regularization term, the value of the joint loss function is calculated, and the value of the joint loss function is backpropagated layer by layer along the concatenated network through the backpropagation algorithm. At the same time, the partial derivatives of the joint loss function with respect to each trainable parameter in the multi-scale spectral feature extraction network, spatial context aggregation network, and abundance prediction network are calculated.

[0098] Furthermore, a gradient descent optimizer (such as Adam or SGD) is used to synchronously update the weights and bias parameters of all layers in the initialized multi-scale spectral feature extraction network, the initialized spatial context aggregation network, and the initialized abundance prediction network, based on the partial derivatives calculated in the above steps. The update magnitude is controlled by the learning rate hyperparameter set by the optimizer.

[0099] Furthermore, all the above forward propagation, loss calculation, backpropagation and parameter update steps are repeated until the joint loss function converges, thereby obtaining the pre-trained spectral feature extraction network, the pre-trained spatial context aggregation network and the pre-trained abundance prediction network.

[0100] For example, in a certain training iteration, 1000 labeled hyperspectral image patches of size 64×64×150 are input into the network; then, the abundance prediction error term is calculated as 0.031, the spatial continuity regularization term as 0.0042, and the joint loss function is calculated to be 0.0315; during backpropagation, the gradient signal of this loss value simultaneously flows to the prior relation constraint layer and fully connected layer of the abundance prediction network, the adaptive attention gate and dual-branch convolutional layer of the spatial context aggregation network, and each convolutional branch and 1×1 convolutional fusion layer of the multi-scale spectral feature extraction network. The optimizer performs a synchronous update of all parameters accordingly: for example, the weights of the large convolutional kernel branch in the multi-scale spectral feature extraction network responsible for capturing the broad peak features of lipids are fine-tuned to make them more sensitive to the 2930cm peak. −1The response in the spectral range near 1 is sharper; at the same time, the weights of the strip convolution kernels in the axon processing branch of the spatial context aggregation network are also optimized, making them more sensitive to the orientation of the linear myelin sheath structure; after about 150 iterations, the joint loss on the validation set decreased from the initial 0.12 to 0.021 and remained unchanged for 10 consecutive iterations, indicating that the training has converged. At this point, the pre-trained multi-scale spectral feature extraction network, the pre-trained spatial context aggregation network, and the pre-trained abundance prediction network are the pre-trained models that can be used for actual hyperspectral abundance detection of neural cells.

[0101] It should be noted that the above values ​​are for illustrative purposes only and do not constitute a limitation on the present invention.

[0102] In step S100, firstly, a multi-scale spectral feature extraction network, a spatial context aggregation network, and an abundance prediction network are constructed. The multi-scale spectral feature extraction network adaptively captures cross-scale spectral features from hyperspectral data, the spatial context aggregation network fuses cell body clusters and axonal linear structures with dual-branch morphological constraints, and the abundance prediction network outputs the distribution of subcellular components under prior relation constraints. Together, they achieve label-free, high-precision abundance unmixing and spatial mapping of the molecular composition of nerve cells.

[0103] Furthermore, using the reconstruction of hyperspectral data cubes as a self-supervised task, the multi-scale spectral feature extraction network and spatial context aggregation network were pre-trained using unlabeled hyperspectral microscopic image samples. This enabled the network to learn the ability to recover the complete spatial-spectral structure from noise and redundant bands, thereby obtaining network initialization weights with spectral characterization and morphological perception capabilities. This provides a parameter starting point that is far superior to random initialization for the subsequent abundance prediction task with a small number of labeled samples.

[0104] Furthermore, the labeled hyperspectral microscopic image samples are used as labeled hyperspectral microscopic image samples to conduct supervised training on the abundance prediction network, resulting in an initialized abundance prediction network. This allows the network to learn the nonlinear mapping relationship from spatial-spectral joint features to subcellular component abundance vectors, and obtain the initial weights of the transformation matrix containing prior knowledge of the coexistence and mutual exclusion of neural cell components. This provides a parameter starting point with basic abundance unmixing capabilities for subsequent end-to-end joint fine-tuning.

[0105] Furthermore, a joint loss function is constructed. By treating the initialized multi-scale spectral feature extraction network, the initialized spatial context aggregation network, and the initialized abundance prediction network as a whole, end-to-end supervised joint training is performed using the joint loss function until the joint loss function converges. This yields pre-trained spectral feature extraction network, pre-trained spatial context aggregation network, and pre-trained abundance prediction network. This allows spectral feature extraction, spatial morphology aggregation, and abundance prior constraints to be globally optimized under a unified objective, ultimately resulting in a pre-trained model that converges simultaneously in three dimensions: prediction accuracy, structural continuity, and spectral interpretability.

[0106] In summary, this step endows the multi-scale spectral feature extraction and spatial context aggregation network with robust spectral-spatial joint representation capabilities through self-supervised pre-training. Then, driven by a joint loss function that integrates abundance prediction error, spatial continuity regularization, and spectral smoothness regularization, end-to-end supervised collaborative fine-tuning is performed. Ultimately, with only a small number of labeled samples, high-precision abundance unmixing and structure-preserving spatial mapping of subcellular components of nerve cells are achieved. The output abundance map combines numerical accuracy, axonal continuity, and clear cell body boundaries, and the end-to-end inference efficiency meets the requirements of real-time label-free detection.

[0107] S200: Acquire a hyperspectral microscopic image of the nerve cell sample to be detected, and preprocess the hyperspectral microscopic image to obtain a normalized hyperspectral data cube.

[0108] Step S200 in the method provided in this application embodiment includes:

[0109] Acquire a hyperspectral microscopic image of the nerve cell sample to be tested, and preprocess the hyperspectral microscopic image to obtain a normalized hyperspectral data cube, including:

[0110] A hyperspectral microscopic image of a nerve cell sample to be tested is acquired. Based on the position of the characteristic absorption peaks of nerve cell components, a preset number of wavelength bands with the highest information content are selected from the hyperspectral microscopic image. The characteristic absorption peaks include at least the CH stretching vibration peak of lipids, the amide I peak of proteins, and the phosphodiester bond peak of nucleic acids.

[0111] The spectral data corresponding to the selected bands are used to construct a dimension-reduced hyperspectral data cube.

[0112] For each pixel spectrum in the dimension-reduced hyperspectral data cube, extended multivariate scattering correction is used to remove scattering baseline drift, resulting in a scattering-corrected hyperspectral data cube.

[0113] The spectrum of each pixel in the scattering-corrected hyperspectral data cube is normalized to obtain a normalized hyperspectral data cube.

[0114] In this embodiment, for each pixel spectrum in the dimensionality-reduced hyperspectral data cube, extended multivariate scattering correction is used to remove scattering baseline drift, resulting in a scattering-corrected hyperspectral data cube, including:

[0115] Calculate the arithmetic mean of the spectra of all pixels in the dimensionality-reduced hyperspectral data cube as the average spectrum;

[0116] For each pixel spectrum, the scattering correction coefficient and baseline offset are obtained by fitting the linear relationship between the pixel spectrum and the average spectrum using the least squares method.

[0117] The corrected pixel spectrum is obtained by subtracting the corresponding baseline offset from the spectrum of each pixel and dividing by the corresponding scattering correction coefficient. All the corrected pixel spectra are then used to form a scattering-corrected hyperspectral data cube.

[0118] Specifically, firstly, the spectral curves of all pixels in the reduced-dimensional hyperspectral data cube are arithmetically averaged, that is, the average reflectance of all pixels in the corresponding band, to obtain the average spectral vector, which is an average spectrum representing the overall sample spectral response trend.

[0119] Furthermore, for the spectral vector of each pixel, a linear regression model between the spectral vector and the average spectrum is established using the least squares method. Here, a scattering correction coefficient is set, representing the multiplicative scaling of the pixel's spectrum relative to the average spectrum; a baseline offset represents the additive baseline drift of the pixel's spectrum relative to the average spectrum; and a residual vector is used to represent the fitting error. The least squares method obtains the optimal solution for the scattering correction coefficient and the baseline offset by minimizing the sum of squared residuals.

[0120] Furthermore, the corresponding baseline offset is subtracted from the spectrum of each pixel and then divided by the corresponding scattering correction coefficient to obtain the corrected pixel spectrum. All the corrected spectral vectors are rearranged according to their original spatial positions to form a scattering-corrected hyperspectral data cube with the same size as the input.

[0121] For example, when performing extended multivariate scattering correction on the dimensionality-reduced hyperspectral data cube of neural cells, all pixels in the image are first traversed, and the reflectance values ​​at each band are accumulated and averaged to obtain a smooth average spectral curve, which reflects the overall performance of the sample at 750 cm⁻¹. −1 The depression at the cytochrome absorption peak, at 1450 cm⁻¹ −1 The slight fluctuations at the water peak and at 2930cm −1The characteristic trend of the bulge at the broad lipid peaks was observed. Subsequently, for a pixel located at the edge of an axon in the image, due to severe local scattering, its original spectrum showed a high overall reflectance and a significant baseline rise. A least-squares method was used to linearly fit the pixel's spectrum to the average spectrum, yielding a scattering correction coefficient of 1.35 and a baseline shift of 0.12. This indicates that the pixel's scattering intensity is approximately 1.35 times the average level, with an additional background rise of 0.12. For another pixel located at the center of the cell body, its original spectrum was closer to the average spectrum, and a scattering correction coefficient of 1.02 and a baseline shift of 0.01 were obtained through fitting. After the correction transformation, the pixel's spectrum was divided by 1.35 and subtracted by 0.12, resulting in a value of 2930 cm⁻¹. −1 The lipid peak intensity at the point was adjusted from 0.78 before correction to about 0.49, which is basically consistent with the lipid peak intensity of 0.48 after pixel correction, thus obtaining the hyperspectral data cube after scattering correction.

[0122] It should be noted that the above values ​​are for illustrative purposes only and do not constitute a limitation on the present invention.

[0123] In this step, firstly, after acquiring a hyperspectral microscopic image of the nerve cell sample to be tested, a predetermined number of bands with the greatest information discrimination capability are selected from the entire spectrum based on the known characteristic absorption peak positions of the main biochemical components within the nerve cells. The spectral data corresponding to the selected bands constitute a dimension-reduced hyperspectral data cube. Among these, the characteristic absorption peaks include at least the CH stretching vibration peak of lipids (2840-2960 cm⁻¹). -1 The amide I peak of proteins is 1600-1700 cm⁻¹. -1 Phosphodiester bond peaks with nucleic acids: 1080-1240 cm⁻¹ -1 While retaining key biochemical diagnostic information, the data dimensionality was compressed to reduce the computational burden on subsequent networks. The selection principle was to prioritize retaining the bands containing characteristic absorption peaks and their adjacent bands, and to remove invalid bands that were severely affected by water vapor absorption or had excessively low signal-to-noise ratios.

[0124] Furthermore, for the spectral curve of each pixel in the dimension-reduced hyperspectral data cube, extended multivariate scattering correction is used to eliminate multiplicative scattering scaling and additive baseline drift caused by non-uniform scattering on the sample surface and optical path differences, resulting in a scattering-corrected hyperspectral data cube. By fitting the linear relationship of the average spectrum pixel by pixel and subtracting the baseline offset and multiplicative scattering coefficient, spectral pseudo-differences caused by non-uniform scattering on the sample surface and optical path differences are eliminated, so that the same biochemical components exhibit comparable response intensities at different spatial locations.

[0125] Furthermore, normalization is performed on the spectrum of each pixel in the hyperspectral data cube after scattering correction, mapping the spectral values ​​of all bands to a uniform numerical range. This eliminates global intensity variations caused by uneven illumination or detector gain differences during imaging. Eliminating these global intensity variations provides a consistent input to the deep learning network, facilitating stable gradient propagation and rapid convergence.

[0126] In summary, this step, through a three-step cascaded preprocessing process of feature band screening, extended multivariate scattering correction, and pixel spectral normalization, transforms the original hyperspectral microscopic image into a hyperspectral data cube with simplified bands, consistent scattering, and normalized intensity. This effectively compresses the data dimension to reduce the computational burden on subsequent networks, while eliminating spectral pseudo-differences and global intensity variations introduced by non-uniform scattering from the sample surface, optical path differences, and uneven illumination. This allows the same biochemical components to exhibit comparable spectral responses that truly reflect molecular composition at different spatial locations, thus providing high-quality, highly discriminative standardized input for the multi-scale spectral feature extraction network.

[0127] S300: Input the hyperspectral data cube into the pre-trained spectral feature extraction network and output the deep spectral feature map of each pixel.

[0128] In this step, the normalized hyperspectral data cube is input into a pre-trained multi-scale spectral feature extraction network. The network uses a parallel multi-branch one-dimensional convolutional architecture to encode the spectral curve of each pixel across scales. The small convolutional kernel branch accurately captures the sharp absorption features of narrow-peak molecules such as cytochrome c, while the large convolutional kernel branch fully covers the slow-varying envelope trend of broad-peak components such as lipid CH bonds. After channel splicing and 1×1 convolution fusion, a deep spectral feature map of each pixel is output. This feature map compresses the original high-dimensional spectral data into a low-dimensional, high-discriminative feature representation. While removing irrelevant interference bands such as water vapor absorption, it retains the key biochemical fingerprint information required to distinguish neuronal cell bodies, axonal myelin sheaths, and background matrix to the greatest extent, providing a condensed and physically interpretable feature foundation for subsequent spatial structure modeling.

[0129] S400: Input the deep spectral feature map into the pre-trained spatial context aggregation network, and output a spatial-spectral joint feature map by aggregating the spectral features in the neighborhood of each pixel.

[0130] After inputting the deep spectral feature map into the pre-trained spatial context aggregation network, the network differentially models the spatial morphology of nerve cells through a dual-branch parallel architecture. The cell body processing branch constructs a circular receptive field using dilated convolutions, extracts the clustered continuous features of the neuronal perinuclear body, and outputs a smooth cell body spatial feature map. The axon processing branch captures the directional components of linear fibers using horizontal and vertical strip convolution kernels, and adds the two to obtain an axon spatial feature map that is continuous along the axon direction and inhibits the cell body response. Subsequently, an adaptive spatial attention gate dynamically learns the fusion weights of the two branches at each pixel position through 1×1 convolutions, generating a natural gradient weight allocation in morphological transition regions such as the axon hill. The final output spatial-spectral joint feature map achieves pixel-level adaptive unification of spectral discrimination information and prior cellular spatial structure. That is, the cell body region maintains clustered uniformity, the axon region maintains linear continuity, and the background region achieves noise suppression, providing a joint feature expression with both biochemical discrimination and morphological integrity for abundance prediction.

[0131] S500: Input the spatial-spectral joint feature map into the pre-trained abundance prediction network and output the abundance map of the neural cell sample to be detected as the abundance detection result of the neural cell sample to be detected.

[0132] After inputting the spatial-spectral joint feature map into the pre-trained abundance prediction network, the fully connected layer first maps the joint features to the original abundance logical values ​​of each neuronal component. Then, the prior relation constraint layer uses the embedded transformation matrix to explicitly model the symbiotic and mutual exclusion relationships between components. When mitochondrial signals are detected, the abundance score of cytoplasmic proteins is actively enhanced, and when lipid myelin signals appear in the nucleus region, they are actively suppressed. After normalization by the Softmax activation layer, the abundance vector of each pixel satisfies the non-negative and sum-to-one constraints. Finally, the abundance map reconstruction layer arranges the abundance vectors of all pixels according to their spatial positions to generate a multi-channel abundance map with the same size as the input image. This abundance map is the abundance detection result of the neuronal sample to be detected. It intuitively presents the quantitative proportion and spatial location of subcellular components such as the nucleus, cytoplasmic proteins, lipid myelin, and mitochondria in the sample in the form of a pixel-level spatial distribution map. It provides a label-free, high-precision, and structurally preserved biochemical composition map for the assessment of the physiological state of neuronal cells, the discrimination of differentiation stages, and the analysis of pathological changes.

[0133] In summary, the embodiments of this application achieve complete intelligent computation from raw hyperspectral microscopic images to subcellular component abundance maps by jointly designing and optimizing the multi-scale spectral feature extraction network, spatial context aggregation network, and prior relation constraint abundance prediction network in an end-to-end manner. First, the multi-scale spectral feature extraction network adaptively captures cross-scale spectral features such as the broad peak of lipid CH and the narrow peak of protein amide I using parallel one-dimensional convolutional branches, significantly improving the ability to distinguish neural cell subtypes with similar spectral features but different biochemical properties. Second, the spatial context aggregation network matches clustered and linear cell morphologies respectively through the dilated convolution of the cell body branch and the strip convolution of the axon branch, and achieves pixel-level dynamic fusion through adaptive spatial attention gates, effectively solving the boundary blurring and structural breakage problems caused by the fragmentation of spectral-spatial information in traditional methods. The output spatial-spectral joint feature map has the uniformity of cell body, the continuity of axonal orientation, and the ability to suppress background noise. Finally, the abundance prediction network embeds the transformation matrix encoding the co-occurrence and mutual exclusion relationships of components in the prior relation constraint layer, so that the abundance prediction results strictly conform to the real biological composition logic of neural cells. At the same time, the joint loss function introduces component adaptive spatial regularization and endmember spectral smoothing regularization, further ensuring the spatial consistency of the abundance map and the physical interpretability of the endmember curves. With only a small number of labeled samples, it can output subcellular abundance maps with high numerical accuracy, intact spatial structure, and smooth and realistic endmember spectra. Moreover, end-to-end inference only requires a single forward propagation to complete the mapping from the original hyperspectral data cube to the abundance map, which balances detection accuracy, computational efficiency and biological rationality, and meets the application requirements for label-free, non-invasive, real-time quantitative molecular imaging of nerve cells.

[0134] Example 2, as Figure 2 As shown in the figure, this application provides a deep learning-based neural cell spectral detection system, the system comprising:

[0135] Joint pre-training module 11: Construct and jointly train the spectral feature extraction network, spatial context aggregation network, and abundance prediction network to obtain the pre-trained spectral feature extraction network, the pre-trained spatial context aggregation network, and the pre-trained abundance prediction network;

[0136] A spectral feature extraction network, a spatial context aggregation network, and an abundance prediction network are constructed and jointly trained to obtain pre-trained spectral feature extraction networks, pre-trained spatial context aggregation networks, and pre-trained abundance prediction networks, including:

[0137] Construct a multi-scale spectral feature extraction network, a spatial context aggregation network, and an abundance prediction network;

[0138] Using the reconstruction of hyperspectral data cubes as a self-supervised task, the multi-scale spectral feature extraction network and the spatial context aggregation network are pre-trained using unlabeled hyperspectral microscopic image samples to obtain an initialized multi-scale spectral feature extraction network and an initialized spatial context aggregation network.

[0139] The abundance prediction network is trained in a supervised manner using labeled hyperspectral microscopic image samples to obtain an initialized abundance prediction network. The labeled hyperspectral microscopic image samples are labeled with real abundance maps, and the real abundance maps contain the real abundance vector corresponding to each pixel.

[0140] Construct a joint loss function, which includes an abundance prediction error term, a spatial continuity regularization term, and a spectral smoothness regularization term;

[0141] The initialized multi-scale spectral feature extraction network, the initialized spatial context aggregation network, and the initialized abundance prediction network are treated as a whole and subjected to end-to-end supervised joint training using the joint loss function until the joint loss function converges, resulting in pre-trained spectral feature extraction network, pre-trained spatial context aggregation network, and pre-trained abundance prediction network.

[0142] This includes constructing a multi-scale spectral feature extraction network and a spatial context aggregation network, including:

[0143] Multiple parallel convolutional branches are set up, each containing a one-dimensional convolutional layer. Different convolutional kernel sizes are assigned to the one-dimensional convolutional layers of different branches, wherein the convolutional kernel sizes correspond to the typical widths of different characteristic spectral peaks of nerve cells.

[0144] The outputs of all convolutional branches are concatenated along the channel dimension to obtain the concatenated multi-scale feature vector;

[0145] A 1×1 convolutional layer is set after the spliced ​​multi-scale feature vector to obtain a multi-scale spectral feature extraction network.

[0146] A dual-branch spatial context aggregation network is constructed, which includes a cell body processing branch and an axon processing branch;

[0147] One or more two-dimensional convolutional layers with small convolutional kernels are set in the cell body processing branch, and dilated convolution is introduced in the two-dimensional convolutional layers with small convolutional kernels to extract the clustered spatial features of neuronal cell bodies and output the cell body spatial feature map.

[0148] A two-dimensional convolutional layer with strip convolutional kernels is set in the axon processing branch. The strip convolutional kernels include horizontal strip convolutional kernels and vertical strip convolutional kernels. The output feature maps of the horizontal strip convolutional kernels and the vertical strip convolutional kernels are added together to obtain the axon spatial feature map.

[0149] An adaptive spatial attention gate is set in the dual-branch spatial context aggregation network. The adaptive spatial attention gate receives the cell body spatial feature map and the axon spatial feature map, calculates the fusion weight at each spatial location through a 1×1 convolutional layer, and weights and sums the cell body spatial feature map and the axon spatial feature map according to the fusion weight to obtain the spatial-spectral joint feature map.

[0150] Constructing an abundance prediction network includes:

[0151] Construct a transformation matrix whose dimension is equal to the number of types of nerve cell components. Initialize the diagonal elements of the transformation matrix to 1, and initialize the off-diagonal elements of the transformation matrix according to the symbiotic or mutually exclusive relationship between nerve cell components: for two components with a symbiotic relationship, initialize the corresponding off-diagonal elements to positive numbers; for two components with a mutually exclusive relationship, initialize the corresponding off-diagonal elements to negative numbers.

[0152] An abundance prediction network is constructed, comprising a fully connected layer, a prior relation constraint layer, a softmax activation layer, and an abundance map reconstruction layer connected in sequence. The prior relation constraint layer contains the transformation matrix, the softmax activation layer is used to output the abundance vector of each pixel, and the abundance map reconstruction layer is used to arrange the abundance vectors of all pixels according to their pixel positions and output the abundance map.

[0153] Using the reconstruction of a hyperspectral data cube as a self-supervised task, the multi-scale spectral feature extraction network and the spatial context aggregation network are pre-trained using unlabeled hyperspectral microscopic image samples to obtain an initialized multi-scale spectral feature extraction network and an initialized spatial context aggregation network, including:

[0154] Multiple unlabeled hyperspectral microscopic image samples were acquired. Each unlabeled hyperspectral microscopic image sample was preprocessed by denoising and normalization to obtain a normalized hyperspectral data cube corresponding to each unlabeled hyperspectral microscopic image sample.

[0155] Construct a decoder network and connect the decoder network to the spatial context aggregation network;

[0156] The normalized hyperspectral data cube is sequentially input into the multi-scale spectral feature extraction network and the spatial context aggregation network to obtain a spatial-spectral joint feature map.

[0157] The spatial-spectral joint feature map is input into the decoder network, and a reconstructed hyperspectral data cube is output.

[0158] Calculate the mean square error between the reconstructed hyperspectral data cube and the normalized hyperspectral data cube, and use the mean square error as a self-supervised loss function.

[0159] Based on the self-supervised loss function, the parameters of the multi-scale spectral feature extraction network, the spatial context aggregation network, and the decoder network are iteratively optimized using the backpropagation algorithm until the self-supervised loss function converges, resulting in the initialized multi-scale spectral feature extraction network and the initialized spatial context aggregation network.

[0160] Constructing the joint loss function includes:

[0161] The mean square error between the abundance map output by the abundance prediction network and the actual abundance map is used as the abundance prediction error term.

[0162] The total variation loss of the abundance map is used as a spatial continuity regularization term, and different weights are assigned to the abundance channels corresponding to different neural cell components in the abundance map, wherein the channel weights corresponding to axonal components are higher than those corresponding to cell body components.

[0163] A learnable endmember spectral matrix is ​​constructed, and the second difference norm of the adjacent bands of the endmember spectral matrix is ​​used as the spectral smoothness regularization term. The dimension of the endmember spectral matrix is ​​equal to the number of types of neural cell components multiplied by the number of bands of the hyperspectral image.

[0164] The abundance prediction error term, the spatial continuity regularization term, and the spectral smoothness regularization term are added together to obtain the joint loss function.

[0165] The initialized multi-scale spectral feature extraction network, the initialized spatial context aggregation network, and the initialized abundance prediction network are treated as a whole and subjected to end-to-end supervised joint training using the joint loss function until the joint loss function converges, resulting in pre-trained spectral feature extraction network, pre-trained spatial context aggregation network, and pre-trained abundance prediction network, including:

[0166] The hyperspectral data cube of the labeled hyperspectral microscopic image sample is sequentially input into the initialized multi-scale spectral feature extraction network, the initialized spatial context aggregation network, and the initialized abundance prediction network. The abundance prediction error term between the abundance map output by the abundance prediction network and the true abundance map is calculated.

[0167] The spatial continuity regularization term is calculated based on the abundance map output by the abundance prediction network, and the second difference norm of the adjacent bands of the endmember spectral matrix is ​​calculated as the spectral smoothness regularization term.

[0168] The value of the joint loss function is obtained by adding the abundance prediction error term, the spatial continuity regularization term, and the spectral smoothness regularization term.

[0169] The gradient of the joint loss function with respect to all trainable parameters in the initialized multi-scale spectral feature extraction network, the initialized spatial context aggregation network, and the initialized abundance prediction network is calculated simultaneously using the backpropagation algorithm.

[0170] A gradient descent optimizer is used to synchronously update all trainable parameters of the initialized multi-scale spectral feature extraction network, the initialized spatial context aggregation network, and the initialized abundance prediction network according to the gradient.

[0171] Repeat the forward propagation, loss calculation, backpropagation, and parameter update steps until the joint loss function converges to obtain the pre-trained spectral feature extraction network, the pre-trained spatial context aggregation network, and the pre-trained abundance prediction network.

[0172] Hyperspectral image acquisition and preprocessing module 12: acquires hyperspectral microscopic images of the nerve cell sample to be detected, preprocesses the hyperspectral microscopic images, and obtains a normalized hyperspectral data cube;

[0173] Acquire a hyperspectral microscopic image of the nerve cell sample to be tested, and preprocess the hyperspectral microscopic image to obtain a normalized hyperspectral data cube, including:

[0174] A hyperspectral microscopic image of a nerve cell sample to be tested is acquired. Based on the position of the characteristic absorption peaks of nerve cell components, a preset number of wavelength bands with the highest information content are selected from the hyperspectral microscopic image. The characteristic absorption peaks include at least the CH stretching vibration peak of lipids, the amide I peak of proteins, and the phosphodiester bond peak of nucleic acids.

[0175] The spectral data corresponding to the selected bands are used to construct a dimension-reduced hyperspectral data cube.

[0176] For each pixel spectrum in the dimension-reduced hyperspectral data cube, extended multivariate scattering correction is used to remove scattering baseline drift, resulting in a scattering-corrected hyperspectral data cube.

[0177] The spectrum of each pixel in the scattering-corrected hyperspectral data cube is normalized to obtain a normalized hyperspectral data cube.

[0178] Specifically, for each pixel spectrum in the dimensionality-reduced hyperspectral data cube, extended multivariate scattering correction is applied to remove scattering baseline drift, resulting in a scattering-corrected hyperspectral data cube, including:

[0179] Calculate the arithmetic mean of the spectra of all pixels in the dimensionality-reduced hyperspectral data cube as the average spectrum;

[0180] For each pixel spectrum, the scattering correction coefficient and baseline offset are obtained by fitting the linear relationship between the pixel spectrum and the average spectrum using the least squares method.

[0181] The corrected pixel spectrum is obtained by subtracting the corresponding baseline offset from the spectrum of each pixel and dividing by the corresponding scattering correction coefficient. All the corrected pixel spectra are then used to form a scattering-corrected hyperspectral data cube.

[0182] Spectral feature extraction module 13: Inputs the hyperspectral data cube into the pre-trained spectral feature extraction network and outputs a deep spectral feature map of each pixel;

[0183] Spatial context aggregation module 14: Inputs the deep spectral feature map into the pre-trained spatial context aggregation network, and outputs a spatial-spectral joint feature map by aggregating the spectral features in the neighborhood of each pixel;

[0184] Abundance prediction output module 15: Inputs the spatial-spectral joint feature map into the pre-trained abundance prediction network and outputs the abundance map of the neural cell sample to be detected as the abundance detection result of the neural cell sample to be detected.

[0185] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0186] This specification and accompanying drawings are merely illustrative examples of this application and are intended to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from its scope. Therefore, if such modifications and modifications fall within the scope of this application and its equivalents, this application intends to include such modifications and modifications.

Claims

1. A deep learning-based neural cell spectral detection method, characterized in that, The method includes: A spectral feature extraction network, a spatial context aggregation network, and an abundance prediction network are constructed and jointly trained to obtain pre-trained spectral feature extraction network, pre-trained spatial context aggregation network, and pre-trained abundance prediction network. A hyperspectral microscopic image of the nerve cell sample to be detected is acquired, and the hyperspectral microscopic image is preprocessed to obtain a normalized hyperspectral data cube. The hyperspectral data cube is input into the pre-trained spectral feature extraction network, which outputs a deep spectral feature map for each pixel. The deep spectral feature map is input into the pre-trained spatial context aggregation network, and the spatial-spectral joint feature map is output by aggregating the spectral features in the neighborhood of each pixel. The spatial-spectral joint feature map is input into the pre-trained abundance prediction network, and the abundance map of the neural cell sample to be detected is output as the abundance detection result of the neural cell sample to be detected. This includes constructing a multi-scale spectral feature extraction network and a spatial context aggregation network, including: Multiple parallel convolutional branches are set up, each containing a one-dimensional convolutional layer. Different convolutional kernel sizes are assigned to the one-dimensional convolutional layers of different branches, wherein the convolutional kernel sizes correspond to the typical widths of different characteristic spectral peaks of nerve cells. The outputs of all convolutional branches are concatenated along the channel dimension to obtain the concatenated multi-scale feature vector; A 1×1 convolutional layer is set after the spliced ​​multi-scale feature vector to obtain a multi-scale spectral feature extraction network. A dual-branch spatial context aggregation network is constructed, which includes a cell body processing branch and an axon processing branch; One or more 3×3 small convolutional kernels are set in the cell body processing branch, and dilated convolution is introduced in the two-dimensional convolutional layer of the small convolutional kernels to extract the clustered spatial features of the neuron cell body and output the cell body spatial feature map. A two-dimensional convolutional layer with strip convolutional kernels is set in the axon processing branch. The strip convolutional kernels include horizontal strip convolutional kernels and vertical strip convolutional kernels. The output feature maps of the horizontal strip convolutional kernels and the vertical strip convolutional kernels are added together to obtain the axon spatial feature map. An adaptive spatial attention gate is set in the dual-branch spatial context aggregation network. The adaptive spatial attention gate receives the cell body spatial feature map and the axon spatial feature map. The fusion weight at each spatial location is calculated through a 1×1 convolutional layer. The cell body spatial feature map and the axon spatial feature map are weighted and summed according to the fusion weight to obtain a spatial-spectral joint feature map. The construction of the abundance prediction network includes: Construct a transformation matrix whose dimension is equal to the number of types of nerve cell components. Initialize the diagonal elements of the transformation matrix to 1, and initialize the off-diagonal elements of the transformation matrix according to the symbiotic or mutually exclusive relationship between nerve cell components: for two components with a symbiotic relationship, initialize the corresponding off-diagonal elements to positive numbers; for two components with a mutually exclusive relationship, initialize the corresponding off-diagonal elements to negative numbers. An abundance prediction network is constructed, comprising a fully connected layer, a prior relation constraint layer, a softmax activation layer, and an abundance map reconstruction layer connected in sequence. The prior relation constraint layer contains the transformation matrix, the softmax activation layer is used to output the abundance vector of each pixel, and the abundance map reconstruction layer is used to arrange the abundance vectors of all pixels according to their pixel positions and output the abundance map.

2. The neural cell spectral detection method based on deep learning according to claim 1, characterized in that, A spectral feature extraction network, a spatial context aggregation network, and an abundance prediction network are constructed and jointly trained to obtain pre-trained spectral feature extraction networks, pre-trained spatial context aggregation networks, and pre-trained abundance prediction networks, including: Construct a multi-scale spectral feature extraction network, a spatial context aggregation network, and an abundance prediction network; Using the reconstruction of hyperspectral data cubes as a self-supervised task, the multi-scale spectral feature extraction network and the spatial context aggregation network are pre-trained using unlabeled hyperspectral microscopic image samples to obtain an initialized multi-scale spectral feature extraction network and an initialized spatial context aggregation network. The abundance prediction network is trained in a supervised manner using labeled hyperspectral microscopic image samples to obtain an initialized abundance prediction network. The labeled hyperspectral microscopic image samples are labeled with real abundance maps, and the real abundance maps contain the real abundance vector corresponding to each pixel. Construct a joint loss function, which includes an abundance prediction error term, a spatial continuity regularization term, and a spectral smoothness regularization term; The initialized multi-scale spectral feature extraction network, the initialized spatial context aggregation network, and the initialized abundance prediction network are treated as a whole and subjected to end-to-end supervised joint training using the joint loss function until the joint loss function converges, resulting in pre-trained spectral feature extraction network, pre-trained spatial context aggregation network, and pre-trained abundance prediction network.

3. The deep learning-based neural cell spectral detection method according to claim 2, characterized in that, Using the reconstruction of a hyperspectral data cube as a self-supervised task, the multi-scale spectral feature extraction network and the spatial context aggregation network are pre-trained using unlabeled hyperspectral microscopic image samples to obtain an initialized multi-scale spectral feature extraction network and an initialized spatial context aggregation network, including: Multiple unlabeled hyperspectral microscopic image samples were acquired. Each unlabeled hyperspectral microscopic image sample was preprocessed by denoising and normalization to obtain a normalized hyperspectral data cube corresponding to each unlabeled hyperspectral microscopic image sample. Construct a decoder network and connect the decoder network to the spatial context aggregation network; The normalized hyperspectral data cube is sequentially input into the multi-scale spectral feature extraction network and the spatial context aggregation network to obtain a spatial-spectral joint feature map. The spatial-spectral joint feature map is input into the decoder network, and a reconstructed hyperspectral data cube is output. Calculate the mean square error between the reconstructed hyperspectral data cube and the normalized hyperspectral data cube, and use the mean square error as a self-supervised loss function. Based on the self-supervised loss function, the parameters of the multi-scale spectral feature extraction network, the spatial context aggregation network, and the decoder network are iteratively optimized using the backpropagation algorithm until the self-supervised loss function converges, resulting in the initialized multi-scale spectral feature extraction network and the initialized spatial context aggregation network.

4. The deep learning-based neural cell spectral detection method according to claim 2, characterized in that, Constructing the joint loss function includes: The mean square error between the abundance map output by the abundance prediction network and the actual abundance map is used as the abundance prediction error term. The total variation loss of the abundance map is used as a spatial continuity regularization term, and different weights are assigned to the abundance channels corresponding to different neural cell components in the abundance map, wherein the channel weights corresponding to axonal components are higher than those corresponding to cell body components. A learnable endmember spectral matrix is ​​constructed, and the second difference norm of the adjacent bands of the endmember spectral matrix is ​​used as the spectral smoothness regularization term. The dimension of the endmember spectral matrix is ​​equal to the number of types of neural cell components multiplied by the number of bands of the hyperspectral image. The abundance prediction error term, the spatial continuity regularization term, and the spectral smoothness regularization term are added together to obtain the joint loss function.

5. The deep learning-based neural cell spectral detection method according to claim 2, characterized in that, The initialized multi-scale spectral feature extraction network, the initialized spatial context aggregation network, and the initialized abundance prediction network are treated as a whole and subjected to end-to-end supervised joint training using the joint loss function until the joint loss function converges, resulting in pre-trained spectral feature extraction network, pre-trained spatial context aggregation network, and pre-trained abundance prediction network, including: The hyperspectral data cube of the labeled hyperspectral microscopic image sample is sequentially input into the initialized multi-scale spectral feature extraction network, the initialized spatial context aggregation network, and the initialized abundance prediction network. The abundance prediction error term between the abundance map output by the abundance prediction network and the true abundance map is calculated. The spatial continuity regularization term is calculated based on the abundance map output by the abundance prediction network, and the second difference norm of the adjacent bands of the endmember spectral matrix is ​​calculated as the spectral smoothness regularization term. The value of the joint loss function is obtained by adding the abundance prediction error term, the spatial continuity regularization term, and the spectral smoothness regularization term. The gradient of the joint loss function with respect to all trainable parameters in the initialized multi-scale spectral feature extraction network, the initialized spatial context aggregation network, and the initialized abundance prediction network is calculated simultaneously using the backpropagation algorithm. A gradient descent optimizer is used to synchronously update all trainable parameters of the initialized multi-scale spectral feature extraction network, the initialized spatial context aggregation network, and the initialized abundance prediction network according to the gradient. Repeat the forward propagation, loss calculation, backpropagation, and parameter update steps until the joint loss function converges to obtain the pre-trained spectral feature extraction network, the pre-trained spatial context aggregation network, and the pre-trained abundance prediction network.

6. The deep learning-based neural cell spectral detection method according to claim 1, characterized in that, Acquire a hyperspectral microscopic image of the nerve cell sample to be tested, and preprocess the hyperspectral microscopic image to obtain a normalized hyperspectral data cube, including: A hyperspectral microscopic image of a nerve cell sample to be tested is acquired. Based on the position of the characteristic absorption peaks of nerve cell components, a preset number of wavelength bands with the highest information content are selected from the hyperspectral microscopic image. The characteristic absorption peaks include at least the CH stretching vibration peak of lipids, the amide I peak of proteins, and the phosphodiester bond peak of nucleic acids. The spectral data corresponding to the selected bands are used to construct a dimension-reduced hyperspectral data cube. For each pixel spectrum in the dimension-reduced hyperspectral data cube, extended multivariate scattering correction is used to remove scattering baseline drift, resulting in a scattering-corrected hyperspectral data cube. The spectrum of each pixel in the scattering-corrected hyperspectral data cube is normalized to obtain a normalized hyperspectral data cube.

7. The deep learning-based neural cell spectral detection method according to claim 6, characterized in that, For each pixel spectrum in the dimensionality-reduced hyperspectral data cube, extended multivariate scattering correction is applied to remove scattering baseline drift, resulting in a scattering-corrected hyperspectral data cube, including: Calculate the arithmetic mean of the spectra of all pixels in the dimensionality-reduced hyperspectral data cube as the average spectrum; For each pixel spectrum, the scattering correction coefficient and baseline offset are obtained by fitting the linear relationship between the pixel spectrum and the average spectrum using the least squares method. The corrected pixel spectrum is obtained by subtracting the corresponding baseline offset from the spectrum of each pixel and dividing by the corresponding scattering correction coefficient. All the corrected pixel spectra are then used to form a scattering-corrected hyperspectral data cube.

8. A deep learning-based neural cell spectral detection system, characterized in that, The system for implementing the deep learning-based neural cell spectral detection method according to any one of claims 1 to 7, the system comprising: Joint pre-training module: Construct and jointly train the spectral feature extraction network, spatial context aggregation network, and abundance prediction network to obtain pre-trained spectral feature extraction network, pre-trained spatial context aggregation network, and pre-trained abundance prediction network; Hyperspectral image acquisition and preprocessing module: acquires hyperspectral microscopic images of the nerve cell sample to be detected, preprocesses the hyperspectral microscopic images to obtain a normalized hyperspectral data cube; Spectral feature extraction module: Inputs the hyperspectral data cube into the pre-trained spectral feature extraction network and outputs a deep spectral feature map for each pixel; Spatial context aggregation module: Inputs the deep spectral feature map into the pre-trained spatial context aggregation network, and outputs a spatial-spectral joint feature map by aggregating the spectral features in the neighborhood of each pixel; Abundance prediction output module: Input the spatial-spectral joint feature map into the pre-trained abundance prediction network, and output the abundance map of the neural cell sample to be detected as the abundance detection result of the neural cell sample to be detected.

Images