Three-dimensional neural signal extraction method based on four-dimensional light field

Abstract

The invention relates to the technical field of image processing, and discloses a three-dimensional neural signal extraction method based on a four-dimensional light field, and the method comprises the following steps: S1, obtaining time sequence multi-view in-vivo neural data D under the four-dimensional light field; s2, analyzing neuron distribution according to the time sequence multi-view in-vivo neural data D, and obtaining multi-view data representing neuron positions; s3, reconstructing the multi-view data based on PSF to obtain three-dimensional space position data of neuron distribution; s4, performing neuron segmentation based on the three-dimensional space position data to obtain a segmentation mask body of each neuron; s5, performing full-frame-rate frame-by-frame reconstruction on the time sequence multi-view living neural data D based on PSF to obtain a time sequence three-dimensional body stack, and obtaining a neuron activity matrix of each neuron in the time sequence three-dimensional body stack according to the segmentation mask body; according to the invention, accurate capture and segmentation of neurons can be realized from the neural data of the four-dimensional light field, so that extraction of neuron activities is realized more accurately.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more particularly to a three-dimensional neural signal extraction method based on a four-dimensional light field. Background Art

[0002] Neural signals are the core carriers of information transmission in the nervous system, and their dynamic changes directly reflect neuronal activity, circuit function, and even higher-level cognitive behavior. In the biomedical field, the significance of neural signal analysis is reflected in multiple dimensions: in basic research, neural signals can be used to reveal neural coding mechanisms (such as action potential timing and synaptic plasticity) and analyze the neural basis of complex functions such as perception, memory, and decision-making; in clinical medicine, they can assist in the localization of epileptic lesions, the selection of deep brain stimulation targets for Parkinson's disease, or help patients with motor dysfunction rebuild control through brain-computer interfaces; in drug development, they can be used to evaluate the intervention effects of neuromodulatory tools (such as optogenetics and chemogenetics) or to screen candidate compounds targeting ion channels. However, the spatiotemporal dynamics of neural signals, characterized by millisecond-level transients and three-dimensional network synergy, place extremely high demands on observation technology.

[0003] Traditional microscopic imaging techniques, such as light-sheet microscopy and two-photon microscopy, achieve optical tomography through scanning imaging. While they offer high spatial resolution, they suffer from significant limitations: insufficient temporal resolution: point-by-point scanning limits frame rates (typically <30 Hz), making it difficult to capture rapid neural events (e.g., action potential firing rates can reach the kHz level); cumulative phototoxicity: prolonged exposure to high-power lasers can cause photobleaching or cell damage, limiting long-term observations in vivo; and loss of spatial information: single-focal-plane imaging struggles to synchronously record the activity of populations of three-dimensionally distributed neurons, while mechanical Z-axis scanning further reduces temporal resolution. These issues present traditional methods with a dilemma: balancing speed, accuracy, and activity when analyzing highly dynamic, large-scale neural signals.

[0004] Light field microscopy (LFM) is a new imaging paradigm that makes up for the shortcomings of traditional methods through light field encoding and computational reconstruction technology. Light field microscopy uses a microlens array to record four-dimensional light field information at one time, achieving a kHz-level frame rate without mechanical scanning, perfectly matching the transient characteristics of neural signals. In addition, the multi-perspective characteristics of the light field of light field microscopy allow it to obtain three-dimensional volumetric imaging results of the subject with only a single exposure, greatly reducing the impact of phototoxicity on long-term live filming and supporting continuous observation of live samples for several hours.

[0005] Therefore, how to fully utilize four-dimensional light field information to accurately extract and confirm three-dimensional neural signals is an urgent problem that technicians in this field need to solve. Summary of the Invention

[0006] In view of this, the present invention provides a three-dimensional neural signal extraction method based on four-dimensional light field, which can accurately capture and segment neurons from the neural data of the four-dimensional light field, thereby more accurately extracting neuronal activity.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] A three-dimensional neural signal extraction method based on a four-dimensional light field comprises the following steps:

[0009] S1: Acquire time-series multi-view in vivo neural data D under a four-dimensional light field.

[0010] S2: Analyze neuron distribution according to the time-series multi-view in vivo neural data D, and obtain multi-view data representing neuron positions.

[0011] S3: Reconstruct the multi-view data based on the PSF to obtain three-dimensional spatial position data of neuron distribution.

[0012] S4: Perform neuron segmentation based on the three-dimensional spatial position data to obtain a segmentation mask for each neuron.

[0013] S5: Based on the PSF data, the temporal multi-view in vivo neural data D is reconstructed frame by frame at full frame rate to obtain a temporal three-dimensional volume stack, and the neuronal activity matrix of each neuron in the temporal three-dimensional volume stack is obtained according to the segmentation mask volume.

[0014] Preferably, the S2 specifically includes: calculating the standard deviation of all frame images at each viewing angle in time sequence based on the time sequence multi-view in vivo neural data to obtain the multi-view data.

[0015] Preferably, the calculation formula for the neuron position data at each viewing angle in S2 is:

[0016]

[0017] in, represents the neuron position data under the current perspective, that is, the data corresponding to a single perspective in the multi-perspective data, is the 2D image of the i-th frame under the first viewing angle, is the temporal mean of all two-dimensional images at the current viewing angle, and t is the total number of data frames.

[0018] Preferably, the S3 specifically includes:

[0019] Performing sparse sampling on the multi-view data to obtain sparse view data;

[0020] The PSF_sparse data corresponding to the sparse perspective is extracted from the PSF, and the sparse perspective data is reconstructed to obtain the three-dimensional space position data.

[0021] Preferably, the S5 specifically includes:

[0022] For each frame in the temporal multi-view neural data D, sparse view sampling is performed and then sent to SeReNet together with the corresponding PSF_sparse to complete light field reconstruction.

[0023] Preferably, the SeReNet algorithm is used for reconstruction in S3.

[0024] Preferably, the neuron segmentation step in S4 includes:

[0025] S41: Use UNet3D to perform preliminary neuron candidate point coordinates p1, feature vector f1, neuron radius s1 and binary mask u1 detection on the three-dimensional spatial position data std_volume;

[0026] S42: For each neuron candidate point, extract the feature vectors of the five closest points in its neighborhood and send them to EXNet for clustering and deduplication;

[0027] S43: According to the clustering result index, the corresponding neuron coordinates, radius and mask are taken from p1, s1 and u1;

[0028] S44: After traversing the indices of all neurons obtained by clustering, multiplying the corresponding mask by the corresponding index and putting the result into the all-zero matrix mask_volume to obtain the final three-dimensional mask volume of all neurons.

[0029] A three-dimensional neural signal extraction system based on a four-dimensional light field, comprising:

[0030] The data acquisition module is used to obtain time-series multi-view in vivo neural data and corresponding PSF data under a four-dimensional light field.

[0031] The distribution analysis module is connected to the data acquisition module and is used to analyze the neuron distribution according to the time-series multi-view living neural data D and obtain multi-view data representing the neuron position.

[0032] An image reconstruction module is connected to the distribution analysis module and is used to reconstruct the multi-view data based on the PSF data to obtain three-dimensional spatial position data of neuron distribution.

[0033] A segmentation module is connected to the image reconstruction module and is used to perform neuron segmentation based on the three-dimensional spatial position data to obtain a segmentation mask volume of each neuron.

[0034] The image reconstruction module is also used to reconstruct the temporal multi-view living neural data D frame by frame at full frame rate based on the PSF data to obtain a temporal three-dimensional volume stack, and obtain the neuronal activity matrix of each neuron in the temporal three-dimensional volume stack based on the segmentation mask volume.

[0035] Preferably, it also includes a sparse sampling module, which is used to perform sparse perspective sampling on the multi-perspective data to obtain sparse perspective data; extract PSF_sparse data corresponding to the sparse perspective from the PSF data, reconstruct the sparse perspective data, and obtain the three-dimensional spatial position data.

[0036] Preferably, the sparse sampling module is further used to perform perspective sampling on the PSF data and the time-series multi-perspective living neural data D before performing the full frame rate frame-by-frame reconstruction.

[0037] It can be seen from the above technical solution that compared with the existing technology, the present invention discloses a three-dimensional neural signal extraction method based on a four-dimensional light field, which uses a light field microscope to replace the traditional microscopic imaging mode. Four-dimensional light field information can be obtained with a single exposure. This four-dimensional information can be reconstructed into a three-dimensional sample volume at the corresponding moment through a light field reconstruction algorithm. Compared with the traditional scanning imaging mode, the imaging efficiency is improved by more than an order of magnitude. Moreover, since the light intensity during exposure is evenly distributed in the sample, the local light dose of the sample in the scanning imaging can be diluted, and the overall phototoxicity can be reduced by more than 90%.

[0038] In addition, since the light field images captured by the present invention need to be post-processed by the light field reconstruction algorithm for neuronal activity extraction, accelerating the post-processing speed is crucial for analyzing neuronal activity. The present invention introduces a sparse sampling strategy to downsample the original light field data by more than 4 times in the perspective dimension, accelerating the post-processing speed while ensuring the parallax information between perspectives. The computation and subsequent reconstruction process improves the efficiency of neural analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0040] Figure 1 A schematic diagram of a three-dimensional neural signal extraction method based on a four-dimensional light field provided in an embodiment of the present invention.

[0041] Figure 2Schematic diagram of a sparse perspective sampling method in an embodiment of the present invention.

[0042] Figure 3 A schematic diagram of a three-dimensional neural signal extraction system based on a four-dimensional light field provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0044] Example 1

[0045] like Figure 1 The embodiment of the present invention discloses a three-dimensional neural signal extraction method based on a four-dimensional light field, comprising the following steps:

[0046] S1: Acquire time-series multi-view in vivo neural data D under a four-dimensional light field.

[0047] S2: Analyze neuron distribution according to the time-series multi-view in vivo neural data D, and obtain multi-view data representing neuron positions.

[0048] S3: Reconstruct the multi-view data based on the PSF to obtain three-dimensional spatial position data of neuron distribution.

[0049] S4: Perform neuron segmentation based on the three-dimensional spatial position data to obtain a segmentation mask for each neuron.

[0050] S5: Based on the PSF, the temporal multi-view in vivo neural data D is reconstructed frame by frame at full frame rate to obtain a temporal three-dimensional volume stack, and the neuronal activity matrix of each neuron in the temporal three-dimensional volume stack is obtained according to the segmentation mask volume.

[0051] In this embodiment, the present invention utilizes time series data of multiple perspectives in a four-dimensional light field to achieve accurate positioning and segmentation of neurons and accurate extraction of group activities of multiple neurons.

[0052] For S1, data acquisition can use a light field microscope system to shoot time-series multi-view in vivo neural data D, the shape of D is ,in is the total number of frames captured in the time series, 81 is the number of viewing angles of the light field microscopy system capturing data, is the pixel height of the two-dimensional image of each frame and each perspective, is the pixel width of the 2D image for each viewpoint in each frame.

[0053] In order to further implement the above technical solution, the calculation formula for the neuron distribution data at each perspective in S2 is:

[0054]

[0055] in, is the 2D image of the i-th frame under the first viewing angle, is the temporal mean of all two-dimensional images at the current viewing angle, and t is the total number of data frames.

[0056] In this embodiment, since the pixel intensity of flickering neurons in the multi-view in vivo neural data captured by the light field microscope system is higher than the pixel intensity of the background such as blood vessels and cerebral fluid in the in vivo neural data, in order to obtain the spatial position of each neuron in the in vivo neural data, precise positioning is achieved by calculating the standard deviation (STD) of all frame images in each viewpoint in time sequence.

[0057] In another embodiment, sparse view sampling is utilized to speed up the extraction process. Figure 2 Schematic diagram of sparse perspective selection, based on Figure 2 The perspective shown is sparsely sampled from the multi-perspective data to obtain sparse perspective data. The data corresponding to the sparse perspective is extracted from the PSF, namely PSF_sparse data, and the sparse perspective data is reconstructed to obtain 3D spatial position data. The red area is the selected perspective.

[0058] Select from the 81 viewpoints in the example Figure 2 The 13 viewing angles with large parallax are shown. The standard deviation of each viewing angle is calculated using the neuron distribution data calculation formula to obtain the shape Multi-view data .

[0059] Then, for multi-view data Reconstruction, in order to To restore the three-dimensional spatial position information of neurons from the multi-view information, it is necessary to According to the optical point spread function (PSF) of the light field microscope system, the shape is , where 81 is the number of viewing angles, 61 is the depth, and 2 is the offset along the height and width directions) is reconstructed into a shape of of The three-dimensional body is used for the next three-dimensional segmentation. This invention uses the open source light field image reconstruction algorithm SeReNet to complete the high-fidelity light field image reconstruction task. The speed of this reconstruction process is closely related to the number of viewing angles of the light field data. Therefore, the extraction of PSF and S2 The corresponding sparse perspective obtains PSF_sparse for three-dimensional light field reconstruction to accelerate the reconstruction process.

[0060] Specifically, the SeReNet reconstruction process is as follows:

[0061] 1) Each two-dimensional angle information is copied d times, where d is the depth of the PSF, and the three-dimensional angle information of d layers and 13 light field angles per layer is obtained. , whose shape is At the same time, the light field microscopy system can obtain the PSF while acquiring images, which contains the offset information at each angle in each depth layer.

[0062] 2) According to the offset of the corresponding layer of PSF corresponding to the viewing angle, it is translated a certain distance in the horizontal and vertical directions to obtain , whose shape is ;

[0063] 3) It is fed into 4 convolutional layers, a 2x upsampling layer, 2 convolutional layers, a 2x upsampling layer, three convolutional layers, and a 1.25x upsampling layer in sequence, and the final shape is of Three-dimensional body.

[0064] To further implement the above technical solution, the neuron segmentation step in S4 includes:

[0065] S41: Perform preliminary neuron detection on the three-dimensional spatial position data std_volume using UNet3D. The U-Net3D module in the present invention adopts a symmetrical encoding-decoding structure and includes the following main components:

[0066] Encoder path (ContextPathway):

[0067] It contains multiple downsampling levels (Level1~Level4), each level includes: a set of convolutional layers (Conv3D), instance normalization layers (InstanceNorm3D), LeakyReLU activation functions, and residual connections (ResidualConnection); convolution with a step size of (2,2,1) is used to achieve downsampling of the spatial dimension, and the number of output feature map channels is doubled in sequence.

[0068] Bottleneck layer:

[0069] It consists of a convolutional layer with further downsampling and a residual structure to extract high-dimensional semantic information and serve as the starting point of the decoder.

[0070] Decoder path (Localization Pathway):

[0071] Upsampling is performed using nearest neighbor interpolation (Upsample), and each decoding layer fuses the skip connection information of the corresponding encoding layer. The final output resolution is consistent with the original input, and multi-scale features are retained during the decoding process to assist in subsequent clustering analysis. After std_volume passes through the above structure, the network aggregates low-dimensional and high-dimensional features in the input 3D volume, performs coarse instance segmentation on the foreground and background in the 3D volume, and outputs the neuron candidate point coordinates p1, feature vector f1, neuron radius s1 and binary mask u1 for subsequent processing.

[0072] S42: For each neuron candidate point feature vector , extract the feature vectors of the 5 closest points in its neighborhood , and sent to EXNet for clustering and deduplication. During this process, the semantic relevance score f between the candidate point and its neighbors is calculated. EXNet is a deep learning network that stacks fully connected layers and uses the LeakyRelu activation function. Its final output is a 5-dimensional binary classification score. If the score in a dimension is higher than the threshold of 0.9, the neighboring point is considered to belong to the same neuron as the candidate point.

[0073] S43: According to the clustering result index, the corresponding neuron coordinates, radius and mask are taken out from the neuron candidate point coordinates p1, neuron radius s1 and binary mask u1.

[0074] S44: After traversing the indices of all neurons obtained by clustering, multiplying the corresponding mask by the corresponding index and putting the result into the all-zero matrix mask_volume to obtain the final three-dimensional mask volume of all neurons.

[0075] Example 2

[0076] like Figure 3 Based on the same inventive concept, an embodiment of the present invention discloses a three-dimensional neural signal extraction system based on a four-dimensional light field, comprising:

[0077] The data acquisition module is used to obtain time-series multi-view in vivo neural data and corresponding PSF data under a four-dimensional light field.

[0078] The distribution analysis module is connected to the data acquisition module and is used to analyze the neuron distribution based on the time-series multi-view in vivo neural data D and obtain multi-view data representing the neuron position.

[0079] The image reconstruction module is connected to the distribution analysis module and is used to reconstruct the multi-view data based on the PSF data to obtain the three-dimensional spatial position data of the neuron distribution.

[0080] The segmentation module is connected to the image reconstruction module and is used to perform neuron segmentation based on the three-dimensional spatial position data to obtain the segmentation mask of each neuron.

[0081] The image reconstruction module is also used to reconstruct the temporal multi-view in vivo neural data D frame by frame at full frame rate based on the PSF data to obtain a temporal three-dimensional volume stack, and obtain the neuronal activity matrix of each neuron in the temporal three-dimensional volume stack based on the segmentation mask volume.

[0082] In order to further implement the above technical solution, a sparse sampling module is also included, which is used to perform sparse perspective sampling on multi-perspective data to obtain sparse perspective data; the PSF_sparse data corresponding to the sparse perspective is extracted from the PSF data, and the sparse perspective data is reconstructed to obtain three-dimensional spatial position data.

[0083] Furthermore, the sparse sampling module is also used to perform perspective sampling on the PSF data and the time-series multi-perspective living neural data D before performing full frame rate frame-by-frame reconstruction.

[0084] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0085] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A three-dimensional neural signal extraction method based on four-dimensional light field, characterized in that: The following steps are involved: S1: Acquire time-series multi-view in vivo neural data D and PSF data under four-dimensional light field; S2: Analyze neuron distribution according to the time-series multi-view in vivo neural data D, and obtain multi-view data representing neuron positions; S3: reconstructing the multi-view data based on the PSF data to obtain three-dimensional spatial position data of neuron distribution; S4: performing neuron segmentation based on the three-dimensional spatial position data to obtain a segmentation mask for each neuron; S5: Based on the PSF data, the temporal multi-view in vivo neural data D is reconstructed frame by frame at full frame rate to obtain a temporal three-dimensional volume stack, and the neuronal activity matrix of each neuron in the temporal three-dimensional volume stack is obtained according to the segmentation mask volume.

2. The three-dimensional neural signal extraction method based on four-dimensional light field according to claim 1, characterized in that: The S2 specifically includes: based on the time-series multi-view in vivo neural data, calculating the standard deviation of all frame images at each view in time sequence to obtain the multi-view data.

3. The method for extracting three-dimensional neural signals based on a four-dimensional light field according to claim 2, wherein: The calculation formula for the neuron position data at each viewing angle in S2 is: ； in, Represents the neuron position data under the current viewing angle, is the i-th two-dimensional image at the current viewing angle, is the temporal mean of all two-dimensional images at the current viewing angle, and t is the total number of data frames.

4. The method for extracting three-dimensional neural signals based on four-dimensional light fields according to claim 1, characterized in that: The S3 specifically includes: Performing sparse sampling on the multi-view data to obtain sparse view data; The PSF_sparse data corresponding to the sparse perspective is extracted from the PSF data, and the sparse perspective data is reconstructed to obtain the three-dimensional space position data.

5. A three-dimensional neural signal extraction method based on a four-dimensional light field according to claim 1 or 4, characterized in that: The S5 specifically includes: For each frame in the temporal multi-view in vivo neural data D, sparse view sampling is performed and then sent to SeReNet together with the corresponding PSF_sparse to complete light field reconstruction.

6. The method for extracting three-dimensional neural signals based on four-dimensional light fields according to claim 1, characterized in that: In S3, the SeReNet algorithm is used for reconstruction.

7. The method for extracting three-dimensional neural signals based on four-dimensional light fields according to claim 1, characterized in that: The neuron segmentation step in S4 includes: S41: Use UNet3D to perform preliminary detection on the three-dimensional spatial position data std_volume to obtain the neuron candidate point coordinates p1, feature vector f1, neuron radius s1 and binary mask u1; S42: For each candidate neuron point, extract the feature vectors of multiple points in its neighborhood, perform clustering and remove duplicates; S43: According to the clustering result index, the corresponding neuron coordinates, radius and mask are taken from p1, s1 and u1; S44: After traversing the indices of all neurons obtained by clustering, multiplying the corresponding mask by the corresponding index and putting the result into the all-zero matrix mask_volume to obtain the final three-dimensional mask volume of all neurons.

8. A three-dimensional neural signal extraction system based on four-dimensional light field, characterized in that: include: Data acquisition module, used to acquire time-series multi-view in vivo neural data and corresponding PSF data under four-dimensional light field; a distribution analysis module, connected to the data acquisition module, for analyzing neuron distribution based on the time-series multi-view in vivo neural data D and obtaining multi-view data representing neuron positions; an image reconstruction module, connected to the distribution analysis module, for reconstructing the multi-view data based on the PSF data to obtain three-dimensional spatial position data of neuron distribution; a segmentation module, connected to the image reconstruction module, for performing neuron segmentation based on the three-dimensional spatial position data to obtain a segmentation mask for each neuron; The image reconstruction module is also used to reconstruct the temporal multi-view living neural data D frame by frame at full frame rate based on the PSF data to obtain a temporal three-dimensional volume stack, and obtain the neuronal activity matrix of each neuron in the temporal three-dimensional volume stack based on the segmentation mask volume.

9. The three-dimensional neural signal extraction system based on four-dimensional light field according to claim 8, characterized in that: It also includes a sparse sampling module, which is used to perform sparse perspective sampling on the multi-perspective data to obtain sparse perspective data; extract PSF_sparse data corresponding to the sparse perspective from the PSF data, reconstruct the sparse perspective data, and obtain the three-dimensional spatial position data.

10. The three-dimensional neural signal extraction system based on four-dimensional light field according to claim 9, characterized in that: The sparse sampling module is further configured to perform perspective sampling on the PSF data and the time-series multi-perspective living neural data D before performing the full frame rate frame-by-frame reconstruction.

Images