A three-dimensional neural signal extraction method based on a four-dimensional light field

Abstract

The application relates to the technical field of image processing, and discloses a three-dimensional neural signal extraction method based on a four-dimensional light field, which comprises the following steps: S1: acquiring time-series multi-view living body neural data D under a four-dimensional light field; S2: analyzing neuron distribution according to the time-series multi-view living body neural data D, and obtaining multi-view data representing neuron positions; S3: reconstructing the multi-view data based on PSF to obtain three-dimensional spatial position data of the neuron distribution; S4: carrying out neuron segmentation based on the three-dimensional spatial position data to obtain segmentation mask bodies of various neurons; and S5: carrying out full-frame-rate frame-by-frame reconstruction on the time-series multi-view living body neural data D based on PSF to obtain a time-series three-dimensional body stack, and obtaining a neuron activity matrix of each neuron in the time-series three-dimensional body stack according to the segmentation mask bodies; the application can realize accurate capture and segmentation of neurons from neural data of the four-dimensional light field, and further realizes more accurate extraction of neuron activity.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to a method for extracting three-dimensional neural signals based on a four-dimensional light field. Background Technology

[0002] Neural signals are the core carriers of information transmission in the nervous system, and their dynamic changes directly reflect neuronal activity, circuit function, and even higher cognitive behaviors. In the biomedical field, the significance of neural signal analysis is multi-dimensional: in basic research, neural signals can be used to reveal neural coding mechanisms (such as action potential timing and synaptic plasticity) and analyze the neural basis of complex functions such as perception, memory, and decision-making; in clinical medicine, they can assist in the localization of epileptic lesions, the selection of deep brain stimulation targets for Parkinson's disease, or help patients with motor dysfunction regain control through brain-computer interfaces; in drug development, they can be used to evaluate the intervention effects of neuromodulation tools (such as optogenetics and chemogenetics) or to screen candidate compounds targeting ion channels. However, the spatiotemporal dynamics of neural signals exhibit millisecond-level transients and three-dimensional network synergy, which places extremely high demands on observation techniques.

[0003] Traditional microscopic imaging techniques such as light sheet microscopy and two-photon microscopy achieve optical tomography through scanning imaging. While providing high spatial resolution, they have significant limitations: insufficient temporal resolution: point-by-point scanning limits the frame rate (typically <30Hz), making it difficult to capture rapid neural events (such as action potentials with firing frequencies reaching kHz); cumulative phototoxicity: prolonged exposure to high-power lasers causes photobleaching or cell damage, limiting long-term in vivo observations; and loss of spatial information: single-focal-plane imaging struggles to simultaneously record the collective activity of three-dimensionally distributed neurons, while mechanical Z-axis scanning further reduces temporal resolution. These problems lead to a dilemma where speed, accuracy, and activity are mutually exclusive when resolving highly dynamic, large-scale neural signals using traditional methods.

[0004] Light field microscopy (LFM) is a novel imaging paradigm that overcomes the shortcomings of traditional methods by using light field encoding and computational reconstruction techniques. LFM uses a microlens array to record four-dimensional light field information in one go, achieving kHz-level frame rates without mechanical scanning, perfectly matching the transient characteristics of neural signals. In addition, the multi-view characteristics of light field microscopy allow it to obtain three-dimensional volumetric imaging results of the subject with only a single exposure, greatly reducing the impact of phototoxicity on long-term live imaging and supporting continuous observation of live samples for several hours.

[0005] Therefore, how to make full use of four-dimensional light field information to achieve accurate extraction and confirmation of three-dimensional neural signals is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] In view of this, the present invention provides a three-dimensional neural signal extraction method based on a four-dimensional light field, which can accurately capture and segment neurons from neural data of a four-dimensional light field, thereby achieving more accurate extraction of neuronal activity.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A method for extracting three-dimensional neural signals based on a four-dimensional light field includes the following steps:

[0009] S1: Acquire temporal multi-view live neural data D under a four-dimensional light field.

[0010] S2: Analyze the neuron distribution based on the temporal multi-view live neural data D, and obtain multi-view data representing the neuron locations.

[0011] S3: Reconstruct the multi-view data based on PSF to obtain the three-dimensional spatial location data of the neuron distribution.

[0012] S4: Perform neuron segmentation based on the three-dimensional spatial location data to obtain the segmentation mask of each neuron.

[0013] S5: Based on the PSF data, the temporal multi-view live neural data D is reconstructed frame by frame at full frame rate to obtain a temporal three-dimensional volume stack, and the neuron activity matrix of each neuron in the temporal three-dimensional volume stack is obtained according to the segmentation mask volume.

[0014] Preferably, S2 specifically includes: calculating the standard deviation of all frame images in time under each viewpoint based on the temporal multi-view live neural data to obtain the multi-view data.

[0015] Preferably, the formula for calculating the neuron position data from each viewpoint in S2 is:

[0016]

[0017] in, This represents the neuron location data from the current viewpoint, that is, the data corresponding to a single viewpoint in the multi-view data. This is a two-dimensional image of the i-th frame from the first perspective. t represents the temporal mean of all frames of 2D images from the current perspective, and t represents the total number of data frames.

[0018] Preferably, S3 specifically includes:

[0019] Sparse sampling is performed on the multi-view data to obtain sparse view data;

[0020] Extract the PSF_sparse data corresponding to the sparse viewpoint from the PSF, reconstruct the sparse viewpoint data, and obtain the three-dimensional spatial location data.

[0021] Preferably, S5 specifically includes:

[0022] For each frame in the temporal multi-view neural data D, sparse view sampling is performed and then fed into SeReNet along with the corresponding PSF_sparse to complete the light field reconstruction.

[0023] Preferably, the SeReNet algorithm is used for reconstruction in step S3.

[0024] Preferably, the neuron segmentation step in S4 includes:

[0025] S41: Use UNet3D to perform preliminary detection of neuron candidate point coordinates p1, feature vector f1, neuron radius s1, and binarization mask u1 on the three-dimensional spatial location data std_volume;

[0026] S42: For each candidate neuron, extract the feature vectors of its 5 closest neighbors and feed them into EXNet for clustering and deduplication.

[0027] S43: Based on the clustering result index, extract the corresponding neuron coordinates, radius and mask from p1, s1 and u1;

[0028] S44: Traverse the indices of all neurons obtained after clustering, multiply the corresponding mask by the corresponding index, and put the result into the all-zero matrix mask_volume to obtain the final three-dimensional mask volume of all neurons.

[0029] A three-dimensional neural signal extraction system based on a four-dimensional light field includes:

[0030] The data acquisition module is used to acquire temporal multi-view live neural data and corresponding PSF data under a four-dimensional light field.

[0031] The distribution analysis module, connected to the data acquisition module, is used to analyze the neuron distribution based on the temporal multi-view live neural data D, and obtain multi-view data representing the neuron locations.

[0032] The image reconstruction module, connected to the distribution analysis module, is used to reconstruct the multi-view data based on the PSF data to obtain the three-dimensional spatial location data of the neuron distribution.

[0033] The segmentation module, connected to the image reconstruction module, is used to segment neurons based on the three-dimensional spatial location data to obtain the segmentation mask of each neuron.

[0034] The image reconstruction module is also used to perform full-frame-rate frame-by-frame reconstruction of the temporal multi-view live neural data D based on the PSF data to obtain a temporal three-dimensional volume stack, and to obtain the neuron activity matrix of each neuron in the temporal three-dimensional volume stack based on the segmentation mask volume.

[0035] Preferably, it also includes a sparse sampling module, which is used to perform sparse perspective sampling on the multi-view data to obtain sparse perspective data; extract the PSF_sparse data corresponding to the sparse perspective from the PSF data, and reconstruct the sparse perspective data to obtain the three-dimensional spatial position data.

[0036] Preferably, the sparse sampling module is further configured to perform viewpoint sampling on the PSF data and the temporal multi-view live neural data D before performing the full frame rate frame-by-frame reconstruction.

[0037] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a three-dimensional neural signal extraction method based on four-dimensional light field. It uses light field microscopy to replace the traditional microscopic imaging mode, and four-dimensional light field information can be obtained in a single exposure. This four-dimensional information can be reconstructed into a three-dimensional sample volume at the corresponding time through light field reconstruction algorithm. Compared with the traditional scanning imaging mode, the imaging efficiency is improved by more than an order of magnitude. Moreover, since the light intensity is uniformly distributed in the sample during exposure, the local light dose of the sample in the scanning imaging can be diluted, and the overall phototoxicity is reduced by more than 90%.

[0038] Furthermore, since the light field images captured by this invention need to undergo post-processing light field reconstruction algorithms for neuronal activity extraction, accelerating the post-processing speed is crucial for analyzing neuronal activity. This invention introduces a sparse sampling strategy to downsample the original light field data by more than four times in the viewpoint dimension, thereby accelerating the process while preserving the parallax information between viewpoints. The computation and subsequent reconstruction process improve the efficiency of neuroanalysis. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0040] Figure 1 This is a schematic diagram of a three-dimensional neural signal extraction method based on a four-dimensional light field, provided as an embodiment of the present invention.

[0041] Figure 2This is a schematic diagram of the sparse view sampling method in an embodiment of the present invention.

[0042] Figure 3 This is a schematic diagram of a three-dimensional neural signal extraction system based on a four-dimensional light field, provided as an embodiment of the present invention. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] Example 1

[0045] like Figure 1 This invention discloses a method for extracting three-dimensional neural signals based on a four-dimensional light field, comprising the following steps:

[0046] S1: Acquire temporal multi-view live neural data D under a four-dimensional light field.

[0047] S2: Analyze the neuron distribution based on the temporal multi-view live neural data D, and obtain multi-view data representing the neuron locations.

[0048] S3: Reconstruct the multi-view data based on PSF to obtain the three-dimensional spatial location data of the neuron distribution.

[0049] S4: Perform neuron segmentation based on the three-dimensional spatial location data to obtain the segmentation mask of each neuron.

[0050] S5: Based on PSF, the temporal multi-view live neural data D is reconstructed frame by frame at full frame rate to obtain a temporal three-dimensional volume stack, and the neuron activity matrix of each neuron in the temporal three-dimensional volume stack is obtained according to the segmentation mask volume.

[0051] In this embodiment, the present invention utilizes temporal data from multiple perspectives in a four-dimensional light field to achieve precise localization and segmentation of neurons, and to accurately extract the collective activity of multiple neurons.

[0052] For S1, data acquisition can be performed using a light field microscopy system to capture temporal multi-view live neural data D, where the shape of D is... ,in 81 represents the total number of frames captured in the time sequence, and 81 represents the number of angles of view captured by the light field microscopy system. For each frame, the pixel height of the 2D image at each viewpoint, The width of the 2D image is pixels for each frame and each viewpoint.

[0053] To further implement the above technical solution, the calculation formula for neuron distribution data from each perspective in S2 is as follows:

[0054]

[0055] in, This is a two-dimensional image of the i-th frame from the first perspective. t represents the temporal mean of all frames of 2D images from the current perspective, and t represents the total number of data frames.

[0056] In this embodiment, since the pixel intensity of the flickering neurons in the multi-view live neural data captured by the light field microscopy system is higher than the pixel intensity of the background such as blood vessels and cerebrospinal fluid in the live neural data, in order to obtain the spatial position of each neuron in the live neural data, the standard deviation (STD) of all frames in time under each viewpoint is calculated to achieve precise positioning.

[0057] In another embodiment, sparse view sampling is used to accelerate the extraction process. Figure 2 This is a schematic diagram illustrating the selection of sparse viewpoints, based on... Figure 2 The viewpoint shown is obtained by sparse sampling in multi-view data to obtain sparse viewpoint data; the data corresponding to the sparse viewpoint is extracted from the PSF, i.e., PSF_sparse data, and the sparse viewpoint data is reconstructed to obtain three-dimensional spatial position data. The red area represents the selected viewpoint.

[0058] Select from the 81 perspectives in the example, such as Figure 2 The 13 viewpoints with significant parallax shown were analyzed using a formula based on neuron distribution data. The standard deviation for each viewpoint was calculated to obtain the shape shown. Multi-perspective data .

[0059] Then, the multi-view data... To carry out reconstruction, in order to... To reconstruct the three-dimensional spatial location information of neurons from multi-view information, it is necessary to... According to the optical point spread function (PSF) of the light field microscopy system, the shape is... (Where 81 represents the number of viewpoints, 61 represents the depth, and 2 represents the offsets along the height and width directions) is reconstructed into a shape as of A three-dimensional volume is generated for subsequent three-dimensional segmentation. This invention uses the open-source light field image reconstruction algorithm SeReNet to complete a high-fidelity light field image reconstruction task. The reconstruction speed is closely related to the number of viewpoints in the light field data; therefore, extracting data from PSF and S2... The corresponding sparse viewpoint is used to obtain PSF_sparse for 3D light field reconstruction to accelerate the reconstruction process.

[0060] Specifically, the SeReNet reconstruction process is as follows:

[0061] 1) Each two-dimensional angle information is copied d times, where d is the depth of the PSF, resulting in d layers of three-dimensional angle information, with 13 light field angles per layer. Its shape is Meanwhile, while acquiring images, the light field microscopy system can obtain the PSF, which contains offset information at various angles in each depth layer.

[0062] 2) Based on the offset of the corresponding layer and viewpoint of the PSF, it is obtained by translating it a certain distance in the horizontal and vertical directions. Its shape is ;

[0063] 3) The input is sequentially fed into four convolutional layers, one 2x upsampling layer, two convolutional layers, one 2x upsampling layer, three convolutional layers, and one 1.25x upsampling layer, ultimately resulting in a shape of... of Three-dimensional volume.

[0064] To further implement the above technical solution, the neuron segmentation step in S4 includes:

[0065] S41: Use UNet3D to perform preliminary neuron detection on the three-dimensional spatial location data std_volume. In this invention, the U-Net3D module adopts a symmetric encoder-decoder structure and includes the following main components:

[0066] Encoder path (ContextPathway):

[0067] It contains multiple downsampling levels (Level 1 to Level 4), each level including: a set of convolutional layers (Conv3D), instance normalization layers (InstanceNorm3D), LeakyReLU activation function, and residual connections (ResidualConnection); it uses convolutions with strides of (2,2,1) to achieve spatial dimension downsampling, and doubles the number of channels in the output feature map in turn.

[0068] Bottleneck layer:

[0069] It consists of a further downsampled convolutional layer and a residual structure, used to extract high-dimensional semantic information and serve as the starting point for the decoder.

[0070] Decoder Path (Localization Pathway):

[0071] Upsampling is performed using nearest neighbor interpolation. Each decoding layer fuses skip connection information from the corresponding encoding layer, resulting in an output resolution consistent with the original input. Multi-scale features are preserved during the decoding process to assist subsequent clustering analysis. After passing std_volume through the above structure, the network aggregates low-dimensional and high-dimensional features from the input 3D volume, performs coarse instance segmentation of the foreground and background in the 3D volume, and outputs the coordinates p1 of the candidate neuron point, feature vector f1, neuron radius s1, and binarized mask u1 for subsequent processing.

[0072] S42: For each neuron candidate point feature vector Extract the feature vectors of the 5 nearest points in its neighborhood. The data is fed into EXNet for clustering and deduplication. During the process, the semantic relevance score f between the candidate point and its neighbors is calculated. EXNet is a deep learning network with stacked fully connected layers and LeakyReLU activation functions. Its final output is a 5-dimensional binary classification score. If the score of a certain dimension is higher than the threshold of 0.9, it is considered that this point and the candidate point belong to the same neuron among the neighbors.

[0073] S43: Based on the clustering result index, extract the corresponding neuron coordinates, radius and mask from the neuron candidate point coordinates p1, neuron radius s1 and binarization mask u1.

[0074] S44: Traverse the indices of all neurons obtained after clustering, multiply the corresponding mask by the corresponding index, and put the result into the all-zero matrix mask_volume to obtain the final three-dimensional mask volume of all neurons.

[0075] Example 2

[0076] like Figure 3 Based on the same inventive concept, this invention discloses a three-dimensional neural signal extraction system based on a four-dimensional light field, comprising:

[0077] The data acquisition module is used to acquire temporal multi-view live neural data and corresponding PSF data under a four-dimensional light field.

[0078] The distribution analysis module, connected to the data acquisition module, is used to analyze neuron distribution based on temporal multi-view live neural data D and obtain multi-view data representing neuron locations.

[0079] The image reconstruction module, connected to the distribution analysis module, is used to reconstruct multi-view data based on PSF data to obtain three-dimensional spatial location data of neuron distribution.

[0080] The segmentation module, connected to the image reconstruction module, is used to segment neurons based on three-dimensional spatial location data to obtain the segmentation mask of each neuron.

[0081] The image reconstruction module is also used to perform full-frame-rate frame-by-frame reconstruction of the temporal multi-view live neural data D based on the PSF data, to obtain a temporal three-dimensional volume stack, and to obtain the neuron activity matrix of each neuron in the temporal three-dimensional volume stack based on the segmentation mask volume.

[0082] To further implement the above technical solution, a sparse sampling module is also included. The sparse sampling module is used to perform sparse perspective sampling on multi-view data to obtain sparse perspective data; extract the PSF_sparse data corresponding to the sparse perspective from the PSF data, reconstruct the sparse perspective data, and obtain three-dimensional spatial position data.

[0083] Furthermore, the sparse sampling module is also used to perform viewpoint sampling on the PSF data and the temporal multi-view live neural data D before performing full-frame-rate frame-by-frame reconstruction.

[0084] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0085] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for extracting three-dimensional neural signals based on a four-dimensional light field, characterized in that, Includes the following steps: S1: Acquire temporal multi-view live neural data D and PSF data under a four-dimensional light field; S2: Analyze the neuron distribution based on the temporal multi-view live neural data D, and obtain multi-view data representing the neuron locations; S3: Reconstruct the multi-view data based on the PSF data to obtain the three-dimensional spatial location data of the neuron distribution; S4: Perform neuron segmentation based on the three-dimensional spatial location data to obtain the segmentation mask of each neuron; S5: Based on the PSF data, the temporal multi-view live neural data D is reconstructed frame by frame at full frame rate to obtain a temporal three-dimensional volume stack, and the neuron activity matrix of each neuron in the temporal three-dimensional volume stack is obtained according to the segmentation mask volume.

2. The method for extracting three-dimensional neural signals based on a four-dimensional light field according to claim 1, characterized in that, S2 specifically includes: based on the temporal multi-view live neural data, calculating the temporal standard deviation of all frame images under each viewpoint to obtain the multi-view data.

3. The method for extracting three-dimensional neural signals based on a four-dimensional light field according to claim 2, characterized in that, The formula for calculating the neuron position data from each viewpoint in S2 is as follows: ； in, This represents the neuron location data from the current perspective. This is the i-th two-dimensional image from the current perspective. t represents the temporal mean of all frames of 2D images from the current perspective, and t represents the total number of data frames.

4. The method for extracting three-dimensional neural signals based on a four-dimensional light field according to claim 1, characterized in that, S3 specifically includes: Sparse sampling is performed on the multi-view data to obtain sparse view data; Extract the PSF_sparse data corresponding to the sparse viewpoint from the PSF data, and reconstruct the sparse viewpoint data to obtain the three-dimensional spatial location data.

5. A method for extracting three-dimensional neural signals based on a four-dimensional light field according to claim 1 or 4, characterized in that, S5 specifically includes: For each frame in the temporal multi-view live neural data D, sparse view sampling is performed and then fed into SeReNet along with the corresponding PSF_sparse to complete the light field reconstruction.

6. The method for extracting three-dimensional neural signals based on a four-dimensional light field according to claim 1, characterized in that, The S3 uses the SeReNet algorithm for reconstruction.

7. The method for extracting three-dimensional neural signals based on a four-dimensional light field according to claim 1, characterized in that, The neuron segmentation step in S4 includes: S41: Use UNet3D to perform preliminary detection on the three-dimensional spatial location data std_volume to obtain the coordinates p1 of the neuron candidate point, the feature vector f1, the neuron radius s1, and the binarization mask u1. S42: For each candidate neuron, extract the feature vectors of multiple points in its neighborhood, and perform clustering and deduplication. S43: Based on the clustering result index, extract the corresponding neuron coordinates, radius and mask from p1, s1 and u1; S44: Traverse the indices of all neurons obtained after clustering, multiply the corresponding mask by the corresponding index, and put the result into the all-zero matrix mask_volume to obtain the final three-dimensional mask volume of all neurons.

8. A three-dimensional neural signal extraction system based on a four-dimensional light field, characterized in that, include: The data acquisition module is used to acquire temporal multi-view live neural data and corresponding PSF data under a four-dimensional light field. The distribution analysis module, connected to the data acquisition module, is used to analyze the neuron distribution based on the temporal multi-view live neural data D, and obtain multi-view data representing the neuron positions. An image reconstruction module, connected to the distribution analysis module, is used to reconstruct the multi-view data based on the PSF data to obtain three-dimensional spatial location data of neuron distribution; The segmentation module, connected to the image reconstruction module, is used to segment neurons based on the three-dimensional spatial location data to obtain the segmentation mask of each neuron. The image reconstruction module is also used to perform full-frame-rate frame-by-frame reconstruction of the temporal multi-view live neural data D based on the PSF data to obtain a temporal three-dimensional volume stack, and to obtain the neuron activity matrix of each neuron in the temporal three-dimensional volume stack based on the segmentation mask volume.

9. A three-dimensional neural signal extraction system based on a four-dimensional light field according to claim 8, characterized in that, It also includes a sparse sampling module, which is used to perform sparse perspective sampling on the multi-view data to obtain sparse perspective data; extract the PSF_sparse data corresponding to the sparse perspective from the PSF data, and reconstruct the sparse perspective data to obtain the three-dimensional spatial position data.

10. A three-dimensional neural signal extraction system based on a four-dimensional light field according to claim 9, characterized in that, The sparse sampling module is also used to perform viewpoint sampling on the PSF data and the temporal multi-view live neural data D before performing the full frame rate frame-by-frame reconstruction.

Images