Wide-view-field multi-photon microscopic imaging method, device and equipment

Abstract

The invention relates to the technical field of optical microscopic imaging, in particular to a wide-view-field multi-photon microscopic imaging method, device and equipment, and the method comprises the steps: obtaining training data collected by a multi-photon microscope according to a rolling sub-sampling scanning strategy, the strategy covers the entire field of view by successively and longitudinally moving a periodic scan path of high frame rate, low spatial sampling rate. Preprocessing such as normalization, time sequence registration or space division is carried out on the data, and a mask matrix is generated; and performing cross supervision by using the complementary region to construct a self-supervised training sample and a pseudo tag. And training a three-dimensional convolutional neural network based on the training samples and the pseudo labels, and performing reasoning reconstruction on the rolling sub-sampling data by using the trained network to obtain a target image sequence. Therefore, the problem that in the prior art, the high spatial sampling density causes the low frame rate, and the high frame rate needs the small imaging area or the low spatial sampling rate, that is, the imaging view field and the imaging temporal-spatial resolution are difficult to consider at the same time is solved.

Description

Technical Field

[0001] This application relates to the field of optical microscopy imaging technology, and in particular to a wide field-of-view multiphoton microscopy imaging method, apparatus and equipment. Background Technology

[0002] Multiphoton microscopy is an important means of observing the dynamics of organisms in vivo. However, due to physical limitations such as the mechanical inertia of the galvanometer, tissue scattering, and noise, increasing the sampling density of related technologies will significantly reduce the frame rate. Increasing the frame rate often requires shrinking the imaging area or sacrificing spatial resolution, making it difficult to simultaneously balance imaging speed, imaging field of view, and imaging quality. Summary of the Invention

[0003] This application provides a wide field-of-view multiphoton microscopy imaging method, apparatus, and device to solve the technical challenge of achieving both wide field of view and high spatiotemporal resolution in wide field-of-view multiphoton microscopy imaging.

[0004] The first aspect of this application provides a wide-field-of-view multiphoton microscopy imaging method, comprising the following steps: acquiring training data collected by a multiphoton microscope according to a rolling subsampling scanning strategy, wherein the rolling subsampling scanning strategy includes, within multiple cycles of the multiphoton microscope performing rolling subsampling scanning along a target direction, the scanning paths of adjacent cycle image frames are staggered sequentially in the target direction, and the scanning paths of all cycle image frames cover all points in the entire field of view; performing at least one preprocessing step of normalization, temporal registration, and spatial partitioning on the training data; generating a mask matrix based on the preprocessed data; forming self-supervised training samples through cross-supervision of complementary regions of the mask matrix; constructing pseudo-labels based on the spatial complementary relationships of image frames; training a pre-constructed three-dimensional convolutional neural network based on the self-supervised training samples and pseudo-labels; and using the trained three-dimensional convolutional neural network to infer and reconstruct the rolling subsampling data to obtain an image sequence, wherein the multiphoton microscope is controlled to collect rolling subsampling data according to the rolling subsampling scanning strategy.

[0005] Optionally, in one embodiment of this application, both the training data and the subsampling data include prior information on the scanning path, neuronal spatial location morphology information, and neural activity firing time information during the multiphoton microscopy scanning process.

[0006] Optionally, in one embodiment of this application, the formula for calculating the mask matrix is: X=Y⊙M Where X represents subsampling observations, Y represents complete neural activity imaging data, including neural calcium signal imaging, neural voltage signal imaging, and neurochemical signal imaging, and ⊙ represents element-wise multiplication. Represents a binary sampling mask; The formula for calculating the complementary region of the mask matrix is:

[0007]

[0008] in, Represents an even-numbered column index. Represents an index for odd-numbered columns. Basic sampling mode (e.g., period is 1, period is 2) Triangle or rectangle pattern, (for subsampling step size) Indicator Frame pixels Whether it was sampled, Represents the even-numbered column mask. .

[0009] Optionally, in one embodiment of this application, the complementary regions of the mask matrix satisfy a spatial complementarity condition, wherein the spatial complementarity condition is:

[0010] in, An indicator function representing the complete sampling. It represents the empty set.

[0011] Optionally, in one embodiment of this application, the reconstruction target of the three-dimensional convolutional neural network is a parameterized reconstruction function, through which the function enables... The reconstruction function is:

[0012] in, It provides a time-averaged image with global spatial context. It is the reconstructed complete data. Indicates by parameters The mapping function for control.

[0013] Optionally, in one embodiment of this application, the three-dimensional convolutional neural network includes an encoder, a decoder, and an output layer, wherein the encoder performs stepwise downsampling and feature extraction on the input data, the decoder performs stepwise upsampling on the extracted features, the output layer outputs the image sequence, and there are skip connections between corresponding layers of the encoder and the decoder. Layered mathematical expression is as follows:

[0014] in, Representing the Layer encoder,

[0015] The decoder's first Layered mathematical expression is as follows:

[0016] in, Representing the Layer decoder, where [ , [] represents splicing; The mathematical expression for the output layer is:

[0017] in, Represents the final reconstruction result. This represents a single-pixel convolution kernel.

[0018] Optionally, in one embodiment of this application, the size of the l-th layer of the three-dimensional convolutional neural network is... × × The convolution kernel extracts temporal features, among which, The temporal features extracted by the convolutional kernel are:

[0019] in, Represents the sense of space. Represents the temporal scale of the convolution kernel. Representing the The time pooling factor of the layer; The spatial features extracted by the convolutional kernel are:

[0020] in, Represents the spatial size of the convolution kernel. Represents the spatial pooling factor.

[0021] Optionally, in one embodiment of this application, the three-dimensional convolutional neural network calculates the training loss using a total reconstruction loss function during training, and updates the network parameters of the three-dimensional convolutional neural network based on the training loss, wherein the total reconstruction loss function is:

[0022]

[0023]

[0024] in, Represents the total reconstruction losses. Represents the combination of L1 and L2 losses. Represents the target in the left-hand domain. Represents the target in the right-hand domain.

[0025] A second aspect of this application provides a wide-field-of-view multiphoton microscopy imaging device, comprising: an acquisition module for acquiring training data collected by a multiphoton microscope according to a rolling subsampling scanning strategy, wherein the rolling subsampling scanning strategy includes, within multiple cycles of the multiphoton microscope performing rolling subsampling scanning along a target direction, the scanning paths of adjacent cycle image frames are staggered sequentially in the target direction, and the scanning paths of all cycle image frames cover all points in the entire field of view; a processing module for performing at least one preprocessing operation on the training data, including normalization, temporal registration, and spatial partitioning, generating a mask matrix based on the preprocessed data, forming self-supervised training samples through cross-supervision of complementary regions of the mask matrix, and constructing pseudo-labels based on the spatial complementary relationships of the image frames; and an output module for training a pre-constructed three-dimensional convolutional neural network based on the self-supervised training samples and pseudo-labels, and using the trained three-dimensional convolutional neural network to infer and reconstruct the rolling subsampling data to obtain an image sequence, wherein the multiphoton microscope is controlled to acquire rolling subsampling data according to the rolling subsampling scanning strategy.

[0026] Optionally, in one embodiment of this application, both the training data and the subsampling data include prior information on the scanning path, spatial location and topography information, and neural activity firing time information during the multiphoton microscopy scanning process.

[0027] Optionally, in one embodiment of this application, the formula for calculating the mask matrix is: X=Y⊙M Where X represents subsampling observations, Y represents complete imaging volume data, including neural calcium signal imaging, neural voltage signal imaging, neurochemical substance signal imaging, etc., and ⊙ represents element-wise multiplication. Represents a binary sampling mask; The formula for calculating the complementary region of the mask matrix is:

[0028]

[0029] in, Represents an even-numbered column index. Represents an index for odd-numbered columns. Basic sampling mode, Indicator Frame pixels Whether it was sampled, Represents the even-numbered column mask. .

[0030] Optionally, in one embodiment of this application, the complementary regions of the mask matrix satisfy a spatial complementarity condition, wherein the spatial complementarity condition is:

[0031] in, An indicator function representing the complete sampling. It represents the empty set.

[0032] Optionally, in one embodiment of this application, the reconstruction target of the three-dimensional convolutional neural network is a parameterized reconstruction function, through which the function enables... The reconstruction function is:

[0033] in, It provides a time-averaged image with global spatial context. It is the reconstructed complete data. Indicates by parameters The mapping function for control.

[0034] Optionally, in one embodiment of this application, the three-dimensional convolutional neural network includes an encoder, a decoder, and an output layer, wherein the encoder performs stepwise downsampling and feature extraction on the input data, the decoder performs stepwise upsampling on the extracted features, the output layer outputs the image sequence, and there are skip connections between corresponding layers of the encoder and the decoder. Layered mathematical expression is as follows:

[0035] in, Representing the Layer encoder, Representing the Layer encoder; The decoder's first Layered mathematical expression is as follows:

[0036] in, Representing the Layer decoder, where [ , [] represents splicing; The mathematical expression for the output layer is:

[0037] in, Represents the final reconstruction result. This represents a single-pixel convolution kernel.

[0038] Optionally, in one embodiment of this application, the size of the l-th layer of the three-dimensional convolutional neural network is... × × The convolution kernel extracts temporal features, among which, The temporal features extracted by the convolutional kernel are:

[0039] in, Represents the sense of space. Represents the temporal scale of the convolution kernel. Representing the The time pooling factor of the layer; The spatial features extracted by the convolutional kernel are:

[0040] in, Represents the spatial size of the convolution kernel. Represents the spatial pooling factor.

[0041] Optionally, in one embodiment of this application, the three-dimensional convolutional neural network calculates the training loss using a total reconstruction loss function during training, and updates the network parameters of the three-dimensional convolutional neural network based on the training loss, wherein the total reconstruction loss function is:

[0042]

[0043]

[0044] in, Represents the total reconstruction losses. Represents the combination of L1 and L2 losses. Represents the target in the left-hand domain. Represents the target in the right-hand domain.

[0045] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the wide field-of-view multiphoton microscopy imaging method as described in the above embodiments.

[0046] Therefore, this application includes the following beneficial effects: First, this application controls a multiphoton microscope to acquire training data according to a rolling subsampling scanning strategy. This rolling subsampling scanning strategy involves sequentially staggering the scanning paths of adjacent periodic image frames along the target direction within multiple scanning cycles, ensuring that all periodic image frames collectively cover all points in the entire field of view. This achieves high frame rate data acquisition while maintaining complete field-of-view coverage, improving imaging speed without sacrificing spatial coverage of the field of view. Second, the acquired training data undergoes at least one preprocessing step, including normalization, temporal registration, and spatial partitioning, and a mask matrix is ​​generated based on the preprocessing results. The complementary regions of the mask matrix are then utilized... Cross-supervision generates self-supervised training samples, and pseudo-labels are constructed based on the spatial complementarity of image frames. Reliable training signals are generated through self-supervision, which not only preserves important spatiotemporal information but also avoids dependence on high-quality labeled data, achieving high-fidelity training. Then, a pre-constructed three-dimensional convolutional neural network is trained based on the self-supervised training samples and pseudo-labels. The trained network is then used to infer and reconstruct subsampled data acquired by a rolling subsampled scanning strategy to obtain high-quality image sequences. This effectively restores the spatial details and dynamic signals of the subsampled images, thereby solving the technical challenge of achieving both wide field of view and high spatiotemporal resolution in wide-field multiphoton microscopy.

[0047] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0048] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a comparative schematic diagram of two-photon galvanometer scanning methods according to embodiments of this application; Figure 2 This is a flowchart of a wide-field multiphoton microscopy imaging method according to an embodiment of this application; Figure 3 This is a schematic diagram of the scanning path, neuronal spatial morphology, and firing time of the rolling subsampling scan and synchronous recording according to an embodiment of this application; Figure 4 This is a schematic diagram of the subsampling reconstruction results according to an embodiment of this application; Figure 5 This is a comparison diagram of neuronal activity trajectories according to embodiments of this application; Figure 6 This is a block diagram illustrating the overall architecture and workflow according to an embodiment of this application; Figure 7 This is an example diagram of a wide field-of-view multiphoton microscopy imaging apparatus according to an embodiment of this application; Figure 8 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0049] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0050] Multiphoton microscopy, including multiphoton fluorescence microscopy, harmonic microscopy, and coherent Raman microscopy, effectively overcomes the influence of scattering from biological tissues by using long-wavelength ultrashort pulse excitation, making it an important technique for in vivo imaging and a crucial method for dynamic observation of living organisms. However, multiphoton microscopy typically requires point-by-point scanning imaging. The mechanical inertia of the scanning elements (such as galvanometers) affects the frame rate of wide-field imaging, resulting in a low frame rate under high spatial sampling density. Conversely, a high frame rate requires a small imaging area or a low spatial sampling rate. This presents a challenge of simultaneously achieving optimal imaging field of view and spatiotemporal resolution, making it difficult to meet practical application requirements.

[0051] On the other hand, wide-field microscopy is of great significance for systems biology research. For example, in systems neuroscience research, wide-field microscopy can be used to dynamically observe neural networks across brain regions, enhancing our understanding of brain information processing mechanisms; in systems immunology research, wide-field microscopy can be used to observe the long-range migration dynamics of immune cells, etc. Recently, various mesoscale microscopy and other wide-field microscopy techniques have emerged, among which the contradiction between imaging field of view and imaging spatiotemporal resolution has become increasingly prominent.

[0052] To achieve wide-field-of-view multiphoton microscopy, random scanning multiphoton microscopy has been developed. This technique involves first performing a full-field-of-view traversal scan to identify several target sub-regions, and then performing skip scanning only on these sub-regions (e.g., doi: 10.7554 / eLife.14472 and US Patents 10,295,811 and 10,901,194), effectively increasing the imaging frame rate. However, it should be noted that this method sacrifices the imaging area for a high frame rate and is unsuitable for applications requiring wide field of view and high spatiotemporal resolution.

[0053] Therefore, achieving microscopic imaging with both wide field of view and high spatiotemporal resolution is a technical challenge that urgently needs to be overcome in the field of wide field of view multiphoton microscopy.

[0054] The following description, with reference to the accompanying drawings, describes a wide-field-of-view multiphoton microscopy imaging method, apparatus, and device according to embodiments of this application. Addressing the problems mentioned in the background art, this application provides a wide-field-of-view multiphoton microscopy imaging method. In this method, firstly, the application controls a multiphoton microscope to acquire training data according to a rolling subsampling scanning strategy. This rolling subsampling scanning strategy includes, within multiple scanning cycles, sequentially staggering the scanning paths of adjacent period image frames along the target direction, so that all period image frames collectively cover all points in the entire field of view. This achieves high frame rate data acquisition while ensuring complete field-of-view coverage, improving imaging speed without sacrificing spatial coverage of the field of view. Secondly, the acquired training data undergoes at least one preprocessing step, including normalization, temporal registration, and spatial partitioning, and a mask moment is generated based on the preprocessing results. The system utilizes the cross-supervision of complementary regions of the mask matrix to form self-supervised training samples, and constructs pseudo-labels based on the spatial complementarity of image frames. Reliable training signals are generated through self-supervision, which not only preserves important spatiotemporal information but also avoids dependence on high-quality labeled data, achieving high-fidelity training. Then, a pre-constructed three-dimensional convolutional neural network is trained based on the self-supervised training samples and pseudo-labels. The trained network is then used to infer and reconstruct subsampled data acquired using a rolling subsampled scanning strategy to obtain high-quality image sequences. This effectively restores the spatial details and dynamic signals of the subsampled images, thereby solving the technical challenge of achieving both wide field of view and high spatiotemporal resolution in wide-field multiphoton microscopy.

[0055] In related technologies, multiphoton microscopy achieves two-dimensional spatial point scanning by driving a biaxial galvanometer, the specific process of which is as follows: Figure 1 As shown. The two mirrors in the biaxial mirror system control the excitation beam at... , Movement in two-dimensional space: Galvanometer 1 is responsible for Directional scanning, galvanometer 2 is responsible for Directional scan. Reference Figure 1 The traditional scanning path in the image requires the following to complete the scanning of the entire 2D image: The galvanometer 1 in the direction is scanned multiple times, while the one responsible for The galvanometer 2 scans once in the direction, therefore defining... The direction is the fast axis. The direction is the slow axis. Because... The scanning speed in a given direction is usually the fastest speed achievable by the galvanometer; therefore, the frame imaging speed depends on the size of the imaging field of view and the spatial sampling rate, i.e., the number of rows contained in each frame. This is achieved by artificially introducing... Oriented subsampling scanning, which increases the frame rate at the expense of spatial sampling rate and spatial resolution, can achieve faster two-dimensional microscopic imaging.

[0056] To make up for Information lost in directional subsampling imaging is typically recovered using deep generative models trained on large amounts of experimental data. However, even though deep generative models can learn rich morphological features from large datasets, image reconstruction at high subsampling magnifications remains challenging. Due to information sparsity, generative models often struggle to converge or are prone to artifacts, resulting in unsatisfactory reconstruction results.

[0057] To better achieve the restoration of subsampled images and provide as much prior information as possible for the deep generation model, this application proposes a wide-field-of-view multiphoton microscopy imaging method. This method covers the entire field of view by successively moving a periodic scanning path with a high frame rate and low spatial sampling rate longitudinally. At the same time, it records the prior information of the scanning path, spatial position and morphology information, and neural activity firing time information during the multiphoton microscopy scanning process, providing rich prior information for the deep generation model, thereby improving the quality and reliability of subsampled image reconstruction.

[0058] Specifically, Figure 2 This is a schematic flowchart of a wide-field multiphoton microscopy imaging method provided in an embodiment of this application.

[0059] like Figure 2 As shown, this wide-field-of-view multiphoton microscopy imaging method includes the following steps: In step S101, training data collected by the multiphoton microscope according to the rolling subsampling scanning strategy is acquired. The rolling subsampling scanning strategy includes multiple cycles of rolling subsampling scanning along the target direction by the multiphoton microscope, in which the scanning paths of adjacent cycle image frames are staggered in the target direction, and the scanning paths of all cycle image frames cover all points of the entire field of view.

[0060] In this context, subsampling refers to acquiring data from only a portion of the spatial or temporal points within a complete scan or imaging dataset, rather than acquiring data from every pixel or every temporal point. The rolling subsampling scanning strategy is an acquisition method in multiphoton microscopy, and in this application, it represents multi-cycle scanning of the field of view along the target direction at subsampling intervals. An image frame is a two-dimensional or three-dimensional image generated in a single scan, and in this application, it represents single-sampling data acquired by the multiphoton microscope. The scanning path is the trajectory of the multiphoton microscope in space, and in this application, it represents the trajectory of the microscope beam moving along the target direction, used to generate the image frame. Subsampling is an acquisition method, and in this application, it represents a method of acquiring only a portion of the pixels in the field of view to improve acquisition speed. The complete field of view is the entire sampling range of the microscope imaging area, and in this application, it represents all points in the entire imaging area that can be covered by multiple cycle scans. The target direction represents the direction in which the scanning paths of adjacent scan cycles offset in the rolling subsampling scanning strategy. This direction can be the X direction, the Y direction, or any direction along a preset path, used to ensure that the scanning paths of multiple cycles are staggered sequentially in the target direction to cover the entire field of view.

[0061] Understandably, this application employs a rolling subsampling scanning strategy for multiphoton microscopy data acquisition, which can significantly improve the imaging frame rate while ensuring coverage of the entire field of view. The sequentially staggered scanning paths in multiple cycles ensure that only partial spatial information is acquired per frame, but complete spatial and temporal information can be obtained through cycle accumulation, thus providing features for subsequent self-supervised reconstruction. In addition, subsampling acquisition reduces the number of scanning points per frame, alleviating the scanning load of the galvanometer and the energy consumption of the optical system. At the same time, the complementary use of staggered scanning paths achieves high-fidelity image restoration, balancing high frame rate and imaging quality.

[0062] In one embodiment of this application, both the training data and the subsampling data include prior information on the scanning path, spatial location and morphology information of neurons, and information on the firing time of neural activity during the multiphoton microscopy scanning process.

[0063] Among them, the scanning path prior information represents the spatial position and scanning sequence information of each scanning point during the rolling subsampling scanning process of the multiphoton microscope; the neuron spatial position morphology information is the spatial distribution and morphological characteristics of the neuron in the three-dimensional imaging field of view, which represents the structural prior of the neuron in this application; the neuron firing time information is the dynamic signal change and firing time sequence of the neuron during the acquisition process, which represents the temporal prior of the neuron activity in this application.

[0064] It is understood that this application simultaneously includes training data and subsampled data containing prior information on scanning paths, spatial location and morphology information of neurons, and firing time information of neurons. This enables the three-dimensional convolutional neural network to fully utilize multi-source priors in space and time during the reconstruction process, thereby improving its ability to recover sparse subsampled data. The combination of multi-dimensional information enhances the model's ability to perceive the morphological features of neurons and improves the accuracy of capturing dynamic signals.

[0065] like Figure 3 As shown, the relevant technology completes one scan cycle, while The double-roll subsampling scan is then completed. This allows for a frame rate increase of up to [number] cycles. Times. In subsampling In each cycle, the scan path of each cycle is offset downwards relative to the previous cycle, making... A single scan cycle can completely cover all points in the entire field of view. All image frames in a given period are treated as a three-dimensional matrix, serving as input to the depth generation model. This input includes not only the spatial location and topography information of neurons in the current field of view and neuron firing time information, but also the corresponding prior information of the scanning path. Thus, under the joint guidance of multiple prior information, higher-quality subsampled image restoration is achieved.

[0066] In step S102, the training data is preprocessed by at least one of normalization, temporal registration and spatial partitioning. A mask matrix is ​​generated based on the preprocessed data. Self-supervised training samples are formed by cross-supervision of the complementary regions of the mask matrix. Pseudo-labels are constructed based on the spatial complementary relationship of image frames.

[0067] Normalization is used to adjust the numerical scale of the acquired multiphoton microscopy imaging data; temporal registration represents aligning consecutive image frames in the time dimension, which in this application means eliminating temporal misalignment caused by scanning delay or sample motion, so that the dynamic signals of neurons correspond correctly in time; spatial partitioning is the operation of dividing the image into complementary regions or blocks in the spatial dimension; the mask matrix represents a binary matrix of pixel sampling state, which in this application is used to indicate that some pixels in the input data have been sampled and some pixels have not been sampled, thereby providing spatial constraints for the prediction target of the neural network; cross-supervision of complementary regions is a mechanism for training unsampled regions using the sampled information of complementary regions; pseudo-labels are training targets generated based on complementary regions or time averaging.

[0068] Understandably, by preprocessing the training data through normalization, temporal registration, and spatial partitioning, and by constructing complementary regions based on the mask matrix for cross-supervision, generating self-supervised training samples and pseudo-labels, the method in this application enables 3D convolutional neural networks to learn neuron morphology and dynamic signal features without high-quality labeled data. This strategy fully utilizes spatial complementarity and temporal redundancy information, improving the accuracy and stability of subsampled data reconstruction while reducing dependence on a large amount of labeled data. This makes the training process more efficient and adaptable to various imaging conditions, providing a reliable self-supervised learning foundation for wide-field multiphoton microscopy.

[0069] The normalization operation is specifically performed on each image patch according to the scaling factor. Intensity normalization is performed to ensure uniform amplitude of the input data, eliminating the influence of different acquisition conditions on model training and improving the model's convergence speed and stability. The formula is as follows:

[0070] in, It can be the maximum intensity, the 95th percentile, or a fixed constant, depending on the imaging modality.

[0071] In addition, this application employs image patch extraction and data augmentation methods when training the data.

[0072] Image patch extraction represents stacking each volume data. The image is segmented into overlapping 3D image patches, as shown in the following formula:

[0073] in, Represents spatial and temporal indexes. Represents the dimensions of the image patch (default is 48×150×150). Represents the overlap factor (default is 0.25). Represents the image patch stride: in, , , .

[0074] To ensure the integrity of temporal and spatial local information, and to control the coverage relationship between image patches through an overlap factor, this application enhances the quantity and diversity of training samples. For spatial redundancy modes, this application will also... Double to 2 To accommodate data input after left and right splitting.

[0075] Data augmentation involves randomly rotating image patches, such as by 0°, 90°, 180°, and 270°, as well as horizontal / vertical flipping, while ensuring consistency between the input and target data. This increases the diversity of training data and improves the model's generalization ability.

[0076] After preprocessing, a mask matrix is ​​generated based on the training data. This mask matrix represents the sampled pixels and the regions to be predicted, guiding the 3D convolutional neural network to distinguish between usable information in the input data and the regions that need to be reconstructed. Building upon this, this application further constructs self-supervised training samples: cross-supervision is performed using complementary regions of the mask matrix to form the network's training objective. Simultaneously, corresponding pseudo-labels are generated based on the spatial complementarity of image frames to provide the supervision information required for network training.

[0077] In one embodiment of this application, the formula for calculating the mask matrix is: X=Y⊙M Where X represents subsampling observations, Y represents complete imaging volume data, including neural calcium signal imaging, neural voltage signal imaging, neurochemical substance signal imaging, etc., and ⊙ represents element-wise multiplication. Represents a binary sampling mask; The formula for calculating the complementary region of the mask matrix is:

[0078]

[0079] in, Represents an even-numbered column index. Represents an index for odd-numbered columns. Basic sampling mode (e.g., period is 1, period is 2) Triangle or rectangle pattern, (for subsampling step size) Indicator Frame pixels Whether it was sampled, Represents the even-numbered column mask. .

[0080] Among them, the basic sampling mode In this application, a binary sampling mask is used to generate the corresponding frame, defining the sampling rules or weighting patterns at each pixel location. Understandably, by constructing a mask matrix of complementary regions and generating self-supervised training samples and pseudo-labels based on spatial complementarity, this application is able to automatically learn spatial dependencies and structural features between pixels without the need for high-quality paired annotations; the mask matrix clearly distinguishes between sampled and unsampled regions, enabling the network to focus on predicting missing information during training and reducing the risk of trivial solutions and overfitting.

[0081] In one embodiment of this application, Represents complete imaging volume data, where For the time dimension (number of frames). For spatial dimensions, subsampling observation Using a binary sampling mask Obtain the mask matrix:

[0082] in, This represents element-wise multiplication. Indicator Frame pixels Whether it has been sampled.

[0083] Sampling mask The design features spatial complementarity; specifically, in embodiments of this application, each frame is spatially divided into two complementary regions along the width dimension:

[0084]

[0085] in, and These represent the indices of the even-numbered and odd-numbered columns, respectively. It is the basic sampling mode (e.g., period is 1). Triangle or rectangle pattern, (Subsampling step size).

[0086] In one embodiment of this application, the complementary regions of the mask matrix satisfy the spatial complementarity condition, which is:

[0087] in, An indicator function representing the complete sampling. It represents the empty set.

[0088] Among them, the spatial complementarity condition is a spatial sampling constraint condition, which in this application means that the pixel regions identified by the left and right complementary mask matrices do not overlap in each frame and cover the entire field of view after being combined.

[0089] Understandably, by ensuring that the complementary regions of the mask matrix do not overlap and cover the entire field of view, this application can make full use of the complementary information of different frames to perform pixel prediction during the training process, thereby effectively recovering the data of the unsampled area and reducing the risk of information loss.

[0090] In one embodiment of this application, to ensure that the complementary regions on the left and right sides can effectively cover the entire field of view during subsampling observations, the complementary regions of the mask matrix satisfy the spatial complementarity condition. Specifically, in each sampling period... Inside, the complementary area on the left complementary region to the right The intersection of the two is approximately an empty set, ensuring that the left and right regions do not overlap. Simultaneously, their union is approximately equal to the indicator function of the complete sampling. :

[0091] Through the above-described spatial complementarity design, the embodiments of this application achieve cross-supervision during the self-supervised training process.

[0092] In step S103, a pre-constructed three-dimensional convolutional neural network is trained based on self-supervised training samples and pseudo-labels. The trained three-dimensional convolutional neural network is then used to infer and reconstruct the rolling subsampling data to obtain an image sequence. In this process, the multiphoton microscope is controlled to acquire rolling subsampling data according to the rolling subsampling scanning strategy.

[0093] Among them, the three-dimensional convolutional neural network is a deep learning network structure; the self-supervised training samples are training data that can be generated without manual annotation, and in this application, they represent input-prediction pairs constructed through complementary regions of the mask matrix; the pseudo-labels are virtual labels generated based on the input data; and reasoning and reconstruction are the process by which the neural network predicts new input data to generate a complete image.

[0094] Understandably, the neural network in this application can learn the mapping relationship to recover the complete image from subsampled data without manual annotation, thereby achieving high-quality reconstruction of multiphoton microscopy imaging data at high subsampled rates, improving imaging speed, and ensuring the continuity of image sequences in time and space as well as the fidelity of fine structures.

[0095] In one embodiment of this application, the reconstruction target of the three-dimensional convolutional neural network is a parameterized reconstruction function, through which the function enables... The reconstruction function is:

[0096] in, It provides a time-averaged image with global spatial context. It is the reconstructed complete data. Indicates by parameters The mapping function for control.

[0097] Here, the reconstruction target is the parameterized reconstruction function, which represents the mapping relationship learned by the neural network.

[0098] Understandably, through parameterized reconstruction functions, 3D convolutional neural networks can represent the mapping from subsampled data to complete image sequences in a unified mathematical form, improving reconstruction accuracy and enabling the network to adaptively process data under different sampling modes, reducing reliance on long-term full-sampling scans.

[0099] In one embodiment of this application, the reconstruction target of the three-dimensional convolutional neural network is a parameterized reconstruction function. This function is used to process subsampled observation data. Mapped to complete imaging volume data Specifically, it is expressed as:

[0100] in, It is a time-averaged image that provides global spatial context, used to provide global spatial context information and assist the network in referencing overall structural features during reconstruction; Indicates by parameters The control mapping function, in this application, represents a reconstruction model implemented by a 3D convolutional neural network, used to convert subsampled observation data into complete volume data. Through this parameterized reconstruction function, this application can utilize local subsampled information and global temporal average context information to simultaneously recover missing pixel values, thereby achieving the reconstruction of subsampled images.

[0101] In one embodiment of this application, the three-dimensional convolutional neural network includes an encoder, a decoder, and an output layer. The encoder performs stepwise downsampling and feature extraction on the input data, the decoder performs stepwise upsampling on the extracted features, and the output layer outputs the image sequence. There are skip connections between corresponding layers of the encoder and the decoder. Layered mathematical expression is as follows:

[0102] in, Representing the Layer encoder, Representing the Layer encoder; The decoder's first Layered mathematical expression is as follows:

[0103] in, Representing the Layer decoder, where [ , [] represents splicing; The mathematical expression for the output layer is:

[0104] in, Represents the final reconstruction result. This represents a single-pixel convolution kernel.

[0105] In this application, the encoder corresponds to the feature extraction of the neural network, which represents the network part that performs layer-by-layer downsampling of the input subsampled data and extracts multi-scale spatiotemporal features; the decoder corresponds to the feature reconstruction of the neural network, which represents the network part that performs layer-by-layer upsampling of the features extracted by the encoder and fuses skip connection information to recover the complete image; the skip connection is the direct feature transfer channel between the encoder and the decoder.

[0106] Understandably, by combining an encoder-decoder structure with skip connections, this application can effectively capture the multi-scale spatiotemporal features of the input subsampled data while preserving detailed information and avoiding information loss caused by downsampling. The encoder downsamples step by step to compress redundant information and extract global features, while the decoder upsamples step by step to restore spatial resolution, enabling the network to generate high-quality complete image sequences. Skip connections further enhance the preservation of local details, ensuring that the reconstructed image is spatially continuous and refined. The final output layer realizes the mapping from subsampled observations to high-precision reconstruction.

[0107] In one embodiment of this application, a three-dimensional convolutional neural network employs a 3D U-Net architecture to jointly process spatiotemporal information. The network architecture includes an encoder, a decoder, and an output layer, wherein... The encoder path includes: four coding blocks, with progressive downsampling; each coding block contains two 3D convolutional layers, with a kernel size that can be [missing value]. The number of feature map channels is ,in The default value of 16 can be used; the space pooling used between coding blocks can be 16. The activation function used is ReLU; the feature maps output by the 3D convolutional layer are divided into several groups (e.g., 8 groups) and normalized.

[0108] The decoder path includes: three decoding blocks, which are upsampled step by step; each decoding block receives feature maps from the corresponding encoder layer and concatenates them through skip connections to fuse feature information from different levels; the upsampling operation uses nearest neighbor interpolation, and the upsampling factor can be (2,2,2) to increase the spatial size of the feature map; each decoding block contains two 3D convolutional layers for extracting and fusing the concatenated features, thereby enhancing the feature representation capability; the final layer includes convolution to produce a single-channel output.

[0109] Furthermore, the input configuration of this application is a dual-channel input for network reception, wherein channel 1 is for subsampled volume data. Channel 2 is the time-averaged image. Replicating in the time dimension to provide global spatial context information, the first The layer encoder is:

[0110] in, As input, and:

[0111] in, This represents 3D convolution, GN represents grouped normalization, and Pool is 3D max pooling (the first layer is omitted). The layer decoder is:

[0112] in, Indicates splicing, ,and:

[0113] Final reconstruction result:

[0114] In one embodiment of this application, the size of the l-th layer of the three-dimensional convolutional neural network is × × The convolution kernel extracts temporal features, among which, The temporal features extracted by the convolutional kernel are:

[0115] in, Represents the sense of space. Represents the temporal scale of the convolution kernel. Representing the The time pooling factor of the layer; The spatial features extracted by the convolutional kernel are:

[0116] in, Represents the spatial size of the convolution kernel. Represents the spatial pooling factor.

[0117] Understandably, by precisely calculating the temporal and spatial receptive fields of the convolutional kernels, this application can fully capture the temporal and spatial correlations of multiphoton microscopy data, improving the ability to perceive the dynamics of neuronal activity. A larger temporal receptive field enables the network to identify the temporal patterns of neuronal firing using information from multiple frames of images, while a reasonable spatial receptive field ensures the effective extraction of local and global structural information. The receptive field design, combined with pooling factors, controls information convergence and feature representation capabilities, enabling the 3D convolutional neural network to preserve details and understand overall dynamics in reconstruction tasks using wide-field-of-view subsampled data.

[0118] In one embodiment of this application, the third-dimensional convolutional neural network... Layer size is The convolutional kernels are used to jointly extract temporal and spatial features, and their effective receptive field increases exponentially with network depth. Specifically, the temporal features extracted by the convolutional kernels are:

[0119] in, Representing the The time perception field of the layer, For the first The size of the convolutional kernel in the time dimension, For the first The time pooling factor of the layer.

[0120] The spatial features extracted by the convolution kernel are:

[0121] in, The size of the convolution kernel in the spatial dimension. For the first Spatial pooling factor of the layer.

[0122] when When the pooling factor is 2, the effective receptive field of the deepest layer (layer 4) of the network is approximately: temporal receptive field: 24–48 frames; spatial receptive field: 48×48 pixels.

[0123] This joint representation of multi-scale spatiotemporal features is crucial for capturing local dynamics (such as calcium transients) as well as global structural information (such as neuronal morphology), and helps improve the reconstruction accuracy of subsampled images.

[0124] In one embodiment of this application, the three-dimensional convolutional neural network calculates the training loss using a total reconstruction loss function during training, and updates the network parameters of the three-dimensional convolutional neural network based on the training loss, wherein the total reconstruction loss function is:

[0125]

[0126]

[0127] in, Represents the total reconstruction losses. Represents the combination of L1 and L2 losses. Represents the target in the left-hand domain. Represents the target in the right-hand domain.

[0128] Understandably, by introducing a total reconstruction loss function and combining it with cross-supervision of complementary mask regions, 3D convolutional neural networks can simultaneously consider information from the left and right complementary regions during training, thereby enhancing the network's ability to predict unsampled regions and reducing reconstruction errors.

[0129] Specifically, for each training sample, this application divides the spatiotemporal volume data into two complementary spatial domains:

[0130]

[0131] in and These represent the indices for even-numbered and odd-numbered columns, respectively.

[0132] Similarly, the segmentation mask is:

[0133] And the target (same as the subsampled data in the self-supervised setting):

[0134] This application applies a mask to the left-hand domain. Construct the mask input:

[0135] Similarly, for the right-hand domain:

[0136] The network input for each domain also includes a time-averaged image:

[0137] Subsequently, this application employs a spatial rematching strategy; for the left-hand domain, the target... The construction is as follows:

[0138] For the right-hand domain, the target for:

[0139] in, and For pseudo-labels generated by self-supervision, Represents the unsampled pixels in the left-hand domain. This represents the sampled pixels at the corresponding spatial location in the right-hand domain. The re-matching strategy forces the network to use information from complementary regions to predict unsampled pixels in a given spatial region, thus preventing the network from simply memorizing sampling patterns.

[0140] This application also employs a combination of L1 and L2 losses:

[0141] in The two terms are balanced. L1 loss is robust to outliers (such as sharp signal transients), while L2 loss provides smooth gradients for stable optimization.

[0142] For spatial partitioning, the total reconstruction loss function of this application is:

[0143] The overall training objective function is this reconstruction loss:

[0144] After completing the design of the reconstruction loss function, this application further provides the network optimization configuration and training process, as described below.

[0145] This application uses the Adam optimizer for network training, and the specific optimization parameters are set as follows: (1) Learning rate: The basic learning rate is set to ; (2) Momentum coefficient: , ; (3) Weight decay: Do not use weight decay.

[0146] Regarding learning rate scheduling, a fixed learning rate (without decay) is used during the baseline training phase; during the fine-tuning phase, the learning rate is reduced to a fixed rate. .

[0147] Since 3D volume data consumes a lot of video memory, this application sets the single-card training batch size to 1; in multi-GPU scenarios, a larger effective batch size is obtained through data parallelism.

[0148] Regarding the training duration, in one embodiment of this application, a total of 10 training rounds are conducted, with each round containing approximately 6,000 training image blocks, resulting in a total of approximately 60,000 iterations throughout the entire training process.

[0149] Regarding the convergence criteria, training is performed in a fixed number of rounds, and the convergence of the model is monitored by the validation loss of the validation set.

[0150] Finally, the experimental results of rolling subsampling and algorithmic recovery in this application are presented. Figure 4 and Figure 5 As shown. Among them, Figure 4 The original subsampling imaging results of this application are shown; Figure 5 Typical neurons are labeled with triangles, showing their neural activity trajectories under subsampling, algorithmic reconstruction, and ground truth conditions, with a scale bar of 60 micrometers.

[0151] from Figure 4 and Figure 5 As can be seen, this application downsamples the original image through rolling subsampling, resulting in significant noise and spatial information loss in the sampled image. Applying the wide-field-of-view multiphoton microscopy imaging method proposed in this application, the structural details in the subsampled image are effectively restored, and the overall imaging clarity is close to the original full-sampled true value. Even with high noise in the original image, this application can still accurately reconstruct detailed structures in spatial imaging. Simultaneously, experimental results show that subsampled signals suffer from significant noise interference and loss of weak signals. This application, however, can recover near-true neural activity trajectories from low-quality subsampled data. Even weak or high-frequency neural signals can be accurately reconstructed, thus ensuring the integrity and reliability of the temporal information of neural activity.

[0152] Based on the aforementioned self-supervised reconstruction framework and optimization strategies, this application can be adapted to a variety of important biological imaging systems in a plug-and-play manner. For example, this application can be applied to mesoscopic microscopy systems to improve imaging speed and quality in large-field-of-view volumetric imaging tasks; it can also be applied to conventional multiphoton microscopes to enhance 3D imaging speed and reconstruction resolution. Furthermore, this application can be seamlessly integrated into light-bead microscopes for ultra-high-speed, large-scale volumetric imaging. This application can also be extended to other scanning-based imaging methods, thus providing a universal reconstruction solution for high-speed 3D imaging tasks.

[0153] In summary, as Figure 6As shown, the wide-field multiphoton microscopy imaging method of this application includes two parts: a self-supervised offline training stage and an online application reconstruction stage, as detailed below: During the self-supervised offline training phase, training data is first acquired on an existing multiphoton microscopy system using a rolling subsampling scanning strategy, with fluorescence signals, scan path indices, and timestamps recorded simultaneously. The acquired data is then normalized, temporally registered, and spatially partitioned to generate a mask matrix corresponding to the hardware sampling mode. Pseudo-labels are constructed based on spatial complementarity, and self-supervised training samples are formed through cross-supervision of left and right complementary regions. The subsampling images and their temporally averaged maps are used as input to train a three-dimensional convolutional neural network, employing reconstruction loss, mask consistency loss, and optional periodic constraint loss to optimize network parameters. Finally, the optimal model weights are selected and solidified based on the validation set performance, resulting in a deployable self-supervised reconstruction model.

[0154] During the online application reconstruction phase, the scanning control module performs scanning in the same rolling subsampling mode as the training phase, while the data acquisition module simultaneously acquires fluorescence signals and galvanometer position information. The collaborative computing module formats, normalizes, and performs mask matching on the acquired data to generate model input. The self-supervised learning module loads the trained model weights and performs real-time inference on the subsampling data to output a high-fidelity restored image sequence. The imaging reconstruction module stitches and displays the reconstruction results and feeds back the quality assessment results to the scanning control module for dynamic optimization of scanning parameters.

[0155] Through the above process, this application achieves a deep integration of offline self-supervised learning and online real-time reconstruction, constructing a complete closed loop from optical sampling, data acquisition, algorithm reconstruction to imaging feedback.

[0156] According to the wide field-of-view multiphoton microscopy imaging method proposed in this application, firstly, the multiphoton microscope is controlled to acquire training data according to a rolling subsampling scanning strategy. This rolling subsampling scanning strategy involves sequentially staggering the scanning paths of adjacent periodic image frames along the target direction within multiple scanning cycles, ensuring that all periodic image frames collectively cover all points in the entire field of view. This achieves high frame rate data acquisition while maintaining complete field-of-view coverage, improving imaging speed without sacrificing spatial coverage of the field of view. Secondly, the acquired training data undergoes at least one preprocessing step, including normalization, temporal registration, and spatial partitioning, and a mask matrix is ​​generated based on the preprocessing results. The mask matrix is ​​then used for mutual... Cross-supervision of the complementary regions forms self-supervised training samples, and pseudo-labels are constructed based on the spatial complementarity of image frames. Reliable training signals are generated through self-supervision, which not only preserves important spatiotemporal information but also avoids dependence on high-quality labeled data, thus achieving high-fidelity training. Then, the pre-constructed three-dimensional convolutional neural network is trained according to the self-supervised training samples and pseudo-labels, and the trained network is used to infer and reconstruct subsampled data collected by the rolling subsampled scanning strategy to obtain high-quality image sequences. This effectively restores the spatial details and dynamic signals of the subsampled images, thereby solving the technical difficulty of achieving both wide field of view and high spatiotemporal resolution in wide field-of-view multiphoton microscopy.

[0157] Next, with reference to the accompanying drawings, a wide field-of-view multiphoton microscopy imaging apparatus according to an embodiment of this application is described.

[0158] Figure 7 This is a block diagram of a wide field-of-view multiphoton microscopy imaging device according to an embodiment of this application.

[0159] like Figure 7 As shown, the wide field-of-view multiphoton microscopy imaging device 10 includes: an acquisition module 100, a processing module 200, and an output module 300.

[0160] The system includes an acquisition module 100 for acquiring training data collected by a multiphoton microscope according to a rolling subsampling scanning strategy. This rolling subsampling scanning strategy involves the scanning paths of adjacent image frames being staggered sequentially during multiple cycles of rolling subsampling scanning along the target direction, with the scanning paths of all image frames covering all points in the entire field of view. A processing module 200 performs at least one preprocessing step on the training data, including normalization, temporal registration, and spatial partitioning. It generates a mask matrix based on the preprocessed data, forms self-supervised training samples through cross-supervision of complementary regions of the mask matrix, and constructs pseudo-labels based on the spatial complementarity of the image frames. An output module 300 trains a pre-constructed three-dimensional convolutional neural network using the self-supervised training samples and the pseudo-labels. It then uses the trained three-dimensional convolutional neural network to infer and reconstruct the subsampling data to obtain an image sequence. The system controls the multiphoton microscope to collect the subsampling data according to the rolling subsampling scanning strategy.

[0161] In one embodiment of this application, both the training data and the subsampling data include prior information on the scanning path, spatial location and morphology information of neurons, and information on the firing time of neural activity during the multiphoton microscopy scanning process.

[0162] In one embodiment of this application, the formula for calculating the mask matrix is: X=Y⊙M Where X represents subsampling observations, Y represents complete imaging volume data, including neural calcium signal imaging, neural voltage signal imaging, neurochemical substance signal imaging, etc., and ⊙ represents element-wise multiplication. Represents a binary sampling mask; The formula for calculating the complementary region of the mask matrix is:

[0163]

[0164] in, Represents an even-numbered column index. Represents an index for odd-numbered columns. Basic sampling mode (e.g., period is 1, period is 2) Triangle or rectangle pattern, (for subsampling step size) Indicator Frame pixels Whether it was sampled, Represents the even-numbered column mask. .

[0165] In one embodiment of this application, the complementary regions of the mask matrix satisfy the spatial complementarity condition, which is:

[0166] in, An indicator function representing the complete sampling. It represents the empty set.

[0167] In one embodiment of this application, the reconstruction target of the three-dimensional convolutional neural network is a parameterized reconstruction function, through which the function enables... The reconstruction function is:

[0168] in, It provides a time-averaged image with global spatial context. It is the reconstructed complete data. Indicates by parameters The mapping function for control.

[0169] In one embodiment of this application, the three-dimensional convolutional neural network includes an encoder, a decoder, and an output layer. The encoder performs stepwise downsampling and feature extraction on the input data, the decoder performs stepwise upsampling on the extracted features, and the output layer outputs the image sequence. There are skip connections between corresponding layers of the encoder and the decoder. Layered mathematical expression is as follows:

[0170] in, Representing the Layer encoder, Representing the Layer encoder; The decoder's first Layered mathematical expression is as follows:

[0171] in, Representing the Layer decoder, where [ , [] represents splicing; The mathematical expression for the output layer is:

[0172] in, Represents the final reconstruction result. This represents a single-pixel convolution kernel.

[0173] In one embodiment of this application, the size of the l-th layer of the three-dimensional convolutional neural network is × × The convolution kernel extracts temporal features, among which, The temporal features extracted by the convolutional kernel are:

[0174] in, Represents the sense of space. Represents the temporal scale of the convolution kernel. Representing the The time pooling factor of the layer; The spatial features extracted by the convolutional kernel are:

[0175] in, Represents the spatial size of the convolution kernel. Represents the spatial pooling factor.

[0176] In one embodiment of this application, the three-dimensional convolutional neural network calculates the training loss using a total reconstruction loss function during training, and updates the network parameters of the three-dimensional convolutional neural network based on the training loss, wherein the total reconstruction loss function is:

[0177]

[0178]

[0179] in, Represents the total reconstruction losses. Represents the combination of L1 and L2 losses. Represents the target in the left-hand domain. Represents the target in the right-hand domain.

[0180] It should be noted that the foregoing explanation of the wide field-of-view multiphoton microscopy imaging method embodiment also applies to the wide field-of-view multiphoton microscopy imaging device of this embodiment, and will not be repeated here.

[0181] According to the wide field-of-view multiphoton microscopy imaging device proposed in this application, firstly, this application controls the multiphoton microscope to acquire training data according to a rolling subsampling scanning strategy. This rolling subsampling scanning strategy includes, within multiple scanning cycles, sequentially staggering the scanning paths of adjacent period image frames along the target direction, so that all period image frames collectively cover all points in the entire field of view. This achieves high frame rate data acquisition while ensuring complete field-of-view coverage, improving imaging speed without sacrificing spatial coverage of the field of view. Secondly, the acquired training data undergoes at least one preprocessing step, including normalization, temporal registration, and spatial partitioning, and a mask matrix is ​​generated based on the preprocessing results. The mask matrix is ​​then used for mutual... Cross-supervision of the complementary regions forms self-supervised training samples, and pseudo-labels are constructed based on the spatial complementarity of image frames. Reliable training signals are generated through self-supervision, which not only preserves important spatiotemporal information but also avoids dependence on high-quality labeled data, thus achieving high-fidelity training. Then, the pre-constructed three-dimensional convolutional neural network is trained according to the self-supervised training samples and pseudo-labels, and the trained network is used to infer and reconstruct subsampled data collected by the rolling subsampled scanning strategy to obtain high-quality image sequences. This effectively restores the spatial details and dynamic signals of the subsampled images, thereby solving the technical difficulty of achieving both wide field of view and high spatiotemporal resolution in wide field-of-view multiphoton microscopy.

[0182] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include: The memory 801, the processor 802, and the computer program stored on the memory 801 and capable of running on the processor 802.

[0183] When the processor 802 executes the program, it implements the wide field-of-view multiphoton microscopy imaging method provided in the above embodiments.

[0184] Furthermore, electronic devices also include: Communication interface 803 is used for communication between memory 801 and processor 802.

[0185] The memory 801 is used to store computer programs that can run on the processor 802.

[0186] The memory 801 may include high-speed RAM (Random Access Memory) memory, and may also include non-volatile memory, such as at least one disk storage.

[0187] If the memory 801, processor 802, and communication interface 803 are implemented independently, then the communication interface 803, memory 801, and processor 802 can be interconnected via a bus to complete communication between them. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 8 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0188] Optionally, in a specific implementation, if the memory 801, processor 802, and communication interface 803 are integrated on a single chip, then the memory 801, processor 802, and communication interface 803 can communicate with each other through an internal interface.

[0189] The processor 802 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of this application.

[0190] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0191] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0192] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0193] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any of the following techniques known in the art, or a combination thereof: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (FPGAs), field-programmable gate arrays (FPGAs), etc.

[0194] Those skilled in the art will understand that all or part of the steps of the methods implementing the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0195] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A wide-field-of-view multiphoton microscopy imaging method, characterized in that, Includes the following steps: Acquire training data collected by a multiphoton microscope according to a rolling subsampling scanning strategy, wherein the rolling subsampling scanning strategy includes that, in multiple cycles of rolling subsampling scanning by the multiphoton microscope along the target direction, the scanning paths of adjacent cycle image frames are staggered in the target direction, and the scanning paths of all cycle image frames cover all points of the entire field of view. The training data is preprocessed by at least one of normalization, temporal registration and spatial partitioning. A mask matrix is ​​generated based on the preprocessed data. Self-supervised training samples are formed by cross-supervision of the complementary regions of the mask matrix. Pseudo-labels are constructed based on the spatial complementarity of image frames. The pre-constructed three-dimensional convolutional neural network is trained based on the self-supervised training samples and the pseudo-labels. The trained three-dimensional convolutional neural network is then used to infer and reconstruct the rolling subsampled data to obtain an image sequence. The multiphoton microscope is controlled to acquire the rolling subsampled data according to the rolling subsampled scanning strategy.

2. The wide-field-of-view multiphoton microscopy imaging method according to claim 1, characterized in that, Both the training data and the subsampling data include prior information on the scanning path, spatial location and topography, and neural activity firing time information during the multiphoton microscopy scanning process.

3. The wide-field-of-view multiphoton microscopy imaging method according to claim 1, characterized in that, The formula for calculating the mask matrix is: X=Y⊙M Where X represents subsampled observation data, Y represents complete neural activity imaging data, and ⊙ represents element-wise multiplication. Represents a binary sampling mask; The formula for calculating the complementary region of the mask matrix is: in, Index for even-numbered columns, Index for odd-numbered columns, Basic sampling mode, frame pixels Whether it was sampled, For even-numbered column masks, .

4. The wide-field-of-view multiphoton microscopy imaging method according to claim 3, characterized in that, The complementary regions of the mask matrix satisfy the spatial complementarity condition, which is: in, For the indicator function of full sampling, It is an empty set.

5. The wide-field-of-view multiphoton microscopy imaging method according to claim 1, characterized in that, The reconstruction target of the three-dimensional convolutional neural network is a parameterized reconstruction function, such that... ; in, It provides a time-averaged image with global spatial context. It is the reconstructed complete data. Indicates by parameters The mapping function for control.

6. The wide-field-of-view multiphoton microscopy imaging method according to claim 1, characterized in that, The three-dimensional convolutional neural network includes an encoder, a decoder, and an output layer. The encoder performs step-by-step downsampling and feature extraction on the input data; the decoder performs step-by-step upsampling on the extracted features; and the output layer outputs the image sequence. There are skip connections between corresponding layers of the encoder and decoder. Layered mathematical expression is as follows: in, For the first Layer encoder, For the first Layer encoder; The decoder's first Layered mathematical expression is as follows: in, For the first Layer decoder, where [ , [] represents splicing; The mathematical expression for the output layer is: in, For the final reconstruction result, It is a single-pixel convolution kernel.

7. The wide-field-of-view multiphoton microscopy imaging method according to claim 6, characterized in that, The third-dimensional convolutional neural network l Layer size is × × The convolution kernel extracts temporal features, among which, The temporal features extracted by the convolutional kernel are: in, To create a sense of space and wilderness The temporal scale of the convolution kernel. For the first The time pooling factor of the layer; The spatial features extracted by the convolutional kernel are: in, The size of the convolution kernel in space. This is the spatial pooling factor.

8. The wide-field-of-view multiphoton microscopy imaging method according to claim 6, characterized in that, The three-dimensional convolutional neural network calculates the training loss using a total reconstruction loss function during training, and updates the network parameters based on the training loss. The total reconstruction loss function is: in, For the total reconstruction losses, For the combination of Manhattan loss and Euclidean loss, For the left-hand domain target, For the right-hand domain target.

9. A wide-field-of-view multiphoton microscopy imaging device, characterized in that, include: The acquisition module is used to acquire training data collected by the multiphoton microscope according to the rolling subsampling scanning strategy, wherein the rolling subsampling scanning strategy includes that in multiple cycles of the multiphoton microscope performing rolling subsampling scanning along the target direction, the scanning paths of adjacent cycle image frames are staggered in the target direction, and the scanning paths of all cycle image frames cover all points of the entire field of view. The processing module is used to perform at least one preprocessing on the training data, including normalization, temporal registration, and spatial partitioning; generate a mask matrix based on the preprocessed data; form self-supervised training samples by cross-supervision of the complementary regions of the mask matrix; and construct pseudo-labels based on the spatial complementarity of image frames. The output module is used to train a pre-constructed three-dimensional convolutional neural network based on the self-supervised training samples and the pseudo-labels, and to use the trained three-dimensional convolutional neural network to infer and reconstruct the rolling subsampled data to obtain an image sequence. The multiphoton microscope is controlled to acquire the rolling subsampled data according to the rolling subsampled scanning strategy.

10. An electronic device, characterized in that, include: Storage hardware, a processor, and a computer program stored on the storage hardware and executable on the processor, the processor executing the program to implement the wide field-of-view multiphoton microscopy imaging method as described in any one of claims 1-8.

Images