Two-photon light field supervised network reconstruction method and apparatus, electronic device and medium

Abstract

The present application relates to the technical field of computational imaging, in particular to a two-photon light field supervised network reconstruction method and apparatus, an electronic device and a storage medium. The method comprises: acquiring paired two-photon light field data and 3D volume data; preprocessing the paired two-photon light field data and 3D volume data to obtain preprocessed data; on the basis of the preprocessed data, constructing a supervised reconstruction network until a preset iteration stop condition is reached, so as to obtain a two-photon light field supervised network reconstruction result. Thus, the present application solves the problems in the prior art that due to the involvement of a large amount of operations, the computational complexity is increased, the consumed time is long and the reconstruction efficiency is reduced, and that as the resolution gradually reduces along with the increase of the distance from the original object plane, the complexity of the light field distribution mode is increased, detailed information loss easily occurs, and the details and clarity of reconstructed images are reduced.

Description

Two-photon light field supervised network reconstruction method, device, electronic device and medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on the Chinese patent application with application number 202410064498.1 and application date on January 16, 2024, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby introduced into this application as a reference. Technical Field

[0003] The present application relates to the field of computational imaging technology, and in particular to a method, device, electronic device, and medium for supervised network reconstruction of a two-photon light field. Background Art

[0004] The 2PSA microscope can flexibly switch between two configurations: traditional 2PSA mode and 2PSA mode. Traditional 2PSA mode primarily uses 3D point scanning to acquire high-resolution 3D volumes, but suffers from issues such as high phototoxicity and low temporal resolution. 2PSA uses a needle-shaped beam to capture multiple angles of the entire 3D volume to obtain two-photon light field data, significantly reducing phototoxicity and enabling millisecond-level subcellular 3D imaging in deep tissues. It also exhibits strong robustness to various optical aberrations.

[0005] In related technologies, traditional deconvolution methods based on light propagation models are mainly used to deconvolve and reconstruct two-photon light field data, including the Richardson-Lucy deconvolution algorithm, the phase space deconvolution algorithm, etc., and then according to the point spread function and light propagation model of the optical system, the 3D structure can be restored from the two-photon light field data.

[0006] However, in related technologies, due to the large number of operations involved, the computational complexity is increased, which can easily lead to a long time consumption and reduced reconstruction efficiency. In addition, since the resolution gradually decreases with the increase in distance from the original object plane, it may lead to an increase in the complexity of the light field distribution pattern, resulting in the loss of detail information, reducing the details and clarity of the reconstructed image, and affecting the quality of image reconstruction, which urgently needs to be improved.

[0007] Summary of the Invention

[0008] The present application provides a supervised network reconstruction method, device, electronic device and storage medium for two-photon light field to solve the problems in related technologies, such as involving a large number of operations, increasing computational complexity, resulting in long reconstruction time and reduced reconstruction efficiency. In addition, since the resolution gradually decreases with increasing distance from the original object plane, it may lead to increased complexity of the light field distribution pattern, easily causing loss of detailed information, and reducing the details and clarity of the reconstructed image.

[0009] A first aspect of the present application provides a supervised network reconstruction method for a two-photon light field, comprising the following steps: acquiring paired two-photon light field data and 3D volume data; preprocessing the paired two-photon light field data and 3D volume data to obtain preprocessed data; and constructing a supervised reconstruction network based on the preprocessed data until a preset iteration stop condition is reached to obtain a supervised network reconstruction result of the two-photon light field.

[0010] Optionally, in one embodiment of the present application, the preprocessing of the paired two-photon light field data and 3D volume data includes: linearly transforming the multi-angle data in the two-photon light field data and the 3D volume data so that the pixel values ​​are within a preset range to obtain normalized data; and performing random angle selection, flipping and / or cropping on the normalized data to obtain amplified data as the preprocessed data.

[0011] Optionally, in one embodiment of the present application, a supervised reconstruction network is constructed based on the preprocessed data until a preset iteration stop condition is reached to obtain a two-photon light field supervised network reconstruction result, including: generating a network output and a supervisory signal based on the preprocessed data; calculating the first and second norms of the difference between the network output and the supervisory signal, and calculating the first norm of the difference between the partial derivatives of the network output and the supervisory signal in three spatial dimensions, and calculating the second and second norm of the difference between the feature maps of the network output and the supervisory signal when the network has been pre-trained; weighting the first and second norms, and calculating the supervised loss function of the supervised reconstruction network.

[0012] Optionally, in one embodiment of the present application, before obtaining the two-photon light field supervised network reconstruction result, it also includes: inputting a test set obtained from the preprocessed data into the trained supervised reconstruction network, and outputting the network result, so as to output the two-photon light field supervised network reconstruction result when the network result meets the preset test conditions, wherein the test set does not overlap with the data of the training set for constructing the supervised reconstruction network and has a different size.

[0013] Optionally, in one embodiment of the present application, the preset iteration stopping condition is that the number of iterations reaches a preset iteration number threshold.

[0014] The second aspect of the present application provides a two-photon light field supervised network reconstruction device, including: an acquisition module for acquiring paired two-photon light field data and 3D volume data; a processing module for preprocessing the paired two-photon light field data and 3D volume data to obtain preprocessed data; and a reconstruction module for constructing a supervised reconstruction network based on the preprocessed data until a preset iteration stop condition is reached to obtain a two-photon light field supervised network reconstruction result.

[0015] Optionally, in one embodiment of the present application, the processing module includes: a normalization unit, used to perform linear transformation on the two-photon light field data and multi-angle data in the 3D volume data so that the pixel value is between a preset range to obtain normalized data; an amplification unit, used to perform random angle selection, flipping and / or cropping on the normalized data to obtain amplified data as the preprocessed data.

[0016] Optionally, in one embodiment of the present application, the reconstruction module includes: a generation unit for generating a network output and a supervisory signal based on the preprocessed data; a calculation unit for calculating the first and second norms of the difference between the network output and the supervisory signal, and calculating the first norm of the difference between the partial derivatives of the network output and the supervisory signal in three spatial dimensions, and calculating the second and second norm of the difference between the feature maps of the network output and the supervisory signal when the network has been pre-trained; a weighting unit for weighting according to the first and second norms to generate a supervised loss function of the supervised reconstruction network.

[0017] Optionally, in one embodiment of the present application, the reconstruction module is also used to input a test set obtained from the preprocessed data into the trained supervised reconstruction network before obtaining the two-photon light field supervised network reconstruction result, and output the network result, so as to output the two-photon light field supervised network reconstruction result when the network result meets the preset test conditions, wherein the test set does not overlap with the data of the training set for constructing the supervised reconstruction network and has a different size.

[0018] Optionally, in one embodiment of the present application, the preset iteration stopping condition is that the number of iterations reaches a preset iteration number threshold.

[0019] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the two-photon light field supervised network reconstruction method as described in the above embodiment.

[0020] The fourth aspect of the present application provides a computer-readable storage medium, which stores a computer program. When the program is executed by a processor, it implements the above-mentioned two-photon light field supervised network reconstruction method.

[0021] The embodiments of the present application can obtain paired two-photon light field data and 3D volume data, pre-process the paired two-photon light field data and 3D volume data, and construct a supervised reconstruction network based on the pre-processed data until a preset iteration stop condition is reached, thereby obtaining a supervised network reconstruction result of the two-photon light field, improving the network prediction speed, achieving real-time reconstruction, and achieving a relatively robust effect under different structures and signal-to-noise ratios, thereby improving universality. This solves the problems in the related art that, due to the large number of operations involved, the computational complexity is increased, resulting in a long time consumption and reduced reconstruction efficiency, and because the resolution gradually decreases with the distance from the original object plane, the complexity of the light field distribution pattern may increase, which easily causes the loss of detailed information and reduces the details and clarity of the reconstructed image.

[0022] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0024] FIG1 is a schematic structural diagram of a two-photon light field supervised network reconstruction method according to one embodiment of the present application;

[0025] FIG2 is a flow chart of a two-photon light field supervised network reconstruction method provided according to an embodiment of the present application;

[0026] FIG3 is a schematic diagram showing the principle of a supervised loss function calculation process according to one embodiment of the present application;

[0027] FIG4 is a schematic diagram showing a comparison between an original two-photon light field image and an image reconstructed through a network according to an embodiment of the present application;

[0028] FIG5 is a schematic structural diagram of a two-photon light field supervised network reconstruction device provided according to an embodiment of the present application;

[0029] FIG6 is a schematic structural diagram of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0031] The following describes a two-photon light field supervised network reconstruction method, device, electronic device and storage medium of an embodiment of the present application with reference to the accompanying drawings. In view of the related technologies mentioned in the above background technology center, since a large number of operations are involved, the computational complexity is increased, resulting in a long time consumption and reduced reconstruction efficiency. Moreover, since the resolution gradually decreases with the distance from the original object plane, the complexity of the light field distribution pattern may increase, which may easily cause the loss of detail information and reduce the details and clarity of the reconstructed image. The present application provides a two-photon light field supervised network reconstruction method, in which paired two-photon light field data and 3D volume data can be obtained, the paired two-photon light field data and 3D volume data can be preprocessed, and a supervised reconstruction network can be constructed based on the preprocessed data until a preset iteration stop condition is reached, thereby obtaining a two-photon light field supervised network reconstruction result, improving the network prediction speed, achieving real-time reconstruction, and achieving a more robust effect under different structures and signal-to-noise ratios, thereby improving universality. This solves the problems in related technologies, such as the large number of operations involved, which increases the computational complexity, resulting in a long time consumption and reduced reconstruction efficiency. In addition, since the resolution gradually decreases with increasing distance from the original object plane, the complexity of the light field distribution pattern may increase, which may easily cause the loss of detail information and reduce the details and clarity of the reconstructed image.

[0032] Before explaining the two-photon light field supervised network reconstruction method provided in the embodiment of the present application, the structure of the two-photon light field supervised network reconstruction method involved in the embodiment of the present application is first explained.

[0033] As shown in FIG1 , the structure of the supervised network reconstruction method for two-photon light fields includes: a two-photon light field data acquisition unit, a data preprocessing unit, a supervised reconstruction network training unit, and a supervised reconstruction network testing unit.

[0034] Among them, the two-photon light field data acquisition unit includes the 2pSAM (Two-photon Synthetic Aperture Microscopy) light field data acquisition function and the TPM (Two-photon Microscopy) 3D volume data acquisition function, which are used to provide data for network training and testing.

[0035] Among them, the two-photon light field data obtained by 2pSAM is used as the input of the network, and the 3D volume data obtained by TPM is used as the supervision signal for supervised deep learning.

[0036] Data preprocessing unit, including data normalization and data amplification functions.

[0037] The data normalization function is used to perform linear transformation on multi-angle data so that the pixel values ​​are between 0 and 1 to obtain normalized data. The data augmentation function is used to perform random angle rotation, flipping, cropping and other operations on the normalized data to obtain the final amplified data. The final data will be used for supervised reconstruction network training or supervised reconstruction network testing.

[0038] The supervised reconstruction network training unit includes the network input forward transmission function, the supervised loss function calculation function and the supervised loss function return function. Completing the above three steps is called an iteration.

[0039] Among them, the network input forward transmission function receives the final light field input data output by the data preprocessing unit through the input layer, passes it through the network, and obtains the network output through the output layer; the supervised loss function calculation function can be obtained by the corresponding 3D true value data output by the network output and the data preprocessing unit, and is formed by the weighted mean square error, gradient loss and perceptual loss; the supervised loss function return function obtains the loss function value through supervised loss function calculation 320, and performs gradient return to update the network parameters. Usually, a threshold of the number of iterations is set. When the value is not exceeded, it is considered that the network training has not ended. When the network training has not ended, it enters the next iteration, that is, the next batch of data undergoes network input forward transmission, supervised loss function calculation and supervised loss function return;

[0040] The supervised reconstruction network testing unit is performed after the supervised reconstruction network training is completed, and is used to test the final performance of the selected data on the network.

[0041] Among them, the selected data often does not overlap with the data used by the supervised reconstruction network training unit, and the size is not necessarily the same. If the size is different, the selected data should be cropped with overlap and cropped into several images that meet the size requirements of the network input layer. Let the network predict separately and then splice the prediction results.

[0042] Next, the two-photon light field supervised network reconstruction method of the embodiment of the present application is described in detail.

[0043] Specifically, FIG2 is a flow chart of a two-photon light field supervised network reconstruction method provided in an embodiment of the present application.

[0044] As shown in FIG2 , the two-photon light field supervised network reconstruction method includes the following steps:

[0045] In step S201 , paired two-photon light field data and 3D volume data are acquired.

[0046] It can be understood that paired two-photon light field data refers to light field measurement data obtained through two-photon entanglement, and 3D volume data refers to the three-dimensional spatial measurement of an object or scene, and the data on the position and characteristics of each point obtained.

[0047] Specifically, the embodiments of the present application can quickly switch between the two modes of the two-photon synthetic aperture microscope to obtain paired two-photon light field data and 3D volume data. For example, by quickly switching between the two modes of the two-photon synthetic aperture microscope, detailed morphology and detailed information of immune cells can be obtained, such as the shape, size, outline and arrangement of cells, and three-dimensional structure and tissue information of immune cells, such as the position and distribution of major organelles such as the nucleus and mitochondria inside the immune cells.

[0048] The embodiment of the present application can quickly switch between the two modes of the two-photon synthetic aperture microscope to obtain paired two-photon light field data and 3D volume data, improve the quality of the data and the richness of the information, and provide an accurate basis for subsequent operations.

[0049] In step S202 , the paired two-photon light field data and 3D volume data are preprocessed to obtain preprocessed data.

[0050] It can be understood that preprocessing includes data normalization, data amplification, etc.

[0051] The embodiment of the present application preprocesses paired two-photon light field data and 3D volume data, such as data normalization and data amplification, to obtain preprocessed data, which helps to reduce interference factors and improve the accuracy and reliability of the data.

[0052] Optionally, in one embodiment of the present application, paired two-photon light field data and 3D volume data are preprocessed, including: linearly transforming the multi-angle data in the two-photon light field data and the 3D volume data so that the pixel values ​​are within a preset range to obtain normalized data; and performing random angle selection, flipping and / or cropping on the normalized data to obtain amplified data as the preprocessed data.

[0053] It is understandable that the preset range refers to a preset pixel value range of data, such as scaling the pixel value to between 0 and 1.

[0054] Specifically, the embodiments of the present application can perform linear transformation on the multi-angle data in the two-photon light field data and the 3D volume data, can scale the pixel values ​​from the original range [0, 255] to between [0, 1], and then can perform data amplification operations on the normalized data, including random angle selection, flipping and cropping, etc., so as to obtain preprocessed data.

[0055] In an embodiment of the present application, two-photon light field data and multi-angle data in 3D volume data are linearly transformed so that pixel values ​​are within a preset range to obtain normalized data, and random angle selection, flipping and / or cropping are performed to obtain amplified data, which can be used as preprocessed data, can effectively adjust the data range, help improve robustness, and can better cope with data variability under different acquisition conditions, thereby increasing the diversity and stability of data samples.

[0056] In step S203, a supervised reconstruction network is constructed based on the preprocessed data until a preset iteration stop condition is reached to obtain a two-photon light field supervised network reconstruction result.

[0057] It is understandable that the preset iteration stopping condition may be that the number of iterations reaches a preset iteration threshold, such as the number of iterations reaches 1000 times.

[0058] Specifically, the embodiments of the present application can utilize preprocessed data for supervised reconstruction network training or supervised reconstruction network testing, wherein the supervised reconstruction network training includes network input forward transmission, supervised loss function calculation and supervised loss function return transmission. Completion of the above three steps is called an iteration. When the network training is not completed, the next iteration is entered, that is, the next batch of network input forward transmission, supervised loss function calculation and supervised loss function return transmission; supervised reconstruction network testing is performed after the supervised reconstruction network training is completed, and is used to test the final performance of the selected data on the network.

[0059] The embodiment of the present application can construct a supervised reconstruction network based on the preprocessed data until the preset iteration stop condition is reached, thereby obtaining a supervised network reconstruction result of the two-photon light field, which helps to improve the accuracy of the reconstruction result. At the same time, through continuous optimization, it can improve robustness and increase flexibility in adapting to different application scenarios.

[0060] Optionally, in one embodiment of the present application, a supervised reconstruction network is constructed based on the preprocessed data until a preset iteration stop condition is reached to obtain a supervised network reconstruction result of a two-photon light field, including: generating a network output and a supervisory signal based on the preprocessed data; calculating the first and second norms of the difference between the network output and the supervisory signal, and calculating the first norm of the difference between the partial derivatives of the network output and the supervisory signal in three spatial dimensions, and calculating the second and second norms of the difference between the feature maps of the network output and the supervisory signal when the network has been pre-trained; weighting the first and second norms, the first norm and the second and second norms to calculate the supervised loss function of the supervised reconstruction network.

[0061] Specifically, in combination with what is shown in FIG3 , the embodiment of the present application can calculate the first and second norms of the difference between the network output and the supervisory signal, the first norm of the difference between the partial derivatives of the network output and the supervisory signal in three spatial dimensions, and the second and second norms of the difference between the feature maps of the network output and the supervisory signal when they pass through the pre-trained network through a supervised loss function, and weight them to form a supervised loss function, wherein the network output can be obtained by passing the input multi-view data through multiple layers of convolution, transformer modules, forward propagation layers and activation layers.

[0062] Therefore, the relationship between the supervised loss function and the first and second norms of the difference between the network output and the supervisory signal, the first norm of the difference between the partial derivatives of the network output and the supervisory signal in the three spatial dimensions, and the second and second norms of the difference between the feature maps of the network output and the supervisory signal after the pre-trained network can be shown as follows: L(Output,Gt)=αMSE(Output,Gt)+βGradient3D(Output,Gt)+γPerceptual(Output,Gt)

[0063] Among them, Output is the network output, Gt is the 3D true value, L is the supervised loss function, MSE is the mean square error between the forward projection and the network input, Gradient3D is the mean absolute error after the first-order difference between the network output and the 3D true value in each dimension, Perceptual is the sum of the mean square error between the network output and the 3D true value in each feature map after input into the pre-trained network, α is the weight of the mean square error, β is the weight of the second norm of the second-order derivative, and γ is the weight of the continuity constraint.

[0064] The embodiments of the present application can obtain relatively robust and fast three-dimensional reconstruction data based on a two-photon light field supervised reconstruction network, which is beneficial to improving the accuracy and stability of reconstruction.

[0065] Optionally, in one embodiment of the present application, before obtaining the supervised network reconstruction result of the two-photon light field, it also includes: inputting a test set obtained from the preprocessed data into the trained supervised reconstruction network, and outputting the network result, so as to output the supervised network reconstruction result of the two-photon light field when the network result meets the preset test conditions, wherein the data of the test set does not overlap with the training set for constructing the supervised reconstruction network and has a different size.

[0066] It is understandable that the preset test conditions refer to the conditions for evaluating and judging the reconstruction results, which can be the degree of structural preservation, reconstruction accuracy, etc.

[0067] In actual implementation, as shown in FIG4 , the embodiment of the present application can utilize multi-angle test images to obtain a high-resolution reconstructed image after network reconstruction, wherein the image reconstructed by the network is a 3D image, but is displayed as a projection along the z-axis.

[0068] For example, the embodiment of the present application can use a two-photon synthetic aperture microscope to capture immune cell data, and then normalize and amplify the data to generate 1,000 multi-angle images of size 1024×1024×13 for supervised reconstruction network training; a supervised reconstruction network can be built using the Pytorch deep learning framework and Python programming language. Specifically, after the input layer, it first passes through multiple 2D convolutions to fuse the information between 13 perspectives, and gradually increases the number of channels in the perspective dimension to 256; then a new feature vector dimension is added, where the initial size of the dimension is 1. Through a structure similar to UNet (Fully Convolutional Network, U-shaped network in the full convolutional network), the number of channels in the perspective dimension is compressed layer by layer, and the number of channels in the feature vector dimension is increased simultaneously; in the central part of the UNet, several transformer blocks are inserted to perform self-attention operations in the spatial and angular dimensions respectively. Further, by training the network, where the initial learning rate is 1×10 -4 The training batch size is 1, and the Adam (Adaptive Moment Estimation) optimizer is used for back-propagation iterative optimization. A total of 1,000 iterations are trained, and the learning rate is reduced to half every 200 iterations. This allows multi-angle test images to be input into the trained supervised reconstruction network to obtain reconstructed images.

[0069] The embodiment of the present application inputs a test set obtained from preprocessed data into a trained supervised reconstruction network and outputs a network result, so as to output a supervised network reconstruction result of a two-photon light field when the network result meets preset test conditions, thereby facilitating adaptive adjustment and further improving the accuracy and stability of the reconstruction result.

[0070] Optionally, in one embodiment of the present application, the preset iteration stopping condition is that the number of iterations reaches a preset iteration number threshold.

[0071] The embodiment of the present application can effectively control the training time by setting the iteration number threshold, improve the training efficiency, reduce the consumption of computing resources, and save costs.

[0072] According to the two-photon light field supervised network reconstruction method proposed in the embodiment of the present application, it is possible to obtain paired two-photon light field data and 3D volume data, pre-process the paired two-photon light field data and 3D volume data, and construct a supervised reconstruction network based on the pre-processed data until a preset iteration stop condition is reached, thereby obtaining a two-photon light field supervised network reconstruction result, improving the network prediction speed, achieving real-time reconstruction, and achieving a more robust effect under different structures and signal-to-noise ratios, thereby improving universality. This solves the problems in the related art that, due to the large number of operations involved, the computational complexity is increased, resulting in a long time consumption and reduced reconstruction efficiency, and because the resolution gradually decreases with the distance from the original object plane, the complexity of the light field distribution pattern may increase, which easily causes the loss of detail information and reduces the details and clarity of the reconstructed image.

[0073] Next, a two-photon light field supervised network reconstruction device proposed according to an embodiment of the present application will be described with reference to the accompanying drawings.

[0074] FIG5 is a schematic structural diagram of a two-photon light field supervised network reconstruction device according to an embodiment of the present application.

[0075] As shown in FIG5 , the two-photon light field supervised network reconstruction device 10 includes: an acquisition module 100 , a processing module 200 and a reconstruction module 300 .

[0076] Specifically, the acquisition module 100 is used to acquire paired two-photon light field data and 3D volume data.

[0077] The processing module 200 is used to preprocess the paired two-photon light field data and 3D volume data to obtain preprocessed data.

[0078] The reconstruction module 300 is used to construct a supervised reconstruction network based on the preprocessed data until a preset iteration stop condition is reached to obtain a supervised network reconstruction result of the two-photon light field.

[0079] Optionally, in one embodiment of the present application, the processing module 200 includes: a normalization unit and an amplification unit.

[0080] The normalization unit is used to perform linear transformation on the two-photon light field data and the multi-angle data in the 3D volume data so that the pixel values ​​are within a preset range to obtain normalized data;

[0081] The amplification unit is used to perform random angle selection, flipping and / or cropping on the normalized data to obtain amplified data as preprocessed data.

[0082] Optionally, in one embodiment of the present application, the reconstruction module 300 includes: a generation unit, a calculation unit and a weighting unit.

[0083] Wherein, the generation unit is used to generate network output and supervision signal based on the preprocessed data;

[0084] A calculation unit, configured to calculate the first and second norms of the difference between the network output and the supervisory signal, calculate the first norm of the difference between the partial derivatives of the network output and the supervisory signal in three spatial dimensions, and calculate the second and second norm of the difference between the feature maps of the network output and the supervisory signal when the network has been pre-trained;

[0085] The weighting unit is used to perform weighting according to the first binary norm, the first norm and the second binary norm to generate a supervised loss function of the supervised reconstruction network.

[0086] Optionally, in one embodiment of the present application, the reconstruction module 300 is also used to input a test set obtained from the preprocessed data into the trained supervised reconstruction network before obtaining the two-photon light field supervised network reconstruction result, and output the network result, so as to output the two-photon light field supervised network reconstruction result when the network result meets the preset test conditions, wherein the test set does not overlap with the data of the training set for constructing the supervised reconstruction network and has a different size.

[0087] Optionally, in one embodiment of the present application, the preset iteration stopping condition is that the number of iterations reaches a preset iteration number threshold.

[0088] It should be noted that the aforementioned explanation of the embodiment of the two-photon light field supervised network reconstruction method is also applicable to the two-photon light field supervised network reconstruction device of this embodiment, and will not be repeated here.

[0089] According to the two-photon light field supervised network reconstruction device proposed in the embodiment of the present application, it is possible to obtain paired two-photon light field data and 3D volume data, pre-process the paired two-photon light field data and 3D volume data, and construct a supervised reconstruction network based on the pre-processed data until a preset iteration stop condition is reached. In this way, a two-photon light field supervised network reconstruction result can be obtained, the network prediction speed can be improved, and real-time reconstruction can be achieved. Moreover, it can achieve a relatively robust effect under different structures and signal-to-noise ratios, thereby improving universality. This solves the problems in the related art that, due to the large number of operations involved, the computational complexity is increased, the reconstruction efficiency is reduced, and the resolution gradually decreases with the distance from the original object plane, which may lead to an increased complexity of the light field distribution pattern, easily resulting in the loss of detailed information, and reducing the details and clarity of the reconstructed image.

[0090] FIG6 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0091] A memory 601 , a processor 602 , and a computer program stored in the memory 601 and executable on the processor 602 .

[0092] When the processor 602 executes the program, the two-photon light field supervised network reconstruction method provided in the above embodiment is implemented.

[0093] Furthermore, the electronic device further includes:

[0094] The communication interface 603 is used for communication between the memory 601 and the processor 602 .

[0095] The memory 601 is used to store computer programs that can be run on the processor 602 .

[0096] The memory 601 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0097] If the memory 601, processor 602, and communication interface 603 are implemented independently, the communication interface 603, memory 601, and processor 602 can be interconnected via a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be divided into address buses, data buses, control buses, etc. For ease of illustration, FIG6 shows only one thick line, but this does not mean that there is only one bus or only one type of bus.

[0098] Optionally, in a specific implementation, if the memory 601, the processor 602 and the communication interface 603 are integrated on a chip, the memory 601, the processor 602 and the communication interface 603 can communicate with each other through an internal interface.

[0099] The processor 602 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0100] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned two-photon light field supervised network reconstruction method.

[0101] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0102] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of this application, "N" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0103] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing a custom logical function or process step, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed in a different order than shown or discussed, including performing functions in a substantially simultaneous manner or in a reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application pertain.

[0104] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or N wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program can be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing it in other suitable ways as necessary, and then storing it in a computer memory.

[0105] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiment, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0106] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0107] In addition, the functional units in the various embodiments of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into a module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0108] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A supervised network reconstruction method for two-photon optical fields, characterized in that, It includes the following steps: Obtain paired two-photon light field data and 3D volume data; Preprocess the paired two-photon light field data and 3D volume data to obtain preprocessed data; And Construct a supervised reconstruction network according to the preprocessed data until a preset iteration stop condition is reached, and obtain the supervised network reconstruction result of the two-photon light field.

2. The supervised network reconstruction method for two-photon optical fields according to claim 1, characterized in that The preprocessing of the paired two-photon light field data and 3D volume data includes: Perform a linear transformation on the multi-angle data in the two-photon light field data and 3D volume data so that the pixel values are within a preset range to obtain normalized data; Perform random angle selection, flipping, and / or cropping on the normalized data to obtain augmented data as the preprocessed data.

3. The supervised network reconstruction method for two-photon optical fields according to claim 1, wherein The constructing of a supervised reconstruction network according to the preprocessed data until a preset iteration stop condition is reached and obtaining the supervised network reconstruction result of the two-photon light field includes: Generate a network output and a supervision signal based on the preprocessed data; Calculate the first two-norm of the difference between the network output and the supervision signal, calculate the first norm of the difference between the partial derivatives of the network output and the supervision signal in three spatial dimensions, and calculate the second two-norm of the difference between the feature maps of the network output and the supervision signal when passing through a pre-trained network; Weight the first two-norm, the first norm, and the second two-norm, and calculate the supervised loss function of the supervised reconstruction network.

4. The supervised network reconstruction method for two-photon optical field according to claim 1, characterized in that Before obtaining the supervised network reconstruction result of the two-photon light field, it further includes: Input a test set obtained from the preprocessed data into the trained supervised reconstruction network, output a network result, and output the supervised network reconstruction result of the two-photon light field when the network result meets a preset test condition, where the test set does not overlap with the data of the training set for constructing the supervised reconstruction network and has a different size.

5. The supervised network reconstruction method for two-photon optical fields according to claim 1, wherein The preset iteration stop condition is that the number of iterations reaches a preset iteration number threshold.

6. A supervised network reconstruction device for a two-photon optical field, characterized in that, It includes: An acquisition module for obtaining paired two-photon light field data and 3D volume data; A processing module for preprocessing the paired two-photon light field data and 3D volume data to obtain preprocessed data; And A reconstruction module for constructing a supervised reconstruction network according to the preprocessed data until a preset iteration stop condition is reached, and obtaining the supervised network reconstruction result of the two-photon light field.

7. The two-photon optical field supervised network reconstruction device according to claim 6, characterized in that, The processing module includes: A normalization unit for performing a linear transformation on the multi-angle data in the two-photon light field data and 3D volume data so that the pixel values are within a preset range to obtain normalized data; An augmentation unit for performing random angle selection, flipping, and / or cropping on the normalized data to obtain augmented data as the preprocessed data.

8. The two-photon optical field supervised network reconstruction device according to claim 6, characterized in that, The reconstruction module includes: A generation unit for generating a network output and a supervision signal based on the preprocessed data; A calculation unit for calculating the first L2 norm of the difference between the network output and the supervision signal, calculating the L1 norm of the difference between the partial derivatives of the network output and the supervision signal in three spatial dimensions, and calculating the second L2 norm of the difference between the feature maps of the network output and the supervision signal when passing through the pre-trained network; A weighting unit for weighting according to the first L2 norm, the L1 norm, and the second L2 norm to generate a supervised loss function of the supervised reconstruction network.

9. The two-photon optical field supervised network reconstruction device according to claim 6, characterized in that The reconstruction module is further configured to, before obtaining the reconstruction result of the supervised network of the two-photon light field, input a test set obtained from the preprocessed data into the trained supervised reconstruction network and output a network result, so as to output the reconstruction result of the supervised network of the two-photon light field when the network result meets a preset test condition, wherein the test set does not overlap with the data of the training set for constructing the supervised reconstruction network and has a different size.

10. The two-photon optical field supervised network reconstruction device according to claim 6, wherein The preset iteration stop condition is that the number of iterations reaches a preset iteration number threshold.

11. An electronic device, characterized in that, Comprising: A memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the method for reconstructing a supervised network of a two-photon light field according to any one of claims 1-5.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to be used for implementing the method for reconstructing a supervised network of a two-photon light field according to any one of claims 1-5.

Images