Light field microscopic image super-resolution method and device based on deep learning, equipment and medium

Abstract

The invention discloses a light field microscopic image super-resolution method and device based on deep learning, equipment and a medium. The method comprises the following steps: acquiring a light field image to be super-resolved of a sample; the light field image to be subjected to super-resolution is input into a convolutional neural network framework, super-resolution light field image data are obtained, the convolutional neural network framework comprises a parallax estimation network and a view fusion network, the parallax estimation network is used for estimating parallax between a target view and other views, and the view fusion network is used for fusing the target view and the other views. And the view fusion network is used for carrying out view fusion on the distorted image and outputting a super-resolution target view, and other views are distorted to the target view so as to realize alignment of spatial features, and the view fusion network is used for carrying out view fusion on the distorted image and outputting the super-resolution target view. Therefore, the light field image resolution can be improved under the condition that other hardware is not required to be changed, and the long-time observation phototoxicity is reduced.

Description

Technical Field

[0001] The present invention relates to a light field microscopy image super-resolution method, device, equipment and medium based on deep learning, and relates to the field of light field microscopy technology. Background Art

[0002] Modern life science research requires faster 3D volume imaging. In this context, light-field microscopy has become a popular method due to its ability to simultaneously obtain spatial and angular information of the sample and reconstruct the 3D scene.

[0003] The combination of light field technology and microscopy technology plays an increasingly important role in optical imaging methods for three-dimensional biological problems such as calcium signal imaging and hemodynamics. Although light field microscopy has the ability to quickly obtain three-dimensional information of samples, the pixel-multiplexed light field microscope based on microlens array has a trade-off between spatial resolution and angular resolution, resulting in low spatial resolution, which seriously limits the application range of light field microscopy. The main problems are: First, in order to improve the light field resolution, more dense pixels need to be provided for sampling, which will reduce the image signal-to-noise ratio. Second, a microlens array with a smaller aperture is required, and the numerical aperture must be large enough, which poses a challenge to micro-nano processing technology. Third, due to the reconstruction algorithm based on deconvolution, there are mosaic artifacts at the focal plane, and traditional methods cannot effectively solve this problem.

[0004] In summary, although light field microscopy has significant advantages in some aspects compared with other three-dimensional imaging methods, its spatial resolution still needs to be improved. Summary of the Invention

[0005] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, in response to the above-mentioned problems, the present invention aims to provide a deep learning-based light field microscopy image super-resolution method, device, equipment, and medium capable of improving the spatial resolution of light field images.

[0006] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is: In a first aspect, the present invention provides a light field microscopy image super-resolution method based on deep learning, comprising: Collecting a light field image of the sample to be super-resolved; The light field image to be super-resolved is input into a convolutional neural network framework to obtain super-resolved light field image data, wherein the convolutional neural network framework includes a disparity estimation network and a view fusion network. The disparity estimation network is used to estimate the disparity between the target view and other views, and warp the other views onto the target view to achieve alignment of spatial features. The view fusion network is used to perform view fusion on the warped image and output a super-resolved target view.

[0007] In some possible implementations, the acquiring of the light field image of the sample to be super-resolved includes: Set up a light-field microscopy system, including a dichroic mirror, objective lens, tube lens, microlens array, relay lens, and camera; An illumination light source excites the sample fluorescence, the dichroic mirror is used to separate the fluorescence and the excitation light, the tube lens maps the image formed by the objective lens at infinity onto the microlens array, the microlens array modulates the image, and the modulated image is imaged on the camera through the relay lens to obtain a time-sequential light field image; Pixels at different angles in the time-series light field image are extracted and rearranged, and the single-frame light field image is rearranged into a two-dimensional angle-two-dimensional space image to obtain a four-dimensional light field image, which is the light field image to be super-resolved.

[0008] In some possible implementations, the training process of the convolutional neural network framework includes: Generate a simulated four-dimensional light field image dataset; Building a two-stage super-resolution framework based on convolutional neural networks, including the disparity estimation network and the view fusion network; The convolutional neural network framework is trained using simulated four-dimensional light field images as input to the disparity estimation network and super-resolved target views as output of the view fusion network.

[0009] In some possible implementations, the disparity estimation network includes a feature extraction layer and a disparity estimation layer, wherein the feature extraction layer uses a convolution unit with a convolution kernel size of 1 to extract features of the simulated four-dimensional light field image, and the disparity estimation layer includes cascaded convolution layers with different convolution kernel sizes, and the convolution layer and the linear rectification activation function constitute a convolution unit. The image features extracted by the feature extraction layer are learned, and the mapping relationship between images at different angles and the target image is learned from the features through convolution kernels with decreasing sizes layer by layer.

[0010] In some possible implementations, the training process of the view fusion network is as follows: the super-resolution input image is a high-resolution light field, the high-resolution image is downsampled in the view fusion stage, the network learns the mapping from the downsampled image to the high-resolution light field image, and the network loss function is designed based on the difference between the network super-resolution result and the high-resolution light field image to perform network training.

[0011] In some possible implementations, the loss function of the network during training is divided into two parts: disparity estimation loss and super-resolution loss, wherein the loss function of the disparity estimation network is trained in a self-supervised manner, based on the idea that the distorted view and the target view are as close as possible; the super-resolution loss includes two parts: recursive loss and back propagation loss within a single view, wherein the recursive loss is trained using downsampled images, which may lose high-frequency information, and the back propagation loss samples the original size images for training, which can retain some high-frequency information.

[0012] In a second aspect, the present invention further provides a light field microscopic image super-resolution device based on deep learning, comprising: An image acquisition unit, configured to acquire a light field image of a sample to be super-resolved; The high-resolution light field unit is configured to input the light field image to be super-resolved into a convolutional neural network framework to obtain super-resolved light field image data, wherein the convolutional neural network framework includes a disparity estimation network and a view fusion network. The disparity estimation network is used to estimate the disparity between the target view and other views, and warp the other views onto the target view to achieve alignment of spatial features. The view fusion network is used to perform view fusion on the warped image and output a super-resolved target view.

[0013] In some possible implementations, the image acquisition unit includes: Light-field microscopy system, including dichroic mirror, objective, tube lens, microlens array, relay lens, and camera; The time-sequential light field image acquisition system is configured such that an illumination light source excites the sample fluorescence, the dichroic mirror is used to separate the fluorescence and the excitation light, the tube lens projects the image formed by the objective lens at infinity onto the microlens array, the microlens array modulates the image, and the modulated image is imaged onto the camera through the relay lens to obtain a time-sequential light field image; The four-dimensional light field image acquisition system is configured to extract pixels at different angles in the time-sequential light field image and rearrange them, and rearrange the single-frame light field image into a two-dimensional angle-two-dimensional spatial image to obtain a four-dimensional light field image, which is the light field image to be super-resolved.

[0014] In a third aspect, the present invention also provides an electronic device comprising: at least one processor; and a memory communicatively connected to the processor; wherein the memory stores instructions executable by the processor, and the instructions are executed by the processor to enable the processor to execute the described method.

[0015] In a fourth aspect, the present invention further provides a computer-readable storage medium storing one or more programs, wherein the one or more programs include computer instructions, and the computer instructions are used to enable a computer to execute the method described.

[0016] The present invention adopts the above technical solution, which has the following characteristics: 1. The network structure designed in the present invention can explore information between different angles and use this information to perform self-supervised light field super-resolution. It can improve the resolution of light field images without other hardware changes and reduce the phototoxicity of long-term observation.

[0017] 2. The light field super-resolution method based on the deep learning network of view alignment loss and view fusion loss provided by the present invention does not require benchmark real data for pairing, and has a small demand on the number of training samples. Only one four-dimensional light field image is needed for network training. In addition, the network design has a physical basis and is versatile, and can be extended to different biological scenarios.

[0018] 3. Based on the existing light-field microscope, the present invention combines deep learning methods to improve the spatial resolution of light-field images. It can be applied to a variety of samples and has better versatility than traditional end-to-end super-resolution networks. It solves the trade-off between spatial resolution and angular resolution in traditional light-field microscopes.

[0019] In summary, the present invention can be widely used in the biopharmaceutical industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present invention. Throughout the drawings, the same reference numerals are used to denote the same components. In the drawings: Figure 1 This is a flowchart of a light field microscopy image super-resolution method based on a deep learning algorithm according to an embodiment of the present invention; Figure 2 Schematic diagram of the structure of a light field microscopy imaging system according to an embodiment of the present invention; Figure 3 A schematic diagram of the deep learning principle of an embodiment of the present invention; Figure 4 Schematic diagram of a convolutional neural network framework according to an embodiment of the present invention; Figure 5 FIG. 4 is a structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0021] It should be understood that the terms used herein are for the purpose of describing specific example embodiments only and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "comprise", "include", "contain" and "have" are inclusive and therefore specify the presence of stated features, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not to be construed as necessarily requiring them to be performed in the specific order described or illustrated, unless the order of execution is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0022] Although the terms first, second, third, etc. can be used in the text to describe multiple elements, components, regions, layers and / or sections, these elements, components, regions, layers and / or sections should not be limited by these terms. These terms can only be used to distinguish an element, component, region, layer or section from another region, layer or section. Unless the context clearly indicates otherwise, terms such as "first", "second" and other numerical terms do not imply order or sequence when used in the text. Therefore, the first element, component, region, layer or section discussed below can be referred to as the second element, component, region, layer or section without departing from the teaching of the example embodiments.

[0023] For ease of description, spatially relative terms may be used herein to describe the relationship of one element or feature relative to another element or feature as shown in the figures, such as "inside," "outside," "inner side," "outer side," "lower," "upper," etc. Such spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures.

[0024] Because the spatial resolution of existing light field microscopy technology still needs to be improved, the present invention provides a deep learning-based light field microscopy image super-resolution method, apparatus, device, and medium, comprising: collecting a sample light field image to be super-resolved; inputting the light field image to be super-resolved into a convolutional neural network framework to obtain super-resolved light field image data. The convolutional neural network framework includes a disparity estimation network and a view fusion network. The disparity estimation network is used to estimate the disparity between the target view and other views and warp the other views onto the target view to achieve spatial feature alignment. The view fusion network is used to perform view fusion on the warped image and output a super-resolved target view. Therefore, based on existing light field microscopy systems, the present invention combines deep learning methods to improve the spatial resolution of light field images, explore information between different angles, and super-resolve the target view. This method is compatible with any hardware form of light field microscopes. The deep learning method of the present invention is based on a "divide and conquer" approach. The network framework adopts a two-stage training strategy: a disparity estimation network and a view fusion network. The disparity estimation loss function and the view fusion loss function can be trained in an unsupervised manner. Since the network training takes into account the physical basis of parallax between different views, it can be applied to a variety of samples. Its versatility is better than that of traditional end-to-end super-resolution networks, and it solves the trade-off problem between spatial resolution and angular resolution in traditional light-field microscopy.

[0025] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. Rather, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0026] Example 1: Figure 1 As shown, the light field microscopy image super-resolution method based on deep learning provided in this embodiment includes: S101. Build a light field microscopy system to collect four-dimensional light field images.

[0027] In this embodiment, Figure 2As shown, the light field microscopy system includes components such as an objective lens 1, a dichroic mirror 2, a tube lens 3, a microlens array 4, a relay lens 5, and a camera 6. Specifically, an illumination light source excites a sample 7 to emit fluorescence, and a dichroic mirror 2 separates the fluorescence and excitation light. The tube lens 3 projects the image formed by the objective lens 1 at infinity onto the microlens array 4, which modulates the image. The modulated image is then imaged by the relay lens 5 onto the camera 6 to produce a time-sequential light field image. The time-sequential light field image is obtained by simultaneously capturing both angular and spatial information through the microlens array 4, acquiring three-dimensional information about the object through a single exposure. This acquisition method has very high temporal resolution.

[0028] In this embodiment, the time-sequential light field image also needs to be processed. By extracting pixels at different angles in the time-sequential light field image and rearranging them, the single-frame light field image is rearranged into a two-dimensional angle-two-dimensional space image to obtain a four-dimensional light field image.

[0029] S102: Generate a simulated four-dimensional light field image data set.

[0030] In this embodiment, a simulated sample volume distribution dataset is generated based on the distribution characteristics of the three-dimensional sample data. That is, a simulation algorithm is used to introduce reasonable randomness and, in accordance with the distribution characteristics that may exist in actual biological samples, such as spatial distribution, temporal variation, morphological structure, and intensity distribution, to generate a corresponding simulated sample dataset, so that it basically conforms to the same rules and properties as the samples in the experimental process. The simulated sample dataset is then convolved with the point spread function generated by the simulation to generate a simulated light field image dataset. Specifically, the generation is performed by specifying the distribution density of the simulated data, the actual light intensity distribution, and the point spread function of the system. The point spread function of the system can be obtained by collecting images of small balls, such as 100 nm, or can be obtained by computational simulation using the Debye model. For example, when generating a small ball sample, the number of small balls per unit area can be specified to obtain a three-dimensional stacked image and the image can be convolved with the point spread function image to generate random small ball data. When generating microtubule structures, the generation of straight lines with random orientations and length distributions can be specified, and distortion deformation can be introduced to simulate cytoskeleton data. This is an example and is not limited to this.

[0031] Furthermore, in the simulation, the image is corrected taking into account the camera noise introduced in the actual acquisition. The camera noise satisfies the Poisson distribution. The generated simulated light field image data can generate Poisson noise according to the intensity of the image and add it to the image to simulate the noise in the actual acquired image.

[0032] Furthermore, it is necessary to perform pixel rearrangement on the simulated light field image data set to generate a simulated four-dimensional light field image set.

[0033] S103. Build a convolutional neural network framework and complete network training.

[0034] In this embodiment, the idea of ​​"divide and conquer" is adopted to decompose the light field super-resolution task into two tasks: disparity estimation and view fusion, and a two-stage super-resolution framework based on convolutional neural networks is built. Figure 2 、 Figure 3 The simulated four-dimensional light field image is used as the input of the convolutional neural network framework, and the super-resolution target view is used as the output of the convolutional neural network framework. The convolutional neural network is trained based on a loss function designed according to the microscopic sample. In this embodiment, the convolutional neural network framework comprises two networks: a disparity estimation network and a view fusion network, forming a two-stage network structure from a four-dimensional light field image to a target super-resolution view. The disparity estimation network explicitly estimates the disparity between the target view and other views (disparity is the pixel offset between images of the same scene point at different viewpoints) and warps the other views onto the target view to align spatial features. The warped images are then fused and trained using the view fusion network, which then employs downsampling to obtain paired super-resolution views.

[0035] Furthermore, the disparity estimation network learns the distortions of images from different viewpoints to the target view, thereby obtaining a representation of the displayed disparity, namely, the pixel offset between images from different angles. Once trained, the disparity estimation network can directly align input light field images from different angles to the target view. The disparity estimation network is implemented using common neural network structures such as convolutional layers and warping layers. It comprises a feature extraction layer and a disparity estimation layer. The feature extraction layer uses convolution units with a kernel size of 1 to extract features from a simulated four-dimensional light field image. The disparity estimation layer comprises cascaded convolution units with different kernel sizes, composed of convolution layers and linear rectifier activation functions. The disparity estimation layer explicitly estimates the disparity between the target view and other views. The disparity estimation layer learns the features extracted by the feature extraction layer, using convolution kernels of decreasing size layer by layer to learn the mapping relationship between images from different angles and the target image. This means that the pixel offset between different views is used to represent the light field disparity structure.

[0036] Furthermore, the view fusion network includes cascaded convolution units with different convolution kernel sizes. After the views are aligned, the pixels of the same scene point at different angles are spatially closer, which is beneficial for the view fusion network to learn complementary information between different angles. The super-resolution input image is a high-resolution light field. The image will be downsampled by 2 times in the view fusion stage. The network learns the mapping from the downsampled image to the high-resolution light field image, and designs the network loss function based on the difference between the network super-resolution result and the high-resolution light field image for network training. Specifically, the L2 loss is used in this embodiment. By downsampling the high-resolution light field to obtain high-low paired light field data and inputting it into the disparity estimation network, aligned high-low resolution paired light field data can be generated. The low-resolution aligned light field is input into the view fusion network to predict the super-resolution view, and the difference between it and the high-resolution target view is used as the learning loss function. In this way, the fusion network can be trained in a self-supervised manner.

[0037] Furthermore, this embodiment combines the characteristics of biological microscopic samples and specially designs a loss function suitable for biological microscopic samples to train the network model. This is mainly reflected in the design of a self-supervised disparity distortion constraint term in the loss function to constrain the disparity between different sub-apertures. The network loss function can be divided into two parts: disparity estimation loss and super-resolution loss. Among them, the loss function of the disparity estimation network can be trained in a self-supervised manner. It is based on the idea that the distorted view and the target view are as close as possible, and can be specifically expressed as:

[0038] in, is the disparity estimation loss, N is the number of input images, The mapping from different perspectives to the target perspective learned by the network, For all views, For high-resolution light field target view, is the target view.

[0039] The super-resolution loss consists of two parts: the recursive loss within a single view and the backpropagation loss. The recursive loss uses downsampled images for training, which may lose high-frequency information. The backpropagation loss samples the original size images for training, so it can retain some high-frequency information. It can be expressed as: , in,

[0040]

[0041] in, is the input light field image block, It is the super-resolution result of the network downsampling the target view data. is the super-resolution result of the input light field target view, N is the number of samples involved in the loss function calculation, and α is the network super-resolution factor.

[0042] Furthermore, this embodiment sets appropriate parameters to train the convolutional neural network. The input of the convolutional neural network framework is the generated simulated light field image, and the output is the super-resolution target view. The parameters that need to be trained mainly include two parts. One part is the parameters related to the network model, including but not limited to the number of layers of the network, the size of the convolution kernel, the number of channels with different convolution kernel sizes when extracting features, etc.; the other part is the parameters related to the training process, including but not limited to the weights of each item in the loss function, the number of training samples, the number of training iterations, the relationship between the number of training times of the generating network and the discriminating network, the training step size, etc. It is worth noting that the network model parameters and the training parameters are closely related to the final training results. Therefore, different adjustments and optimizations need to be made for different tasks. For example, for samples with small differences, some parameters can be frozen for network fine-tuning; when the sample differences are large, it is necessary to select a suitable learning rate based on the network convergence situation, and select a suitable batch size based on the local computing configuration to accelerate network training.

[0043] S104: Inputting the four-dimensional light field image to be super-resolved into the trained convolutional neural network framework for super-resolution testing to obtain super-resolution light field image data.

[0044] Example 2: The above-mentioned Example 1 provides a light field microscopic image super-resolution method based on deep learning. Correspondingly, this embodiment provides a light field microscopic image super-resolution device based on deep learning. The device provided in this embodiment can implement the light field microscopic image super-resolution method based on deep learning of Example 1, and the device can be implemented by software, hardware, or a combination of software and hardware. For the convenience of description, this embodiment is described by dividing the functions into various units and describing them separately. Of course, the functions of each unit can be implemented in the same or multiple software and / or hardware during implementation. For example, the device may include integrated or separate functional modules or functional units to perform the corresponding steps in each method of Example 1. Since the device of this embodiment is basically similar to the method embodiment, the description process of this embodiment is relatively simple. For relevant matters, please refer to the partial description of Example 1. The embodiment of the light field microscopic image super-resolution device based on deep learning provided by the present invention is only schematic.

[0045] Specifically, the present invention also provides a light field microscopic image super-resolution device based on deep learning, comprising: An image acquisition unit, configured to acquire a light field image of a sample to be super-resolved; The high-resolution light field unit is configured to input the light field image to be super-resolved into a convolutional neural network framework to obtain super-resolved light field image data, wherein the convolutional neural network framework includes a disparity estimation network and a view fusion network. The disparity estimation network is used to estimate the disparity between the target view and other views, and warp the other views onto the target view to achieve alignment of spatial features. The view fusion network is used to perform view fusion on the warped image and output a super-resolved target view.

[0046] Example 3: This example provides an electronic device corresponding to the deep learning-based light field microscopy image super-resolution method provided in Example 1. The electronic device may be an electronic device used for a client, such as a mobile phone, a laptop computer, a tablet computer, a desktop computer, etc., to execute the method of Example 1.

[0047] like Figure 5 As shown, the electronic device includes a processor, a memory, a communication interface and a bus. The processor, the memory and the communication interface are connected via the bus to complete communication between them. The memory stores a computer program that can be run on the processor. When the processor runs the computer program, it executes the method of embodiment 1. Its implementation principle and technical effect are similar to those of embodiment 1 and will not be repeated here. It can be understood by those skilled in the art that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computing device to which the solution of the present application is applied. The specific computing device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0048] In a preferred embodiment, the logic instructions in the aforementioned memory can be implemented in the form of a software functional unit and, when sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the portion that contributes to the prior art, or the portion of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), and an optical disk.

[0049] In a preferred embodiment, the processor may be a central processing unit (CPU), a digital signal processor (DSP), or other general-purpose processors, which are not limited herein.

[0050] Embodiment 4: This embodiment provides a computer-readable storage medium storing one or more programs, wherein the one or more programs include computer instructions. When the computer instructions are executed by a computer, the computer executes the method provided in the above embodiment 1.

[0051] In a preferred embodiment, a computer-readable storage medium may be a tangible device that retains and stores instructions executed by the computer, such as, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any combination thereof. The computer-readable storage medium stores computer program instructions that cause a computer to execute the method provided in the first embodiment.

[0052] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (apparatus), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0053] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0054] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0055] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In the description of this specification, the reference terms "a preferred embodiment", "further", "specifically", "in this embodiment", etc. mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of this specification. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0056] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A light field microscopy image super-resolution method based on deep learning, characterized in that: include: Collecting a light field image of the sample to be super-resolved; The light field image to be super-resolved is input into a convolutional neural network framework to obtain super-resolved light field image data, wherein the convolutional neural network framework includes a disparity estimation network and a view fusion network. The disparity estimation network is used to estimate the disparity between the target view and other views, and warp the other views onto the target view to achieve alignment of spatial features. The view fusion network is used to perform view fusion on the warped image and output a super-resolved target view.

2. The light field microscopy image super-resolution method based on deep learning according to claim 1, characterized in that: The light field image to be super-resolved of the collected sample includes: Set up a light-field microscopy system, including a dichroic mirror, objective lens, tube lens, microlens array, relay lens, and camera; An illumination light source excites the sample fluorescence, the dichroic mirror is used to separate the fluorescence and the excitation light, the tube lens maps the image formed by the objective lens at infinity onto the microlens array, the microlens array modulates the image, and the modulated image is imaged on the camera through the relay lens to obtain a time-sequential light field image; Pixels at different angles in the time-series light field image are extracted and rearranged, and the single-frame light field image is rearranged into a two-dimensional angle-two-dimensional space image to obtain a four-dimensional light field image, which is the light field image to be super-resolved.

3. The light field microscopy image super-resolution method based on deep learning according to claim 1, characterized in that: The training process of the convolutional neural network framework includes: Generate a simulated four-dimensional light field image dataset; Building a two-stage super-resolution framework based on convolutional neural networks, including the disparity estimation network and the view fusion network; The convolutional neural network framework is trained using simulated four-dimensional light field images as input to the disparity estimation network and super-resolved target views as output of the view fusion network.

4. The light field microscopy image super-resolution method based on deep learning according to claim 3, characterized in that: The disparity estimation network includes a feature extraction layer and a disparity estimation layer, wherein the feature extraction layer uses a convolution unit with a convolution kernel size of 1 to extract features of a simulated four-dimensional light field image, and the disparity estimation layer includes cascaded convolution layers with different convolution kernel sizes. The convolution layer and the linear rectification activation function constitute a convolution unit, which learns the image features extracted by the feature extraction layer and learns the mapping relationship between images at different angles and the target image from the features through convolution kernels with decreasing sizes layer by layer.

5. The light field microscopic image super-resolution method based on deep learning according to claim 4, characterized in that: The training process of the view fusion network is as follows: the super-resolution input image is a high-resolution light field, the high-resolution image is downsampled in the view fusion stage, the network learns the mapping from the downsampled image to the high-resolution light field image, and the network training is performed by designing the network loss function based on the difference between the network super-resolution result and the high-resolution light field image.

6. The light field microscopy image super-resolution method based on deep learning according to claim 5, characterized in that: During the training process, the network's loss function is divided into two parts: disparity estimation loss and super-resolution loss. The loss function of the disparity estimation network is trained in a self-supervised manner, based on the idea that the distorted view and the target view are as close as possible. The super-resolution loss consists of two parts: recursive loss within a single view and back-propagation loss. The recursive loss uses downsampled images for training, which may lose high-frequency information. The back-propagation loss samples the original size images for training, which can retain some high-frequency information.

7. A light field microscopic image super-resolution device based on deep learning, characterized in that: include: An image acquisition unit, configured to acquire a light field image of a sample to be super-resolved; The high-resolution light field unit is configured to input the light field image to be super-resolved into a convolutional neural network framework to obtain super-resolved light field image data, wherein the convolutional neural network framework includes a disparity estimation network and a view fusion network. The disparity estimation network is used to estimate the disparity between the target view and other views, and warp the other views onto the target view to achieve alignment of spatial features. The view fusion network is used to perform view fusion on the warped image and output a super-resolved target view.

8. The light field microscopic image super-resolution device based on deep learning according to claim 7, characterized in that: The image acquisition unit includes: Light-field microscopy system, including dichroic mirror, objective, tube lens, microlens array, relay lens, and camera; The time-sequential light field image acquisition system is configured such that an illumination light source excites the sample fluorescence, the dichroic mirror is used to separate the fluorescence and the excitation light, the tube lens projects the image formed by the objective lens at infinity onto the microlens array, the microlens array modulates the image, and the modulated image is imaged onto the camera through the relay lens to obtain a time-sequential light field image; The four-dimensional light field image acquisition system is configured to extract pixels at different angles in the time-sequential light field image and rearrange them, and rearrange the single-frame light field image into a two-dimensional angle-two-dimensional spatial image to obtain a four-dimensional light field image, which is the light field image to be super-resolved.

9. An electronic device, characterized in that: include: at least one processor; and a memory communicatively connected to the processor; wherein the memory stores instructions executable by the processor, and the instructions are executed by the processor to enable the processor to perform the method according to any one of claims 1-6.

10. A computer-readable storage medium storing one or more programs, characterized in that: The one or more programs include computer instructions for causing a computer to execute the method according to any one of claims 1 to 6.

Images