A low-light microscopic image enhancement method and system based on scanning light field

By using scanning light field microscopy and neural network processing, combined with depth reconstruction and image enhancement models, the impact of phototoxicity on the quality of microscopic images was resolved, enabling high-quality live-cell microscopic observation.

CN118587111BActive Publication Date: 2024-11-05ZHEJIANG HEHU TECH CO LTD
View PDF 12 Cites 0 Cited by

Patent Information

Application Number
CN202411055036.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-02
Publication Date
2024-11-05
Estimated Expiration
2044-08-02

AI Technical Summary

Technical Problem

Existing technologies struggle to reduce phototoxicity while maintaining the quality of microscopic images, especially in the observation of living tissues, where the impact of illumination on image quality remains unresolved.

Method used

By employing a scanning light field-based method, combined with a depth reconstruction model and an image enhancement model, image enhancement is achieved through depth information extraction and neural network processing of multi-angle low-light microscopic images.

Benefits of technology

While reducing optical damage, it improves the quality of microscopic images, providing an effective means for long-term, high-resolution live-body observation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118587111B_ABST
    Figure CN118587111B_ABST
Patent Text Reader

Abstract

The present invention discloses a low-light microscopic image enhancement method and system based on scanning light field, which relates to the field of image enhancement technology, and the specific steps are: obtaining data to be enhanced, wherein the data to be enhanced is low-light microscopic images of any sample at multiple angles; inputting the low-light microscopic images of any sample at multiple angles into a depth reconstruction model to obtain a depth map of any sample; pairing the depth map of any sample with the low-light microscopic images of any sample at multiple angles to obtain multiple depth map-low-light microscopic image pairs; inputting multiple depth map-low-light microscopic image pairs into an image enhancement model to obtain high signal-to-noise ratio images of any sample at multiple angles. The present invention achieves high-quality microscopic image capture while greatly reducing phototoxicity, and provides an effective technical means for long-term high-resolution in vivo microscopic observation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image enhancement technology, and more particularly to a low-light microscopic image enhancement method and system based on scanning light field. Background Art

[0002] How to reduce phototoxicity has always been one of the key issues that researchers in the field of microscopic imaging have focused on. Since light can cause irreversible photodamage to living tissues or cells, long-term in vivo microscopic observation technology has been severely limited, greatly restricting the development of research fields such as brain science, immunology, tumor science and pathology.

[0003] In the existing technology, phototoxicity is mostly reduced by changing the illumination method (such as light sheet microscopy) and the acquisition method (such as light field microscopy). Although light field microscopy can achieve three-dimensional imaging of samples through a single shot, it sacrifices a certain spatial resolution. Scanning light field microscopy can make up for the lack of spatial resolution through scanning, but it cannot solve the impact of reduced illumination on image quality. Therefore, how to ensure the quality of microscopic images while reducing phototoxicity is a problem that technical personnel in this field urgently need to solve. Summary of the invention

[0004] In view of this, the present invention provides a low-light microscopic image enhancement method and system based on scanning light field, which overcomes the above-mentioned defects.

[0005] In order to achieve the above object, the present invention adopts the following technical solution:

[0006] A low-light microscopic image enhancement method based on scanning light field, the specific steps are:

[0007] Acquiring data to be enhanced, wherein the data to be enhanced is low-light microscopic images of any sample at multiple angles;

[0008] Inputting the low-light microscopic images of any sample at multiple angles into a depth reconstruction model to obtain a depth map of any sample;

[0009] Pairing the depth map of any sample with the low-light microscopic images of any sample at multiple angles to obtain a plurality of depth map-low-light microscopic image pairs;

[0010] A plurality of the depth map-low-light microscopic image pairs are input into an image enhancement model to obtain high signal-to-noise ratio images of any sample at multiple angles.

[0011] Optionally, the same training set is used in the training process of the depth reconstruction model and the image enhancement model, and the training set is the low-light microscopic images, the high signal-to-noise ratio images and the depth maps of various types of samples at multiple angles.

[0012] Optionally, the method for acquiring each data in the training set is:

[0013] Perform multi-angle shooting based on various samples to obtain the high signal-to-noise ratio images at multiple angles;

[0014] Processing the high signal-to-noise ratio images at multiple angles using a degradation and noise model to obtain the low-light microscopic images at multiple angles;

[0015] The depth map is obtained by using a light field depth estimation algorithm based on the high signal-to-noise ratio images at multiple angles.

[0016] Optionally, the steps of constructing the depth reconstruction model are:

[0017] Build an initial deep reconstruction model based on a convolutional neural network;

[0018] The initial depth reconstruction model is iteratively trained based on the low-light microscopic images at multiple angles and the corresponding depth maps to obtain the depth reconstruction model.

[0019] Optionally, the depth reconstruction model includes an image feature extraction module, a feature fusion module and a disparity regression module.

[0020] Optionally, the steps of constructing the image enhancement model are:

[0021] Build an initial image enhancement model based on convolutional neural network;

[0022] The initial image enhancement model is iteratively trained based on the depth map-low-light microscopic image pair and the high signal-to-noise ratio images at multiple angles to obtain the image enhancement model.

[0023] Optionally, the image enhancement model includes an image feature extraction module, a depth feature extraction module, a feature fusion and enhancement module and a feature regression reconstruction module.

[0024] A low-light microscopic image enhancement system based on scanning light field, comprising:

[0025] A data acquisition module, used to acquire data to be enhanced, wherein the data to be enhanced is a low-light microscopic image of any sample at multiple angles;

[0026] A depth reconstruction module, used for inputting the low-light microscopic images of any sample at multiple angles into a depth reconstruction model to obtain a depth map of any sample;

[0027] A data grouping module, used for grouping the depth map of any sample with the low-light microscopic images of any sample at multiple angles to obtain a plurality of depth map-low-light microscopic image pairs;

[0028] The image enhancement module is used to input a plurality of the depth map-low-light microscopic image pairs into an image enhancement model to obtain high signal-to-noise ratio images of any sample at multiple angles.

[0029] It can be seen from the above technical scheme that compared with the prior art, the present invention discloses a low-light microscopic image enhancement method and system based on scanning light field, in which low-light intensity illumination is performed at the illumination end, and the acquisition end is photographed through a scanning light field microscope, thereby reducing light damage from illumination to acquisition, and combining with a neural network to extract depth information of multi-angle low-light images, and enhancing the low-light images with the extracted depth information in combination with the neural network, thereby greatly reducing phototoxicity and achieving high-quality microscopic image capture, providing an effective technical means for long-term and high-resolution in vivo microscopic observations. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0031] Figure 1 The present invention provides a flow chart of the method. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] On the one hand, this embodiment discloses a low-light microscopic image enhancement method based on scanning light fields, which utilizes the unique feature of scanning light fields that can provide multi-angle high-resolution microscopic images, extracts information from multi-angle low-light microscopic images through a neural network to obtain corresponding depth maps, and inputs the depth maps and low-light images into another neural network for image enhancement to obtain high-resolution and high-signal-to-noise ratio microscopic images at different angles. The specific steps are as follows: Figure 1 As shown, including:

[0034] Step 1, obtaining data to be enhanced, where the data to be enhanced is a low-light microscopic image of any sample at multiple angles;

[0035] Step 2, inputting the low-light microscopic images of any sample at multiple angles into the depth reconstruction model to obtain a depth map of any sample;

[0036] Step 3: Pairing the depth map of any sample with the low-light microscopic images of any sample at multiple angles to obtain a plurality of depth map-low-light microscopic image pairs;

[0037] Step 4: Input multiple depth map-low-light microscopic image pairs into the image enhancement model to obtain high signal-to-noise ratio images of any sample at multiple angles.

[0038] In one embodiment, the same training set is used in the training process of the depth reconstruction model and the image enhancement model, and the training set is low-light microscopic images, high signal-to-noise ratio images, and depth maps of various samples at multiple angles.

[0039] In one embodiment, the method for obtaining data in the training set is:

[0040] Multi-angle shooting based on various samples to obtain high signal-to-noise ratio images at multiple angles;

[0041] The high signal-to-noise ratio images at multiple angles are processed using degradation and noise models to obtain low-light microscopic images at multiple angles.

[0042] The depth map is obtained by using the light field depth estimation algorithm based on high signal-to-noise ratio images at multiple angles.

[0043] Furthermore, the specific steps of constructing the training set are as follows: establishing a light field image dataset I of various biological tissue samples, which includes low-light images of various cell tissue samples at K angles , , …, , high signal-to-noise ratio images and depth map , where a scanning light field microscope is used to capture high signal-to-noise ratio images of various biological tissue samples , …, , and then use the degradation and noise model to classify the high signal-to-noise ratio image Processing to obtain low-light images , where H represents the degradation model, including but not limited to linear motion degradation, Gaussian degradation, color gamut space gamma transformation to reduce image brightness and other methods, N represents noise, including but not limited to Poisson noise, Gaussian noise, etc.; at the same time, the depth map of the corresponding cell tissue sample is obtained by using the light field depth estimation algorithm and multi-view images. ; Among them, the light field depth estimation algorithm includes but is not limited to an EPI-based light field depth estimation algorithm, a multi-view based method, a focus stack based method and a deep learning based method.

[0044] In one embodiment, the steps of constructing the depth reconstruction model are:

[0045] Build an initial deep reconstruction model based on a convolutional neural network;

[0046] The initial depth reconstruction model is iteratively trained based on low-light microscopic images at multiple angles and the corresponding depth maps to obtain a depth reconstruction model.

[0047] In one embodiment, the depth reconstruction model includes an image feature extraction module, a feature fusion module and a disparity regression module.

[0048] Furthermore, the specific steps of building the depth reconstruction model are as follows: building a convolutional neural network Depth_CNN for depth reconstruction, transforming the low-light images at K angles Input Depth_CNN, output estimated depth map , where Depth_CNN consists of an image feature extraction module, a feature fusion module, and a disparity regression module. First, the low-light images at K angles are , The input image feature extraction module performs feature extraction. The extracted features include but are not limited to color features, texture features, shape features and spatial relationship features. The extracted features are then input into the feature fusion module for feature fusion. The feature fusion methods used include but are not limited to weighted average, feature connection, feature selection, feature transformation and other methods. Finally, the disparity regression module is used to reconstruct the disparity and depth map of the fused features:

[0049] ;

[0050] In this embodiment, according to the depth map output by the convolutional neural network Depth_CNN And the true depth map in the training set Calculate the corresponding loss function :

[0051] ;

[0052] based on Train the convolutional neural network Depth_CNN. After the training, the network can output the corresponding depth map through the input low-light images of cell tissue samples from multiple angles.

[0053] In one embodiment, the steps of constructing the image enhancement model are:

[0054] Build an initial image enhancement model based on convolutional neural network;

[0055] The initial image enhancement model is iteratively trained based on depth map-low-light microscopic image pairs and high signal-to-noise ratio images at multiple angles to obtain an image enhancement model.

[0056] In one embodiment, the image enhancement model includes an image feature extraction module, a deep feature extraction module, a feature fusion and enhancement module, and a feature regression reconstruction module.

[0057] In this embodiment, the specific steps of constructing the image enhancement model are as follows: building an enhanced convolutional neural network E_CNN to transform the low-light images at K angles Respectively with the depth map The images are paired and input into the enhanced convolutional neural network E_CNN in turn to obtain high-resolution and high-signal-to-noise ratio microscopic images at K angles output by the network. , where E_CNN consists of an image feature extraction module, a deep feature extraction module, a feature fusion and enhancement module, and a feature regression and reconstruction module. First, the low-light images at K angles are Input image feature extraction module to extract features, and the depth map The deep feature extraction module is input to perform deep feature extraction. The extracted features include but are not limited to color features, geometric structure features, and feature descriptors based on key points. The extracted image features and depth map features are then input into the feature fusion and enhancement module. The fusion methods include but are not limited to weighted average, feature connection, feature selection, feature transformation, etc. Image enhancement is to perform operations such as flipping, rotation, cropping, padding, scaling, rotation, noise blurring, and random color transformation on the data. Finally, the output features are input into the feature regression reconstruction module to obtain high-resolution and high-signal-to-noise ratio microscopic images at K angles. :

[0058] ;

[0059] ;

[0060]

[0061] ;

[0062] High-resolution and high-signal-to-noise ratio microscopic images at K angles output by the enhanced convolutional neural network E_CNN and the high signal-to-noise ratio ground truth images in the training set Calculate the corresponding loss function:

[0063] ;

[0064] based on The enhanced convolutional neural network E_CNN is trained. After the training, the network can perform image enhancement on the input depth map and low-light image to obtain a high-resolution and high-signal-to-noise ratio microscopic image.

[0065] After the training of the deep reconstruction neural network Depth_CNN and the enhanced convolutional neural network E_CNN is completed, a scanning light field microscope is used to collect low-light microscopic images of the sample at multiple angles. The depth map of the sample is first input into Depth_CNN, and then paired with the low-light microscopic images at different angles and input into E_CNN to obtain high signal-to-noise ratio images at different angles.

[0066] The present invention performs low-intensity illumination at the illumination end and photographs through a scanning light field microscope at the collection end, thereby reducing light damage from illumination to collection, and extracting depth information from multi-angle low-light images in combination with a neural network. The extracted depth information is combined with a neural network to enhance the low-light image, thereby greatly reducing phototoxicity and achieving high-quality microscopic image capture, providing an effective technical means for long-term and high-resolution in vivo microscopic observation.

[0067] On the other hand, this embodiment further discloses a low-light microscopic image enhancement system based on scanning light field, including:

[0068] A data acquisition module is used to obtain data to be enhanced, where the data to be enhanced is a low-light microscopic image of any sample at multiple angles;

[0069] A depth reconstruction module is used to input low-light microscopic images of any sample at multiple angles into a depth reconstruction model to obtain a depth map of any sample;

[0070] A data grouping module, used for pairing a depth map of any sample with low-light microscopic images of any sample at multiple angles to obtain a plurality of depth map-low-light microscopic image pairs;

[0071] The image enhancement module is used to input multiple depth map-low-light microscopic image pairs into the image enhancement model to obtain high signal-to-noise ratio images of any sample at multiple angles.

[0072] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0073] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A low-light microscopic image enhancement method based on scanning light field, characterized in that: The specific steps are: Acquiring data to be enhanced, wherein the data to be enhanced is low-light microscopic images of any sample at multiple angles; Inputting the low-light microscopic images of any sample at multiple angles into a depth reconstruction model to obtain a depth map of any sample; Pairing the depth map of any sample with the low-light microscopic images of any sample at multiple angles to obtain a plurality of depth map-low-light microscopic image pairs; Inputting a plurality of the depth map-low-light microscopic image pairs into an image enhancement model to obtain high signal-to-noise ratio images of any sample at multiple angles; The steps of constructing the depth reconstruction model are as follows: Build an initial deep reconstruction model based on a convolutional neural network; Iteratively training the initial depth reconstruction model based on the low-light microscopic images at multiple angles and the corresponding depth maps to obtain the depth reconstruction model; The steps for constructing the image enhancement model are: Build an initial image enhancement model based on convolutional neural network; The initial image enhancement model is iteratively trained based on the depth map-low-light microscopic image pair and the high signal-to-noise ratio images at multiple angles to obtain the image enhancement model.

2. The low-light microscopic image enhancement method based on scanning light field according to claim 1, characterized in that: The same training set is used during the training process of the depth reconstruction model and the image enhancement model, and the training set is the low-light microscopic images, the high signal-to-noise ratio images and the depth maps at multiple angles of various samples.

3. The low-light microscopic image enhancement method based on scanning light field according to claim 2, characterized in that: The method for obtaining each data in the training set is: Perform multi-angle shooting based on various samples to obtain the high signal-to-noise ratio images at multiple angles; Processing the high signal-to-noise ratio images at multiple angles using a degradation and noise model to obtain the low-light microscopic images at multiple angles; The depth map is obtained by using a light field depth estimation algorithm based on the high signal-to-noise ratio images at multiple angles.

4. The low-light microscopic image enhancement method based on scanning light field according to claim 1, characterized in that: The depth reconstruction model includes an image feature extraction module, a feature fusion module and a disparity regression module.

5. The low-light microscopic image enhancement method based on scanning light field according to claim 1, characterized in that: The image enhancement model includes an image feature extraction module, a depth feature extraction module, a feature fusion and enhancement module and a feature regression reconstruction module.

6. A low-light microscopic image enhancement system based on scanning light field, characterized in that: include: A data acquisition module, used to acquire data to be enhanced, wherein the data to be enhanced is a low-light microscopic image of any sample at multiple angles; A depth reconstruction module, used for inputting the low-light microscopic images of any sample at multiple angles into a depth reconstruction model to obtain a depth map of any sample; the depth reconstruction model is obtained by iteratively training an initial depth reconstruction model constructed based on a convolutional neural network based on the low-light microscopic images at multiple angles and the corresponding depth maps; A data grouping module, used for grouping the depth map of any sample with the low-light microscopic images of any sample at multiple angles to obtain a plurality of depth map-low-light microscopic image pairs; An image enhancement module, used for inputting a plurality of the depth map-low-light microscopic image pairs into an image enhancement model to obtain high signal-to-noise ratio images of any sample at multiple angles; The image enhancement model is obtained by iteratively training an initial image enhancement model based on a convolutional neural network based on the depth map-low-light microscopic image pair and the high signal-to-noise ratio image pair at multiple angles.

Citation Information

Patent Citations

  • Low-illumination image enhancement method based on saliency foreground content

    CN111275642A

  • Fluorescence microscopic image super-resolution reconstruction method based on deep learning

    CN111524064A

  • Low-illumination image enhancement method based on deep learning

    CN114092352A

  • Weak light multi-view geometric reconstruction method based on deep learning

    CN114332355A

  • Low-illumination image enhancement method based on convolutional neural network

    CN116523794A