Unsupervised learning method, system, device and medium for live microscopic super-resolution imaging

Abstract

This invention relates to the field of computer vision technology, and discloses an unsupervised learning method, system, device, and medium for live-body microscopic super-resolution imaging. The method includes: simulating diffraction on a shallow super-resolution structural image to obtain a diffraction degradation image; convolving the diffraction degradation image to obtain a scattering degradation image; adding image noise to the scattering degradation image to obtain a simulated fluorescence image; training a super-resolution reconstruction network using the shallow super-resolution structural image and the simulated fluorescence image to obtain a target super-resolution reconstruction network; and using the target super-resolution reconstruction network to perform super-resolution reconstruction on the live-body image to be reconstructed, thereby obtaining live-body microscopic super-resolution imaging. Through the implementation of this invention, the true morphology of biological structures can be preserved, reducing the problems of pseudo-structures and missing details that easily occur in traditional super-resolution reconstruction processes, thus achieving accurate microscopic super-resolution imaging.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to an unsupervised learning method, system, device, and medium for live microscopic super-resolution imaging. Background Technology

[0002] Fluorescence microscopy enables non-contact, highly specific imaging observation of biological samples, and has significant application value in fields such as neuroscience, biomedicine, and disease research. However, in the process of in vivo deep tissue microscopy imaging, due to factors such as optical diffraction limit, tissue scattering, and attenuation of effective fluorescence signal, existing fluorescence microscopy techniques still struggle to simultaneously meet the imaging requirements of high spatial resolution, deep-depth imaging, and high signal-to-noise ratio.

[0003] On the one hand, traditional wide-field fluorescence microscopy and confocal microscopy are constrained by the optical diffraction limit, and their spatial resolution is usually insufficient to meet the needs of observing nanoscale biological structures. To overcome the diffraction limit, existing technologies have developed a variety of super-resolution fluorescence microscopy imaging methods, including stimulated emission depletion microscopy (STED), structured illumination microscopy (SIM), and single-molecule localization microscopy (SMLM).

[0004] STED (Spatial Depletion Therapy) technology, by spatially modulating a depleted beam to confine the fluorescence emission region, can achieve spatial resolution superior to traditional microscopy systems. Under shallow or low-scattering conditions, STED can achieve lateral resolution at the tens of nanometer scale. However, when this technology is applied to complex scattering media such as living brain tissue, the effective imaging depth is extremely limited. During propagation, the depleted light is affected by non-homogeneous media such as internal cellular structures, myelin sheaths, blood vessels, and the extracellular matrix, resulting in significant multiple scattering and wavefront distortion. This prevents the depleted region from effectively confining the fluorescence emission range, leading to a decrease in spatial resolution. Simultaneously, as the imaging depth increases, the number of effectively detected photons decreases, significantly reducing the image signal-to-noise ratio.

[0005] To improve deep imaging capabilities, existing technologies have proposed an improved scheme combining adaptive optics correction (STED) and two-photon excitation. Adaptive optics STED corrects the incident light field by measuring and compensating for wavefront aberrations caused by tissue. Two-photon excitation STED utilizes near-infrared two-photon excitation to reduce scattering effects and improve fluorescence excitation efficiency in deep tissues. While these methods can reduce aberration effects to some extent and increase imaging depth to over 100 micrometers, they still rely on the optical system to directly acquire super-resolution information. Their imaging performance is constrained by the tissue scattering limit, and their effective imaging depth and quality remain limited.

[0006] In existing technologies, traditional SIM (Structured Illumination) techniques can improve lateral resolution, but their effective depth is typically only a few micrometers. Hessian structured illumination microscopy and triangular beam interference structured illumination microscopy can improve frame rate, but they do not effectively extend imaging depth. While adaptive optics two-photon multifocal structured illumination microscopy has achieved super-resolution imaging in 300 μm brain slices, its single-frame time is on the order of minutes, making it unsuitable for in vivo dynamic observation. Single-molecule localization microscopy series, such as stochastic optical reconstruction microscopy and photoactivated localization microscopy, can improve resolution. However, their reconstruction nature based on the SMLM principle determines the long imaging time, limiting their application in in vivo dynamic imaging. Therefore, existing methods based on optical system optimization struggle to achieve continuous and stable super-resolution imaging of deep regions of living tissue.

[0007] On the other hand, to overcome the insufficient ability of traditional optical systems to acquire super-resolution information, existing technologies have further proposed methods for super-resolution reconstruction of microscopic images using deep learning. These methods are typically based on supervised learning frameworks, establishing a mapping relationship between low-quality microscopic images and high-quality ground truth (GT) images to recover super-resolution structural information from low-resolution images. The success of supervised learning highly depends on massive amounts of high-quality and precisely paired training data. In shallow or artificially prepared samples, super-resolution GT images can be directly acquired through super-resolution microscopy systems, thus these methods can achieve good reconstruction results. However, in deep tissues of living organisms, due to the combined effects of tissue scattering, aberrations, and light signal attenuation, it is experimentally difficult to simultaneously acquire both a deep scattering-degraded image and a corresponding scatter-free super-resolution GT image at the same spatial location. This data hunger limitation makes existing supervised learning paths ineffective in deep super-resolution tasks, leading to…

[0008] To address the issue of supervised learning methods relying on ground truth (GT) data, existing technologies have proposed unsupervised and self-supervised learning methods. For example, the excitation mode descattering sum-of-squares self-heuristic noise-to-noise method has achieved descattering and denoising without GT data. However, the exploration of such methods is still in its early stages, and their core objective is to improve the image signal-to-noise ratio. This cannot meet the requirements of deep tissue super-resolution reconstruction for nanoscale structure restoration, resulting in poor accuracy of existing in vivo depth microscopic super-resolution imaging.

[0009] Therefore, improving the accuracy of in vivo super-resolution microscopy has become an urgent problem to be solved. Summary of the Invention

[0010] This invention provides an unsupervised learning method, system, device, and medium for in vivo super-resolution microscopy, the main purpose of which is to solve the problem of poor accuracy in in vivo super-resolution microscopy.

[0011] To achieve the above objectives, this invention provides an unsupervised learning-based live-body microscopic super-resolution imaging method, comprising: acquiring a shallow super-resolution structural image; calculating the relative pixel intensity of each pixel in the shallow super-resolution structural image based on the imaging parameters of the shallow super-resolution structural image; constructing a discrete diffraction point spread function template based on the pixel relative intensity; performing Gaussian approximation on the discrete diffraction point spread function template to obtain the diffraction point spread function corresponding to the shallow super-resolution structural image; performing convolution processing on the shallow super-resolution structural image using the diffraction point spread function to obtain a diffraction degradation image; simulating the propagation trajectory of photons within the tissue corresponding to the diffraction degradation image to construct a scattering point spread function; performing convolution processing on the diffraction degradation image using the scattering point spread function to obtain a scattering degradation image; adding image noise to the scattering degradation image to obtain a simulated fluorescence image; training a pre-constructed super-resolution reconstruction network using the shallow super-resolution structural image and the simulated fluorescence image to obtain a target super-resolution reconstruction network; and performing super-resolution reconstruction on the live-body image to be reconstructed using the target super-resolution reconstruction network to obtain live-body microscopic super-resolution imaging.

[0012] This invention also provides an unsupervised learning-based live-body microscopic super-resolution imaging system, comprising: a simulated diffraction module for acquiring shallow super-resolution structure images; calculating the relative pixel intensity of each pixel in the shallow super-resolution structure image based on imaging parameters of the shallow super-resolution structure image; constructing a discrete diffraction point spread function template based on the pixel relative intensity; performing Gaussian approximation on the discrete diffraction point spread function template to obtain the diffraction point spread function corresponding to the shallow super-resolution structure image; and performing convolution processing on the shallow super-resolution structure image using the diffraction point spread function to obtain a diffraction degradation image; and a simulated scattering module for... The system simulates the propagation trajectory of photons within the corresponding tissue of a diffraction degradation image to construct a scattering point spread function. This scattering point spread function is then used to convolve the diffraction degradation image to obtain a scattering degradation image. A noise addition module adds image noise to the scattering degradation image to obtain a simulated fluorescence image. A model training module trains a pre-constructed super-resolution reconstruction network using shallow super-resolution structural images and simulated fluorescence images to obtain a target super-resolution reconstruction network. A super-resolution reconstruction module uses the target super-resolution reconstruction network to perform super-resolution reconstruction on the live image to be reconstructed, resulting in live microscopic super-resolution imaging.

[0013] The present invention also provides an electronic device, comprising: a memory communicatively connected to at least one processor; wherein the processor is configured to execute a computer program stored in the memory; the memory stores a computer program executable by at least one processor, the computer program being executed by at least one processor to enable at least one processor to perform the above-described unsupervised learning method for super-resolution microscopy.

[0014] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements an unsupervised learning method for live-body super-resolution microscopy based on any of the above-mentioned methods.

[0015] This invention utilizes deep learning networks to learn the mapping relationship between deep low-resolution images and high-resolution structures by diffraction and scattering shallow super-resolution structural images. This allows for the recovery of high-frequency structural information from deep two-photon fluorescence images affected by diffraction and scattering, improving the spatial resolution of deep imaging. Furthermore, by training a pre-constructed super-resolution reconstruction network using shallow super-resolution structural images and simulated fluorescence images, the network can simulate the propagation, scattering, absorption, and noise detection processes of photons in biological tissues, achieving a positive mapping from shallow super-resolution images to deep low-quality fluorescence images. Finally, by using a target super-resolution reconstruction network to perform super-resolution reconstruction on the in vivo image to be reconstructed, the true biological structural morphology can be preserved, reducing the problems of pseudo-structures and detail loss that easily occur in traditional super-resolution reconstruction, thereby achieving accurate microscopic super-resolution imaging. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating an unsupervised learning method for live-body super-resolution microscopy provided in an embodiment of the present invention. Figure 2 A visualization of the target scattering image generation process provided in an embodiment of the present invention; Figure 3 A model architecture diagram of a live-cell microscopic super-resolution imaging method provided in an embodiment of the present invention; Figure 4 This is a functional block diagram of an unsupervised learning live microscopic super-resolution imaging system provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of an electronic device for implementing an unsupervised learning method for live-body microscopic super-resolution imaging, according to an embodiment of the present invention.

[0017] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0018] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0019] This application provides an unsupervised learning-based live-body super-resolution microscopy imaging method. The execution subject of this unsupervised learning-based live-body super-resolution microscopy imaging method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application: a server, a terminal, etc. In other words, the unsupervised learning-based live-body super-resolution microscopy imaging method can be executed by software or hardware installed on a terminal device or a server device; the software can be a blockchain platform. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster. The server can be an independent server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0020] Reference Figure 1 The diagram shown is a flowchart illustrating an unsupervised learning-based in vivo super-resolution microscopy imaging method according to an embodiment of the present invention. In this embodiment, the unsupervised learning-based in vivo super-resolution microscopy imaging method includes: S1. Obtain a shallow super-resolution structure image. Calculate the relative intensity of each pixel in the shallow super-resolution structure image based on its imaging parameters. Construct a discrete diffraction point spread function template based on the relative pixel intensities. Perform Gaussian approximation on the discrete diffraction point spread function template to obtain the diffraction point spread function corresponding to the shallow super-resolution structure image. Convolve the shallow super-resolution structure image using the diffraction point spread function to obtain a diffraction degradation image.

[0021] In this embodiment of the invention, the shallow super-resolution structural image is a shallow two-photon stimulated emission depletion super-resolution image, which is a nanoscale super-resolution structural image directly acquired in the shallow layer of tissue (tens of micrometers deep, in living samples) by combining two-photon infrared nonlinear excitation and stimulated emission depletion microscopy (STED) ring-loss light compression fluorescence emission region.

[0022] Furthermore, simulating diffraction in shallow super-resolution structural images simulates the spatial resolution degradation process caused by optical diffraction limitations in traditional fluorescence microscopy systems under unorganized scattering conditions. For example, it simulates the spatial resolution degradation process caused by finite numerical aperture and optical diffraction effects.

[0023] In this embodiment of the invention, the original fluorescence grayscale data of the shallow super-resolution structure image and the corresponding imaging system parameters are read. The parameters include: imaging lateral sampling step size (pixel physical size), fluorescent molecular label concentration distribution, detector grayscale response curve, and imaging baseline noise floor. The system baseline grayscale is subtracted from the grayscale values ​​of all pixels in the image to obtain the net fluorescence grayscale after removing the background. All pixels in the entire shallow super-resolution structure image are traversed, and the maximum value of the net fluorescence grayscale is extracted as the intensity normalization benchmark. The net fluorescence grayscale of each pixel is divided by the intensity benchmark value to obtain the relative intensity of the pixel in the range of 0 to 1.

[0024] Furthermore, based on the numerical aperture of the objective lens and the excitation wavelength of the microscope system used for imaging shallow super-resolution structural images, the effective radius of the diffraction spot is determined. Using the pixel physical size of the shallow super-resolution image as the spatial sampling interval of the discrete template, a two-dimensional square discrete grid is established as the template carrier. With the center of the template grid as the ideal point source position, based on Abbe diffraction theory, discrete sampling values ​​of the Airy disk intensity on the discrete grid are constructed to obtain the original discrete diffraction intensity distribution. The diffraction intensity of all grid points within the template is normalized by dividing it by the total intensity of the template. This normalized discrete diffraction grid is used as the template for the discrete diffraction point diffusion function.

[0025] In this embodiment of the invention, the light intensity sampling coordinates and corresponding light intensity values ​​of the discrete diffraction template are extracted to obtain the full width at half maximum (FWHM) of the diffraction spot. Based on the FWHM, a two-dimensional isotropic Gaussian function is used to perform least-squares fitting on the light intensity distribution of the discrete diffraction template to obtain a fitted continuous Gaussian function. The continuous Gaussian function is then discretized and resampled again, with the sampling interval consistent with the physical size of the pixels in the shallow super-resolution image, to obtain the diffraction point spread function of the imaging system and image sampling grid adapted to the shallow super-resolution structure image.

[0026] Furthermore, the diffraction point spread function is converted into a two-dimensional object distribution matrix composed of the relative intensities of pixels, where the matrix rows and columns correspond one-to-one with the image pixel grid. Using the diffraction point spread function as a convolution kernel, a two-dimensional linear convolution method is employed to perform pixel-by-pixel convolution between the object distribution matrix and the diffraction point spread function, resulting in a diffraction degradation image. Specifically, a pre-constructed diffraction convolution formula can be used to perform convolution processing on shallow super-resolution structure images. The diffraction convolution formula is expressed as follows:

[0027] in, This is a diffraction degradation image. The diffraction point spread function, This represents the convolution operation. This is a shallow super-resolution structural image. For shallow super-resolution structural images on the x-axis and ordinate The coordinates at that location.

[0028] In this embodiment of the invention, the diffraction degradation image can simulate the diffraction-limited blurred observation image acquired by a real microscope, realizing the simulation degradation from the nanoscale super-resolution true image to the ordinary diffraction blurred image, thereby improving the accuracy of super-resolution imaging.

[0029] S2. Simulate the propagation trajectory of photons within the tissue corresponding to the diffraction degradation image to construct the scattering point spread function. Use the scattering point spread function to convolve the diffraction degradation image to obtain the scattering degradation image.

[0030] In this embodiment of the invention, a Monte Carlo photon transmission model is established based on the photon propagation law inside biological tissues in the scattering degradation image. By randomly simulating the absorption, scattering and direction change process of a large number of photons during propagation inside the tissue, the scattering point spread function corresponding to different imaging depth conditions is obtained. The obtained scattering point spread function is used to further convolve the diffraction degradation image to simulate the multiple scattering effect in deep tissues.

[0031] Specifically, simulating the propagation trajectory of photons within the tissue corresponding to the diffraction degradation image to construct a scattering point diffusion function includes: obtaining the medium optical parameters corresponding to the shallow super-resolution structural image, generating a virtual photon packet based on the medium optical parameters; simulating the scattering and absorption trajectory of the virtual photon packet to filter out the outgoing photons in the virtual photon packet; statistically analyzing the two-dimensional simulation coordinates of the back projection of the outgoing photons, calculating the statistical histogram of the two-dimensional simulation coordinates; and generating the scattering point diffusion function based on the statistical histogram.

[0032] In this embodiment of the invention, the medium optical parameters include the medium absorption coefficient, scattering coefficient, anisotropy factor, and medium refractive index, which are used to determine the three-dimensional spatial range of the photon simulation. The relative intensity of pixels in the shallow super-resolution structural image is used as the distribution of photon emission sources. A number of photons matching the relative intensity of the pixels are generated at each emitting pixel to construct a virtual photon packet carrying initial position, propagation direction, and photon weight information, thereby completing the simulation initialization.

[0033] Furthermore, the scattering and absorption trajectory tracking simulation utilizes biological tissue optical Monte Carlo photon tracking software (such as MCML, MultiLayer Monte Carlo, and other open-source software) to simulate photon trajectories. Specifically, the free propagation step size of each virtual photon packet is randomly sampled, and the propagation direction is updated after the photon reaches the scattering point. During propagation, the photon weight is attenuated according to the medium absorption coefficient. The photon trajectory is continuously tracked, and when a photon exits the simulation area from the upper surface of the shallow tissue layer, it is determined to be a valid outgoing photon and retained. Photons absorbed by the medium or escaping from other boundaries are discarded, thus achieving outgoing photon screening.

[0034] In detail, a two-dimensional receiving plane with the sampling interval consistent with the shallow super-resolution image is established. All effective emitted photons are back-projected onto the receiving plane along their respective emission directions to obtain the two-dimensional simulation coordinates corresponding to each photon. Using the receiving plane grid as the statistical interval, the photon weights are accumulated by the grid to generate a statistical histogram of the two-dimensional simulation coordinates, and the histogram is energy normalized.

[0035] In this embodiment of the invention, the normalized two-dimensional statistical histogram is used as the scattering point spread function; the scattering point spread function is used as the convolution kernel to perform two-dimensional convolution operation on the diffraction degradation image. During the convolution process, the image boundary is extended to suppress edge artifacts. The secondary diffusion blurring of fluorescence signal caused by scattering of shallow tissue in living organism is simulated by convolution, and the scattering degradation image is output to complete the simulation of tissue scattering effect.

[0036] In detail, by superimposing scattering degradation after diffraction degradation, the two-stage imaging degradation mechanism of diffraction blurring of the microscopic system and scattering blurring of living tissue is reproduced in sequence, which is closer to the actual acquired images of shallow microscopy in living tissue. It can be used to construct a high-fidelity simulation dataset, which is beneficial to improving the accuracy of super-resolution reconstruction network training.

[0037] S3. Add image noise to the scattering degradation image to obtain a simulated fluorescence image.

[0038] In this embodiment of the invention, the image noise is achieved by adding a random noise model that conforms to the fluorescence microscopy acquisition process to the scattering degradation image, including Gaussian noise and Poisson noise, to simulate the electronic noise, photon statistical noise and detector noise generated in the actual microscopic imaging process.

[0039] In detail, adding image noise to a scattering degradation image to obtain a simulated fluorescence image includes: randomly generating a noise signal including Gaussian noise and Poisson noise; converting the noise signal into a noise signal image; and pixel-wise superimposing the noise signal image with the scattering degradation image to obtain a simulated fluorescence image.

[0040] In detail, Gaussian noise can be generated by randomly sampling the pixel grid of the scattering degradation image to simulate detector readout noise and circuit noise; Poisson noise can be generated by randomly sampling pixel by pixel based on the grayscale signal of each pixel in the scattering degradation image as the expectation of the Poisson distribution to simulate photon shot noise in fluorescence imaging. The Poisson noise component and the Gaussian noise component are added to randomly generate a mixed noise signal containing Gaussian and Poisson noise.

[0041] Furthermore, a two-dimensional grid of the same size as the scattering degradation image is constructed. The amplitudes of the mixed noise signals are then filled into the corresponding pixel positions of the two-dimensional grid to form a two-dimensional noise intensity matrix. This two-dimensional noise intensity matrix is ​​mapped onto the detector's grayscale range to obtain an independent noise signal image. Point-to-point pixel superposition is then performed on the noise signal image and the scattering degradation image, adding the grayscale values ​​of the pixels in the scattering degradation image and the noise signal image at the same spatial coordinates to obtain a simulated fluorescence image.

[0042] In this embodiment of the invention, the step of generating a simulated fluorescence image can be specifically represented as follows:

[0043]

[0044] in, For scattering degradation images, This is a shallow super-resolution structural image. The diffraction point spread function, For convolution processing, Let be the scattering point spread function. To simulate fluorescence images, This is an image of a noise signal.

[0045] In detail, the signal value of the noise signal can be changed to obtain simulated fluorescence images with different noise values, and the tissue optical parameters and imaging depth of the shallow super-resolution structure image can be changed. The shallow super-resolution structure image and the corresponding simulated fluorescence image are used as a virtual paired dataset as the training set for the subsequent super-resolution reconstruction network.

[0046] Specifically, see Figure 2 The diagram illustrates the generation process of the target scattering image. The shallow super-resolution structure image serves as the scatter-free super-resolution image, which is then processed... Convolution blurring is performed to obtain a diffraction-degraded image (low-resolution, non-scattering image), then... Convolutional scattering yields a scattering degradation image (low-resolution scattering image). Finally, image noise is added to the low-resolution scattering image to obtain a simulated fluorescence image, i.e., the target scattering image.

[0047] S4. Train the pre-constructed super-resolution reconstruction network using shallow super-resolution structural images and simulated fluorescence images to obtain the target super-resolution reconstruction network.

[0048] In this embodiment of the invention, the pre-constructed super-resolution reconstruction network is a deep super-resolution image reconstruction network based on the attention mechanism.

[0049] Specifically, a pre-constructed super-resolution reconstruction network is trained using shallow super-resolution structural images and simulated fluorescence images to obtain a target super-resolution reconstruction network. This process includes: performing feature transformation on the simulated fluorescence image to obtain initial image features; downsampling the initial image features using the encoder in the super-resolution reconstruction network to obtain downsampled image features; performing attention enhancement processing on the downsampled image features using the attention enhancement module in the super-resolution reconstruction network to obtain attention-enhanced features; upsampling the attention-enhanced features using the decoder in the super-resolution reconstruction network to obtain downsampled image features; performing activation processing on the downsampled image features to obtain a predicted super-resolution image; calculating the multimodal loss function value between the predicted super-resolution image and the shallow super-resolution structural image using a pre-defined multimodal loss function; and optimizing the parameters of the super-resolution reconstruction network using the multimodal loss function value to obtain the target super-resolution reconstruction network.

[0050] In this embodiment of the invention, the super-resolution reconstruction network takes the simulated fluorescence image after tissue scattering degradation as the network input and first performs feature transformation through a convolutional layer with a kernel size of 2 × 2 and a stride of 2 to obtain the initial image features, thereby reducing the image dimension and computational cost.

[0051] Secondly, to progressively extract multi-scale structural information from deep fluorescence images, an encoding feature extraction mechanism is implemented. Preferably, the encoder consists of four downsampling modules, each responsible for extracting spatial feature information. Each downsampling module contains two residual channel attention modules and one convolutional layer. The residual channel attention modules utilize a channel attention mechanism to highlight structural and key image details. This mechanism enables the network to focus on information channels that are important for descattering and super-resolution tasks, enhancing the perception of high-frequency information such as edges and textures.

[0052] In order for the network to capture long-range dependencies across image regions and further improve its ability to restore fine details, an attention enhancement module is set between the encoder and decoder.

[0053]

[0054] in, , , These are the query matrix, key matrix, and value matrix corresponding to the features of the downsampled image, respectively. For feature dimensions; This represents the positional bias term. Through the above calculation method, the network can comprehensively consider the correlation information between different spatial regions, improving its ability to identify weak structural signals under low signal-to-noise ratio conditions.

[0055] Furthermore, the attention-enhancing features are upsampled using the decoder in the super-resolution reconstruction network to obtain downsampled image features. This includes: performing residual channel attention processing on the attention-enhancing features using the decoder to obtain residual channel attention features; performing deconvolution on the residual channel attention features to obtain deconvolution features; performing residual connection between the deconvolution features and the attention-enhancing features to obtain residual connection features; and performing spatial upsampling and feature channel mapping on the residual connection features using the deconvolution layer in the decoder to obtain downsampled image features.

[0056] The decoder network upsamples the attention-enhanced features to predict the image and symmetrically contains four upsampling modules. Each upsampling module consists of two residual channel attention modules and a deconvolution layer to achieve fine-grained feature reconstruction. Corresponding layers of the encoder and decoder are concatenated via skip connections, which helps retain more detail during decoding. After the last upsampling module in the decoder, the network further performs spatial upsampling through a deconvolution layer with a kernel size of 2 × 2 and a stride of 2, doubling the spatial size of the feature map and integrating the 64-channel feature maps into a single channel to obtain downsampled image features, which are then used to generate the predicted super-resolution image through the Tanh activation function.

[0057] To eliminate scattering and restore image detail, a composite loss function is used to optimize the network parameters. The loss function includes at least the following: mean squared error (MSE), structural similarity index (SSIM), and dice similarity coefficient (MCE_DSC) based on threshold segmentation.

[0058] The multimodal loss function is expressed as:

[0059] in, This represents the value of the multimodal loss function. This represents the preset weight parameters for the first loss term. Indicates the prediction of super-resolution images With shallow super-resolution structural images The predicted loss value between This represents the preset weight parameters for the second loss term. Indicates the prediction of super-resolution images With shallow super-resolution structural images Image quality loss values ​​between The preset weight parameters for the third loss term, Indicates the prediction of super-resolution images With shallow super-resolution structural images The minimum cross-entropy loss value between them.

[0060] The MSE loss penalizes the difference between the network output and the ground truth to ensure prediction accuracy, while the SSIM loss improves the perceptual quality fidelity of the output. To further optimize model performance, MCE_DSC is introduced into the loss function. Minimum cross-entropy (MCE) is a widely used global thresholding technique in image segmentation, dividing an image into foreground and background. This technique is used to segment the network output and the ground truth, obtaining their respective segmentation maps, and the dice similarity coefficient (DSC) between them is calculated.

[0061] Specifically, the process iterates through all pixels, calculates the difference between the grayscale values ​​of pixels in the predicted super-resolution image and those in the shallow super-resolution structure image, squares the difference, sums the squared differences of all pixels, and divides the sum by the total number of pixels in the image to obtain the mean square error (MSE). The smaller the MSE value, the smaller the difference in the overall grayscale distribution between the two images.

[0062] In detail, the Structural Similarity Index (SSIM) is calculated as follows: A sliding local window is set up on both images, moving position by position along the image plane. Within each window, the mean gray level, standard deviation gray level, and pixel covariance of the ground truth image and the predicted image are calculated. The local SSIM value of the window is calculated using a combination of luminance, contrast, and structure terms. The average of the local SSIM values ​​for all sliding windows is then used to obtain the global SSIM index for the entire image. The closer the SSIM is to 1, the higher the similarity in local texture, edges, and spatial structure between the two images.

[0063] Further calculation of the dice similarity coefficient MCE_DSC based on threshold segmentation: A uniform grayscale threshold was selected, and threshold segmentation was performed on both the shallow super-resolution structure image and the predicted super-resolution image to generate corresponding binary mask images. Fluorescent structure regions were labeled as foreground pixels, and other regions were labeled as background pixels. The number of pixels overlapping between the ground truth and predicted foreground, the number of predicted false positive pixels, and the number of missed false negative pixels were counted. MCE_DSC was then calculated using the dice similarity coefficient formula. The closer MCE_DSC is to 1, the higher the degree of overlap between the reconstructed biofluorescent structure contour and the ground truth structure.

[0064] In this embodiment of the invention, an encoder-decoder network structure comprising a residual channel attention module and a spatial attention enhancement module is employed. Through multi-scale feature extraction, channel feature selection, and long-range spatial dependency modeling, the network can effectively identify weak signal structures in deep fluorescence images and enhance the recovery capability of high-frequency information such as edges and textures. Simultaneously, by combining mean squared error loss, structural similarity loss, and segmentation consistency-based loss values, the intensity and structural information of the predicted super-resolution image are jointly constrained. This ensures that the recovery result not only has high pixel consistency but also maintains the morphology of realistic biological structures, reducing the pseudo-structure and detail loss problems that easily occur in traditional super-resolution reconstruction. The network improves spatial resolution while maintaining the realism of biological structures, avoiding structural artifacts caused by over-enhancement, thereby improving the accuracy of subsequent super-resolution reconstruction.

[0065] S5. Use the target super-resolution reconstruction network to perform super-resolution reconstruction on the live image to be reconstructed, and obtain live microscopic super-resolution imaging.

[0066] In this embodiment of the invention, the trained target super-resolution reconstruction network is applied to actual live in vivo deep imaging data. The original two-photon fluorescence image to be reconstructed, i.e., the live in vivo image to be reconstructed, is input into the trained target super-resolution reconstruction network. The target super-resolution reconstruction network performs feature extraction, scattering compensation, and super-resolution reconstruction on the input live in vivo image based on the learned optical degradation rules and structural restoration mapping relationship, and outputs the corresponding super-resolution image. This process can achieve the recovery from a low-quality deep fluorescence image to a super-resolution image without the need to acquire an additional deep super-resolution gold standard image.

[0067] See Figure 3 The diagram shown illustrates the model architecture of the unsupervised learning-based live-body microscopy super-resolution imaging method of this invention. Wherein, Figure 3 The simulated dataset represents shallow super-resolution structural images and simulated fluorescence images. The Monte Carlo physical model represents the pre-constructed super-resolution reconstruction network. The shallow super-resolution structural images are used as ground truth images. The error between the predicted super-resolution images and the ground truth images is calculated. The backpropagation algorithm is used to iteratively update the network parameters to complete the model convergence.

[0068] Finally, experimental data verification was carried out: real low-resolution images acquired by live microscopy were input into the trained network, and the network output super-resolution reconstructed images; the reconstructed images were compared with shallow super-resolution structure (gold standard) images pre-acquired in the experiment, and the quality of super-resolution reconstruction was quantitatively evaluated using multimodal methods, realizing the complete process from simulation learning to super-resolution restoration of real microscopic images.

[0069] like Figure 4 The diagram shown is a functional block diagram of an unsupervised learning live microscopic super-resolution imaging system provided in an embodiment of the present invention.

[0070] The unsupervised learning-based live-body microscopy super-resolution imaging system 100 of this invention can be installed in an electronic device. Depending on the functions implemented, the unsupervised learning-based live-body microscopy super-resolution imaging system 100 may include a simulated diffraction module 101, a simulated scattering module 102, a noise addition module 103, a model training module 104, and a super-resolution reconstruction module 105. A module, also referred to as a unit, is a series of computer program segments that can be executed by the processor of an electronic device and perform a fixed function, and is stored in the memory of the electronic device.

[0071] In this embodiment, the functions of each module / unit are as follows: The simulated diffraction module 101 is used to acquire shallow super-resolution structure images. It calculates the relative intensity of each pixel in the shallow super-resolution structure image based on the imaging parameters of the image. A discrete diffraction point spread function template is constructed based on the pixel relative intensities. A Gaussian approximation is applied to the discrete diffraction point spread function template to obtain the diffraction point spread function corresponding to the shallow super-resolution structure image. The shallow super-resolution structure image is then convolved using the diffraction point spread function to obtain a diffraction degradation image. The simulated scattering module 102 is used to simulate the propagation trajectory of photons within the tissue corresponding to the diffraction degradation image to construct the scattering point spread function. The scattering point spread function is then used to convolve the diffraction degradation image to obtain the scattering degradation image. The noise addition module 103 is used to add image noise to the scattering degradation image to obtain a simulated fluorescence image; The model training module 104 is used to train a pre-constructed super-resolution reconstruction network using shallow super-resolution structural images and simulated fluorescence images to obtain the target super-resolution reconstruction network. The super-resolution reconstruction module 105 is used to perform super-resolution reconstruction of the live image to be reconstructed using the target super-resolution reconstruction network, so as to obtain live microscopic super-resolution imaging.

[0072] In detail, each module in the unsupervised learning live-body microscopic super-resolution imaging system 100 of this embodiment of the invention adopts the same approach as described above during use. Figures 1 to 3 The technique is the same as that used in unsupervised learning for live-body microscopic super-resolution imaging, and can produce the same technical effects, so it will not be elaborated here.

[0073] like Figure 5 The diagram shown is a schematic representation of an electronic device for implementing an unsupervised learning method for super-resolution microscopy in vivo, according to an embodiment of the present invention.

[0074] The electronic device 500 may include a processor 501, a memory 502, a communication bus 503, and a communication interface 504. It may also include a computer program stored in the memory and capable of running on the processor, such as an unsupervised learning live microscopic super-resolution imaging method program.

[0075] In some embodiments, the processor 501 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips.

[0076] The memory 502 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 502 may be an internal storage unit of an electronic device, such as a portable hard drive of the electronic device.

[0077] The communication bus 503 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 502 and at least one processor 501.

[0078] The communication interface 504 is used for communication between the aforementioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, Bluetooth interface, etc.), typically used to establish communication connections between the electronic device and other electronic devices. The user interface may be a display or an input unit (such as a keyboard). Optionally, the user interface may also be a standard wired interface or a wireless interface.

[0079] Figure 5 The image only shows electronic devices with components; it will be understood by those skilled in the art that... Figure 5 The structure shown does not constitute a limitation on the electronic device 500, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0080] Specifically, the processor's implementation method for the above instructions can be found in the description of the relevant steps in the corresponding embodiments in the accompanying drawings, and will not be repeated here.

[0081] In the several embodiments provided by this invention, it should be understood that the disclosed devices, systems, and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0082] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0083] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0084] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0085] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0086] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0087] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or systems stated in a system claim may also be implemented by a single unit or system through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. An unsupervised learning method for in vivo super-resolution microscopy, characterized in that, The method includes: A shallow super-resolution structure image is acquired. The relative pixel intensity of each pixel in the shallow super-resolution structure image is calculated based on the imaging parameters of the image. A discrete diffraction point spread function template is constructed based on the relative pixel intensity. A Gaussian approximation is applied to the discrete diffraction point spread function template to obtain the diffraction point spread function corresponding to the shallow super-resolution structure image. The shallow super-resolution structure image is then convolved using the diffraction point spread function to obtain a diffraction degradation image. The propagation trajectory of photons within the tissue corresponding to the diffraction degradation image is simulated to construct a scattering point spread function. The scattering point spread function is then used to convolve the diffraction degradation image to obtain a scattering degradation image. Adding image noise to the scattering degradation image yields a simulated fluorescence image; The pre-constructed super-resolution reconstruction network is trained using the shallow super-resolution structural image and the simulated fluorescence image to obtain the target super-resolution reconstruction network; The target super-resolution reconstruction network is used to perform super-resolution reconstruction of the live image to be reconstructed, resulting in live microscopic super-resolution imaging.

2. The unsupervised learning-based live-body microscopic super-resolution imaging method as described in claim 1, characterized in that, The step of convolving the diffraction degradation image using a pre-constructed scattering point spread function to obtain a scattering degradation image includes: Obtain the medium optical parameters corresponding to the shallow super-resolution structure image, and generate virtual photonic packets based on the medium optical parameters; The scattering and absorption trajectory of the virtual photon packet is tracked and simulated to filter out the outgoing photons in the virtual photon packet; The two-dimensional simulated coordinates of the back projection of the emitted photons are statistically analyzed, and the statistical histogram of the two-dimensional simulated coordinates is calculated. A scattering point spread function is generated based on the statistical histogram, and the diffraction degradation image is convolved using the scattering point spread function to obtain a scattering degradation image.

3. The unsupervised learning-based live-body microscopic super-resolution imaging method as described in claim 1, characterized in that, Adding image noise to the scattering degradation image to obtain a simulated fluorescence image includes: Randomly generate noise signals including Gaussian noise and Poisson noise; Convert the noise signal into a noise signal image; The noise signal image and the scattering degradation image are superimposed pixel by pixel to obtain a simulated fluorescence image.

4. The unsupervised learning-based live-body microscopic super-resolution imaging method as described in claim 1, characterized in that, The step of training a pre-constructed super-resolution reconstruction network using the shallow super-resolution structural image and the simulated fluorescence image to obtain the target super-resolution reconstruction network includes: The simulated fluorescence image is subjected to feature transformation to obtain initial image features; The encoder in the super-resolution reconstruction network is used to downsample the initial image features to obtain downsampled image features. The attention enhancement module in the super-resolution reconstruction network is used to perform attention enhancement processing on the downsampled image features to obtain attention-enhanced features; The attention-enhanced features are upsampled using the decoder in the super-resolution reconstruction network to obtain downsampled image features. The downsampled image features are activated to obtain the predicted super-resolution image; The multimodal loss function value between the predicted super-resolution image and the shallow super-resolution structure image is calculated using a preset multimodal loss function; The parameters of the super-resolution reconstruction network are optimized using the multimodal loss function value to obtain the target super-resolution reconstruction network.

5. The unsupervised learning-based live-body super-resolution microscopy imaging method as described in claim 4, characterized in that, The step of upsampling the attention-enhanced features using the decoder in the super-resolution reconstruction network to obtain downsampled image features includes: The attention enhancement features are processed using the decoder to obtain residual channel attention features; The residual channel attention features are deconvolved to obtain deconvolution features; The deconvolutional features and the attention-enhanced features are then residually connected to obtain residual connected features. The residual connection features are spatially upsampled and feature channel mapped using the deconvolution layer in the decoder to obtain downsampled image features.

6. The unsupervised learning-based live-body microscopic super-resolution imaging method as described in claim 4, characterized in that, The multimodal loss function is expressed as: in, This represents the value of the multimodal loss function. This represents the preset weight parameters for the first loss term. Indicates the prediction of super-resolution images With shallow super-resolution structural images The predicted loss value between This represents the preset weight parameters for the second loss term. Indicates the prediction of super-resolution images With shallow super-resolution structural images Image quality loss values ​​between The preset weight parameters for the third loss term, Indicates the prediction of super-resolution images With shallow super-resolution structural images The minimum cross-entropy loss value between them.

7. An unsupervised learning live-body microscopic super-resolution imaging device, characterized in that, The unsupervised learning live-body super-resolution microscopy imaging device includes: A simulated diffraction module is used to acquire shallow super-resolution structure images. Based on the imaging parameters of the shallow super-resolution structure image, the module calculates the relative intensity of each pixel in the image. A discrete diffraction point spread function template is constructed based on the relative pixel intensities. A Gaussian approximation is applied to the discrete diffraction point spread function template to obtain the diffraction point spread function corresponding to the shallow super-resolution structure image. The shallow super-resolution structure image is then convolved using the diffraction point spread function to obtain a diffraction degradation image. The simulated scattering module is used to simulate the propagation trajectory of photons within the tissue corresponding to the diffraction degradation image to construct a scattering point spread function. The scattering point spread function is then used to convolve the diffraction degradation image to obtain a scattering degradation image. A noise addition module is used to add image noise to the scattering degradation image to obtain a simulated fluorescence image; The model training module is used to train a pre-constructed super-resolution reconstruction network using the shallow super-resolution structure image and the simulated fluorescence image to obtain the target super-resolution reconstruction network. The super-resolution reconstruction module is used to perform super-resolution reconstruction of the live image to be reconstructed using the target super-resolution reconstruction network, so as to obtain live microscopic super-resolution imaging.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The processor is used to execute computer programs stored in the memory; The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the unsupervised learning live-body super-resolution microscopy method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the unsupervised learning method for live-body microscopic super-resolution imaging as described in any one of claims 1 to 6.

Images