Structured illumination obvious micro-imaging method and device based on AFM imaging prior

Abstract

The invention provides a structured illumination obvious micro-imaging method and device based on AFM imaging priori, the method is applied to the field of micro-imaging, and the method comprises the steps: obtaining continuous multi-frame original SIM images corresponding to a target sample; based on a first image reconstruction method, performing artifact elimination processing on the multiple frames of original SIM images to obtain multiple frames of reconstructed SIM images corresponding to the multiple frames of original SIM images; and based on a second image reconstruction method and the AFM image of the target sample, carrying out artifact elimination processing on the multiple frames of reconstructed SIM images to obtain multiple frames of target imaging images corresponding to the multiple frames of reconstructed SIM images. According to the method, in the imaging process, multi-modal imaging can be achieved in combination with the SIM and the AFM, high-precision surface contour information is provided through the AFM, the reconstruction algorithm of the SIM image is optimized, artifacts are reduced, and the imaging efficiency of the image is improved.

Description

Technical Field

[0001] The present application relates to the field of microscopy imaging, and more particularly, to a structured illumination microscopy imaging method and device based on AFM imaging prior in the field of microscopy imaging. Background Art

[0002] Currently in the field of microscopy imaging, with the continuous development of technology, compared with traditional optical microscopes, structured illumination microscopes (SIM) have been widely applied in the fields of biological research, materials science, or medical diagnosis due to their advantages of high resolution and high-speed imaging.

[0003] Although SIM has the above advantages in microscopy imaging, in the actual operation process, due to the influence of various factors, the image quality may decline, resulting in artifacts in SIM imaging. Artifacts not only reduce the resolution and contrast of the structured illumination microscope imaging, affecting the accurate analysis of the sample structure, but even local artifacts will spread to the entire imaging range at the wrong angle, seriously affecting the imaging effect of the structured illumination microscope.

[0004] Based on this, how to effectively reduce the artifacts in SIM imaging and improve the SIM imaging quality has become an urgent problem to be solved. Summary of the Invention

[0005] The present application provides a structured illumination microscopy imaging method and device based on AFM imaging prior. This method can combine SIM and AFM to achieve multimodal imaging during the imaging process, provide high-precision surface profile information through AFM, optimize the reconstruction algorithm of SIM images, reduce artifacts and improve the imaging efficiency of the images.

[0006] In the first aspect, a structured illumination microscopy imaging method based on AFM imaging prior is provided. The method includes: acquiring a continuous multi-frame of original SIM images corresponding to a target sample; performing artifact elimination processing on the multi-frame of original SIM images based on a first image reconstruction method to obtain a multi-frame of reconstructed SIM images corresponding to the multi-frame of original SIM images; performing artifact elimination processing on the multi-frame of reconstructed SIM images based on a second image reconstruction method and the AFM image of the target sample to obtain a multi-frame of target imaging images corresponding to the multi-frame of reconstructed SIM images.

[0007] In the above technical solution, in the process of SIM microscopic imaging, the present application proposes a structured illumination microscopic imaging method based on AFM imaging prior. In the imaging process, first, a first artifact elimination process is performed on a single SIM image to reduce the artifacts generated during the reconstruction process, improving the quality and resolution of the image. Further, on the basis of the single SIM reconstructed image, an AFM image is combined for multimodal reconstruction, further improving the accuracy and reliability of imaging, making full use of the advantages of different imaging technologies, providing more comprehensive sample information, and achieving the purpose of improving the image reconstruction efficiency and reconstruction quality.

[0008] Combined with the first aspect, in some possible implementation manners, the first image reconstruction method includes a Wiener deconvolution method and a rolling reconstruction method. Based on the first image reconstruction method, artifact elimination processing is performed on the multiple frames of original SIM images to obtain multiple frames of reconstructed SIM images corresponding to the multiple frames of original SIM images, including: for any one of the multiple frames of original SIM images, based on the Wiener deconvolution method, deconvolution processing is performed on the original SIM image to obtain a restored SIM image of the original SIM image; based on the rolling reconstruction method, rolling reconstruction processing is performed on the restored SIM images of the multiple frames of original SIM images to obtain the multiple frames of reconstructed SIM images.

[0009] In the above technical solution, in the process of reconstructing the SIM image, using the Wiener deconvolution method for image reconstruction can reduce the noise and blur of the image and enhance the resolution of the image. By performing rolling reconstruction on multiple frames of restored SIM images, the information in multiple frames of images can be further integrated, the limitations of single-frame images can be reduced, and a higher-resolution and higher-quality reconstructed image can be obtained.

[0010] Combined with the first aspect and the above implementation manner, in some possible implementation manners, the deconvolution processing is performed on the original SIM image based on the Wiener deconvolution method to obtain a restored SIM image of the original SIM image, including: obtaining the modal wave vector of the microscopic imaging system; determining the point spread function of the microscopic imaging system according to the modal wave vector; obtaining the noise power spectrum of the original SIM image and the power spectrum of the original SIM image; determining the restored image of the original SIM image according to the point spread function, the original SIM image, the noise power spectrum, and the power spectrum.

[0011] Combined with the first aspect and the above implementation manners, in some possible implementation manners, based on the rolling reconstruction method, performing rolling reconstruction processing on the restored images of the multi-frame original SIM images to obtain the multi-frame reconstructed SIM images, including: for the restored image of any one of the multi-frame original SIM images in the restored images of the multi-frame original SIM images, traversing the restored image of the original SIM image based on a preset rolling window to determine a target local restored image corresponding to the preset rolling window at the current moment; extracting features of the target local restored image to obtain local image features of the target local restored image; based on the local image features, performing filtering processing on the target local restored image to obtain a filtered target local restored image; performing reconstruction on the filtered target local restored image to obtain a reconstructed target local restored image; when the traversal of the restored image of the original SIM image is completed, splicing the multiple reconstructed local restored images corresponding to the restored image of the original SIM image to obtain a reconstructed SIM image corresponding to the original SIM image.

[0012] Combined with the first aspect and the above implementation manners, in some possible implementation manners, the second image reconstruction method includes a Hessian reconstruction method. Based on the second image reconstruction method and the AFM image of the target sample, performing artifact removal processing on the multi-frame reconstructed SIM images to obtain multi-frame target imaging images corresponding to the multi-frame reconstructed SIM images, including: for any one of the multi-frame reconstructed SIM images in the multi-frame reconstructed SIM images, determining the Hessian matrix of the reconstructed SIM image according to the AFM image and the reconstructed SIM image; based on the Hessian matrix of the reconstructed SIM image, performing artifact removal processing on the reconstructed SIM image to obtain a target imaging image corresponding to the reconstructed SIM image.

[0013] In the above technical solution, during the joint imaging process based on the SIM image and the AFM image, by introducing the Hessian matrix, artifacts in the multi-frame reconstructed SIM images can be effectively identified and removed. Artifacts are usually caused by noise, distortion during the imaging process, or imperfections in the reconstruction algorithm. The Hessian matrix can capture local structural information in the image, thereby helping to distinguish real signals from artifacts. By combining the AFM image and the reconstructed SIM image and using the Hessian matrix for processing, the quality of the final target imaging image can be significantly improved. The AFM image provides high-resolution surface topography information. Combining with the super-resolution characteristics of the SIM image, a clearer and more accurate imaging result can be generated, ensuring that the reconstructed target imaging image has higher resolution and richer details, thereby ensuring the consistency and accuracy of the entire image sequence.

[0014] Combined with the first aspect and the above implementation manners, in some possible implementation manners, determining the Hessian matrix of the reconstructed SIM image according to the AFM image and the reconstructed SIM image includes: obtaining the acquisition time of the reconstructed SIM image; for any target pixel point in the reconstructed SIM image, determining a plurality of neighboring pixel points within a local neighborhood window centered on the target pixel point; obtaining the pixel coordinates of the target pixel point and the pixel coordinates of the plurality of neighboring pixel points; determining a target sample point corresponding to the target pixel point from the AFM image, and obtaining the height information of the target sample point; determining a plurality of neighboring sample points corresponding to the plurality of neighboring pixel points from the AFM image, and obtaining the height information of the plurality of neighboring sample points; and determining the Hessian matrix of the target pixel point according to the acquisition time, the pixel coordinates of the target pixel point, the height information of the target sample point, the pixel coordinates of the plurality of neighboring pixel points, and the height information of the plurality of neighboring sample points.

[0015] Combined with the first aspect and the above implementation manners, in some possible implementation manners, performing artifact removal processing on the reconstructed SIM image according to the Hessian matrix of the reconstructed SIM image to obtain a target imaging image corresponding to the reconstructed SIM image includes: determining the eigenvalues and eigenvectors of the Hessian matrix of the target pixel point; determining whether the target pixel point is an artifact point according to the eigenvalues and the eigenvectors; and in the case where the target pixel point is an artifact point, performing artifact removal processing on the reconstructed SIM image to obtain the target imaging image.

[0016] Combined with the first aspect and the above implementation manners, in some possible implementation manners, in the case where the target pixel point is an artifact point, performing artifact removal processing on the reconstructed SIM image to obtain the target imaging image includes: in the case where the target pixel point is an artifact point, obtaining the pixel values of the plurality of neighboring pixel points; determining the average pixel value of the pixel values of the plurality of neighboring pixel points; replacing the pixel value of the target pixel point with the average pixel value to obtain a reconstructed SIM image with corrected pixels; and optimizing the reconstructed SIM image with corrected pixels to obtain the target imaging image.

[0017] In the above technical solution, by replacing the pixel value of the target artifact point with the average value of the neighboring pixels, the noise and artifacts in the image are effectively reduced, the clarity and accuracy of the image are improved, local mutations are reduced, and the image becomes more natural and coherent.

[0018] Combined with the first aspect and the above implementation manners, in some possible implementation manners, optimizing the reconstructed SIM image after pixel correction to obtain the target imaging image includes: determining an image reconstruction loss according to the reconstructed SIM image after pixel correction; determining an image constraint loss based on the height information of the reconstructed SIM image after pixel correction and the height information of the AFM image; determining an objective loss function based on the image reconstruction loss, the image constraint loss, a regularization term, a first weight corresponding to the image reconstruction loss, a second weight corresponding to the image constraint loss, and a third weight corresponding to the regularization term; performing multiple iterations on the objective loss function until the objective loss function meets a preset iteration condition, and outputting the target imaging image corresponding to the objective loss function.

[0019] In the above technical solution, by using the height information of the reconstructed SIM image after pixel correction and the AFM image, combining the image reconstruction loss and the image constraint loss, the target image can be reconstructed more accurately and the error can be reduced. By utilizing the different characteristics of the SIM image and the AFM image, through weight assignment and the regularization term, the effective fusion of multi-source information is achieved, and the robustness and accuracy of image reconstruction are improved. By optimizing the objective loss function through multiple iterations, it is ensured that the finally output target imaging image meets the preset conditions, further improving the image quality.

[0020] In a second aspect, a structured illumination microscopy imaging device based on AFM imaging prior is provided. The device includes: an image acquisition module configured to acquire a continuous multi-frame original SIM image corresponding to a target sample; a first reconstruction module configured to perform artifact removal processing on the multi-frame original SIM images based on a first image reconstruction method to obtain multi-frame reconstructed SIM images corresponding to the multi-frame original SIM images; a second reconstruction module configured to perform artifact removal processing on the multi-frame reconstructed SIM images based on a second image reconstruction method and the AFM image of the target sample to obtain multi-frame target imaging images corresponding to the multi-frame reconstructed SIM images.

[0021] Combined with the second aspect, in some possible implementation manners, the first image reconstruction method includes a Wiener deconvolution method and a rolling reconstruction method. The first reconstruction module is specifically configured to: for any one of the multi-frame original SIM images, perform deconvolution processing on the original SIM image based on the Wiener deconvolution method to obtain a restored SIM image of the original SIM image; perform rolling reconstruction processing on the restored SIM images of the multi-frame original SIM images based on the rolling reconstruction method to obtain the multi-frame reconstructed SIM images.

[0022] Combined with the second aspect and the above implementation manners, in some possible implementation manners, the first reconstruction module is further configured to: obtain the mode wave vector of the microscopic imaging system; determine the point spread function of the microscopic imaging system according to the mode wave vector; obtain the noise power spectrum of the original SIM image and the power spectrum of the original SIM image; and determine the restored image of the original SIM image according to the point spread function, the original SIM image, the noise power spectrum, and the power spectrum.

[0023] Combined with the second aspect and the above implementation manners, in some possible implementation manners, the first reconstruction module is further configured to: for any restored image of the multi-frame original SIM images, traverse the restored image of the original SIM image based on a preset rolling window, and determine a target local restored image corresponding to the preset rolling window at the current moment; extract features from the target local restored image to obtain local image features of the target local restored image; perform filtering processing on the target local restored image based on the local image features to obtain a filtered target local restored image; perform reconstruction on the filtered target local restored image to obtain a reconstructed target local restored image; and when the traversal of the restored image of the original SIM image is completed, splice the multiple reconstructed local restored images corresponding to the restored image of the original SIM image to obtain a reconstructed SIM image corresponding to the original SIM image.

[0024] Combined with the second aspect and the above implementation manners, in some possible implementation manners, the second image reconstruction method includes a Hessian reconstruction method, and the second reconstruction module is specifically configured to: for any reconstructed SIM image in the multi-frame reconstructed SIM images, determine the Hessian matrix of the reconstructed SIM image according to the AFM image and the reconstructed SIM image; and perform artifact removal processing on the reconstructed SIM image according to the Hessian matrix of the reconstructed SIM image to obtain a target imaging image corresponding to the reconstructed SIM image.

[0025] Combined with the second aspect and the above implementation manners, in some possible implementation manners, the second reconstruction module is further configured to: obtain the acquisition time of the reconstructed SIM image; for any target pixel point in the reconstructed SIM image, determine a plurality of neighboring pixel points within a local neighborhood window centered on the target pixel point; obtain the pixel coordinates of the target pixel point and the pixel coordinates of the plurality of neighboring pixel points; determine a target sample point corresponding to the target pixel point from the AFM image, and obtain the height information of the target sample point; determine a plurality of neighboring sample points corresponding to the plurality of neighboring pixel points from the AFM image, and obtain the height information of the plurality of neighboring sample points; and determine the Hessian matrix of the target pixel point according to the acquisition time, the pixel coordinates of the target pixel point, the height information of the target sample point, the pixel coordinates of the plurality of neighboring pixel points, and the height information of the plurality of neighboring sample points.

[0026] Combined with the second aspect and the above implementation manners, in some possible implementation manners, the second reconstruction module is further configured to: determine the eigenvalues and eigenvectors of the Hessian matrix of the target pixel point; determine whether the target pixel point is an artifact point according to the eigenvalues and the eigenvectors; and in the case that the target pixel point is an artifact point, perform artifact removal processing on the reconstructed SIM image to obtain the target imaging image.

[0027] Combined with the second aspect and the above implementation manners, in some possible implementation manners, the second reconstruction module is further configured to: in the case that the target pixel point is an artifact point, obtain the pixel values of the plurality of neighboring pixel points; determine the average pixel value of the pixel values of the plurality of neighboring pixel points; replace the pixel value of the target pixel point with the average pixel value to obtain a reconstructed SIM image with pixel correction; and optimize the reconstructed SIM image with pixel correction to obtain the target imaging image.

[0028] Combined with the second aspect and the above implementation manners, in some possible implementation manners, the second reconstruction module is further configured to: determine an image reconstruction loss according to the reconstructed SIM image with pixel correction; determine an image constraint loss based on the height information of the reconstructed SIM image with pixel correction and the height information of the AFM image; determine a target loss function based on the image reconstruction loss, the image constraint loss, a regularization term, a first weight corresponding to the image reconstruction loss, a second weight corresponding to the image constraint loss, and a third weight corresponding to the regularization term; and perform multiple iterations on the target loss function until the target loss function meets a preset iteration condition, and output the target imaging image corresponding to the target loss function.

[0029] In a third aspect, an electronic device is provided, including a memory and a processor. The memory is used to store executable program code, and the processor is used to call and run the executable program code from the memory, so that the electronic device executes the method in the first aspect or any possible implementation manner of the first aspect above.

[0030] In a fourth aspect, a computer program product is provided, including: computer program code, when the computer program code runs on a computer, enabling the computer to execute the method in the first aspect or any possible implementation manner of the first aspect above.

[0031] In a fifth aspect, a computer-readable storage medium is provided, storing computer program code, when the computer program code runs on a computer, enabling the computer to execute the method in the first aspect or any possible implementation manner of the first aspect above. Description of the Drawings

[0032] Figure 1 is a schematic structural diagram of a structured illumination microscopy imaging system based on AFM imaging prior provided by an embodiment of the present application;

[0033] Figure 2 is a schematic scene diagram of an implementation process of structured illumination microscopy imaging based on AFM imaging prior provided by an embodiment of the present application;

[0034] Figure 3 is a schematic flowchart of a method for structured illumination microscopy imaging based on AFM imaging prior provided by an embodiment of the present application;

[0035] Figure 4 is a schematic structural diagram of a structured illumination microscopy imaging device based on AFM imaging prior provided by an embodiment of the present application;

[0036] Figure 5 is a schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed Embodiments

[0037] Next, the technical solutions in the present application will be clearly and elaborately described in conjunction with the drawings. Among them, in the description of the embodiments of the present application, unless otherwise specified, " / " means "or". For example, A / B may mean A or B. The "and / or" in the text is only a description of the association relationship between associated objects, indicating that there can be three relationships. For example, A and / or B may mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of the present application, "a plurality" means two or more than two.

[0038] Hereinafter, the terms "first" and "second" are for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of such features.

[0039] Before introducing the solutions of the embodiments of the present application, first, a glossary of professional terms that may be involved in the embodiments of the present application will be provided.

[0040] Structured Illumination Microscopy (SIM): SIM microscopes utilize super-resolution microscopy techniques to break through the diffraction limit of traditional optical microscopes by introducing an interference grating in the illumination path. Its basic principle is to utilize the Moiré effect. By changing the angle or moving direction of the grating, multiple images with different phases are captured, and then a super-resolution image is reconstructed after imaging. SIM microscopes can achieve a resolution of up to 20 nm, significantly superior to the diffraction limit of traditional optical microscopes (i.e., the Abbe limit, approximately 200 nm laterally and approximately 500 nm axially).

[0041] Image reconstruction: The process of converting the acquired raw data (such as projection data) into a two-dimensional or three-dimensional image through mathematical algorithms.

[0042] Atomic Force Microscope (AFM): AFM microscopes utilize scanning probe microscopy techniques to detect the surface topography and mechanical properties of a sample through the interaction force between the tip and the sample surface. Its working principle is to use a microcantilever fixed above the sample. The minute force changes generated when the tip contacts the sample surface are detected and recorded by a sensor, thereby generating a surface topography map.

[0043] Currently, in some research fields, based on the respective advantages of SIM microscopes and AFM microscopes, researchers have begun to combine SIM microscopes and AFM microscopes to achieve multimodal imaging in order to obtain more comprehensive sample information. For example, in the field of life science research, during the analysis of cell dynamics and structure, the combination of SIM microscopes and AFM microscopes can simultaneously capture high-resolution images and mechanical properties of cells.

[0044] Wiener deconvolution algorithm: A non-blind linear image restoration method widely used in fields such as image deblurring, noise smoothing, and signal processing. Its core idea is to restore the original signal or image by optimizing the objective function while balancing deconvolution and noise suppression. In the field of optical imaging, the Wiener deconvolution algorithm is often used to improve spatial resolution and reduce the impact of noise.

[0045] Rolling reconstruction method: A technique for super-resolution imaging of dynamic scenes, aiming to generate high-resolution images by continuously acquiring multiple low-resolution images in a short time and combining image processing algorithms. The rolling reconstruction method is particularly suitable for scenarios that require high frame rate and high temporal resolution, such as the rapid dynamic change process of living cells.

[0046] Artifact: Refers to abnormal images that appear in the image during the imaging process due to various reasons and are unrelated to the actual object being scanned. These images may appear as spots, stripes, shadows, rings or other irregular forms, and they do not reflect the true structure or pathological changes of the object being examined. The presence of artifacts will reduce the quality of the image and affect the imaging effect of the image.

[0047] Wave Vector: A mathematical vector that describes the spatial and directional properties of a wave.

[0048] Point Spread Function (PSF): Used to describe the response characteristics of an imaging system to a point light source.

[0049] Hessian algorithm: An optimization method based on the Hessian matrix, usually used to solve unconstrained optimization problems. Its core idea is to achieve optimization by constructing a function closer to the objective function.

[0050] After introducing the professional terms, the application scenarios of the embodiments of this application are introduced below.

[0051] Currently, nanotechnology has been widely applied in various fields, such as the chemical field, the physics field, the materials science field, the biology field, etc. The development of nanotechnology has driven many emerging industries, such as nanomaterials, nanodevices, and nanomedicine, etc.

[0052] In a possible implementation, in the fields of materials science and biology, nanotechnology is used to detect samples to obtain clear imaging images. As a nanoscale detection microscope, the SIM microscope is widely used to detect samples due to its advantages such as super-resolution imaging, fast imaging speed, and the ability to adapt to a variety of fluorescent dyes.

[0053] Although the SIM microscope has many advantages in the imaging process, there are also obvious defects in the imaging process of the SIM microscope, that is, artifacts are likely to appear during the image reconstruction process. The reasons mainly include but are not limited to the following points:

[0054] (1) Inaccurate parameter estimation. The SIM microscope relies on the estimation of accurate illumination parameters (for example, modulation wavelength, modulation amplitude, and phase). If these parameter estimations are inaccurate, artifacts will appear during the image reconstruction process.

[0055] (2) Noise amplification during Fourier transform and deconvolution. SIM microscope imaging requires converting the acquired original image into the frequency domain for processing and then back into the spatial domain. This process involves Fourier transform and inverse Fourier transform, which are prone to introducing artifacts.

[0056] (3) Low signal-to-noise ratio (SNR) and insufficient image preprocessing. SIM imaging usually requires shortening the exposure time to reduce photobleaching and phototoxic effects, which results in a decrease in the SNR of the original image. The low SNR exacerbates the impact of noise, thus introducing artifacts during the reconstruction process.

[0057] The artifacts generated by SIM microscope imaging may reduce the clarity of the image, affect the imaging speed and resolution, and ultimately lead to a decrease in imaging efficiency and a deterioration of the imaging effect.

[0058] Based on the above problems, the embodiments of the present application provide a structured illumination microscopy imaging method based on AFM imaging prior. This method can combine SIM and AFM to achieve multimodal imaging during the imaging process, provide high-precision surface profile information through AFM, optimize the reconstruction algorithm of SIM images, reduce artifacts, and improve the imaging efficiency of the images.

[0059] Before introducing the method of the embodiments of the present application, the hardware devices and working principles involved in the embodiments of the present application will be introduced first.

[0060] Figure 1 is a schematic structural diagram of a structured illumination microscopy imaging system based on AFM imaging prior provided by the embodiments of the present application.

[0061] Exemplarily, as Figure 1 shown, the structured illumination microscopy imaging system based on AFM imaging prior provided by the embodiments of the present application includes a SIM microscope 101, an electronic device 102, and an AFM microscope 103.

[0062] Optionally, the type of the electronic device 102 includes a desktop computer, a portable computer, an embedded computer, etc. The embodiments of the present application do not limit the type of the electronic device 102.

[0063] Among them, the SIM microscope 101 is used to collect multiple frames of original SIM images of the target sample at different illumination angles or different phases, and send the multiple frames of SIM images to the electronic device 102.

[0064] The electronic device 102 is used to receive the multiple frames of original SIM images, reconstruct the multiple frames of original SIM images to remove the artifacts in the original SIM images, and obtain multiple frames of reconstructed SIM images after reconstruction.

[0065] The AFM microscope 102 is used to collect topographic information of a target sample, such as height information of the target sample, and send the topographic information of the target sample to the electronic device 102 .

[0066] After the electronic device 102 reconstructs multiple frames of original SIM images collected based on a single SIM microscope to obtain multiple frames of reconstructed SIM images, it can further combine the AFM image of the target sample and reconstruct the multiple frames of reconstructed SIM images to eliminate artifacts in the SIM image and obtain the final multi-frame target imaging image.

[0067] Below through Figure 2 The overall implementation process of the embodiment of the present application is introduced.

[0068] Figure 2 This is a scene diagram of the process of implementing structured illumination microscopy imaging based on AFM imaging prior provided in an embodiment of the present application.

[0069] For example, Figure 2 As shown, combined Figure 1 During the implementation of the method of the embodiment of the present application, the overall process is as follows:

[0070] First, the electronic device 102 collects the target sample through the SIM microscope 101 to obtain a series of continuous multi-frame original SIM images. The multi-frame original SIM images serve as the input images of the embodiment of the present application. In the embodiment of the present application, the processing process of the input image is divided into two paths: Figure 2 As shown in the figure, one path is used to initialize multiple frames of raw SIM images to obtain the wave vectors of the target sample under different illumination modes. The other path is used for Wiener deconvolution. During Wiener deconvolution, the wave vectors obtained above can be combined to calculate the PSF of the microscopic imaging system. Then, Wiener deconvolution is performed on the multiple frames of raw SIM images based on the point spread function to obtain the multiple frames of restored SIM images corresponding to the multiple frames of raw SIM images.

[0071] Furthermore, in order to further enhance the signal in the image, suppress random noise in the image, and improve the signal-to-noise ratio and detail clarity of the image, since the multi-frame restored SIM image is a continuous multi-frame image, the multi-frame restored SIM image can be superimposed through the rolling reconstruction method to enhance the signal in the image, thereby obtaining a multi-frame image that is continuous in time series after rolling reconstruction.

[0072] The above-mentioned reconstruction methods using Wiener deconvolution and rolling reconstruction are both performed on a single SIM image.

[0073] After obtaining multiple consecutive frames of images after rolling reconstruction, in the embodiments of the present application, the AFM image of the target sample can be combined, and then the multiple consecutive frames of images can be reconstructed for the second time, that is Figure 2 the Hessian reconstruction shown in FIG. 3, so as to further optimize the image quality and obtain the final output image.

[0074] The above process is the general flow of the method in the embodiments of the present application. Next, Figure 3 a detailed introduction to the method in the embodiments of the present application will be given.

[0075] Figure 3 FIG. 4 is a schematic flowchart of a structured illumination microscopy method based on AFM imaging prior provided by an embodiment of the present application. It should be understood that the execution subject of this method can be Figure 1 the electronic device 102 in FIG. 1.

[0076] Exemplarily, as Figure 3 shown in FIG. 5, the method 300 includes the following steps 301 to 303.

[0077] 301. Obtain multiple consecutive frames of original SIM images corresponding to the target sample.

[0078] It should be understood that in the implementation process of the method in the embodiments of the present application, first, the image collected by the SIM microscope can be used as the input image, that is, the original image to be processed.

[0079] Specifically, when obtaining multiple consecutive frames of original SIM images corresponding to the target sample, the SIM microscope captures the high-frequency spatial information of the sample by using illumination patterns with different directions and different phases, and generates SIM images. Therefore, in the embodiments of the present application, the SIM microscope can be controlled to quickly switch different illumination patterns, so as to collect multiple consecutive frames of SIM images of the target sample under different illumination patterns.

[0080] Optionally, the illumination pattern includes illumination direction and phase. One illumination direction corresponds to multiple phases, such as 0, 2π / 5, 4π / 5, etc. One illumination direction and a specific phase in this illumination direction are called an illumination pattern.

[0081] 302. Based on the first image reconstruction method, perform artifact removal processing on the multiple frames of original SIM images to obtain multiple frames of reconstructed SIM images corresponding to the multiple frames of original SIM images.

[0082] After the electronic device obtains multiple consecutive frames of original SIM images, it can reconstruct them to remove the blur and noise in the original images, so as to achieve the purpose of removing artifacts in the images.

[0083] Exemplarily, as Figure 2As shown, the electronic device performs artifact removal processing on the original SIM image based on the first image reconstruction method, corresponding to Figure 2 the first reconstruction process in

[0084] Optionally, the first image reconstruction method includes the Wiener deconvolution method and the rolling reconstruction method.

[0085] In a possible implementation, based on the first image reconstruction method, artifact removal processing is performed on multiple frames of original SIM images to obtain multiple frames of reconstructed SIM images corresponding to the multiple frames of original SIM images, including:

[0086] For any one of the multiple frames of original SIM images, based on the Wiener deconvolution method, deconvolution processing is performed on the original SIM image to obtain a restored SIM image of the original SIM image;

[0087] Based on the rolling reconstruction method, rolling reconstruction processing is performed on the restored SIM images of the multiple frames of original SIM images to obtain multiple frames of reconstructed SIM images.

[0088] Exemplarily, as Figure 2 shown, the first image reconstruction method is for reconstructing the image collected by a single SIM microscope, and the process does not involve fusion with the AFM image. In the first reconstruction process, for the input multiple frames of consecutive original SIM images, the electronic device first restores each frame of the original SIM image through the Wiener deconvolution method to obtain multiple frames of restored SIM images.

[0089] It should be understood that if the distance that the target sample moves between two exposures exceeds the resolution of the SIM, motion artifacts may occur. In addition, when the signal-to-noise ratio of the original SIM image is low, the Wiener deconvolution algorithm may amplify the random noise in the image, resulting in artifacts in the reconstructed image. Therefore, in order to achieve high-frame-rate super-resolution imaging, especially in the fast dynamic process of living cells, through hardware configurations such as high-frame-rate cameras and high-numerical-aperture objectives, and data acquisition devices such as multi-channel imaging, a synchronization mechanism is set to coordinate the generation of the high-speed spatial light modulator mode and the camera readout interval, capture the fast dynamic process inside the cell, ensure the acquisition of continuous image data with high frame rate and short exposure time, and then perform real-time processing on the image data at each time point in a rolling reconstruction manner, extract dynamic change information, and capture the fast changes in the intracellular structure at a time resolution of milliseconds, so as to effectively monitor the biological process.

[0090] Therefore, since the time intervals between multiple frames of the acquired original SIM images are short, after obtaining the restored image through Wiener deconvolution, the electronic device can perform rolling reconstruction processing to enhance the signal of the image and suppress random noise, thereby obtaining multiple frames of reconstructed SIM images.

[0091] First, the process of specifically restoring the image by the Wiener deconvolution method will be introduced below.

[0092] In a possible implementation manner, based on the Wiener deconvolution method, deconvolution processing is performed on the original SIM image to obtain the restored SIM image of the original SIM image, including:

[0093] Obtain the modal wave vector of the microscopic imaging system;

[0094] Determine the point spread function of the microscopic imaging system according to the modal wave vector;

[0095] Obtain the noise power spectrum of the original SIM image and the power spectrum of the original SIM image;

[0096] Determine the restored image of the original SIM image according to the point spread function, the original SIM image, the noise power spectrum, and the power spectrum.

[0097] Specifically, the specific implementation process of the Wiener deconvolution method can be represented by the following formula (1).

[0098]

[0099] Among them, in the above formula (1):

[0100] The restored image of the original SIM image;

[0101] F -1 : Fourier transform;

[0102] F(u,v): The Fourier transform of the original SIM image;

[0103] S n (u,v): The power spectrum of the noise;

[0104] S f (u,v): The noise power spectrum of the original SIM image;

[0105] H(u,v): The Fourier transform of the point spread function.

[0106] As can be seen from the above formula (1), when restoring an image based on the Wiener deconvolution algorithm, it is first necessary to obtain the point spread function of the microscopic imaging system. In the embodiments of the present application, the point spread function can be determined by the modal wave vector of the microscopic imaging system. Among them, the modal wave vector describes the propagation direction and wavelength of light waves in space.

[0107] The following details the steps for obtaining the modal wave vector and the point spread function.

[0108] The process of obtaining the modal wave vector.

[0109] It should be understood that during the SIM microscopic imaging process, a single-frame image is often affected by random noise, resulting in a decrease in image quality. Therefore, in the embodiments of the present application during SIM microscopic imaging, the SIM microscope can be controlled to continuously acquire multiple frames of SIM images in the same illumination mode under multiple illumination modes. Thus, through time-series acquisition, the SIM microscope can obtain multiple frames of original SIM images, that is, the input images of the embodiments of the present application.

[0110] Exemplarily, assume that in the embodiments of the present application, the SIM microscope uses 9 illumination modes (for example, 3 illumination directions, and each illumination direction corresponds to three phases) during the acquisition of SIM images, and N frames of images are obtained for each illumination mode. Then, after the acquisition is completed, 9*N frames of original SIM images can be obtained.

[0111] Furthermore, through a series of initialization processes on multiple frames of original SIM images, the electronic device can obtain the wave vectors of multiple illumination modes. The specific initialization process can be divided into the following steps:

[0112] (1) Normalization

[0113] After obtaining multiple frames of original SIM images, in order to eliminate the amplitude differences between different SIM images, first normalize the amplitudes of multiple frames of original SIM images. The specific normalization operation is as follows: for each frame of original SIM image, first determine the maximum pixel value in the original SIM image. Then calculate the ratio between the pixel value of each pixel point in the original SIM image and the above-mentioned maximum pixel value, so as to obtain the original SIM image after normalization for each frame of original SIM image. The pixel value range of the original SIM image after normalization is within [0, 1].

[0114] (2) Phase information extraction

[0115] After obtaining multiple frames of original SIM images after normalization, further extract the phase information from multiple frames of original SIM images after normalization.

[0116] The phase information extraction steps can be as follows: For a frame of original SIM image after normalization, the normalized frame of original SIM image can be converted into a complex image. The real part of the complex image is the normalized frame of original SIM image, and the imaginary part is 0.

[0117] Specifically, assume that the normalized frame of original SIM image is I(x, y), where x and y represent the horizontal and vertical coordinates of the frame of original SIM image respectively. The complex image can be expressed as: C(x, y) = I(x, y) + 0i. Here, i represents the imaginary unit.

[0118] Perform a two-dimensional Fourier transform on the above complex image to obtain a complex frequency spectrum F(u, v). Here, u and v represent the horizontal and vertical coordinates of the complex image in the frequency domain form respectively.

[0119] Furthermore, the electronic device can calculate the phase image of the complex frequency spectrum F(u, v) Specifically expressed as: where, ∠ represents calculating the phase of the complex number.

[0120] Thus, by performing the above processing on each frame of original SIM image after normalization, the phase information of each frame of original SIM image after normalization can be obtained.

[0121] (3) Multi-frame averaging

[0122] In the embodiments of the present application, a set of original SIM images can be obtained under each illumination mode. Therefore, a set of original SIM images can be averaged to obtain an average image.

[0123] Specifically, for the average image of each set of images, it can be obtained by adding all the images in each set and then dividing by the number of images.

[0124] (4) Wave vector estimation

[0125] Specifically, in the embodiments of the present application, the wave vector is calculated by normalized cross-correlation.

[0126] First, perform a two-dimensional Fourier transform on each set of average images to obtain the complex frequency spectrum of each set of images. Calculate the complex conjugate product of the Fourier transforms of two average images, and perform an inverse Fourier transform on the complex conjugate product to obtain a normalized cross-correlation image.

[0127] In the normalized cross-correlation image, find the peak position, calculate the difference between the peak position and the center of the image, determine the relative displacement between the two average images, and finally calculate the modal wave vector according to the above difference. In the embodiments of the present application, the modal wave vector is represented by "k".

[0128] The process of obtaining the point spread function.

[0129] After calculating the modal wave vector, in the embodiments of the present application, the following two methods can be used to calculate the point spread function.

[0130] The first one is based on the Debye-Wolf theory.

[0131] When calculating the PSF based on the Debye-Wolf theory, the specific process can be expressed by the following formula (2).

[0132] PSF(X) = ∫∫F(k)e ik·x dk Formula (2)

[0133] Among them, in formula (2):

[0134] F(k): The spectral function of the light source, and the modal wave vector k is used to describe the spectral distribution of the light source;

[0135] k: The modal wave vector.

[0136] The second one is based on the Richards-Wolf theory.

[0137] When calculating the PSF based on the Richards-Wolf theory, the specific process can be expressed by the following formula (3).

[0138] PSF(X) = ∫∫E(ρ, φ)e ik·x dρdφ Formula (3)

[0139] Among them, in formula (3):

[0140] E(ρ, φ): The pupil function, and the modal wave vector k is used to describe the distribution of the pupil function;

[0141] k: The modal wave vector.

[0142] Thus, through the above steps, the electronic device can calculate the PSF of the microscopic imaging system.

[0143] As shown in formula (1), during the restoration process of the original SIM image, in addition to obtaining the PSF of the microscopic imaging system, it is also necessary to obtain the noise power spectrum and power spectrum of the current original SIM image.

[0144] Noise Power Spectrum (NPS): Used to describe the distribution of noise in the frequency domain.

[0145] Power spectrum, specifically referring to the Image Power Spectrum (IPS) of an image: used to describe the distribution of the image signal in the frequency domain.

[0146] The process of obtaining the noise power spectrum is as follows: Select a noise sample region from the original SIM image. This region should contain as little image signal as possible, or no image signal at all. Perform a two-dimensional Fourier transform on the noise sample region to obtain the frequency domain representation of the noise signal, and then further calculate the square of the magnitude of the frequency domain representation of the noise to obtain the noise power spectrum.

[0147] The process of obtaining the image power spectrum is as follows: Similar to the method of calculating the noise power spectrum above, when calculating the image power spectrum, an image sample region containing the image signal can be selected from the original SIM image. Perform a two-dimensional Fourier transform on this image sample region to obtain the frequency domain representation of the image signal, and calculate the square of the magnitude of the frequency domain representation of the image signal to obtain the image power spectrum.

[0148] After obtaining the noise power spectrum and the image power spectrum, as shown in formula (1), the electronic device can perform a two-dimensional Fourier transform on the original SIM image respectively to obtain the frequency domain representation of the original SIM image, and perform a two-dimensional Fourier transform on the point spread function to obtain the frequency domain representation of the point spread function. Further, combine the image power spectrum and the noise power spectrum to obtain the frequency domain representation result of the restored image, and finally, through the inverse Fourier transform, obtain the restored image of the original SIM image.

[0149] By performing the above Wiener deconvolution processing on each frame of the original SIM image, the restored SIM image corresponding to each frame of the original SIM image can be obtained.

[0150] Furthermore, on this basis, since multiple frames of the original SIM images are continuously acquired in a time series, in the embodiments of the present application, a rolling reconstruction method can also be used to perform rolling reconstruction on the restored SIM images of multiple frames of the original SIM images to enhance the image signal and thus suppress random noise.

[0151] In a possible implementation manner, based on the rolling reconstruction method, perform rolling reconstruction processing on the restored images of multiple frames of the original SIM images to obtain multiple frames of reconstructed SIM images, including:

[0152] For the restored image of any frame of the original SIM images among the restored images of multiple frames of the original SIM images, traverse the restored image of the original SIM image based on a preset rolling window to determine the target local restored image corresponding to the preset rolling window at the current moment;

[0153] Extract features from the target local restored image to obtain the local image features of the target local restored image;

[0154] Based on local image features, perform filtering on the target local restored image to obtain the filtered target local restored image;

[0155] Perform reconstruction on the filtered target local restored image to obtain the reconstructed target local restored image;

[0156] When the traversal of the restored image of the original SIM image is completed, splice the multiple reconstructed local restored images corresponding to the restored image of the original SIM image to obtain the reconstructed SIM image corresponding to the original SIM image.

[0157] It should be understood that when reconstructing the restored image of each frame of the original SIM image based on the rolling reconstruction method, the following steps are included: block processing, local reconstruction, and splicing processing.

[0158] Specifically, for the restored image of any one frame of the original SIM image among the restored images of multiple frames of the original SIM image, during the rolling reconstruction process, those skilled in the art of the embodiments of the present application can, according to requirements, preset a preset rolling window size and store it in the electronic device.

[0159] Exemplarily, the preset rolling window size can be 32×32 pixels, or 64×64 pixels, etc., and the embodiments of the present application do not limit the preset rolling window size.

[0160] Further, the electronic device starts from the upper left corner of the restored image of the original SIM image, slides the preset rolling window at a certain step length, and at the preset rolling window position at the current moment, the target local restored image covered by the preset rolling window at the current moment can be extracted.

[0161] For the target local restored image, the electronic device can perform feature extraction on the target local restored image to obtain the local image features of the target local restored image.

[0162] Optionally, the method of feature extraction includes calculating local gradients, local texture feature extraction, etc.

[0163] After obtaining the local image features of the target restored image, according to the local image features, perform filtering on the target local restored image to obtain the filtered target local restored image, thereby achieving the effect of enhancing the details and clarity of the image.

[0164] Optionally, the filtering method includes Gaussian filtering, median filtering, etc.

[0165] Based on the filtered target local restored image, interpolation or fitting and other methods can be further used to perform reconstruction on the target local restored image to obtain the reconstructed target local restored image.

[0166] During the continuous sliding of the preset scrolling window, for the local restored image corresponding to the preset scrolling window at each moment, the above operations can be repeated. When the current restored image is traversed, it indicates that the reconstruction processes of multiple local images in the restored image are completed. Therefore, the electronic device can splice multiple reconstructed local restored images corresponding to the restored image of the original SIM image to obtain the reconstructed SIM image corresponding to the original SIM image.

[0167] It should be understood that during the sliding of the preset scrolling window, there may be an overlapping area between the local restored images of two adjacent preset scrolling windows. To ensure the continuity and consistency of the image, in the embodiments of the present application, for two reconstructed local restored images with an overlapping area, weighted average processing can be performed to obtain an average image and then splice it.

[0168] Thus, through the above process, the electronic device can obtain multiple reconstructed SIM images corresponding to multiple frames of the original SIM image, that is, complete Figure 1 the first reconstruction process shown.

[0169] In the above technical solution, during the reconstruction process of the SIM image, the Wiener deconvolution method is used for image reconstruction, which can reduce the noise and blur of the image and enhance the resolution of the image. By performing rolling reconstruction on multiple frames of restored SIM images, the information in multiple frames of images can be further integrated, the limitations of single-frame images can be reduced, and higher-resolution and higher-quality reconstructed images can be obtained.

[0170] 303. Based on the second image reconstruction method and the AFM image of the target sample, perform artifact removal processing on multiple frames of reconstructed SIM images to obtain multiple frames of target imaging images corresponding to multiple frames of the original SIM image.

[0171] In the above first reconstruction process, the combination of the SIM image and the AFM image is not involved. Therefore, in the second reconstruction process, the embodiments of the present application can combine the reconstructed SIM image and the AFM image to obtain more information of the target sample to improve the efficiency of image reconstruction.

[0172] The second image reconstruction method in the embodiments of the present application is specifically the Hessian reconstruction method.

[0173] In a possible implementation manner, based on the second image reconstruction method and the AFM image of the target sample, performing artifact removal processing on multiple frames of reconstructed SIM images to obtain multiple frames of target imaging images corresponding to multiple frames of reconstructed SIM images includes:

[0174] For any frame of the reconstructed SIM image in the multi-frame reconstructed SIM image, determine the Hessian matrix of the reconstructed SIM image according to the AFM image and the reconstructed SIM image;

[0175] Perform artifact removal processing on the reconstructed SIM image according to the Hessian matrix of the reconstructed SIM image to obtain the target imaging image corresponding to the reconstructed SIM image.

[0176] When performing secondary artifact removal processing on the reconstructed SIM image obtained from the first reconstruction based on the Hessian reconstruction method, any one frame of the reconstructed SIM image is still used as an example for illustration.

[0177] It should be understood that the core of the Hessian reconstruction algorithm is to use the Hessian matrix to analyze and process the image, effectively capturing features such as edges, corners, and spots in the image, so as to facilitate image reconstruction and denoising.

[0178] Therefore, for any one frame of the reconstructed SIM image in the multi-frame reconstructed SIM image, first, the Hessian matrix of the reconstructed SIM image can be calculated to facilitate artifact removal processing on the reconstructed SIM image, and finally, the target imaging image corresponding to the reconstructed SIM image can be obtained.

[0179] In the above technical solution, during the joint imaging process of the SIM image and the AFM image, by introducing the Hessian matrix, artifacts in the multi-frame reconstructed SIM image can be effectively identified and eliminated. Artifacts are usually caused by noise, distortion during the imaging process, or imperfections in the reconstruction algorithm. The Hessian matrix can capture the local structural information in the image, thereby helping to distinguish real signals from artifacts. By combining the AFM image and the reconstructed SIM image and using the Hessian matrix for processing, the quality of the final target imaging image can be significantly improved. The AFM image provides high-resolution surface topography information. Combining with the super-resolution characteristics of the SIM image, a clearer and more accurate imaging result can be generated, ensuring that the reconstructed target imaging image has higher resolution and richer details, and thus ensuring the consistency and accuracy of the entire image sequence. This is particularly important for dynamic imaging or time series analysis.

[0180] It should also be understood that in the embodiments of the present application, when denoising the reconstructed SIM image, it is mainly achieved by analyzing whether the pixel points in the reconstructed SIM image are artifact points (for example, noise points or isolated points). Therefore, for the Hessian matrix of the reconstructed SIM image, it can be regarded as calculating the Hessian matrix of each pixel point in the reconstructed SIM image.

[0181] The calculation process of the Hessian matrix of each pixel point is introduced below.

[0182] In a possible implementation, according to the AFM image and the reconstructed SIM image, determining the Hessian matrix of the reconstructed SIM image includes:

[0183] Obtaining the acquisition time of the reconstructed SIM image;

[0184] For any target pixel point in the reconstructed SIM image, determining a plurality of neighboring pixel points within a local neighborhood window centered on the target pixel point;

[0185] Obtaining the pixel coordinates of the target pixel point and the pixel coordinates of the plurality of neighboring pixel points;

[0186] Determining a target sample point corresponding to the target pixel point from the AFM image, and obtaining the height information of the target sample point;

[0187] Determining a plurality of neighboring sample points corresponding to the plurality of neighboring pixel points from the AFM image, and obtaining the height information of the plurality of neighboring sample points;

[0188] Determining the Hessian matrix of the target pixel point according to the acquisition time, the pixel coordinates of the target pixel point, the height information of the target sample point, the pixel coordinates of the plurality of neighboring pixel points, and the height information of the plurality of neighboring sample points.

[0189] It should be understood that for any pixel point, since the pixel points are discrete, when calculating the Hessian matrix of the pixel point, it is necessary to calculate with the pixel points within the neighborhood window of the pixel point.

[0190] Specifically, for any target pixel point in the reconstructed SIM image, the electronic device can center on the target pixel point and obtain a plurality of neighboring pixel points within the local neighborhood window of the target pixel point.

[0191] Among them, the size of the neighborhood window can be determined according to actual needs. For example, 3×3 (corresponding to 9 pixels), and the embodiments of the present application do not limit the size of the neighborhood window.

[0192] Since the AFM image can provide the height information of each sample point on the target sample, therefore, when calculating the Hessian matrix, the embodiments of the present application also combine the height information obtained from the AFM image, and perform artifact removal processing on the reconstructed SIM image based on the pixel information of the pixel points on the SIM image and the height information obtained from the AFM.

[0193] After selecting the target pixel point and the neighborhood window centered on the target pixel point, for the target pixel point and the plurality of neighboring pixel points within the neighborhood window, the electronic device can obtain the pixel coordinates of the target pixel point and the pixel coordinates of the plurality of neighboring pixel points based on the reconstructed SIM image.

[0194] In addition, each pixel corresponds to a sample point on the AFM image. Therefore, based on the target pixel, the electronic device can determine the target sample point corresponding to the target pixel on the target sample, and obtain the height information of the target sample point based on the AFM image. Similarly, based on multiple neighboring pixels, the electronic device can determine multiple neighboring sample points corresponding to the multiple neighboring pixels, and obtain the height information of the multiple neighboring sample points based on the AFM image.

[0195] In addition, when calculating the Hessian matrix, in order to optimize the reconstruction process, the time dimension also needs to be considered. For the currently reconstructed SIM image, the corresponding original SIM image has an acquisition time when it is acquired, that is, the acquisition time, which can be directly obtained by the electronic device.

[0196] When determining the Hessian matrix of a pixel based on the pixel coordinates, height information, and time, its expression can be represented by the following formula (4).

[0197]

[0198] When calculating the Hessian matrix of the target pixel, the acquisition time of the reconstructed SIM image, the pixel coordinates and height information of the target pixel, and the pixel coordinates and height information of multiple neighboring pixels can be used to approximately calculate the second-order partial derivatives and mixed partial derivatives within the neighborhood window using the discrete difference formula, and substitute them into the above formula (4) to obtain the Hessian matrix of the target pixel.

[0199] After obtaining the Hessian matrix of the target pixel, the electronic device can perform artifact removal processing on the reconstructed SIM image based on the Hessian matrix of the target pixel to obtain the final target imaging image.

[0200] Specifically, when performing artifact removal processing on the reconstructed SIM image, for each pixel, the electronic device can first determine whether the current pixel is an artifact point to be removed according to the Hessian matrix of the pixel.

[0201] In a possible implementation, performing artifact removal processing on the reconstructed SIM image according to the Hessian matrix of the reconstructed SIM image to obtain the target imaging image corresponding to the reconstructed SIM image includes:

[0202] Determining the eigenvalues and eigenvectors of the Hessian matrix of the target pixel;

[0203] Determining whether the target pixel is an artifact point according to the eigenvalues and eigenvectors;

[0204] In the case where the target pixel is an artifact pixel, artifact removal processing is performed on the reconstructed SIM image to obtain a target imaging image.

[0205] For any target pixel in the reconstructed SIM image, after determining the Hessian matrix of the target pixel, the electronic device can obtain the eigenvalues and eigenvectors of the Hessian matrix of the target pixel, and based on the eigenvalues and eigenvectors, determine whether the target pixel is an artifact pixel.

[0206] Specifically, the Hessian matrix of the above target pixel is a fourth-order matrix, and there are 4 corresponding eigenvalues, denoted as "λ1, λ2, λ3, λ4". If the absolute values of all eigenvalues are small, it indicates that the target pixel may be an artifact pixel. In addition, if the direction of the eigenvector of the Hessian matrix is inconsistent with the direction of the eigenvectors of the surrounding pixels, it indicates that the target pixel may be an artifact pixel.

[0207] Therefore, in the embodiments of the present application, a threshold a can be preset to distinguish the magnitudes of the eigenvalues. After calculating the four eigenvalues of the Hessian matrix, the electronic device can compare the absolute values of all eigenvalues with the above a respectively. When the absolute values of all eigenvalues are less than a and the direction of the eigenvector of the current Hessian matrix is inconsistent with the direction of the eigenvectors of the surrounding pixels, the electronic device can determine that the current target pixel is an artifact pixel.

[0208] When the target pixel is an artifact pixel, the electronic device can adjust the pixel value of the target pixel to reduce artifacts, thereby obtaining the target imaging image corresponding to the current reconstructed SIM image.

[0209] In a possible implementation, in the case where the target pixel is an artifact pixel, artifact removal processing is performed on the reconstructed SIM image to obtain a target imaging image, including:

[0210] In the case where the target pixel is an artifact pixel, obtain the pixel values of multiple neighboring pixels;

[0211] Determine the average pixel value of the pixel values of the multiple neighboring pixels;

[0212] Replace the pixel value of the target pixel with the average pixel value to obtain a reconstructed SIM image with corrected pixels;

[0213] Optimize the reconstructed SIM image with corrected pixels to obtain a target imaging image.

[0214] It should be understood that pixel values in a real image usually have local continuity, that is, the pixel values of adjacent pixel points are correlated. Therefore, when the target pixel point is an artifact point, the electronic device can adjust the pixel value of the target pixel point to be the same as the pixel values of the surrounding pixel points, thereby correcting the unreasonable pixel values in the image to make it more conform to the real physical structure or statistical law, achieving the effect of reducing artifact points.

[0215] Specifically, the electronic device can calculate the average pixel value of the pixel values of multiple neighboring pixel points within the neighborhood window centered on the target pixel point, and thereby replace the pixel value of the target pixel point with the average pixel value, so that the pixel value of the adjusted target pixel point is relatively consistent with the pixel values of multiple pixel points in the surrounding neighborhood.

[0216] Thus, for any target pixel point in the reconstructed SIM image, when the target pixel point is an artifact point, through the above pixel correction process, a reconstructed SIM image with pixel correction can be obtained.

[0217] In the above technical solution, by replacing the pixel value of the target artifact point with the average value of neighboring pixels, the noise and artifacts in the image are effectively reduced, the clarity and accuracy of the image are improved, local mutations are reduced, and the image becomes more natural and coherent.

[0218] Further, based on the reconstructed SIM image after correction, the electronic device can be further optimized to obtain the final target imaging image.

[0219] In a possible implementation manner, optimizing the reconstructed SIM image after pixel correction to obtain a target imaging image includes:

[0220] Determining an image reconstruction loss according to the reconstructed SIM image after pixel correction;

[0221] Determining an image constraint loss based on the height information of the reconstructed SIM image after pixel correction and the height information of the AFM image;

[0222] Determining a target loss function based on the image reconstruction loss, the image constraint loss, the regularization term, the first weight corresponding to the image reconstruction loss, the second weight corresponding to the image constraint loss, and the third weight corresponding to the regularization term;

[0223] Performing multiple iterations on the target loss function until the target loss function meets a preset iteration condition, and outputting the target imaging image corresponding to the target loss function.

[0224] After obtaining the reconstructed SIM image after pixel correction, the embodiments of the present application can further optimize it to improve the imaging quality.

[0225] Specifically, in the embodiments of the present application, the reconstructed SIM image after pixel correction is optimized by constructing a target loss function.

[0226] The expression of the target loss function is shown in the following formula (5).

[0227] L = αL SIM + βL AFM + γR Formula (5)

[0228] Among them, in formula (5):

[0229] L: Target loss function;

[0230] α: First weight;

[0231] β: Second weight;

[0232] γ: Third weight;

[0233] L SIM : Reconstruction loss based on the SIM image, that is, image reconstruction loss;

[0234] L AFM : Reconstruction loss based on the AFM image, that is, image constraint loss;

[0235] R: Regularization term.

[0236] The above L AFM can be represented by the following formula (6).

[0237] L AFM = ||H SM - H AFM || 2 Formula (6)

[0238] Among them, in formula (6):

[0239] H SIM : Height information of the reconstructed SIM image after pixel correction;

[0240] H AFM : Height information of the target sample in the AFM image.

[0241] Exemplarily, for the height information of the reconstructed SIM image after pixel correction, after obtaining the reconstructed SIM image after pixel correction, a person skilled in the art can collect multiple SIM images from different angles and different focal planes. After the electronic device obtains multiple SIM images, it preprocesses the obtained multi-angle and multi-plane SIM images, and uses the SIM reconstruction algorithm to obtain the three-dimensional structure of the target sample, and extracts the height information of the target sample from the reconstructed three-dimensional structure, that is, H SIM .

[0242] It should be understood that H SIM is usually a two-dimensional matrix. The dimensions of the matrix depend on the resolution and scanning range of SIM imaging. For example, assuming the resolution of SIM imaging is m×n, then H SIM has dimensions of m×n. Each value in the matrix represents the height information at the corresponding pixel position.

[0243] For an AFM image, since the target sample does not change throughout the process, H AFM is the height information of each sample point included in the AFM image. H AFM is also a two-dimensional matrix. The dimensions of the matrix depend on the resolution and scanning range of AFM imaging. For example, assuming the resolution of AFM imaging is p×q, then H AFM has dimensions of p×q. Each value in the matrix represents the height information at the corresponding sample point position.

[0244] In the embodiments of the present application, in order to ensure the feasibility of the calculation of formula (6), the resolution of AFM imaging and the resolution of SIM imaging can be set to be the same. When the resolution of SIM imaging and the resolution of AFM imaging are inconsistent, the resolution of one of the matrices can be adjusted by difference or other image processing methods to make it consistent with the resolution of the other matrix.

[0245] After obtaining the height information obtained by each of the two microscopic imagings, the image constraint loss can be calculated through formula (6).

[0246] For the image reconstruction loss, the electronic device can obtain the image reconstruction loss by calculating the difference between the reconstructed SIM image after pixel correction and the reference image. Among them, the reference image can be an ideal imaging image generated by the model from the aforementioned original SIM image.

[0247] Optionally, the calculation methods of the image reconstruction loss include Mean Squared Error (MSE) and Structural Similarity Index Measure (SSIM).

[0248] Specifically, the electronic device can obtain the mean square error MSE as the target reconstruction loss by calculating the sum of the squares of the differences between each pixel value of the reconstructed SIM image after the above pixel correction and the reference image.

[0249] In the process of initially calculating the target loss function, an initial first weight, second weight, third weight, and regularization term can be set, and the target loss function in the first iteration process is calculated. If the current does not meet the preset iteration condition, continue to iterate the target loss function.

[0250] Optionally, the preset iteration condition can be that the number of iterations reaches a certain number or the target loss function is less than a preset threshold.

[0251] In the continuous iteration process, the above-mentioned various weights and regularization terms are continuously adjusted, so as to dynamically balance the importance of different loss terms, change the characteristics of the image, optimize the reconstruction quality, and enhance the structural continuity of the image. When the target loss function meets the preset iteration condition, the SIM image corresponding to the currently calculated target loss function is output as the final target imaging image.

[0252] Through the above steps 301 - step 303, the electronic device can obtain the target imaging image after two reconstructions of each frame of the original SIM image.

[0253] In the above technical solution, through the height information of the reconstructed SIM image and the AFM image after pixel correction, combined with the image reconstruction loss and the image constraint loss, the target image can be reconstructed more accurately and the error can be reduced. By using the different characteristics of the SIM image and the AFM image, through weight allocation and regularization terms, the effective fusion of multi-source information is realized, and the robustness and accuracy of image reconstruction are improved. By iteratively optimizing the target loss function multiple times, it is ensured that the finally output target imaging image meets the preset conditions, further improving the image quality.

[0254] In summary, in the process of SIM microscopy imaging, the present application proposes a structured illumination microscopy imaging method based on AFM imaging prior. In the imaging process, first, the first artifact removal process is performed on a single SIM image to reduce the artifacts generated in the reconstruction process and improve the quality and resolution of the image. Further, on the basis of the single SIM reconstructed image, combined with the AFM image for multimodal reconstruction, the accuracy and reliability of imaging are further improved, making full use of the advantages of different imaging technologies, providing more comprehensive sample information, and achieving the purpose of improving the image reconstruction efficiency and reconstruction quality.

[0255] Figure 4 It is a schematic structural diagram of a structured illumination microscopy imaging device based on AFM imaging prior provided by an embodiment of the present application.

[0256] Exemplarily, as Figure 4 shown, the device 400 includes:

[0257] An image acquisition module 401, configured to acquire a continuous multi-frame original SIM image corresponding to a target sample;

[0258] A first reconstruction module 402, configured to perform artifact removal processing on the multi-frame original SIM images based on a first image reconstruction method, to obtain multi-frame reconstructed SIM images corresponding to the multi-frame original SIM images;

[0259] A second reconstruction module 403, configured to perform artifact removal processing on the multi-frame reconstructed SIM images based on a second image reconstruction method and an AFM image of the target sample, to obtain multi-frame target imaging images corresponding to the multi-frame reconstructed SIM images.

[0260] In a possible implementation, the first image reconstruction method includes a Wiener deconvolution method and a rolling reconstruction method. The first reconstruction module 402 is specifically configured to: for any one of the multi-frame original SIM images, perform deconvolution processing on the original SIM image based on the Wiener deconvolution method, to obtain a restored SIM image of the original SIM image; perform rolling reconstruction processing on the restored SIM images of the multi-frame original SIM images based on the rolling reconstruction method, to obtain the multi-frame reconstructed SIM images.

[0261] In a possible implementation, the first reconstruction module 402 is further configured to: obtain a modal wave vector of the microscopic imaging system; determine a point spread function of the microscopic imaging system according to the modal wave vector; obtain a noise power spectrum of the original SIM image, and a power spectrum of the original SIM image; determine a restored image of the original SIM image according to the point spread function, the original SIM image, the noise power spectrum, and the power spectrum.

[0262] In a possible implementation, the first reconstruction module 402 is further configured to: for any one of the restored images of the multi-frame original SIM images, traverse the restored image of the original SIM image based on a preset rolling window, to determine a target local restored image corresponding to the preset rolling window at the current moment; perform feature extraction on the target local restored image, to obtain local image features of the target local restored image; perform filtering processing on the target local restored image based on the local image features, to obtain a filtered target local restored image; perform reconstruction on the filtered target local restored image, to obtain a reconstructed target local restored image; when the traversal of the restored image of the original SIM image is completed, splice a plurality of reconstructed local restored images corresponding to the restored image of the original SIM image, to obtain a reconstructed SIM image corresponding to the original SIM image.

[0263] In a possible implementation manner, the second image reconstruction method includes a Hessian reconstruction method. Specifically, the second reconstruction module 403 is configured to: for any frame of the multi-frame reconstructed SIM image, determine the Hessian matrix of the reconstructed SIM image according to the AFM image and the reconstructed SIM image; perform artifact removal processing on the reconstructed SIM image according to the Hessian matrix of the reconstructed SIM image to obtain the target imaging image corresponding to the reconstructed SIM image.

[0264] In a possible implementation manner, the second reconstruction module 403 is further configured to: obtain the acquisition time of the reconstructed SIM image; for any target pixel point in the reconstructed SIM image, determine a plurality of neighboring pixel points within a local neighborhood window centered on the target pixel point; obtain the pixel coordinates of the target pixel point and the pixel coordinates of the plurality of neighboring pixel points; determine the target sample point corresponding to the target pixel point from the AFM image, and obtain the height information of the target sample point; determine the plurality of neighboring sample points corresponding to the plurality of neighboring pixel points from the AFM image, and obtain the height information of the plurality of neighboring sample points; determine the Hessian matrix of the target pixel point according to the acquisition time, the pixel coordinates of the target pixel point, the height information of the target sample point, the pixel coordinates of the plurality of neighboring pixel points, and the height information of the plurality of neighboring sample points.

[0265] In a possible implementation manner, the second reconstruction module 403 is further configured to: determine the eigenvalues and eigenvectors of the Hessian matrix of the target pixel point; determine whether the target pixel point is an artifact point according to the eigenvalues and the eigenvectors; in the case where the target pixel point is an artifact point, perform artifact removal processing on the reconstructed SIM image to obtain the target imaging image.

[0266] In a possible implementation manner, the second reconstruction module 403 is further configured to: in the case where the target pixel point is an artifact point, obtain the pixel values of the plurality of neighboring pixel points; determine the average pixel value of the pixel values of the plurality of neighboring pixel points; replace the pixel value of the target pixel point with the average pixel value to obtain a reconstructed SIM image with corrected pixels; optimize the reconstructed SIM image with corrected pixels to obtain the target imaging image.

[0267] In a possible implementation, the second reconstruction module 403 is further configured to: determine an image reconstruction loss according to the reconstructed SIM image after pixel correction; determine an image constraint loss for the image reconstructed based on the AFM image based on the height information of the reconstructed SIM image after pixel correction and the height information of the AFM image; determine a target loss function based on the image reconstruction loss, the image constraint loss, a regularization term, a first weight corresponding to the image reconstruction loss, a second weight corresponding to the image constraint loss, and a third weight corresponding to the regularization term; perform multiple iterations on the target loss function until the target loss function meets a preset iteration condition, and output the target imaging image corresponding to the target loss function.

[0268] Figure 5 FIG. 4 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.

[0269] Exemplarily, as Figure 5 shown, the electronic device 102 includes: a memory 501 and a processor 502. Among them, an executable program code 5011 is stored in the memory 501, and the processor 502 is configured to call and execute the executable program code 5011 to execute a structured illumination microscopy imaging method based on AFM imaging prior.

[0270] In this embodiment, the electronic device can be divided into functional modules according to the above method example. For example, it can correspond to each functional module, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is illustrative, only a logical function division, and there can be other division methods in actual implementation.

[0271] In the case of dividing each functional module according to each function, the electronic device may include: an image acquisition module, a first reconstruction module, a second reconstruction module, etc. It should be noted that all relevant contents of each step involved in the above method embodiment can be cited in the function description of the corresponding functional module, and will not be repeated here.

[0272] The electronic device provided in this embodiment is used to execute the above-mentioned structured illumination microscopy imaging method based on AFM imaging prior, so it can achieve the same effect as the above implementation method.

[0273] In the case of adopting an integrated unit, the electronic device may include a processing module and a storage module. Among them, the processing module can be used to control and manage the actions of the electronic device. The storage module can be used to support the electronic device to execute relevant program codes and data, etc.

[0274] Among them, the processing module can be a processor or a controller, which can implement or execute various exemplary logical blocks, modules, and circuits shown in combination with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc. The storage module can be a memory.

[0275] This embodiment also provides a computer-readable storage medium, in which computer program code is stored. When the computer program code runs on a computer, it causes the computer to execute the above-related method steps to implement a structured illumination microscopy imaging method based on AFM imaging prior in the above embodiment.

[0276] This embodiment also provides a computer program product. When the computer program product runs on a computer, it causes the computer to execute the above-related steps to implement a structured illumination microscopy imaging method based on AFM imaging prior in the above embodiment.

[0277] In addition, the electronic device provided in the embodiment of this application can specifically be a chip, a component, or a module. The electronic device can include a processor and a memory connected to each other. Among them, the memory is used to store instructions. When the electronic device runs, the processor can call and execute the instructions to cause the chip to execute a structured illumination microscopy imaging method based on AFM imaging prior in the above embodiment.

[0278] Among them, the electronic device, computer-readable storage medium, computer program product, or chip provided in this embodiment are all used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be elaborated here.

[0279] Through the description of the above embodiments, those skilled in the art can understand that for the convenience and simplicity of description, only the above division of each functional module is used as an example. In practical applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0280] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of devices or units can be in electrical, mechanical or other forms.

[0281] The above content is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed in the present application can easily think of changes or substitutions, which should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A structured illumination microscopy imaging method based on AFM imaging prior, characterized in that The method includes: Obtaining a series of consecutive original SIM images corresponding to a target sample; Performing artifact removal processing on the series of original SIM images based on a first image reconstruction method to obtain a series of reconstructed SIM images corresponding to the series of original SIM images; Performing artifact removal processing on the series of reconstructed SIM images based on a second image reconstruction method and an AFM image of the target sample to obtain a series of target imaging images corresponding to the series of reconstructed SIM images.

2. The method according to claim 1, characterized in that The first image reconstruction method includes a Wiener deconvolution method and a rolling reconstruction method. The performing artifact removal processing on the series of original SIM images based on the first image reconstruction method to obtain a series of reconstructed SIM images corresponding to the series of original SIM images includes: For any one of the series of original SIM images, performing deconvolution processing on the original SIM image based on the Wiener deconvolution method to obtain a restored SIM image of the original SIM image; Performing rolling reconstruction processing on the restored SIM images of the series of original SIM images based on the rolling reconstruction method to obtain the series of reconstructed SIM images.

3. The method according to claim 2, characterized in that, The performing deconvolution processing on the original SIM image based on the Wiener deconvolution method to obtain a restored SIM image of the original SIM image includes: Obtaining the modal wave vector of a microscopy imaging system; Determining the point spread function of the microscopy imaging system according to the modal wave vector; Obtaining the noise power spectrum of the original SIM image and the power spectrum of the original SIM image; Determining the restored image of the original SIM image according to the point spread function, the original SIM image, the noise power spectrum, and the power spectrum.

4. The method according to claim 2, wherein The performing rolling reconstruction processing on the restored images of the series of original SIM images to obtain the series of reconstructed SIM images includes: For the restored image of any one of the restored images of the series of original SIM images, traversing the restored image of the original SIM image based on a preset rolling window to determine a target local restored image corresponding to the preset rolling window at the current moment; Performing feature extraction on the target local restored image to obtain local image features of the target local restored image; Performing filtering processing on the target local restored image based on the local image features to obtain a filtered target local restored image; Performing reconstruction on the filtered target local restored image to obtain a reconstructed target local restored image; When the traversal of the restored images of the original SIM image is completed, splicing a plurality of reconstructed local restored images corresponding to the restored images of the original SIM image to obtain a reconstructed SIM image corresponding to the original SIM image.

5. The method according to claim 1, characterized in that The second image reconstruction method includes a Hessian reconstruction method. Based on the second image reconstruction method and the AFM image of the target sample, artifact removal processing is performed on the multi-frame reconstructed SIM images to obtain multi-frame target imaging images corresponding to the multi-frame reconstructed SIM images, including: For any one of the multi-frame reconstructed SIM images, a Hessian matrix of the reconstructed SIM image is determined according to the AFM image and the reconstructed SIM image; Based on the Hessian matrix of the reconstructed SIM image, artifact removal processing is performed on the reconstructed SIM image to obtain a target imaging image corresponding to the reconstructed SIM image.

6. The method according to claim 1, characterized in that, The determining of the Hessian matrix of the reconstructed SIM image according to the AFM image and the reconstructed SIM image includes: Obtaining the acquisition time of the reconstructed SIM image; For any target pixel point in the reconstructed SIM image, determining a plurality of neighboring pixel points within a local neighborhood window centered on the target pixel point; Obtaining the pixel coordinates of the target pixel point and the pixel coordinates of the plurality of neighboring pixel points; Determining a target sample point corresponding to the target pixel point from the AFM image, and obtaining the height information of the target sample point; Determining a plurality of neighboring sample points corresponding to the plurality of neighboring pixel points from the AFM image, and obtaining the height information of the plurality of neighboring sample points; Based on the acquisition time, the pixel coordinates of the target pixel point, the height information of the target sample point, the pixel coordinates of the plurality of neighboring pixel points, and the height information of the plurality of neighboring sample points, determining the Hessian matrix of the target pixel point.

7. The method according to claim 6, wherein The performing of artifact removal processing on the reconstructed SIM image based on the Hessian matrix of the reconstructed SIM image to obtain a target imaging image corresponding to the reconstructed SIM image includes: Determining the eigenvalues and eigenvectors of the Hessian matrix of the target pixel point; Based on the eigenvalues and the eigenvectors, determining whether the target pixel point is an artifact point; In the case where the target pixel point is an artifact point, performing artifact removal processing on the reconstructed SIM image to obtain the target imaging image.

8. The method according to claim 7, characterized in that, In the case where the target pixel point is an artifact point, the performing of artifact removal processing on the reconstructed SIM image to obtain the target imaging image includes: In the case where the target pixel point is an artifact point, obtaining the pixel values of the plurality of neighboring pixel points; Determining the average pixel value of the pixel values of the plurality of neighboring pixel points; Replacing the pixel value of the target pixel point with the average pixel value to obtain a reconstructed SIM image with pixel correction; Optimizing the reconstructed SIM image with pixel correction to obtain the target imaging image.

9. The method according to claim 8, characterized in that, The optimizing of the reconstructed SIM image with pixel correction to obtain the target imaging image includes: Determining an image reconstruction loss according to the reconstructed SIM image with pixel correction; Determine an image constraint loss reconstructed based on the AFM image based on the height information of the reconstructed SIM image after pixel correction and the height information of the AFM image; Determine a target loss function based on the image reconstruction loss, the image constraint loss, a regularization term, a first weight corresponding to the image reconstruction loss, a second weight corresponding to the image constraint loss, and a third weight corresponding to the regularization term; Iterate the target loss function multiple times until the target loss function meets a preset iteration condition, and output the target imaging image corresponding to the target loss function.

10. An electronic device, characterized in that, The electronic device includes: A memory for storing executable program code; A processor for calling and running the executable program code from the memory, so that the electronic device executes the method according to any one of claims 1 to 9.

Images