Super-resolution imaging method, apparatus, electronic device, storage medium, and program product

Abstract

The present application relates to the technical field of optical microscopic imaging, and discloses a super-resolution imaging method, device, electronic equipment, storage medium and program product, the method comprising: acquiring multiple Gaussian illumination images and multiple annular illumination images; for any focal point in any frame Gaussian illumination image corresponding to the local Gaussian illumination image, determining the local annular illumination image corresponding to the local Gaussian illumination image; and performing pixel-by-pixel complementary operation on the local Gaussian illumination image and the local annular illumination image to obtain a local super-resolution image, so as to obtain multiple super-resolution images, the super-resolution image is obtained by pixel backfilling the local super-resolution image, and the super-resolution image has the same size as the Gaussian illumination image and the annular illumination image; and generating a global fusion image of a target object based on the multiple super-resolution images.

Description

Technical Field

[0001] This invention relates to the field of optical microscopy imaging technology, specifically to super-resolution imaging methods, devices, electronic devices, storage media, and software products. Background Technology

[0002] Gaussian and ring-shaped light spots exhibit complementary spatial responses. Gaussian spots have the highest intensity at the focal center, while ring-shaped or hollow light spots have lower intensity at the center and stronger response in the peripheral region. Based on this complementary response, differential, divisional, or fluorescence radiation differential methods can compress the equivalent point spread function, enhance edges, or improve resolution under certain conditions.

[0003] In related technologies, the target object is generally scanned using a single-focus Gaussian spot and a single-focus annular spot to obtain Gaussian and annular images, and the Gaussian and annular images are then differentially analyzed. However, when the target object is a dense tissue sample (such as tissue sample, organelle network, collagen fiber, mitochondria, microtubules, nerve fiber), the signals at each pixel are often not local signals of a single structure because the responses of adjacent structures are superimposed in the complete field of view image, thus distorting the imaging results. Summary of the Invention

[0004] This invention provides a super-resolution imaging method, apparatus, electronic device, storage medium, and program product to solve the problem in related technologies where the responses of adjacent structures are superimposed in the complete field of view image, and the signal at each pixel is often not a local signal of a single structure, thus causing the imaging results to be distorted.

[0005] In a first aspect, the present invention provides a super-resolution imaging method, the method comprising: acquiring multiple frames of Gaussian illumination images and multiple frames of annular illumination images, wherein the Gaussian illumination images and the annular illumination images are obtained by scanning a target object, both the Gaussian illumination images and the annular illumination images include multiple focal points, the positions of the multiple focal points in each frame of the multiple Gaussian illumination images are different, the positions of the multiple focal points in each frame of the multiple annular illumination images are different, and the focal point positions of the Gaussian illumination images corresponding to the same frame correspond to the focal point positions of the annular illumination images; determining a local annular illumination image corresponding to a local Gaussian illumination image for a local Gaussian illumination image corresponding to any focal point in any frame of the Gaussian illumination images; performing a pixel-by-pixel complementary operation on the local Gaussian illumination images and the local annular illumination images to obtain a weight matrix; obtaining multiple frames of super-resolution images based on the weight matrix and the local Gaussian illumination images, wherein the super-resolution images are of the same size as the Gaussian illumination images and the annular illumination images; and generating a global fusion image of the target object based on the multiple frames of super-resolution images.

[0006] According to embodiments of the present invention, by acquiring a Gaussian illumination image with multiple focal points and an annular illumination image with multiple focal points, and performing pixel-complementary processing on the local Gaussian illumination image corresponding to any focal point in any frame of the Gaussian illumination image and the local annular illumination image, the mutual influence between pixels is reduced, so that each local super-resolution image only reflects the signal response of the local area where the corresponding focal point is located. This effectively avoids the signal aliasing problem caused by the superposition of adjacent structural responses in the complete field of view image. By supplementing the image information of the same spatial position under the two illumination modes at the pixel level, the resolution and detail richness of the local image are effectively improved. At the same time, by backfilling the pixels of each local super-resolution image according to its corresponding focal position to obtain a multi-frame super-resolution image of the same size as the original image, and then fusing the multi-frame super-resolution images to generate a global fusion image, the signal of each pixel position in the global fusion image comes from the local real response at the corresponding focal point in each frame. This solves the imaging distortion problem caused by signal superposition in the overall difference and ensures the accuracy of the imaging results of dense tissue samples.

[0007] In an optional implementation, before determining the local annular illumination image corresponding to the local Gaussian illumination image, the method further includes: Using image localization algorithms, the positions of multiple focal points in any of the above frames of Gaussian-illuminated images are determined; Using image localization algorithms, the positions of multiple focal points in any of the above-mentioned frame of ring-lit images are determined; For any focal point in any frame of Gaussian illumination image, the local Gaussian illumination image is extracted from the aforementioned Gaussian illumination image based on a preset local image pixel size and the position of the focal point; For any focal point in any frame of ring illumination image, the local ring illumination image is extracted from the aforementioned frame of ring illumination image based on a preset local image pixel size and the position of the focal point.

[0008] In an optional implementation, the pixel-by-pixel complementary operation on the local Gaussian illumination image and the local annular illumination image to obtain the weight matrix includes: Perform pixel-by-pixel complementation on the above local Gaussian illumination image and the above local ring illumination image using any of the following formulas:

[0009]

[0010]

[0011]

[0012]

[0013] in, This is the weight matrix. This is the preprocessed local Gaussian illumination image. This is the preprocessed local ring illumination image. For fixed coefficients, It is a stable term.

[0014] In one optional implementation, obtaining multiple super-resolution images based on the aforementioned weight matrix and the aforementioned local Gaussian illumination image includes: A local super-resolution image is obtained based on the aforementioned weight matrix and the aforementioned local Gaussian illumination image. The above-mentioned multi-frame super-resolution images are obtained based on all the above-mentioned local super-resolution images; The aforementioned local super-resolution image was obtained using the following formula:

[0015] in, For local super-resolution images, Here, s is the weight matrix, s is the scan position, x is the x-coordinate, and y is the y-coordinate. This is the background estimate of the weight matrix. The normalization factor for the local ring illumination image. For the stability term, F represents local standardization, R() is the constraint function, and G is the stability term. i (s,x,y) is a Gaussian illumination image.

[0016] In an optional implementation, the super-resolution image is obtained through the following steps: In the case where at least two local super-resolution images have overlapping regions in any frame, the weighting coefficients corresponding to the local super-resolution images are determined based on the signal-to-noise ratio of each of the local super-resolution images with overlapping regions. Using the aforementioned weighting coefficients, the aforementioned local super-resolution images with overlapping regions are weighted and fused to obtain the aforementioned super-resolution image.

[0017] In an optional embodiment, the above method is applied to an imaging system, which includes a light field modulation module for modulating an excitation beam to generate a Gaussian illumination beam and a ring illumination beam. The Gaussian illumination beam forms multiple Gaussian excitation focal points on the target object, and the ring illumination beam forms multiple ring excitation focal points on the target object.

[0018] In a second aspect, the present invention provides a super-resolution imaging device, comprising: The acquisition module is used to acquire multiple frames of Gaussian illumination images and multiple frames of ring illumination images. The Gaussian illumination images and the ring illumination images are obtained by scanning the target object. Both the Gaussian illumination images and the ring illumination images include multiple focal points. The positions of the multiple focal points in each frame of the multiple Gaussian illumination images are different, and the positions of the multiple focal points in each frame of the multiple ring illumination images are different. The focal point positions of the Gaussian illumination images corresponding to the same frame have a corresponding relationship with the focal point positions of the ring illumination images. The determination module is used to determine a local ring illumination image corresponding to the local Gaussian illumination image corresponding to any focal point in any frame of Gaussian illumination image; The complementarity module is used to perform pixel-by-pixel complementarity operations on the aforementioned local Gaussian illumination image and the aforementioned local ring illumination image to obtain a weight matrix. The super-resolution image generation module is used to obtain multiple super-resolution images based on the weight matrix and the local Gaussian illumination image, wherein the super-resolution images are the same size as the Gaussian illumination image and the ring illumination image. The generation module is used to generate a global fused image of the target object based on the aforementioned multi-frame super-resolution images.

[0019] Thirdly, the present invention provides an electronic device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the super-resolution imaging method of the first aspect or any corresponding embodiment described above.

[0020] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to perform the super-resolution imaging method of the first aspect or any corresponding embodiment thereof.

[0021] Fifthly, the present invention provides a computer program product, including computer instructions for causing a computer to execute the super-resolution imaging method of the first aspect or any corresponding embodiment thereof. Attached Figure Description

[0022] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1This is a schematic diagram of the imaging system used in the super-resolution imaging method according to an embodiment of the present invention; Figure 2 This is a flowchart of a super-resolution imaging method according to an embodiment of the present invention; Figure 3 This is a schematic diagram of imaging processing of a target object according to an embodiment of the present invention; Figure 4 This is a structural block diagram of a super-resolution imaging device according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] It is understood that before using the technical solutions disclosed in the various embodiments of the present invention, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in the present invention and their authorization should be obtained in accordance with relevant laws and regulations through appropriate means.

[0026] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0027] Figure 1 This is a schematic diagram of an imaging system used in the super-resolution imaging method according to an embodiment of the present invention.

[0028] like Figure 1 As shown, the imaging system includes a laser 101, a half-wave plate 102, a polarizing beam splitter 103, a lens 104, a mirror 105, a light field control module 106, a scanning module 107, a dichroic mirror 108, a detector 109, an objective lens 110, a stage 111, and a terminal 112.

[0029] Among them, laser 101, also known as excitation source module, is used to provide excitation beam. The excitation beam can be continuous laser, picosecond laser or femtosecond laser, and is suitable for excitation by single photon, two photon, multi-photon, SHG or THG signals.

[0030] The half-wave plate 102 is used to rotate the polarization direction of the excitation beam. By rotating the angle of the half-wave plate, the intensity ratio of light transmitted through the subsequent polarization beam splitter can be adjusted, thereby achieving continuous adjustment of the excitation light power.

[0031] The polarization beam splitter 103 is used to split the incident light beam according to its polarization state. When used in conjunction with a half-wave plate, it can reflect a certain proportion of the light energy into the subsequent optical path, while preventing the return light from damaging the laser, thus acting as an optical isolator.

[0032] Lens 104 is used to expand and collimate the excitation beam, so that the beam size matches the effective working area of ​​the optical field control module 106, and improves the parallelism of the beam to improve the optical field control efficiency.

[0033] The reflector 105 is used to change the propagation direction of the excitation beam and guide the beam to the light field control module 106. The surface of the reflector 105 can be coated with a high reflectivity film to reduce the loss of light energy during propagation.

[0034] The light field manipulation module 106 may include a spatial light modulator, a digital micromirror device, a diffractive optical element, a microlens array, a vortex phase plate, a 0 / π annular phase plate, or a combination thereof, for applying phase modulation or amplitude modulation to the excitation beam, shaping a single laser beam into a multi-focus array, and switching between Gaussian focuses and annular focuses by loading different modulation phase maps. The light field manipulation module 106 may be a liquid crystal spatial light modulator, a digital micromirror device, or a deformable mirror. The light field manipulation module 106 may further include a multi-focus Gaussian lattice generation module and a multi-focus annular lattice generation module. The multi-focus Gaussian lattice generation module is used to form multiple Gaussian excitation focuses on the sample surface, each focus having a strong central response. The multi-focus annular lattice generation module is used to form multiple annular or hollow excitation focuses on the sample surface, each focus having a weak central response and a strong peripheral response, and forming a one-to-one correspondence or pairing relationship with the focuses in the multi-focus Gaussian lattice.

[0035] The scanning module 107 is used to drive the multi-focus Gaussian array and the multi-focus ring array to scan the sample along the same or corresponding scanning path. It can be a galvanometer, a resonant galvanometer, an acousto-optic deflector, an electro-optic deflector, a piezoelectric platform or a sample displacement stage.

[0036] Dichroic mirror 108 is used to reflect the excitation beam into objective lens 110, while allowing the fluorescence signal emitted by the sample to pass through and enter detector 109. The cutoff wavelength of the dichroic mirror 108 is selected according to the difference between the excitation wavelength and the fluorescence emission wavelength to achieve effective separation of the excitation light path and the fluorescence collection path.

[0037] The detector 109 can be a camera used to record two-dimensional image frames containing multiple focal local responses at each scanning position. It can be an sCMOS (scientific complementary metal-oxide-semiconductor) camera, an EMCCD (electron-multiplying charge-coupled device) camera, a CCD (charge-coupled device) camera, a CMOS (complementary metal-oxide-semiconductor) camera, or other area array detectors.

[0038] Objective lens 110 is used to focus the excitation beam after passing through light field modulation module 106 and scanning module 107 onto the sample surface to form a multifocal illumination spot, while collecting the fluorescence signal emitted by the sample. The numerical aperture of objective lens 110 can be selected according to the requirements of imaging resolution and fluorescence collection efficiency.

[0039] The stage 111 is used to carry the target object and perform three-dimensional displacement to achieve precise positioning and focal plane adjustment of the sample. The stage 111 can be a manual stage or an electric stage. The electric stage, together with the scanning module 107, can achieve large-area stitching imaging.

[0040] Terminal 112 is used to control the operating parameters of each component in the imaging system and to receive and process image data acquired by detector 109. Terminal 112 can be electrically connected to light field modulation module 106, scanning module 107, detector 109, and stage 111, respectively. Terminal 112 stores a computer program, which, when executed by a processor, implements a super-resolution imaging method and ultimately generates a global fused image of the target object. Terminal 112 may include a control and synchronization module and an image processing module. The control and synchronization module is used to control the switching between Gaussian and ring dot arrays, scan coordinate synchronization, camera exposure synchronization, dot array frame numbering, and data pairing. The image processing module is used to perform focus localization, local image block extraction, focus pairing, per-focus and per-pixel complementary operations, local result fusion, and final image output.

[0041] According to a feasible embodiment of the present invention, excitation light output from a laser is used to form a multi-focus Gaussian illumination array in a first operating mode and a multi-focus annular illumination array in a second operating mode via a light field control module 106. The two types of arrays have the same number of focal points, focal spacing, array arrangement, and scanning trajectory, or a predetermined correspondence exists. Further, a detector 109 is controlled to scan the target object, recording a frame of Gaussian illumination image at each scanning position s. This image simultaneously contains multiple Gaussian focal local responses, rather than a single intensity value acquired point-by-point by a single-point detector. Further, the multi-focus annular array is controlled to scan the target object at the same or corresponding scanning positions. At each scanning position s, the detector 109 records a frame of annular illumination image. This image simultaneously contains multiple annular focal local responses. The Gaussian illumination image and the annular illumination image are paired in terms of scanning position, exposure sequence, and focal point number.

[0042] In related technologies, a single-focus Gaussian spot and a single-focus annular spot are typically used to scan the target object sequentially to obtain the Gaussian image and the annular image. Then, global difference or global complementation operations are performed on the entire Gaussian image and the entire annular image. The above method has at least the following problems: On the one hand, single-focus scanning imaging is slow and cannot meet the rapid imaging requirements of wide field, large field of view, or dynamic samples; on the other hand, when the target object is a dense tissue sample, a cell cluster sample, or a sample with a densely distributed structure, the responses of adjacent structures are easily superimposed in the complete field of view image. This results in the signal at each pixel not being a response of a single focus or a single local structure. Directly performing whole-image complementation operations can easily cause artifacts, signal distortion, or false enhancement of local structures.

[0043] Meanwhile, traditional single-focus confocal or FED systems mostly employ single-point detectors such as PMTs and APDs. These single-point detectors record signals point-by-point according to the scanning coordinates, ultimately forming a full-field image. This imaging method obtains a two-dimensional image accumulated over the scanning process, rather than multiple local focal images recorded by the camera simultaneously. Due to the lack of parallel spatial recording of multi-focal local responses by a wide-field camera, it is difficult to perform independent complementary calculations on each focal local region at the data level.

[0044] Multifocal scanning microscopy significantly improves imaging speed by simultaneously forming multiple excitation focal points on the sample surface and using a camera to detect the signals generated by these focal points in parallel. More importantly, in multifocal scanning, each focal local region in each frame of the camera image corresponds to a finite-range local sample response. Compared to a complete field-of-view image formed by continuous scanning from a single focal point, multifocal camera images inherently possess a "locally segmented" characteristic, meaning that complex and dense samples can be decomposed into multiple local response units surrounding a single focal point. Although each focal local region is still affected by neighboring structures and adjacent focal points, this influence is generally less than the cumulative mixing between pixels in a complete field-of-view image. Therefore, applying Gaussian-ring complementarity directly to each focal local region, rather than to the already stitched full-field image, is expected to significantly improve the applicability of this type of method to densely structured samples.

[0045] In view of this, embodiments of the present invention provide a super-resolution imaging method, the method comprising: acquiring multiple frames of Gaussian illumination images and multiple frames of annular illumination images, wherein the Gaussian illumination images and the annular illumination images are obtained by scanning a target object, both the Gaussian illumination images and the annular illumination images include multiple focal points, the positions of the multiple focal points in each frame of the multiple Gaussian illumination images are different, the positions of the multiple focal points in each frame of the multiple annular illumination images are different, and the focal point positions of the Gaussian illumination images corresponding to the same frame correspond to the focal point positions of the annular illumination images; determining a local annular illumination image corresponding to a local Gaussian illumination image for a local Gaussian illumination image corresponding to any focal point in any frame of the Gaussian illumination images; performing a pixel-by-pixel complementary operation on the local Gaussian illumination images and the local annular illumination images to obtain a weight matrix; obtaining multiple frames of super-resolution images based on the weight matrix and the local Gaussian illumination images, wherein the super-resolution images are of the same size as the Gaussian illumination images and the annular illumination images; and generating a global fusion image of the target object based on the multiple frames of super-resolution images.

[0046] This invention is applicable to single-photon fluorescence, two-photon fluorescence, multiphoton fluorescence, second harmonic fluorescence, third harmonic fluorescence, and other nonlinear optical microscopy imaging scenarios. It is especially suitable for samples with dense spatial structures such as tissue samples, organelle networks, collagen fibers, mitochondria, microtubules, and nerve fibers, where conventional full-field differential imaging is unstable.

[0047] According to an embodiment of the present invention, a super-resolution imaging method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0048] This embodiment provides a super-resolution imaging method that can be used in the aforementioned terminal. Figure 2 This is a flowchart of a super-resolution imaging method according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps: Step S201: Obtain multiple frames of Gaussian illumination images and multiple frames of ring illumination images.

[0049] Gaussian illumination images and ring illumination images are obtained by scanning the target object. Both Gaussian and ring illumination images include multiple focal points. The positions of multiple focal points differ between frames in a multi-frame Gaussian illumination image, and the positions of multiple focal points also differ between frames in a multi-frame ring illumination image. There is a correspondence between the focal points of the Gaussian illumination image and the focal points of the ring illumination image for the same frame.

[0050] The target object can be a biological sample, cell sample, tissue section, fluorescently labeled sample, micro / nano structure sample, material surface sample, or other object requiring microscopic imaging. The target object may include multiple fluorescence emission structures or scattering structures.

[0051] A Gaussian illumination image can be an image acquired after illuminating a target object using multiple Gaussian excitation focal points. A ring illumination image can be an image acquired after illuminating a target object using multiple ring excitation focal points or a hollow excitation focal point. Gaussian excitation focal points and ring excitation focal points can be generated by the light field control module 106, and the focal arrays formed by the two on the target object have a corresponding relationship.

[0052] In one feasible implementation, the Gaussian illumination image corresponding to the s-th scan position can be represented as:

[0053] in, Let represent the Gaussian illumination image corresponding to the s-th scan position; Gi(s) represents the local Gaussian response corresponding to the i-th focus in the s-th scan position; N represents the number of focuses in a single frame of Gaussian illumination image.

[0054] The annular illumination image corresponding to the s-th scan position can be represented as:

[0055] in, Ai(s) represents the ring illumination image corresponding to the s-th scan position; Ai(s) represents the local ring response corresponding to the i-th focus in the s-th scan position; N represents the number of focuses in a single frame of ring illumination image.

[0056] During the scanning process, the positions of multiple focal points in each frame of a multi-frame Gaussian illumination image can be different, as can the positions of multiple focal points in each frame of a multi-frame annular illumination image. By changing the position of the multifocal array in each frame, different spatial regions of the target object can be covered. Scanning methods can include row-by-row scanning, column-by-column scanning, raster scanning, spiral scanning, random scanning, sparse sampling scanning, or preset coded scanning.

[0057] According to an embodiment of the present invention, the i-th Gaussian focus in the s-th frame of the Gaussian illumination image and the i-th annular focus in the s-th frame of the annular illumination image correspond to the same scanning position or the same local region on the target object. This correspondence can be determined by optical path calibration, focus positioning, coordinate transformation, image registration, or preset scanning control parameters.

[0058] Step S202: For any Gaussian illumination image in any frame of Gaussian illumination image, determine the local ring illumination image corresponding to the local Gaussian illumination image.

[0059] According to embodiments of the present invention, any frame of Gaussian illumination image may include multiple Gaussian focal points, each Gaussian focal point may correspond to a local Gaussian illumination image. A local Gaussian illumination image may be a local image patch cropped from the Gaussian illumination image, centered at or referencing a Gaussian focal point. Similarly, any frame of annular illumination image may include multiple annular focal points, each annular focal point may correspond to a local annular illumination image.

[0060] If the two have a one-to-one correspondence through calibration, then... A local Gaussian illumination image extracted centered on, and compared with... The local ring illumination image extracted from the center is determined as the corresponding local image pair.

[0061] In some implementations, the local Gaussian illumination image and the local annular illumination image can be the same size, for example, both being local image patches of w×h pixels. The preset local image pixel size can be determined based on the focal spot size, point spread function width, focal spot spacing, imaging system magnification, detector pixel size, and sample structure density. For example, the pixel size of the local image patch can be set to cover the main lobe of a single focal spot and its surrounding response region, while minimizing the entry of adjacent focal spot responses into the current local image patch.

[0062] Step S203: Perform pixel-by-pixel complementary operation on the local Gaussian illumination image and the local ring illumination image to obtain the weight matrix.

[0063] Step S204: Based on the weight matrix and the local Gaussian illumination image, obtain multi-frame super-resolution images.

[0064] The super-resolution image is the same size as the Gaussian illumination image and the ring illumination image.

[0065] Because locally Gaussian illuminated images and locally annularly illuminated images have complementary spatial responses—locally Gaussian illuminated images have a stronger response at the focal center, while locally annularly illuminated images have a weaker response at the focal center and a stronger response in the surrounding areas—pixel-by-pixel complementary operations can highlight the effective signal near the focal center, weaken the response around the focal point or the out-of-focus background, thereby compressing the equivalent point spread function and improving resolution.

[0066] A local super-resolution image can be obtained based on the weight matrix and the local Gaussian illumination image, and then the super-resolution image can be obtained through pixel backfilling.

[0067] Pixel backfilling refers to the process of refilling the processed pixel values ​​into the corresponding spatial positions in the original or target image. In this embodiment of the invention, pixel backfilling means that each local super-resolution image is filled back into the corresponding pixel positions of the super-resolution image one by one according to its corresponding spatial coordinates (provided by the local Gaussian illumination image or the local ring illumination image), the scan position relative to the target object, and the pixel geometry, thereby forming a complete super-resolution image based on multiple local Gaussian illumination images of the current frame's Gaussian illumination image and multiple local ring illumination images corresponding to the corresponding ring illumination image.

[0068] Step S205: Generate a global fused image of the target object based on multi-frame super-resolution images.

[0069] In one implementation, multiple frames of super-resolution images can be mapped to a global coordinate system based on the scanning position, and operations such as averaging, weighted averaging, signal-to-noise ratio weighted fusion, and confidence-weighted fusion can be performed on pixels at the same global coordinates. In a feasible embodiment, the globally fused image can be represented as:

[0070] in, A global fused image of the target object; Represents global image coordinates; This represents the pixel value of the s-th frame super-resolution image after being mapped to global coordinates; This represents the fusion weight of the super-resolution image at this location in the s-th frame; S is the number of super-resolution image frames; ε is a stabilization term, which can be set to a minimum value, for example, 1×10. -6 .

[0071] The fusion weights can be determined based on factors such as focal center distance, local signal-to-noise ratio, local contrast, number of scans, light field intensity calibration results, or image quality evaluation results. If a pixel location is covered by multiple scan frames, the enhanced pixel value with a higher signal-to-noise ratio or closer to the focal center can be prioritized. If a pixel location is covered by only one scan frame, the enhanced pixel value of that scan frame can be directly used.

[0072] According to embodiments of the present invention, subpixel registration, drift correction, distortion correction, intensity equalization, or denoising processing can be performed on multiple super-resolution images before global fusion to improve the continuity and accuracy of the final globally fused image.

[0073] Optionally, after obtaining the globally fused image, mild denoising, deconvolution, background correction, frequency domain enhancement, or intensity normalization can be performed as needed to obtain the final image.

[0074] According to embodiments of the present invention, by acquiring a Gaussian illumination image with multiple focal points and an annular illumination image with multiple focal points, and performing pixel-complementary processing on the local Gaussian illumination image corresponding to any focal point in any frame of Gaussian illumination image and the local annular illumination image, the mutual influence between pixels is reduced, so that each local super-resolution image only reflects the signal response of the local area where the corresponding focal point is located, effectively avoiding the signal aliasing problem caused by the superposition of adjacent structural responses in the complete field of view image. By complementing each other at the pixel level through the image information of the same spatial position under the two illumination modes, the resolution and detail richness of the local image are effectively improved. At the same time, by backfilling the pixels of each local super-resolution image according to its corresponding focal position to obtain a multi-frame super-resolution image of the same size as the original image, and then fusing the multi-frame super-resolution images to generate a global fusion image, the signal of each pixel position in the global fusion image comes from the local real response at the corresponding focal point in each frame, solving the imaging distortion problem caused by signal superposition in the overall difference and ensuring the accuracy of the imaging results of dense tissue samples.

[0075] According to an embodiment of the present invention, before determining the local annular illumination image corresponding to the local Gaussian illumination image, the method further includes: using an image localization algorithm to determine the positions of multiple focal points in any frame of the Gaussian illumination image; using an image localization algorithm to determine the positions of multiple focal points in any frame of the annular illumination image; for any focal point in any frame of the Gaussian illumination image, extracting the local Gaussian illumination image from the Gaussian illumination image based on a preset local image pixel size and the position of the focal point; for any focal point in any frame of the annular illumination image, extracting the local annular illumination image from the annular illumination image based on a preset local image pixel size and the position of the focal point.

[0076] Image localization algorithms can be used to determine the positions of multiple focal points in each frame of an image. These algorithms can include peak localization, centroid localization, template matching, two-dimensional Gaussian fitting, circular template fitting, cross-correlation, or phase correlation methods.

[0077] In one feasible embodiment, the scanning position can also be determined according to a preset geometric lattice relationship, that is, the theoretical position of each focal point can be directly calculated according to the preset parameters of the scanner or the light field control module (such as a rectangular arrangement with a fixed spacing).

[0078] For the s-th frame of the Gaussian illumination image, we can extract... Centered local Gaussian illumination image Similarly, it is possible to extract using Centered local ring illumination image In a localized ring illumination image With local Gaussian illumination image In different coordinate systems, it is possible to... Perform subpixel translation, rotation, scaling, or affine correction to make it consistent with... They are in the same local coordinate system.

[0079] Here, X and Y can represent the local image coordinates with the focal center as the origin, which are determined by a preset local image pixel size. The preset local image pixel size can be determined based on the multifocal distance. To reduce crosstalk between adjacent focal points, the local image size can be smaller than the distance between adjacent focal points.

[0080] According to embodiments of the present invention, the positions of multiple focal points in Gaussian illumination images and ring illumination images are determined by an image localization algorithm, and local high-speed illumination images and local ring illumination images are further extracted. This decomposes the multifocal imaging problem in the complete field of view into a complementary enhancement problem of multiple local focal pairs, which can reduce the impact of the superposition of adjacent focal responses on super-resolution computation, thereby improving the realism and stability of local super-resolution images.

[0081] Local Gaussian illumination images and local ring illumination images may be affected by factors such as background fluorescence, dark current, uneven light intensity, detector response differences, focal energy differences, and local brightness differences in the sample during acquisition. Directly performing pixel complementation operations may amplify noise in low-signal areas or produce artifacts due to the inconsistency in intensity scale between Gaussian and ring illumination. Therefore, background subtraction, constraint, normalization, and local standardization should be performed first.

[0082] Local Gaussian illumination image normalization factor and local ring illumination image normalization factor It can correct for intensity differences between different focal points and intensity differences between different illumination modes. The excitation intensity of different focal points in a multifocal array may not be completely consistent. Without normalization, strong focal areas may be over-enhanced in the fused image, while weak focal areas may be underestimated. By normalizing, different local images can be brought to a uniform intensity scale.

[0083] The constraint function is used to limit the influence of local maxima caused by the nonlinear operation of division. For example, it can be a contrast stretching function. This invention does not impose restrictions on the constraint function. For more constraint functions, please refer to relevant super-resolution imaging technology papers, which will not be elaborated here.

[0084] According to an embodiment of the present invention, the above-described pixel-by-pixel complementary operation on the local Gaussian illumination image and the local ring illumination image to obtain a local super-resolution image includes: Perform pixel-by-pixel complementation on local Gaussian illumination images and local ring illumination images using any of the following formulas:

[0085]

[0086]

[0087]

[0088]

[0089] in, For local super-resolution images, This is the preprocessed local Gaussian illumination image. This is the preprocessed local ring illumination image. For fixed coefficients, It is a stable term.

[0090] Fixed coefficient The signal-to-noise ratio can be adaptively determined based on the local energy of the i-th focal spot, the depth of the dark spot at the center of the ring, the background level, or the crosstalk intensity of adjacent focal spots.

[0091] Pixel-wise complementarity can be performed using differential, ratio, normalized ratio, normalized differential, or logarithmic ratio methods (formulas shown above). Different complementarity methods can be selected based on sample type, signal-to-noise ratio, imaging mode, and enhancement target. Differential complementarity can utilize the strong peripheral response in ring illumination images to suppress the peripheral spread response in Gaussian illumination images, thereby compressing the equivalent point spread function. Normalized ratio complementarity can constrain the enhancement result within a relatively stable numerical range, reducing the impact of overall intensity variations on the imaging result. Normalized differential complementarity can simultaneously utilize the difference information and total intensity information of Gaussian and ring images, making it suitable for use in scenes with large light intensity variations. Logarithmic ratio complementarity can compress the dynamic range, resulting in a more balanced enhancement of strong and weak signal regions. Therefore, complementary operation forms suitable for different imaging conditions can be flexibly selected for calculation to obtain superior imaging results.

[0092] In one feasible embodiment of the present invention, to reduce potential negative values ​​and over-subtraction issues in the difference calculation, a ratio method or a normalized ratio method can be used for pixel-by-pixel complementary operations. Since the ring illumination is weaker at the focal center and stronger at the periphery, the aforementioned ratio can enhance the relative contribution at the focal center while suppressing the focal periphery and out-of-focus background. This method is particularly suitable for tissue samples with weak signals or strong backgrounds.

[0093] According to an embodiment of the present invention, a multi-frame super-resolution image is obtained based on a weight matrix and a local Gaussian illumination image, including: Local super-resolution images are obtained based on weight matrices and local Gaussian illumination images; Multiple super-resolution images are obtained based on all local super-resolution images; Local super-resolution images are obtained using the following formula:

[0094] in, For local super-resolution images, Here, s is the weight matrix, s is the scan position, x is the x-coordinate, and y is the y-coordinate. This is the background estimate of the weight matrix. The normalization factor for the local ring illumination image. For the stability term, F represents local standardization, R() is the constraint function, and G is the stability term. i (s,x,y) is a Gaussian illumination image.

[0095] In one feasible embodiment The background region can be determined based on local Gaussian illumination images and local ring illumination images. For example, the median pixel value of the edge region of the local Gaussian illumination image can be selected as the background estimate, or the low percentile pixel value can be selected as the background estimate to avoid the influence of strong signal regions of the sample on the background estimate.

[0096] According to an embodiment of the present invention, a super-resolution image is obtained by the following steps: when at least two local super-resolution images in any frame have overlapping regions, a weighting coefficient corresponding to the local super-resolution image is determined based on the signal-to-noise ratio of each local super-resolution image with overlapping regions; and the local super-resolution images with overlapping regions are weighted and fused using the weighting coefficients to obtain a super-resolution image.

[0097] Since a single frame image includes multiple focal points, the local super-resolution images corresponding to adjacent focal points may overlap during backfilling. Simply overlaying or directly adding them may lead to abrupt changes in brightness in the overlapping areas, boundary artifacts, or duplicate signal calculations. Therefore, weighted fusion can be performed based on weighted averaging, maximum value fusion, energy normalization fusion, window fusion based on focal center distance, or weighted fusion based on local signal-to-noise ratio.

[0098] Taking weighted fusion based on local signal-to-noise ratio as an example, the signal-to-noise ratio of the i-th local super-resolution image in the s-th frame can be expressed as:

[0099] in, Let be the signal-to-noise ratio of the i-th local super-resolution image in the s-th frame. This represents the average intensity of a local signal region. The standard deviation of noise in the local background region. It is a stable term.

[0100] For a pixel location (X', Y') with certain overlapping global coordinates, if this location is covered by multiple local super-resolution images, the weighting coefficient of the i-th local super-resolution image can be expressed as:

[0101] in, Let represent the weighting coefficients at pixel position (X', Y') of the i-th local super-resolution image. Let be the signal-to-noise ratio of the i-th local super-resolution image in the s-th frame. Let represent the set of local super-resolution image indices covering pixel positions (X', Y') in frame s. Let be the signal-to-noise ratio of the k-th local super-resolution image in the s-th frame.

[0102] Then the pixel values ​​of the super-resolution image in the overlapping region of the s-th frame can be expressed as:

[0103] in, This represents the pixel value at the global coordinate (X', Y') of the super-resolution image in frame s. This represents the weighting coefficient of the pixel position (X', Y') of the i-th local super-resolution image in global coordinates. This represents the local pixel value corresponding to the pixel position (X', Y') in the i-th local super-resolution image with respect to the global coordinates.

[0104] In one feasible implementation, the weighting coefficients can also be determined in conjunction with distance weights. For example, the closer a pixel location is to the focal center of a local super-resolution image, the greater the weight of that local super-resolution image; the closer a pixel location is to the edge of the local image, the smaller the weight. The distance weights can be normalized after being multiplied by the signal-to-noise ratio weights.

[0105] According to embodiments of the present invention, weighting coefficients are determined by the signal-to-noise ratio corresponding to each local super-resolution image, and then the local super-resolution images are weighted and fused. This can smooth the overlapping areas during the local super-resolution image backfilling process, avoid boundary abrupt changes and repeated superposition between local super-resolution images, and thus improve the continuity and uniformity of the super-resolution image.

[0106] According to an embodiment of the present invention, the above method is applied to an imaging system, which includes a light field modulation module. The light field modulation module is used to modulate the excitation beam to generate a Gaussian illumination beam and a ring illumination beam. The Gaussian illumination beam forms multiple Gaussian excitation focal points on the target object, and the ring illumination beam forms multiple ring excitation focal points on the target object.

[0107] According to embodiments of the present invention, the above-described light field modulation module has been described in detail in the imaging system to which the present invention is applied, and will not be repeated here.

[0108] In one feasible embodiment of the present invention, the excitation light is as follows: Figure 1 The imaging system shown propagates the light, forming a Gaussian focal array on the surface of the target object. The same excitation light, modulated by another phase map, forms a corresponding annular focal array on the sample surface. The system uses an sCMOS camera as the detector, recording one frame of multifocal Gaussian image and one frame of multifocal annular image at each scanning position.

[0109] For each scan position, each frame can be divided into multiple local images based on the geometric relationship of the focal array. For the i-th focal point, a local Gaussian illumination image and a local annular illumination image are extracted respectively. After background subtraction and intensity normalization are performed on both, local weighted difference is performed. Multiple pixels are backfilled to the global image coordinates according to the focal array position and fused to obtain a globally fused image.

[0110] Figure 3 This is a schematic diagram of imaging processing of a target object according to an embodiment of the present invention.

[0111] like Figure 3 As shown, the target object 301 is imaged through an objective lens to obtain a Gaussian illumination image 302 and an annular illumination image 303. Multiple Gaussian illumination images 302 are processed using the aforementioned related techniques to obtain a related image 305. The Gaussian illumination image 302 and the annular illumination image 303 are then processed by the super-resolution imaging method provided by this invention to obtain a super-resolution image 304. The multiple super-resolution images are then fused to obtain a globally fused image 306. It can be seen that the super-resolution imaging method proposed in this invention, compared to related techniques, can at least partially improve the imaging clarity and has a better effect.

[0112] This embodiment also provides a super-resolution imaging device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0113] This embodiment provides a super-resolution imaging device, such as... Figure 4 As shown, it includes: The acquisition module 401 is used to acquire multiple frames of Gaussian illumination images and multiple frames of ring illumination images. The Gaussian illumination images and the ring illumination images are obtained by scanning the target object. Both the Gaussian illumination images and the ring illumination images include multiple focal points. The positions of the multiple focal points in each frame of the multiple Gaussian illumination images are different. The positions of the multiple focal points in each frame of the multiple ring illumination images are also different. The focal point positions of the Gaussian illumination images corresponding to the same frame have a corresponding relationship with the focal point positions of the ring illumination images. The determining module 402 is used to determine a local ring illumination image corresponding to the local Gaussian illumination image corresponding to any focal point in any frame of Gaussian illumination image; The complementary module 403 is used to perform pixel-by-pixel complementary operations on the aforementioned local Gaussian illumination image and the aforementioned local ring illumination image to obtain a weight matrix. The super-resolution image generation module 404 is used to obtain multiple super-resolution images based on the weight matrix and the local Gaussian illumination image, wherein the super-resolution images are the same size as the Gaussian illumination image and the ring illumination image. The generation module 405 is used to generate a global fused image of the target object based on the above multi-frame super-resolution images.

[0114] The super-resolution imaging apparatus provided in this embodiment of the invention can execute the super-resolution imaging method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects for executing the method. Further functional descriptions of the various modules and units described above are the same as in the corresponding embodiments described above, and will not be repeated here.

[0115] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0116] The following is a detailed reference. Figure 5 The diagram illustrates a structural schematic suitable for implementing an electronic device according to embodiments of the present invention. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 502 or a program loaded from memory 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device. The processor 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0117] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0118] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a memory 508, or installed from a ROM 502. When the computer program is executed by the processor 501, it performs the functions defined in the super-resolution imaging method of the embodiments of the present invention.

[0119] Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0120] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the super-resolution imaging method shown in the above embodiments is implemented.

[0121] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0122] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A super-resolution imaging method, characterized in that, The method includes: Acquire multiple frames of Gaussian illumination images and multiple frames of annular illumination images, wherein the Gaussian illumination images and the annular illumination images are obtained by scanning the target object. Both the Gaussian illumination images and the annular illumination images include multiple focal points. The positions of the multiple focal points in each frame of the multiple Gaussian illumination images are different, and the positions of the multiple focal points in each frame of the multiple annular illumination images are different. The focal point positions of the Gaussian illumination images corresponding to the same frame have a corresponding relationship with the focal point positions of the annular illumination images. For any local Gaussian illumination image corresponding to any focal point in any frame of Gaussian illumination image, determine the local annular illumination image corresponding to the local Gaussian illumination image; And perform pixel-by-pixel complementary operations on the local Gaussian illumination image and the local annular illumination image to obtain a weight matrix; Based on the weight matrix and the local Gaussian illumination image, a multi-frame super-resolution image is obtained, wherein the super-resolution image is the same size as the Gaussian illumination image and the ring illumination image. A global fused image of the target object is generated based on the multi-frame super-resolution images.

2. The method according to claim 1, characterized in that, Before determining the local annular illumination image corresponding to the local Gaussian illumination image, the method further includes: Using an image localization algorithm, the positions of multiple focal points in any frame of Gaussian illumination image are determined; Using an image localization algorithm, the positions of multiple focal points in any frame of the ring illumination image are determined; For any focal point in any frame of Gaussian illumination image, the local Gaussian illumination image is extracted from the Gaussian illumination image based on a preset local image pixel size and the position of the focal point; For any focal point in any frame of ring illumination image, the local ring illumination image is extracted from the any frame of ring illumination image based on a preset local image pixel size and the position of the focal point.

3. The method according to claim 1, characterized in that, The step of performing pixel-by-pixel complementary operations on the local Gaussian illumination image and the local annular illumination image to obtain a weight matrix includes: Perform a pixel-by-pixel complementary operation on the local Gaussian illumination image and the local annular illumination image using any of the following formulas: in, This is the weight matrix. This is the preprocessed local Gaussian illumination image. This is the preprocessed local ring illumination image. For fixed coefficients, It is a stable term.

4. The method according to claim 1, characterized in that, The process of obtaining multiple super-resolution images based on the weight matrix and the local Gaussian illumination image includes: A local super-resolution image is obtained based on the weight matrix and the local Gaussian illumination image; The multi-frame super-resolution image is obtained based on all the local super-resolution images; The local super-resolution image is obtained using the following formula: in, For local super-resolution images, Here, s is the weight matrix, s is the scan position, x is the x-coordinate, and y is the y-coordinate. This is the background estimate of the weight matrix. The normalization factor for the local ring illumination image. For the stability term, F represents local standardization, R() is the constraint function, and G is the stability term. i (s,x,y) is a Gaussian illumination image.

5. The method according to claim 4, characterized in that, The super-resolution image is obtained through the following steps: In the case where at least two local super-resolution images have overlapping regions in any frame, the weighting coefficients corresponding to the local super-resolution images are determined based on the signal-to-noise ratio of each of the local super-resolution images with overlapping regions. Using the weighting coefficients, the local super-resolution images with overlapping regions are weighted and fused to obtain the super-resolution image.

6. The method according to claim 1, characterized in that, The method is applied to an imaging system, which includes a light field modulation module. The light field modulation module is used to modulate the excitation beam to generate a Gaussian illumination beam and a ring illumination beam. The Gaussian illumination beam forms multiple Gaussian excitation focal points on the target object, and the ring illumination beam forms multiple ring excitation focal points on the target object.

7. A super-resolution imaging device, characterized in that, The device includes: The acquisition module is used to acquire multiple frames of Gaussian illumination images and multiple frames of annular illumination images. The Gaussian illumination images and the annular illumination images are obtained by scanning the target object. Both the Gaussian illumination images and the annular illumination images include multiple focal points. The positions of the multiple focal points in each frame of the multiple Gaussian illumination images are different, and the positions of the multiple focal points in each frame of the multiple annular illumination images are different. The focal point positions of the Gaussian illumination images corresponding to the same frame have a corresponding relationship with the focal point positions of the annular illumination images. The determination module is used to determine a local ring illumination image corresponding to a local Gaussian illumination image corresponding to a local Gaussian illumination image for any focal point in any frame of Gaussian illumination image; The complementary module is used to perform pixel-by-pixel complementary operations on the local Gaussian illumination image and the local ring illumination image to obtain a weight matrix; A super-resolution image generation module is used to obtain multiple frames of super-resolution images based on the weight matrix and the local Gaussian illumination image, wherein the super-resolution images are the same size as the Gaussian illumination image and the ring illumination image; The generation module is used to generate a global fused image of the target object based on the multi-frame super-resolution images.

8. An electronic device, characterized in that, include: A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the super-resolution imaging method according to any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the super-resolution imaging method according to any one of claims 1 to 6.

10. A computer program product, characterized in that, Includes computer instructions for causing a computer to perform the super-resolution imaging method according to any one of claims 1 to 6.

Images