Super-resolution analysis system and method, and corresponding imaging device and model training method

Abstract

The invention belongs to the technical field of image processing and artificial intelligence, and discloses a super-resolution analysis system and method based on deep learning. The super-resolution analysis system comprises an analysis unit, the analysis unit comprises a super-resolution implementation model, the analysis unit can be executed by a processor and is used for constructing a super-resolution image based on an input wide-field image, and the super-resolution implementation model is a trained deep learning model. A training data set for training the super-resolution implementation model and an input wide-field image come from the same imaging module. The invention further discloses corresponding imaging equipment and a model training method. According to the method, only one wide-field image needs to be reconstructed, the super-resolution image based on the mapping relation can be output through the deep learning algorithm, the number of collected images is reduced, the resolution is improved, and additional time does not need to be increased.

Description

Super-resolution analysis system, method, corresponding imaging device and model training method Technical Field

[0001] The present application belongs to the field of image processing and artificial intelligence technology. More specifically, it provides a super-resolution analysis system and method based on deep learning, as well as corresponding imaging equipment and model training methods. Background Art

[0002] The cost of personal genome sequencing has hovered around $1,000 for five years and has not seen any significant decline. This is because the mainstream high-throughput gene sequencing technology (second-generation gene sequencing) is based on traditional optical microscopes. Its resolution is limited by the "optical diffraction limit," which limits the sample spacing to at most 500nm (objective lens NA = 1.0). This restricts the ability to further increase gene sequencing throughput in line with "More Than Moore's Law," making the development of new high-throughput gene sequencing technologies imperative.

[0003] Gene sequencing throughput depends primarily on resolution, objective lens field of view, and camera speed. The objective lens field of view and camera speed are limited by existing optical design and processing techniques and semiconductor technology, respectively, making breakthroughs difficult. Modern optical super-resolution technology can break through the "optical diffraction limit" by leveraging the interaction between light and matter to achieve super-resolution. Super-resolution involves improving the resolution of an original image through hardware or software methods. The process of obtaining a high-resolution image from a series of low-resolution images is known as super-resolution reconstruction.

[0004] Since sequencing throughput and reagent and consumable costs are inversely proportional to the square of sample spot density, how to leverage super-resolution technology to increase the sample spot density on sequencing chips, thereby driving sequencing costs down in line with "More Than Moore's Law," is a key issue urgently awaiting research in sequencing equipment R&D. Currently, one super-resolution technology for high-throughput sequencing exists: structured illumination super-resolution fluorescence microscopy. This technology can approximately double resolution simply by improving illumination and methods. However, the core of super-resolution technology—high-density stripe structured illumination and fast image reconstruction algorithms—requires further advancements to be applicable to high-throughput sequencing.

[0005] Summary of the Invention

[0006] The purpose of this application is to propose a resolution enhancement method based on deep learning, so as to acquire a wide-field image and obtain a corresponding super-resolution image, so as to achieve the purpose of improving spatial resolution without increasing image acquisition time. Its application on sequencers can improve sequencing throughput.

[0007] In a first aspect, the present application provides a super-resolution analysis system, comprising an analysis unit, wherein the analysis unit includes a super-resolution implementation model, and the analysis unit can be executed by a processor to construct a super-resolution image based on an input wide-field image, wherein the super-resolution implementation model is a trained deep learning model, and the training data set for training the super-resolution implementation model and the input wide-field image come from the same imaging module.

[0008] In one embodiment, the analysis unit comprises:

[0009] A receiving module, configured to receive the wide-field image from the imaging module;

[0010] A processing module is connected to the receiving module and is used to implement the super-resolution implementation model and construct a super-resolution image based on the wide-field image.

[0011] In a second aspect, the present application provides an imaging device, which includes an imaging module and the analysis system described in the first aspect of the present application, and the super-resolution implementation model is trained using a training data set from the imaging module.

[0012] In a third aspect, the present application provides a super-resolution analysis method, the analysis method comprising:

[0013] 1) Obtaining a wide-field image of the imaging module;

[0014] 2) constructing a super-resolution image based on the wide-field image through a super-resolution implementation model, wherein the super-resolution implementation model is a trained deep learning model, and a training data set for training the super-resolution implementation model comes from the imaging module.

[0015] In a fourth aspect, the present application provides a method for training a super-resolution implementation model, the method comprising:

[0016] 1) Obtain at least one set of wide-field images and super-resolution images of the imaging module to form a training dataset;

[0017] 2) completing feature parameter training of the deep learning model based on the training data set, and obtaining a super-resolution implementation model for the imaging module based on the trained feature parameters.

[0018] In one embodiment, the super-resolution image is obtained by:

[0019] 1) Using single-molecule localization technology, the fluorophore signals of the target sample are collected over time, and super-resolution images are created by integrating the fluorophore signals;

[0020] 2) Using structured illumination microscopy to construct phase and amplitude images of the target sample in a wide-field light source, and then creating a super-resolution image by integrating multiple phase and amplitude images; or

[0021] 3) Using stimulated emission depletion microscopy, two lasers are used to excite fluorescence into the target sample, one for exciting the fluorophore and the other for emitting a laser beam to deplete the emission from the same fluorophore, thereby forming an image of the local capture area. A super-resolution image is created by integrating the images of the local capture area.

[0022] This application only needs to reconstruct a wide-field image and output a super-resolution image based on the mapping relationship through a deep learning algorithm, reducing the number of image acquisitions and truly achieving both improved resolution and no additional time. This application is compatible with existing non-super-resolution methods. It only needs to import the trained model into the system. Under the condition of unified parameters and unified equipment types, the non-super-resolution system can also produce super-resolution effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] To more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. Those skilled in the art can also derive other drawings based on these drawings without inventive effort.

[0024] FIG1 shows a schematic diagram of a super-resolution analysis system according to one embodiment of the present application.

[0025] FIG2 shows a schematic diagram of an imaging system according to an embodiment of the present application.

[0026] FIG3 shows a traditional super-resolution algorithm in a sequencer and a deep learning algorithm model based on an embodiment of the present application.

[0027] FIG4 shows a super-resolution reconstruction algorithm model in one embodiment.

[0028] FIG5 shows a schematic diagram of the training process of a super-resolution implementation model according to an embodiment of the present application.

[0029] FIG6 shows a scheme for constructing a training data set according to one embodiment of the present application.

[0030] Figure 7 shows that the exemplary deep learning model used in the embodiments of the present application adopts a Transformer-based architecture.

[0031] Figure 8 shows that in an embodiment, a wide-field image (a, ordinary wide-field microscopy) is input into a trained deep learning model, and the corresponding super-resolution image (c) is output; b is a structured illumination microscopy super-resolution image for comparison.

[0032] FIG9 shows a comparison of imaging resolution performance of different methods under the same imaging conditions. DETAILED DESCRIPTION

[0033] In order to make the above and other features and advantages of the present application clearer, the present application is further described below in conjunction with the accompanying drawings. The accompanying drawings constitute a part of this application and are used to illustrate this application together with the embodiments of the present application. For the purpose of clarity and simplification, detailed specific descriptions of the known functions and structures of the devices described herein will be omitted when they may obscure the subject matter of the present application. It should be understood that the specific embodiments given herein are for the purpose of explaining to those skilled in the art and are merely exemplary and non-restrictive. For those of ordinary skill in the art, the specific meaning of the terms in this application can be understood according to the specific circumstances, unless otherwise clearly defined.

[0034] In the following description, many specific details are set forth to provide a thorough understanding of the present application. However, it will be apparent to one of ordinary skill in the art that these specific details do not need to be employed to practice the present application. In other cases, well-known steps or operations are not described in detail to avoid obscuring the present application.

[0035] The purpose of this application is to propose a resolution enhancement scheme based on deep learning, which uses wide-field images to obtain super-resolution images. Therefore, the scheme proposed in this application can be used to implement a deep learning-based super-resolution imaging gene detection method that only requires wide-field images (such as fluorescence microscopy images) and high spatiotemporal resolution, overcoming the problems that the resolution in gene detection imaging is limited by the diffraction limit, structured illumination super-resolution imaging requires large data volume acquisition, and the reconstruction speed is limited. The scheme of this application can significantly improve the data throughput of genetic detection without losing details and imaging field of view, while reducing the risk of light drift and phototoxic contamination of genetic samples by illumination light. Compared with the super-resolution reconstruction method based on structured illumination, consistent, fidelity, and reliable super-resolution reconstruction results can be obtained.

[0036] Without wishing to be bound by any theory, the present applicant has found that due to the differences in optical systems, noise, etc. of different imaging modules, the optical systems of different imaging modules have their own individual uniqueness. It is often difficult to obtain good results on another optical system by using the data of one optical system for learning, even for imaging systems of the same model produced in the same batch. This is especially true for the optical systems of imaging modules equipped on gene sequencing systems. The deep learning algorithm of the super-resolution imaging system proposed in this application is trained based on the image data provided by the same imaging module, and then reconstructed based on a specific imaging module. The reconstruction results are stable, fidelity, and credible. Through verification testing, as long as the hardware system affecting this imaging module is guaranteed to remain unchanged, a wide-field picture can be directly collected later, and a super-resolution image can be generated through this model. If the hardware system needs to be replaced, the deep learning algorithm of the super-resolution imaging system proposed in this application can be used again for training.

[0037] In the present application, the proposed super-resolution analysis system may include an analysis unit, the analysis unit including a super-resolution implementation model, the analysis unit being executable by a processor for constructing a super-resolution image based on an input widefield image, wherein the super-resolution implementation model is a trained deep learning model, and the training dataset for training the super-resolution implementation model and the input widefield image are from the same imaging module. FIG1 shows a schematic diagram of a super-resolution analysis system according to one embodiment of the present application. In FIG1 , the analysis unit receives a widefield image from an imaging module and constructs a super-resolution image based on the received widefield image. Optionally, as shown in FIG1 , the analysis unit may include a receiving module, a processor, and a memory, wherein the memory is loaded with a graphics module, i.e., a super-resolution reconstruction module. The processor and memory constitute a processing module. The receiving module is configured to receive the widefield image, and the processing module is configured to implement the super-resolution implementation model and construct a super-resolution image based on the widefield image. The receiving module communicates with the imaging module via a wired or wireless interface and receives the widefield image data generated by the imaging module. The processing module is connected to the receiving module and obtains the widefield image data from the receiving module for subsequent analysis. The processing module includes a CPU (or GPU) and memory, loaded with computer code for a deep learning model containing trained parameters. The CPU (or GPU) generates a corresponding super-resolution image based on the wide-field image obtained by the receiving module and the computer code and associated training parameters of the deep learning model.

[0038] In the present application, the super-resolution analysis system can be independent of the imaging module system, or can form an integrated imaging system together with the imaging module system. The independent super-resolution analysis system can be independent at the hardware system level, for example, the super-resolution analysis system is operated by an independent processor and preferably includes an independent memory. The independent super-resolution analysis system can communicate with the imaging module through a wired interface or a wireless interface to receive the wide-field image data generated by the imaging module. The wired interface includes but is not limited to a standard serial port (RS232), Ethernet and USB. The protocols used by the wireless interface include but are not limited to LoRa, NB-IOT, ZigBee, WiFi, Bluetooth and BLE. Optionally, the super-resolution analysis system sends a request to the imaging module system to request the other party to provide a wide-field image, and the imaging model system sends the wide-field image to the super-resolution analysis system. Similarly, the super-resolution analysis system can be in the same hardware system as the imaging module system. For example, the super-resolution analysis system is deployed in the imaging system as a graphics module, sharing a processor and memory with the imaging module. During the operation of the super-resolution analysis system, the wide-field image generated by the imaging module is called by computer instructions. In the present application, the wide-field image generated by the imaging module can be directly transmitted to the super-resolution analysis system; alternatively, the wide-field image generated by the imaging module can be stored in a storage device and transmitted to the super-resolution analysis system upon receiving a request or based on other triggering conditions.

[0039] In the present application, the imaging module can image different types of light. In one example, the imaging module can be a conventional optical microscope that utilizes natural light sources. In another example, the imaging module images light of specific wavelengths, such as a fluorescence microscope that uses lasers to excite fluorescence. For a fluorescence microscope, the imaging module can include different light sources. For example, the light source can include an ultra-high-pressure mercury lamp and a filter system. The high-pressure mercury lamp produces intense light, containing significant amounts of ultraviolet and blue-violet light, which is then passed through the filter system to produce monochromatic light. The light source can also include a natural light source, performing the same functions as a conventional optical microscope. The imaging module can include an excitation filter to filter a portion of the visible light in the light source, providing excitation light of a specific wavelength. The imaging module can include a blocking filter to transmit fluorescence within a corresponding wavelength range while blocking or absorbing the remaining excitation light. The imaging module can include a dichroic mirror to transmit long-wavelength light and reflect short-wavelength light. The imaging module can also include an aperture stop to determine the resolution and contrast of the microscope image. The imaging module may include a field diaphragm, which is used to control the size of the specimen illumination area and prevent light that is not necessary for image formation from entering the specimen. The imaging module may include an excitation light converter, also known as a mirror arm rotation stage. The rotation stage is a disc-shaped structure used to set the fluorescence excitation mode and the filter combination. The imaging principle of the fluorescence microscope: First, the light source used in the fluorescence microscope can emit strong ultraviolet rays, which pass through the excitation filter to filter out a portion of the visible light in the light source; then the ultraviolet light is focused on the sample through the condenser to induce the fluorescent substance on the sample to emit fluorescence; finally, the blocking filter after the objective lens prevents all ultraviolet rays from passing through, filters out a portion of the excess excitation light, and only allows the induced fluorescence to pass through, so that the fluorescence emitted by the stained part of the fluorescent substance can be observed.

[0040] FIG2 shows a schematic diagram of an exemplary imaging system according to the present application. As shown in the figure, the imaging module is a microscope system, including a laser, an optical path system, a camera and a structured lighting device. When imaging the sample, the laser emits laser light to illuminate the sample, and the sample generates fluorescence, which is captured by the camera through the optical path system to form a wide-field image. The super-resolution analysis system includes a processor (CPU, GPU) and a memory, and the memory is loaded with a graphics module (super-resolution reconstruction model). The super-resolution analysis system also includes a peripheral interface and a memory controller. The peripheral interface is used to connect to the microscope system and receive the wide-field image formed in the camera from the microscope system. The memory controller is used to control the loading and rewriting of programs and data in the memory. The optional structured lighting device is used to form structured lighting to generate a super-resolution image of the sample. The structured lighting device can be used in the model training stage and removed after the model training is completed. The microscope system can share a processor and memory with the analysis system.

[0041] In the present application, gene sequencing generally refers to analyzing the base sequence of a specific DNA fragment or RNA fragment, such as determining the arrangement of adenine (A), thymine (T), cytosine (C) and guanine (G). It will be appreciated by those skilled in the art that base sequences include, but are not limited to, A, C, T and G, and may also include, for example, rare bases or modified bases. Currently, fluorescent labeling is commonly used for gene sequencing. The role of the fluorescence imaging module in a gene sequencer is to use a laser to excite a fluorescent marker on a gene sequencing chip to produce fluorescence, and to collect fluorescence signals. The four bases combine with different fluorescent markers to produce four different fluorescence bands, thereby identifying the bases. Therefore, the fluorescence imaging module is part of a gene sequencer for identifying and imaging gene bases, which is suitable for high-throughput and small-scale gene sequencers. Figure 3 shows an example of the application of the model and method of the present application in sequencing, showing a traditional super-resolution algorithm and a deep learning algorithm model based on the present application. Compared with the existing methods, each of the original base (including but not limited to A, C, T and G) channels needs to collect 6 original images, and a super-resolution image is obtained by reconstruction, which is input into the base recognition Basecall algorithm to complete the sequencing. Based on the method of the present application, only one wide-field image needs to be collected, and a super-resolution image can be directly output through the constructed deep learning model, and then input into the base recognition Basecall algorithm to obtain the sequencing result. This only adds one step from the wide-field image to the Basecall process relative to the non-super-resolution method, that is, the wide-field image is output as a new image with higher resolution through the deep learning model, and then input into the Basecall algorithm. In comparison, it does not add too much extra time. In addition, since the key to the method of the present application is to generate corresponding models for wide-field images and super-resolution images, the final embodiment is basically the same as that of the non-super-resolution system, and both input a wide-field image to complete the subsequent base recognition. Therefore, the real need for super-resolution image original image collection and reconstruction only occurs during the data training process. Therefore, the solution of the present application is fully compatible with existing sequencer systems. It only needs to import the trained model and parameters into the system before leaving the factory, and the super-resolution effect can also be produced using a non-super-resolution system.

[0042] In the biochemical process of gene sequencing, a sequence of a certain length is first "seeded" onto the surface of a sequencing chip through surface chemical treatment techniques. A small area that can be seeded is called a site, and the distance between adjacent sites is called the pitch. For non-super-resolution methods, this pitch is generally greater than the resolution limit of the optical system (λ / (2NA), where λ is the wavelength of light and NA is the numerical aperture of the objective lens. For example, with λ = 684.3 nm and NA = 0.8, the resolution limit is 427.7 nm. In other words, based on this set of optical parameters, the pitch should be greater than 427.5 nm, and readings of smaller pitches are impossible). Therefore, a sequencing chip can have 1 billion such sites. However, when capturing images, the entire chip is not covered at once. Instead, the image is divided into M×N small areas of equal size. Each time, a uniform light source is projected onto this area, and the camera is exposed for a specific time to produce a sequence image of that area. This is the so-called widefield image. After taking the photo, the fluorescent groups in the existing sequence can be removed through a biochemical reaction, and the removed parts can be washed away. Then a new round of biochemical reaction → photo taking → removal process can be carried out.

[0043] In this application, a widefield image refers to an image obtained by an observer or a camera after exposing the target specimen to a light source. A widefield image is an image obtained by an optically limited system without exceeding the optical diffraction limit, and can be captured by a camera and a microscope in combination with uniform illumination or unstructured illumination, for example, by widefield microscopy. Widefield microscopy is a very commonly used microscopic imaging method. Almost every laboratory of a research unit related to the biological environment has a microscope with widefield imaging. Its imaging is convenient and its principle is simple. However, the resolution of widefield images is not high and the background interference is serious. Widefield imaging can be a fluorescence image, in which the fluorescence of the sample is triggered by the outside world, and the camera and laser are controlled to turn on at the same time to obtain the corresponding light signal. Widefield images can be recorded using CMOS or CCD. For structured illumination microscopy (SIM), widefield images can also be obtained by superimposing 6 original images.

[0044] In the present application, super-resolution images can be obtained by super-resolution microscopy. Super-resolution microscopy refers to a microscope whose resolution breaks the resolution limit of an optical microscope. It is a technology that can image outside the resolution limit of about 200nm. Laser confocal scanning microscopy can obtain super-resolution images. This technology improves the optical resolution and contrast of microscopic images by using a spatial pinhole to block defocused light. During the image formation process, the laser confocal scanning microscope can capture multiple two-dimensional images at different depths in the sample to reconstruct the three-dimensional structure (this process is called optical sectioning). Many super-resolution technologies have been developed in recent years, including structured illumination microscopy (SIM). In structured illumination microscopy, phase and amplitude patterns are constructed in a wide-field light source. The target sample being imaged will also produce similar patterns, and the interference between the constructed pattern and the sample-based pattern is used to construct the intracellular structure. The implementation of structured illumination microscopy super-resolution technology is divided into the following three types according to the core device that produces structured illumination: mechanical grating (Grating-SIM), spatial light modulator (SLM-SIM) and digital micromirror device (DMD-SIM). Among them, Grating-SIM and SLM-SIM both use collimated parallel light projected onto the periodic structure to generate ±1-level light to form interference fringes on the focal plane as structured illumination. The difference is that Grating-SIM requires a rotating device and a piezoelectric displacement stage to realize the switching of stripe projection and phase shift in the X / Y directions respectively, while SLM-SIM realizes the direction switching and phase shift by controlling the rotation of the loaded liquid crystal molecules. The switching speed of the two is relatively slow, both of which are 100ms in two levels. DMD-SIM also uses electrical control to realize stripe projection and phase shift in the X / Y directions, but it uses the control of the deflection state of each micro-mirror to realize the "bright" and "dark" states, and the deflection state is determined by whether the voltage is loaded, so the switching speed can be very fast. But no matter which method is used, it is necessary to collect two directions × three sets of phases in each direction, a total of 6 original images (X direction: 0, 2π / 3, 4π / 3; Y direction: 0, 2π / 3, 4π / 3). A super-resolution image can be constructed using the method shown in FIG4, which shows a schematic diagram of a super-resolution reconstruction algorithm model using 6 original images. As shown in the figure, 6 original images (X direction: 0, 2π / 3, 4π / 3; Y direction: 0, 2π / 3, 4π / 3) raw data is used to create a super-resolution image through processes such as Fourier transform, spectrum decoupling, spectrum shifting, spectrum fusion, and inverse Fourier transform. In the frequency domain, the six original images of structured illumination carry the high-frequency and low-frequency components of the sample. The high-frequency and low-frequency components are separated by spectrum decoupling, and then the high-frequency and low-frequency components are moved to the correct position. Finally, spectrum fusion is used to restore the frequency components of the sample, achieving spectrum expansion. Wide-field images generally only contain low-frequency components. The classic super-resolution reconstruction algorithm requires the acquisition of at least three original fringe images in the X / Y direction (X direction: 0, 2π / 3, 4π / 3; Y direction: 0, 2π / 3, 4π / 3) in order to achieve super-resolution reconstruction in the X / Y dimensions. Therefore, it is a typical technology that sacrifices time for space, which is undoubtedly not cost-effective for sequencing applications. Therefore, it is particularly important to develop a technology that can improve spatial resolution without increasing image acquisition time. In this application, deep learning can extract multi-level features of images, and at the same time use a graphics processing unit (GPU) to quickly implement the fitting process from input image to output image mapping, so that deep learning has been widely used in image classification, semantic segmentation, detection and other aspects. In the field of SIM super-resolution imaging, reconstruction is often used to reconstruct super-resolution images from multiple real-time stripe structure illumination images (such as three stripe structure illumination images in the X direction, or a total of 6 stripe structure illumination images in the X / Y directions). At present, the at least number of original images required based on this method is three, and no method or example of super-resolution reconstruction based on a smaller number of images has been found.

[0045] Super-resolution technologies that have emerged in recent years also include single-molecule localization and stimulated emission depletion (STED) microscopy. Single-molecule localization uses specific fluorescent molecular probes to label target samples. By changing the molecules' external environment, the molecules' optical switching properties are effectively controlled. This technique temporally separates spatially overlapping multi-molecule fluorescence images into a series of sub-images, ensuring that only a small number of sparsely distributed single molecules emit fluorescence in each sub-image frame. This means that only one fluorescent molecule is excited within each diffraction-limited range. Thousands of frames of randomly distributed fluorescence signals are collected, and the center position of each molecule is precisely located using a single-molecule localization algorithm. Finally, all the obtained localization points are superimposed to reconstruct a super-resolution image that breaks the diffraction limit. Stimulated emission depletion (STED) microscopy is a confocal technique that uses two laser beams to simultaneously illuminate the target sample. One laser beam is used to excite fluorescent molecules, causing them to be in an excited state within the Airy disk at the focal point of the objective lens. At the same time, another annular depletion laser beam with zero central intensity is superimposed on it, causing the fluorescent molecules in the excited state at the edge of the Airy disk at the focal point of the objective lens to return to the ground state through the stimulated emission depletion process without spontaneously emitting fluorescence. Therefore, only the fluorescent molecules in the central area can spontaneously radiate fluorescence, thereby obtaining fluorescence luminescence points beyond the diffraction limit.

[0046] The present application found that the corresponding dataset model of ordinary wide-field image and super-resolution image can be established by using the deep learning model SwinIR, which is mainly based on the Transformer architecture. Therefore, the present application realized the use of the deep learning model to compare and map the 6 original images of structured illumination (X direction: 0, 2π / 3, 4π / 3; Y direction: 0, 2π / 3, 4π / 3) and the reconstructed super-resolution image. Through the deep learning model, a correspondence is established between the wide-field image (low resolution) obtained by adding the six original images and the reconstructed super-resolution image (high resolution). In this way, in subsequent applications, only one wide-field image needs to be reconstructed, and the deep learning model can output a super-resolution image based on this mapping relationship, reducing the number of image acquisitions and truly achieving both improved resolution and no additional time.

[0047] Therefore, the present application also provides a training system for a super-resolution implementation model, including a training unit, which can be executed by a processor and is used to: receive at least one set of wide-field images and super-resolution images of an imaging module, the at least one set of wide-field images and super-resolution images constituting a training data set, and complete feature parameter training of a deep learning model based on the training data set; and determine a super-resolution implementation model for the imaging module based on the trained feature parameters. The training unit includes: a receiving module, which is used to receive the at least one set of wide-field images and super-resolution images, the at least one set of wide-field images and super-resolution images constituting a training data set; and a processing module, connected to the receiving module, which is used to: complete feature parameter training of the deep learning model based on the training data set. The super-resolution image is obtained by: 1) using single-molecule localization technology to collect the fluorophore signals of the target sample over time, and integrating a large number of fluorophore signals to create a final super-resolution image; 2) using structured illumination microscopy technology to construct phase and amplitude images of the target sample in a wide-field light source, and integrating multiple phase and amplitude original images to create a super-resolution image; or 3) using stimulated emission depletion microscopy technology to excite fluorescence to the target sample using two lasers, one for exciting the fluorophore and the other for emitting a laser beam to deplete the emission from the same fluorophore, reducing the diffraction area of ​​the fluorescent light spot and significantly improving the resolution of the microscope, thereby forming an image of the local capture area, and creating a super-resolution image by integrating the images of the local capture area. The imaging system shown in Figure 2 can be used as a training system for the super-resolution implementation model to complete the training of the super-resolution implementation model.

[0048] Figure 5 illustrates a schematic diagram of the training process for a super-resolution model according to one embodiment of the present application. Referring to Figures 2 and 5 , a microscope system acquires a widefield image and loads a structured illumination device to acquire six corresponding structured illumination microscopy images. The six structured illumination microscopy images are super-resolved to obtain a widefield image and a corresponding super-resolution image dataset, which serves as a training dataset. This training dataset is then used for offline training of a deep learning model. Those skilled in the art will appreciate that super-resolution images can be obtained using any other technique, such as the single-molecule localization technique or stimulated emission depletion microscopy mentioned above. A dataset formed by super-resolution images and corresponding widefield images obtained using any technique can serve as a training dataset. The reconstruction quality of the deep learning model is tested. If the test passes, the deep learning model and its associated learning parameters are saved. If the test fails, data analysis is performed, and super-resolution reconstruction and analysis of the structured illumination microscopy images are performed, followed by retraining. During the use of the deep learning model, the microscope system turns off the structured illumination device, loads the deep learning model, and performs super-resolution imaging by acquiring widefield images. In one example, for a specific model with fixed hardware parameters, the following process is employed to achieve super-resolution. Those skilled in the art will understand that the following examples use structured illumination microscopy technology to obtain super-resolution images, and other super-resolution image acquisition methods, such as single-molecule localization technology or stimulated emission depletion microscopy technology, can be used instead of structured illumination microscopy technology to obtain super-resolution images.

[0049] 1) Training Phase: First, the optical path of the structured illumination module is coupled between the light source and objective lens of the original non-super-resolution system. This adds a structured illumination module to the original system, while maintaining compatibility with other components that determine the optical system. This module then generates six types of structured light in two directions, each with three phase-shifted groups. These are projected sequentially onto the same region during the same reaction cycle. This replaces the original biochemical reaction → single image → excision image cycle with six separate images, resulting in six original images, each corresponding to a different type of structured illumination. These six original images are summed to create a widefield image (equivalent to a single image captured in the non-super-resolution method), which is then reconstructed using a standard algorithm to create a super-resolution image. This process is repeated, collecting data from multiple biochemical reaction cycles to form a rich dataset, which serves as the training and test sets for the model.

[0050] 2) Testing phase: A wide-field image from the test set is selected, and the corresponding super-resolution image is obtained through the trained model. This image is then compared with the super-resolution image reconstructed using the standard algorithm. Model training is considered complete when the similarity reaches a certain level.

[0051] 3) In actual use, this super-resolution module is first removed from the sequencing system. Since the trained model and the optical system have formed a one-to-one correspondence, a single wide-field image can be input into the super-resolution image output. The corresponding biochemical process returns to the process of biochemical reaction → single photo → excision, but the low-resolution wide-field image obtained by this photo can be converted into a high-resolution image through the trained model, thus achieving a super-resolution effect.

[0052] Figure 6 illustrates a schematic diagram of a method for constructing a training dataset according to one embodiment of the present application. As shown, a structured light field is generated in two different spatial directions with an angle θ = 90° between adjacent directions. In each spatial direction, three structured light fields with different phase shifts are generated with adjacent shifts of π / 2 or 2π / 3. This generates a total of six structured light fields at different locations, which are used to illuminate and excite the sample to produce a fluorescence signal. Therefore, in this application, structured illumination refers to non-uniform illumination with a specific structure. In this application, six fluorescence images obtained with structured illumination can be recorded using a CMOS or CCD. In this application, low-resolution and high-resolution data pairs are data pairs consisting of a super-resolution image and a widefield image. The specific generation method is as follows: the widefield image is used as the low-resolution image, and the six original structured illumination images are reconstructed using the OpenSim algorithm as the high-resolution image. The low-resolution and high-resolution images constitute the dataset. The data includes images excited by red and green light. For example, the low-resolution image size is 48*48, and the high-resolution image size is 96*96. Multiple data pairs form the dataset, which serves as the input for the deep learning model. The dataset size is typically greater than 200,000 pairs.

[0053] Figure 7 shows that the exemplary deep learning model used in the embodiment of the present application adopts a Transformer-based architecture. As shown in Figure 7, the deep learning model includes three parts: shallow feature extraction, deep feature extraction and upsampling reconstruction, which are respectively implemented by the shallow feature extraction module, the deep feature extraction module and the reconstruction module. The first convolution layer is shallow feature extraction, the middle deep feature extraction is the deep feature extraction module, and the rest is upsampling reconstruction. This deep model adopts a [4,2,2,2] structure, that is, there are 4 RSTB blocks, the first RSTB has 4 STLs, and the other three RSTBs are composed of 2 STLs. The window size is 8. Shallow feature extraction mainly includes low-frequency components, and the deep feature extraction module is used to restore high-frequency components. Through the jump link of the shallow feature extraction layer, the low-frequency components and high-frequency components are input into the reconstruction module composed of the convolution layer and the Pixel Shuffle layer to reconstruct a high-resolution image. Shallow feature extraction is extracted by a 3*3 convolution layer. The deep feature extraction module consists of multiple residual Swin Transformer modules (RSTB) and a 3*3 convolutional layer. Each RSTB consists of L Swin Transformer layers (STL). Let the result of the j-th STL in the i-th RSTB be Fi,j, and the j-th STL is denoted as HSwin i,j(), so Fi,j = HSwin i,j(Fi,j-1). The last layer of each RSTB is a convolutional layer, denoted as HCONVi(), and then the residual connection is performed, so Fi,out = HCONVi(Fi,L)+Fi,0.

[0054] Through this process, a corresponding dataset model for ordinary wide-field images and super-resolution images was established. Through verification testing, as long as the hardware system affecting this model remains unchanged, a wide-field image can be directly collected in the future and a super-resolution image can be generated directly through this model.

[0055] The method of this application is compatible with existing non-super-resolution methods. It only needs to import the trained model into the system before leaving the factory, and the non-super-resolution system can also produce super-resolution effects. The traditional method requires the acquisition of 6 stripe structured illumination images, while this method only requires the acquisition of one spacious image, which increases the time utilization by 5 / 6. This application can be applied to any super-resolution imaging field. The method of this application is not limited to the fields of fluorescence imaging and gene sequencing, but is applicable to all systems that achieve super-resolution effects based on structured illumination.

[0056] In one embodiment of the present application, the method of the present application is used to utilize a set of 6 super-resolution images, which are first reconstructed using a traditional super-resolution reconstruction algorithm to form a deep learning model for this system. Then, a wide-field image is input to directly obtain the corresponding super-resolution image. The deep learning model based on the Transformer architecture is shown in Figure 7. The data collection method is as follows: a wide-field image and a corresponding super-resolution image obtained by structured illumination microscopy are obtained (the size, width, and height are twice that of the wide-field image respectively), and the image is divided into 200,000 sets of low-resolution and high-resolution sub-data pairs, with the low-resolution and high-resolution image sizes being 48*48 and 96*96 respectively. The training data and test data in this example are grayscale images of fluorescence excitation of DNA nanospheres (DNBs) on the chip with a spacing of 360nm collected by a CMOS camera. The fluorescence excitation wavelengths include 670nm-700nm and 550nm-580nm. The numerical aperture NA of the microscope is 0.8. The specific training process is as follows: use the Adam optimization algorithm to train on a training set containing 200,000 pairs of low-resolution and high-resolution images, traversing the training set at least 100 times; the mini-batch size N of each optimization traversal of the network is 32, and the initial learning rate is 0.0002; the loss function used is L1 norm loss and L2 regularization, with weights of 1 and 1e-5 respectively; in order to make the error converge to a smaller value, a learning rate decay strategy is used to reduce the learning rate when the training error is stable. This training process is implemented using TRX 3090; the prediction process is to directly reconstruct the wide-field image based on the trained model.

[0057] Figure 8 exemplifies the input of a wide-field image (a, ordinary wide-field microscopy) into the trained deep learning model and the output of the corresponding super-resolution image (c); wherein b is a structured illumination microscopy super-resolution image for comparison. Figure 9 exemplifies the comparison of imaging resolution performance of different methods under the same imaging conditions. As shown in Figure 8, the imaging results are given as an example. Compared with ordinary wide-field microscopy, the method of the present application can effectively improve the imaging resolution, and the imaging quality is comparable to that of the traditional method. As shown in Figure 9, a certain area is randomly selected and its grayscale distribution curve is drawn. It can be seen that the resolution of the method of the present application is comparable to that of the traditional method, and the two DNBs can be clearly separated.

[0058] As shown in the reference table, the structural similarity (SSIM) and correlation coefficient (CC) are used to measure the imaging quality of the method of the present application. Their index ranges are both 0 to 1, and a high value indicates a high similarity. Specifically, in this example, the SSIM value between the imaging results of the method of the present application and the traditional method is 0.892, and the CC value is 0.904, which quantitatively indicates a high degree of consistency between the imaging results of the two. The peak signal-to-noise ratio (PSNR) is also used to evaluate the performance of the imaging results. Specifically, in this example, the PSNR between the method of the present application and the traditional method is 30.13dB, which once again proves the effectiveness of the method of the present application.

[0059]

[0060] The various technical features described above can be combined arbitrarily. Although not all possible combinations of these technical features are described, any combination of these technical features should be considered to be covered by this specification as long as such combination does not conflict.

[0061] Those skilled in the art will appreciate that the embodiments of the present application can be provided as a system or a computer program product. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete module embodiment, or an embodiment in combination with module and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0062] The present application is described with reference to the flow chart and / or block diagram of the method, device (system), and computer program product according to the embodiment of the present application. It should be understood that each flow process and / or box in the flow chart and / or block diagram and the combination of the flow process and / or box in the flow chart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processing machine or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one flow chart flow or multiple flows and / or one box or multiple boxes of the block diagram.

[0063] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0064] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0065] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0066] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0067] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0068] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0069] Although the present application has been described in conjunction with the embodiments, it should be understood by those skilled in the art that the above description and the accompanying drawings are merely exemplary and non-restrictive, and the present application is not limited to the disclosed embodiments. Various modifications and variations are possible without departing from the spirit of the present application.

Claims

1. A super-resolution analysis system, comprising an analysis unit, wherein the analysis unit includes a super-resolution implementation model, and the analysis unit can be executed by a processor to construct a super-resolution image based on an input wide-field image, wherein the super-resolution implementation model is a trained deep learning model, and a training dataset for training the super-resolution implementation model and the input wide-field image are from the same imaging module.

2. The analysis system according to claim 1, wherein the analysis unit comprises: A receiving module, configured to receive the wide-field image from the imaging module; A processing module is connected to the receiving module and is used to implement the super-resolution implementation model and construct a super-resolution image based on the wide-field image.

3. The analysis system according to claim 2, wherein the processing module comprises a processor and a memory, wherein the memory is loaded with code for implementing the super-resolution implementation model.

4. The analysis system according to claim 2, wherein the receiving module communicates with the imaging module via a wired interface or a wireless interface.

5. According to the analysis system according to any one of claims 1 to 4, the deep learning model adopts a Transformer architecture, including a shallow feature extraction module, a deep feature extraction module and a reconstruction module, the shallow feature extraction module is used to extract low-frequency components of the image, the deep feature extraction module is used to restore high-frequency components of the image, and the reconstruction module is used to reconstruct a high-resolution image.

6. An imaging device, comprising an imaging module and the analysis system according to any one of claims 1 to 5, wherein the super-resolution implementation model is trained using a training data set from the imaging module. 7 . The imaging device according to claim 6 , wherein the imaging module comprises a fluorescence microscope system, which emits laser light to illuminate a sample, causing the sample to generate fluorescence, so as to capture a wide-field image. 8 . The imaging device according to claim 7 , wherein the imaging device is a sequencer, and the training dataset comprises wide-field images and super-resolution images generated for different bases.

9. A super-resolution analysis method, comprising: 1) Obtaining a wide-field image of the imaging module; 2) constructing a super-resolution image based on the wide-field image through a super-resolution implementation model, wherein the super-resolution implementation model is a trained deep learning model, and a training data set for training the super-resolution implementation model comes from the imaging module.

10. According to the method of claim 9, the deep learning model adopts a Transformer architecture, including a shallow feature extraction module, a deep feature extraction module and a reconstruction module, the shallow feature extraction module is used to extract low-frequency components of the image, the deep feature extraction module is used to restore high-frequency components of the image, and the reconstruction module is used to reconstruct a high-resolution image.

11. A method for training a super-resolution implementation model, the method comprising: 1) Obtain at least one set of wide-field images and super-resolution images of the imaging module to form a training dataset; 2) completing feature parameter training of the deep learning model based on the training data set, and obtaining a super-resolution implementation model for the imaging module based on the trained feature parameters.

12. The method according to claim 11, wherein the super-resolution image is obtained by: 1) using single-molecule localization technology to collect fluorophore signals of the target sample over time, and establishing the super-resolution image by integrating the fluorophore signals; 2) using structured illumination microscopy to construct phase and amplitude images of a target sample in a wide-field light source, and creating the super-resolution image by integrating multiple phase and amplitude images; or 3) Using stimulated emission depletion microscopy, two lasers are used to excite fluorescence to the target sample, one for exciting the fluorophore and the other for emitting a laser beam to deplete the emission from the same fluorophore, thereby forming an image of the local capture area, and the super-resolution image is established by integrating the images of the local capture area.

13. The method of claim 12, wherein for structured illumination microscopy, a super-resolution image is created by integrating six phase and amplitude images.

14. The method according to claim 13, wherein the phases of the six phase and amplitude images are in the X direction: 0, 2π / 3, 4π / 3; Y direction: 0, 2π / 3, 4π / 3.

Images