Super-resolution processing method, super-resolution processing device, and program

The method uses GANs and virtual reality simulations to convert low-resolution AFM images into high-resolution images, addressing the challenges of cost and skill requirements in AFM operation and noise limitations, enabling efficient high-resolution image generation.

JP7583475B1Active Publication Date: 2024-11-14MOLECULAR ROBOT RES INST CO LTD

Patent Information

Application Number
JP2023198727
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2023-11-23
Publication Date
2024-11-14
Estimated Expiration
2043-11-23

AI Technical Summary

Technical Problem

Existing super-resolution processing methods for scanning probe microscopes, such as AFM, face limitations in generating high-resolution images due to noise and the need for multiple low-resolution images, which are costly and require skilled operation, making it difficult to obtain sufficient training data for deep learning.

Method used

A method involving neural networks, specifically generative adversarial networks (GANs), uses simulation images created on a virtual reality system based on error-free X-ray crystal structure data to convert low-resolution AFM images into high-resolution images by learning from virtual and real data pairs, adjusting the images to achieve almost one-to-one correspondence.

Benefits of technology

This approach allows for generating high-resolution images from a single low-resolution AFM image, reducing the need for expensive consumables and skilled operation, while effectively overcoming noise limitations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007583475000001_ABST
    Figure 0007583475000001_ABST
Patent Text Reader

Abstract

Provided is a method, device, and program for super-resolution processing that converts low-resolution images captured using a microscope such as an atomic force microscope (AFM) into high-resolution images. [Solution] The method comprises the steps of acquiring a low-resolution real image, performing a simulation under different conditions to create a simulated image, and acquiring a high-resolution image that is clearer than the real image by learning a neural network (NN) using the real image and the simulated image as training data.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a method, device, and program for super-resolution processing that converts low-resolution images captured using a scanning probe microscope into high-resolution images by combining neural networks and virtual reality. [Background technology]

[0002] Scanning probe microscopes (hereinafter referred to as "SPM"), including atomic force microscopes (hereinafter referred to as "AFM"), are devices that measure the surface shape of minute objects such as DNA and cancer cells. AFM and SPM are existing technologies, and there are many related patent applications, including Patent Document 1, so a detailed explanation will be omitted. When using an SPM to capture images of tiny objects such as DNA samples, depending on the diameter of the SPM probe and the scanning resolution, only a blurry image like the one shown in Figure 1 can be obtained, making it impossible to observe details such as major and minor grooves. Therefore, attempts are being made to capture multiple images using an AFM and convert these into high-resolution images using super-resolution processing (a method of generating an image that is clearer than the original image).

[0003] Meanwhile, super-resolution processing is not limited to SPM images, and many patent applications have been filed for it as a general image processing technique (Patent Document 2, Patent Document 3, etc.). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] WO2017 / 145381 publication [Patent Document 2] WO2020 / 175446 publication [Patent Document 3] Patent Publication No. 2021-170284 Summary of the Invention [Problem to be solved by the invention]

[0005] There is a method for super-resolution processing that uses deep learning to learn from multiple images, and not only has a patent application been filed, but papers on this subject have already been published. However, these studies are biased toward the super-resolution processing method of superimposing a large amount of microscopic image data, and have a fundamental limitation in that they cannot completely remove noise from the image data. This is like applying the "law of large numbers" in statistics, and uses the principle that increasing the number of trials will bring you closer to the average value, but it is not possible to remove errors that are close to normal distribution. Figure 2 shows how multiple images I1 to I3 are superimposed to obtain an average image J. Although images I1 to I3 each contain noise, by superimposing them, image J is obtained in which the noise is not noticeable. However, in principle, it is impossible to completely remove errors with this method.

[0006] Furthermore, the probe, an essential part of AFM and other SPMs, is an expensive consumable item, and there is also the problem that SPM operation requires skill. There is a unique operation method, such as scanning the DNA surface underwater, and unless you are proficient in the operation, it is not easy to even obtain something that resembles an image. Since it is not easy to even obtain a blurry, low-resolution image, it is not realistic to obtain the multiple training data (also synonymous with "learning data" or "data for learning") required for deep learning.

[0007] In view of such problems, an object of the present invention is to obtain an error-free high-resolution image even if there is only one low-resolution SPM image. [Means for solving the problem]

[0008] In order to solve the above problems, the image processing method of the present invention comprises the steps of: 1. A method for performing super-resolution processing, comprising: obtaining a low-resolution real image taken by a microscope; performing a simulation under different conditions to generate a simulation image; and acquiring a high-resolution image that is clearer than the real image by learning a neural network (hereinafter, "NN") using the real image and the simulation image as training data.

[0009] In the image processing method of the present invention, the real image is An image of DNA obtained from a scanning probe microscope (hereinafter "SPM"), preferably an atomic force microscope (hereinafter "AFM"), The simulation image is A virtual DNA created on a virtual reality (hereinafter, "VR") system based on error-free X-ray crystal structure data is scanned with a virtual probe of a virtual SPM with different diameters and at different resolutions. The simulation images include virtual low-resolution images and virtual high-resolution images. R . The processing method includes a step of performing super-resolution processing by training a first NN to convert a low-resolution image into a high-resolution image using a plurality of the simulation images as training data; a step of performing domain transformation by learning a second NN using the real image and the virtual low-resolution image as training data so that an image of a real area and an image of a simulation area correspond to each other approximately one-to-one; The method may also include inputting a low-resolution image generated by domain transformation from the image of the real domain into the trained first NN to generate a high-resolution image, and training a third NN so that the generated high-resolution image matches the real image as closely as possible, thereby fine-tuning the generated high-resolution image.

[0010] The first to third NNs each include a generator that generates data from training data; It may also be a generative adversarial network (hereinafter, "GAN") that trains the generated data and a discriminator that determines the closeness between the generated data and the training data in an adversarial manner. Effect of the Invention

[0011] Once a single low-resolution image has been obtained by SPM, a virtual high-resolution image can be generated from this SPM image. Since the actual operation of the SPM can be kept to a minimum, two difficulties can be alleviated: the high cost associated with the use of SPM consumables and the high level of proficiency required to operate the SPM. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating a real AFM image. [Diagram 2] FIG. 1 is a diagram illustrating conventional image processing in which multiple real AFM images are superimposed to generate an average image. [Diagram 3] FIG. 2 is a diagram illustrating an outline of the functions of the present embodiment. [Figure 4] 1 is a diagram showing a system configuration and a block configuration of the present embodiment; [Diagram 5] FIG. 13 is a diagram for explaining a simulation executed to obtain virtual training data in this embodiment. [Figure 6] This is a diagram for providing a general explanation of cycleGAN. [Figure 7] 1A to 1C are diagrams illustrating super-resolution processing according to an embodiment of the present invention. [Figure 8] 11A to 11C are diagrams illustrating a region conversion process according to the present embodiment. [Figure 9] 11A and 11B are diagrams for explaining an appropriate input image size used in the present embodiment. [Figure 10] 1A to 1C are diagrams illustrating the high-resolution AFM image fine-adjustment process of the present embodiment. [Figure 11] FIG. 2 is a diagram showing an image obtained by converting the real AFM image shown in FIG. 1 into a high-resolution image by super-resolution processing. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] The super-resolution processor according to an embodiment of the present invention will be described in the following order. 1. Overview of the Present Invention 2. Configuration of the super-resolution processor 2-1. System Configuration [2-2. Functional block configuration] 3. Operation of the super-resolution processor [3-1. Acquiring real image data from AFM] [3-2. Generating training data through simulation] [3-3. Super-resolution processing of real AFM images] [3-3-1. General explanation of cycleGAN] [3-3-2. Super-resolution processing using the first cycleGAN] [3-3-3. Domain transformation processing using the second cycleGAN] [3-3-4. Fine-tuning of high-resolution AFM images using the third cycleGAN]

[0014] 1. Overview of the Present Invention The outline of this embodiment will be described with reference to FIG. A virtual DNA is created in a VR system, and a low-resolution image (VLI) and a high-resolution image (VHI) are generated by simulating the virtual DNA. The VLI and VHI images are used as training data to learn a first neural network (GAN is used in this embodiment, so it is also referred to as "GAN" below) that converts low-resolution images into high-resolution images. On the other hand, the second GAN learns using the real AFM image (RAI) and the image VLI as training data. Through this learning, the generated AFM image (GAI) and the image VLI are ultimately brought into an almost one-to-one correspondence, and the image RAI and the virtual low-resolution image (GLI) are brought into an almost one-to-one correspondence. Through this learning, if the image RAI and the image GAI, and the image GLI and the image VLI are similar enough to almost match, then the image RAI and the image GLI may be considered to be similar enough to almost match. If the image GLI has an almost one-to-one correspondence with the image VLI, a virtual high-resolution AFM image (GHI) can be generated from the image GLI using the learning results of the first GAN, which has learned using the image VLI and image VHI as training data. Here, an image GLI that has a one-to-one correspondence with the image RAI is generated by the learning results of the second GAN, and an image GHI that has a one-to-one correspondence with the generated image GLI is generated. This image GHI can be considered to be similar to the image RAI to the extent that it almost matches, but in order to further improve the accuracy of the image GHI, a third GAN is trained. The ultimate goal is to obtain a high-resolution image GHI that can be almost identical to the input real AFM image (RAI).

[0015] In Figure 3, the images surrounded by thick circles are training data, and the rest are images generated by GAN. In Figure 3, images, including RAI, are expressed using three English letters, each of which has the following meaning: RAI stands for "Real Afm Image." VLI stands for "Virtual Low-resolution Image." VHI stands for "Virtual High-resolution Image." GAI stands for "Generated Afm Image." GLI stands for "Generated Low-resolution AFM Image." GHI stands for "Generated High-resolution IImage."

[0016] 2. Configuration of the super-resolution processor The configuration of the super-resolution processor of this embodiment will be described with reference to the drawings.

[0017] 2-1. System Configuration 4, the super-resolution processor 1 receives as input data a real image (RAI) 3 captured using an AFM 2 via a communication line or a storage medium such as a USB memory. In this embodiment, a real image captured using an AFM is called a "real AFM image" (RAI), while images generated by simulation under various conditions from a virtual DNA created by a VR (virtual reality) system 4 are called "simulation images" (VLI and VHI). The super-resolution processing device 1 performs super-resolution processing using a real AFM image (RAI) 3 and the generated simulation images (VLI and VHI) as training data, and ultimately generates a high-resolution image (GHI) 7 by super-resolution processing the real AFM image (RAI) 3.

[0018] [2-2. Functional block configuration] 4, the functional blocks of the super-resolution processor 1 are broadly divided into a simulation image generator 11 and a high-resolution image generator 12. These units are realized by developing a computer program in memory and executing it with a CPU. The super-resolution processor 1 also includes a storage unit, an input / output unit, a communication interface unit, and the like, but these are not shown and will not be described. The super-resolution processor 1 uses virtual DNA data 6, which is created based on X-ray crystal structure data 5, from the VR system 4 to obtain simulation images (VLI and VHI). For this virtual DNA data 6, the simulation image generator 11 executes scanning simulation under various conditions. The high-resolution image generating unit 12 includes a super-resolution processing unit 121 corresponding to a first NN, a domain conversion processing unit 122 corresponding to a second NN, and a high-resolution AFM image fine adjustment processing unit 123 corresponding to a third NN.

[0019] The super-resolution processor 121 performs deep learning so that there is a one-to-one correspondence between the simulation low-resolution image (VLI) 8 and the simulation high-resolution image (VHI) 9. The learning in the super-resolution processor 121 is performed by taking one image from among multiple images VLI, combining this with the image VHI to create multiple pairs of images, and using each pair as training data. The results of this learning are also used in the high-resolution AFM image fine-tuning processor 123. The super-resolution processor 1 performs learning using the real image (RAI) of the AFM 2 and the simulated images (VLI and VHI) acquired by scanning simulation as training data. Since the RAI and VLI / VHI are data from regions with different characteristics, region conversion processing is required. Therefore, the region conversion processing unit 122 performs deep learning so that the images of both regions correspond one-to-one as much as possible. 3, a virtual high-resolution AFM image (GHI) is generated from a virtual low-resolution AFM image (GLI) by utilizing the results of prior learning by the super-resolution processor 121. This image GHI is the result of super-resolution processing of the image RAI, but is fine-tuned to more closely resemble the image RAI.

[0020] 3. Operation of the super-resolution processor The operation of the super-resolution processor 1 will be explained, focusing on the simulation image generation unit 11 and processing by three types of GANs.

[0021] [3-1. Acquiring real image data from AFM] The super-resolution processor 1 inputs a real AFM image (RAI) from the AFM 2 via a communication network or a storage medium such as a USB memory. This image RAI is a two-dimensional image on an orthogonal XY plane, and information in the Z direction is represented by shading (see FIG. 1). The input image data is stored in a memory unit (not shown) of the super-resolution processor 1. Simulation image data (VLI / VHI) and image data generated by the three types of GANs during learning (GLI, GAI, GHI) are also stored in the memory unit in the same way.

[0022] The purpose of the present invention is to obtain an image obtained by converting a real AFM image (RAI) into a high-resolution image, and this image RAI is also training data for unsupervised learning. However, since a large number of images required for learning cannot be obtained from the AFM, simulation images (VLI and VHI) are also used as training data to compensate for this. Therefore, the generation of images VLI and VHI is explained below.

[0023] [3-2. Generating training data through simulation] DNA data is extracted appropriately from a dsDNA PDB file, that is, a file that stores X-ray crystal structure data 5 of the double helix of DNA. This is bent and twisted on an existing VR system 4 that designs DNA to create a shape similar to the image RAI. For example, in the technology disclosed in JP 2023-107369 A, virtual DNA data 6 can be easily created by wearing a head-mounted display and simply moving your fingers. Using this technology, the shape can be freely changed on the VR system.

[0024] Once a virtual DNA with the desired shape is created, simulations are performed under various conditions, such as the diameter of the tip of the virtual AFM (hereafter referred to as the "tip diameter"), which is a computer program that mimics an actual AFM, and the resolution during scanning. The results of the simulation are shown in Figure 5. Looking at the right side of Figure 5, we can see that the thickness (width) of the image generated from the same DNA model changes when the tip diameter is changed from 1 nm to 5 nm. If the resolution of this AFM image is changed from 0.1 nm (high-resolution) to 0.7 nm (low-resolution), a blurry-looking AFM image can be generated. In the figure, the image with a tip diameter of 1 nm and a resolution of 0.1 nm is the VHI, and the other multiple images are the VLI. The high-resolution image VHI is combined with any one low-resolution image VLI to create a pair of training data.

[0025] It is cost- and time-consuming to obtain multiple real image data by changing the probe diameter or the resolution with an actual AFM2. Therefore, a virtual DNA with a shape similar to the image RAI is created on the VR system from the X-ray crystal structure data, and a simulation is performed in which this is scanned under various conditions. By using simulation, for example, even if the tip diameter of the actual AFM2 probe is 2 nm, it is possible to obtain components with twice the frequency by performing a scanning simulation with a virtual probe with a tip diameter of 1 nm. In the following description, we will consider that real training data and training data created by simulation exist in opposing domains. The domain to which the real training data belongs will be called the "real domain," and the domain to which the training data obtained by simulation belongs will be called the "simulation domain."

[0026] [3-3. Super-resolution processing of real AFM images] We explain the process of converting the blurry real image acquired from AFM into a clear image with high resolution.

[0027] [3-3-1. General explanation of cycleGAN] As explained above, three types of GAN, which is an unsupervised NN, are used in the super-resolution processor 1. Since GAN is a well-known technology, there is no need to explain it in detail, but for the convenience of describing the operation of the super-resolution processor 1 described later, the main points will be explained here. In addition, there are various types of GANs (Generative Adversarial Networks), but in this embodiment, cycleGAN is used.

[0028] GANs are composed of two types of networks, a generator and a discriminator, and are mainly implemented on a computer as a program module, although some or all of them may be implemented in hardware. The network used to generate data from training data is the generator (synonyms: "generator" and "generative model"). The parameters must be updated until the desired data is generated. This is the learning process for the generator. Since there is no teacher, it is not known whether the data generated by the generator is correct or not. Therefore, the discriminator (synonyms: "classifier" and "discriminative model") distinguishes whether the data is from the training data or the data generated by the generator. If the generated data is determined to be different from the training data (the generator is discovered by the discriminator), the generator learns to deceive the discriminator. The discriminator then continues learning through deep learning so as not to be deceived. In this way, the generator and discriminator repeat adversarial learning with each other, and the goal of learning is for both the generator and the discriminator to improve their capabilities and for the generator to be able to generate data similar to the training data.

[0029] Here, we will briefly explain cycleGAN with reference to Figure 6. Let G be a generator from the X domain to the Y domain. Generator G takes an element x in the X domain as input data and generates an element G(x) in the Y domain. On the other hand, let F be a generator from the Y domain to the X domain. Using an element G(x) in the Y domain as input data, generator F generates an element F((G(x)) in the X domain (Figure 6(1)). The discriminator D1 on the X domain side takes the element x in the X domain and the element F((G(x)) as input data and judges whether the element F((G(x)) is similar to the element x. The discriminator D1 evaluates the error between the element x and the element F((G(x)) using a loss function, and makes the above judgment based on this evaluation. In addition, based on the loss function, it updates the generator parameters, i.e., the weights of the connections between nodes and the biases of the nodes. Deep learning training of the generator and discriminator continues until it is determined that the input x and the output F(G(x)) are similar when transformed x-->G(x)-->F(G(x)). On the other hand, generator F takes an element y in the domain Y as input and generates an element F(y) in the domain X, which is input to generator G to generate G(F(y)) in the domain Y (Figure 6(2)). The generators F and G and the discriminator D2 continue adversarial learning until the discriminator D2 on the Y domain side determines that the element y in the Y domain is similar to G(F(y)). In both (1) and (2) of Figure 6, if the learning goal is achieved, it can be said that there is a one-to-one correspondence between the X domain and the Y domain. In this way, since there is bidirectional conversion from the X domain to the Y domain and from the Y domain to the X domain, two generators and two discriminators are required. Note that generator G in Figure 6 (1) and Figure 6 (2) has the same structure, since it generates data in the Y domain from data in the X domain. The same is true for generator F.

[0030] Incidentally, one of the important terms in deep learning is loss. This loss is used to mean whether error propagation during the learning process is correctly transmitted to nodes close to the input layer. Learning in deep learning involves adjusting the weights between each node in deep learning so that when the output result of deep learning differs from the training data, the difference is eliminated. In this case, the learning efficiency changes greatly depending on how much of the output error is transmitted to the previous node. Also, in cycleGAN, as explained above, training is performed so that the input and output are basically the same. Figure 6(1) shows two types of loss. The "cycle-consistency loss" refers to the difference between the data F(G(x)) generated by cycling (X region --> Y region --> X region) and the training data x. This means that the input image x and the generated image F(G(x)) are fed back with information on how similar or different they are when they go around in one cycle. On the other hand, the "shape-consistency loss" refers to the feedback on how similar the input image x and the generated image G(x) are to each other. The same is true for Figure 6(2). CycleGAN continues learning until these errors are nearly zero. Deep learning mechanisms such as error evaluation and parameter updating are well known, and many reference documents have been published, so detailed explanations will be omitted.

[0031] [3-3-2. Super-resolution processing using the first cycleGAN] The operation of the super-resolution processor 121 including the first cycleGAN will be described with reference to FIG. A large number of simulation image data were created in the simulation data generating unit 11. As already explained, the most detailed image is called VHI, and the other images are called VLI. An arbitrary image is taken from multiple VLI images, and learning is performed using this image VLI and image VHI pair as training data. Another image VLI is then taken, and deep learning is continued using this image VLI and image VHI pair as training data. Since there are a maximum of as many pairs of training data as there are VLI images, precise learning is possible. FIG. 7 illustrates learning when a pair of images VLI and VHI are used as training data. In FIG. 7, Gl2h is a generator that converts a low-resolution image into a high-resolution image, and Gh2l is a generator that converts a high-resolution image into a low-resolution image. Dli is a discriminator that determines how similar the generated low-resolution AFM image is to a virtual low-resolution image (VLI), and Dhi is a discriminator that determines how similar the generated high-resolution AFM image is to a virtual high-resolution image (VHI).

[0032] Gl2h takes a virtual low-resolution image (VLI) as input and generates a high-resolution AFM image. Gh2l takes this high-resolution AFM image as input and generates a low-resolution AFM image. Dli judges whether the low-resolution AFM image is similar to the VLI based on an evaluation of the error between the VLI image and the low-resolution AFM image. Gl2h and Gh2l continue to learn until they are judged to be similar. At the same time, Dli also continues to learn so that it can make an appropriate judgment. Meanwhile, Gh2l takes the virtual high-resolution image VHI as input and generates a low-resolution AFM image. Gl2h takes this low-resolution AFM image as input and generates a high-resolution AFM image. Dhi judges whether the high-resolution AFM image is similar to image VHI based on an evaluation of the error between image VHI and the high-resolution AFM image. The generators Gh2l and Gl2h continue to learn until it is judged to be similar. At the same time, Dhi also continues to learn so that it can make an appropriate judgment. If the generators Gl2h and Gh2l and the discriminators Dli and Dhi achieve the goal, the parameters at this time are saved and learning using one set of training data is completed. This learning is performed the number of times equal to the number of pairs of training data. The parameters obtained by the above learning are stored in the storage unit and are used when the high-resolution AFM image fine adjustment processing unit 123 described later generates the image GHI from the image GLI. In other words, the super-resolution processing unit 121 performs pre-learning for generating the high-resolution AFM image GHI.

[0033] [3-3-3. Domain transformation processing using the second cycleGAN] A domain conversion processing unit 122 converts the domain between the real domain and the simulation domain. It takes some ingenuity to use image data from regions with different characteristics, image RAI and image VLI, as training data. The second GAN is trained to generate an image GLI of a simulated domain that corresponds to an image RAI of a real domain.

[0034] The operation of the domain conversion processing unit 122 including the second cycleGAN will be described with reference to FIG. Gr2s is a generator that converts images in the real domain into images in the simulation domain, and Gs2r is a generator that converts images in the simulation domain into images in the real domain. Dri and Dsi are discriminators that determine whether an image is included in a specific region. Dri determines how similar the generated AFM image (GAI) is to the real AFM image (RAI), and thus determines whether the generated image GAI is included in the real region. Dsi determines how similar the generated virtual AFM image (GLI) is to the virtual low-resolution image (VLI), and thus determines whether the generated image GLI is included in the simulation region.

[0035] Gr2s takes image RAI as input and generates image GLI. Gs2r takes image GLI as input and generates image GAI. Dri judges image RAI and image GAI based on error evaluation, and the learning of generators Gr2s and Gs2r continues until image GAI is judged to be similar to image RAI. At the same time, Dri also continues learning so that it can make an appropriate judgment. On the other hand, Gs2r takes the image VLI as input and generates an image GAI. Gr2s takes this image GAI as input and generates an image GLI. Dsi judges the image VLI and the image GLI based on an error evaluation, and Gr2s and Gs2r continue learning until it is judged that the image GLI is similar to the image VLI. At the same time, Dsi also continues learning so that it can make an appropriate judgment.

[0036] When the generators Gr2s, Gs2r and the discriminators Dri, Dsi achieve the final goal, the parameters at this time are saved and deep learning ends. If the generator is in the early stages of learning, the generated image data will be completely dissimilar to the training data, and the discriminator will not mistakenly identify the generated data as training data. However, by repeating deep learning, the generator will be able to generate similar images, and the discriminator will be able to determine that the generated data is similar to the training data. At the end of the learning process, a low-resolution image (GLI) of the simulation region side that corresponds almost one-to-one to the real AFM image (RAI) is generated. This image GLI is input to the next high-resolution AFM image fine-tuning process 123.

[0037] Regarding "one-to-one," a further comment will be made with reference to FIG. Even if the learning of GAN progresses, it is difficult to realize a perfect one-to-one correspondence between the image taken by the microscope and the image generated by GAN for all images. The learning of GAN realizes the mapping from area (a) to area (b) in Figure 9 and the inverse mapping from area (b) to area (a), but the range in which the one-to-one relationship can be realized (within the dashed line in Figure 9) depends heavily on the number and type of training data given, and in the case of AFM images, it depends heavily on the image resolution and the diameter of the probe. Therefore, it is desirable to perform super-resolution processing using AFM images taken within the range of the resolution and diameter of the probe of the virtual low-resolution image (VLI) used as training data.

[0038] [3-3-4. Fine-tuning of high-resolution AFM images using the third cycleGAN] The operation of the third cycleGAN will be described with reference to FIG. The high-resolution AFM image fine-tuning processor 123 including the third GAN generates a virtual high-resolution AFM image (GHI) from the virtual AFM image (GLI) generated by the second GAN using the first GAN. The first GAN used here has already been pre-trained. Image RAI is input to trained Gr2s to generate image GLI. Image GLI is input to trained Gl2h to generate image GHI. Therefore, image RAI should correspond one-to-one with image GHI. However, image GHI is fine-tuned to further improve its accuracy. As shown in Fig. 10, the generator Gh2r of the third GAN generates an image GAI. The discriminator Dri determines whether the image RAI and the image GAI are similar. When Dri determines that the image GAI is data derived from the image RAI, it is determined that the learning of the third GAN is completed, and the image GHI at this time is the finally generated virtual high-resolution AFM image. Figure 1 is an example of a real AFM image (RAI), and the image GHI generated from this RAI provides accurate information including the major and minor grooves of DNA, as shown in Figure 11.

[0039] The above embodiment is merely an example. For example, the simulation data generation unit 11 that acquires the simulation images VLI and VHI may be executed by an information processing device other than the super-resolution processing device 1. In addition, the functions of the three types of GAN may be distributed and executed by multiple computers instead of one computer. What is important in the present invention is that virtual simulation images such as VLI and VHI are used as training data to compensate for the lack of real training data required for super-resolution. A super-resolution processing method provided with this and an apparatus and computer program for implementing this method belong to the scope of the present invention.

[0040] The present invention can be applied not only to SPMs but also to other instruments for observing nano- and micro-scale objects (e.g., confocal microscopes, electron microscopes, etc.) Furthermore, the object is not limited to DNA, and the present invention can also be used for observing the surface of cancer cells. [Industrial Applicability]

[0041] SPMs can observe the surfaces of nano-level objects cheaply and easily, and are therefore expected to contribute to research on DNA, cells, and other related technical fields. [Explanation of symbols]

[0042] 1: Super-resolution processing device 11: Simulation image generation section 12: High-resolution image generation section 121: Super-resolution processing section (first GAN) 122: Domain transformation processing section (second GAN) 123: High-resolution AFM image fine-tuning processor (third GAN) 2: AFM 4: VR system (for DNA design) RAI: Real AFM Image GAI: Generated AFM images VLI: Virtual Low-resolution Image VHI: Virtual High Resolution Image GLI: Generated Virtual Low-Resolution AFM Images GHI: Generated virtual high-resolution AFM images

Claims

1. 1. A method for performing super-resolution processing, comprising: obtaining a low-resolution real image taken by a microscope; performing a simulation under different conditions to generate a simulation image; acquiring a high-resolution image that is clearer than the real image by learning a neural network (hereinafter, "NN") using the real image and the simulation image as training data; The real image is an image of DNA obtained from a scanning probe microscope (hereinafter "SPM"), preferably an atomic force microscope (hereinafter "AFM"); The simulation image is An image processing method characterized in that the image processing method includes a plurality of simulated images obtained by simulating scanning a virtual DNA created on a virtual reality (hereinafter referred to as "VR") system based on error-free X-ray crystal structure data with a virtual probe of a virtual SPM having a different diameter and at different resolutions, the simulated images including a virtual low-resolution image and a virtual high-resolution image.

2. A step of performing super-resolution processing by training a first neural network to convert a low-resolution image into a high-resolution image using a plurality of the simulation images as training data; a step of performing domain transformation by training a second NN using the real image and the virtual low-resolution image as training data so that an image of a real area and an image of a simulation area correspond to each other approximately one-to-one; the low-resolution image generated by domain conversion from the image of the real domain is input to the trained first NN to generate a high-resolution image, and a third NN is trained so that the generated high-resolution image matches the real image as closely as possible, thereby fine-tuning the generated high-resolution image.

3. The first to third NNs each include a generator that generates data from training data; 3. The image processing method according to claim 2, characterized in that the generated data and a discriminator that determines the closeness of the generated data to the training data are trained in an adversarial manner to each other by a generative adversarial network (hereinafter referred to as "GAN").

4. A means for acquiring a low-resolution real image taken by a microscope; A means for generating a simulation image by performing a simulation under different conditions; and a means for acquiring a high-resolution image that is clearer than the real image by learning a neural network (hereinafter, referred to as "NN") using the real image and the simulation image as training data. The real image is an image of DNA obtained from a scanning probe microscope (hereinafter "SPM"), preferably an atomic force microscope (hereinafter "AFM"); The simulation image is A super-resolution processing device characterized in that the multiple simulation images include virtual low-resolution images and virtual high-resolution images, the simulation images being obtained by simulating scanning a virtual DNA created on a virtual reality (hereinafter referred to as "VR") system based on error-free X-ray crystal structure data with a virtual probe of a virtual SPM having a different diameter and at different resolutions.

5. A computer A means for acquiring low-resolution real images taken with a microscope; A means for generating a simulation image by performing a simulation under different conditions; a means for acquiring a high-resolution image that is clearer than the real image by learning a neural network (hereinafter, referred to as "NN") using the real image and the simulation image as training data; Then, super-resolution processing is performed. The real image is an image of DNA obtained from a scanning probe microscope (hereinafter "SPM"), preferably an atomic force microscope (hereinafter "AFM"); The simulation image is A computer program comprising a plurality of simulated images obtained by simulating scanning a virtual DNA created on a virtual reality (hereinafter referred to as "VR") system based on error-free X-ray crystal structure data with a virtual probe of a virtual SPM having a different diameter and at different resolutions, the simulated images including virtual low-resolution images and virtual high-resolution images.

Citation Information

Patent Citations

  • Microscopic image super-resolution method based on mathematical imaging theory and generative adversarial network

    CN114897693A

  • Method, apparatus, and computer program for improving the reconstruction of high-density super-resolution images from diffraction-limited images acquired by single-molecule localization microscopy

    JP2020529083A

  • Information processing device and program

    JP2021170284A

  • Image inspection device and image inspection method

    JP2022013226A

  • Observation method using compound microscope having inverted optical microscope and atomic force microscope, and program and compound microscope for carrying out observation method

    WO2017145381A1

Cited By

  • Method for optimizing fast scanning imaging of atomic force microscope

    CN116819128A