Apparatus and Method for Virtual Staining of Pathology Sample using Quantitative Phase Image based on Near-Infrared Ray

Abstract

The present disclosure provides a pathology sample virtual staining apparatus and method that can easily acquire virtual staining sample images for unlabeled samples as well as decolorized samples, by including a sample image acquisition module that irradiates near-infrared light toward a placed sample to acquire a near-infrared-based phase image of the sample, and a virtual staining module that receives the near-infrared-based phase image and performs neural network computation to acquire a virtual staining image similar to an image of a stained sample in which the sample is stained.

Description

Technology Field

[0001] The present disclosure relates to an apparatus and method for virtual staining of pathological samples, and more specifically, to an apparatus and method for virtual staining of pathological samples using near-infrared-based phase imaging. Background Technology

[0002] Generally, a biopsy is taken for pathological diagnosis, and chemical staining is performed on the specimen to make the tissue structure visible. However, in the case of Hematoxylin & Eosin Staining (H&E staining), which is the universally used tissue staining method as the gold standard in pathology, the stain fades over time or when exposed to light and moisture, causing difficulties in diagnosis or further analysis.

[0003] While chemical re-staining can be performed to restore this, it is highly inefficient for processing large volumes of samples due to the significant time and cost involved. Therefore, techniques have been proposed to restore virtually stained colors without performing re-staining. Virtual color restoration methods primarily utilize color normalization algorithms to restore decolorized colors to their original stained colors; however, they have limitations in that their application is restricted to samples that have become nearly transparent due to extensive decolorization or to unstained samples.

[0004] To overcome these limitations, a virtual staining technology has recently been developed that can convert label-free images of biological specimens into stained images using deep learning-based image conversion techniques.

[0005] Figure 1 is a diagram showing a comparison between a conventional sample staining technique and a virtual staining technique.

[0006] As shown in (a) of Fig. 1, both the sample staining technique and the virtual staining technique first collect an unstained sample to be examined and place it in the sample. Then, in the case of the sample staining technique, as shown in (b), chemical staining is performed on the collected sample, and a high-resolution image of the chemically stained sample is obtained using optical equipment such as a microscope.

[0007] However, as illustrated in (c), while virtual staining technology does not use a stained sample, label-free optical imaging technology is applied to acquire a label-free sample image by optically capturing unique information of the biological sample that can identify important features within the tissue of the label-free sample. Then, a virtual stained sample image is acquired by using a Deep Neural Network (DNN) to assign colors based on the unique information captured in the label-free sample image.

[0008] When an image of a stained sample or an image of a virtual stained sample is obtained, an expert such as a doctor analyzes the obtained image as in (d) to determine whether a disease has occurred.

[0009] Previous studies on virtual staining technology have confirmed that label-free images captured using autofluorescence, multiphoton microscopy, photoacoustic microscopy, and holographic microscopy can be successfully converted into stained color images using trained deep neural networks, demonstrating the advantages of virtual staining in terms of speed, labor and cost reduction, eco-friendliness, and versatility.

[0010] However, to perform virtual staining using a deep neural network, the network must be trained in advance. To train the neural network, images of unlabeled samples and stained samples are first acquired. Then, using the stained sample images as ground truth, the neural network must be supervised so that the unlabeled sample images input to the network are converted into the stained sample images corresponding to the ground truth.

[0011] However, to perform such learning, it is necessary to first acquire label-free sample images and then separately acquire stained sample images after staining the samples; this requires image acquisition twice, consuming time and cost. Furthermore, to use stained sample images as truth values, alignment between the label-free and stained sample images must be performed first. However, accurately aligning high-resolution, precise sample images captured with a microscope is very difficult. Moreover, the structure or tissue morphology of the sample may change due to time differences or external factors during the staining process; this further complicates image alignment and, in some cases, may render the data unusable for training. Due to these issues, training neural networks with virtual staining requires significant time, cost, and effort.

[0012] Furthermore, even if a neural network is trained to convert unlabeled sample images into stained sample images, additional training is required to enable the network to acquire virtual stained sample images from decolorized samples that have faded in color after staining. However, since the color of decolorized samples decolorizes to varying degrees over time and storage conditions, and decolorization proceeds slowly over years to decades, it is practically impossible to acquire a large volume of diverse samples with different decolorization states. Nevertheless, since decolorized samples stored for a long period hold significant value for long-term pathological research, acquiring stained images of them is crucial. The problem to be solved

[0013] The object of the present disclosure is to provide a pathology sample virtual staining apparatus and method that can easily acquire virtual stained sample images for unlabeled samples as well as decolorized samples.

[0014] The object of the present disclosure is to provide a pathology sample virtual staining apparatus and method capable of acquiring a virtual stained sample image based on a phase image obtained according to the tissue structure of the sample using near-infrared rays, without being affected by the color state of the sample stained.

[0015] The objective of the present disclosure is to provide a pathology sample virtual staining apparatus and method capable of acquiring RGB images and near-infrared-based phase images of a stained sample together, and learning virtual staining by using the acquired RGB images as truth values ​​for the phase images. means of solving the problem

[0016] A pathology sample virtual staining device according to one embodiment of the present disclosure includes: a sample image acquisition module that irradiates near-infrared light toward a placed sample to acquire a near-infrared-based phase image of the sample; and a virtual staining module that receives the near-infrared-based phase image and performs neural network computation to acquire a virtual staining image similar to an image of a stained sample in which the sample is stained.

[0017] The virtual staining module described above can be trained using a near-infrared-based training phase image obtained for a training sample, which is a sample pre-stained for training, and an RGB-based training intensity image obtained by irradiating RGB light toward the training sample as a training dataset.

[0018] When a near-infrared-based learning phase image is input during training, the above virtual staining module obtains a learning virtual staining image by performing neural network operations, and can be trained by backpropagating the loss calculated by comparing the obtained learning virtual staining image with the learning intensity image.

[0019] When acquiring the training data set, the sample image acquisition module can acquire the near-infrared-based phase image and the training intensity image by maintaining the training sample in its arranged state and alternately applying the RGB light and the near-infrared light to the sample.

[0020] The sample image acquisition module can acquire intensity images for each of the R, G, and B colors by irradiating light according to each of the R, G, and B colors toward the sample during training, and can acquire the training intensity image by synthesizing the acquired intensity images for each of the R, G, and B colors.

[0021] The above sample image acquisition module can acquire near-infrared-based phase images of unlabeled samples that are not stained or decolorized samples that have been stained and then decolorized.

[0022] The sample image acquisition module may include: a sample slide on which the sample is placed; an illumination module that irradiates near-infrared light toward the sample slide; a sensing module that detects the light irradiated from the illumination module and transmitted through the sample to generate a detection signal; and an image acquisition module that receives the detection signal and generates a near-infrared-based phase image.

[0023] The above lighting module can be implemented as an array light source in which a plurality of light sources emitting near-infrared light or R, G, and B light are arranged.

[0024] The above lighting module may be implemented as a plurality of replaceable modules, each including a light source that emits near-infrared light or R, G, and B light.

[0025] A method for virtual staining of a pathological sample according to another embodiment of the present disclosure is a method performed by a processor, comprising: a step of irradiating near-infrared light toward a sample placed in a sample image acquisition module to acquire a near-infrared-based phase image of said sample; and a step of performing neural network operations on said near-infrared-based phase image with a virtual staining module implemented as a neural network model to acquire a virtual staining image similar to an image of a stained sample in which said sample is stained. Effects of the invention

[0026] The pathology sample virtual staining apparatus and method of the present disclosure can acquire virtual stained sample images based on phase images obtained according to the tissue structure of the sample using near-infrared rays without being affected by color, thereby enabling easy acquisition of virtual stained samples for both unlabeled and decolorized samples. Furthermore, by acquiring RGB images and near-infrared-based phase images together to perform learning, training data can be easily acquired and training can be performed. Brief explanation of the drawing

[0027] Figure 1 is a diagram showing a comparison between a conventional sample staining technique and a virtual staining technique. FIG. 2 shows a schematic configuration divided according to the operation of performing a virtual staining device for pathological samples according to one embodiment. Figure 3 shows an example of the implementation of the sample image acquisition module of Figure 2. Figure 4 is a diagram illustrating the difference between a near-infrared-based phase image and a red light-based phase image. FIG. 5 illustrates a method for virtual staining of a pathological sample according to one embodiment. Figure 6 is a diagram illustrating the learning process of the virtual dyeing module of Figure 5. Figure 7 is a diagram illustrating the operation of the virtual dyeing module of Figure 5. FIG. 8 is a drawing for explaining a computing environment including a computing device according to one embodiment. Specific details for implementing the invention

[0028] Hereinafter, specific embodiments according to embodiments of the present disclosure will be described with reference to the drawings. The following detailed description is provided to facilitate a comprehensive understanding of the methods, apparatuses, and / or systems described herein. However, this is merely illustrative and the present invention is not limited thereto.

[0029] In describing the embodiments of the present disclosure, detailed descriptions of known technology related to the present invention are omitted if it is determined that such detailed descriptions would unnecessarily obscure the essence of the embodiments. Furthermore, terms described below are defined with consideration of their functions in the present invention, and these may vary depending on the intentions or practices of the user or operator. Therefore, such definitions should be based on the content throughout this specification. Terms used in the detailed description are intended merely to describe specific embodiments and should not be limiting. Unless explicitly stated otherwise, expressions in the singular form include the meaning of the plural form. In this description, expressions such as “include” or “compose” are intended to refer to certain characteristics, numbers, steps, actions, elements, parts thereof, or combinations thereof, and should not be interpreted to exclude the existence or possibility of one or more other characteristics, numbers, steps, actions, elements, parts thereof, or combinations thereof other than those described. Additionally, terms such as “...part,” “...unit,” “module,” and “block” described in the specification refer to a unit that processes at least one function or operation, and this may be implemented in hardware, software, or a combination of hardware and software.

[0030] FIG. 2 shows a schematic configuration divided according to the operation of performing a virtual staining device for a pathology sample according to one embodiment, FIG. 3 shows an example of the implementation of the sample image acquisition module of FIG. 2, and FIG. 4 is a diagram for explaining the difference between a near-infrared-based phase image and a red light-based phase image.

[0031] As illustrated in FIG. 2, the pathology sample virtual staining device according to the embodiment may be largely composed of a sample image acquisition module (10) and a virtual staining module (20).

[0032] The sample image acquisition module (10) acquires an image of a sample for which a pathological diagnosis is to be performed. In one embodiment, the sample image acquisition module (10) acquires a phase image of the sample, and in particular, can acquire a quantitative phase image based on near-infrared light. Here, the sample may be an unlabeled sample that has not been stained, or a sample that has been stained and has decolorized over time, but it may also be a stained sample. However, in the learning process for training the virtual staining module (20) implemented as a deep neural network, a stained sample is used so that a learning data set can be easily acquired.

[0033] Conventionally, intensity images of samples are generally acquired, and in the case of intensity images of unlabeled samples, the tissue structure of the sample is not clearly visible. Therefore, to allow experts performing the diagnosis to easily verify the information, stained samples must be used to ensure that the tissue structure is clearly represented in the image. However, in the pathology sample virtual staining device of one embodiment, the virtual staining module (20) performs virtual staining or virtual re-staining on the sample image acquired by the sample image acquisition module (10), thereby overcoming the inefficiency of staining the sample. Accordingly, in the embodiment, it is acceptable for the sample image acquisition module (10) to acquire an image of an unlabeled sample or a decolorized sample. However, as mentioned above, when an intensity image of the sample is acquired, there is a problem in that the tissue structure of the unlabeled sample or the decolorized sample is not clearly visible. To resolve this problem, the sample image acquisition module (10) according to the embodiment acquires a phase image in which the tissue structure can be clearly visible.

[0034] Phase imaging allows for the visualization of changes in light paths caused by refractive indices resulting from the tissue structure of the sample—specifically, differences in sample thickness and local protein density—thereby clearly expressing meaningful spatial contrasts between subcellular organelles such as the nucleus, cytoplasm, and lipids. Compared to intensity imaging, phase imaging is relatively less affected by color, allowing for the acquisition of images with minimal variation due to the sample's staining state.

[0035] In addition, the sample image acquisition module (10) irradiates the sample with near-infrared light to acquire a near-infrared-based phase image. Here, the sample image acquisition module (10) acquires a near-infrared-based phase image to minimize the influence of the staining state of the sample. As mentioned above, although the phase image is relatively less affected by color compared to the intensity image, due to the difference in absorption based on the stained color of the stained or decolorized sample, the light undergoes phase delay differently depending on the color as well as the thickness or refractive index of the sample. Accordingly, a phase difference based on color occurs in the phase image regardless of the tissue structure.

[0036] In Figure 4, (a) shows an example of a decolorized sample that has been decolorized to different levels over time after dyeing, and (b) shows an example of a newly dyed sample. A dyed sample that has been re-dyed after decolorization can also be seen as similar to (b).

[0037] And (c) and (d) represent examples of near-infrared-based phase images acquired for the decolorized sample of (a) and the dyed sample of (b), respectively, and (e) and (f) represent examples of red light (R)-based phase images acquired for the decolorized sample of (a) and the dyed sample of (b).

[0038] As mentioned above, the representative staining technique for samples is the H&E staining technique. When acquiring a phase image of a sample stained by the H&E staining technique, among the light of each color (R, G, and B), the absorption of the R color light by the stained color is the lowest. In other words, a phase image that is relatively less affected by the staining state of the sample can be acquired. Accordingly, Figure 4 illustrates a comparison between a near-infrared-based phase image and a red light-based phase image. As shown in Figures 4 (c) and (d), it can be seen that the phase images acquired from decolorized samples and stained samples are obtained at a nearly similar level in the case of the near-infrared-based phase image. Also, as shown in (e), the red light-based phase image can also be obtained at a level similar to the near-infrared-based phase image for decolorized samples. However, as shown in (f), it can be seen that the red light-based phase image for a heavily stained sample is obtained in a form that differs significantly from the phase image for the decolorized sample in (e). That is, even if a phase image is acquired using red light, which is relatively least affected among the light of each color of R, G, and B, the phase image will appear significantly different depending on the staining state of the sample. In contrast, if a phase image is acquired using near-infrared light, a stable phase image can be acquired because it is hardly affected by the staining state of the sample. Accordingly, in the pathology sample virtual staining device of the embodiment, the sample image acquisition module (10) is configured to acquire a near-infrared-based phase image. However, during the learning process of training the virtual staining module (20), the sample image acquisition module (10) is configured to acquire an RGB-based intensity image to be used as a truth value along with the near-infrared-based phase image, thereby enabling the acquisition of a learning data set.

[0039] The sample image acquisition module (10) can be implemented with various devices such as differential phase contrast (DPC), transport-of-intensity (TIE), and Fourier typography imaging devices capable of acquiring quantitative phase images, but here, as an example, it is assumed to be implemented with a Fourier typography imaging device shown in FIG. 3.

[0040] Fourier tychography microscopy (FPM) is one of the devices that utilizes a space-bandwidth product imaging technique based on computational imaging technology, which simultaneously increases the observation area and resolution through structural illumination. It uses a low-magnification, low-numerical-aperture objective lens capable of acquiring large-area images, and acquires multiple low-resolution images by taking multiple photos while changing the angle of incidence of the illumination incident on the sample. Then, a single large-area high-resolution image is reconstructed by combining the acquired multiple low-resolution images using a computation-based reconstruction algorithm.

[0041] Generally, high-resolution images of a sample are acquired using high-magnification optical imaging devices. However, in optical imaging devices, resolution and the size of the acquireable image are inversely proportional. Therefore, while using a high-magnification objective lens with a high numerical aperture (NA) allows for the acquisition of high-resolution images, the acquireable image area is significantly reduced. Conversely, using a low-magnification objective lens with a low numerical aperture greatly expands the image area that can be acquired at once, but this results in a decrease in resolution. Fourier tychography imaging devices overcome these physical limitations of optical devices through computation, thereby enabling the acquisition of large-area, high-resolution reconstructed images.

[0042] In this process, light phase information that is not measured by the image sensor is restored. Phase information is associated with the optical path, which can be modeled using the refractive index and thickness of transparent samples, such as pathological specimens. Therefore, by using a Fourier Tichography imaging device, high-contrast images of transparent pathological specimens, which are difficult to observe with conventional bright-field imaging devices, can be acquired. Furthermore, the refractive index and thickness profiles of the specimen can be easily obtained from the restored phase information.

[0043] Referring to FIGS. 2 and 3, a sample image acquisition module (10) implemented as a Fourier Tichography imaging device may include an illumination module (11), a sample slide (12), an optical system (13), a sensing module (14), and an image acquisition module (15).

[0044] The lighting module (11) generates light and irradiates it toward the sample slide (12). The lighting module (11) may be configured to have a plurality of light sources arranged to emit light toward the sample slide (12), and may be implemented as an array of light sources, for example. It is assumed that each of the plurality of light sources is implemented as an LED, but is not limited thereto. The lighting module (11) can generate light in various patterns by individually turning on or off each of the plurality of light sources. For example, the lighting module (11) can control the angle of incidence of the light incident on the sample slide (12) by sequentially turning on one or more of the arrayed light sources.

[0045] In addition, the lighting module (11) in the present disclosure may be configured to emit light having a near-infrared wavelength. Furthermore, the lighting module (11) may be configured to irradiate not only near-infrared light but also light of wavelengths for each color of RGB or white light that is a composite of RGB, in order to obtain training data for training the virtual dyeing module (20).

[0046] At this time, the lighting module (11) may be configured such that a plurality of light sources emitting light of different wavelengths are arranged in an array so that light of wavelengths of each RGB color or white light can be irradiated alternately along with near-infrared light. However, light of wavelengths other than near-infrared light is used only when training the virtual staining module (20) of the pathology sample virtual staining device, and is not used when actually operating the pathology sample virtual staining device. Therefore, it is inefficient to configure the lighting module (11) to always emit light other than near-infrared light. Accordingly, the lighting module (11) may be implemented as a replaceable module configured to emit light of the required wavelengths.

[0047] The sample slide (12) is a sample container in which a sample is placed, and is positioned between the lighting module (11) and the optical system (13) in the light path. The sample slide (12) may be made of a transparent material so that light irradiated from the lighting module (11) can be transmitted and incident on the objective lens of the optical system (13). The type of sample to be measured is not limited, but here it is assumed to be a pathological sample for pathological diagnosis as an example. In addition, the sample may be a transparent sample through which light can be transmitted, but it may be at least partially stained so that the structure of the sample is clearly contrasted, and it may also be a decolorized sample that has been decolorized over time after being stained. However, in the learning process for training the virtual staining module (20), a stained sample is used to obtain an RGB-based intensity image used as a truth value.

[0048] The optical system (13) receives light irradiated from the illumination module (11) and transmitted through the sample slide (12), and focuses the received light onto the sensing module (14). The optical system (13) may include an objective lens, and the objective lens focuses the light irradiated from the illumination module (11) and transmitted through the sample slide (12). The objective lens of the Fourier Tichography imaging device is implemented as a low-magnification lens having a low numerical aperture (NA), and can significantly expand the sample area detected by the sensing module (14) compared to a typical high-magnification imaging device. Additionally, the optical system (13) may further include a barrel lens, etc. The barrel lens can be used together with the objective lens to enable the image to be formed more effectively on the sensing module (14).

[0049] The sensing module (14) detects the intensity of light that is focused and transmitted from the optical system (13), including an image sensor, and generates a detection signal, which is an electrical signal. At this time, since the objective lens is implemented as a low-magnification lens having a low numerical aperture (NA), the detection signal transmitted from the sensing module (14) is a signal that detects a wide area of ​​the sample, and depending on the direction in which the light generated by the illumination module (11) is irradiated, that is, the angle of incidence of the light generated as it enters the sample slide (12), multiple detection signals that are different from each other can be generated even for the same sample.

[0050] And the image acquisition module (15) receives a plurality of detection signals from the sensing module (14) and acquires a plurality of low-resolution images. The image acquisition module (15) can acquire a low-resolution image of a wide area of ​​the sample from each of the plurality of detection signals, and can acquire a high-resolution sample image of a wide area by applying a restoration algorithm to the acquired plurality of low-resolution images.

[0051] When light is irradiated from the lighting module (11) with a changed angle of incidence and passes through the sample slide (12) to the objective lens of the optical system (13), the phase or intensity of the light changes according to the refractive index and thickness of the sample. That is, the light changes according to the sample's response to light and is incident on the objective lens. This can be seen as the physical information of the sample, such as its refractive index or thickness, being reflected differently in the form of spatial frequency according to the angle of incidence of the incident light and transmitted to the objective lens. Therefore, the light focused by the objective lens appears similar to the Fourier plane, which is the frequency space in which the sample information is projected, and is subjected to a Fourier transform. Furthermore, by applying a reconstruction algorithm that obtains information about the change in light generated by the sample through computation while the light passes through the sample at each angle of incidence, a high-resolution, large-area sample image can be obtained from multiple low-resolution images.

[0052] In other words, when the lighting module (11) changes the angle of incidence of light on the sample, the spatial frequency information of the object is shifted according to the angle and can be measured by the sensing module (14). If information regarding the angle of incidence is known, the degree of shift of the spatial frequency information can be calculated using the angle of incidence. Therefore, a reconstructed image with a wide spatial frequency synthesis bandwidth can be obtained by utilizing the spatial frequency information contained in multiple images.

[0053] For spatial frequency synthesis, phase information of the light passing through the sample must also be restored, but the sensing module (14) can only measure the intensity of the light. Accordingly, in Fourier typography, a restoration algorithm is performed by utilizing superimposed information between multiple images obtained using multiple light sources provided in the illumination module (11) to simultaneously perform phase restoration and resolution enhancement.

[0054] Accordingly, when the image acquisition module (15) acquires a sample image by applying a restoration algorithm to a plurality of low-resolution images, an intensity image expressing the change in light intensity while the light irradiated from the lighting module (11) passes through the sample and is detected, and a phase image expressing the change in the phase of the light are acquired together.

[0055] The technique by which a Fourier Typography device acquires intensity and phase images by applying a reconstruction algorithm is a known technology, so it will not be explained in detail here.

[0056] When the sample image acquisition module (10) is actually operated, the illumination module (11) irradiates near-infrared light, and transmits the phase image among the intensity image and phase image of the sample generated based on the irradiated near-infrared light to the virtual staining module (20).

[0057] However, during the learning process of training the virtual staining module (20), the sample image acquisition module (10) can acquire a phase image of the stained sample based on near-infrared light, along with an intensity image based on light according to each RGB color. Then, the RGB-based intensity image to be used as a truth value during the training of the virtual staining module (20) is transmitted to the learning module (30).

[0058] As mentioned above, conventionally, an intensity image of an unlabeled sample is acquired first, and then an intensity image of a stained sample is acquired separately to construct a training dataset. However, in order to acquire a training dataset in this manner, a process of staining the sample must be added between the process of acquiring the unlabeled sample image and the process of acquiring the stained sample image. Due to the addition of this staining process, it is inefficient in terms of time and cost, and there is a limitation in that it is difficult to acquire a training dataset for decolorized samples that have been decolorized to different levels. Furthermore, a complex process of matching the acquired unlabeled sample image and the stained sample image must also be performed. Additionally, when the unlabeled sample image is acquired as an intensity image, the tissue structure of the sample is not clearly represented, so there is a problem that it is difficult to train the virtual staining module (20).

[0059] In contrast, in the embodiment, the sample image acquisition module (10) acquires a near-infrared-based phase image that clearly shows the tissue structure of the sample while being hardly affected by the staining state. Therefore, a phase image that clearly shows the tissue structure of the sample can be acquired regardless of whether it is an unlabeled sample or a decolorized sample. Furthermore, when acquiring a training data set, both a near-infrared-based phase image and an RGB-based intensity image can be acquired by maintaining the stained sample in the state placed on the sample slide (12) and having the illumination module (11) alternately irradiate near-infrared and R, G, B (or white light). At this time, since the sample remains in the state placed on the sample slide (12), it is not necessary to separately align the acquired near-infrared-based phase image and the RGB-based intensity image. That is, the training data set can be acquired very easily.

[0060] Meanwhile, the virtual staining module (20) is implemented with a pre-trained neural network model and receives a phase image obtained from the sample image acquisition module (10), performs neural network operations to obtain a virtual staining image similar to the staining image obtained from the stained sample.

[0061] And the pathology sample virtual staining device according to the embodiment may further include a learning module (30) that is used only during the learning process to train a virtual staining module (20) implemented as a neural network model. During the learning process, the learning module (30) receives a virtual staining image converted by the virtual staining module (20) along with an RGB-based intensity image of a sample acquired by the sample image acquisition module (10) and a near-infrared-based phase image, compares them with each other, calculates a loss based on the comparison result, and backpropagates to train the virtual staining module (20).

[0062] The learning module (30) may include a staining image acquisition module (31) and a backpropagation module (32). The staining image acquisition module (31) acquires an RGB-based intensity image by combining intensity images acquired for each RGB color from the stained sample by the sample image acquisition module (10) during the learning process. The RGB-based intensity image can be used as a truth value to train the virtual staining module (20) as a stained sample image.

[0063] Here, it is assumed that the lighting module (11) individually irradiates light according to each color of R, G, and B to acquire color-specific intensity images for each of R, G, and B, so the dyeing image acquisition module (31) is described as acquiring a dyed sample image by combining the color-specific intensity images of R, G, and B. However, as described above, if the lighting module (11) irradiates white light in which RGB is synthesized and the sensing module (14) is configured to detect the deformed light transmitted through the sample, the intensity image acquired by the image acquisition module (15) can be used as is as the dyed sample image, so the dyeing image acquisition module (31) may be omitted.

[0064] The backpropagation module (32) calculates the difference between the stained image obtained as a truth value and the virtual stained image obtained from the virtual staining module (20) as a loss, and trains the virtual staining module (20) implemented as a deep neural network by backpropagating the calculated loss to the virtual staining module (20). Here, the loss can be calculated in various ways, and since a method of calculating the loss based on the difference between previously disclosed images can be used, a detailed explanation is omitted here.

[0065] Consequently, the pathology sample virtual staining device according to the embodiment can easily acquire virtual staining images by acquiring near-infrared-based phase images so as not to be affected by the staining state of the sample. Therefore, accurate pathology diagnosis can be performed using the virtual staining images acquired by the pathology sample virtual staining device without staining newly collected unlabeled samples or re-staining decolorized samples that have been interpolated for a long period. In addition, since near-infrared-based phase images and RGB-based intensity images can be acquired together by utilizing the stained sample as is during training, a large amount of training data sets can be smoothly acquired without a separate image registration process, thereby significantly reducing the time and cost required for training.

[0066] In the illustrated embodiments, each component may have different functions and capabilities in addition to those described above and may include additional components not described. Additionally, in one embodiment, each component may be implemented using one or more physically separated devices, or by one or more processors or a combination of one or more processors and software, and may not be clearly distinguished in specific operation as in the illustrated examples.

[0067] And the pathology sample virtual staining device illustrated in FIG. 2 may be implemented within a logic circuit by hardware, firmware, software, or a combination thereof, or may be implemented using a general-purpose or specific-purpose computer. The device may be implemented using a hardwired device, a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), etc. Additionally, the device may be implemented as a system-on-chip (SoC) including one or more processors and controllers.

[0068] In addition, the virtual staining device for pathological samples may be installed on a computing device or server equipped with hardware elements in the form of software, hardware, or a combination thereof. A computing device or server may refer to various devices that include, in whole or in part, communication devices such as communication modems for communicating with various devices or wired / wireless communication networks, memory for storing data for executing programs, and microprocessors for executing programs to perform calculations and commands.

[0069] FIG. 5 illustrates a method for virtual staining of a pathological sample according to one embodiment, FIG. 6 is a diagram for explaining the learning step of FIG. 5, and FIG. 7 is a diagram for explaining the virtual staining step of FIG. 5.

[0070] Referring to FIG. 5, a method for virtual staining of a pathological sample according to one embodiment is broadly divided into a learning step of learning a virtual staining module (20) and a step of virtual staining a near-infrared-based phase image of a sample using the learned virtual staining module (20).

[0071] In the learning phase, a learning sample for acquiring a learning data set is first placed on the sample slide (12) of the sample image acquisition module (10) (51). Here, the learning sample may be a dyed sample (S) from which an RGB-based intensity image used as a truth value can be acquired. As an example, the dyed sample (S) may be a sample dyed using the H&E dyeing technique as described above.

[0072] Then, when a dyed sample (S) is placed on the sample slide (12) of the sample image acquisition module (10) as a learning sample, the lighting module (11) is set to irradiate RGB light (52). Then, when the lighting module (11) irradiates the set RGB light and the light transmitted through the sample is incident on the sensing module (14) to generate a detection signal, an RGB-based intensity image to be used as a truth value during learning is acquired as a learning intensity image according to the generated detection signal (53). In FIG. 5, for convenience, it is described that the lighting module (11) irradiates RGB light and acquires a learning intensity image according to the irradiated RGB light. However, as described above, the learning intensity image can be acquired by synthesizing the R, G, and B-based intensity images obtained from the R, G, and B-colored light obtained sequentially by the lighting module (11) irradiating R, G, and B-colored light.

[0073] When an RGB-based intensity image to be used as a truth value is obtained, the lighting is set so that the lighting module (11) irradiates near-infrared light while the stained sample placed on the sample slide (12) remains as is (54). Then, the near-infrared light transmitted through the sample is detected to obtain a learning phase image (55). The learning intensity image and the learning phase image obtained here are used as a learning data set to train the virtual staining module (20).

[0074] That is, during learning, as shown in Fig. 6, an RGB-based intensity image (|S) from a stained sample RGB |) and near-infrared-based phase image (∠S NIR ) together to create a training dataset.

[0075] When a training data set is obtained, as shown in FIG. 6, a training phase image among the training data set is input into a virtual staining module (20) that is the target of training, and a neural network operation is performed to obtain a virtual stained image ( It is converted into ) (56). The virtual stained image obtained therefrom ( RGB-based training intensity images (|S) obtained with ) and truth values RGB The virtual staining module (20) is trained (57) by comparing the values ​​of |) to calculate the loss and backpropagating the calculated loss. Afterward, it is determined whether to terminate the training (58). Training can be terminated if the calculated loss is less than or equal to the reference loss, or if the number of training iterations is greater than or equal to the reference number. If it is determined that training is not to be terminated, the training step can be performed again. However, if it is determined that training is to be terminated, the virtual staining step is performed.

[0076] In the virtual staining step, a sample (F) to be virtually stained is first placed on a sample slide (12) (61). Here, the sample may be an unlabeled sample that has not been stained or a decolorized sample that has been decolorized after staining.

[0077] When the sample (F) is placed, the lighting module (11) irradiates near-infrared light toward the sample (F) (62). Then, by detecting the near-infrared light that has passed through the sample (F), a phase image (∠F) of the sample is formed. NIR ) is obtained (63). A near-infrared-based phase image (∠F) of the sample (F) is obtained. NIR When ) is acquired, the acquired near-infrared-based phase image (∠F NIR By inputting ) into the learned virtual staining module (20) and converting it into a neural network operation, the virtual stained image ( ) obtains (64).

[0078] Although FIG. 5 describes each process as being executed sequentially, this is merely an illustrative description, and a person skilled in the art can apply various modifications and variations by changing the order described in FIG. 5, executing one or more processes in parallel, or adding other processes, within the scope of not departing from the essential characteristics of the embodiment of the present invention.

[0079] FIG. 8 is a drawing for explaining a computing environment including a computing device according to one embodiment.

[0080] In the illustrated embodiments, each component may have different functions and capabilities in addition to those described below, and may include additional components in addition to those described below. The illustrated computing environment (70) may include a computing device (71) to perform the pathology sample virtual staining method illustrated in FIG. 8. In one embodiment, the computing device (71) may be one or more components included in the pathology sample virtual staining device illustrated in FIG. 2.

[0081] A computing device (71) includes at least one processor (72), a computer-readable storage medium (73), and a communication bus (75). The processor (72) can cause the computing device (71) to operate according to the exemplary embodiment described above. For example, the processor (72) can execute one or more programs (74) stored in the computer-readable storage medium (73). The one or more programs (74) may include one or more computer-executable instructions, and the computer-executable instructions may be configured to cause the computing device (71) to perform operations according to the exemplary embodiment when executed by the processor (72).

[0082] The communication bus (75) interconnects various other components of the computing device (71), including the processor (72) and the computer-readable storage medium (73).

[0083] The computing device (71) may also include one or more input / output interfaces (76) and one or more communication interfaces (77) that provide an interface for one or more input / output devices (78). The input / output interfaces (76) and the communication interfaces (77) are connected to a communication bus (75). The input / output devices (78) may be connected to other components of the computing device (71) through the input / output interfaces (76). An exemplary input / output device (78) may include an input device such as a pointing device (such as a mouse or trackpad), a keyboard, a touch input device (such as a touchpad or touchscreen), a voice or sound input device, various types of sensor devices and / or imaging devices, and / or an output device such as a display device, a printer, a speaker and / or a network card. An exemplary input / output device (78) may be included inside the computing device (71) as a component constituting the computing device (71), or it may be connected to the computing device (71) as a separate device distinct from the computing device (71).

[0084] Although the present invention has been described in detail above through representative embodiments, those skilled in the art will understand that various modifications and equivalent alternative embodiments are possible therefrom. Accordingly, the true technical scope of protection of the present invention should be determined by the technical spirit of the appended claims.

Claims

Claim 1 A sample image acquisition module that irradiates near-infrared light toward a placed sample to acquire a near-infrared-based phase image of the sample; and a virtual staining module that receives the near-infrared-based phase image and performs neural network computation to acquire a virtual stained image similar to an image of a stained sample in which the sample is stained, wherein the virtual staining module is trained using a near-infrared-based learning phase image acquired for a learning sample, which is a sample pre-stained for learning, and an RGB-based learning intensity image acquired by irradiating RGB light toward the learning sample as a learning data set, and the sample image acquisition module comprises a sample slide on which the sample is placed; an illumination module that irradiates near-infrared light toward the sample slide; and a sensing module that detects light irradiated from the illumination module and transmitted through the sample to generate a detection signal. A pathology sample virtual staining device comprising an image acquisition module that receives the detection signal and generates the near-infrared-based phase image, wherein the illumination module comprises a plurality of light sources emitting near-infrared light and R, G, and B light arranged in an array form, and sequentially turns on the arranged light sources to apply light with an incident angle controlled to be incident on the sample slide, and acquires the near-infrared-based phase image and the learning intensity image through Fourier Typography imaging for near-infrared light and R, G, and B light of different incident angles. Claim 2 In claim 1, the virtual staining module is a pathology sample virtual staining device that is trained using a near-infrared-based learning phase image obtained for a learning sample, which is a sample pre-stained for learning, and an RGB-based learning intensity image obtained by irradiating RGB light toward the learning sample as a learning data set. Claim 3 In paragraph 2, the virtual staining module acquires a learning virtual staining image by performing neural network operations when a near-infrared-based learning phase image is input during learning, and a loss calculated by comparing the acquired learning virtual staining image with the learning intensity image is backpropagated to learn a pathology sample virtual staining device. Claim 4 delete Claim 5 In paragraph 2, the sample image acquisition module acquires intensity images for each of the R, G, and B colors by irradiating light according to each of the R, G, and B colors toward the sample during learning, and synthesizes the acquired intensity images for each of the R, G, and B colors to acquire the learning intensity image, thereby forming a virtual staining device for a pathology sample. Claim 6 In claim 1, the sample image acquisition module is a pathology sample virtual staining device that acquires near-infrared-based phase images of an unlabeled sample that is not stained or a decolorized sample that has been stained and then decolorized. Claim 7 delete Claim 8 delete Claim 9 A pathology sample virtual staining device according to claim 1, wherein a plurality of light sources emitting near-infrared light and R, G, and B light are implemented as a plurality of replaceable modules. Claim 10 delete Claim 11 delete Claim 12 delete Claim 13 delete Claim 14 delete Claim 15 delete Claim 16 delete

Images