Information processing apparatus, information processing method, method for generating learning model, and program

Abstract

To provide an information processing apparatus capable of more easily reducing fluorescence crosstalk of an image acquired by a fluorescence microscope.SOLUTION: The information processing apparatus 10 according to the present disclosure includes a control unit 11 for acquiring a learning model constructed by learning a third image according to a fourth image on the basis of learning data in which the third image is related to the fourth image including fluorescence crosstalk, the third image not including a fluorescence crosstalk acquired by a fluorescence microscope 1, and for generating a second image in which fluorescence crosstalk of a first image of a sample (S) having been acquired by a fluorescence microscope (1), is reduced based on the acquired learning model.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, a learning model generation method, and a program. [Background technology]

[0002] Conventionally, techniques related to image processing of images acquired by a fluorescence microscope have been known. For example, Patent Document 1 discloses a spectroscopic optical unit that eliminates pixel shifts between single-color images when acquiring a multicolor image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2008-065144 Summary of the Invention [Problem to be solved by the invention]

[0004] The conventional technology described in Patent Document 1 leaves room for improvement in terms of reducing fluorescent crosstalk in images acquired by a fluorescent microscope.

[0005] The present disclosure aims to provide an information processing device, an information processing method, a learning model generation method, and a program that can more easily reduce fluorescent crosstalk in images acquired by a fluorescence microscope. [Means for solving the problem]

[0006] An information processing device according to some embodiments includes a control unit that acquires a learning model constructed by learning a third image corresponding to a fourth image that does not contain fluorescent crosstalk and is acquired by a fluorescence microscope based on learning data that associates the third image with the fourth image that contains the fluorescent crosstalk, and generates a second image in which the fluorescent crosstalk of a first image of a sample acquired by the fluorescence microscope has been reduced based on the acquired learning model.

[0007] This makes it easier to reduce fluorescence crosstalk in images acquired by a fluorescence microscope. The information processing device acquires a learning model constructed by learning a third image corresponding to a fourth image based on the learning data. The information processing device generates a second image from the first image based on the acquired learning model. As a result, unlike conventional technology, the information processing device can reduce fluorescence crosstalk using only images without acquiring fluorescence spectra. The information processing device can more easily reduce fluorescence crosstalk during simultaneous fluorescence imaging at multiple wavelengths, without relying on changes in the fluorescence spectra of the fluorescent dyes used in the sample. The information processing device can accurately and simply reduce fluorescence crosstalk.

[0008] In one embodiment, the control unit may acquire each of the first image and the fourth image as fluorescence images of multiple wavelengths when excitation light of multiple wavelengths is simultaneously irradiated onto the sample by an optical system of the fluorescence microscope, thereby enabling the information processing device to efficiently acquire fluorescence images of different wavelengths in a short time both in an observation stage in which the sample is actually observed and in a learning stage in which learning data is acquired and a learning model is constructed.

[0009] In one embodiment, the control unit may acquire the third image as a fluorescence image of a single wavelength when the excitation light is irradiated onto the sample by the optical system for each of the plurality of wavelengths, thereby enabling the information processing device to accurately acquire a third image that does not include any fluorescence crosstalk as training data.

[0010] In one embodiment, the information processing device may include a pair of images captured in the same field of view by the fluorescence microscope, where the third image and the fourth image are captured. This allows the information processing device to acquire a pair of images in the same field of view, where the positional information of the sample in the images is maintained. This improves learning accuracy when constructing a learning model based on the third image and the fourth image. The information processing device can accurately generate a second image with reduced fluorescence crosstalk from a first image of the sample captured by the fluorescence microscope.

[0011] In one embodiment, the control unit may acquire multiple sets of images captured by the fluorescence microscope in multiple different fields of view for each set. This allows the information processing device to acquire training data including more sets of third and fourth images. Increasing the amount of training data improves the learning accuracy when constructing a learning model based on the third and fourth images. The information processing device can accurately generate a second image with reduced fluorescence crosstalk from a first image of a sample acquired by the fluorescence microscope.

[0012] In one embodiment of the information processing device, the control unit may acquire a set of the first image and the third and fourth images, in which at least one of the intensity and irradiation time of excitation light irradiated onto the sample by the optical system of the fluorescence microscope is different from each other.

[0013] This allows the information processing device to increase the intensity of the excitation light and / or lengthen the irradiation time when capturing a set of third and fourth images used for learning, thereby obtaining images with a high S / N ratio. This improves the learning accuracy when constructing a learning model based on the third and fourth images. Meanwhile, the information processing device can also reduce damage to the sample caused by the excitation light by decreasing the intensity of the excitation light and / or shortening the irradiation time when capturing the first image used for actual observation. Additionally, by shortening the irradiation time of the excitation light, the information processing device can also shorten the exposure time of the imaging camera included in each light-receiving unit, thereby shortening the imaging time.

[0014] In one embodiment of the information processing device, the fluorescence microscope may be a confocal microscope, which is capable of acquiring high-resolution images with little blur and high contrast.

[0015] An information processing method according to some embodiments is an information processing method executed by an information processing device, and includes: acquiring a learning model constructed by learning a third image corresponding to a fourth image that does not contain fluorescent crosstalk and is acquired by a fluorescence microscope, based on learning data associating the third image with the fourth image that contains the fluorescent crosstalk; and generating a second image in which the fluorescent crosstalk of a first image of a sample acquired by the fluorescence microscope has been reduced, based on the acquired learning model.

[0016] This makes it easier to reduce fluorescence crosstalk in images acquired by a fluorescence microscope. The information processing device acquires a learning model constructed by learning a third image corresponding to a fourth image based on the learning data. The information processing device generates a second image from the first image based on the acquired learning model. As a result, unlike conventional technology, the information processing device can reduce fluorescence crosstalk using only images without acquiring fluorescence spectra. The information processing device can more easily reduce fluorescence crosstalk during simultaneous fluorescence imaging at multiple wavelengths, without relying on changes in the fluorescence spectra of the fluorescent dyes used in the sample. The information processing device can accurately and simply reduce fluorescence crosstalk.

[0017] A method for generating a learning model according to some embodiments is a method for generating the learning model used in the information processing method described above, and includes acquiring the learning data and constructing the learning model by learning the third image corresponding to the fourth image based on the acquired learning data. This allows the information processing device to acquire the learning data and construct the learning model by itself. Therefore, the information processing device can complete the processes from the learning stage of constructing the learning model to the observation stage of actually observing the sample within a single device.

[0018] In one embodiment, in the method for generating a learning model, the learning model may be a machine learning model trained based on the acquired learning data. This improves learning accuracy when constructing the learning model based on the third image and the fourth image. The information processing device can accurately generate a second image with reduced fluorescence crosstalk from a first image of a sample acquired by a fluorescence microscope.

[0019] A program according to some embodiments causes the information processing device to execute any one of the above-described information processing method and the above-described method for generating a learning model.

[0020] This makes it easier to reduce fluorescence crosstalk in images acquired by a fluorescence microscope. The information processing device acquires a learning model constructed by learning a third image corresponding to a fourth image based on the learning data. The information processing device generates a second image from the first image based on the acquired learning model. As a result, unlike conventional technology, the information processing device can reduce fluorescence crosstalk using only images without acquiring fluorescence spectra. The information processing device can more easily reduce fluorescence crosstalk during simultaneous fluorescence imaging at multiple wavelengths, without relying on changes in the fluorescence spectra of the fluorescent dyes used in the sample. The information processing device can accurately and simply reduce fluorescence crosstalk. [Effects of the Invention]

[0021] According to the present disclosure, it is possible to provide an information processing device, an information processing method, a learning model generation method, and a program that can more easily reduce fluorescent crosstalk in images acquired by a fluorescence microscope. [Brief explanation of the drawings]

[0022] [Figure 1] FIG. 1 is a schematic diagram illustrating an example of the configuration of a fluorescence microscope including an information processing device according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is a first schematic diagram for explaining an example of the operation of the information processing device in FIG. [Figure 3] 1. FIG. 4 is a second schematic diagram illustrating an example of the operation of the information processing device in FIG. [Figure 4] FIG. 10 is a third schematic diagram illustrating an example of the operation of the information processing device in FIG. [Figure 5] FIG. 4 is a fourth schematic diagram illustrating an example of the operation of the information processing device in FIG. [Figure 6] FIG. 5 is a fifth schematic diagram illustrating an example of the operation of the information processing device in FIG. [Figure 7] FIG. 6 is a sixth schematic diagram illustrating an example of the operation of the information processing device in FIG. [Figure 8] FIG. 7 is a seventh schematic diagram illustrating an example of the operation of the information processing device in FIG. [Figure 9] 2 is a flowchart illustrating an example of an information processing method executed by the information processing device of FIG. 1. [Figure 10] FIG. 1 is a schematic diagram for explaining problems with the prior art. DETAILED DESCRIPTION OF THE INVENTION

[0023] The background and problems of the prior art will now be described in more detail.

[0024] In conventional fluorescence microscopes, excitation light of multiple wavelengths is irradiated onto a sample, such as a cell stained with multiple fluorescent dyes, and multiple fluorescent lights corresponding to the wavelengths of the excitation light are emitted from the sample and detected. The multiple fluorescent lights emitted from the sample pass through various optical elements and reach a wavelength-separating optical element, such as a dichroic mirror. The dichroic mirror transmits fluorescent lights with one wavelength and reflects fluorescent lights with other wavelengths. The multiple wavelength-separated fluorescent lights are then imaged on imaging devices, such as multiple cameras.

[0025] 10 is a schematic diagram for explaining the problems of the conventional technology. With reference to FIG. 10, the wavelength separation of the fluorescence in the dichroic mirror will be mainly explained.

[0026] As shown in FIG. 10, it is assumed that the dichroic mirror has wavelength characteristics as indicated by the dashed line. In this case, fluorescence a and fluorescence b are not completely separated from each other by the dichroic mirror, and a portion of fluorescence b is mixed into the optical path of fluorescence a guided to one camera. In the present disclosure, this mixing of a portion of fluorescence is referred to as "fluorescence crosstalk." A portion of fluorescence b due to fluorescence crosstalk is not removed even by a bandpass filter placed in front of one camera. Meanwhile, a portion of fluorescence a is similarly mixed into the optical path of fluorescence b guided to another camera. A portion of fluorescence a due to fluorescence crosstalk is not removed even by a bandpass filter placed in front of another camera.

[0027] As described above, with conventional fluorescence microscopes, it has been difficult to accurately capture fluorescence images due to fluorescence crosstalk.

[0028] As a method for solving such problems, an image processing technique such as the conventional technique described in Patent Document 1 is known. In this conventional technique, a fluorescence image that expresses only the individual emission of each fluorescent dye is obtained by performing matrix division on the fluorescence spectrum of each fluorescent dye for each pixel of a plurality of fluorescence images in which fluorescence is mixed.

[0029] However, such conventional techniques have the following problems. Generally, the fluorescence spectrum of a fluorescent dye changes depending on the state of the molecules contained in the fluorescent dye. The state of the molecules changes depending on conditions such as the ambient pH, temperature, and the molecule to which the molecule is bound. In other words, the fluorescence spectrum changes with each experiment. Therefore, in order to accurately perform the image processing method described in Patent Document 1, it is necessary to acquire a fluorescence spectrum for each experiment. However, acquiring an accurate fluorescence spectrum requires a great deal of time and effort.

[0030] In order to solve the above-mentioned problems, the present disclosure aims to provide an information processing device, a fluorescence microscope, an information processing method, a method for generating a learning model, and a program that can more easily reduce fluorescent crosstalk in images acquired by a fluorescence microscope.

[0031] Hereinafter, one embodiment of the present disclosure will be mainly described with reference to the accompanying drawings.

[0032] 1 is a schematic diagram showing an example of the configuration of a fluorescence microscope 1 including an information processing device 10 according to an embodiment of the present disclosure. An example of the configuration and functions of a fluorescence microscope 1 including an information processing device 10 according to an embodiment will be mainly described with reference to FIG.

[0033] A fluorescence microscope 1 according to one embodiment is, for example, a confocal microscope. The fluorescence microscope 1 includes an information processing device 10 and an optical system 20 connected to the information processing device 10. The optical system 20, for example, simultaneously irradiates a sample S with excitation light L of multiple wavelengths. In the present disclosure, "excitation light L of multiple wavelengths" may mean, for example, multiple excitation light L corresponding to multiple different wavelengths, or a single excitation light L having a wide optical spectrum encompassing multiple different wavelengths. "Irradiating simultaneously" may mean that the start and end points of irradiation are exactly the same, or that at least one of the start and end points of irradiation is different while the irradiation times overlap within a predetermined time range.

[0034] The fluorescence microscope 1 uses the optical system 20 to obtain fluorescent images of the sample S at multiple wavelengths, each corresponding to a multiple wavelength of excitation light L. In the present disclosure, the term "fluorescent images at multiple wavelengths" refers to, for example, multiple images based on multiple fluorescent lights corresponding to multiple different wavelengths emitted from the sample S stained with multiple fluorescent dyes.

[0035] The optical system 20 includes an irradiation unit 21, a first dichroic mirror 22, an objective lens 23, a second dichroic mirror 24, a first light receiving unit 25a, and a second light receiving unit 25b.

[0036] The irradiation unit 21 has a light source such as a laser or a light-emitting diode (LED). In addition to the light source, the irradiation unit 21 may further have a focusing element such as a lens. The irradiation unit 21 irradiates the first dichroic mirror 22 with excitation light L having wavelengths corresponding to multiple fluorescent dyes that stain the sample S to be observed using the fluorescence microscope 1. The excitation light L includes, for example, a first excitation light L1 and a second excitation light L2. The light source of the irradiation unit 21 may irradiate the first excitation light L1 having a first wavelength and the second excitation light L2 having a second wavelength individually or simultaneously based on a control signal from the information processing device 10. The wavelength of the excitation light L irradiated by the irradiation unit 21 is, for example, within the visible range. However, the wavelength of the excitation light L may be within the ultraviolet range, the near-infrared range, or other infrared ranges, for example.

[0037] The first dichroic mirror 22 is a mirror that reflects the excitation light L irradiated from the irradiation unit 21 toward the objective lens 23 and transmits the fluorescence E, which will be described later, emitted from the sample S toward the second dichroic mirror 24.

[0038] The objective lens 23 is disposed to face the sample S. The objective lens 23 guides the first excitation light L1 and the second excitation light L2 reflected by the first dichroic mirror 22 to the sample S. The objective lens 23 collects the first fluorescence E1 emitted from the sample S in response to the first excitation light L1 and the second fluorescence E2 emitted from the sample S in response to the second excitation light L2, and forms images on the first light receiving unit 25a and the second light receiving unit 25b, respectively. At this time, an imaging lens (not shown) may be used to form images on each light receiving unit.

[0039] The second dichroic mirror 24 is a mirror that receives the fluorescence E, including the first fluorescence E1 and the second fluorescence E2, that has been transmitted through the first dichroic mirror 22. The second dichroic mirror 24 reflects the first fluorescence E1 toward the first light receiving unit 25a and transmits the second fluorescence E2 toward the second light receiving unit 25b. The second dichroic mirror 24 separates the wavelengths of the first fluorescence E1 and the second fluorescence E2.

[0040] The first light receiving unit 25a has a photodetector such as a first imaging camera. The first light receiving unit 25a receives the first fluorescence E1 from the sample S, which is collected by the objective lens 23. The first imaging camera of the first light receiving unit 25a converts the first fluorescence E1, which is imaged by the objective lens 23, into an electrical signal and outputs it to the information processing device 10 as first fluorescence image information. The wavelength band that can be received by the first imaging camera of the first light receiving unit 25a includes wavelength bands corresponding to the multiple fluorescent dyes that stain the sample S. In addition to the above configuration, the first light receiving unit 25a may further have other optical elements, such as a band-pass filter.

[0041] The second light receiving unit 25b has a photodetector such as a second imaging camera. The second light receiving unit 25b receives the second fluorescence E2 from the sample S, which is collected by the objective lens 23. The second imaging camera of the second light receiving unit 25b converts the second fluorescence E2, which is imaged by the objective lens 23, into an electrical signal and outputs it to the information processing device 10 as second fluorescence image information. The wavelength band that can be received by the second imaging camera of the second light receiving unit 25b includes wavelength bands corresponding to the multiple fluorescent dyes that stain the sample S. In addition to the above configuration, the second light receiving unit 25b may further have other optical elements, such as a band-pass filter.

[0042] The information processing device 10 includes any processing device that is integrally built into the fluorescence microscope 1 together with the optical system 20, or that is externally electrically connected to the optical system 20. The information processing device 10 may be configured, for example, as a device having dedicated digital circuits that can realize the functions described below. The information processing device 10 may include any general-purpose electronic device, such as a PC (Personal Computer), a tablet PC, a smartphone, or a smartwatch. Without being limited to these, the information processing device 10 may also include other electronic devices dedicated to the fluorescence microscope 1.

[0043] The information processing device 10 includes a control unit 11 and a storage unit 12.

[0044] The control unit 11 includes one or more processors, one or more programmable circuits, one or more dedicated circuits, or a combination thereof. In this disclosure, a "processor" refers to, but is not limited to, a general-purpose processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), or a dedicated processor specialized for a specific process. A "programmable circuit" refers to, but is not limited to, a FPGA (Field-Programmable Gate Array). A "dedicated circuit" refers to, but is not limited to, an ASIC (Application Specific Integrated Circuit). The control unit 11 is communicatively connected to the memory unit 12 of the information processing device 10, and the irradiation unit 21, first light receiving unit 25a, and second light receiving unit 25b of the optical system 20, and controls the operation of each component.

[0045] The storage unit 12 includes storage devices such as a hard disk drive (HDD), a solid state drive (SSD), an electrically erasable programmable read-only memory (EEPROM), a read-only memory (ROM), and a random access memory (RAM). The storage unit 12 stores information necessary to realize the operation of the information processing device 10. The storage unit 12 stores information obtained by the operation of the information processing device 10. For example, the storage unit 12 stores system programs, application programs, and various data obtained by any means such as communication.

[0046] The storage unit 12 may function as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 12 is not limited to being built into the information processing device 10, and may include an external storage device connected via a digital input / output port such as a USB (Universal Serial Bus).

[0047] The control unit 11 outputs a control signal to the first imaging camera of the first light receiving unit 25a, and acquires a first fluorescence image based on the first fluorescence E1. The control unit 11 stores image data of the first fluorescence image in the storage unit 12. The control unit 11 outputs a control signal to the second imaging camera of the second light receiving unit 25b, and acquires a second fluorescence image based on the second fluorescence E2. The control unit 11 stores image data of the second fluorescence image in the storage unit 12.

[0048] The control unit 11 generates a second image in which the fluorescent crosstalk of the first image of the sample S acquired by the fluorescence microscope 1 has been reduced. The control unit 11 acquires a learning model constructed by learning the third image corresponding to the fourth image based on learning data that associates a third image that does not include fluorescent crosstalk with a fourth image that includes fluorescent crosstalk. The control unit 11 generates a second image from the first image based on the acquired learning model.

[0049] For example, the control unit 11 constructs a learning model by learning a third image corresponding to a fourth image based on the acquired learning data. That is, the control unit 11 acquires the learning model by constructing the learning model in the information processing device 10 itself. The control unit 11 stores the constructed learning model in the storage unit 12.

[0050] The learning model is a machine learning model based on deep learning or the like, trained based on acquired training data. For example, the learning model is a supervised learning model, which is a mathematical model that uses a fourth image as input data and a third image as training data to learn a third image corresponding to the fourth image. The supervised learning model is, for example, but is not limited to, a convolutional neural network including an input layer, one or more hidden layers, and an output layer.

[0051] The learning of the supervised learning model is performed by the control unit 11. The learning of the supervised learning model may be batch learning or online learning. In the present disclosure, "constructing a learning model" means a state in which batch learning has been completed or online learning has been performed to a certain extent. In the case of online learning, learning may be continued even after learning has been performed to a certain extent.

[0052] The above-mentioned processing by the control unit 11 will be explained below in order.

[0053] Fig. 2 is a first schematic diagram for explaining an example of the operation of the information processing device 10 in Fig. 1. The following mainly explains the process of acquiring a third image corresponding to the first fluorescence image P13, which is executed by the control unit 11 of the information processing device 10. The control unit 11 acquires the third image as a fluorescence image of a single wavelength when excitation light L is irradiated onto the sample S by the optical system 20 for each single wavelength among the multiple wavelengths.

[0054] The control unit 11 controls the irradiation unit 21 to irradiate the first excitation light L1. The first excitation light L1 is reflected by the first dichroic mirror 22, passes through the objective lens 23, and is guided to the sample S. One of the multiple fluorescent dyes that stain the sample S is excited by the first excitation light L1 and emits first fluorescence E1. The first fluorescence E1 passes through the objective lens 23, transmits through the first dichroic mirror 22, is reflected by the second dichroic mirror 24, and is incident on the first imaging camera of the first light receiving unit 25a.

[0055] The control unit 11 stores the first fluorescence image P13 acquired from the first light receiving unit 25a as a third image in the storage unit 12. The first fluorescence image P13 is an image obtained using only the first fluorescence E1 and does not include the second fluorescence E2. The control unit 11 acquires a plurality of third images of the sample S captured in a plurality of different fields of view in the fluorescence microscope 1.

[0056] Fig. 3 is a second schematic diagram for explaining an example of the operation of the information processing device 10 in Fig. 1. With reference to Fig. 3, the process of acquiring a third image corresponding to the second fluorescence image P23, which is executed by the control unit 11 of the information processing device 10, will be mainly explained.

[0057] The control unit 11 controls the irradiation unit 21 to irradiate the second excitation light L2. The second excitation light L2 is reflected by the first dichroic mirror 22, passes through the objective lens 23, and is guided to the sample S. Other fluorescent dyes among the multiple fluorescent dyes that stain the sample S are excited by the second excitation light L2 and emit second fluorescence E2. The second fluorescence E2 passes through the objective lens 23, transmits through the first dichroic mirror 22, and further transmits through the second dichroic mirror 24, and is incident on the second imaging camera of the second light receiving unit 25b.

[0058] The control unit 11 stores the second fluorescence image P23 acquired from the second light receiving unit 25b as a third image in the storage unit 12. The second fluorescence image P23 is an image obtained only from the second fluorescence E2 and does not include the first fluorescence E1. The control unit 11 acquires multiple third images of the sample S captured in multiple different fields of view in the fluorescence microscope 1. The multiple fields of view at this time are the same as the multiple fields of view described above with reference to FIG. 2 in the acquisition process of the third image corresponding to the first fluorescence image P13.

[0059] 4 is a third schematic diagram for explaining an example of the operation of the information processing device 10 of FIG. 4, the process of acquiring a fourth image executed by the control unit 11 of the information processing device 10 will be mainly described with reference to FIG. The control unit 11 acquires the fourth image as a fluorescent image of multiple wavelengths when excitation light L of multiple wavelengths is simultaneously irradiated onto the sample S by the optical system 20 of the fluorescence microscope 1. As an example, the operation of simultaneously acquiring fluorescence at two wavelengths by the information processing device 10 will mainly be described.

[0060] The control unit 11 controls the irradiation unit 21 to irradiate the first excitation light L1 and the second excitation light L2. The first excitation light L1 and the second excitation light L2 are reflected by the first dichroic mirror 22, pass through the objective lens 23, and are guided to the sample S. One of the multiple fluorescent dyes staining the sample S is excited by the first excitation light L1 and emits first fluorescence E1. The other of the multiple fluorescent dyes staining the sample S is excited by the second excitation light L2 and emits second fluorescence E2.

[0061] The first fluorescence E1 and the second fluorescence E2 pass through the objective lens 23 and are transmitted through the first dichroic mirror 22. The first fluorescence E1 is reflected by the second dichroic mirror 24 and enters the first imaging camera of the first light receiving unit 25a. At this time, a portion of the second fluorescence E2 is also reflected by the second dichroic mirror 24 and enters the first imaging camera of the first light receiving unit 25a. The second fluorescence E2 is transmitted through the second dichroic mirror 24 and enters the second imaging camera of the second light receiving unit 25b. At this time, a portion of the first fluorescence E1 is also transmitted through the second dichroic mirror 24 and enters the second imaging camera of the second light receiving unit 25b.

[0062] The control unit 11 stores the first fluorescence image P14 acquired from the first light receiving unit 25a in the storage unit 12 as a fourth image. The first fluorescence image P14 is an image in which the first fluorescence E1 includes fluorescence crosstalk from the second fluorescence E2. The control unit 11 stores the second fluorescence image P24 acquired from the second light receiving unit 25b in the storage unit 12 as a fourth image. The second fluorescence image P24 is an image in which the second fluorescence E2 includes fluorescence crosstalk from the first fluorescence E1.

[0063] The control unit 11 acquires a plurality of fourth images of the sample S, each captured in a plurality of different fields of view, in the fluorescence microscope 1. The plurality of fields of view at this time are the same as the plurality of fields of view described above using FIGS. 2 and 3 in the process of acquiring the third image. The third image and the fourth image are a set of images captured in the same field of view in the fluorescence microscope 1. The control unit 11 acquires a set of images, each set of images captured in a plurality of different fields of view in the fluorescence microscope 1.

[0064] 5 is a fourth schematic diagram for explaining an example of the operation of the information processing device 10 of FIG. 1. The control unit 11 includes an image processing unit 111 and a comparison unit 112 as multiple functional units that realize its functions. With reference to FIG. 5, a first example of the learning model construction process executed by the control unit 11 of the information processing device 10 will be mainly described. The learning process of the control unit 11 for generating, from the first fluorescence image P14, a second image similar to the first fluorescence image P13 in which fluorescence crosstalk has been reduced will be mainly described.

[0065] The image processing unit 111 of the control unit 11 references the first fluorescence image P14 stored in the storage unit 12. The image processing unit 111 may perform processing based on a convolutional neural network, which has, for example, an encoder that captures image features using multiple convolutional layers and a decoder that generates an image by performing inverse operations from the image features using deconvolutional layers with the same number of stages. The image processing unit 111 applies an action to remove fluorescence crosstalk from the second fluorescence E2 to the first fluorescence image P14, and outputs a first fluorescence image P141 in the middle of learning.

[0066] The comparison unit 112 of the control unit 11 refers to the first fluorescence image P13 and the first fluorescence image P141 in the middle of learning, both stored in the storage unit 12. The comparison unit 112 compares the first fluorescence image P13 with the first fluorescence image P141 in the middle of learning, and adjusts the internal parameters of the image processing unit 111 so that the first fluorescence image P141 approaches the first fluorescence image P13.

[0067] The control unit 11 repeatedly executes the above-described processing by the image processing unit 111 and the comparison unit 112. The control unit 11 executes the same processing on a pair of first fluorescence image P13 and first fluorescence image P14 captured in another field of view. As described above, the control unit 11 optimizes the image processing unit 111 through repeated learning.

[0068] Fig. 6 is a fifth schematic diagram for explaining an example of the operation of the information processing device 10 in Fig. 1. A second example of the learning model construction process executed by the control unit 11 of the information processing device 10 will be mainly described with reference to Fig. 6. The main description will be directed to the learning process of the control unit 11 that generates, from the second fluorescence image P24, a second image similar to the second fluorescence image P23 in which fluorescence crosstalk has been reduced.

[0069] The image processing unit 111 of the control unit 11 references the second fluorescence image P24 stored in the storage unit 12. The image processing unit 111 may perform processing based on a convolutional neural network, which has, for example, an encoder that captures image features using multiple convolutional layers and a decoder that generates an image by performing inverse operations from the image features using deconvolutional layers with the same number of stages. The image processing unit 111 applies an action to the second fluorescence image P24 to remove fluorescence crosstalk from the first fluorescence E1, and outputs a second fluorescence image P241 in the middle of learning.

[0070] The comparison unit 112 of the control unit 11 refers to the second fluorescence image P23 and the second fluorescence image P241 in the middle of learning, both of which are stored in the storage unit 12. The comparison unit 112 compares the second fluorescence image P23 with the second fluorescence image P241 in the middle of learning, and adjusts the internal parameters of the image processing unit 111 so that the second fluorescence image P241 approaches the second fluorescence image P23.

[0071] The control unit 11 repeatedly executes the above-described processing by the image processing unit 111 and the comparison unit 112. The control unit 11 executes the same processing on a pair of second fluorescence images P23 and P24 captured in another field of view. As described above, the control unit 11 optimizes the image processing unit 111 through repeated learning.

[0072] FIG. 7 is a sixth schematic diagram illustrating an example of the operation of the information processing device 10 of FIG. 1. FIG. 8 is a seventh schematic diagram illustrating an example of the operation of the information processing device 10 of FIG. 1. With reference to FIGS. 7 and 8, the following will mainly describe a process for generating a second image from a first image of the sample S acquired by the fluorescence microscope 1, which is executed by the control unit 11 of the information processing device 10 based on the constructed learning model. The control unit 11 acquires the first image as a fluorescent image of multiple wavelengths when excitation light L of multiple wavelengths is simultaneously irradiated onto the sample S by the optical system 20 of the fluorescence microscope 1. As an example, the following will mainly describe the operation of the information processing device 10 to simultaneously acquire fluorescence at two wavelengths and reduce fluorescence crosstalk.

[0073] 7, the control unit 11 controls the irradiation unit 21 to irradiate the first excitation light L1 and the second excitation light L2. The first excitation light L1 and the second excitation light L2 are reflected by the first dichroic mirror 22, pass through the objective lens 23, and are guided to the sample S. One of the multiple fluorescent dyes staining the sample S is excited by the first excitation light L1 and emits first fluorescence E1. The other of the multiple fluorescent dyes staining the sample S is excited by the second excitation light L2 and emits second fluorescence E2.

[0074] The first fluorescence E1 and the second fluorescence E2 pass through the objective lens 23 and are transmitted through the first dichroic mirror 22. The first fluorescence E1 is reflected by the second dichroic mirror 24 and enters the first imaging camera of the first light receiving unit 25a. At this time, a portion of the second fluorescence E2 is also reflected by the second dichroic mirror 24 and enters the first imaging camera of the first light receiving unit 25a. The second fluorescence E2 is transmitted through the second dichroic mirror 24 and enters the second imaging camera of the second light receiving unit 25b. At this time, a portion of the first fluorescence E1 is also transmitted through the second dichroic mirror 24 and enters the second imaging camera of the second light receiving unit 25b.

[0075] The control unit 11 stores the first fluorescence image P11 acquired from the first light receiving unit 25a as a first image in the storage unit 12. The first fluorescence image P11 is an image in which the first fluorescence E1 includes fluorescence crosstalk of the second fluorescence E2. The control unit 11 stores the second fluorescence image P21 acquired from the second light receiving unit 25b as a first image in the storage unit 12. The second fluorescence image P21 is an image in which the second fluorescence E2 includes fluorescence crosstalk of the first fluorescence E1.

[0076] 8, the image processing unit 111 for which a learning model has been constructed references the first fluorescence image P11 stored in the storage unit 12, and outputs, as the second image, the first fluorescence image P12 in which the fluorescence crosstalk due to the second fluorescence E2 has been reduced. The image processing unit 111 for which a learning model has been constructed references the second fluorescence image P21 stored in the storage unit 12, and outputs, as the second image, the second fluorescence image P22 in which the fluorescence crosstalk due to the first fluorescence E1 has been reduced.

[0077] Fig. 9 is a flowchart for explaining an example of an information processing method executed by the information processing device 10 of Fig. 1. With reference to Fig. 9, the example of the information processing method executed by the information processing device 10 of Fig. 1 will be mainly explained.

[0078] In step S100, the control unit 11 of the information processing device 10 acquires learning data that associates a third image that does not include fluorescent crosstalk with a fourth image that includes fluorescent crosstalk.

[0079] In step S101, the control unit 11 of the information processing device 10 constructs a learning model by learning a third image corresponding to the fourth image based on the learning data acquired in step S100. In this way, the control unit 11 acquires the learning model.

[0080] In step S102, the control unit 11 of the information processing device 10 controls the optical system 20 of the fluorescence microscope 1 to acquire a first image.

[0081] In step S103, the control unit 11 of the information processing device 10 generates a second image from the first image acquired in step S102, based on the learning model acquired in step S101.

[0082] According to the information processing device 10 of the embodiment described above, it is possible to more easily reduce fluorescence crosstalk in images acquired by the fluorescence microscope 1. The information processing device 10 acquires a learning model constructed by learning a third image corresponding to a fourth image based on learning data. The information processing device 10 generates a second image from a first image based on the acquired learning model. As described above, unlike conventional technology, the information processing device 10 is able to reduce fluorescence crosstalk using only images without acquiring fluorescence spectra. The information processing device 10 can more easily reduce fluorescence crosstalk during simultaneous fluorescence imaging at multiple wavelengths, without relying on changes in the fluorescence spectrum of the fluorescent dye used in the sample S. The information processing device 10 can accurately and simply reduce fluorescence crosstalk.

[0083] The information processing device 10 acquires each of the first image and the fourth image as fluorescent images of multiple wavelengths when excitation light L of multiple wavelengths is simultaneously irradiated onto the sample S by the optical system 20 of the fluorescence microscope 1. This allows the information processing device 10 to efficiently acquire fluorescent images of different wavelengths in a short time both in the observation stage in which the sample S is actually observed and in the learning stage in which learning data is acquired and a learning model is constructed.

[0084] The information processing device 10 acquires a third image as a fluorescent image of a single wavelength when excitation light L of each of the plurality of wavelengths is irradiated onto the sample S by the optical system 20. This allows the information processing device 10 to accurately acquire a third image that does not contain any fluorescent crosstalk as training data in the learning data.

[0085] The third image and the fourth image are a pair of images captured in the same field of view by the fluorescence microscope 1. This allows the information processing device 10 to acquire a pair of images in the same field of view, in which the positional information of the sample S in the images is maintained. This improves the learning accuracy when constructing a learning model based on the third image and the fourth image. The information processing device 10 can accurately generate a second image in which fluorescence crosstalk has been reduced from the first image of the sample S captured by the fluorescence microscope 1.

[0086] The information processing device 10 acquires a plurality of sets of images captured in a plurality of fields of view that differ from each other for each set using the fluorescence microscope 1. This enables the information processing device 10 to acquire training data that includes more sets of third and fourth images. Increasing the amount of training data improves the learning accuracy when constructing a training model based on the third and fourth images. The information processing device 10 can accurately generate a second image with reduced fluorescence crosstalk from the first image of the sample S acquired by the fluorescence microscope 1.

[0087] The fluorescence microscope 1 is a confocal microscope, and is therefore capable of acquiring high-resolution images with little blur and high contrast.

[0088] The information processing device 10 acquires learning data and constructs a learning model by itself, thereby enabling the processing from the learning stage of constructing the learning model to the observation stage of actually observing the sample S to be completed by a single device.

[0089] The learning model is a machine learning model trained based on the acquired learning data. This improves the learning accuracy when constructing the learning model based on the third image and the fourth image. The information processing device 10 can accurately generate a second image with reduced fluorescence crosstalk from the first image of the sample S acquired by the fluorescence microscope 1.

[0090] It will be apparent to those skilled in the art that the present disclosure may be embodied in other specific forms other than the above-described embodiments without departing from the spirit or essential characteristics thereof. Therefore, the foregoing description is illustrative and not limiting. The scope of the disclosure is defined not by the foregoing description but by the appended claims. All modifications within the range of equivalents of any modifications are intended to be embraced therein.

[0091] For example, the shape, pattern, size, arrangement, orientation, type, and number of each of the above-mentioned components are not limited to those shown in the above description and drawings. The shape, pattern, size, arrangement, orientation, type, and number of each component may be configured arbitrarily as long as the function can be realized. The components of the illustrated fluorescence microscope 1 and information processing device 10 are functional concepts, and the specific form of each component is not limited to those shown.

[0092] The functions contained in each of the above-mentioned components or steps can be rearranged so as not to cause logical contradictions, and multiple components or steps can be combined into one or divided.

[0093] For example, a general-purpose electronic device such as a smartphone or a computer can function as the information processing device 10 according to the embodiment described above. Specifically, a program describing the processing content for realizing each function of the information processing device 10 according to the embodiment is stored in the memory of the electronic device, and the program is read and executed by a processor of the electronic device. Therefore, the present disclosure can also be realized as a program executable by a processor.

[0094] Alternatively, the present disclosure may be realized as a non-transitory computer-readable medium storing a program executable by one or more processors to cause the information processing device 10 according to an embodiment to execute each function, etc. It should be understood that these are also included within the scope of the present disclosure.

[0095] In the above embodiment, the third image and the fourth image are described as a pair of images captured in the same field of view by the fluorescence microscope 1, but are not limited to this. The third image and the fourth image may be a pair of images captured in different fields of view by the fluorescence microscope 1, as long as the learning accuracy is maintained when constructing the learning model.

[0096] In the above embodiment, the information processing device 10 has been described as acquiring a plurality of sets of images captured in a plurality of different fields of view for each set using the fluorescence microscope 1, but this is not limited to this. The information processing device 10 may acquire a plurality of sets of images in the same field of view for each set, as long as the learning accuracy when constructing a learning model is maintained. For example, the information processing device 10 may perform image processing such as rotating the sample S on a set of a third image and a fourth image captured in one field of view, and acquire a plurality of sets of images. The information processing device 10 may increase the amount of learning data using a method based on such image processing.

[0097] In addition to the processing described in the above embodiment, the control unit 11 of the information processing device 10 may acquire a set of a first image and a third image and a fourth image in which at least one of the intensity and irradiation time of the excitation light L irradiated onto the sample S by the optical system 20 of the fluorescence microscope 1 is different from each other. In other words, at least one of the intensity and irradiation time of the excitation light L may be different from each other between the set of the third image and the fourth image used for learning by the image processing unit 111 and the first image used for actual observation.

[0098] This enables the information processing device 10 to acquire images with a high S / N ratio by increasing the intensity of the excitation light L and / or lengthening the irradiation time when capturing a set of the third and fourth images used for learning by the image processing unit 111. This improves the learning accuracy when constructing a learning model based on the third and fourth images. On the other hand, the information processing device 10 can also reduce damage to the sample S caused by the excitation light L by decreasing the intensity of the excitation light L and / or shortening the irradiation time when capturing the first image used for actual observation. In addition, by shortening the irradiation time of the excitation light L, the information processing device 10 can also shorten the exposure time of the imaging camera included in each light receiving unit and shorten the imaging time.

[0099] In the above embodiment, the fluorescence microscope 1 has been described as having the information processing device 10, but this is not limiting. The information processing device 10 does not have to be provided in the fluorescence microscope 1. For example, the information processing device 10 may be an external device communicably connected to the fluorescence microscope 1 via a network including a mobile communication network and the Internet. The external device may include, for example, one or more server devices that can communicate with each other.

[0100] In the above embodiment, the fluorescence microscope 1 is described as a confocal microscope, but is not limited to this. The fluorescence microscope 1 may be any other microscope capable of acquiring an image as a fluorescent image.

[0101] In the above embodiment, the information processing device 10 has been described as constructing a learning model by itself, but the present invention is not limited to this. The information processing device 10 may acquire a learning model by receiving an already constructed learning model from any other external device via a network including a mobile communication network and the Internet.

[0102] In the above embodiment, the learning model is described as a convolutional neural network, but is not limited to this. The learning model may be any other machine learning model. Alternatively, the learning model is described as a machine learning model trained based on acquired training data, but is not limited to this. The learning model may be a model based on any other statistical method other than machine learning.

[0103] In the above embodiment, the operation of the information processing device 10 has been described based on an example of simultaneous fluorescence imaging at two wavelengths, but the invention is not limited to this. The information processing device 10 can also be applied to simultaneous fluorescence imaging at three or more wavelengths, depending on the configuration of the optical system 20 of the fluorescence microscope 1.

[0104] The image processing unit 111 described in the above embodiment does not need to re-learn and construct a learning model if the sample S is cultured and stained under similar conditions, and may maintain the same learning model and apply the learning model to the first image containing fluorescent crosstalk. Without being limited to this, the image processing unit 111 may further execute a learning process to update the learning model.

[0105] In the above embodiment, the optical system 20 of the fluorescence microscope 1 has a plurality of light receiving units corresponding to the number of fluorescence E, and each light receiving unit individually performs fluorescence imaging, but this is not limiting. The optical system 20 may have only a single imaging camera, and may individually perform fluorescence imaging in different areas of the single imaging camera.

[0106] Some embodiments of the present disclosure will be described below as examples, however, it should be noted that the embodiments of the present disclosure are not limited to these. [Appendix 1] acquiring a learning model constructed by learning a third image corresponding to a fourth image including the fluorescent crosstalk, the third image being acquired by a fluorescence microscope and not including the fluorescent crosstalk, based on learning data in which the third image is associated with the fourth image including the fluorescent crosstalk; generating a second image of the sample acquired by the fluorescence microscope, in which the fluorescence crosstalk of the first image of the sample has been reduced, based on the acquired learning model; a control unit; Information processing device. [Appendix 2] 10. The information processing device according to claim 1, the control unit acquires each of the first image and the fourth image as fluorescent images of multiple wavelengths when excitation light of multiple wavelengths is simultaneously irradiated onto the sample by an optical system of the fluorescence microscope. Information processing device. [Appendix 3] 10. The information processing device according to claim 2, the control unit acquires the third image as a fluorescent image of a single wavelength when the excitation light is irradiated onto the sample by the optical system for each single wavelength among the plurality of wavelengths. Information processing device. [Appendix 4] An information processing device according to any one of Supplementary Notes 1 to 3, The third image and the fourth image are a pair of images captured in the same field of view on the fluorescence microscope. Information processing device. [Appendix 5] 5. The information processing device according to claim 4, the control unit acquires a plurality of sets, each set being imaged in a plurality of fields of view different from one another, using the fluorescence microscope; Information processing device. [Appendix 6] 6. An information processing device according to any one of Supplementary Notes 1 to 5, the control unit acquires a set of the first image and the third and fourth images, each of which differs from the other in at least one of intensity and irradiation time of excitation light irradiated onto the sample by the optical system of the fluorescence microscope. Information processing device. [Appendix 7] 7. An information processing device according to any one of Supplementary Notes 1 to 6, The fluorescence microscope is a confocal microscope. Information processing device. [Appendix 8] An information processing method executed by an information processing device, acquiring a learning model constructed by learning a third image corresponding to a fourth image including the fluorescent crosstalk, the third image being acquired by a fluorescence microscope and not including the fluorescent crosstalk, based on learning data in which the third image is associated with the fourth image including the fluorescent crosstalk; generating a second image of the sample acquired by the fluorescence microscope in which the fluorescence crosstalk has been reduced based on the acquired learning model; Including, Information processing methods. [Appendix 9] A method for generating the learning model used in the information processing method described in Supplementary Note 8, acquiring the training data; learning the third image corresponding to the fourth image based on the acquired learning data to construct the learning model; Including, How to generate a learning model. [Appendix 10] A method for generating a learning model according to Supplementary Note 9, The learning model is a machine learning model trained based on the acquired learning data. How to generate a learning model. [Appendix 11] A program that causes the information processing device to execute any one of the information processing method described in Supplementary Note 8 and the learning model generation method described in Supplementary Notes 9 and 10. [Explanation of symbols]

[0107] 1. Fluorescence Microscopy 10. Information processing equipment 11 Control section 111 Image processing unit 112 Comparison section 12 Storage section 20 Optical system 21 Irradiation unit 22 First dichroic mirror 23 Objective Lens 24 Second dichroic mirror 25a 1st light receiving section 25b 2nd light receiving section E fluorescence E1 1st fluorescence E2 Second Fluorescence L excitation light L1 First excitation light L2 Second excitation light P11 First fluorescent image (first image) P21 Second fluorescent image (first image) P12 First fluorescent image (second image) P22 Second fluorescent image (second image) P13 First fluorescent image (third image) P23 Second fluorescent image (third image) P14 First fluorescent image (fourth image) P141 First fluorescent image P24 Second fluorescence image (fourth image) P241 Second fluorescent image S sample a Fluorescence b Fluorescence

Claims

1. acquiring a learning model constructed by learning a third image corresponding to a fourth image including the fluorescent crosstalk, based on learning data in which the third image obtained by a fluorescent microscope and not including the fluorescent crosstalk is associated with the fourth image including the fluorescent crosstalk; generating a second image of the sample acquired by the fluorescence microscope, in which the fluorescence crosstalk of the first image of the sample has been reduced, based on the acquired learning model; a control unit; Information processing device.

2. 2. The information processing device according to claim 1, the control unit acquires each of the first image and the fourth image as fluorescent images of multiple wavelengths when excitation light of multiple wavelengths is simultaneously irradiated onto the sample by an optical system of the fluorescence microscope. Information processing device.

3. 3. The information processing device according to claim 2, the control unit acquires the third image as a fluorescent image of a single wavelength when the excitation light is irradiated onto the sample by the optical system for each single wavelength among the plurality of wavelengths. Information processing device.

4. 4. The information processing device according to claim 1, the third image and the fourth image are a pair of images captured in the same field of view on the fluorescence microscope; Information processing device.

5. 5. The information processing device according to claim 4, the control unit acquires a plurality of sets, each set being imaged in a plurality of fields of view different from one another, using the fluorescence microscope; Information processing device.

6. 4. The information processing device according to claim 1, the control unit acquires the first image and a set of the third and fourth images, each of which differs from the other in at least one of intensity and irradiation time of excitation light irradiated onto the sample by the optical system of the fluorescence microscope. Information processing device.

7. 4. The information processing device according to claim 1, The fluorescence microscope is a confocal microscope. Information processing device.

8. An information processing method executed by an information processing device, acquiring a learning model constructed by learning a third image corresponding to a fourth image including the fluorescent crosstalk, based on learning data in which the third image not including the fluorescent crosstalk is associated with the fourth image obtained by a fluorescent microscope; generating a second image of the sample acquired by the fluorescence microscope in which the fluorescence crosstalk has been reduced based on the acquired learning model; Including, Information processing methods.

9. A method for generating the learning model used in the information processing method according to claim 8, acquiring the training data; constructing the learning model by learning the third image corresponding to the fourth image based on the acquired learning data; Including, How to generate a learning model.

10. The learning model generation method according to claim 9, The learning model is a machine learning model trained based on the acquired learning data. How to generate a learning model.

11. A program that causes the information processing device to execute any one of the information processing method according to claim 8 and the learning model generation method according to claims 9 and 10.

Images