Information processing device and program

Abstract

The present invention analyzes a sample with high accuracy. An information processing device and a program according to the present invention, the information processing device comprising: an extraction unit (20E) that extracts fluorescence correction information from a bright field image of a sample; and a generation unit (20F) that generates a fluorescence correction image on the basis of the fluorescence information and the fluorescence correction information of the sample.

Description

[0001] This application is a divisional application of a Chinese application with application number 202080047218.6, filing date May 26, 2020, and invention title "Information Processing Apparatus and Program", the entire content of which is incorporated herein by reference. Technical Field

[0002] The present disclosure relates to an information processing apparatus and a program. Background Art

[0003] A technique for analyzing a sample using a fluorescence image of the sample has been disclosed. For example, a technique for analyzing the type, size, etc. of cells by analyzing a fluorescence signal or a fluorescence spectrum is known (for example, Patent Document 1 and Patent Document 2).

[0004] Prior Art Documents

[0005] Patent Documents

[0006] Patent Document 1: JP 2005-91895A

[0007] Patent Document 2: JP 2015-145829 A Summary of the Invention

[0008] Technical Problem to be Solved by the Invention

[0009] However, a fluorescence image has factors that affect analysis caused by autofluorescence, light absorption, etc. from substances contained in the sample. Therefore, in the prior art, it is sometimes difficult to analyze a sample with high accuracy.

[0010] Accordingly, the present disclosure provides an information processing apparatus and a program capable of analyzing a sample with high accuracy.

[0011] Solution to the Problem

[0012] To solve the above problems, an information processing apparatus according to one aspect of the present disclosure includes: an extraction unit that extracts fluorescence correction information from a bright-field image of a sample; and a generation unit that generates a fluorescence correction image based on the fluorescence information and the fluorescence correction information of the sample. Brief Description of the Drawings

[0013] Figure 1A is a schematic diagram showing an example of an information processing apparatus according to an embodiment of the present disclosure;

[0014] Figure 1B is a schematic diagram showing an example of a measurement unit according to an embodiment of the present disclosure;

[0015] Figure 2 is a diagram showing an example of a bright-field image according to an embodiment of the present disclosure;

[0016] Figure 3A is a diagram showing an example of a fluorescence image according to an embodiment of the present disclosure;

[0017] Figure 3B is a diagram showing an example of a fluorescence image according to an embodiment of the present disclosure;

[0018] Figure 3C is a diagram showing an example of a fluorescence image according to an embodiment of the present disclosure;

[0019] Figure 3D is a diagram showing an example of a fluorescence image according to an embodiment of the present disclosure;

[0020] Figure 4 is an explanatory diagram of an example of generating a combined image according to an embodiment of the present disclosure;

[0021] Figure 5 is a diagram showing an example of a bright-field image according to an embodiment of the present disclosure, in which a calibration target area is identified;

[0022] Figure 6A is a diagram showing an example of a fluorescence calibration image according to an embodiment of the present disclosure;

[0023] Figure 6B is a diagram showing an example of a fluorescence calibration image according to an embodiment of the present disclosure;

[0024] Figure 6C is a diagram showing an example of a fluorescence calibration image according to an embodiment of the present disclosure;

[0025] Figure 6D is a diagram showing an example of a fluorescence calibration image according to an embodiment of the present disclosure;

[0026] Figure 7 is a diagram showing the spectrum of a fluorescent dye according to an embodiment of the present disclosure;

[0027] Figure 8A is an explanatory diagram of an example of the staining states of a bright-field image and a fluorescence image according to an embodiment of the present disclosure;

[0028] Figure 8B is an explanatory diagram of an example of the staining states of a bright-field image and a fluorescence image according to an embodiment of the present disclosure;

[0029] Figure 8C is an explanatory diagram of an example of the staining states of a bright-field image and a fluorescence image according to an embodiment of the present disclosure;

[0030] Figure 8D is an explanatory diagram of an example of the staining states of a bright-field image and a fluorescence image according to an embodiment of the present disclosure;

[0031] Figure 9 It is a diagram showing the positional relationship and staining state between tumor cells and lymphocytes according to an embodiment of the present disclosure;

[0032] Figure 10A It is a schematic diagram of an example of a display screen according to an embodiment of the present disclosure;

[0033] Figure 10B It is a schematic diagram of an example of a display screen according to an embodiment of the present disclosure;

[0034] Figure 10C It is a schematic diagram of an example of a display screen according to an embodiment of the present disclosure;

[0035] Figure 10D It is a schematic diagram of an example of a display screen according to an embodiment of the present disclosure;

[0036] Figure 10E It is a schematic diagram of an example of a display screen according to an embodiment of the present disclosure;

[0037] Figure 10F It is a schematic diagram of an example of a display screen according to an embodiment of the present disclosure;

[0038] Figure 11 It is a schematic diagram showing an example of switching of a display screen according to an embodiment of the present disclosure;

[0039] Figure 12 It is a flowchart showing an example of a process of information processing according to an embodiment of the present disclosure;

[0040] Figure 13 It is a hardware configuration diagram showing an example of a computer for implementing the functions of the analysis device of the present disclosure according to an embodiment of the present disclosure. Detailed implementation manners

[0041] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same parts are denoted by the same reference numerals, and thus repeated descriptions will be omitted.

[0042] Figure 1A It is a schematic diagram showing an example of the information processing device 1 according to the present embodiment.

[0043] The information processing device 1 includes an analysis device 10 and a measurement unit 12. The analysis device 10 is connected to the measurement unit 12 so as to be able to exchange data or signals.

[0044] The measurement unit 12 photographs a sample and obtains a bright-field image and a fluorescence image of the sample.

[0045] The sample is a sample to be analyzed by the analysis device 10. The sample is, for example, a tissue sample for pathological diagnosis or the like. Specifically, the specimen is a biological tissue including tumor cells, lymphocytes (T cells, B cells, natural killer cells (NK cells)), and the like. The sample may also be a sample including a complex of one or more proteins, amino acids, carbohydrates, lipids, and their modified molecules. In addition, the specimen may be a sample including an antigen (tumor marker, signal transducer, hormone, cancer growth regulator, metastasis regulator, growth regulator, inflammatory cytokine, virus-related molecule, etc.) related to the disease to be pathologically diagnosed. In addition, the sample may be a sample including metabolites, deoxyribonucleic acid (DNA), ribonucleic acid (RNA), microRNA, polynucleotide, toxin, drug, virion, cell, hemoglobin, and the like. Note that the sample is not limited to the above. For example, the sample may be a sample different from a living tissue. For example, the sample may be an abiotic sample composed of multiple molecules.

[0046] In the present embodiment, as an example, the case where the sample is a biological tissue will be described.

[0047] The bright-field image is a captured image of the specimen. The captured image is captured image data that defines a pixel value for each pixel. That is, the bright-field image is captured image data that defines a color value for each pixel. The color value is represented, for example, by the gray-scale values of each of red (R), green (G), and blue (B). Hereinafter, the captured image data will be simply referred to as the captured image.

[0048] In the present embodiment, a stained sample is used when capturing the bright-field image. The stained specimen specifically includes a specimen stained with hematoxylin and eosin (HE). The measurement unit 12 obtains a bright-field image by capturing the sample, and the bright-field image is a captured image of the light transmitted through or reflected by the HE-stained sample. The measurement unit 12 outputs the obtained bright-field image to the analysis device 10.

[0049] The fluorescence image is a captured image of the fluorescently stained sample. Specifically, the bright-field image is captured image data that defines a fluorescence intensity value for each pixel.

[0050] The fluorescently stained sample is a sample in which a target is labeled with a fluorescent dye or included in the stained sample. For example, the target is the target to be analyzed. The target is used for, for example, tumors, cells, pathological diagnosis, and the like.

[0051] Targets are, for example, tumor markers, lymphocyte markers, immune cells, immune checkpoint molecules, molecules used as molecular target drug indicators, receptors, cell surface markers, etc. As these targets, various antigens are used, for example. Antigens are, for example, CD3, CD20, CD68, etc. For the fluorescence labeling of these antigens, known antibodies fluorescently labeled with known fluorescent dyes (for example, fluorescein isothiocyanate (FITC)) can be used. In addition, there is a method of accumulating fluorescent dyes by using an enzyme, but this method is not limited thereto. Examples thereof include nucleic acids, for example, DNA and RNA. Examples of fluorescent dyes for staining these specific tissues include DAPI (4’,6-diamidino-2-phenylindole) (DAPI), Alexa Fluor (registered trademark) (AF) 405, Brilliant Violet (BV) 421, BV480, BV510, BV510, etc.

[0052] The measurement unit 12 obtains a fluorescence image by irradiating a sample in which a target is fluorescently stained with light (excitation light) in a wavelength region that excites the fluorescence of the fluorescent dye and photographing the sample. A known mechanism can be used as the light irradiation and photographing mechanism. The measurement unit 12 outputs the obtained fluorescence image to the analysis device 10.

[0053] Figure 1B is a schematic diagram showing an example of the specific configuration of the measurement unit 12. The measurement unit 12 is a microscope system that photographs a sample in an enlarged state.

[0054] The measurement unit 12 includes a microscope 70 and a data processing unit 80.

[0055] The microscope 70 includes a stage 71, an irradiation unit 73, and an imaging element 74. The stage 71 has a placement surface on which a sample SPL as a sample can be placed. The stage 71 is movable in a parallel direction (x-y plane direction) and a vertical direction (z-axis direction) under the control of a stage drive unit 75.

[0056] The irradiation unit 73 irradiates the sample with excitation light. The irradiation unit 73 includes an optical system 72, a light source drive unit 76, and a light source 78.

[0057] The optical system 72 is provided above the stage 71. The optical system 72 includes an objective lens 72A, an imaging lens 72B, a dichroic mirror 72C, an emission filter 72D, and an excitation filter 72E. For example, the light source 78 is a bulb such as a mercury lamp, a light emitting diode (LED), etc.

[0058] The excitation filter 72E is a filter that selectively transmits light in the wavelength region that excites the fluorescence of the fluorescent dye from the light emitted by the light source 78. The microscope 70 is equipped with a plurality of excitation filters 72E, which have different wavelength regions of transmitted light.

[0059] The dichroic mirror 72C guides the light emitted from the light source 78 and transmitted through the excitation filter 72E to the objective lens 72A. The objective lens 72A focuses the light on the sample SPL. Then, the objective lens 72A and the imaging lens 72B form a magnified image obtained by magnifying the image of the sample SPL to a predetermined magnification on the imaging surface of the imaging element 74.

[0060] The light source drive unit 76 controls the light source 78 and controls the switching of the excitation filter 72E.

[0061] The imaging element 74 obtains a captured image of the sample. The magnified image of the sample is formed on the imaging element 74 via the objective lens 72A and the imaging lens 72B. The imaging element 74 obtains a captured image obtained by imaging the magnified sample.

[0062] The imaging element 74 is an imager having a photoelectric conversion element and obtaining an image from incident light. The imaging element 74 has an image device, for example, a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) image sensor, etc. Note that the imaging lens 72B and the emission filter 72D can be changed to spectral elements. In this case, a spectral camera of a running scan type as a spatial scan type or a two-dimensional spectral camera of a time scan type is used.

[0063] The imaging element 74 obtains a captured image obtained by photographing the sample under the control of the imaging control unit 77, and outputs the captured image to the data processing unit 80.

[0064] The data processing unit 80 includes an irradiation control unit 80A and an acquisition unit 80B. The irradiation control unit 80A controls the irradiation unit 73. Under the control of the irradiation control unit 80A, the light source drive unit 76 controls the position of the excitation filter 72E so that the light emitted from the light source 78 is transmitted through the excitation filter 72E, and then causes the light source 78 to emit light.

[0065] The acquisition unit 80B acquires a captured image of the sample from the imaging control unit 77. Specifically, the acquisition unit 80B acquires a bright-field image by acquiring a captured image of the stained sample. In addition, the acquisition unit 80B acquires a fluorescence image by acquiring a captured image captured in a state where the sample stained with fluorescence by the excitation light irradiates the target. Then, the acquisition unit 80B outputs the fluorescence image and the bright-field image to the analysis device 10.

[0066] Back to Figure 1A, which will be described below. The analysis device 10 is an analysis device for analyzing a sample.

[0067] The analysis device 10 includes a control unit 20, a storage unit 22, a user interface (UI) unit 24, and a communication unit 26. The control unit 20, the storage unit 22, the UI unit 24, and the communication unit 26 are connected so as to be able to exchange data or signals.

[0068] The storage unit 22 stores various data. In this embodiment, the storage unit 22 stores various data, for example, a learning model 23 and a composite image 38. Details of the learning model 23 and the composite image 38 will be described later.

[0069] The UI unit 24 receives various operation inputs from the user to output various types of information. In this embodiment, the UI unit 24 includes a display unit 24A and an input unit 24B.

[0070] The display unit 24A displays various types of information. The display unit 24A is, for example, an organic electroluminescent device (EL), a liquid crystal display (LCD), etc. The input unit 24B receives various operation inputs from the user. The input unit 24B is, for example, a pointing device, a mouse, a keyboard, an input button, etc. Note that the display unit 24A and the input unit 24B can be integrally configured as a touch panel.

[0071] The communication unit 26 is a communication interface that communicates with an external device via a network in a wired or wireless manner.

[0072] The control unit 20 includes a fluorescence image acquisition unit 20A, a bright-field image acquisition unit 20B, a storage control unit 20C, a learning unit 20D, an extraction unit 20E, an identification unit 20F, a generation unit 20G, an analysis unit 20H, and a display control unit 20I. Some or all of the fluorescence image acquisition unit 20A, the bright-field image acquisition unit 20B, the storage control unit 20C, the learning unit 20D, the extraction unit 20E, the identification unit 20F, the generation unit 20G, the analysis unit 20H, and the display control unit 20I can be implemented by causing a processing device (e.g., a central processing unit (CPU)) to execute a program (i.e., implemented by software), implemented by hardware (e.g., an integrated circuit (IC)), or can be implemented by a combination of software and hardware.

[0073] The fluorescence image acquisition unit 20A acquires a fluorescence image including fluorescence information of the sample. The fluorescence information is information about the fluorescently stained sample. For example, the fluorescence image acquisition unit 20A acquires a fluorescence image from the measurement unit 12. Note that the fluorescence image acquisition unit 20A can acquire a fluorescence image by reading a fluorescence image stored in the storage unit 22.

[0074] The bright-field image acquisition unit 20B acquires a bright-field image. For example, the bright-field image acquisition unit 20B acquires a bright-field image from the measurement unit 12. Note that the bright-field image acquisition unit 20B may acquire a bright-field image by reading the bright-field image stored in the storage unit 22.

[0075] Note that the fluorescence image acquisition unit 20A and the bright-field image acquisition unit 20B acquire a fluorescence image and a bright-field image, respectively, as captured images of the same sample. That is, the fluorescence image acquired by the fluorescence image acquisition unit 20A and the bright-field image acquired by the bright-field image acquisition unit 20B are captured images of the same sample. Note that the fluorescence image acquisition unit 20A and the bright-field image acquisition unit 20B may acquire a fluorescence image and a bright-field image, respectively, of the same tissue section of the same sample. In addition, the fluorescence image acquisition unit 20A and the bright-field image acquisition unit 20B may acquire a fluorescence image and a bright-field image, respectively, of different tissue sections of the same sample. In this case, it is preferable that the fluorescence image acquisition unit 20A and the bright-field image acquisition unit 20B acquire a fluorescence image and a bright-field image, respectively, of each consecutive section of the same sample. That is, the sample is preferably the same section or consecutive tissue sections.

[0076] As described above, in the present embodiment, each of the fluorescence image and the bright-field image can be acquired using the same sample or samples similar to each other. At this time, as a sample that is the same as or similar to a certain sample, any section of an unstained section and a stained section can also be used. For example, when using an unstained section, a section before staining can also be used as a stained section, a section adjacent to the stained section, a section different from the stained section in the same block (sampled from the same place as the stained section), a section in a different block in the same tissue (sampled from a place different from the stained section), etc.

[0077] In the present embodiment, as an example, a mode in which the fluorescence image acquisition unit 20A and the bright-field image acquisition unit 20B acquire a fluorescence image and a bright-field image, respectively, of the same tissue section of the same sample will be described. Hereinafter, the tissue section may be simply referred to as a section.

[0078] Figure 2 is a diagram showing an example of the bright-field image 30. Figure 2 An example of the bright-field image 30 of a section of the sample 40 as a living tissue is shown. In the present embodiment, a case where the sample 40 includes components that emit autofluorescence will be described. Examples of components that emit autofluorescence are red blood cells, vascular endothelium, necrotic regions, adipose regions, collagen, elastin, debris, dirt, contaminants, artifacts, etc. On the other hand, components that absorb fluorescence as a light absorption region are toner, etc.

[0079] Figures 3A to 3D This is a figure showing an example of the fluorescence image 32.

[0080] Figure 3A This is an image showing the fluorescence image 32A. The fluorescence image 32A is a fluorescence image 32 of a section of the sample 40, where CD20 as an antigen is fluorescently labeled using a fluorescently labeled antibody specific to cluster of differentiation (CD) 20.

[0081] Figure 3B This is an image showing the fluorescence image 32B. The fluorescence image 32B is a fluorescence image 32 of a section of the sample 40, where CD3 as an antigen is fluorescently labeled using a fluorescently labeled antibody specific to CD3.

[0082] Figure 3C This is an image showing the fluorescence image 32C. The fluorescence image 32C is a fluorescence image 32 of a section of the sample 40, where CD68 as an antigen is fluorescently labeled using a fluorescently labeled antibody specific to CD68.

[0083] Figure 3D This is an image showing the fluorescence image 32D. The fluorescence image 32D is a fluorescence image 32 of a section of the sample 40, where specific tissues (e.g., DNA and RNA) are fluorescently stained with 4′,6-diamidino-2-phenylindole (DAPI).

[0084] Note that Figures 3A to 3D shows the Figure 2 fluorescence image 32 of the same section of the sample 40 corresponding to the bright-field image 30 shown in

[0085] Specifically, both the bright-field image 30 and the fluorescence image 32 (fluorescence images 32A to 32D) are captured images obtained by photographing the same section of the same sample 40. However, when photographing the bright-field image 30, the section may be at least stained with HE. In addition, when photographing the fluorescence image 32, the section is fluorescently stained with a fluorescent dye (or fluorescently labeled antigen) specific to each of a plurality of different targets (antigens, specific tissues), and photographed in a state of being irradiated with light in the wavelength region excited by the fluorescent dye.

[0086] Returning to Figure 1A , the description will continue. The storage control unit 20C stores the fluorescence image 32 acquired by the fluorescence image acquisition unit 20A and the bright-field image 30 acquired by the bright-field image acquisition unit 20B in the storage unit 22 in association with each other for the same sample 40.

[0087] For example, identification information (e.g., a barcode) of a sample 40 is provided to a slide on which a section of the sample 40 for capturing each of a bright-field image 30 and a fluorescence image 32 is placed. Then, the measurement unit 12 can capture the identification information as well as each of the bright-field image 30 and the fluorescence image 32 when capturing the sample 40. The storage control unit 20C can store the bright-field image 30 and the fluorescence image 32 of each sample 40 in the storage unit 22 by identifying the bright-field image 30 and the fluorescence image 32 having the same identification information.

[0088] Therefore, whenever the fluorescence image acquisition unit 20A and the bright-field image acquisition unit 20B acquire the fluorescence image 32 and the bright-field image 30, the fluorescence image 32 and the bright-field image 30 corresponding to the sample 40 are sequentially stored in the storage unit 22.

[0089] Note that the storage control unit 20C can generate a composite image 38 using the bright-field image 30 and the fluorescence image 32 of each sample 40 and store the composite image in the storage unit 22.

[0090] Figure 4 is an explanatory diagram of an example of generating the composite image 38.

[0091] The composite image 38 is an image in which a color value of the bright-field image 30 and a fluorescence intensity value of the fluorescence image 32 are defined for each pixel. Specifically, the composite image 38 is an image obtained by combining the bright-field image 30 and the fluorescence image 32 of each corresponding pixel. The corresponding pixels are pixels that indicate the same region at the same position in the same section 41 of the same sample 40. Therefore, preferably, the plurality of images (bright-field image 30, fluorescence image 32) constituting the composite image 38 are captured images of the same region of the same section 41 except that at least one of the staining conditions and the imaging conditions is different.

[0092] For example, assume that the bright-field image 30 is composed of bright-field images 30 (bright-field image 30R, bright-field image 30G, and bright-field image 30B) of three channels of red (R), green (G), and blue (B). The bright-field image 30R is a bright-field image 30 in which a color value of red (R) is defined for each pixel. The bright-field image 30G is a bright-field image 30 in which a color value of green (G) is defined for each pixel. The bright-field image 30B is a bright-field image 30 in which a color value of B (blue) is defined for each pixel.

[0093] For example, the section 41 of the HE-stained sample 40 is adjusted (step S1). When the measurement unit 12 photographs the section 41, the bright-field image acquisition unit 20B acquires the bright-field image 30 of the section 41 (bright-field image 30R, bright-field image 30G, and bright-field image 30B) (step S2).

[0094] In addition, the section 41 of the fluorescently stained sample 40 is adjusted (step S3). The measurement unit 12 irradiates the section 41 with light having a wavelength that excites fluorescence and photographs the section 41. At this time, the measurement unit 12 fluorescently stains the section 41 with a fluorescent dye for each of a plurality of types of targets, and performs photographing in a state where light having a wavelength that excites each fluorescent dye is sequentially emitted (steps S3 and S4). As a result, the fluorescence image acquisition unit 20A acquires a plurality of fluorescence images 32 of a plurality of types of targets (for example, fluorescence image 32A to fluorescence image 32D) (step S4).

[0095] The storage control unit 20C creates a combined image 38 that defines a combination of the color values of the bright-field image 30 and the fluorescence intensity values of the fluorescence image 32 for each pixel of each sample 40 (steps S5 and S6). The storage control unit 20C generates the combined image 38 of each sample 40 by associating the color value of the bright-field image 30 of each pixel with the fluorescence intensity value of the fluorescence image 32. Then, the storage control unit 20C stores the generated combined image 38 in the storage unit 22.

[0096] Therefore, for the region corresponding to the pixels of the same section 41 of the same sample 40, the combined image 38 that defines the color values of the three channels of R, G, and B defined by the bright-field image 30 and the number of channels equal to the number of target types defined by one or more fluorescence images 32 is stored in the storage unit 22 of each sample 40.

[0097] Whenever the bright-field image 30 and the fluorescence image 32 of a new sample 40 are stored in the storage unit 22, the storage control unit 20C can generate the combined image 38. Note that the time when the storage control unit 20C generates the combined image 38 is not limited to this time. For example, the storage control unit 20C can generate the combined image 38 at every predetermined time period using the bright-field image 30 and the fluorescence image 32 stored in the storage unit 22.

[0098] Back to Figure 1A, will be described below. When the composite image 38 has been stored in the storage unit 22, the fluorescence image acquisition unit 20A can use the bright-field image 30 acquired by the bright-field image acquisition unit 20B and the composite image 38 to acquire the fluorescence image 32. In this case, the fluorescence image acquisition unit 20A can determine, from the composite image 38, the fluorescence intensity value corresponding to each pixel of the bright-field image 30 acquired by the bright-field image acquisition unit 20B, so as to acquire the fluorescence image 32 in which the fluorescence intensity value is defined for each pixel. That is, the fluorescence image acquisition unit 20A can use the composite image 38 as a learning model to acquire the fluorescence image 32.

[0099] Similarly, when the composite image 38 has been stored in the storage unit 22, the bright-field image acquisition unit 20B can use the fluorescence image 32 acquired by the fluorescence image acquisition unit 20A and the composite image 38 to acquire the bright-field image 30. In this case, the bright-field image acquisition unit 20B can determine, from the composite image 38, the color value corresponding to each pixel of the fluorescence image 32 acquired by the fluorescence image acquisition unit 20A, so as to acquire the bright-field image 30 in which the color value is defined for each pixel.

[0100] That is, the bright-field image acquisition unit 20B can use the composite image 38 as a learning model to acquire the bright-field image 30. Specifically, the fluorescence information of the composite image 38 can be learned as teacher data to acquire the bright-field image 30. According to the present embodiment, since the bright-field image and the fluorescence information derived from the same slice are accurately associated on a pixel basis, the bright-field image can be acquired from the fluorescence information with high accuracy. Preferably, the fluorescence information may include information about fluorescence in situ hybridization (FISH).

[0101] Next, the extraction unit 20E will be described. The extraction unit 20E extracts fluorescence correction information from the bright-field image 30 of the sample 40. The fluorescence correction information is information selected from the color information, the morphological information, or the staining information of the bright-field image 30. The morphological information of the bright-field image 30 is information indicating the position, size, and range of the regions indicating each color included in the bright-field image 30. The staining information of the bright-field image 30 is information indicating the staining state of the sample 40 in the bright-field image 30. For example, the extraction unit 20E can extract the fluorescence correction information from the bright-field image 30 by analyzing the bright-field image 30 using a known image analysis method.

[0102] Next, the recognition unit 20F will be described. The recognition unit 20F determines the correction target region of the bright-field image 30 based on the fluorescence correction information extracted by the extraction unit 20E. As Figure 2 shown, the recognition unit 20F determines the correction target region P1 included in the bright-field image 30.

[0103] Figure 5 This is a diagram showing an example of a bright-field image 30 in a state where a calibration target area P1 is recognized.

[0104] The calibration target area P1 is an area where auxiliary information is to be added. For example, the calibration target area P1 is an area including information that obstructs the analysis of the analysis unit 20H, an area that prompts fixation, etc. The calibration target area P1 is, for example, an autofluorescence area or a light absorption area. The analysis unit 20H can determine the calibration target area P1 according to the analysis content.

[0105] For example, assume that the analysis content is the type, distribution, area, size, abundance, etc. of the target being analyzed. In this case, the calibration target area P1 is an area including autofluorescence signals, autofluorescence spectra, toner that absorbs light, etc. Hereinafter, the autofluorescence signal may be referred to as an autofluorescence signal, and the autofluorescence spectrum may be referred to as an autofluorescence spectrum.

[0106] The recognition unit 20F recognizes the calibration target area P1 corresponding to the analysis content in the entire area of the bright-field image 30. For example, the recognition unit 20F determines the calibration target area P1 of the bright-field image 30 based on the fluorescence correction information extracted by the extraction unit 20E. Specifically, an area that matches the feature information for recognizing the calibration target area P1 can be recognized as the calibration target area P1.

[0107] The feature information is information indicating the features of the calibration target area P1. The feature information is, for example, information indicating the features of the autofluorescence signal and information indicating the features of the autofluorescence spectrum. The information indicating the features of the autofluorescence signal is, for example, information indicating the wavelength and intensity of the autofluorescence signal. The autofluorescence signal itself can be used as feature information. The information indicating the features of the autofluorescence spectrum is, for example, information indicating the peak position and peak intensity of the autofluorescence spectrum. The autofluorescence spectrum itself can be used as feature information. In addition, the color information represented by the autofluorescence signal and the autofluorescence spectrum can be used as feature information. The recognition unit 20F can derive the feature information from the fluorescence correction information extracted by the extraction unit 20E.

[0108] Then, the recognition unit 20F recognizes the calibration target area P1 based on the color value of each pixel of the bright-field image 30. Specifically, the recognition unit 20F can recognize such a pixel area in the pixels of the bright-field image 30 as the calibration target area P1, in which the spectrum represented by the color value of each pixel matches the feature information or falls within a predetermined range.

[0109] As an example, Figure 5 and Figure 2This shows a case where the calibration target area P1 is an autofluorescence area of autofluorescence of red blood cells.

[0110] Note that the recognition unit 20F can determine the calibration target area P1 by recognizing the morphological information indicating the position, size, and range of the calibration target area P1 included in the bright-field image 30. The position of the calibration target area P1 indicates the position of the calibration target area P1 in the bright-field image 30. The size of the calibration target area P1 indicates the size of the calibration target area P1 in the bright-field image 30. The range of the calibration target area P1 indicates the range occupied by the calibration target area P1 in the bright-field image 30.

[0111] Note that it is assumed that the shooting angles and shooting magnifications of the bright-field image 30 and the fluorescence image 32 obtained by shooting the same sample 40 are the same.

[0112] Then, the recognition unit 20F stores the recognized calibration target area P1 in the storage unit 22 in association with the bright-field image 30 used to recognize the calibration target area P1. Therefore, the storage unit 22 stores the bright-field image 30, the fluorescence images 32 (fluorescence images 32A to 32D), and the morphological information of the calibration target area P1 in association with each other for each sample 40. Note that the process of storing the calibration target area P1 in the storage unit 22 can be executed by the storage control unit 20C.

[0113] Returning to Figure 1A , the description will continue. The recognition unit 20F can use the learning model 23 to recognize the calibration target area P1. The learning model 23 is generated by the learning unit 20D.

[0114] The learning model 23 is a model that takes the bright-field image 30 as input and the calibration target area P1 as output. The learning unit 20D generates the learning model 23 through learning. Known algorithms such as convolutional neural networks (CNNs) can be used for learning.

[0115] For example, the learning unit 20D learns and updates the learning model 23 at a predetermined time. The predetermined time is, for example, the time when a new bright-field image 30 and a new fluorescence image 32 are stored in the storage unit 22, the time when a new calibration target area P1 is stored in the storage unit 22, a predetermined time period, a predetermined day, etc. The predetermined time can be determined in advance.

[0116] In the case where the learning model 23 has been recorded in the storage unit 22, the recognition unit 20F can recognize the calibration target area P1 by inputting the bright-field image 30 acquired by the bright-field image acquisition unit 20B as input data into the learning model 23 and obtaining the calibration target area P1 output from the learning model 23.

[0117] Next, the generation unit 20G will be described. The generation unit 20G generates a fluorescence correction image based on the fluorescence information and fluorescence correction information of the sample 40. In the present embodiment, the generation unit 20G generates a fluorescence correction image obtained by correcting the correction target region P1 included in the fluorescence image 32 based on the fluorescence image 32 and the correction target region P1 identified based on the extracted fluorescence correction information.

[0118] The fluorescence correction image is a fluorescence image generated by correcting the correction target region P1 in the fluorescence image 32.

[0119] Figures 6A to 6D It is a diagram showing an example of the fluorescence correction image 34 (fluorescence correction images 34A to 34D). The fluorescence correction image 34A is a fluorescence image obtained by correcting the first region PA corresponding to the correction target region P1 of the fluorescence image 32A (see Figure 3A ). The fluorescence correction image 34B is a fluorescence image obtained by correcting the first region PA corresponding to the correction target region P1 of the fluorescence image 32B (see Figure 3B ). The fluorescence correction image 34C is a fluorescence image obtained by correcting the first region PA corresponding to the correction target region P1 of the fluorescence image 32C (see Figure 3C ). The fluorescence correction image 34D is a fluorescence image obtained by correcting the first region PA corresponding to the correction target region P1 of the fluorescence image 32D (see Figure 3D ).

[0120] The generation unit 20G identifies, as the correction target region P1 (i.e., the first region PA) on the fluorescence correction image 34, the region indicating the position, size, and range of the correction target region P1 identified by the identification unit 20F in the fluorescence correction image 34.

[0121] As described above, the bright-field image 30 and the fluorescence image 32 obtained by photographing the same sample 40 have the same photographing angle and photographing magnification. In addition, it is assumed that the bright-field image 30 and the fluorescence image 32 obtained by photographing the same sample 40 have the same region to be imaged in the sample 40.

[0122] Therefore, the generation unit 20G can identify, as a region of the position, size, and range on the fluorescence image 32 defined by the position, size, and range of the correction target region P1 on the bright-field image 30 identified by the identification unit 20F, the region corresponding to the correction target region P1 as the first region PA.

[0123] Then, the generation unit 20G corrects the fluorescence intensity value of each pixel in the first region PA identified in the fluorescence image 32.

[0124] That is, the generation unit 20G identifies a first region PA corresponding to the calibration target region P1 of the bright-field image 30 in the fluorescence image 32 by using a superimposed image obtained by superimposing the bright-field image 30 and the fluorescence image 32, and corrects the first region PA.

[0125] For example, when the amount of the fluorescence component caused by autofluorescence is high, the generation unit 20G corrects the fluorescence intensity value in the first region PA included in the fluorescence image 32 to a lower value.

[0126] Specifically, the generation unit 20G prestores the autofluorescence spectra of each type of autofluorescence. The type of autofluorescence is, for example, the type of component that emits autofluorescence (red blood cells, collagen, elastin, etc.).

[0127] Then, the generation unit 20G reads the autofluorescence spectrum corresponding to the type of autofluorescence included in the slice 41 of the fluorescence correction image 34. The type of autofluorescence can be input by the user through the operation of the input unit 24B, or can be identified by the generation unit 20G that performs image analysis or the like by a known method. The generation unit 20G can generate the fluorescence correction image 34 by removing the autofluorescence spectrum from the spectrum of the color value of each pixel in the first region PA in the fluorescence image 32. For example, the generation unit 20G can generate the fluorescence correction image 34 by multiplying the spectrum of the color value of the nuclear pixel in the first region PA in the fluorescence image 32 by a weight value according to the autofluorescence spectrum.

[0128] At this time, the stored autofluorescence spectrum can be the autofluorescence spectrum of one channel. That is, there is a case where an autofluorescence spectrum of one channel obtained by synthesizing the autofluorescence spectra obtained from each of the plurality of fluorescence images 32 is stored, and each fluorescence image includes various types of fluorescent staining targets.

[0129] In this case, the generation unit 20G can divide the autofluorescence spectrum into the number of channels of the fluorescence images 32 (fluorescence images 32A to 32D) (in this case, four channels). The separation can be performed using a known method. Then, the generation unit 20G can generate each fluorescence correction image 34A to 34D by using the separated autofluorescence spectrum corresponding to each target to correct the first region PA of each fluorescence image 32A to the fluorescence image 32D corresponding to the fluorescent staining target.

[0130] Note that the generation unit 20G can obtain a weight value according to a standard spectrum derived from a sample from the learning model 23, and use the weight value to generate the fluorescence correction image 34.

[0131] In this case, the learning unit 20D can pre-generate a model, as the learning model 23, which has the bright-field image 30 as the input and the weight values according to the calibration target region P1 and the standard spectrum derived from the sample as the output. Then, the generation unit 20G obtains the weight values output from the learning model 23 by inputting the bright-field image 30 as the input data into the learning model 23. Then, the generation unit 20G can use the obtained weight values to correct the first region PA (calibration target region P1) of the fluorescence image 32 to generate a fluorescence-corrected image 34. At this time, the generation unit 20G can generate the fluorescence-corrected image 34 obtained by removing the first region PA corresponding to the calibration target region P1 corrected by the weight values.

[0132] In addition, the generation unit 20G can correct the fluorescence intensity values in the first region PA included in the fluorescence image 32 to be equal to or less than a predetermined threshold (e.g., "0"). This threshold can be determined in advance. In this case, the generation unit 20G can generate the fluorescence-corrected image 34 obtained by removing the region corresponding to the calibration target region P1.

[0133] In addition, the generation unit 20G can generate the fluorescence-corrected image 34 based on the peripheral information of the calibration target region P1. For example, the generation unit 20G can use the fluorescence intensity values around the first region PA in the fluorescence image 32 to correct the fluorescence intensity values in the first region PA included in the fluorescence image 32. Specifically, the generation unit 20G can generate the fluorescence-corrected image 34 by interpolating the fluorescence intensity values of the first region PA in the fluorescence image 32 using the fluorescence intensity values of the pixels around the first region PA in the fluorescence image 32. Known image processing techniques can be used for interpolation.

[0134] Note that, as described above, for example, when the amount of the fluorescence component caused by autofluorescence is high, the generation unit 20G corrects the fluorescence intensity values in the first region PA included in the fluorescence image 32 to lower values. Therefore, the fluorescence intensity values required for analysis included in the fluorescence-corrected image 34 may also decrease due to the correction.

[0135] Therefore, preferably, the fluorescence image acquisition unit 20A acquires the fluorescence image 32 under the shooting conditions where the brightness of the calibration target region P1 in the fluorescence image 32 is saturated. The shooting conditions can be preset in the measurement unit 12. For example, the shooting conditions where the brightness of the calibration target region P1 is saturated can be adjusted by adjusting the exposure time, the detector sensitivity, the intensity of the illumination light during shooting, etc.

[0136] As described above, the generation unit 20G determines a first region PA corresponding to the calibration target region P1 of the bright-field image 30 in the fluorescence image 32 by using the superimposed image in which the bright-field image 30 and the fluorescence image 32 are superimposed, and corrects the first region PA.

[0137] That is, the analysis device 10 of the present embodiment uses the bright-field image 30 to determine the first region PA that is the calibration target region P1 included in the fluorescence image 32, and corrects the first region PA of the fluorescence image 32. Therefore, the analysis device 10 of the present embodiment can easily correct the calibration target region P1 included in the fluorescence image 32 with high accuracy, and set the first region PA as a region not to be analyzed, etc.

[0138] Note that the generation unit 20G can generate the fluorescence correction image 34 by using the learning model 23. In this case, a model that can take the fluorescence information of the sample 40 as input and the fluorescence correction information as output can be used as the learning model 23. In this case, the learning unit 20D can generate and update the learning model 23 by learning the correspondence between the fluorescence image 32 including the fluorescence information of the sample 40 and the fluorescence correction information extracted by the extraction unit 20E at a predetermined time. Then, the generation unit 20G can generate the fluorescence correction image 34 based on the learning result of the learning unit 20D. Specifically, the generation unit 20G inputs the fluorescence image 32 acquired by the fluorescence image acquisition unit 20A into the learning model 23 as input data, and obtains the fluorescence correction information output from the learning model 23. The generation unit 20G can generate the fluorescence correction image 34 in the same manner as described above based on the obtained fluorescence correction information.

[0139] Here, among fluorescent dyes such as AF405, BV421, BV480, BV510, BV510, and DAPI, the spectrum of DAPI has a wide fluorescence wavelength. For this reason, in the prior art, it is sometimes difficult to separate it from the signals caused by other fluorescent dyes.

[0140] Figure 7 is a diagram showing the spectra of fluorescent dyes. Line 36A represents the spectrum of AF405, and line 36B represents the spectrum of BV421. Line 36C represents the DAPI spectrum. Line 36D represents the spectrum of BV480, and line 36E represents the spectrum of BV510.

[0141] As Figure 7 shown, the DAPI spectrum has a wide fluorescence wavelength, and in the prior art, it is difficult to separate it from the signals caused by other fluorescent dyes.

[0142] On the other hand, in the present embodiment, the analysis device 10 identifies the first region PA corresponding to the calibration target region P1 of the bright-field image 30 in the fluorescence image 32 by using the superimposed image in which the bright-field image 30 and the fluorescence image 32 are superimposed. Therefore, even when DAPI is used as the staining dye for obtaining the fluorescence image 32, the analysis device 10 of the present embodiment can accurately identify the calibration target region P1 (the first region PA). In addition, the analysis device 10 of the present embodiment can obtain the bright-field image 30 of nuclear staining without using an embedding dye, such as DAPI, as the fluorescence dye. Therefore, when displaying the display screen to be described later, the nuclear staining image based on the bright-field image 30 can be displayed on the fluorescence image 32 by using the bright-field image 30.

[0143] Note that the above combined image 38 may further include a fluorescence correction image 34. In this case, the storage control unit 20C can create the combined image 38 in which the color value of the bright-field image 30, the fluorescence intensity value of the fluorescence image 32, and the fluorescence intensity value of the fluorescence correction image 34 are defined for each pixel of each sample 40.

[0144] Return to Figure 1A , and the description will continue. Next, the analysis unit 20H will be described. The analysis unit 20H analyzes the fluorescence correction image 34 and derives an analysis result. The analysis unit 20H can analyze the entire fluorescence correction image 34, or can separately analyze the first region PA and the second region PB included in the fluorescence correction image 34. In addition, not limited to the difference between the first region PA and the second region PB, the analysis unit 20H can perform analysis on each region obtained by segmenting the fluorescence correction image 34 according to at least one of the color value of the bright-field image 30 and the fluorescence intensity value of the fluorescence correction image 34. For example, the analysis unit 20H can use the color value of the bright-field image 30 to identify each region of a specific cell included in the bright-field image 30, and analyze each region of the characteristic cell in the fluorescence correction image 34.

[0145] The analysis result is information indicating the analysis result of the analysis content. The analysis unit 20H derives the analysis result by analyzing the distribution (fluorescence signal, fluorescence spectrum) of the fluorescence intensity values included in the fluorescence correction image 34 (fluorescence correction images 34A to 34D).

[0146] The analysis results are, for example, the type, distribution, region, size, abundance, fraction, etc. of the target being analyzed. The target being analyzed is the above-mentioned target, etc. Specifically, the target is a tumor marker, lymphocyte marker, immune cell, immune checkpoint molecule, and specific tissue. In addition, the target being analyzed can be a tumor region, the target present in the tumor region (immune cell, immune checkpoint molecule), the number of tumor cells, the target-positive tumor cells per total number of tumor cells, etc.

[0147] The abundance of the target being analyzed as an analysis result is represented, for example, by quantitative evaluation values such as the number of cells, the number of molecules, the cell density in the region, the molecular density in the region, the number of biomarker-positive cells per specific number of cells, and the ratio between the number of biomarker-positive cells per specific number of cells and the number of biomarker-positive cells per specific number of cells. The distribution and region of the target being analyzed as an analysis result are represented, for example, by the intercellular distance, intercellular interaction, etc.

[0148] The analysis unit 20H can identify the target being analyzed by analyzing the bright-field image 30. In this case, for example, the analysis unit 20H can identify the target being analyzed by a known image processing method based on the color value of each pixel of the bright-field image 30G, the positional relationship of the pixels of each color value, etc. In addition, the bright-field image 30G can identify the target being analyzed based on the target information input by the user through the operation instruction of the input unit 24B. Then, the analysis unit 20H can analyze the target being analyzed in the fluorescence-corrected image 34.

[0149] Note that the analysis unit 20H can analyze the target being analyzed by a known method based on the distribution of the fluorescence intensity values in at least one of the first region PA and the second region PB which is a region other than the first region PA in the fluorescence-corrected image 34.

[0150] In addition, the analysis result can also include the analysis result of the bright-field image 30. The analysis result of the bright-field image 30 preferably includes, for example, information that is difficult to analyze from the fluorescence-corrected image 34. For example, the analysis result of the bright-field image 30 includes, but is not limited to, cell nuclei, intracellular organelles, cell membranes, tissue matrices, adipose sites, necrotic regions, carbon powder, etc.

[0151] Here, as described above, the bright-field image 30 and the fluorescence image 32 are not limited to the captured images of the same section 41 of the same sample 40, and can be the captured images of consecutive sections of the same sample 40.

[0152] In this case, the section 41 for the bright-field image 30 and the section 41 for the fluorescence image 32 are different sections 41 of the same sample 40. Therefore, an analysis result different from the actual state of the sample 40 can be obtained.

[0153] For example, assume that the analysis unit 20H derives the degree of infiltration of lymphocytes into tumor cells as an analysis result. In this case, the actual infiltration state of lymphocytes into tumor cells may be different from the infiltration state derived as the analysis result.

[0154] Therefore, in the analysis device 10 of the present embodiment, when different sections 41 (e.g., serial sections) of the same sample 40 are used as the bright-field image 30 and the fluorescence image 32, it is preferable to use the section 41 in the following staining state. Specifically, the analysis device 10 preferably uses the section 41 stained with HE and a specific cell marker as the section 41 for the bright-field image 30 and the section 41 for the fluorescence image 32.

[0155] Figures 8A to 8D It is an explanatory diagram of an example of the staining state of the bright-field image 30 and the fluorescence image 32. Figures 8A to 8D It shows a case where the sample 40 is a living tissue and includes tumor cells 42 and a plurality of lymphocytes 44. The lymphocyte 44 is an example of a specific cell.

[0156] Figure 8A It is a diagram showing an example of a cross-sectional view of the sample 40. As Figure 8A shown, assume that the sample 40 includes tumor cells 42 and lymphocytes 44. In addition, assume that among the plurality of lymphocytes 44 (lymphocytes 44A to 44F), lymphocytes 44A, 44D, and 44E are present in the tumor cells 42. In addition, among the plurality of lymphocytes 44, assume that lymphocytes 44B, 44C, and 44F are present outside the tumor cells 42.

[0157] Then, assume that the sample 40 is cut to prepare a first section 41A and a second section 41B as sections 41 that are continuous in the thickness direction. The first section 41A is an example of the section 41 for photographing the bright-field image 30. The second section 41B is an example of the section 41 for photographing the fluorescence image 32.

[0158] Then, the first section 41A and the second section 41B are stained, and the measurement unit 12 photographs the stained first section 41A and second section 41B, thereby obtaining the bright-field image 30 and the fluorescence image 32. Then, the generation unit 20G determines a first region PA corresponding to the calibration target region P1 of the bright-field image 30 in the fluorescence image 32 by using the superimposed image in which the bright-field image 30 and the fluorescence image 32 are superimposed.

[0159] Figures 8B to 8D It is an explanatory diagram of the staining state of the section 41. Figures 8B to 8D It shows different staining conditions.

[0160] Figure 8B It is an explanatory diagram of a superimposed image 50A of a bright-field image 30 of a first section 41A and a fluorescence image 32B of a second section 41B. In the bright-field image 30 of the first section 41A, tumor cells 42 are stained with HE. In the fluorescence image 32B of the second section 41B, a marker of lymphocytes 44 as specific cells is stained with a fluorescent dye.

[0161] In this case, in the superimposed image 50A of the bright-field image 30 and the fluorescence image 32, lymphocytes 44E are present in tumor cells 42. However, in the actual specimen 40, lymphocytes 44E are present outside tumor cells 42. Therefore, in this case, the generation unit 20G determines that lymphocytes 44E actually present in tumor cells 42 are lymphocytes 44 present outside tumor cells 42. Therefore, the actual infiltration state of lymphocytes 44 into tumor cells 42 is different from the infiltration state derived as an analysis result.

[0162] Figure 8C It is an explanatory diagram of a superimposed image 50B of a bright-field image 30 of a first section 41A and a fluorescence image 32B of a second section 41B. In the first section 41A, a marker of lymphocytes 44 is stained with a fluorescent dye, and in the second section 41B, tumor cells 42 are stained with HE.

[0163] In this case, in the superimposed image 50B of the bright-field image 30 and the fluorescence image 32, lymphocytes 44B are present in tumor cells 42. However, in the actual specimen 40, lymphocytes 44B are present outside tumor cells 42. Therefore, in this case, the generation unit 20G determines that lymphocytes 44B that do not actually exist in tumor cells 42 are lymphocytes 44 present in tumor cells 42. Therefore, the actual infiltration state of lymphocytes 44 into tumor cells 42 is different from the infiltration state derived as an analysis result.

[0164] Figure 8D It is an explanatory diagram of a bright-field image 30 of a first section 41A in which tumor cells 42 are stained with HE and a marker of lymphocytes 44 is stained with a fluorescent dye, and a fluorescence image 32B of a second section 41B in which tumor cells 42 are stained with HE and a marker of lymphocytes 44 is stained with a fluorescent dye.

[0165] In this case, a superimposed image 50 (not shown) of the bright-field image 30 and the fluorescence image 32 is an image whose state matches the actual state of the sample 40. Therefore, in this case, the actual infiltration state of lymphocytes 44 into tumor cells 42 matches the infiltration state derived as an analysis result.

[0166] Figure 9 is a diagram showing the positional relationship between tumor cells 42 and lymphocytes 44 and the staining states described Figures 8A to 8D In Figure 9 , "actual" indicates Figure 8A the positional relationship between tumor cells 42 and lymphocytes 44 in the actual specimen 40 shown. In Figure 9 , patterns 1 to 3 respectively correspond to Figure 8B Figures 8A to 8D.

[0167] In Figure 9 , "A" to "F" respectively correspond to lymphocytes 44A to 44F. In Figure 9 , "N.D." indicates that the superimposed image 50 does not include the expression information of the marker. In Figure 9 , "inside" indicates that it is determined that the corresponding lymphocyte 44 exists inside the tumor cell 42. In Figure 9 , "outside" indicates that it is determined that the corresponding lymphocyte 44 exists outside the tumor cell 42.

[0168] As Figure 9 shown, when using the section 41 in which the tumor cells 42 are stained with HE and the markers of the lymphocytes 44 are stained with a fluorescent dye in both the bright-field image 30 and the fluorescence image 32, the positional relationship between the lymphocytes 44 and the tumor cells 42 can be determined in the same state as the actual state of the sample 40.

[0169] Therefore, in the case where different sections 41 of the same sample 40 are used as the section 41 for the bright-field image 30 and the section 41 for the fluorescence image 32, the analysis device 10 preferably uses the section 41 stained under the following staining conditions.

[0170] That is, the bright-field image acquisition unit 20B preferably acquires the bright-field image 30 of the first section 41A of the sample 40, in which the cells are stained with HE and specific cells are stained (cells, for example, lymphocytes or their markers). In addition, the fluorescence image acquisition unit 20A preferably acquires the fluorescence image 32 of the second section 41B of the sample 40, in which the cells are stained with HE and specific cells are stained.

[0171] Returning to Figure 1A , the description will continue. Next, the display control unit 20I will be described.

[0172] The display control unit 20I generates a display screen on the display unit 24A, and the display screen includes at least one of the bright-field image 30, the fluorescence image 32, the fluorescence correction image 34, the superimposed image obtained by superimposing the bright-field image 30, the fluorescence image 32, or the fluorescence correction image 34, and the analysis result.

[0173] Figures 10A to 10F It is a schematic diagram showing an example of the display screen 60.

[0174] Figure 10A It is a diagram showing an example of the display screen 60A. The display screen 60A is an example of the display screen 60. As Figure 10A shown, the display control unit 20I displays a display screen 60 including a superimposed image 62A on the display unit 24A, where a fluorescence image 32 of a sample 40 corresponding to the bright-field image 30 is superimposed on the bright-field image 30. Note that the superimposed image 62A can be an image in which the bright-field image 30 is superimposed on the fluorescence image 32.

[0175] Figure 10B It is a diagram showing an example of the display screen 60B. The display screen 60B is an example of the display screen 60. For example, the display control unit 20I displays the bright-field image 30 and a region selection box Q for receiving selection of a specific region on the bright-field image 30. The user operates the input unit 24B to adjust the position and size of the region selection box Q on the bright-field image 30. For example, when an execution instruction signal or the like is received from the input unit 24B, the display control unit 20I generates a magnified image 62B in which the inside of the region selection box Q on the bright-field image 30 is magnified and displayed, and displays the magnified image on the display unit 24A. Note that the display control unit 20I can display a display screen 60B including a superimposed image obtained by superimposing the magnified image 62B on the bright-field image 30 on the display unit 24A.

[0176] Figure 10C It is a diagram showing an example of the display screen 60C. The display screen 60C is an example of the display screen 60. For example, as Figure 10B shown, the display control unit 20I displays a region selection box Q on the bright-field image 30. The user operates the input unit 24B to adjust the position and size of the region selection box Q on the bright-field image 30. For example, when an execution instruction signal or the like is received from the input unit 24B, the display control unit 20I generates a superimposed image 62C in which the inside of the region selection box Q of the fluorescence image 32 magnified at the same magnification is superimposed on the magnified image obtained by magnifying the inside of the region selection box Q on the bright-field image 30, and displays the superimposed image on the display unit 24A.

[0177] Figure 10D It is a diagram showing an example of the display screen 60D. The display screen 60D is an example of the display screen 60. For example, as Figure 10CAs shown, the display control unit 20I displays the bright-field image 30 and the superimposed image 62C on the display unit 24A. At this time, the display control unit 20I can further display a display screen 60D including the analysis result 64 on the display unit 24A. Figure 10D Information indicating the distribution of the analyzed target as an example of the analysis result 64 is shown. In addition, the display control unit 20I can display fluorescence signal information in the calibration target area P1 as the analysis result on the display unit 24A. In this case, the display unit 24A can display the calibration target area P1 and the fluorescence signal information in the calibration target area P1 on the bright-field image 30.

[0178] Note that the display form of the analysis result 64 is not limited to Figure 10D the display form shown. For example, the display control unit 20I can display the analysis result 64 near the corresponding area as information (annotation information) indicating an annotation of the corresponding area (for example, the first area PA or the second area PB). That is, the display control unit 20I can cause the display unit 24A to display annotation information based on the information about the calibration target area P1.

[0179] In addition, the display control unit 20I can display the analysis result 64 in three-dimensional (3D) display or bar graph display according to the analysis result 64.

[0180] Figure 10E is a diagram showing an example of the display screen 60E. The display screen 60E is an example of the display screen 60. For example, as Figure 10B shown, the display control unit 20I displays a region selection box Q on the bright-field image 30. The user operates the input unit 24B to adjust the position and size of the region selection box Q on the bright-field image 30. The display control unit 20I receives, for example, an execution instruction signal from the input unit 24B. Then, the display control unit 20I generates a superimposed image 62D in which the inside of the region selection box Q of the fluorescence image 32 magnified at the same magnification is superimposed on the magnified image obtained by magnifying the inside of the region selection box Q on the bright-field image 30, and displays the superimposed image on the display unit 24A.

[0181] At this time, the display control unit 20I can display an image schematically showing at least one of the magnified image in the region selection box Q of the bright-field image 30 and the magnified image in the region selection box Q of the fluorescence image 32, as Figure 10E shown. At this time, in the superimposed image 62D, the first region PA as the calibration target area P1 and the second region PB as the area outside the calibration target area P1 can be displayed in different display forms (see Figure 10E)。In addition, the display control unit 20I can also display a display screen 60D including the analysis result 64 on the display unit 24A. Although not shown, the display control unit 20I can also display the depth information of the display unit. In addition, the display control unit 20I can also display a display screen 60E including the analysis result 64 on the display unit 24A.

[0182] Figure 10F FIG. is a diagram showing an example of the display screen 60F. The display screen 60F is an example of the display screen 60. For example, as Figure 10B shown, the display control unit 20I displays a region selection frame Q on the bright-field image 30. The user operates the input unit 24B to adjust the position and size of the region selection frame Q on the bright-field image 30. For example, when receiving an execution instruction signal or the like from the input unit 24B, the display control unit 20I generates a superimposed image 62D in which a region selection frame Q of a fluorescence image 32 magnified at the same magnification is superimposed on a magnified image obtained by magnifying the inside of the region selection frame Q on the bright-field image 30, and displays the superimposed image on the display unit 24A.

[0183] At this time, the display control unit 20I can display an image schematically showing at least one of the magnified image in the region selection frame Q of the bright-field image 30 and the magnified image in the region selection frame Q of the fluorescence image 32, as Figure 10F shown. The display control unit 20I can remove the first region PA that is the calibration target region P1 and selectively display the second region PB that is a region other than the calibration target region P1 (see Figure 10F ). In addition, the display control unit 20I can also display a display screen 60D including the analysis result 64 on the display unit 24A.

[0184] Note that the display control unit 20I can display the display screen 60 on the display unit 24A, and the display screen also includes the morphological information of at least one of the first region PA and the second region PB that are the calibration target regions P1 included in the fluorescence correction image 34. Note that the analysis result 64 can include morphological information.

[0185] The morphological information of the first region PA and the second region PB is information indicating the position, size, and range of each of the first region PA and the second region PB on the fluorescence-corrected image 34. Specifically, the morphological information of the first region PA and the second region PB is represented by a display form indicating its position, size, and range. The display form is, for example, a frame line indicating the outer shape of the first region PA and the second region PB, a specific color indicating the first region PA and the second region PB, a blinking display or a highlighting display of the first region PA and the second region PB, a display with increased brightness of the first region PA and the second region PB, etc. The blinking display includes blinking, in which the brightness of at least one of the first region PA and the second region PB changes periodically. The highlighting display includes a display that attracts attention by color or size.

[0186] Note that when a user or the like indicates a specific position in the display screen 60 by an operation instruction of the input unit 24B, the display control unit 20I can further display an image related to the image displayed at the indicated position. For example, assume that the analysis result 64 is displayed at the indicated position. In this case, the display control unit 20I can search the storage unit 22 for another analysis result 64 that matches or is similar to the analysis result 64, at least one of the bright-field image 30, the fluorescence image 32, and the fluorescence-corrected image 34 used for analyzing the analysis result 64, and further display the searched result on the display unit 24A.

[0187] In addition, the display control unit 20I can display the analysis result 64 of each analysis region.

[0188] Specifically, when analysis is performed on each of the first region PA and the second region PB, the display control unit 20I can display the analysis result of the first region PA and the analysis result of the second region PB on the display unit 24A.

[0189] In addition, assume that the display control unit 20I has performed analysis on each region obtained by segmenting the fluorescence-corrected image 34 based on at least one of the color value of the bright-field image 30 and the fluorescence intensity value of the fluorescence-corrected image 34. In this case, the display control unit 20I can display the analysis result 64 of each region on the display unit 24A. By displaying the analysis result 64 of each region, an annotation of each region can be displayed. At this time, the display control unit 20I can generate a line that divides the cell regions of each cell included in the bright-field image 30, and display the cell regions so that they are superimposed on the fluorescence image 32.

[0190] In addition, the display control unit 20I can display a histogram as the analysis result 64, in which the measurement parameters of each cell are plotted on the channel axis. Specifically, a dot plot similar to the curve graph of a flow cytometer can be displayed.

[0191] Note that when receiving a screen switching signal, the display control unit 20I can switch the display screen 60. For example, the user operates the input unit 24B to indicate screen switching. In response to the screen switching instruction of the operation of the input unit 24B, the display control unit 20I receives the screen switching signal from the input unit 24B.

[0192] When receiving the screen switching signal, the display control unit 20I changes one of the bright-field image 30, the fluorescence image 32, the fluorescence correction image 34, the superimposed image, and the analysis result 64 displayed on the display screen 60 to any other image. In addition, the display control unit 20I can switch the displayed display screen 60 (display screens 60A to 60F) to another display screen 60 (display screens 60A to 60F).

[0193] In addition, the display control unit 20I can display at least one of the bright-field image 30, the fluorescence image 32, and the fluorescence correction image 34 during the user's reception of the operation of the input unit 24B (e.g., mouse operation), and display the analysis result 64 when the operation stops. In addition, the display control unit 20I can display the analysis result 64 during the user's reception of the operation of the input unit 24B (e.g., mouse operation), and display at least one of the bright-field image 30, the fluorescence image 32, and the fluorescence correction image 34 when the operation stops.

[0194] In addition, the display control unit 20I can use the combined image 38 to switch the display screen 60.

[0195] Figure 11 is a schematic diagram showing an example of switching the display screen 60 using the combined image 38. As described above, the combined image 38 is an image obtained by combining the bright-field image 30 and the fluorescence image 32 of each corresponding pixel.

[0196] The display control unit 20I receives, for example, a display instruction signal from the input unit 24B. Then, the display control unit 20I displays on the display unit 24A a display screen 60G including the bright-field image 30 (bright-field image 30R, bright-field image 30G, and bright-field image 30B) included in the combined image 38 (step S10). Then, the display control unit 20I can switch the display of the display screen 60G including the bright-field image 30 and the display of the display screen 60H including the fluorescence image 32 each time it receives the screen switching signal (step S12).

[0197] At this time, the display control unit 20I can easily display a display screen 60H including the fluorescence image 32 on the display unit 24A by extracting the fluorescence image 32 from the combined image 38 including the bright-field image 30 being displayed. Similarly, by extracting the bright-field image 30 from the combined image 38 including the fluorescence image 32 being displayed, the display control unit 20I can easily display a display screen 60H including the bright-field image 30 on the display unit 24A. That is, the display control unit 20I can easily switch the display screen 60 by using the combined image 38.

[0198] Next, an example of the process of the information processing executed by the analysis device 10 will be described.

[0199] Figure 12 It is a flowchart showing an example of the process of the information processing executed by the analysis device 10.

[0200] First, the fluorescence image acquisition unit 20A acquires a fluorescence image 32 (step S100). Next, the bright-field image acquisition unit 20B acquires a bright-field image 30 of the sample 40 corresponding to the fluorescence image 32 acquired in step S100 (step S102).

[0201] The storage control unit 20C stores the fluorescence image 32 acquired in step S100 and the bright-field image 30 acquired in step S102 in the storage unit 22 in association with each other for the same sample 40 (step S104).

[0202] Therefore, whenever the fluorescence image acquisition unit 20A and the bright-field image acquisition unit 20B acquire the fluorescence image 32 and the bright-field image 30, the fluorescence image 32 and the bright-field image 30 corresponding to the sample 40 are sequentially stored in the storage unit 22.

[0203] Note that at this time, the storage control unit 20C can generate a combined image 38 using the bright-field image 30 and the fluorescence image 32 of each sample 40 and store the combined image in the storage unit 22.

[0204] The extraction unit 20E extracts fluorescence correction information from the bright-field image 30 acquired in step S102 (step S106).

[0205] The recognition unit 20F identifies a calibration target region P1 included in the bright-field image 30 acquired in step S102 based on the fluorescence correction information extracted in step S106 (step S108). The recognition unit 20F stores the calibration target region P1 determined in step S108 in the storage unit 22 in association with the bright-field image 30 used to identify the calibration target region P1 (step S110).

[0206] Therefore, the storage unit 22 stores the bright-field image 30, the fluorescence images 32 (fluorescence images 32A to 32D), and the morphological information of the calibration target region P1 in association with each other for each sample 40.

[0207] At this time, the learning unit 20D can learn the learning model 23. Note that the learning time of the learning unit 20D for the learning model 23 is not limited to this time.

[0208] Next, the generation unit 20G generates a fluorescence corrected image 34 based on the fluorescence image 32 obtained in step S100 and the fluorescence correction information extracted in step S106 (step S112). The generation unit 20G generates the fluorescence corrected image 34 by using the fluorescence image 32 obtained in step S100 and the calibration target region P1 identified based on the fluorescence correction information in step S108.

[0209] The analysis unit 20H analyzes the fluorescence corrected image 34 generated in step S112 and derives an analysis result 64 (step S114).

[0210] The display control unit 20I generates a display screen 60 on the display unit 24A, and the display screen includes at least one of the analysis result 64 derived in step S114, the fluorescence image 32 obtained in step S100, and the bright-field image 30 obtained in step S102 (step S116).

[0211] The display control unit 20I determines whether a screen switching signal has been received (step S118). When it is determined that a screen switching signal has been received (step S118: Yes), the process proceeds to step S120.

[0212] In step S120, the display control unit 20I switches the display screen 60 displayed on the input unit 24B to another display screen 60 (step S120). Then, the process proceeds to step S122. When the determination made in step S118 is negative (step S118: No), the process similarly proceeds to step S122.

[0213] In step S122, the display control unit 20I determines whether to end the display of the display screen 60 on the display unit 24A (step S122). For example, the display control unit 20I determines whether a signal indicating an end display instruction has been received from the input unit 24B, thereby making a determination in step S122. When the determination made in step S122 is negative (step S122: No), the process returns to step S118. When the determination made in step S122 is affirmative (step S122: Yes), the process proceeds to step S124.

[0214] The display control unit 20I determines whether to end the analysis (step S124). For example, the display control unit 20I determines whether a signal indicating an analysis end instruction has been received from the input unit 24B, and thus makes a determination in step S124. When the determination made in step S124 is negative (step S124: No), the process returns to step S100. When the determination made in step S124 is positive (step S124: Yes), the process ends.

[0215] Note that there is a case where the display control unit 20I receives a display instruction signal during the processing of any one of steps S100 to S114. In this case, when the display instruction signal is received, the display control unit 20I may interrupt the processing and execute the processing of steps S116 to S122.

[0216] As described above, the information processing apparatus 1 of the present embodiment includes an extraction unit 20E and a generation unit 20G. The extraction unit 20E extracts fluorescence correction information from the bright-field image 30 of the sample 40. The generation unit 20G generates a fluorescence correction image 34 based on the fluorescence information and the fluorescence correction information of the sample 40.

[0217] Here, in the prior art, it is sometimes difficult to analyze a sample with high accuracy. Specifically, for example, in the prior art, a fluorescence image 32 is used for analysis, and autofluorescence components included in the fluorescence image 32 and the like may affect the analysis result. That is, in the prior art, there is a case where the fluorescence dye component and the autofluorescence component cannot be completely distinguished. Specifically, such a problem has occurred in cases where the spectrum of the fluorescence dye and the autofluorescence spectrum are close in wavelength, the shapes of these spectra are similar, and the intensity of the autofluorescence spectrum is stronger than the spectrum of the fluorescence dye.

[0218] Specifically, in ordinary filter-type photography, rather than multi-channel photography (e.g., spectroscopy), it is difficult to distinguish the spectrum of the fluorescence dye and the spectrum of the autofluorescence component. In addition, even when the wavelength resolution is improved as in spectroscopic photography, the autofluorescence signal may remain depending on the shooting position.

[0219] For example, in a case where a region having a partial weak fluorescence intensity exists in the edge portion, the central portion, etc. of the red blood cell region, the shape of the autofluorescence spectrum may be changed due to the influence of noise or the like during shooting. In addition, it is known that the preservation state of the sample 40, the tissue fixation method, etc. affect the shape and intensity of the autofluorescence spectrum. Therefore, the shape of the autofluorescence spectrum may be changed. In this case, even when standard autofluorescence spectrum data is acquired in advance and autofluorescence correction is performed, sufficient autofluorescence correction may not be possible because the shape of the photographed autofluorescence spectrum is changed.

[0220] As described above, in the prior art, it is sometimes difficult to analyze a sample with high accuracy.

[0221] On the other hand, in the information processing apparatus 1 of the present embodiment, the extraction unit 20E extracts fluorescence correction information from the bright-field image 30 of the sample. The generation unit 20G generates a fluorescence correction image 34 based on the fluorescence information and the fluorescence correction information of the sample.

[0222] That is, the information processing apparatus 1 of the present embodiment uses the fluorescence correction information extracted from the bright-field image 30 to generate a fluorescence correction image 34 from the fluorescence information (e.g., fluorescence image 32) of the sample. Therefore, by using the fluorescence correction image 34 for analysis, the sample 40 can be analyzed with high accuracy.

[0223] Therefore, the information processing apparatus 1 of the present embodiment can analyze the sample 40 with high accuracy.

[0224] In addition, the fluorescence image 32 may include local fluorescence dye noise derived from debris, dirt, fluorescence dye aggregates, debris generated due to tissue section peeling, and the like. However, the information processing apparatus 1 of the present embodiment corrects the calibration target region P1 of the fluorescence image 32 using the calibration target region P1 of the bright-field image 30 identified based on the fluorescence correction information. Since the information processing apparatus 1 of the present embodiment performs analysis using the fluorescence correction image 34 obtained by easily correcting the calibration target region P1 including noise and the like included in the fluorescence image 32, the sample 40 can be analyzed with high accuracy.

[0225] Here, cells or regions in the sample 40 that are not fluorescently stained are regarded as regions where nothing exists in the fluorescence image 32. Therefore, in the conventional analysis technique using the fluorescence image 32, the analysis accuracy may deteriorate. For example, in the case of analyzing the ratio of negative cells expressing a marker contained in the sample 40, when the presence or absence of cells cannot be identified from the fluorescence image 32, it is impossible to analyze the ratio. Therefore, there is also a method of using the fluorescence image 32 of the sample 40 stained with cell nuclei. However, for the sample 40 stained only with cell nuclei, the segmentation of cells is insufficient, and it is difficult to analyze the accurate ratio.

[0226] In addition, in the case where the fluorescence image 32 includes a region with a weak fluorescence signal or a region without a fluorescence signal, when analyzing the fluorescence image 32, it is difficult to identify whether there is no tissue or the fluorescence signal is weak. For this reason, in the prior art, it is difficult to obtain the morphological information and the overall image of the target to be analyzed, and it is difficult to analyze the sample 40 with high accuracy.

[0227] On the other hand, the information processing apparatus 1 of the present embodiment uses the bright-field image 30 to identify the calibration target region P1, and generates a fluorescence corrected image 34 obtained by correcting the calibration target region P1 included in the fluorescence image 32. Then, the analysis unit 20H analyzes the fluorescence corrected image 34. Therefore, when the display control unit 20I displays, the nucleus, intracellular organelles, cell membrane, tissue matrix, fat part, necrotic region, etc. can be further displayed as the analysis result 64 based on the bright-field image 30, and necessary information can be added to and provided for the fluorescence corrected image 34.

[0228] In addition, in the present embodiment, the identification unit 20F identifies the calibration target region P1 as the autofluorescence region or light absorption region included in the bright-field image 30. Therefore, the generation unit 20G can accurately correct the autofluorescence region or light absorption region included in the fluorescence image 32. Therefore, the information processing apparatus 1 of the present embodiment can accurately analyze the sample 40.

[0229] Here, it is difficult to distinguish the signal from the fluorescent dye and the signal from the autofluorescence only by the fluorescence image 32. On the other hand, the autofluorescence region such as red blood cells can be determined by the bright-field image 30. Therefore, the identification unit 20F identifies the calibration target region P1 as the autofluorescence region or light absorption region included in the bright-field image 30, and uses the calibration target region P1 to correct the fluorescence image 32, so that the sample 40 can be accurately analyzed.

[0230] In addition, the identification unit 20F identifies the position, size, and range of the calibration target region P1 included in the bright-field image 30. Therefore, the generation unit 20G can accurately identify and correct the first region PA corresponding to the calibration target region P1 included in the fluorescence image 32. Therefore, the information processing apparatus 1 of the present embodiment can accurately analyze the sample 40 by using the fluorescence corrected image 34 for analysis.

[0231] In addition, the identification unit 20F uses the learning model 23 and the acquired bright-field image 30 to determine the calibration target region P1, where the bright-field image 30 is used as the input and the calibration target region P1 is used as the output. By using the learning model 23 to determine the calibration target region P1, the calibration target region P1 can be determined with high accuracy and high speed.

[0232] In addition, the generation unit 20G generates a fluorescence corrected image 34 obtained by removing the region (first region PA) corresponding to the calibration target region P1. Therefore, the information processing apparatus 1 of the present embodiment can accurately analyze the sample 40 by analyzing the fluorescence corrected image 34.

[0233] In addition, the generation unit 20G generates a fluorescence correction image 34 based on the peripheral information of the correction target region P1. Therefore, the generation unit 20G can generate the fluorescence correction image 34 obtained by easily and accurately correcting the fluorescence image 32.

[0234] In addition, the fluorescence image acquisition unit 20A acquires a fluorescence image 32 captured under the imaging conditions where the brightness of the correction target region P1 is saturated.

[0235] Here, in the conventional color separation method based on the fluorescence spectrum shape, for example, when correcting the autofluorescence component from red blood cells, it is necessary to perform imaging under the imaging conditions where a part of the red blood cell component is not saturated during observation. However, since the autofluorescence component of the red blood cell component indicates a high value, it is necessary to perform imaging under the imaging conditions for suppressing the fluorescence signal, for example, shortening the exposure time and using an ND filter. However, in this conventional method, it may be difficult to image the fluorescence signal with a weak fluorescence intensity included in the sample 40.

[0236] On the other hand, in the present embodiment, the fluorescence image 32 is corrected by identifying the correction target region P1 based on the bright-field image 30, and the corrected fluorescence correction image 34 is used for analysis. That is, in the present embodiment, the bright-field image 30 is used to determine the correction target region P1, which is the autofluorescence region derived from red blood cells and the like. Therefore, in the information processing apparatus 1 of the present embodiment, it is not necessary to adjust the imaging conditions of the fluorescence image 32 to the imaging conditions where the brightness of the correction target region P1 is not saturated. In addition, the information processing apparatus 1 of the present embodiment can acquire the fluorescence image 32 captured under the imaging conditions where the brightness of the correction target region P1 is saturated, so that the fluorescence signal with a weak fluorescence intensity included in the sample 40 can be imaged.

[0237] In addition, the bright-field image acquisition unit 20B acquires a bright-field image 30 of the section 41 of the HE-stained sample 40.

[0238] Here, in the prior art, an embedding dye (for example, DAPI) is used as a fluorescence dye for nuclear staining. However, DAPI has a wide fluorescence spectrum and causes leakage to the wavelengths of other fluorescence dyes. In addition, since the presence or absence of nuclear staining is usually determined by the wavelength shift during embedding, the degree of embedding is affected by the wavelength shift, and thus the shape of the standard spectrum tends to change. For this reason, in the prior art, there is a problem of reduced color separation accuracy.

[0239] On the other hand, in the present embodiment, the bright-field image 30 of the section 41 of the HE-stained sample 40 is used as the bright-field image 30. Therefore, in the information processing apparatus 1 of the present embodiment, the position of the cell nucleus can be determined based on the bright-field image 30 which is an HE-stained image. Thus, in the information processing apparatus 1 of the present embodiment, the display control unit 20I can display the cell nucleus stained image on the fluorescence image 32 according to the bright-field image 30 without using an embedding dye. Further, by using the bright-field image 30, even when the embedding dye is used in the fluorescence image 32, the analysis unit 20H can easily determine the position of the cell nucleus.

[0240] Note that the display control unit 20I uses an image display control program to display the above fluorescence image or bright-field image. However, the present invention is not limited thereto, and the image display control program can be downloaded from a server or installed from a storage medium such as a digital versatile disc (DVD) into a general-purpose computer to implement the processing by the display control unit 20I described below. Further, the processing executed by the display control unit 20I can be implemented by executing the processing by two or more devices. For example, some processing is executed on a server, and other processing is executed on a computer (e.g., the display control unit H). Further, the image display control program can be operated in the cloud to implement the processing by the display control unit 20I described below.

[0241] In addition, the bright-field image acquisition unit 20B acquires the bright-field image 30 of the first section 41A of the sample 40 which is HE-stained and in which a specific cell is stained, and the fluorescence image acquisition unit 20A acquires the fluorescence image 32 of the second section 41B of the sample 40 which is HE-stained and in which a specific cell is stained.

[0242] Therefore, it is possible to suppress obtaining an analysis result different from the positional relationship between the tumor cells 42 and the lymphocytes 44 in the actual sample 40.

[0243] Specifically, in the prior art, the bright-field image 30 of the HE-stained sample 40 is used as the bright-field image 30 for pathological diagnosis. Therefore, it is difficult to determine the target from the bright-field image 30. Thus, in the prior art, it is difficult to accurately grasp the target region expressing the tumor cells 42, and it is difficult to determine the type of immune cells, for example, helper T cells or cytotoxic T cells.

[0244] On the other hand, fluorescence staining is used to determine the target. However, conventional fluorescence images are dark-field images, and unstained objects are dark-field. Therefore, the morphological information of tissues and cells is lost. Thus, it is difficult to determine the tumor area in conventional fluorescence images. In addition, a method using anti-cytokeratin antibody to assist in determining the tumor area has been disclosed, but since anti-cytokeratin antibody is an epithelial marker, the tumor area and the number of tumor cells cannot be accurately determined.

[0245] In melanoma, the biomarker SOX10 for determining the tumor area is used as the target, and research on determining the tumor area by immunostaining has been conducted. As a lung cancer marker, thyroid transcription factor-1 (TTF-1) is used for immunostaining. However, although TTF-1 has a high positive rate of approximately 80% in lung adenocarcinoma and can be used to determine the tumor area, it has a low expression level in squamous cell carcinoma of the lung and usually indicates negative. Therefore, it cannot be applied to the determination of the tumor area. In addition, there are few targets that can determine the tumor area, and the tumor area cannot be determined only by fluorescence staining.

[0246] On the other hand, in the present embodiment, the bright-field image 30 of the first section 41A of the sample 40 stained with HE and with specific cells stained and the fluorescence image 32 of the second section 41B of the sample 40 stained with HE and with specific cells stained are used. Therefore, the superimposed image 50 of the bright-field image 30 and the fluorescence image 32 is an image whose state matches the actual state of the sample 40. Thus, the information processing device 1 of the present embodiment can determine the positional relationship between the lymphocytes 44 and the tumor cells 42 in a state identical to the actual state of the sample 40.

[0247] In addition, for example, the information processing device 1 of the present embodiment can determine immune cells (such as lymphocytes 44) that have infiltrated into the tumor cells 42, measure the number and distance of the immune cells, and the information processing device 1 of the present embodiment can accurately analyze the distribution of biomarkers (immune checkpoint molecules, molecules used as indicators of molecular targeted drugs, etc.), which are examples of targets expressed by the tumor cells 42, etc. In addition, when determining the tumor area (for example, correcting the target area P1), the information processing device 1 of the present embodiment can accurately determine the tumor area by using the superimposed image 50 of the bright-field image 30 and the fluorescence image 32.

[0248] In addition, in the information processing apparatus 1 of the present embodiment, since tumor cells can be determined from the bright-field image 30, not only can the tumor region be calculated, but also the number of tumor cells can be calculated. Further, in the information processing apparatus 1 of the present embodiment, the expression of a biomarker can be detected by the fluorescence image 32. Further, the information processing apparatus 1 of the present embodiment can calculate the number of biomarker-positive or -negative cells per tumor cell number by superimposing the bright-field image 30 and the fluorescence image 32. Incidentally, machine learning can be used to determine the number of tumor cells and the area of each tumor cell. For example, the information processing apparatus 1 according to the present embodiment can calculate the number of HER2-positive cells per total number of tumor cells in a tissue section. Specifically, for example, by analyzing the fluorescence-corrected image 34, the information processing apparatus 1 can obtain an analysis result in which 50 out of 100 tumor cells present in a tissue sample are HER2-positive cells, and the ratio of HER2-positive tumor cells is 50%.

[0249] Further, the analysis unit 20H analyzes the analyzed target included based on the distribution of the fluorescence intensity values in at least one of the first region PA, which is the calibration target region P1 included in the fluorescence-corrected image 34, and the second region PB other than the first region PA. Thus, the information processing apparatus 1 of the present embodiment can accurately analyze the first region PA and the second region PB.

[0250] Further, the display control unit 20I displays a display screen 60 on the display unit 24A, and the display screen also includes the morphological information of at least one of the first region PA, which is the calibration target region P1 included in the fluorescence-corrected image 34, and the second region PB other than the first region PA. The display control unit 20I also displays the morphological information of at least one of the first region PA and the second region PB, whereby more detailed analysis results 64 can be provided to the user.

[0251] Further, the morphological information of the first region PA and the second region PB is represented by a display form indicating their positions, sizes, and ranges. The display form is, for example, a frame line indicating the outer shapes of the first region PA and the second region PB, a specific color indicating the first region PA and the second region PB, a blinking display (flashing) or a highlighting display of the first region PA and the second region PB, a display with increased brightness of the first region PA and the second region PB, or the like.

[0252] In the information processing apparatus 1 of the present embodiment, the bright-field image 30 is used to correct the fluorescence image 32. The display control unit 20I displays a display screen 60 on the display unit 24A, and the display screen further includes morphological information of at least one of a first region PA that is a calibration target region P1 included in the fluorescence correction image 34 and a second region PB other than the first region PA. Therefore, in the information processing apparatus 1 of the present embodiment, information about non-fluorescently stained cells can be easily further displayed as morphological information of at least one of the first region PA and the second region PB. In addition, the information processing apparatus 1 according to the present embodiment can easily provide information that can be used for cell segmentation by HE staining and fluorescence staining and for analyzing the same cells.

[0253] In addition, when a screen switching signal is received, the display control unit 20I changes one of the bright-field image 30, the fluorescence image 32, the fluorescence correction image 34, and the analysis result 64 displayed on the display screen 60 to any other image. Therefore, the display control unit 20I of the present embodiment can change the image according to the user's intention.

[0254] In addition, the storage control unit 20C stores the combined image 38 in the storage unit 22 for each sample 40, where the color value of the bright-field image 30 and the fluorescence intensity value of the fluorescence image 32 are defined for each pixel. By using the combined image 38 for display, it is possible to easily perform display switching between the bright-field image 30 and the fluorescence image 32. Note that the combined image 38 may be a combined image 38 in which the color value of the bright-field image 30, the fluorescence intensity value of the fluorescence image 32, and the fluorescence intensity value of the fluorescence correction image 34 are defined for each pixel.

[0255] In addition, the fluorescence image acquisition unit 20A can acquire a fluorescence image 32 in which the fluorescence intensity value corresponding to each pixel of the bright-field image 30 acquired by the bright-field image acquisition unit 20B in the combined image 38 is defined for each pixel. In addition, the bright-field image acquisition unit 20B can acquire a bright-field image 30 in which the color value corresponding to each pixel of the fluorescence image 32 acquired by the fluorescence image acquisition unit 20A in the combined image 38 is defined for each pixel.

[0256] Note that the application target of the information processing apparatus 1 of the present embodiment is not limited. For example, the information processing apparatus 1 can be applied to a fluorescence in situ hybridization method (FISH method) or the like.

[0257] Note that although the embodiments of the present disclosure have been described above, the processes according to the above embodiments can be performed in various different embodiments other than the above embodiments. In addition, the above embodiments can be appropriately combined within a range where the processing contents do not conflict with each other.

[0258] In addition, the effects described in this specification are merely examples and are not limited thereto. There may be other effects.

[0259] (Hardware Configuration)

[0260] Figure 13 FIG. is a hardware configuration diagram showing an example of a computer 1000 that implements the functions of the information processing apparatus 1 according to the above-described embodiment.

[0261] The computer 1000 includes a CPU 1100, a random access memory (RAM) 1200, a read only memory (ROM) 1300, a hard disk drive (HDD) 1400, a communication interface 1500, and an input / output interface 1600. Each unit of the computer 1000 is connected via a bus 1050.

[0262] The CPU 1100 performs operations based on programs stored in the ROM 1300 or the HDD 1400 and controls each unit. For example, the CPU 1100 develops programs stored in the ROM 1300 or the HDD 1400 in the RAM 1200 and executes processing corresponding to various programs.

[0263] The ROM 1300 stores a boot program, for example, a basic input / output system (BIOS) executed by the CPU 1100 when the computer 1000 is activated, programs depending on the hardware of the computer 1000, and the like.

[0264] The HDD 1400 is a non-transitory computer-readable recording medium that records programs executed by the CPU 1100, data used by the programs, and the like. Specifically, the HDD 1400 is a recording medium that records programs according to the present disclosure, and the program is an example of program data 1450.

[0265] The communication interface 1500 is an interface for the computer 1000 to connect to an external network 1550 (for example, the Internet). For example, the CPU 1100 receives data from another device or transmits data generated by the CPU 1100 to another device via the communication interface 1500.

[0266] The input / output interface 1600 is an interface for connecting an input / output device 1650 and a computer 1000. For example, the CPU 1100 receives data from input devices such as a keyboard and a mouse via the input / output interface 1600. In addition, the CPU 1100 transmits data to output devices such as a display, a speaker, or a printer via the input / output interface 1600. Further, the input / output interface 1600 can be used as a medium interface for reading a program or the like recorded in a predetermined recording medium (medium). The medium is, for example, an optical recording medium such as a digital versatile disc (DVD) or a phase change rewritable disc (PD), a magneto-optical recording medium such as a magneto-optical disc (MO), a tape medium, a magnetic recording medium, a semiconductor memory, or the like.

[0267] For example, when the computer 1000 functions as the information processing device 1 according to the above-described embodiment, the CPU 1100 of the computer 1000 executes an information processing program loaded on the RAM 1200 to implement functions such as the fluorescence image acquisition unit 20A. Further, the HDD 1400 stores programs and data according to the present disclosure in the storage unit 22. The CPU 1100 reads the program data 1450 from the HDD 1400 and executes the program data. However, as another example, the program may be acquired from another device via the external network 1550.

[0268] Note that the present technology may also be configured as follows.

[0269] (1) An information processing device, comprising:

[0270] an extraction unit that extracts fluorescence correction information from a bright-field image of a sample; and

[0271] a generation unit that generates a fluorescence correction image based on the fluorescence information and the fluorescence correction information of the sample.

[0272] (2) The information processing device according to (1), further comprising:

[0273] a fluorescence image acquisition unit that acquires a fluorescence image including the fluorescence information of the sample; and

[0274] a bright-field image acquisition unit that acquires a bright-field image.

[0275] (3) The information processing device according to (1), wherein the fluorescence correction information is selected from color information, morphological information, or staining information of the bright-field image.

[0276] (4) The information processing device according to (1), further comprising:

[0277] An identification unit that determines a calibration target area of a bright-field image based on fluorescence calibration information, where

[0278] A generation unit

[0279] Generates a fluorescence calibration image obtained by removing the area corresponding to the calibration target area.

[0280] (5) The information processing device according to (1) further includes:

[0281] An identification unit that determines a calibration target area of a bright-field image based on fluorescence calibration information, where

[0282] A generation unit

[0283] Generates a fluorescence calibration image based on the peripheral information of the calibration target area.

[0284] (6) The information processing device according to (4), where

[0285] A generation unit

[0286] Generates a fluorescence calibration image obtained by removing the area corresponding to the calibration target area corrected based on the weight value according to the standard spectrum derived from the sample.

[0287] (7) The information processing device according to (4), where

[0288] The calibration target area

[0289] Is an autofluorescence area or a light absorption area included in the bright-field image.

[0290] (8) The information processing device according to (1) further includes:

[0291] A learning unit that learns the correspondence between fluorescence information and fluorescence calibration information, where

[0292] A generation unit

[0293] Generates a fluorescence calibration image based on the learning result of the learning unit.

[0294] (9) The information processing device according to (2) further includes:

[0295] A display unit that displays at least one or more of the bright-field image, the fluorescence image including the fluorescence information of the sample, the fluorescence calibration image, and the superimposed image obtained by superimposing the bright-field image and the fluorescence image or the fluorescence calibration image.

[0296] (10) The information processing device according to (9), where

[0297] Prompts the display unit

[0298] Display annotation information based on information about the calibration target area of the bright-field image.

[0299] (11) The information processing apparatus according to (9), wherein

[0300] Cause the display unit

[0301] To display the calibration target area of the bright-field image and the fluorescence signal information in the calibration target area in the bright-field image.

[0302] (12) The information processing apparatus according to (1), wherein the sample includes the same tissue section or consecutive tissue sections.

[0303] (13) The information processing apparatus according to (9), wherein

[0304] The display unit

[0305] Change one of the bright-field image, fluorescence image, fluorescence calibration image, and superimposed image being displayed when a screen switching signal is received to any other image.

[0306] (14) The information processing apparatus according to (1), wherein

[0307] The sample includes a plurality of samples, and wherein

[0308] The information processing apparatus further includes a storage control unit that stores, for each sample, a combined image that defines a fluorescence intensity value of a fluorescence image for each pixel, the fluorescence image including a color value of the bright-field image and fluorescence information of each sample.

[0309] (15) The information processing apparatus according to (1), further comprising:

[0310] A learning unit that learns the correspondence between the fluorescence information of the sample and the bright-field image, wherein

[0311] A generation unit

[0312] Generate a bright-field calibration image based on the learning result of the learning unit.

[0313] (16) A program that causes a computer to perform the following steps:

[0314] Extract fluorescence calibration information from the bright-field image of the sample; and

[0315] Generate a fluorescence calibration image based on the fluorescence information and the fluorescence calibration information of the sample.

[0316] List of reference numerals

[0317] 1 Information processing device

[0318] 10 Analysis device

[0319] 20A Fluorescent image acquisition unit

[0320] 20B Bright-field image acquisition unit

[0321] 20C Storage control unit

[0322] 20D Learning unit

[0323] 20E Extraction unit

[0324] 20F Recognition unit

[0325] 20G Generation unit

[0326] 20H Analysis unit

[0327] 20I Display control unit

[0328] 30 Bright-field image

[0329] 32 Fluorescent image

[0330] 34 Fluorescent correction image

[0331] 40 Sample

[0332] 41 Section

[0333] 41A First section

[0334] 41B Second section.

Claims

1. An information processing apparatus, comprising: an extraction unit that extracts fluorescence correction information from a bright-field image of a sample; and a generation unit that generates a fluorescence correction image based on the fluorescence information of the sample and the fluorescence correction information.

2. The information processing apparatus according to claim 1, further comprising: a fluorescence image acquisition unit that acquires a fluorescence image including the fluorescence information of the sample; and a bright-field image acquisition unit that acquires the bright-field image.

3. The information processing apparatus according to claim 1, wherein The fluorescence correction information is selected from color information, morphological information, or staining information of the bright-field image.

4. The information processing apparatus according to claim 1, further comprising: an identification unit that identifies a correction target region of the bright-field image based on the fluorescence correction information, wherein the generation unit: generates a fluorescence correction image obtained by removing a region corresponding to the correction target region.

5. The information processing apparatus according to claim 1, further comprising: an identification unit that identifies a correction target region of the bright-field image based on the fluorescence correction information, wherein the generation unit: generates the fluorescence correction image based on peripheral information of the correction target region.

6. The information processing apparatus according to claim 4, wherein the generation unit: generates a fluorescence correction image obtained by removing a region corresponding to the correction target region corrected based on a weight value, the weight value being obtained from a standard spectrum derived from the sample.

7. The information processing apparatus according to claim 4, wherein the correction target region: is an autofluorescence region or a light absorption region included in the bright-field image.

8. The information processing apparatus according to claim 1, further comprising: a learning unit that learns a correspondence relationship between the fluorescence information and the fluorescence correction information, wherein the generation unit: generates the fluorescence correction image based on a learning result of the learning unit.

9. The information processing apparatus according to claim 2, further comprising: a display unit that displays at least one or more of the bright-field image, the fluorescence image including the fluorescence information of the sample, the fluorescence correction image, and a superimposed image obtained by superimposing the bright-field image and the fluorescence image or the fluorescence correction image.

10. The information processing apparatus according to claim 9, wherein the display unit: displays annotation information based on information about a correction target region of the bright-field image.

11. The information processing apparatus according to claim 9, wherein the display unit: displays the correction target region of the bright-field image and fluorescence signal information in the correction target region in the bright-field image.

12. The information processing apparatus according to claim 1, wherein The sample includes identical tissue sections or consecutive tissue sections.

13. The information processing apparatus according to claim 9, wherein the display unit: when receiving a screen switching signal, changes one of the bright-field image, the fluorescence image, the fluorescence correction image, and the superimposed image being displayed to any other image.

14. The information processing apparatus according to claim 1, wherein The information processing apparatus further includes a storage control unit that stores, for each of the sample storage combination images, a color value of the bright field image and a fluorescence intensity value of a fluorescence image including the fluorescence information of the sample for each pixel.

15. The information processing apparatus according to claim 1, further comprising: a learning unit that learns a correspondence relationship between the fluorescence information of the sample and the bright field image, wherein the generating unit: generates a bright field corrected image based on the learning result of the learning unit.

16. A program that causes a computer to perform the following steps: extract fluorescence correction information from a bright field image of a sample; and generate a fluorescence corrected image based on the fluorescence information of the sample and the fluorescence correction information.

Images