Information processing device and program

Abstract

The present invention analyzes a sample with high precision. An information processing device according to the present invention comprises: an extraction unit (20E) for extracting fluorescence correction information from a bright field image of a sample; and a generation unit (20F) for generating a fluorescence correction image based on the fluorescence information of the sample and the fluorescence correction information.

Description

Technical Field

[0001] The present disclosure relates to an information processing device and a program. Background Art

[0002] A technique for analyzing a sample using a fluorescent image of the sample is disclosed. For example, a technique for analyzing the type, size, etc. of a cell by analyzing a fluorescent signal or a fluorescent spectrum is known (for example, Patent Document 1 and Patent Document 2).

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: JP 2005-91895 A

[0006] Patent Document 2: JP 2015-145829 A Summary of the invention

[0007] Technical Problems to be Solved by the Invention

[0008] However, the fluorescent image has factors that affect the analysis caused by autofluorescence originating from substances contained in the sample, light absorption, etc. Therefore, in the related art, it is sometimes difficult to analyze the sample with high accuracy.

[0009] Therefore, the present disclosure proposes an information processing device and a program capable of analyzing a sample with high accuracy.

[0010] Solution to the problem

[0011] In order to solve the above problems, an information processing apparatus according to one aspect of the present disclosure includes: an extraction unit that extracts fluorescence correction information from a bright field image of a sample; and a generation unit that generates a fluorescence correction image based on fluorescence information of the sample and the fluorescence correction information. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1A is a schematic diagram showing an example of an information processing device according to an embodiment of the present disclosure;

[0013] Figure 1B is a schematic diagram showing an example of a measurement unit according to an embodiment of the present disclosure;

[0014] Figure 2 is a diagram showing an example of a bright field image according to an embodiment of the present disclosure;

[0015] Figure 3A is a diagram showing an example of a fluorescent image according to an embodiment of the present disclosure;

[0016] Figure 3Bis a diagram showing an example of a fluorescent image according to an embodiment of the present disclosure;

[0017] Figure 3C is a diagram showing an example of a fluorescent image according to an embodiment of the present disclosure;

[0018] Figure 3D is a diagram showing an example of a fluorescent image according to an embodiment of the present disclosure;

[0019] Figure 4 is an explanatory diagram of a generation example of a combined image according to an embodiment of the present disclosure;

[0020] Figure 5 is a diagram showing an example of a bright field image according to an embodiment of the present disclosure, in which a correction target area is identified;

[0021] Fig. 6A is a diagram showing an example of a fluorescence correction image according to an embodiment of the present disclosure;

[0022] Figure 6B is a diagram showing an example of a fluorescence correction image according to an embodiment of the present disclosure;

[0023] Figure 6C is a diagram showing an example of a fluorescence correction image according to an embodiment of the present disclosure;

[0024] Fig.6D is a diagram showing an example of a fluorescence correction image according to an embodiment of the present disclosure;

[0025] Figure 7 is a diagram showing the spectrum of a fluorescent dye according to an embodiment of the present disclosure;

[0026] Fig. 8A is an explanatory diagram of an example of a staining state of a bright field image and a fluorescent image according to an embodiment of the present disclosure;

[0027] Figure 8B is an explanatory diagram of an example of a staining state of a bright field image and a fluorescent image according to an embodiment of the present disclosure;

[0028] Figure 8C is an explanatory diagram of an example of a staining state of a bright field image and a fluorescent image according to an embodiment of the present disclosure;

[0029] Fig.8D is an explanatory diagram of an example of a staining state of a bright field image and a fluorescent image according to an embodiment of the present disclosure;

[0030] Fig. 9 is a diagram showing the positional relationship and staining status between tumor cells and lymphocytes according to an embodiment of the present disclosure;

[0031] Fig. 10Ais a schematic diagram of an example of a display screen according to an embodiment of the present disclosure;

[0032] Fig. 10B is a schematic diagram of an example of a display screen according to an embodiment of the present disclosure;

[0033] Fig. 10C is a schematic diagram of an example of a display screen according to an embodiment of the present disclosure;

[0034] Fig. 10D is a schematic diagram of an example of a display screen according to an embodiment of the present disclosure;

[0035] Fig.10E is a schematic diagram of an example of a display screen according to an embodiment of the present disclosure;

[0036] Fig.10F is a schematic diagram of an example of a display screen according to an embodiment of the present disclosure;

[0037] Fig.11 is a schematic diagram showing an example of switching of display screens according to an embodiment of the present disclosure;

[0038] Fig.12 is a flowchart showing an example of a procedure of information processing according to an embodiment of the present disclosure;

[0039] Fig.13 : is a hardware configuration diagram showing an example of a computer that realizes the function of the analysis device of the present disclosure according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0040] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the following embodiments, the same parts are denoted by the same reference numerals, and thus repeated descriptions will be omitted.

[0041] Figure 1A is a schematic diagram showing an example of the information processing device 1 according to the present embodiment.

[0042] The information processing device 1 includes an analyzing device 10 and a measuring unit 12. The analyzing device 10 is connected to the measuring unit 12 so as to be able to exchange data or signals.

[0043] The measuring unit 12 photographs the sample and obtains a bright field image and a fluorescence image of the sample.

[0044] The sample is a sample to be analyzed by the analysis device 10. The sample is, for example, a tissue sample for pathological diagnosis, etc. Specifically, the specimen is a biological tissue including tumor cells, lymphocytes (T cells, B cells, natural killer cells (NK cells)), etc. The sample may also be a sample including a complex of one or more proteins, amino acids, carbohydrates, lipids and their modified molecules. In addition, the specimen may be a sample including antigens (tumor markers, signal converters, hormones, cancer growth regulators, metastasis regulators, growth regulators, inflammatory cytokines, virus-related molecules, etc.) associated with the disease to be pathologically diagnosed. In addition, the sample may be a sample including metabolites, deoxyribonucleic acid (DNA), ribonucleic acid (RNA), microribonucleic acid, polynucleotides, toxins, drugs, virions, cells, hemoglobin, etc. Note that the sample is not limited to the above. For example, the sample may be a sample different from a living tissue. For example, the sample may be a non-biological sample composed of multiple molecules.

[0045] In the present embodiment, a case where the sample is a biological tissue will be described as an example.

[0046] A bright field image is a captured image of a specimen. A captured image is captured image data in which a pixel value is defined for each pixel. That is, a bright field image is captured image data in which a color value is defined for each pixel. The color value is represented by a grayscale value of each of red (R), green (G), and blue (B), for example. Hereinafter, the captured image data will be referred to simply as a captured image.

[0047] In this embodiment, a stained sample is used when capturing a bright field image. The stained sample specifically includes a sample stained with hematoxylin and eosin (HE). The measuring unit 12 obtains a bright field image by capturing the sample, and the bright field image is a captured image of light transmitted through or reflected by the HE-stained sample. The measuring unit 12 outputs the obtained bright field image to the analyzing device 10.

[0048] A fluorescent image is a photographic image of a fluorescently stained sample. Specifically, a bright field image is a photographic image in which a fluorescence intensity value is defined for each pixel.

[0049] Fluorescently stained samples are samples of targets that are labeled with fluorescent dyes or contained in the stained samples. For example, the target is the target being analyzed. The target is used, for example, for tumor, cell, pathological diagnosis, etc.

[0050] Targets are, for example, tumor markers, lymphocyte markers, immune cells, immune checkpoint molecules, molecules used as molecular target drug indicators, receptors, cell surface markers, etc. As these targets, for example, various antigens are used. Antigens are, for example, CD3, CD20, CD68, etc. For the fluorescent labeling of these antigens, known antibodies fluorescently labeled with known fluorescent dyes (e.g., fluorescein isothiocyanate (FITC)) can be used. In addition, there is a method for accumulating fluorescent dyes by using enzymes, but the method is not limited thereto. Examples thereof include nucleic acids, for example, DNA and RNA. Examples of fluorescent dyes for staining these specific tissues include DAPI (4', 6-diamino-2-phenylindole) (DAPI), Alexa Fluor (registered trademark) (AF) 405, Brilliant Violet (BV) 421, BV480, BV510, BV510, etc.

[0051] The measuring unit 12 obtains a fluorescent image by irradiating a sample in which a target is fluorescently stained with light (excitation light) in a wavelength region that excites the fluorescence of the fluorescent dye and photographing the sample. A known mechanism can be used as the light irradiation and photographing mechanism. The measuring unit 12 outputs the obtained fluorescent image to the analyzing device 10.

[0052] Figure 1B is a schematic diagram showing an example of a specific configuration of the measurement unit 12. The measurement unit 12 is a microscope system that captures a sample in a magnified state.

[0053] The measuring unit 12 includes a microscope 70 and a data processing unit 80 .

[0054] The microscope 70 includes a stage 71, an irradiation unit 73, and an imaging element 74. The stage 71 has a placement surface on which a sample SPL as a sample can be placed. The stage 71 can be moved in a parallel direction (xy plane direction) and a vertical direction (z axis direction) under the control of a stage drive unit 75.

[0055] The irradiation unit 73 irradiates the sample with the excitation light. The irradiation unit 73 includes an optical system 72 , a light source driving unit 76 , and a light source 78 .

[0056] The optical system 72 is disposed above the stage 71. The optical system 72 includes an objective lens 72A, an imaging lens 72B, a dichroic mirror 72C, an emission filter 72D, and an excitation filter 72E. The light source 78 is, for example, a bulb such as a mercury lamp, a light emitting diode (LED), or the like.

[0057] The excitation filter 72E is a filter that selectively transmits light in a wavelength region that excites fluorescence of a fluorescent dye among the light emitted from the light source 78. The microscope 70 is equipped with a plurality of excitation filters 72E having different wavelength regions for transmitting light.

[0058] The dichroic mirror 72C guides the light emitted from the light source 78 and transmitted through the excitation filter 72E to the objective lens 72A. The objective lens 72A focuses the light on the sample SPL. Then, the objective lens 72A and the imaging lens 72B form an enlarged image obtained by enlarging the image of the sample SPL to a predetermined magnification on the imaging surface of the imaging element 74.

[0059] The light source driving unit 76 controls the light source 78 and controls switching of the excitation filter 72E.

[0060] The imaging element 74 obtains a captured image of the sample. An enlarged image of the sample is formed on the imaging element 74 via the objective lens 72A and the imaging lens 72B. The imaging element 74 obtains a captured image obtained by imaging the enlarged sample.

[0061] The imaging element 74 is an imager having a photoelectric conversion element and obtaining an image from incident light. The imaging element 74 has an image device, for example, a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) image sensor, etc. Note that the imaging lens 72B and the emission filter 72D can be changed to a spectral element. In this case, a spectral camera of a running scan type or a two-dimensional spectral camera of a time scan type as a spatial scan type is used.

[0062] The imaging element 74 acquires a captured image obtained by capturing a sample under the control of the imaging control unit 77 to output the captured image to the data processing unit 80 .

[0063] The data processing unit 80 includes an irradiation control unit 80A and an acquisition unit 80B. The irradiation control unit 80A controls the irradiation unit 73. Under the control of the irradiation control unit 80A, the light source driving unit 76 controls the position of the excitation filter 72E so that the light emitted from the light source 78 is transmitted through the excitation filter 72E, and then the light source 78 is illuminated.

[0064] The acquisition unit 80B acquires the captured image of the sample from the imaging control unit 77. Specifically, the acquisition unit 80B acquires a bright field image by acquiring a captured image of the stained sample. In addition, the acquisition unit 80B acquires a fluorescent image by acquiring a captured image captured in a state where the sample in which the target is fluorescently stained is irradiated with excitation light. Then, the acquisition unit 80B outputs the fluorescent image and the bright field image to the analysis device 10.

[0065] Back to Figure 1AThe analysis device 10 is an analysis device that analyzes a sample.

[0066] The analysis device 10 includes a control unit 20, a storage unit 22, a user interface (UI) unit 24, and a communication unit 26. The control unit 20, the storage unit 22, the UI unit 24, and the communication unit 26 are connected so as to be able to exchange data or signals.

[0067] The storage unit 22 stores various data. In the present embodiment, the storage unit 22 stores various data, for example, the learning model 23 and the combined image 38. The details of the learning model 23 and the combined image 38 will be described later.

[0068] The UI unit 24 receives various operation inputs from the user to output various types of information. In the present embodiment, the UI unit 24 includes a display unit 24A and an input unit 24B.

[0069] The display unit 24A displays various types of information. The display unit 24A is, for example, an organic electroluminescent device (EL), a liquid crystal display (LCD), etc. The input unit 24B receives various operation inputs from the user. The input unit 24B is, for example, a pointing device, a mouse, a keyboard, an input button, etc. Note that the display unit 24A and the input unit 24B may be configured as a touch panel as a whole.

[0070] The communication unit 26 is a communication interface that communicates with an external device via a network in a wired or wireless manner.

[0071] The control unit 20 includes a fluorescent image acquisition unit 20A, a bright field image acquisition unit 20B, a storage control unit 20C, a learning unit 20D, an extraction unit 20E, a recognition unit 20F, a generation unit 20G, an analysis unit 20H, and a display control unit 20I. Some or all of the fluorescent image acquisition unit 20A, the bright field image acquisition unit 20B, the storage control unit 20C, the learning unit 20D, the extraction unit 20E, the recognition unit 20F, the generation unit 20G, the analysis unit 20H, and the display control unit 20I may be implemented by causing a processing device (e.g., a central processing unit (CPU)) to execute a program (i.e., implemented by software), by hardware (e.g., an integrated circuit (IC)), or by using software and hardware in combination.

[0072] The fluorescent image acquisition unit 20A acquires a fluorescent image including fluorescent information of the sample. The fluorescent information is information about the fluorescently stained sample. For example, the fluorescent image acquisition unit 20A acquires the fluorescent image from the measurement unit 12. Note that the fluorescent image acquisition unit 20A can acquire the fluorescent image by reading the fluorescent image stored in the storage unit 22.

[0073] The bright field image acquisition unit 20B acquires a bright field image. For example, the bright field image acquisition unit 20B acquires a bright field image from the measurement unit 12. Note that the bright field image acquisition unit 20B can acquire a bright field image by reading a bright field image stored in the storage unit 22.

[0074] Note that the fluorescent image acquisition unit 20A and the bright field image acquisition unit 20B acquire a fluorescent image and a bright field image, respectively, as captured images of the same sample. That is, the fluorescent image acquired by the fluorescent image acquisition unit 20A and the bright field image acquired by the bright field image acquisition unit 20B are captured images of the same sample. Note that the fluorescent image acquisition unit 20A and the bright field image acquisition unit 20B can respectively acquire a fluorescent image and a bright field image of the same tissue slice of the same sample. In addition, the fluorescent image acquisition unit 20A and the bright field image acquisition unit 20B can respectively acquire a fluorescent image and a bright field image for different tissue slices of the same sample. In this case, it is preferred that the fluorescent image acquisition unit 20A and the bright field image acquisition unit 20B acquire a fluorescent image and a bright field image, respectively, for each continuous slice of the same sample. That is, the sample is preferably the same slice or continuous tissue slices.

[0075] As described above, in the present embodiment, each of the fluorescent image and the bright field image can be acquired using the same sample or samples similar to each other. At this time, as a sample that is the same as or similar to a certain sample, any slice of the unstained slice and the stained slice can also be used. For example, when using an unstained slice, a slice before staining, a slice adjacent to the stained slice, a slice different from the stained slice in the same block (sampled from the same place as the stained slice), a slice in a different block in the same tissue (sampled from a place different from the stained slice), etc. can also be used.

[0076] In this embodiment, a mode in which the fluorescent image acquisition unit 20A and the bright field image acquisition unit 20B acquire a fluorescent image and a bright field image of the same tissue slice of the same sample, respectively, will be described as an example. Hereinafter, the tissue slice may be simply referred to as a slice.

[0077] Figure 2 is a diagram showing an example of the bright field image 30 . Figure 2 An example of a bright field image 30 of a slice of a sample 40 as a living tissue is shown. In the present embodiment, a case where the sample 40 includes a component that emits autofluorescence will be described. Examples of components that emit autofluorescence are red blood cells, vascular endothelium, necrotic areas, fat areas, collagen, elastin, debris, dirt, contaminants, artifacts, etc. On the other hand, a component that absorbs fluorescence as a light absorbing area is carbon powder, etc.

[0078] FIG. 3A to FIG. 3D is a diagram showing an example of the fluorescent image 32 .

[0079] Figure 3A 3 is an image showing a fluorescent image 32A. The fluorescent image 32A is a fluorescent image 32 of a slice of the sample 40 in which a cluster of differentiation (CD) 20 as an antigen is fluorescently labeled using a fluorescently labeled antibody specific to CD 20.

[0080] Figure 3B 3B is an image showing a fluorescent image 32B. The fluorescent image 32B is a fluorescent image 32 of a slice of the sample 40 in which CD3 as an antigen is fluorescently labeled using a fluorescent-labeled antibody specific to CD3.

[0081] Figure 3C 3 is an image showing a fluorescent image 32C. The fluorescent image 32C is a fluorescent image 32 of a slice of the sample 40 in which CD68 as an antigen is fluorescently labeled using a fluorescent-labeled antibody specific to CD68.

[0082] Figure 3D 3 is an image showing a fluorescent image 32D. The fluorescent image 32D is a fluorescent image 32 of a slice of a sample 40 in which specific tissues (eg, DNA and RNA) are fluorescently stained with 4′,6-diamino-2-phenylindole (DAPI).

[0083] Notice, FIG. 3A to FIG. 3D Shown with Figure 2 The bright field image 30 shown corresponds to a fluorescent image 32 of the same slice of the sample 40 .

[0084] Specifically, both the bright field image 30 and the fluorescent image 32 (fluorescent image 32A to fluorescent image 32D) are captured images obtained by capturing the same slice of the same sample 40. However, when capturing the bright field image 30, the slice may be stained with at least HE. Furthermore, when capturing the fluorescent image 32, the slice is fluorescently stained with a fluorescent dye (or a fluorescently labeled antigen) specific to each of a plurality of targets (antigens, specific tissues) different from each other, and captured in a state of being irradiated with light in a wavelength region excited by the fluorescent dye.

[0085] Back to Figure 1A The storage control unit 20C stores the fluorescent image 32 acquired by the fluorescent image acquisition unit 20A and the bright field image 30 acquired by the bright field image acquisition unit 20B in association with each other in the storage unit 22 for the same sample 40 .

[0086] For example, identification information (e.g., a barcode) of the sample 40 is provided to a slide on which a slice of the sample 40 for capturing each of the bright field image 30 and the fluorescent image 32 is placed. Then, the measurement unit 12 may capture the identification information together with each of the bright field image 30 and the fluorescent image 32 when capturing the sample 40. The storage control unit 20C may store the bright field image 30 and the fluorescent image 32 of each sample 40 in the storage unit 22 by identifying the bright field image 30 and the fluorescent image 32 having the same identification information.

[0087] Therefore, each time the fluorescent image acquisition unit 20A and the bright field image acquisition unit 20B acquire the fluorescent image 32 and the bright field image 30 , the fluorescent image 32 and the bright field image 30 corresponding to the sample 40 are sequentially stored in the storage unit 22 .

[0088] Note that the storage control unit 20C may generate the combined image 38 using the bright field image 30 and the fluorescent image 32 of each sample 40 , and store the combined image in the storage unit 22 .

[0089] Figure 4 3 is an explanatory diagram of an example of generating the combined image 38 .

[0090] The combined image 38 is an image in which the color value of the bright field image 30 and the fluorescence intensity value of the fluorescent image 32 are defined for each pixel. Specifically, the combined image 38 is an image obtained by combining the bright field image 30 and the fluorescent image 32 for each corresponding pixel. The corresponding pixels refer to pixels indicating the same area at the same position in the same slice 41 of the same sample 40. Therefore, the multiple images (bright field image 30, fluorescent image 32) constituting the combined image 38 are preferably captured images of the same area of ​​the same slice 41, except that at least one of the staining conditions and the shooting conditions is different.

[0091] For example, it is assumed that the bright field image 30 is composed of bright field images 30 (bright field image 30R, bright field image 30G, and bright field image 30B) of three channels of red (R), green (G), and blue (B). The bright field image 30R is a bright field image 30 in which a color value of red (R) is defined for each pixel. The bright field image 30G is a bright field image 30 in which a color value of green (G) is defined for each pixel. The bright field image 30B is a bright field image 30 in which a color value of B (blue) is defined for each pixel.

[0092] For example, the slice 41 of the HE-stained sample 40 is adjusted (step S1). When the measurement unit 12 captures the slice 41, the bright field image acquisition unit 20B acquires the bright field image 30 (bright field image 30R, bright field image 30G, and bright field image 30B) of the slice 41 (step S2).

[0093] In addition, the slice 41 of the fluorescent dye sample 40 is adjusted (step S3). The measuring unit 12 irradiates the slice 41 with light having a wavelength that excites fluorescence, and photographs the slice 41. At this time, the measuring unit 12 fluorescently dyes the slice 41 with a fluorescent dye according to each of the plurality of types of targets, and sequentially performs photographing in a state where light having a wavelength that excites each fluorescent dye is sequentially emitted (step S3, step S4). As a result, the fluorescent image acquisition unit 20A acquires a plurality of fluorescent images 32 (e.g., fluorescent images 32A to 32D) of the plurality of types of targets (step S4).

[0094] The storage control unit 20C creates a combined image 38 in which the color value of the bright field image 30 and the fluorescence intensity value of the fluorescent image 32 are defined for each pixel of each sample 40 (step S5, step S6). The storage control unit 20C generates the combined image 38 for each sample 40 by associating the color value of the bright field image 30 for each pixel with the fluorescence intensity value of the fluorescent image 32. Then, the storage control unit 20C stores the generated combined image 38 in the storage unit 22.

[0095] Therefore, for an area corresponding to a pixel of the same slice 41 of the same sample 40, a combined image 38 defining color values ​​of the three channels of R, G and B defined by the bright field image 30 and fluorescence intensity values ​​of the same number of channels as the number of target types defined by one or more fluorescence images 32 is stored in the storage unit 22 of each sample 40.

[0096] The storage control unit 20C may generate the combined image 38 every time the bright field image 30 and the fluorescent image 32 of a new sample 40 are stored in the storage unit 22. Note that the time when the storage control unit 20C generates the combined image 38 is not limited to this time. For example, the storage control unit 20C may generate the combined image 38 using the bright field image 30 and the fluorescent image 32 stored in the storage unit 22 every predetermined time period.

[0097] Back to Figure 1A, will be described further. When the combined image 38 is already stored in the storage unit 22, the fluorescent image acquisition unit 20A can acquire the fluorescent image 32 using the bright field image 30 acquired by the bright field image acquisition unit 20B and the combined image 38. In this case, the fluorescent image acquisition unit 20A can identify the fluorescence intensity value corresponding to each pixel of the bright field image 30 acquired by the bright field image acquisition unit 20B from the combined image 38 to acquire the fluorescent image 32 in which the fluorescence intensity value is defined for each pixel. That is, the fluorescent image acquisition unit 20A can acquire the fluorescent image 32 using the combined image 38 as a learning model.

[0098] Similarly, in the case where the combined image 38 is already stored in the storage unit 22, the bright field image acquisition unit 20B can acquire the bright field image 30 using the fluorescent image 32 acquired by the fluorescent image acquisition unit 20A and the combined image 38. In this case, the bright field image acquisition unit 20B can identify the color value corresponding to each pixel of the fluorescent image 32 acquired by the fluorescent image acquisition unit 20A from the combined image 38 to acquire the bright field image 30 in which the color value is defined for each pixel.

[0099] That is, the bright field image acquisition unit 20B can use the combined image 38 as a learning model to acquire the bright field image 30. Specifically, the fluorescence information of the combined image 38 can be learned as teacher data to acquire the bright field image 30. According to the present embodiment, since the bright field image and the fluorescence information derived from the same slice are accurately associated on a pixel basis, the bright field image can be acquired from the fluorescence information with high accuracy. Preferably, the fluorescence information may include information about fluorescence in situ hybridization (FISH).

[0100] Next, the extraction unit 20E will be described. The extraction unit 20E extracts fluorescence correction information from the bright field image 30 of the sample 40. The fluorescence correction information is information selected from color information of the bright field image 30, morphological information of the bright field image 30, or staining information of the bright field image 30. The morphological information of the bright field image 30 is information indicating the position, size, and range of an area indicating each color included in the bright field image 30. The staining information of the bright field image 30 is information indicating the staining state of the sample 40 in the bright field image 30. For example, the extraction unit 20E can extract the fluorescence correction information from the bright field image 30 by analyzing the bright field image 30 with a known image analysis method.

[0101] Next, the recognition unit 20F will be described. The recognition unit 20F recognizes the correction target area of ​​the bright field image 30 based on the fluorescence correction information extracted by the extraction unit 20E. Figure 2 As shown, the recognition unit 20F recognizes the correction target area P1 included in the bright field image 30 .

[0102] Figure 5 is a diagram showing an example of a bright field image 30 in a state where the correction target region P1 is recognized.

[0103] The correction target area P1 is an area to which auxiliary information is added. For example, the correction target area P1 is an area including information that hinders the analysis of the analysis unit 20H, an area that prompts attention, etc. The correction target area P1 is, for example, an autofluorescence area or a light absorption area. The correction target area P1 can be determined by the analysis unit 20H according to the analysis content.

[0104] For example, it is assumed that the analysis content is the type, distribution, area, size, amount of existence, etc. of the target being analyzed. In this case, the correction target area P1 is an area including a signal of autofluorescence, a spectrum of autofluorescence, carbon powder that absorbs light, etc. Hereinafter, the signal of autofluorescence may be referred to as an autofluorescence signal, and the spectrum of autofluorescence may be referred to as an autofluorescence spectrum.

[0105] The recognition unit 20F recognizes the correction target area P1 corresponding to the analysis content in the entire area of ​​the bright field image 30. For example, the recognition unit 20F recognizes the correction target area P1 of the bright field image 30 based on the fluorescence correction information extracted by the extraction unit 20E. Specifically, the area matching the feature information for identifying the correction target area P1 can be recognized as the correction target area P1.

[0106] The characteristic information is information indicating the characteristics of the correction target area P1. The characteristic information is, for example, information indicating the characteristics of the autofluorescence signal and information indicating the characteristics of the autofluorescence spectrum. The information indicating the characteristics of the autofluorescence signal is, for example, information indicating the wavelength and intensity of the autofluorescence signal. The autofluorescence signal itself can be used as the characteristic information. The information indicating the characteristics of the autofluorescence spectrum is, for example, information indicating the peak position and peak intensity of the autofluorescence spectrum. The autofluorescence spectrum itself can be used as the characteristic information. In addition, color information represented by the autofluorescence signal and the autofluorescence spectrum can be used as the characteristic information. The recognition unit 20F can derive the characteristic information from the fluorescence correction information extracted by the extraction unit 20E.

[0107] Then, the recognition unit 20F recognizes the correction target area P1 based on the color value of each pixel of the bright field image 30. Specifically, the recognition unit 20F may recognize such a pixel area among the pixels of the bright field image 30 as the correction target area P1, in which the spectrum represented by the color value of each pixel matches the characteristic information or falls within a predetermined range.

[0108] As an example, Figure 5 and Figure 2A case where the correction target region P1 is an autofluorescence region of erythrocyte autofluorescence is shown.

[0109] Note that the recognition unit 20F can recognize the correction target area P1 by recognizing the morphological information indicating the position, size, and range of the correction target area P1 included in the bright field image 30. The position of the correction target area P1 indicates the position of the correction target area P1 in the bright field image 30. The size of the correction target area P1 indicates the size of the correction target area P1 in the bright field image 30. The range of the correction target area P1 indicates the range occupied by the correction target area P1 in the bright field image 30.

[0110] Note that it is assumed that the bright field image 30 and the fluorescent image 32 obtained by photographing the same sample 40 have the same photographing angle and photographing magnification.

[0111] Then, the identification unit 20F stores the identified correction target region P1 in the storage unit 22 in association with the bright field image 30 used to identify the correction target region P1. Therefore, the storage unit 22 stores the bright field image 30, the fluorescent image 32 (fluorescent image 32A to fluorescent image 32D), and the morphological information of the correction target region P1 in association with each other for each sample 40. Note that the process of storing the correction target region P1 in the storage unit 22 can be performed by the storage control unit 20C.

[0112] Back to Figure 1A , which will be described further. The recognition unit 20F can recognize the correction target area P1 using the learning model 23. The learning model 23 is generated by the learning unit 20D.

[0113] The learning model 23 is a model that takes the bright field image 30 as an input and takes the correction target area P1 as an output. The learning unit 20D generates the learning model 23 through learning. A known algorithm such as a convolutional neural network (CNN) can be used for the learning.

[0114] For example, the learning unit 20D learns and updates the learning model 23 at a predetermined time. The predetermined time is, for example, the time when the new bright field image 30 and the new fluorescent image 32 are stored in the storage unit 22, the time when the new correction target area P1 is stored in the storage unit 22, a predetermined period of time, a predetermined day, etc. The predetermined time can be determined in advance.

[0115] When the learning model 23 has been recorded in the storage unit 22 , the recognition unit 20F can identify the correction target area P1 by inputting the bright field image 30 acquired by the bright field image acquisition unit 20B as input data into the learning model 23 and acquiring the correction target area P1 output from the learning model 23 .

[0116] Next, the generation unit 20G will be described. The generation unit 20G generates a fluorescence correction image based on the fluorescence information and the fluorescence correction information of the sample 40. In the present embodiment, the generation unit 20G generates a fluorescence correction image obtained by correcting the correction target region P1 included in the fluorescence image 32, based on the fluorescence image 32 and the correction target region P1 identified based on the extracted fluorescence correction information.

[0117] The fluorescence correction image is a fluorescence image generated by correcting the correction target region P1 in the fluorescence image 32 .

[0118] FIG. 6A to FIG. 6D 34A to 34D. The fluorescence correction image 34A is obtained by correcting the fluorescence image 32A (see FIG. Figure 3A ) is a fluorescence image of the first area PA corresponding to the correction target area P1 of the fluorescence image 32B (see FIG. 34B ). Figure 3B ) is a fluorescence image of the first area PA corresponding to the correction target area P1 of the fluorescence image 32C (see FIG. 34A ). Figure 3C ) is a fluorescence image of the first area PA corresponding to the correction target area P1. The fluorescence correction image 34D is obtained by correcting the fluorescence image 32D (see Figure 3D ) is a fluorescent image obtained by focusing on the first area PA corresponding to the correction target area P1.

[0119] The generation unit 20G identifies a region in the fluorescence correction image 34 indicating the position, size, and range of the correction target region P1 identified by the identification unit 20F as the correction target region P1 (ie, the first region PA) on the fluorescence correction image 34 .

[0120] As described above, the bright field image 30 and the fluorescent image 32 obtained by photographing the same sample 40 have the same photographing angle and photographing magnification. In addition, it is assumed that the bright field image 30 and the fluorescent image 32 obtained by photographing the same sample 40 have the same area to be imaged in the sample 40.

[0121] Therefore, the generation unit 20G can identify an area of ​​the position, size and range on the fluorescent image 32 defined by the position, size and range of the correction target area P1 on the bright field image 30 identified by the identification unit 20F as the first area PA corresponding to the correction target area P1.

[0122] Then, the generation unit 20G corrects the fluorescence intensity value of each pixel in the first area PA identified in the fluorescent image 32 .

[0123] That is, the generation unit 20G recognizes the first area PA corresponding to the correction target area P1 of the bright field image 30 in the fluorescent image 32 by using the superimposed image in which the bright field image 30 and the fluorescent image 32 are superimposed, and corrects the first area PA.

[0124] For example, when the amount of the fluorescence component caused by autofluorescence is high, the generation unit 20G corrects the fluorescence intensity value in the first area PA included in the fluorescent image 32 to a lower value.

[0125] Specifically, the generation unit 20G stores in advance the autofluorescence spectrum of each type of autofluorescence. The type of autofluorescence is, for example, the type of component that emits autofluorescence (erythrocytes, collagen, elastin, etc.).

[0126] Then, the generation unit 20G reads the autofluorescence spectrum corresponding to the type of autofluorescence included in the slice 41 of the fluorescence correction image 34. The type of autofluorescence may be input by the user through the operation of the input unit 24B, or may be identified by a known method by the generation unit 20G that performs image analysis, etc. The generation unit 20G may generate the fluorescence correction image 34 by removing the autofluorescence spectrum from the spectrum of the color value of each pixel in the first area PA in the fluorescence image 32. For example, the generation unit 20G may generate the fluorescence correction image 34 by multiplying the spectrum of the color value of the core pixel in the first area PA in the fluorescence image 32 by a weight value according to the autofluorescence spectrum.

[0127] At this time, the stored autofluorescence spectrum may be an autofluorescence spectrum of one channel. That is, there is a case where an autofluorescence spectrum of one channel obtained by synthesizing autofluorescence spectra obtained from each of a plurality of fluorescent images 32, each of which includes a plurality of types of fluorescently stained targets, is stored.

[0128] In this case, the generation unit 20G can separate the autofluorescence spectrum into the number of channels (in this case, four channels) of the fluorescence image 32 (fluorescence image 32A to fluorescence image 32D). The separation can be performed using a known method. Then, the generation unit 20G can generate each of the fluorescence correction images 34A to 34D by correcting the first area PA of each fluorescence image 32A into the fluorescence image 32D corresponding to the fluorescent staining target using the separated autofluorescence spectrum corresponding to each target.

[0129] Note that the generation unit 20G may acquire a weight value from a standard spectrum derived from a sample from the learning model 23 , and use the weight value to generate the fluorescence correction image 34 .

[0130] In this case, the learning unit 20D may generate a model in advance as a learning model 23, which has a bright field image 30 as an input and a weight value according to the correction target area P1 and a standard spectrum derived from a sample as an output. Then, the generation unit 20G acquires the weight value output from the learning model 23 by inputting the bright field image 30 as input data to the learning model 23. Then, the generation unit 20G may use the acquired weight value to correct the first area PA (correction target area P1) of the fluorescence image 32 to generate a fluorescence correction image 34. At this time, the generation unit 20G may generate a fluorescence correction image 34 obtained by removing the first area PA corresponding to the correction target area P1 corrected by the weight value.

[0131] In addition, the generation unit 20G may correct the fluorescence intensity value in the first area PA included in the fluorescence image 32 to be equal to or less than a predetermined threshold value (e.g., "0"). The threshold value may be determined in advance. In this case, the generation unit 20G may generate a fluorescence correction image 34 obtained by removing the area corresponding to the correction target area P1.

[0132] In addition, the generation unit 20G may generate the fluorescence correction image 34 based on the peripheral information of the correction target area P1. For example, the generation unit 20G may correct the fluorescence intensity value of the first area PA included in the fluorescence image 32 using the fluorescence intensity value around the first area PA in the fluorescence image 32. Specifically, the generation unit 20G may generate the fluorescence correction image 34 by interpolating the fluorescence intensity value of the first area PA included in the fluorescence image 32 using the fluorescence intensity values ​​of the pixels around the first area PA in the fluorescence image 32. A known image processing technique may be used for the interpolation.

[0133] Note that, as described above, for example, when the amount of the fluorescence component caused by autofluorescence is high, the generation unit 20G corrects the fluorescence intensity value in the first area PA included in the fluorescence image 32 to a lower value. Therefore, the fluorescence intensity value required for analysis included in the fluorescence correction image 34 may also be reduced due to the correction.

[0134] Therefore, preferably, the fluorescent image acquisition unit 20A acquires the fluorescent image 32 captured under the capturing condition in which the brightness of the correction target area P1 in the fluorescent image 32 is saturated. The capturing condition may be preset in the measurement unit 12. For example, the capturing condition in which the brightness of the correction target area P1 is saturated may be adjusted by adjusting the exposure time, the detector sensitivity, the intensity of the illumination light at the time of capturing, and the like.

[0135] As described above, the generation unit 20G recognizes the first area PA corresponding to the correction target area P1 of the bright field image 30 in the fluorescent image 32 by using the superimposed image in which the bright field image 30 and the fluorescent image 32 are superimposed, and corrects the first area PA.

[0136] That is, the analyzing device 10 of the present embodiment recognizes the first area PA as the correction target area P1 included in the fluorescent image 32 using the bright field image 30, and corrects the first area PA of the fluorescent image 32. Therefore, the analyzing device 10 of the present embodiment can easily correct the correction target area P1 included in the fluorescent image 32 with high accuracy, and set the first area PA as an area not to be analyzed, etc.

[0137] Note that the generation unit 20G can generate the fluorescence correction image 34 using the learning model 23. In this case, a model that takes the fluorescence information of the sample 40 as an input and the fluorescence correction information as an output can be used as the learning model 23. In this case, the learning unit 20D can generate and update the learning model 23 by learning the correspondence between the fluorescence image 32 including the fluorescence information of the sample 40 and the fluorescence correction information extracted by the extraction unit 20E at a predetermined time. Then, the generation unit 20G can generate the fluorescence correction image 34 based on the learning result of the learning unit 20D. Specifically, the generation unit 20G inputs the fluorescence image 32 acquired by the fluorescence image acquisition unit 20A to the learning model 23 as input data, and obtains the fluorescence correction information output from the learning model 23. The generation unit 20G can generate the fluorescence correction image 34 in the same manner as described above based on the acquired fluorescence correction information.

[0138] Here, among fluorescent dyes such as AF405, BV421, BV480, BV510, BV510 and DAPI, the spectrum of DAPI has a wide fluorescence wavelength. For this reason, in the prior art, it is difficult to separate it from signals caused by other fluorescent dyes in some cases.

[0139] Figure 7 36A shows the spectrum of fluorescent dyes. Line 36B shows the spectrum of BV421. Line 36C shows the spectrum of DAPI. Line 36D shows the spectrum of BV480. Line 36E shows the spectrum of BV510.

[0140] like Figure 7 As shown, the DAPI spectrum has a broad fluorescence wavelength and is difficult to separate from signals caused by other fluorescent dyes in the prior art.

[0141] On the other hand, in the present embodiment, the analysis device 10 identifies the first area PA corresponding to the correction target area P1 of the bright field image 30 in the fluorescent image 32 by using the superimposed image in which the bright field image 30 and the fluorescent image 32 are superimposed. Therefore, even when DAPI is used as a staining dye for obtaining the fluorescent image 32, the analysis device 10 of the present embodiment can accurately identify the correction target area P1 (first area PA). In addition, the analysis device 10 of the present embodiment can obtain a bright field image 30 of cell nucleus staining without using an embedding dye, such as DAPI, as a fluorescent dye. Therefore, when displaying a display screen to be described later, a cell nucleus staining image according to the bright field image 30 can be displayed on the fluorescent image 32 using the bright field image 30.

[0142] Note that the above-mentioned combined image 38 may also include the fluorescence correction image 34. In this case, the storage control unit 20C may create the combined image 38 in which the color value of the bright field image 30, the fluorescence intensity value of the fluorescence image 32, and the fluorescence intensity value of the fluorescence correction image 34 are defined for each pixel of each sample 40.

[0143] Back to Figure 1A , will be described further. Next, the analysis unit 20H will be described. The analysis unit 20H analyzes the fluorescence correction image 34 and derives the analysis result. The analysis unit 20H may analyze the entire fluorescence correction image 34, or may analyze the first area PA and the second area PB included in the fluorescence correction image 34 separately. In addition, without being limited to the distinction between the first area PA and the second area PB, the analysis unit 20H may perform analysis on each area obtained by segmenting the fluorescence correction image 34 based on at least one of the color value of the bright field image 30 and the fluorescence intensity value of the fluorescence correction image 34. For example, the analysis unit 20H may use the color value of the bright field image 30 to identify each area of ​​a specific cell included in the bright field image 30, and analyze each area of ​​the characteristic cell in the fluorescence correction image 34.

[0144] The analysis result is information indicating the analysis result of the analysis content. The analysis unit 20H derives the analysis result by analyzing the distribution of fluorescence intensity values ​​(fluorescence signal, fluorescence spectrum) included in the fluorescence correction image 34 (fluorescence correction image 34A to fluorescence correction image 34D).

[0145] The analysis results are, for example, the type, distribution, area, size, amount, score, etc. of the analyzed target. The analyzed targets are the above targets, etc. Specifically, the targets are tumor markers, lymphocyte markers, immune cells, immune checkpoint molecules, and specific tissues. In addition, the analyzed targets can be tumor areas, targets (immune cells, immune checkpoint molecules) present in tumor areas, the number of tumor cells, target-positive tumor cells per total number of tumor cells, etc.

[0146] The amount of the target analyzed as a result of the analysis is represented by quantitative evaluation values ​​such as the number of cells, the number of molecules, the cell density in an area, the molecule density in an area, the number of biomarker-positive cells per a specific number of cells, and the ratio between the number of biomarker-positive cells per a specific number of cells and the number of biomarker-positive cells per a specific number of cells. The distribution and area of ​​the target analyzed as a result of the analysis are represented by, for example, intercellular distance, intercellular interaction, and the like.

[0147] The analysis unit 20H can identify the analyzed target by analyzing the bright field image 30G. In this case, for example, the analysis unit 20H can identify the analyzed target by a known image processing method based on the color value of each pixel of the bright field image 30G, the positional relationship of the pixels of each color value, etc. In addition, the bright field image 30G can identify the analyzed target based on the analyzed target information input by the user through the operation instruction of the input unit 24B. Then, the analysis unit 20H can analyze the analyzed target of the fluorescence correction image 34.

[0148] Note that the analysis unit 20H can analyze the analyzed target by a known method based on the distribution of fluorescence intensity values ​​of at least one of the first area PA and the second area PB which is an area other than the first area PA in the fluorescence corrected image 34 .

[0149] In addition, the analysis results may also include analysis results of the bright field image 30. The analysis results of the bright field image 30 preferably include, for example, information that is difficult to analyze from the fluorescence correction image 34. For example, the analysis results of the bright field image 30 include, but are not limited to, cell nuclei, intracellular organelles, cell membranes, tissue matrix, fat sites, necrotic areas, carbon powder, and the like.

[0150] Here, as described above, the bright field image 30 and the fluorescent image 32 are not limited to captured images of the same slice 41 of the same sample 40 , and may be captured images of consecutive slices of the same sample 40 .

[0151] In this case, the slice 41 for the bright field image 30 and the slice 41 for the fluorescent image 32 are different slices 41 of the same sample 40. Therefore, an analysis result different from the actual state of the sample 40 can be obtained.

[0152] For example, assuming that the analysis unit 20H derives the degree of lymphocyte infiltration into tumor cells as an analysis result, in this case, the actual infiltration state of lymphocytes into tumor cells may be different from the infiltration state derived as the analysis result.

[0153] Therefore, in the analysis device 10 of the present embodiment, when different slices 41 (for example, continuous slices) of the same sample 40 are used as the bright field image 30 and the fluorescent image 32, it is preferable to use the slices 41 in the following staining state. Specifically, the analysis device 10 preferably uses the slices 41 stained with HE and stained with a specific cell marker as the slices 41 for the bright field image 30 and the slices 41 for the fluorescent image 32.

[0154] FIG. 8A to FIG. 8D 3 is an explanatory diagram of an example of a staining state of a bright field image 30 and a fluorescent image 32 . FIG. 8A to FIG. 8D A case is shown in which the sample 40 is a living tissue and includes a tumor cell 42 and a plurality of lymphocytes 44. The lymphocyte 44 is an example of a specific cell.

[0155] Fig. 8A 40 is a diagram showing an example of a cross-sectional view of the sample 40. Fig. 8A As shown, it is assumed that the sample 40 includes a tumor cell 42 and a lymphocyte 44. In addition, it is assumed that among the plurality of lymphocytes 44 (lymphocytes 44A to 44F), lymphocytes 44A, 44D, and 44E exist in the tumor cell 42. In addition, among the plurality of lymphocytes 44, lymphocytes 44B, 44C, and 44F are assumed to exist outside the tumor cell 42.

[0156] Then, assume that the sample 40 is cut to prepare a first slice 41A and a second slice 41B as slices 41 continuous in the thickness direction. The first slice 41A is an example of a slice 41 for capturing the bright field image 30. The second slice 41B is an example of a slice 41 for capturing the fluorescent image 32.

[0157] Then, the first slice 41A and the second slice 41B are stained, and the measurement unit 12 photographs the stained first slice 41A and the second slice 41B, thereby obtaining the bright field image 30 and the fluorescent image 32. Then, the generation unit 20G identifies the first area PA corresponding to the correction target area P1 of the bright field image 30 in the fluorescent image 32 by using the superimposed image on which the bright field image 30 and the fluorescent image 32 are superimposed.

[0158] FIG. 8B to FIG. 8D It is an explanatory diagram of the staining state of the slice 41. FIG. 8B to FIG. 8D Different staining conditions are shown.

[0159] Figure 8B It is an explanatory diagram of a superimposed image 50A of a bright field image 30 of a first slice 41A and a fluorescent image 32B of a second slice 41B. In the bright field image 30 of the first slice 41A, tumor cells 42 are stained with HE, and in the fluorescent image 32B of the second slice 41B, a marker of lymphocytes 44 as specific cells is stained with a fluorescent dye.

[0160] In this case, in the superimposed image 50A of the bright field image 30 and the fluorescent image 32, the lymphocyte 44E exists in the tumor cell 42. However, in the actual specimen 40, the lymphocyte 44E exists outside the tumor cell 42. Therefore, in this case, the generation unit 20G determines that the lymphocyte 44E actually existing in the tumor cell 42 is the lymphocyte 44 existing outside the tumor cell 42. Therefore, the actual infiltration state of the lymphocyte 44 into the tumor cell 42 is different from the infiltration state derived as the analysis result.

[0161] Figure 8C It is an explanatory diagram of a superimposed image 50B of a bright field image 30 of a first slice 41A in which a marker of lymphocytes 44 is stained with a fluorescent dye and a fluorescent image 32B of a second slice 41B. In the first slice 41A, the marker of lymphocytes 44 is stained with a fluorescent dye, and in the second slice 41B, the tumor cells 42 are stained with HE.

[0162] In this case, in the superimposed image 50B of the bright field image 30 and the fluorescent image 32, the lymphocytes 44B are present in the tumor cells 42. However, in the actual specimen 40, the lymphocytes 44B are present outside the tumor cells 42. Therefore, in this case, the generation unit 20G determines that the lymphocytes 44B that are not actually present in the tumor cells 42 are the lymphocytes 44 present in the tumor cells 42. Therefore, the actual infiltration state of the lymphocytes 44 into the tumor cells 42 is different from the infiltration state derived as the analysis result.

[0163] Fig.8D It is an explanatory diagram of a bright field image 30 of a first slice 41A in which tumor cells 42 are stained with HE and a marker of lymphocytes 44 is stained with a fluorescent dye, and a fluorescent image 32B of a second slice 41B in which tumor cells 42 are stained with HE and a marker of lymphocytes 44 is stained with a fluorescent dye.

[0164] In this case, the superimposed image 50 (not shown) of the bright field image 30 and the fluorescent image 32 is an image whose state matches the actual state of the sample 40. Therefore, in this case, the actual infiltration state of the lymphocytes 44 into the tumor cells 42 matches the infiltration state derived as the analysis result.

[0165] Fig. 9 The figure shows the positional relationship between the tumor cell 42 and the lymphocyte 44 and the use FIG. 8A to FIG. 8D Figure 1 describes the dyeing state. Fig. 9 In the above, “actual” means Fig. 8A The positional relationship between tumor cells 42 and lymphocytes 44 in the actual specimen 40 shown. Fig. 9 In the figure, patterns 1 to 3 are respectively Figure 8B 8D corresponds to FIG.

[0166] exist Fig. 9 In the figure, "A" to "F" correspond to lymphocytes 44A to 44F, respectively. Fig. 9 In , "ND" indicates that the overlay image 50 does not include the expression information of the marker. Fig. 9 In the above, “in” indicates that the corresponding lymphocytes 44 are determined to be present in the tumor cells 42. Fig. 9 In the figure, “outside” indicates that the corresponding lymphocyte 44 is determined to be present outside the tumor cell 42.

[0167] like Fig. 9 As shown, when a section 41 in which tumor cells 42 are stained with HE and a marker of lymphocytes 44 is stained with a fluorescent dye is used in both the bright field image 30 and the fluorescent image 32, the positional relationship between the lymphocytes 44 and the tumor cells 42 can be determined in the same state as the actual state of the sample 40.

[0168] Therefore, in the case where different slices 41 of the same sample 40 are used as the slices 41 for the bright field image 30 and the slices 41 for the fluorescent image 32 , the analysis device 10 preferably uses the slices 41 stained under the following staining conditions.

[0169] That is, the bright field image acquisition unit 20B preferably acquires the bright field image 30 of the first slice 41A of the sample 40, in which the cells are stained with HE and specific cells are stained (cells, for example, lymphocytes or their markers). In addition, the fluorescent image acquisition unit 20A preferably acquires the fluorescent image 32 of the second slice 41B of the sample 40, in which the cells are stained with HE and specific cells are stained.

[0170] Back to Figure 1A Next, the display control unit 20I will be described.

[0171] The display control unit 20I generates a display screen on the display unit 24A, which includes at least one of the bright field image 30, the fluorescent image 32, the fluorescence correction image 34, the superimposed image obtained by superimposing the bright field image 30, the fluorescent image 32 or the fluorescence correction image 34, and the analysis result.

[0172] FIG. 10A to FIG. 10F 2 is a schematic diagram showing an example of the display screen 60 .

[0173] Fig. 10A 60A is a diagram showing an example of a display screen 60A. The display screen 60A is an example of the display screen 60. Fig. 10A As shown, the display control unit 20I displays a display screen 60 including an overlay image 62A on the display unit 24A, in which the fluorescent image 32 of the sample 40 corresponding to the bright field image 30 is overlaid on the bright field image 30. Note that the overlay image 62A may be an image in which the bright field image 30 is overlaid on the fluorescent image 32.

[0174] Fig. 10B is a diagram showing an example of a display screen 60B. The display screen 60B is an example of the display screen 60. For example, the display control unit 20I displays the bright field image 30 and the area selection frame Q for receiving the selection of a specific area on the bright field image 30. The user operates the input unit 24B to adjust the position and size of the area selection frame Q on the bright field image 30. For example, when an execution instruction signal or the like is received from the input unit 24B, the display control unit 20I generates an enlarged image 62B in which the inside of the area selection frame Q on the bright field image 30 is enlarged and displayed, and displays the enlarged image on the display unit 24A. Note that the display control unit 20I can display the display screen 60B including the superimposed image obtained by superimposing the enlarged image 62B on the bright field image 30 on the display unit 24A.

[0175] Fig. 10C 60C is a diagram showing an example of a display screen 60C. The display screen 60C is an example of the display screen 60. For example, Fig. 10B As shown, the display control unit 20I displays the area selection frame Q on the bright field image 30. The user operates the input unit 24B to adjust the position and size of the area selection frame Q on the bright field image 30. For example, when receiving an execution instruction signal or the like from the input unit 24B, the display control unit 20I generates a superimposed image 62C in which the inside of the area selection frame Q of the fluorescent image 32 enlarged at the same magnification is superimposed on an enlarged image obtained by enlarging the inside of the area selection frame Q on the bright field image 30, and displays the superimposed image on the display unit 24A.

[0176] Fig. 10D 60D is a diagram showing an example of a display screen 60D. The display screen 60D is an example of the display screen 60. For example, Fig. 10CAs shown, the display control unit 20I displays the bright field image 30 and the superimposed image 62C on the display unit 24A. At this time, the display control unit 20I may further display a display screen 60D including the analysis result 64 on the display unit 24A. Fig. 10D Information indicating the distribution of the analyzed target as an example of the analysis result 64 is shown. In addition, the display control unit 20I can display the fluorescence signal information in the correction target area P1 as the analysis result on the display unit 24A. In this case, the display unit 24A can display the correction target area P1 and the fluorescence signal information in the correction target area P1 on the bright field image 30.

[0177] Note that the display format of the analysis result 64 is not limited to Fig. 10D For example, the display control unit 20I may display the analysis result 64 near the corresponding area as information (annotation information) indicating an annotation of the corresponding area (e.g., the first area PA or the second area PB). That is, the display control unit 20I may cause the display unit 24A to display the annotation information based on the information about the correction target area P1.

[0178] Furthermore, the display control unit 20I may display the analysis result 64 in a three-dimensional (3D) display or a bar graph according to the analysis result 64 .

[0179] Fig.10E 60E is a diagram showing an example of a display screen 60E. The display screen 60E is an example of the display screen 60. For example, Fig. 10B As shown, the display control unit 20I displays the area selection frame Q on the bright field image 30. The user operates the input unit 24B to adjust the position and size of the area selection frame Q on the bright field image 30. The display control unit 20I receives, for example, an execution instruction signal from the input unit 24B. Then, the display control unit 20I generates a superimposed image 62D in which the inside of the area selection frame Q of the fluorescent image 32 enlarged at the same magnification is superimposed on the enlarged image obtained by enlarging the inside of the area selection frame Q on the bright field image 30, and displays the superimposed image on the display unit 24A.

[0180] At this time, the display control unit 20I may display an image schematically showing at least one of the enlarged image in the region selection frame Q of the bright field image 30 and the enlarged image in the region selection frame Q of the fluorescent image 32, such as: Fig.10E At this time, in the superimposed image 62D, the first area PA as the correction target area P1 and the second area PB as the area other than the correction target area P1 may be displayed in different display forms (see Fig.10E). In addition, the display control unit 20I can also display a display screen 60D including the analysis result 64 on the display unit 24A. Although not shown, the display control unit 20I can also display the depth information of the unit. In addition, the display control unit 20I can also display a display screen 60E including the analysis result 64 on the display unit 24A.

[0181] Fig.10F 60F is a diagram showing an example of a display screen 60F. The display screen 60F is an example of the display screen 60. For example, Fig. 10B As shown, the display control unit 20I displays the area selection frame Q on the bright field image 30. The user operates the input unit 24B to adjust the position and size of the area selection frame Q on the bright field image 30. For example, when receiving an execution instruction signal or the like from the input unit 24B, the display control unit 20I generates a superimposed image 62D in which the inside of the area selection frame Q of the fluorescent image 32 enlarged at the same magnification is superimposed on an enlarged image obtained by enlarging the inside of the area selection frame Q on the bright field image 30, and displays the superimposed image on the display unit 24A.

[0182] At this time, the display control unit 20I may display an image schematically showing at least one of the enlarged image in the region selection frame Q of the bright field image 30 and the enlarged image in the region selection frame Q of the fluorescent image 32, such as: Fig.10F The display control unit 20I may remove the first area PA as the correction target area P1 and selectively display the second area PB as an area other than the correction target area P1 (see Fig.10F ). In addition, the display control unit 20I can also display a display screen 60D including the analysis result 64 on the display unit 24A.

[0183] Note that the display control unit 20I may display, on the display unit 24A, a display screen 60 which also includes morphological information of at least one of the first area PA and the second area PB which are the correction target area P1 included in the fluorescence correction image 34. Note that the analysis result 64 may include morphological information.

[0184] The morphological information of the first area PA and the second area PB is information indicating the position, size, and range of each of the first area PA and the second area PB on the fluorescence correction image 34. Specifically, the morphological information of the first area PA and the second area PB is represented by a display form indicating the position, size, and range thereof. The display form is, for example, a frame line indicating the outer shape of the first area PA and the second area PB, a specific color indicating the first area PA and the second area PB, a flashing display or a highlighted display of the first area PA and the second area PB, a display with increased brightness of the first area PA and the second area PB, and the like. The flashing display includes flashing, in which the brightness of at least one of the first area PA and the second area PB changes periodically. The highlighted display includes a display that attracts attention by color or size.

[0185] Note that when a user or the like indicates a specific position in the display screen 60 through an operation instruction of the input unit 24B, the display control unit 20I may further display an image related to the image displayed at the indicated position. For example, it is assumed that the analysis result 64 is displayed at the indicated position. In this case, the display control unit 20I may search the storage unit 22 for at least one of another analysis result 64 that matches or is similar to the analysis result 64, the bright field image 30 used to analyze the analysis result 64, the fluorescence image 32, and the fluorescence correction image 34, and further display the searched result on the display unit 24A.

[0186] Furthermore, the display control unit 20I may display the analysis result 64 for each analysis area.

[0187] Specifically, when analysis is performed on each of the first area PA and the second area PB, the display control unit 20I may display the analysis result of the first area PA and the analysis result of the second area PB on the display unit 24A.

[0188] In addition, it is assumed that the display control unit 20I has performed analysis on each region obtained by segmenting the fluorescence correction image 34 based on at least one of the color value of the bright field image 30 and the fluorescence intensity value of the fluorescence correction image 34. In this case, the display control unit 20I can display the analysis result 64 of each region on the display unit 24A. By displaying the analysis result 64 of each region, an annotation of each region can be displayed. At this time, the display control unit 20I can generate a line that divides the cell region of each cell included in the bright field image 30, and display the cell region so that it is superimposed on the fluorescence image 32.

[0189] Furthermore, the display control unit 20I can display a histogram in which the measurement parameters of each cell are plotted on the channel axis as the analysis result 64. Specifically, a dot graph similar to a graph of a flow cytometer can be displayed.

[0190] Note that when receiving a screen switching signal, the display control unit 20I can switch the display screen 60. For example, the user instructs screen switching by operating the input unit 24B. In response to the screen switching instruction of the operation of the input unit 24B, the display control unit 20I receives the screen switching signal from the input unit 24B.

[0191] When receiving the screen switching signal, the display control unit 20I changes one of the bright field image 30, the fluorescent image 32, the fluorescent correction image 34, the superimposed image, and the analysis result 64 displayed on the display screen 60 to any other image. In addition, the display control unit 20I can switch the displayed display screen 60 (display screen 60A to display screen 60F) to another display screen 60 (display screen 60A to display screen 60F).

[0192] Furthermore, the display control unit 20I may display at least one of the bright field image 30, the fluorescent image 32, and the fluorescence correction image 34 during the period when the user receives the operation of the input unit 24B (e.g., the mouse operation), and display the analysis result 64 when the operation is stopped. Furthermore, the display control unit 20I may display the analysis result 64 during the period when the user receives the operation of the input unit 24B (e.g., the mouse operation), and display at least one of the bright field image 30, the fluorescent image 32, and the fluorescence correction image 34 when the operation is stopped.

[0193] Furthermore, the display control unit 20I can switch the display screen 60 using the combined image 38 .

[0194] Fig.11 3 is a schematic diagram showing an example of switching the display screen 60 using the combined image 38. As described above, the combined image 38 is an image obtained by combining the bright field image 30 and the fluorescent image 32 for each corresponding pixel.

[0195] The display control unit 20I receives, for example, a display instruction signal from the input unit 24B. Then, the display control unit 20I displays a display screen 60G including the bright field images 30 (bright field image 30R, bright field image 30G, and bright field image 30B) included in the combined image 38 on the display unit 24A (step S10). Then, the display control unit 20I may switch the display of the display screen 60G including the bright field image 30 and the display of the display screen 60H including the fluorescent image 32 every time a screen switching signal is received (step S12).

[0196] At this time, the display control unit 20I can easily display the display screen 60H including the fluorescent image 32 on the display unit 24A by extracting the fluorescent image 32 from the combined image 38 including the bright field image 30 being displayed. Similarly, the display control unit 20I can easily display the display screen 60H including the bright field image 30 on the display unit 24A by extracting the bright field image 30 from the combined image 38 including the fluorescent image 32 being displayed. That is, the display control unit 20I can easily switch the display screen 60 by using the combined image 38.

[0197] Next, an example of a procedure of information processing performed by the analysis device 10 will be described.

[0198] Fig.12 is a flowchart showing an example of a procedure of information processing performed by the analysis device 10 .

[0199] First, the fluorescent image acquisition unit 20A acquires the fluorescent image 32 (step S100 ). Next, the bright field image acquisition unit 20B acquires the bright field image 30 of the sample 40 corresponding to the fluorescent image 32 acquired in step S100 (step S102 ).

[0200] The storage control unit 20C stores the fluorescent image 32 acquired in step S100 and the bright field image 30 acquired in step S102 in association with each other in the storage unit 22 for the same sample 40 (step S104 ).

[0201] Therefore, each time the fluorescent image acquisition unit 20A and the bright field image acquisition unit 20B acquire the fluorescent image 32 and the bright field image 30 , the fluorescent image 32 and the bright field image 30 corresponding to the sample 40 are sequentially stored in the storage unit 22 .

[0202] Note that at this time, the storage control unit 20C may generate a combined image 38 using the bright field image 30 and the fluorescent image 32 of each sample 40 , and store the combined image in the storage unit 22 .

[0203] The extraction unit 20E extracts the fluorescence correction information from the bright field image 30 acquired in step S102 (step S106 ).

[0204] The recognition unit 20F recognizes the correction target area P1 included in the bright field image 30 acquired in step S102 based on the fluorescence correction information extracted in step S106 (step S108). The recognition unit 20F stores the correction target area P1 recognized in step S108 in the storage unit 22 in association with the bright field image 30 used to recognize the correction target area P1 (step S110).

[0205] Therefore, the storage unit 22 stores the bright field image 30 , the fluorescent image 32 (fluorescent image 32A to fluorescent image 32D), and the morphological information of the correction target region P1 in association with each other for each sample 40 .

[0206] At this time, the learning unit 20D can learn the learning model 23. Note that the learning time of the learning model 23 by the learning unit 20D is not limited to this time.

[0207] Next, the generation unit 20G generates the fluorescence correction image 34 based on the fluorescence image 32 acquired in step S100 and the fluorescence correction information extracted in step S106 (step S112). The generation unit 20G generates the fluorescence correction image 34 by using the fluorescence image 32 acquired in step S100 and the correction target area P1 identified based on the fluorescence correction information in step S108.

[0208] The analysis unit 20H analyzes the fluorescence correction image 34 generated in step S112 and derives the analysis result 64 (step S114 ).

[0209] The display control unit 20I generates a display screen 60 on the display unit 24A, the display screen including at least one of the analysis result 64 derived in step S114 , the fluorescent image 32 acquired in step S100 , and the bright field image 30 acquired in step S102 (step S116 ).

[0210] The display control unit 20I determines whether a screen switching signal has been received (step S118). When it is determined that a screen switching signal has been received (step S118: Yes), the process proceeds to step S120.

[0211] In step S120, the display control unit 20I switches the display screen 60 displayed on the input unit 24B to another display screen 60 (step S120). Then, the process proceeds to step S122. When the determination made in step S118 is negative (step S118: No), the process similarly proceeds to step S122.

[0212] In step S122, the display control unit 20I determines whether to end the display of the display screen 60 on the display unit 24A (step S122). For example, the display control unit 20I determines whether a signal indicating a display end instruction has been received from the input unit 24B, thereby making a determination in step S122. When the determination made in step S122 is negative (step S122: No), the process returns to step S118. When the determination made in step S122 is positive (step S122: Yes), the process proceeds to step S124.

[0213] The display control unit 20I determines whether to end the analysis (step S124). For example, the display control unit 20I determines whether a signal indicating an analysis end instruction has been received from the input unit 24B, thereby making a determination in step S124. When the determination made in step S124 is negative (step S124: No), the process returns to step S100. When the determination made in step S124 is positive (step S124: Yes), the process ends.

[0214] Note that there is a case where the display control unit 20I receives a display instruction signal during the processing of any of steps S100 to S114. In this case, when the display instruction signal is received, the display control unit 20I may interrupt the processing and execute the processing of steps S116 to S122.

[0215] As described above, the information processing apparatus 1 of this embodiment includes the extraction unit 20E and the generation unit 20G. The extraction unit 20E extracts the fluorescence correction information from the bright field image 30 of the sample 40. The generation unit 20G generates the fluorescence correction image 34 based on the fluorescence information of the sample 40 and the fluorescence correction information.

[0216] Here, in the prior art, it is sometimes difficult to analyze the sample with high accuracy. Specifically, for example, in the prior art, the fluorescent image 32 is used for analysis, and the autofluorescence component and the like included in the fluorescent image 32 may affect the analysis result. That is, in the prior art, there is a case where the fluorescent dye component and the autofluorescence component cannot be completely distinguished. Specifically, such a problem has occurred in the case where the spectrum of the fluorescent dye and the autofluorescence spectrum are close in wavelength, the shapes of these spectra are similar, the intensity of the autofluorescence spectrum is stronger than the spectrum of the fluorescent dye, etc.

[0217] Specifically, in ordinary filter-type photography, rather than multi-channel photography (e.g., spectroscopy), it is difficult to distinguish the spectrum of fluorescent dyes from the spectrum of autofluorescence components. In addition, even when the wavelength resolution is increased as in spectral photography, the autofluorescence signal may remain depending on the photography position.

[0218] For example, in the case where a region with partially weak fluorescence intensity exists at the edge portion of the red blood cell region, the central portion of the red blood cell region, etc., the shape of the autofluorescence spectrum may change due to the influence of noise, etc. at the time of photographing. In addition, it is known that the preservation state of the sample 40, the tissue fixation method, etc. affect the shape and intensity of the autofluorescence spectrum. Therefore, the shape of the autofluorescence spectrum may change. In this case, even when standard autofluorescence spectrum data is acquired in advance and autofluorescence correction is performed, sufficient autofluorescence correction may not be performed because the shape of the photographed autofluorescence spectrum changes.

[0219] As described above, in the prior art, it is sometimes difficult to analyze a sample with high accuracy.

[0220] On the other hand, in the information processing apparatus 1 of the present embodiment, the extraction unit 20E extracts the fluorescence correction information from the sample bright field image 30. The generation unit 20G generates the fluorescence correction image 34 based on the fluorescence information of the sample and the fluorescence correction information.

[0221] That is, the information processing apparatus 1 of the present embodiment generates the fluorescence correction image 34 from the fluorescence information of the sample (eg, the fluorescence image 32) using the fluorescence correction information extracted from the bright field image 30. Therefore, by using the fluorescence correction image 34 for analysis, the sample 40 can be analyzed with high accuracy.

[0222] Therefore, the information processing device 1 of the present embodiment can analyze the sample 40 with high accuracy.

[0223] In addition, the fluorescent image 32 may include local fluorescent dye noise derived from debris, dirt, fluorescent dye aggregates, debris generated due to tissue section peeling, etc. However, the information processing apparatus 1 of the present embodiment uses the correction target area P1 of the bright field image 30 identified based on the fluorescence correction information to correct the correction target area P1 of the fluorescent image 32. Since the information processing apparatus 1 of the present embodiment performs analysis using the fluorescence correction image 34 obtained by easily correcting the correction target area P1 including noise and the like included in the fluorescent image 32, the sample 40 can be analyzed with high accuracy.

[0224] Here, cells or regions that are not fluorescently stained in the sample 40 are regarded as regions where nothing exists in the fluorescent image 32. Therefore, in the conventional analysis technique using the fluorescent image 32, the analysis accuracy may deteriorate. For example, in the case of analyzing the ratio of negative cells expressing a marker contained in the sample 40, when the presence or absence of cells cannot be identified from the fluorescent image 32, it is impossible to analyze the ratio. Therefore, there is also a method of using the fluorescent image 32 of the sample 40 stained with cell nuclei. However, with the sample 40 stained with cell nuclei alone, the segmentation of cells is insufficient, and it is difficult to analyze an accurate ratio.

[0225] Furthermore, in the case where the fluorescent image 32 includes an area with a weak fluorescent signal or an area without a fluorescent signal, it is difficult to identify whether there is no tissue or the fluorescent signal is weak when analyzing the fluorescent image 32. For this reason, in the prior art, it is difficult to obtain morphological information and an overall image of the analyzed target, and it is difficult to analyze the sample 40 with high accuracy.

[0226] On the other hand, the information processing apparatus 1 of the present embodiment uses the bright field image 30 to identify the correction target region P1, and generates a fluorescence correction image 34 obtained by correcting the correction target region P1 included in the fluorescence image 32. Then, the analysis unit 20H analyzes the fluorescence correction image 34. Therefore, when the display control unit 20I displays, the nucleus, intracellular organelles, cell membranes, tissue matrix, fat parts, necrotic areas, etc. can be further displayed as the analysis results 64 based on the bright field image 30, and necessary information can be added to and provided to the fluorescence correction image 34.

[0227] Furthermore, in the present embodiment, the recognition unit 20F recognizes the correction target region P1 which is an autofluorescence region or a light absorption region included in the bright field image 30. Therefore, the generation unit 20G can highly accurately correct the autofluorescence region or the light absorption region included in the fluorescent image 32. Therefore, the information processing apparatus 1 of the present embodiment can analyze the sample 40 with high accuracy.

[0228] Here, it is difficult to distinguish a signal derived from the fluorescent dye and a signal derived from the autofluorescence only by the fluorescent image 32. On the other hand, an autofluorescence region such as red blood cells can be identified by the bright field image 30. Therefore, the recognition unit 20F recognizes the correction target region P1 as the autofluorescence region or the light absorption region included in the bright field image 30, and corrects the fluorescent image 32 using the correction target region P1, so that the sample 40 can be analyzed with high accuracy.

[0229] Furthermore, the recognition unit 20F recognizes the position, size, and range of the correction target area P1 included in the bright field image 30. Therefore, the generation unit 20G can accurately recognize and correct the first area PA corresponding to the correction target area P1 included in the fluorescent image 32. Therefore, the information processing device 1 of the present embodiment can analyze the sample 40 with high accuracy by using the fluorescent correction image 34 for analysis.

[0230] In addition, the recognition unit 20F recognizes the correction target area P1 using the learning model 23 and the acquired bright field image 30 as input and the correction target area P1 as output. By recognizing the correction target area P1 using the learning model 23, the correction target area P1 can be recognized with high accuracy and high speed.

[0231] Furthermore, the generation unit 20G generates the fluorescence corrected image 34 obtained by removing the region (first region PA) corresponding to the correction target region P1. Therefore, the information processing apparatus 1 of the present embodiment can analyze the sample 40 with high accuracy by analyzing the fluorescence corrected image 34.

[0232] Furthermore, the generation unit 20G generates the fluorescence corrected image 34 based on the peripheral information of the correction target region P1. Therefore, the generation unit 20G can generate the fluorescence corrected image 34 obtained by correcting the fluorescence image 32 easily and accurately.

[0233] Furthermore, the fluorescent image acquisition unit 20A acquires the fluorescent image 32 captured under the capturing condition in which the brightness of the correction target region P1 is saturated.

[0234] Here, in the conventional color separation method based on the fluorescence spectrum shape, for example, when correcting the autofluorescence component from red blood cells, it is necessary to perform shooting under shooting conditions in which a part of the red blood cell component is not saturated when observing. However, since the autofluorescence component of the red blood cell component indicates a high value, it is necessary to perform shooting under shooting conditions that suppress the fluorescence signal, for example, shortening the exposure time and using an ND filter. However, in this conventional method, it may be difficult to shoot a fluorescence signal with a weak fluorescence intensity contained in the sample 40.

[0235] On the other hand, in the present embodiment, the fluorescence image 32 is corrected by identifying the correction target area P1 based on the bright field image 30, and the analysis is performed using the corrected fluorescence correction image 34. That is, in the present embodiment, the correction target area P1, which is an autofluorescence area originating from red blood cells and the like, is identified using the bright field image 30. Therefore, in the information processing device 1 of the present embodiment, it is not necessary to adjust the shooting condition of the fluorescence image 32 to a shooting condition in which the brightness of the correction target area P1 is not saturated. In addition, the information processing device 1 of the present embodiment can acquire the fluorescence image 32 shot under the shooting condition in which the brightness of the correction target area P1 is saturated, so that the fluorescence signal with weak fluorescence intensity included in the sample 40 can be shot.

[0236] Furthermore, the bright field image acquisition unit 20B acquires a bright field image 30 of the slice 41 of the HE-stained sample 40 .

[0237] In this article, in the prior art, an embedding dye (e.g., DAPI) is used as a fluorescent dye for nuclear staining. However, DAPI has a wide fluorescence spectrum and causes leakage to other fluorescent dye wavelengths. In addition, since the presence or absence of nuclear staining is usually determined by the wavelength shift during embedding, the degree of embedding is affected by the wavelength shift, so the shape of the standard spectrum tends to change. For this reason, in the prior art, there is a problem of reduced color separation accuracy.

[0238] On the other hand, in the present embodiment, the bright field image 30 of the slice 41 of the HE-stained sample 40 is used as the bright field image 30. Therefore, in the information processing apparatus 1 of the present embodiment, the position of the cell nucleus can be determined based on the bright field image 30 as the HE-stained image. Therefore, in the information processing apparatus 1 of the present embodiment, the display control unit 20I can display the cell nucleus staining image based on the bright field image 30 on the fluorescent image 32 using the bright field image 30 without using the embedding dye. In addition, by using the bright field image 30, the analysis unit 20H can easily recognize the position of the cell nucleus even in the case where the fluorescent image 32 uses the embedding dye.

[0239] Note that the display control unit 20I uses an image display control program to display the above-mentioned fluorescent image or bright field image. However, the present invention is not limited to this, and the image display control program can be downloaded from a server or installed from a storage medium such as a digital versatile disk (DVD) to a general-purpose computer to implement the processing by the display control unit 20I described below. In addition, the processing performed by the display control unit 20I can be implemented by performing the processing by two or more devices, for example, some processing is performed on the server, and other processing is performed on the computer (e.g., display control unit H). In addition, the image display control program can be operated on the cloud to implement the processing by the display control unit 20I described below.

[0240] Furthermore, the bright field image acquisition unit 20B acquires a bright field image 30 of a first slice 41A of the sample 40 stained with HE and specific cells, and the fluorescent image acquisition unit 20A acquires a fluorescent image 32 of a second slice 41B of the sample 40 stained with HE and specific cells.

[0241] Therefore, it is possible to suppress acquisition of analysis results that differ from the positional relationship between the tumor cells 42 and the lymphocytes 44 in the actual sample 40 .

[0242] Specifically, in the prior art, the bright field image 30 of the HE-stained sample 40 is used as the bright field image 30 for pathological diagnosis. Therefore, it is difficult to identify the target from the bright field image 30. Therefore, in the prior art, it is difficult to accurately grasp the target area expressing the tumor cell 42, and it is difficult to identify the type of immune cells, for example, helper T cells or cytotoxic T cells.

[0243] On the other hand, fluorescent staining is used to identify targets. However, conventional fluorescent images are dark field images, and unstained objects are dark fields. Therefore, the morphological information of tissues and cells is lost. Therefore, it is difficult to identify tumor areas in conventional fluorescent images. In addition, a method for using anti-cytokeratin antibodies to assist in identifying tumor areas is disclosed, but since anti-cytokeratin antibodies are epithelial markers, tumor areas and the number of tumor cells cannot be accurately identified.

[0244] In melanoma, SOX10, a biomarker for identifying tumor areas, is used as a target, and studies have been conducted to identify tumor areas by immunostaining. As a lung cancer marker, thyroid transcription factor-1 (TTF-1) is used for immunostaining. However, although TTF-1 has a high positive rate of about 80% in lung adenocarcinoma and can be used to identify tumor areas, it has a low expression level in lung squamous cell carcinoma and generally indicates negative. Therefore, it cannot be applied to the identification of tumor areas. In addition, there are few targets that can identify tumor areas, and tumor areas cannot be identified by fluorescent staining alone.

[0245] On the other hand, in the present embodiment, the bright field image 30 of the first slice 41A of the sample 40 stained with HE and in which specific cells are stained and the fluorescent image 32 of the second slice 41B of the sample 40 stained with HE and in which specific cells are stained are used. Therefore, the superimposed image 50 of the bright field image 30 and the fluorescent image 32 is an image whose state matches the actual state of the sample 40. Therefore, the information processing apparatus 1 of the present embodiment can determine the positional relationship between the lymphocyte 44 and the tumor cell 42 in the same state as the actual state of the sample 40.

[0246] In addition, for example, the information processing device 1 of the present embodiment can determine immune cells (e.g., lymphocytes 44) that have infiltrated into the tumor cells 42, and measure the number and distance of the immune cells, and the information processing device 1 of the present embodiment can accurately analyze the distribution of biomarkers (immune checkpoint molecules, molecules used as indicators of molecular targeted drugs, etc.), which are examples of targets expressing the tumor cells 42, etc. In addition, when determining a tumor region (e.g., the correction target region P1), the information processing device 1 of the present embodiment can accurately determine the tumor region by using the superimposed image 50 of the bright field image 30 and the fluorescent image 32.

[0247] In addition, in the information processing device 1 of the present embodiment, since tumor cells can be determined by the bright field image 30, not only the tumor area but also the number of tumor cells can be calculated. In addition, in the information processing device 1 of the present embodiment, the expression of biomarkers can be detected by the fluorescent image 32. In addition, the information processing device 1 of the present embodiment can calculate the number of biomarker positive or negative cells per number of tumor cells by superimposing the bright field image 30 and the fluorescent image 32. Incidentally, machine learning can be used to determine the number of tumor cells and the area of ​​each tumor cell. For example, the information processing device 1 according to the present embodiment can calculate the number of HER2-positive cells per total number of tumor cells in a tissue section. Specifically, for example, by analyzing the fluorescence correction image 34, the information processing device 1 can obtain an analysis result in which 50 cells out of 100 tumor cells present in the tissue sample are HER2-positive cells, and the ratio of HER2-positive tumor cells is 50%.

[0248] Furthermore, the analysis unit 20H analyzes the analyzed target included based on the distribution of the fluorescence intensity values ​​of at least one of the first area PA as the correction target area P1 included in the fluorescence correction image 34 and the second area PB other than the first area PA. Therefore, the information processing device 1 of the present embodiment can accurately analyze the first area PA and the second area PB.

[0249] Furthermore, the display control unit 20I displays a display screen 60 on the display unit 24A, which also includes morphological information of at least one of the first area PA which is the correction target area P1 included in the fluorescence correction image 34 and the second area PB other than the first area PA. The display control unit 20I also displays the morphological information of at least one of the first area PA and the second area PB, thereby being able to provide a more detailed analysis result 64 to the user.

[0250] In addition, the morphological information of the first area PA and the second area PB is represented by a display form indicating the position, size and range thereof. The display form is, for example, a frame line indicating the outer shape of the first area PA and the second area PB, a specific color indicating the first area PA and the second area PB, a flashing display (flash) or a highlighted display of the first area PA and the second area PB, a display with increased brightness of the first area PA and the second area PB, etc.

[0251] In the information processing device 1 of the present embodiment, the fluorescent image 32 is corrected using the bright field image 30. The display control unit 20I displays a display screen 60 on the display unit 24A, and the display screen also includes morphological information of at least one of the first area PA as the correction target area P1 included in the fluorescent correction image 34 and the second area PB other than the first area PA. Therefore, in the information processing device 1 of the present embodiment, it is easy to further display information about cells that are not fluorescently stained as morphological information of at least one of the first area PA and the second area PB. In addition, the information processing device 1 according to the present embodiment can easily provide information that can be used for cell segmentation and analysis of the same cells by HE staining and fluorescent staining.

[0252] Furthermore, when receiving a screen switching signal, the display control unit 20I changes one of the bright field image 30, the fluorescent image 32, the fluorescent correction image 34, and the analysis result 64 displayed on the display screen 60 to any other image. Therefore, the display control unit 20I of the present embodiment can change the image according to the user's intention.

[0253] Furthermore, the storage control unit 20C stores the combined image 38 in which the color value of the bright field image 30 and the fluorescence intensity value of the fluorescence image 32 are defined for each pixel in the storage unit 22 for each sample 40. By using the combined image 38 for display, display switching between the bright field image 30 and the fluorescence image 32 can be easily performed. Note that the combined image 38 may be a combined image 38 in which the color value of the bright field image 30, the fluorescence intensity value of the fluorescence image 32, and the fluorescence intensity value of the fluorescence correction image 34 are defined for each pixel.

[0254] In addition, the fluorescence image acquisition unit 20A can acquire a fluorescence image 32 in which a fluorescence intensity value corresponding to each pixel of the bright field image 30 acquired by the bright field image acquisition unit 20B in the combined image 38 is defined for each pixel. In addition, the bright field image acquisition unit 20B can acquire a bright field image 30 in which a color value corresponding to each pixel of the fluorescence image 32 acquired by the fluorescence image acquisition unit 20A in the combined image 38 is defined for each pixel.

[0255] Note that the application target of the information processing apparatus 1 of the present embodiment is not limited. For example, the information processing apparatus 1 can be applied to a fluorescent in situ hybridization method (FISH method) or the like.

[0256] Note that although the embodiments of the present disclosure are described above, the processes according to the above embodiments can be performed in various different embodiments other than the above embodiments. In addition, the above embodiments can be appropriately combined within the range where the processing contents do not contradict each other.

[0257] Furthermore, the effects described in this specification are merely examples and are not limited thereto, and other effects may also exist.

[0258] (Hardware Configuration)

[0259] Fig.13 : is a hardware configuration diagram showing an example of a computer 1000 that realizes the functions of the information processing apparatus 1 according to the above-described embodiment.

[0260] The computer 1000 includes a CPU 1100, a random access memory (RAM) 1200, a read only memory (ROM) 1300, a hard disk drive (HDD) 1400, a communication interface 1500, and an input / output interface 1600. The respective units of the computer 1000 are connected via a bus 1050.

[0261] The CPU 1100 performs operations based on the programs stored in the ROM 1300 or the HDD 1400 and controls each unit. For example, the CPU 1100 develops the programs in the ROM 1300 or the HDD 1400 stored in the RAM 1200 and performs processing corresponding to various programs.

[0262] The ROM 1300 stores a boot program, for example, a basic input output system (BIOS) executed by the CPU 1100 when the computer 1000 is activated, a program depending on the hardware of the computer 1000 , and the like.

[0263] The HDD 1400 is a computer-readable recording medium that non-transitorily records a program executed by the CPU 1100 , data used by the program, and the like. Specifically, the HDD 1400 is a recording medium that records a program according to the present disclosure, which is an example of the program data 1450 , and the like.

[0264] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (eg, the Internet). For example, the CPU 1100 receives data from another device or transmits data generated by the CPU 1100 to another device via the communication interface 1500.

[0265] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from input devices such as a keyboard and a mouse via the input / output interface 1600. In addition, the CPU 1100 transmits data to output devices such as a display, a speaker, or a printer via the input / output interface 1600. In addition, the input / output interface 1600 can be used as a medium interface for reading a program recorded in a predetermined recording medium (medium). The medium is, for example, an optical recording medium such as a digital versatile disk (DVD) or a phase-change rewritable disk (PD), a magneto-optical recording medium such as a magneto-optical disk (MO), a magnetic tape medium, a magnetic recording medium, a semiconductor memory, etc.

[0266] For example, in the case where the computer 1000 is used as the information processing apparatus 1 according to the above-described embodiment, the CPU 1100 of the computer 1000 executes the information processing program loaded on the RAM 1200 to realize the functions of the fluorescent image acquisition unit 20A, etc. In addition, the HDD 1400 stores the program and data according to the present disclosure in the storage unit 22. The CPU 1100 reads the program data 1450 from the HDD 1400 and executes the program data, but as another example, the program may be acquired from another device via the external network 1550.

[0267] Note that the present technology can also be configured as follows.

[0268] (1) An information processing device comprising:

[0269] an extraction unit, which extracts fluorescence correction information from a bright field image of the sample; and

[0270] A generating unit generates a fluorescence correction image based on the fluorescence information of the sample and the fluorescence correction information.

[0271] (2) The information processing device according to (1), further comprising:

[0272] a fluorescence image acquisition unit that acquires a fluorescence image including fluorescence information of the sample; and

[0273] The bright field image acquisition unit acquires a bright field image.

[0274] (3) The information processing device according to (1), wherein the fluorescence correction information is selected from color information, morphology information, or staining information of the bright field image.

[0275] (4) The information processing device according to (1), further comprising:

[0276] A recognition unit is used to determine a correction target area of ​​the bright field image based on the fluorescence correction information, wherein:

[0277] Generate Unit

[0278] A fluorescence correction image obtained by removing a region corresponding to the correction target region is generated.

[0279] (5) The information processing device according to (1), further comprising:

[0280] A recognition unit is used to determine a correction target area of ​​the bright field image based on the fluorescence correction information, wherein:

[0281] Generate Unit

[0282] A fluorescence correction image is generated based on the peripheral information of the correction target area.

[0283] (6) The information processing device according to (4), wherein:

[0284] Generate Unit

[0285] Based on the standard spectrum derived from the sample, a fluorescence correction image obtained by removing a region corresponding to the correction target region corrected based on the weight value is generated.

[0286] (7) The information processing device according to (4), wherein:

[0287] Correction target area

[0288] It is the autofluorescent area or light absorbing area included in the bright field image.

[0289] (8) The information processing device according to (1), further comprising:

[0290] A learning unit, wherein the learning unit learns the correspondence between the fluorescence information and the fluorescence correction information, wherein

[0291] Generate Unit

[0292] A fluorescence correction image is generated based on the learning result of the learning unit.

[0293] (9) The information processing device according to (2), further comprising:

[0294] A display unit displays at least one or more of a bright field image, a fluorescent image including fluorescent information of a sample, a fluorescence correction image, and a superimposed image obtained by superimposing the bright field image and the fluorescent image or the fluorescence correction image.

[0295] (10) The information processing device according to (9), wherein:

[0296] Prompt display unit

[0297] Annotation information is displayed based on information about the correction target area of ​​the bright field image.

[0298] (11) The information processing device according to (9), wherein:

[0299] Prompt display unit

[0300] The correction target region of the bright field image and the fluorescence signal information in the correction target region are displayed in the bright field image.

[0301] (12) The information processing device according to (1), wherein the sample includes the same tissue slice or consecutive tissue slices.

[0302] (13) The information processing device according to (9), wherein:

[0303] Display unit

[0304] One of the bright field image, the fluorescence image, the fluorescence correction image, and the superimposed image being displayed when the screen switching signal is received is changed to any other image.

[0305] (14) The information processing device according to (1), wherein:

[0306] The sample includes a plurality of samples, and wherein,

[0307] The information processing device also includes a storage control unit that stores a combined image for each sample, the combined image defining a fluorescence intensity value of a fluorescence image for each pixel, the fluorescence image including a color value of the bright field image and fluorescence information of each sample.

[0308] (15) The information processing device according to (1), further comprising:

[0309] A learning unit, wherein the learning unit learns the correspondence between the fluorescence information of the sample and the bright field image, wherein,

[0310] Generate Unit

[0311] A bright field corrected image is generated based on the learning result of the learning unit.

[0312] (16) A program that causes a computer to execute the following steps:

[0313] extracting fluorescence correction information from a bright field image of the sample; and

[0314] A fluorescence correction image is generated based on the fluorescence information of the sample and the fluorescence correction information.

[0315] Reference numerals list

[0316] 1 Information processing device

[0317] 10 Analysis device

[0318] 20A Fluorescence Image Acquisition Unit

[0319] 20B Bright Field Image Acquisition Unit

[0320] 20C Storage Control Unit

[0321] 20D Learning Unit

[0322] 20E Extraction Unit

[0323] 20F Identification Unit

[0324] 20G Generation Unit

[0325] 20H Analysis Unit

[0326] 20I Display Control Unit

[0327] 30 Bright field images

[0328] 32 Fluorescence images

[0329] 34 Fluorescence correction image

[0330] 40 samples

[0331] 41 Slice

[0332] 41A First Slice

[0333] 41B Second slice.

Claims

1. An information processing device, comprising: The central processing unit CPU is configured as: Extracting fluorescence correction information from bright-field images of the specimen; as well as identifying a correction target area of ​​the bright field image based on the fluorescence correction information; correcting the fluorescence intensity value of each of a plurality of pixels in an area corresponding to the correction target area in the fluorescence image based on the fluorescence information of the sample and the fluorescence correction information; and A fluorescence corrected image is generated based on the correction of the fluorescence intensity value of each of the plurality of pixels in the region.

2. The information processing device according to claim 1, wherein: The CPU is also configured to: acquiring a fluorescent image including the fluorescent information of the sample; and The bright field image is acquired.

3. The information processing device according to claim 1, wherein: The fluorescence correction information is one of color information, morphology information, and staining information of the bright field image. 4 . The information processing apparatus according to claim 1 , the CPU being further configured to generate the fluorescence correction image based on removal of the region corresponding to the correction target region. 5 . The information processing apparatus according to claim 1 , wherein the CPU is further configured to generate the fluorescence correction image based on peripheral information of the correction target area.

6. The information processing device according to claim 4, wherein: The CPU is further configured to remove an area corresponding to the correction target area based on a weight value, and The weight values ​​are based on a standard spectrum derived from the sample.

7. The information processing device according to claim 4, wherein: The correction target region is an autofluorescence region of the bright field image and a light absorption region of the bright field image.

8. The information processing device according to claim 1, wherein: The CPU is also configured to: learning the corresponding relationship between the fluorescence information and the fluorescence correction information to generate a learning result; The fluorescence correction image is generated based on the learning result.

9. The information processing device according to claim 2, further comprising: A display screen displays the bright field image, the fluorescent image including the fluorescence information of the sample, the fluorescence correction image, and at least one of a superimposed image, wherein the superimposed image corresponds to an image obtained based on superposition of the bright field image and one of the fluorescent image or the fluorescence correction image.

10. The information processing device according to claim 9, wherein: The display is also configured to display annotation information based on information associated with a correction target region of the bright field image.

11. The information processing device according to claim 9, wherein: The display screen is further configured to display a correction target region of the bright field image and fluorescence signal information in the correction target region in the bright field image.

12. The information processing device according to claim 1, wherein: The sample includes one of the same tissue section and consecutive tissue sections.

13. The information processing device according to claim 9, wherein: The display screen is further configured to switch display of the bright field image, the fluorescence image, the fluorescence correction image, or the overlay image based on a screen switching signal.

14. The information processing device according to claim 1, wherein: The information processing device further includes a storage control unit that stores the combined image, and The combined image includes the fluorescence intensity value of each pixel of the plurality of pixels, a color value of the bright field image, and the fluorescence information of the sample.

15. The information processing device according to claim 1, further comprising: learning the correspondence between the fluorescence information of the sample and the bright field image, and Generate a bright field corrected image based on the learning results.

16. A computer-readable storage medium having computer-executable instructions stored thereon, wherein when the computer-executable instructions are executed by a computer, the computer performs an operation, the operation comprising: Extracting fluorescence correction information from bright-field images of the specimen; identifying a correction target area of ​​the bright field image based on the fluorescence correction information; correcting the fluorescence intensity value of each of a plurality of pixels in an area corresponding to the correction target area in the fluorescence image based on the fluorescence information of the sample and the fluorescence correction information; and A fluorescence corrected image is generated based on the correction of the fluorescence intensity value of each of the plurality of pixels in the region.

Images