Fluorescence image information processing system, fluorescence image information processing method, and program

Abstract

In imaging flow cytometry, the objective is to improve the color separation accuracy of fluorescence images and to compensate for undetected wavelength information, even when the number of image detection channels is less than the number of staining dyes. [Solution] This technology provides a fluorescence image information processing system comprising: an image information acquisition unit that acquires fluorescence image information of minute particles; a multi-wavelength fluorescence information acquisition unit that acquires multi-wavelength fluorescence information of the minute particles; a multi-wavelength fluorescence information processing unit that performs multi-wavelength fluorescence information processing, including spectral unmixing, based on the multi-wavelength fluorescence information and reference information to generate multi-color fluorescence separation information; and a calculation unit that corrects or complements the fluorescence image information based on the fluorescence image information and the multi-color fluorescence separation information when the number of channels used by the image information acquisition unit is less than the number of channels required to discriminate the fluorescence of the minute particles.

Description

[Technical Field]

[0001] The technology relating to this disclosure (hereinafter also referred to as "this technology") relates to flow cytometry technology, and more particularly to fluorescence image information processing systems, fluorescence image information processing methods, and programs in imaging flow cytometers. [Background technology]

[0002] A flow cytometer is a device that analyzes the characteristics of individual microparticles, such as cells, by flowing them at high speed through a channel and measuring the scattered light and fluorescence generated by irradiating them with laser light. In recent years, imaging flow cytometers that acquire morphological information of cells as images have been developed, contributing to the advancement of cell analysis.

[0003] In an imaging flow cytometer, fluorescence obtained from cells stained with multiple fluorescent dyes is separated using multiple detectors (for example, a combination of filters and photodetectors that transmit different wavelength bands) to generate a fluorescence image corresponding to each dye. This makes it possible to observe in detail the expression distribution of specific molecules within cells. Related technologies are disclosed, for example, in Patent Documents 1 to 3. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Special Publication No. 2021-524579 [Patent Document 2] Japanese Patent Publication No. 2005-283264 [Patent Document 3] Japanese Patent Publication No. 2007-10340 [Overview of the project] [Problems that the invention aims to solve]

[0005] In imaging flow cytometers, the number of channels available for fluorescence detection (e.g., the number of filters) tends to be fewer than the number of dyes detected in a typical flow cytometer, due to factors such as optical design constraints of the image acquisition system, cost, and data transfer capacity. In particular, when performing multiple staining using a large number of dyes (e.g., multi-color staining of 40 or more colors used in high-end sorters), the number of channels in the image acquisition system may fall below the number of staining dyes (or the number of channels required to discriminate fluorescence). In this case, crosstalk (fluorescence leakage) between dyes becomes large, making accurate color separation difficult. Furthermore, fluorescence information in wavelength bands that are not detected cannot be acquired and is excluded from the analysis.

[0006] This technology was developed in light of the problems described above, and its main purpose is to improve the color separation accuracy of fluorescence images and to compensate for undetected wavelength information, even when the number of image detection channels in an imaging flow cytometer is less than the number of staining dyes. [Means for solving the problem]

[0007] This technology is An image information acquisition unit that acquires fluorescence image information of minute particles, A multi-wavelength fluorescence information acquisition unit that acquires multi-wavelength fluorescence information of the aforementioned fine particles, A multi-wavelength fluorescence information processing unit that performs multi-wavelength fluorescence information processing, including spectral unmixing, based on the multi-wavelength fluorescence information and reference information, and generates multi-color fluorescence separation information. The present invention provides a fluorescence image information processing system comprising: an image information processing unit that corrects or supplements the fluorescence image information based on the fluorescence image information and the multicolor fluorescence separation information when the number of channels used in the image information acquisition unit is less than the number of channels required to discriminate the fluorescence of the minute particles. The image information processing unit may generate a virtual detection filter in the image information acquisition unit based on the dye information and device channel information included in the reference information. The image information processing unit may generate a leak matrix based on the reference information and the virtual detection filter. The multi-wavelength fluorescence information processing unit may calculate coefficients for converting the multi-wavelength fluorescence information acquired by the multi-wavelength fluorescence information acquisition unit into measurements taken by the virtual detection filter. The image information processing unit may use the leakage matrix and the coefficients to perform a color separation operation on the fluorescence image information and generate an image corresponding to each dye. The multi-wavelength fluorescence information processing unit may calculate the coefficients by dividing the multi-wavelength fluorescence information acquired by the multi-wavelength fluorescence information acquisition unit into sections. The image information processing unit may generate the virtual detection filter based on filter settings by the user or automatic settings based on dye information. The image information acquisition unit acquires fluorescence image information in at least one wavelength band, The image information processing unit may calculate the ratio of each pigment in the fluorescence image information based on the color fluorescence separation information, the pigment information of each pigment included in the reference information, and the transmission spectrum of the filter in the image information acquisition unit included in the reference information, and correct the fluorescence image information based on the ratio and the pixel values ​​of the fluorescence image information. The image information processing unit may calculate the corrected pixel value as z·j, where z is the pixel value of the fluorescence image information acquired by the image information acquisition unit and j is the ratio. The multi-wavelength fluorescence information processing unit may treat autofluorescence as one of the dyes and include the autofluorescence spectrum in the reference information to remove the influence of autofluorescence. The multi-wavelength fluorescence information processing unit may extract autofluorescence spectra using multivariate analysis or autofluorescence spectrum information previously entered by the user. The multi-wavelength fluorescence information processing unit may acquire autofluorescence spectral information specific to the cell type in advance, and if the cell type can be identified, it may use that autofluorescence spectral information to remove the background. The image information processing unit may perform autofluorescence correction using the luminance distribution pattern of autofluorescence estimated based on the outer shape of a cell, the autofluorescence ratio calculated from the multicolor fluorescence separation information, and the luminance value of the fluorescence image information acquired by the image information acquisition unit. The image information processing unit may substitute the intensity coefficient of autofluorescence calculated using the multi-wavelength fluorescence information and the reference information during color separation of the fluorescence image information acquired by the image information acquisition unit, and perform color separation for dyes other than autofluorescence. The multi-wavelength fluorescence information acquisition unit may acquire the multi-wavelength fluorescence information of the microparticles by an optical system using a diffraction grating and a multi-channel detector. The multi-wavelength fluorescence information acquisition unit may acquire the multi-wavelength fluorescence information of the microparticles by an optical system that separates wavelengths and a plurality of detectors provided for each wavelength. The image information acquisition unit may include a plurality of detectors that detect fluorescence in a plurality of different wavelength regions, and each detector may include a line sensor that detects fluorescence from the microparticles passing through the flow cell in a line shape. The multi-wavelength fluorescence information acquisition unit may acquire multi-wavelength fluorescence information from the microparticles moving in the flow cell at a speed of 0.1 to 10 m / sec. In addition, the present technology acquiring fluorescence image information of microparticles; acquiring multi-wavelength fluorescence information of the microparticles; performing multi-wavelength fluorescence information processing including spectral unmixing based on the multi-wavelength fluorescence information and reference information to generate multicolor fluorescence separation information; when the number of channels used in acquiring the fluorescence image information is smaller than the number of channels required to discriminate the fluorescence of the microparticles, correcting or complementing the fluorescence image information based on the fluorescence image information and the multicolor fluorescence separation information. A fluorescence image information processing method is provided. In addition, the present technology causes a computer to acquire fluorescence image information of microparticles; acquire multi-wavelength fluorescence information of the microparticles; performing multi-wavelength fluorescence information processing including spectral unmixing based on the multi-wavelength fluorescence information and reference information, to generate multicolor fluorescence separation information; when the number of channels used in the acquisition of the fluorescence image information is smaller than the number of channels required to discriminate the fluorescence of the microparticles, correcting or complementing the fluorescence image information based on the fluorescence image information and the multicolor fluorescence separation information; to provide a program for executing [Brief Description of Drawings]

[0008] [Figure 1] FIG. 2 is a block diagram illustrating a configuration example of a fluorescence image information processing system 100 according to an embodiment of the present technology. [Figure 2] FIG. 3 is a conceptual diagram illustrating a configuration example of the fluorescence image information processing system 100 according to an embodiment of the present technology. [Figure 3] FIG. 4 is a block diagram illustrating another configuration example of the fluorescence image information processing system 100 according to an embodiment of the present technology. [Figure 4] FIG. 5 is a block diagram illustrating a configuration example of an arithmetic unit 130 according to an embodiment of the present technology. [Figure 5] FIG. 6 is a flowchart illustrating an example of a processing flow of the fluorescence image information processing system 100 according to an embodiment of the present technology. [Figure 6] FIG. 7 is a graph illustrating an example of multi-wavelength fluorescence information. [Figure 7] FIG. 8 is a graph illustrating an example of multi-wavelength fluorescence information. [Figure 8] FIG. 9 is a schematic diagram for describing image compensation and virtual image generation in the present technology. [Figure 9] FIG. 10 is a schematic diagram for describing a color separation method using a spectrum corresponding to a position inside a microparticle in the present technology. [Figure 10] FIG. 11 is a schematic diagram for describing a color separation method using a spectrum corresponding to a position inside a microparticle in the present technology. [Figure 11] FIG. 12 is a graph illustrating an example of multi-wavelength fluorescence information. [Figure 12]This flowchart shows an example of the processing flow of a fluorescence image information processing system 100 according to one embodiment of this technology. [Figure 13] This is a schematic diagram illustrating the autofluorescence correction method in this technology. [Figure 14] This is a schematic diagram illustrating the autofluorescence correction method in this technology. [Figure 15] This is a schematic diagram illustrating the autofluorescence correction method in this technology. [Figure 16] This graph shows an example of multi-wavelength fluorescence information. [Figure 17] This graph shows the intensity coefficients of each component. [Figure 18] This graph shows an example of multi-wavelength fluorescence information. [Figure 19] This graph shows the intensity coefficients of each component. [Figure 20] This flowchart shows an example of the processing flow for the color separation method of this technology. [Figure 21] This flowchart shows an example of the processing flow for the color separation method of this technology. [Figure 22] This flowchart shows an example of the processing flow of a fluorescence image information processing system 100 according to one embodiment of this technology. [Figure 23] This is a flowchart showing an example of the flow of a fluorescence image information processing method according to one embodiment of this technology. [Figure 24] This graph shows an example of multi-wavelength fluorescence information. [Modes for carrying out the invention]

[0009] Hereinafter, preferred embodiments for carrying out the present technology will be described in detail with reference to the drawings. The embodiments described below are merely examples of typical embodiments of the present technology and do not limit the scope of the present technology. Furthermore, the present technology can be implemented by combining any of the following embodiments and their modifications.

[0010] In the following description of embodiments, configurations may be described using terms with "approximately" attached, such as "approximately parallel" and "approximately orthogonal." For example, "approximately parallel" means not only that they are perfectly parallel, but also that they are substantially parallel, that is, that is, that they are deviated from a perfectly parallel state by, for example, a few percent. The same applies to other terms with "approximately" attached.

[0011] Furthermore, each diagram is a schematic representation and not necessarily a strictly accurate depiction. The scale of the drawings is exaggerated to make the technical features easier to understand. Therefore, it should be noted that the scale of the drawings and the scale of the actual device are not necessarily the same.

[0012] Unless otherwise specified, in drawings, "up" means the upper direction or upper side in the drawing, "down" means the lower direction or lower side in the drawing, "left" means the left direction or left side in the drawing, and "right" means the right direction or right side in the drawing. In addition, in drawings, the same or equivalent elements or components are denoted by the same reference numeral, and redundant explanations are omitted.

[0013] Furthermore, the effects described herein are merely illustrative and not limited to those described herein, and other effects may also occur.

[0014] In this specification, the explanation will be given in the following order. 1. Background technology 2. Overview of this technology (1) System Configuration (2) Operation (3) Definitions of terms (4) Subject of analysis 3. First Embodiment of the Technology (Example 1 of a Fluorescence Image Information Processing System) (1) System Configuration (2) Image information acquisition unit 110 (3) Multi-wavelength fluorescence information acquisition unit 120 (4) Variations (5) Arithmetic unit 130 (6) Effects 4. A second embodiment of this technology (Example 2 of a fluorescence image information processing system) 5. A third embodiment of this technology (Example 3 of a fluorescence image information processing system) 6. A fourth embodiment of this technology (Example 4 of a fluorescence image information processing system) (1) System Configuration (2) Operation (processing flow) (3) Effects 7. Fifth embodiment of the present technology (Example 5 of a fluorescence image information processing system) 8. A sixth embodiment of this technology (Example 6 of a fluorescence image information processing system) 9. Seventh Embodiment of the Technology (Example 7 of a Fluorescence Image Information Processing System) 10. Eighth Embodiment of the Technology (Example 8 of a Fluorescence Image Information Processing System) (1) System Configuration (2) Operation (processing flow) (3) Effects 11. A ninth embodiment of this technology (an example of a fluorescence image information processing method) 12. Tenth Embodiment of the Technology (Program and Recording Medium)

[0015] [1. Background technology] Figure 24 is a graph illustrating the problems with fluorescence spectra and color separation during multiple staining in conventional flow cytometers.

[0016] Figure 24A shows the fluorescence spectra of three dyes A, B, and C excited by 405 nm excitation light, and Figure 24B shows the fluorescence spectra of three dyes D, E, and F excited by 488 nm excitation light. The fluorescence spectra of each dye have different wavelength peaks, but there are overlapping regions.

[0017] For example, near the peak wavelength of dye B (the area enclosed by the rectangular frame in Figure 24A), a signal is detected where the fluorescence of dyes A, B, and C are interfering with each other. Similarly, near the peak wavelengths of other dyes, fluorescence from multiple dyes also interferes with each other.

[0018] To solve this fluorescence crosstalk problem, conventional flow cytometers use methods called compensation or spectral unmixing to separate the fluorescence signals corresponding to each dye and calculate the fluorescence intensity of a single dye.

[0019] [Compensation] In a typical flow cytometer, fluorescence obtained from a sample stained with multiple fluorescent dyes is detected by multiple detectors (a combination of filters and photodetectors). However, because the fluorescence spectra of each dye overlap, each detector also detects fluorescence from dyes other than the target dye. To correct this fluorescence leakage (crosstalk) and calculate the true fluorescence intensity of each dye, a technique called compensation is used.

[0020] Compensation is a technique that separates the fluorescence signals obtained from a sample stained with multiple fluorescent dyes into signals corresponding to each dye.

[0021] As shown in equation (1) below, in compensation, the output (measured value) of each filter is first represented by a matrix F (number of filters × 1 matrix). In this example, the fluorescence intensities measured by the three filters F1, F2, and F3 are x, y, and z, respectively.

[0022]

number

[0023] These measured values ​​x, y, and z are obtained as a result of the mixing of fluorescence from different dyes (in this example, dyes A, B, and C). The purpose of compensation is to calculate the true fluorescence intensities a, b, and c of each dye from this mixed signal. These are also called dye intensity coefficients and are represented as elements of matrix A (a matrix of the number of dyes × 1).

[0024] The amount of fluorescence from each dye that leaks into (mixes with) each filter is represented by the leakage matrix M (a matrix of the number of dyes × the number of filters). In equation (1) above, the matrix located between each filter F1, F2, F3 and the intensity coefficients a, b, c of the dyes is the leakage matrix M. Each element of the leakage matrix M represents how the fluorescence spectrum of a single dye is split and detected across its respective wavelength filter.

[0025] And the inverse matrix M of the leakage matrix M -1 By multiplying this by the filter output matrix F, the fluorescence intensity (separation result) A (number of dyes × 1 matrix) of each dye is calculated. Specifically, the calculation is performed using the following equation (2).

[0026] A=M -1 ·F ···(2)

[0027] Here, A represents the fluorescence intensity of each dye (number of dyes × 1 matrix). M represents the leakage matrix (number of dyes × number of filters matrix). -1 represents the inverse matrix of the leakage matrix (number of filters × number of dyes matrix). F represents the output of each filter (number of filters × 1 matrix).

[0028] Substituting equation (1) above into equation (2), we obtain the following equation (3).

[0029]

number

[0030] However, in order to perform this compensation calculation, the leakage matrix M must be a square matrix, and the number of filters must be equal to or greater than the number of dyes. For example, if the leakage matrix M is not a square matrix, as shown in equation (4) below, the compensation calculation cannot be performed.

[0031]

number

[0032] [Spectral Unmixing] As another method for correcting fluorescence crosstalk, there is a technique called spectral unmixing. In spectral unmixing, the standard fluorescence spectrum of each dye (Spectral Reference, SR) is used.

[0033] Specifically, first, a matrix S (number of dyes n × number of detection channels ch) is prepared by concatenating the spectral references of the respective dyes for the number of dyes. This matrix S represents the fluorescence intensity in each detection channel when each dye is measured individually.

[0034] Next, let the measured spectrum obtained from the multiply stained sample be W (ch × 1 matrix).

[0035] Then, as shown in the following equation (5), the fluorescence intensity A (n × 1 matrix) of each dye is calculated using the Least Squares Method (LSM).

[0036] A=(S T S) -1 S·W ···(5)

[0037] Here, A is the intensity of each dye (n × 1 matrix, n is the number of dyes). S is the spectral reference matrix (number of dyes n × number of detection channels ch). W is the measured spectrum (ch × 1 matrix). S T is the transpose matrix of S. (S T S) -1 is the inverse matrix of (S T S).

[0038] Unlike compensation, spectral unmixing can be applied even when the number of detection channels (ch) is larger than the number of dyes (n).

[0039] However, when applying the conventional color separation techniques described above (compensation, spectral unmixing) to the system shown in Figure 2 or Figure 3, which will be described later, the following problems arise.

[0040] In the configurations shown in Figures 2 and 3, described later, the multi-wavelength fluorescence information acquisition unit 120 has a large number of detection channels (e.g., 32 channels x 2), and the total number of detection channels for the combined dye signal per cell is greater than or equal to the number of dyes. Therefore, it is possible to calculate the fluorescence intensity of each dye using the aforementioned formula. On the other hand, the image information acquisition unit 110 has only a limited number of detection channels (number of filters). As a result, the following relationship holds.

[0041] Number of detection channels for the combined dye signal of a single cell ≥ Number of dyes > Number of fluorescence image filters

[0042] Therefore, as shown in equation (4) above, if the image information acquisition unit has, for example, only three filters (F1, F2, F3) and acquires fluorescence image information of a sample stained with six types of dyes (A, B, C, D, E, F), problems arise with conventional technology.

[0043] In other words, the leakage matrix used for compensation is not a square matrix, and its inverse cannot be calculated, making it impossible to apply the aforementioned formula. Furthermore, spectral unmixing cannot be directly applied either.

[0044] As a result, the fluorescent image information acquired by the image information acquisition unit cannot adequately remove crosstalk (fluorescence leakage) between dyes, making it difficult to separate and extract accurate fluorescent images corresponding to each dye.

[0045] [2. Overview of this technology] To solve the above problems, this technology provides a fluorescence image information processing system comprising: an image information acquisition unit that acquires fluorescence image information of minute particles; a multi-wavelength fluorescence information acquisition unit that acquires multi-wavelength fluorescence information of the minute particles; and a calculation unit that corrects or complements the fluorescence image information based on the fluorescence image information and the multi-wavelength fluorescence information when the number of channels used by the image information acquisition unit is less than the number of channels required to discriminate the fluorescence of the minute particles.

[0046] [(1) System Configuration] Figure 1 is a block diagram showing an example configuration of the fluorescence image information processing system 100 according to this embodiment. The system 100 is broadly composed of an image information acquisition unit 110, a multi-wavelength fluorescence information acquisition unit 120, and a calculation unit 130. The calculation unit 130 includes an image information processing unit 131 and a multi-wavelength fluorescence information processing unit 132.

[0047] The image information acquisition unit 110 acquires fluorescence image information of minute particles flowing through a flow cell (not shown). The multi-wavelength fluorescence information acquisition unit 120 acquires multi-wavelength fluorescence information (raw spectral data) of minute particles flowing through the flow cell. The multi-wavelength fluorescence information processing unit 132 performs multi-wavelength fluorescence information processing, including spectral unmixing, based on the multi-wavelength fluorescence information and reference information to generate multi-color fluorescence separation information. The image information processing unit 131 corrects or complements the fluorescence image information based on the fluorescence image information acquired from the image information acquisition unit 110 and the multi-color fluorescence separation information if the number of channels used by the image information acquisition unit 110 is less than the number of channels required to discriminate the fluorescence of minute particles.

[0048] Here, the arithmetic unit 130 may be configured as an independent device (fluorescence image information processing device) or it may be incorporated into a computer that controls the entire system 100. In other words, in this specification, the term "system" is used as a broad concept that includes "device". Therefore, the technical ideas described herein can be similarly applied even if "system" is replaced with "device".

[0049] [(2) Operation] In the multi-wavelength fluorescence information processing unit 132, first, an unmixing process based on equation (5) is performed using the multi-wavelength fluorescence information obtained from the multi-wavelength fluorescence information acquisition unit 120 and the SR matrix included in the reference information to calculate the intensity of each dye component (part of the multi-color fluorescence separation information).

[0050] Next, the image information processing unit 131 uses this multicolor fluorescence separation information (intensity of each dye component) and the SR information included in the reference information to complete the data on the image detector side. Specifically, it completes the leakage matrix on the image detector side and outputs a corrected image (image with crosstalk removed) and a completed image (virtually generated image) (details will be explained in the embodiment described later).

[0051] The multi-wavelength fluorescence information acquisition unit 120 normally measures and records the spectrum of an entire cell, but because cells move through the linear illumination area, it is also possible to measure the time-series spectral waveform of a single cell. This time-series spectral waveform can be input to the calculation unit 130, and correction and interpolation processing can be performed according to the position within the cell (details will be explained in the embodiment described later).

[0052] Furthermore, the spectral reference provided as SR is not limited to dyes, but may also be the autofluorescence spectrum produced by substances contained within the cells themselves. This allows for the acquisition of fluorescence images with the effects of autofluorescence removed (details will be explained in the embodiments described later).

[0053] [(3) Definitions of Terms] In this specification, "fluorescence image information" refers to information acquired by the image information acquisition unit 110 that represents fluorescence from minute particles in a multidimensional manner. Specifically, it is data showing the distribution of fluorescence intensity (luminance value) in a specific wavelength band for each pixel.

[0054] In this specification, "multi-wavelength fluorescence information" refers to the intensity data (raw spectral data) of fluorescence at each wavelength from minute particles, acquired by the multi-wavelength fluorescence information acquisition unit 120. This data is acquired with a specific wavelength resolution (e.g., 1 nm, 5 nm, 10 nm, etc.).

[0055] In this specification, "reference information" includes at least "dye information (spectral reference: SR)," which is standard fluorescence spectral information of the dye used for fluorescent labeling, and "device channel information," which indicates the characteristics of the detection channels of the image information acquisition unit 110 and the multi-wavelength fluorescence information acquisition unit 120. The reference information may also include other information, such as autofluorescence spectral information, as needed.

[0056] In this specification, "dye information (spectral reference: SR)" refers to standard fluorescence spectral information of dyes used for fluorescent labeling and constitutes part of the reference information. Dye information is obtained or provided by one of the following methods: pre-registration, user input, or system estimation. [Pre-registration] The dye information may be data pre-registered in a database built into the device or in an external database. [User input] The dye information may be data manually entered by the user (for example, in a text file, spreadsheet format, or via a GUI) containing the fluorescence spectrum information of the dyes used in the experiment. [System Estimation] The pigment information may be data estimated by the instrument from the measurement data using a multivariate analysis algorithm such as non-negative matrix factorization (NMF).

[0057] In this specification, "device channel information" refers to information indicating the characteristics of the detection channels of the image information acquisition unit 110 and the multi-wavelength fluorescence information acquisition unit 120, and constitutes part of the reference information. Specifically, it includes information such as the transmission wavelength characteristics of the filters and the sensitivity characteristics of the detectors. The transmission wavelength characteristics of the filters are the wavelength band (transmittance curve) of the light transmitted by each filter. The sensitivity characteristics of the detectors are the sensitivity (quantum efficiency, etc.) of each detector to the wavelength of light it senses.

[0058] In this specification, "multicolor fluorescence separation information" refers to information generated in the multi-wavelength fluorescence information processing unit 132 by performing multi-wavelength fluorescence information processing, which includes at least spectral unmixing (color separation), based on multi-wavelength fluorescence information and reference information, and is used for correction or supplementation of fluorescence image information in the image information processing unit 131. Multicolor fluorescence separation information includes, for example, the intensity coefficient or abundance ratio of each dye component (which may include autofluorescent components) obtained by spectral unmixing. In addition, other information necessary for correction or supplementation (e.g., leakage matrix, coefficient k, etc.) may also be generated in relation to the multicolor fluorescence separation information during the multi-wavelength fluorescence information processing process and may be included.

[0059] In this specification, "multi-wavelength fluorescence information processing" means a series of processes performed in the multi-wavelength fluorescence information processing unit 132 based on multi-wavelength fluorescence information and reference information, including spectral unmixing (color separation), generation of a leakage matrix (if necessary), calculation of coefficients for conversion to measurements by a virtual detection filter in the image information acquisition unit 110 (if necessary), and other processes including the generation of multi-color fluorescence separation information and the generation of information necessary for correcting or supplementing fluorescence image information in the image information processing unit 131.

[0060] In this specification, “acquisition” means obtaining or making available information in any way, such as through measurement, configuration, input, or retrieval from a database.

[0061] In this specification, "correction" refers to a process that removes or reduces the effects of inter-dye crosstalk (fluorescence leakage) and autofluorescence contained in fluorescence image information.

[0062] In this specification, "interpolation" refers to the process of estimating fluorescence information in wavelength bands that cannot be detected by the image information acquisition unit based on multi-wavelength fluorescence information, and generating new fluorescence image information.

[0063] In this specification, "wavelength band" refers to the range of wavelengths of light. A wavelength band may be defined, for example, by a central wavelength and bandwidth (full width at half maximum: FWHM, etc.), or by a lower limit wavelength and an upper limit wavelength. For example, a "wavelength band of 500-550 nm" means the wavelength range from 500 nm to 550 nm. Also, a "wavelength band with a central wavelength of 525 nm and a bandwidth of 50 nm" means the wavelength range from approximately 475 nm to 575 nm (according to the definition of bandwidth).

[0064] In this specification, "fluorescence intensity" refers to a value that quantitatively represents the strength of fluorescence. Fluorescence intensity can be expressed, for example, as the amount of light energy, the number of photons, or the magnitude of the corresponding electrical signal (voltage, current, etc.) detected by a photodetector (e.g., photodiode, PMT (photomultiplier tube), APD, SPAD, HPD (hybrid photodiode), CCD, CMOS). Fluorescence intensity may be expressed in arbitrary units (au) or in calibrated units (e.g., photons per second).

[0065] [(4) Subjects of Analysis] The target of analysis in the fluorescence image information processing system 100 of this technology is a liquid sample containing biological microparticles or microparticles other than biological microparticles.

[0066] Biological microparticles include cells and noncellular biological microparticles. Examples of cells include living cells such as blood cells (red blood cells, white blood cells, etc.) and germ cells (sperm, fertilized eggs, etc.). These cells may be directly collected from a sample such as whole blood, or obtained after culturing. Cells may be in the form of individual cells or cell aggregates (spheroids, organoids, etc.). Microorganisms, including bacteria, can also be used as biological microparticles for analysis. Multiple bacterial species constituting a microbiome can be analyzed together or individually. Examples of noncellular biological microparticles include extracellular vesicles (EVs). Extracellular vesicles include exosomes, microvesicles, and apoptotic bodies, and may be further classified according to their origin, size, and contents (proteins, nucleic acids, lipids, etc.).

[0067] Examples of microparticles other than biological microparticles include beads, modified beads, cell aggregates (spheroids), emulsions, and intracellular encapsulation carriers. These microparticles may be used for calibration or may hold biological components (e.g., cells or cell-derived components (secretions, etc.)). For example, biological components can be captured in beads (e.g., antibody-modified beads, nucleic acid-modified beads), or biological components can be encapsulated in emulsions. Fluorescent beads can be used for system calibration or as standard materials. Modified beads refer to beads that have been given functions according to their purpose, such as the ability to bind to specific molecules or changes in fluorescence intensity under specific conditions. Intracellular encapsulation carriers are structures that hold living cells inside for culture, transport, or analysis, and examples include hydrogels, liposomes, and microcapsules.

[0068] These microparticles may be labeled with one or more labeling substances (e.g., dyes (especially fluorescent dyes) and fluorescently labeled antibodies). The microparticles themselves may also emit autofluorescence. The labeling substances may specifically bind to specific molecules (e.g., cell surface antigens, intracellular proteins, nucleic acids, lipids, etc.) or they may be nonspecific labeling substances (e.g., cell membrane-permeable dyes, etc.).

[0069] These microparticles may, for example, be carriers used for secretion analysis. If the carrier is an emulsion, the biological microparticles may be recovered while contained within the emulsion. In this case, the recovered preparative product is an emulsion, and the dispersed phase constituting the emulsion may be biological microparticles contained within an emulsion containing the microparticles to be preparated (e.g., beads). The dispersion medium constituting the emulsion may be appropriately selected by those skilled in the art depending on the type of emulsion particles. The emulsion may be a multiple emulsion (e.g., oil-in-water ("o / w / o") or water-in-oil ("w / o / w")). The shape of the carrier is not particularly limited and may have any microparticle shape, such as spherical, elliptical, or rod-shaped. The emulsion may be a monodisperse emulsion or a polydisperse emulsion.

[0070] [3. First Embodiment of the Technology (Example 1 of a Fluorescence Image Information Processing System)] This embodiment relates to a fluorescence image information processing system that corrects or complements fluorescence image information using multi-wavelength fluorescence information (spectral information) when the number of channels used in the image information acquisition unit of an imaging flow cytometer is less than the number of channels required to discriminate the fluorescence of minute particles.

[0071] [(1) System Configuration] Figure 2 is a conceptual diagram showing an example configuration of the fluorescence image information processing system 100 according to this embodiment. The system 100 includes an image information acquisition unit 110 that acquires fluorescence image information of minute particles (e.g., cells) 102 flowing in a flow cell 101, a multi-wavelength fluorescence information acquisition unit 120 that acquires multi-wavelength fluorescence information (fluorescence spectrum information) of the minute particles 102, and a calculation unit 130 (not shown) that corrects or complements the fluorescence image information based on the fluorescence image information, the multi-color fluorescence separation information, and the reference information (in particular, dye information and device channel information).

[0072] [(2) Image information acquisition unit 110] [(2-1) Structure] The image information acquisition unit 110 includes a flow cell 101, an excitation light source (not shown), an objective lens 112, an imaging lens 113, a dichroic mirror 114, a bandpass filter 115, and a detector (e.g., a line sensor) 116.

[0073] The flow cell 101 is a channel through which fine particles 102 flow at a constant velocity (0.1 to 10 m / sec).

[0074] An excitation light source (not shown) irradiates the microparticles 102 with fluorescence excitation light 103. In this embodiment, two excitation light sources (e.g., a 405 nm laser and a 488 nm laser) are used, but the number and wavelength of the excitation light sources are not limited thereto. Three or more excitation light sources may be used.

[0075] The objective lens 112 is an optical element that focuses fluorescence from the minute particles 102.

[0076] The imaging lens 113 is an optical element that forms an image of the fluorescence focused by the objective lens 112.

[0077] The dichroic mirror 114 is an optical element that transmits light in a specific wavelength band and reflects light in other wavelength bands. It is used to separate excitation light from fluorescence or to split fluorescence into multiple wavelength bands.

[0078] The bandpass filter 115 is an optical element that transmits only light in a specific wavelength band. The number of filters and the transmitted wavelength band are not limited.

[0079] Detector 116 is a one-dimensional photodetector that detects fluorescence from minute particles 102 passing through the flow cell 101 in a linear fashion. The spatial resolution is, for example, 0.2 to 4 μm / pixel. Two-dimensional fluorescence image information A to C is obtained by rearranging the time-series signals from the line sensor. The number of detectors 116 (and the number of corresponding filters) determines the number of channels used in the image information acquisition unit 110.

[0080] [(2-2) Operation] In this example, lateral excitation light from two excitation light sources (not shown) is irradiated onto minute particles 102 in the flow cell 101 via a dichroic mirror (not shown). The fluorescence generated from the minute particles 102 is focused by the objective lens 112 and separated by wavelength by the dichroic mirror 114. The separated fluorescence is focused by the imaging lens 113, passes through bandpass filters 115 that transmit different wavelength bands, and is incident on the corresponding detector 116. Each detector 116 detects the intensity of the incident fluorescence and converts it into an electrical signal. This electrical signal is sent to the calculation unit 130 (not shown) as fluorescence image information A to C.

[0081] [(3) Multi-wavelength fluorescence information acquisition unit 120] [(3-1) Structure] The multi-wavelength fluorescence information acquisition unit 120 includes a flow cell 101, an excitation light source (not shown), an objective lens 121, imaging lenses 122, 127, a dichroic mirror 123, a slit 124, a collimator lens 125, a diffraction grating 126, and a detector (e.g., a multi-channel detector) 128.

[0082] The flow cell 101 and the excitation light source are shared with the image information acquisition unit 110.

[0083] The objective lens 121 is an optical element that collects fluorescence from minute particles 102. The objective lens 121 may be the same as the objective lens 112 included in the image information acquisition unit 110.

[0084] The imaging lens 122 is an optical element that forms an image of the fluorescence focused by the objective lens 121.

[0085] The dichroic mirror 123 is an optical element that transmits light in a specific wavelength range and reflects light in other wavelength ranges.

[0086] Slit 124 is an optical element that limits the incident range of fluorescence in order to improve spatial resolution. Instead of a slit, an optical fiber may be used, for example.

[0087] The collimator lens 125 is an optical element that converts the fluorescence that has passed through the slit 124 into parallel light.

[0088] The diffraction grating 126 is an optical element that disperses incident light according to its wavelength.

[0089] The imaging lens 127 is an optical element that forms an image of light dispersed by the diffraction grating 126.

[0090] The detector 128 detects the intensity of fluorescence dispersed for each wavelength. In this embodiment, a 32-channel detector is used, but the number of channels is not limited to this. It is sufficient if there are more channels than the number of channels in the image information acquisition unit 110. For example, for excitation at 405 nm, the range of 410 to 730 nm is detected using 32 channels, and for excitation at 488 nm, the range of 500 to 820 nm is detected using 32 channels.

[0091] [(3-2) Operation] Similar to the image information acquisition unit 110, light from an excitation light source (not shown) is irradiated onto minute particles 102 in the flow cell 101, and the fluorescence generated from the minute particles 102 is focused by the objective lens 121. The focused fluorescence is separated by the dichroic mirror 123, focused by the imaging lens 122, and passes through the slit 124. The fluorescence that has passed through the slit 124 is converted into parallel light by the collimator lens 125 and incident on the diffraction grating 126. The diffraction grating 126 disperses the incident light at different angles for each wavelength and images it onto the detector 128 by the imaging lens 127. The detector (32CH×2 in this example) 128 detects the fluorescence intensity in each wavelength band and converts it into an electrical signal. This electrical signal is sent to the calculation unit 130 (not shown) as multi-wavelength fluorescence information.

[0092] As described above, the multi-wavelength fluorescence information acquisition unit 120 according to this embodiment acquires multi-wavelength fluorescence information of minute particles using an optical system that employs a diffraction grating 126 and a multi-channel detector 128. Furthermore, the multi-wavelength fluorescence information acquisition unit 120 according to this embodiment acquires multi-wavelength fluorescence information of minute particles using an optical system (dichroic mirror 123) that separates wavelengths and a plurality of detectors 128 provided for each wavelength.

[0093] [(4) Modified Version] Figure 3 is a block diagram showing another configuration example of the fluorescence image information processing system 100 according to this embodiment.

[0094] This system includes an image information acquisition unit 110 that acquires fluorescence image information of minute particles 102 flowing through a flow cell, a multi-wavelength fluorescence information acquisition unit 120 that acquires multi-wavelength fluorescence information of minute particles 102, and a calculation unit (not shown) that corrects or interpolates fluorescence image information based on fluorescence image information and multi-wavelength fluorescence information.

[0095] In the configuration shown in Figure 2, there is one objective lens 121, and the optical path is shared between the image information acquisition unit 110 and the multi-wavelength fluorescence information acquisition unit 120. Fluorescence from the minute particles 102 is focused by the objective lens 121, then split into multiple wavelength bands by the dichroic mirror 123, and branched into two optical paths by the half-mirror 129. One optical path is led to the image information acquisition unit 110 and enters the detector 116 via the bandpass filter 115 and imaging lens 113. The other optical path is led to the multi-wavelength fluorescence information acquisition unit 120 and enters the detector 128 via the imaging lens 122, slit 124, collimator lens 125, diffraction grating 126, and imaging lens 127.

[0096] The other components (excitation light source, imaging lens, detector, etc.) and operation of the image information acquisition unit and the multi-wavelength fluorescence information acquisition unit are the same as those shown in Figure 2, so a detailed explanation is omitted.

[0097] Furthermore, the multi-wavelength fluorescence information acquisition unit is not limited to the configuration shown in Figures 2 and 3. For example, a method using refractive index wavelength dispersion or a configuration with one path and one detector branched by a filter may be adopted as the optical system for separating wavelengths. In addition, the excitation light may be illuminated by multiple spots with different radio band frequencies using an AOD (acousto-optic deflection element) and an AOFS (acousto-optic frequency shifter), and a configuration may be adopted in which image acquisition and conventional fluorescence signal acquisition are performed simultaneously from a single PMT (photomultiplier tube) by frequency analysis (Non-patent literature: Eric D. Diebold, Digitally synthesized beat frequency multiplexing for sub-millisecond fluorescence microscopy, nature Photonics, 2013).

[0098] [(5) Arithmetic unit 130] [(5-1) Structure] Figure 4 is a block diagram showing an example configuration of the arithmetic unit 130 according to this embodiment. The arithmetic unit 130 comprises an arithmetic device 140, an input interface 141, and an output interface 142.

[0099] The computing unit 140 is a computer equipped with a CPU, RAM, ROM, and storage (e.g., a hard disk drive or solid-state drive), and performs correction or interpolation processing of fluorescence image information according to this embodiment. This storage stores programs executed by the computing unit 140, various data (e.g., image data, multi-wavelength fluorescence information, coefficient k, leakage matrix), and spectral reference (SR).

[0100] The input interface 141 is an interface for receiving data (fluorescence image information and multi-wavelength fluorescence information, respectively) from the image information acquisition unit 110 and the multi-wavelength fluorescence information acquisition unit 120.

[0101] The output interface 142 is an interface for outputting the fluorescence image information corrected or complemented by the arithmetic unit 140 to an external device (e.g., a display, printer, or external storage device).

[0102] [(5-2) Operation (Processing Flow)] Figure 5 is a flowchart showing an example of the processing flow of the fluorescence image information processing system 100 according to this embodiment. The operation of the fluorescence image information processing system 100 will be described in detail with reference to Figure 5.

[0103] [Step S101: Obtain SR (spectral reference) of the staining dye] The user inputs information about the dyes they have previously stained into the device (calculation unit 130) in the form of a spectral reference (SR). The SR is standard multi-wavelength fluorescence information for each dye (for example, fluorescence intensity data for each wavelength with a 10 nm pitch). If autofluorescence is considered, the spectral information of the autofluorescence is also input as part of the SR. The SR is stored in storage.

[0104] [Step S102: Image and spectrum acquisition] The image information acquisition unit 110 acquires fluorescence image information of minute particles 102 flowing within the flow cell 101. Simultaneously, the multi-wavelength fluorescence information acquisition unit 120 acquires multi-wavelength fluorescence information of the same minute particles 102. The acquired data is sent to the computing unit 140 via the input interface 141 and stored in storage.

[0105] [Step S103: Unmixing using SR] The multi-wavelength fluorescence information processing unit 132 performs an unmixing process using the reference information (SR) stored in storage and the multi-wavelength fluorescence information acquired from the multi-wavelength fluorescence information acquisition unit 120 to generate multi-color fluorescence separation information. Unmixing is a process that separates the mixed fluorescence spectrum (multi-wavelength fluorescence information) into components of each dye (or autofluorescence) based on the reference information (SR). Specifically, the intensity of each dye is calculated using the above equation (5) (least squares method).

[0106] [Step S104: Comparison of the number of image filters and the number of staining dyes] The image information processing unit 131 compares the number of detection channels (number of filters) of the image information acquisition unit 110 with the number of staining dyes (including the number of autofluorescent components if autofluorescence is considered). If the number of dyes > the number of image filters (step S104: YES), the process proceeds to step S105 (virtual filter generation). If the number of dyes ≤ the number of image filters (step S104: NO), the process proceeds to step S108 (image compensation).

[0107] [Step S105: Virtual filter generation] The image information processing unit 131 generates a virtual detection filter (virtual filter) corresponding to dyes that cannot be detected by the image information acquisition unit 110, based on the dye information and device channel information included in the reference information. Specifically, one of the following methods, or a combination thereof, can be used for generation.

[0108] [Automatic generation] The image information processing unit 131 automatically determines the wavelength band of the virtual filter to cover the missing dye channels based on the dye information (spectral reference: SR) included in the reference information entered by the user.

[0109] For example, one method is to set up a virtual filter that includes the peak wavelength of each dye, or to set it up considering the broadening of the dye's fluorescence spectrum. Specific generation methods will be described later.

[0110] [Manual generation] The user may input the wavelength band of the virtual filter numerically or set it via a graphical user interface (GUI). The user can set an appropriate wavelength band by referring to the fluorescence spectrum of the dye or the transmission wavelength characteristics of other filters.

[0111] The transmission wavelength characteristics of the generated virtual filter are stored in storage, for example, as transmittance data for each wavelength with a 10nm pitch, similar to the transmission wavelength characteristics of the actual filter.

[0112] [Step S106: Generate leakage matrix] The image information processing unit 131 generates a leakage matrix based on reference information (dye information, instrument channel information) and virtual filters. The leakage matrix is ​​a matrix that represents how much fluorescence from each dye (or autofluorescence component) leaks into (mixes with) each detection filter (real filter and virtual filter).

[0113] If a virtual filter is generated, the leak matrix is ​​generated to include both the real and virtual filters. That is, the number of rows in the leak matrix is ​​equal to the total number of real and virtual filters, and the number of columns is equal to the number of dyes (including the number of autofluorescent components if autofluorescence is considered).

[0114] If we denote the summation with respect to wavelength as Σ, then each element of the leakage matrix is ​​calculated using the following formula (6).

[0115] The element (i,j) of the leakage matrix = Σ{spectrum of dye j * transmission spectrum of filter i} ... (6)

[0116] Here, i is the filter number (including real and virtual filters). j is the dye number (including autofluorescence). Σ is the sum over wavelength. "Spectrum of dye j" is the spectral reference (SR) of dye j included in the reference information. "Transmission spectrum of filter i" is the transmission spectrum of the real or virtual filter included in the reference information. "*" represents the Hadamard product (element product).

[0117] Specifically, the Hadamard product (elemental product) of the matrix storing the spectrum of dye j and the matrix storing the transmission spectrum of filter i is calculated, and the elements (i,j) of the leakage matrix are calculated by multiplying all the elements of the resulting matrix.

[0118] The generated leakage matrix is ​​stored in storage and used for image compensation (color separation operation), which will be described later.

[0119] Note that the leakage matrix does not necessarily need to include virtual filters and may consist only of real filters. However, including virtual filters makes it possible to distinguish more pigments and improves color separation accuracy.

[0120] [Step S107: Simulation of measurement results using a virtual filter (coefficient calculation)] The multi-wavelength fluorescence information processing unit 132 simulates the measurement value at the virtual filter using the multi-wavelength fluorescence information acquired by the multi-wavelength fluorescence information acquisition unit 120 and the transmission spectra of the real and virtual filters (F4~F6) included in the reference information. Specifically, the multi-wavelength fluorescence information processing unit 132 calculates a coefficient k for converting the multi-wavelength fluorescence information and reference information into the measurement value at the virtual filter.

[0121] A coefficient k is used to estimate the fluorescence intensity measured by the virtual filter from the multi-wavelength fluorescence information acquired by the multi-wavelength fluorescence information acquisition unit 120. The coefficient k represents the ratio of the transmission intensity of the virtual filter to the transmission intensity of the actual filter. The calculated coefficient k is stored in storage.

[0122] [Step S108: Image Compensation (Color Separation Calculation)] The image information processing unit 131 performs color separation (compensation) on the fluorescence image information using the leakage matrix and coefficient k. Specifically, it generates images (corrected images) corresponding to each dye using the following equation (7) (inverse matrix calculation).

[0123] A=M -1 F ···(7)

[0124] Here, A is the intensity of each dye (n×1 matrix, where n is the number of dyes (actual number of dyes + virtual number of dyes)). M is the leakage matrix. F is the fluorescence intensity measured by each filter (measured value or value corrected by coefficient k). -1 This is the inverse matrix of M.

[0125] [Step S109: Output Result] The arithmetic unit 131 outputs images corresponding to each pigment obtained by color separation calculation (corrected images) and images generated using virtual filters (interpolated images) via the output interface 142. The output destinations are displays, printers, external storage devices, etc.

[0126] [(5-3) Specific Examples] Figure 6 is a graph showing an example of multi-wavelength fluorescence information. This figure is intended to explain the method of generating virtual filters and interpolating leak matrices in this technology. It shows the fluorescence spectra of six dyes (A-F) for two excitation wavelengths (405 nm and 488 nm) based on a spectral reference (SR) input by the user, and the transmission wavelength bands of the real filters (F1-F3) and virtual filters (F4-F6) in the image information acquisition unit.

[0127] In this embodiment, if the number of channels (number of filters) used in the image information acquisition unit is less than the number of dyes, a virtual filter is generated using the multi-wavelength fluorescence information (fluorescence spectrum information) acquired by the multi-wavelength fluorescence information acquisition unit (spectral detector) and SR to interpolate the leakage matrix.

[0128] Specifically, if the image detector is equipped with filters F1 for dye A, F2 for dye B, and F3 for dye D, then there are no corresponding physical filters for dyes C, E, and F.

[0129] Therefore, the image information processing unit 131 identifies the missing dyes (C, E, F) based on the dye information (SR) input by the user and the filter information (transmission wavelength band) held by the device, and generates virtual filters (F4, F5, F6) corresponding to the identified dyes. The transmission wavelength band of the virtual filters is set automatically (for example, to include the peak wavelength of each dye, or to take into account the broadening of the fluorescence spectrum of the dyes) or manually by the user.

[0130] Next, a leakage matrix is ​​generated that includes all of the real filters (F1-F3) and virtual filters (F4-F6), as shown in equation (8) below.

[0131]

number

[0132] Each element of the leakage matrix is ​​calculated by performing the Hadamard product (elemental product) of the matrix storing the spectrum of dye j and the matrix storing the transmission spectrum of filter i, and then integrating all the elements of the resulting matrix. If the cutoff wavelength of the image detector filter and the wavelength division of the detection channel of the spectral detector do not match, the matrix size is adjusted using various interpolation methods such as linear interpolation or spline interpolation.

[0133] Specific examples of dyes and filters are as follows:

[0134] [405nm excitation] ·Dye: BV421, BV510, BV570, BV605, BV650, BV711, BV750, BV785 Bandpass filters: 415-455nm, 455-555nm, 455-555nm, 565-595nm, 595-625nm, 625-675nm, 685-735nm, 745-805nm, 755-815nm

[0135] [488nm excitation] ·Dye: Alexa Fluor488, Calcein-AM, Calcein RedOrange, PE, PE-Cy5, Calcein DeepRed, PE-Cy7 • Bandpass filters: 500-550nm, 570-600nm, 670-720nm, 760-820nm

[0136] Figure 7 is a graph showing an example of multi-wavelength fluorescence information. This figure explains the method for calculating the coefficient k used to simulate the measurement values ​​at the virtual filter in this technology. It shows the fluorescence spectra (measured spectra) U and V actually measured by the multi-wavelength fluorescence information acquisition unit (spectral detector), and the transmission wavelength bands of the actual filters (F1-F3) and virtual filters (F4-F6). The spectra U and V measured by the spectral detector are the sum of the fluorescence from each dye in one cell. U is the measured spectrum for 405 nm excitation light, and V is the measured spectrum for 488 nm excitation light.

[0137] To simulate the measurements with virtual filters (F4~F6), first, calculate the Hadamard product (element product) of the measured spectra U and V with the transmission spectra of each filter (F1~F6) to determine the transmission intensity of each filter. Next, calculate the ratio k between the transmission intensity of the virtual filter and the transmission intensity of the actual filter. Specifically, calculate the coefficients k1, k2, and k3 using the following equations (9)~(11).

[0138] k1 = Σ{(F4)*U} / Σ{(F2)*U} ···(9)

[0139] k2 = Σ{(F5)*V} / Σ{(F3)*V} ···(10)

[0140] k3 = Σ{(F6)*V} / Σ{(F3)*V} ···(11)

[0141] Here, Σ represents the summation with respect to wavelength, and * represents the Hadamard product (element product). (Fi) represents the transmission spectrum of filter Fi, U represents the measured spectrum with 405 nm excitation light, and V represents the measured spectrum with 488 nm excitation light.

[0142] When calculating the coefficient k1, the denominator is selected from the actual filter (F2 in this example) that has the highest fluorescence intensity in the corresponding wavelength range in the measured spectrum U. Alternatively, the spectral waveforms of several hundred cells or more may be measured, and the actual filter with the highest correlation of transmitted light intensity with the virtual filter F4 may be selected as the denominator. Alternatively, the transmission intensity ratio may be calculated based on the color separation results after spectral unmixing and the measured spectrum reconstructed using a spectral reference (SR), and the filter to be used as the denominator may be selected. Similarly, for coefficients k2 and k3, the actual filter F3 with the highest fluorescence intensity or high correlation in the wavelength range corresponding to the virtual filters F5 and F6, respectively, is selected.

[0143] By using these coefficients k, the fluorescence intensity measured with a virtual filter can be estimated from the measured spectrum with relatively low computational cost and high accuracy.

[0144] Figure 8 is a schematic diagram illustrating image compensation (crosstalk correction) and virtual image generation in this technology, showing the relationship between the fluorescence images acquired by the image information acquisition unit (F1 measurement image, F2 measurement image, F3 measurement image) and the virtual images generated by the virtual filter ((F4) virtual pixel value, (F5) virtual pixel value, (F6) virtual pixel value).

[0145] The image information acquisition unit acquires three fluorescence images (F1 measurement image, F2 measurement image, and F3 measurement image) using real filters F1, F2, and F3. Focusing on one pixel corresponding to the same cell coordinate in these images, the respective pixel values ​​(fluorescence intensity) are denoted as x, y, and z. On the other hand, the pixel values ​​corresponding to the virtual filters (F4), (F5), and (F6) are estimated as k1·y, k2·z, and k3·z, respectively, using the calculated coefficient k.

[0146] This makes it possible to solve the image compensation equation (12) shown below using the measurement matrix, which consists of the measured values ​​x, y, and z and the estimated values ​​k1·y, k2·z, and k3·z, and the generated leakage matrix. This calculation yields the true fluorescence intensities a, b, c, d, e, and f of each dye, corrected for crosstalk between dyes.

[0147]

number

[0148] [(6) Effects] According to this embodiment, the following effects can be obtained.

[0149] [(a) Achieving both high-precision color separation and image acquisition] Even when the number of detection channels (filters) in the image information acquisition unit is less than the number of staining dyes, a virtual detection filter is generated using high-resolution spectral information acquired by the multi-wavelength fluorescence information acquisition unit, and the leakage matrix is ​​compensated for. This makes it possible to separate a large number of dyes with high accuracy and acquire fluorescence images corresponding to each dye, even while using a limited number of image detection channels.

[0150] [(b) Compensation for undetected wavelength information] Fluorescence information in wavelength ranges not detected by the image information acquisition unit can be supplemented based on multi-wavelength fluorescence information. This allows for the restoration of information missing from the image information acquisition unit, resulting in the acquisition of more complete fluorescence image information.

[0151] [(c) Reducing computational costs] Since the measured values ​​from the virtual filter are estimated using the measured spectrum and the coefficient k, there is no need to repeatedly perform complex inverse matrix calculations, enabling highly accurate color separation with relatively low computational cost.

[0152] [(d) Flexible system design] By sharing the optical paths of the image information acquisition unit and the multi-wavelength fluorescence information acquisition unit (Figure 3), the device can be miniaturized and its cost reduced. Furthermore, since various configurations (diffraction grating and multi-channel detector, tunable filter and single detector, etc.) can be adopted for the multi-wavelength fluorescence information acquisition unit, the system offers high flexibility.

[0153] [(e) Wide range of applications] This technology is applicable to various applications that perform multiplex fluorescence staining, including not only imaging flow cytometers but also fluorescence microscopes and spectroscopic analyzers. It can be used for the analysis of diverse microparticles, including not only cells but also extracellular vesicles, bacterial flora, and beads.

[0154] The above description of the fluorescence image information processing system according to this embodiment can be applied to other embodiments of this technology, unless there are any particular technical inconsistencies.

[0155] [4. Second Embodiment of the Technology (Example 2 of a Fluorescence Image Information Processing System)] In this embodiment, the multi-wavelength fluorescence information processing unit 132 calculates coefficients for each section of the multi-wavelength fluorescence information acquired by the multi-wavelength fluorescence information acquisition unit 120.

[0156] Figures 9 and 10 are schematic diagrams illustrating the color separation method using position-dependent spectra within microparticles in this technology. Figure 9 shows the relationship between microparticles 102 flowing through a flow cell and the laser illumination area 104. Figure 10 shows an example of fluorescence spectra (particle position spectra) acquired for each section within the microparticles.

[0157] In the above-described embodiment, a virtual filter was generated and the coefficient k was calculated using the combined spectrum obtained by summing the fluorescence of the entire microparticle. However, the illumination width of the laser that excites fluorescence is about 1 μm to 10 μm, which may be smaller than the size of the microparticle 102.

[0158] When fluorescence staining patterns within microparticles are localized, the ratio of fluorescence intensity from each dye differs depending on the location within the microparticle. Therefore, using a spectrum that corresponds to the location within the microparticle allows for more accurate color separation than using a combined spectrum of the entire microparticle.

[0159] In a flow cytometer, since minute particles flow at a constant velocity within the flow cell, the detection spectrum at each time point corresponds to the position (p) of the minute particles in the direction of flow. In other words, the particle position p and time t can be uniquely correlated depending on the flow velocity of the target.

[0160] Therefore, in this embodiment, instead of treating spectra U and V as a summation spectrum for the entire microparticle, they are treated as spectra for each particle position (particle position spectra), such as U(p) and V(p), depending on the particle position p.

[0161] The temporal resolution of the spectral detector in a flow cytometer is typically around 10 to 100 MHz. Therefore, the spatial resolution of the particle position spectrum can be set based on this temporal resolution and the target flow velocity. By determining the spectral detection interval based on the set spatial resolution, it becomes possible to calculate particle position spectra U(1), U(2), U(3), ..., U(n) corresponding to the position within the minute particle (see Figure 11).

[0162] The particle position spectrum and the pixel values ​​of the fluorescence image acquired by the image detector can be easily correlated. Therefore, in the image compensation described in the flowchart of Figure 5, different coefficients k can be used depending on the pixel number (position within the minute particle). Specifically, the fluorescence intensity of each pixel is corrected using a coefficient k calculated using the particle position spectrum corresponding to each pixel.

[0163] This method allows the localization of fluorescence distribution within minute particles to be reflected in color separation, thereby improving color separation accuracy and virtual image generation accuracy.

[0164] The above description of the fluorescence image information processing system according to this embodiment can be applied to other embodiments of this technology, unless there are any particular technical inconsistencies.

[0165] [5. Third Embodiment of the Technology (Example 3 of a Fluorescence Image Information Processing System)] In the above embodiment, image compensation using a leakage matrix was described, but under certain conditions, image correction can be performed in a simpler way.

[0166] In this embodiment, the image information acquisition unit acquires fluorescence image information in at least one wavelength band. The image information processing unit then calculates the ratio of each pigment in the fluorescence image information based on the multicolor fluorescence separation information, the pigment information of each pigment included in the reference information, and the transmission spectrum of the filter in the image information acquisition unit included in the reference information, and corrects the fluorescence image information based on this ratio and the pixel values ​​of the fluorescence image information.

[0167] For example, consider the case where the image information acquisition unit acquires fluorescence images in only one wavelength band for all excitation wavelengths, as in the 488nm excitation example in Figure 7 (where there is only one real filter). In this case, the measurement value at the virtual filter is estimated by multiplying the measurement value at the real filter by a coefficient k, so there is no need to create a leakage matrix.

[0168] Specifically, the ratio of dyes in each filter is calculated from the color separation results in the multi-wavelength fluorescence information processing unit. For example, the ratio j of dye D in filter F3 can be calculated using the following equation (13), where c, e, and f are the color separation results for dyes D, E, and F, respectively.

[0169] j=c·Σ{D*F3} / [c·Σ{D*F3}+e·Σ{E*F3}+f·{E*F3}] ···(13)

[0170] Here, Σ represents the sum over wavelength, * represents the Hadamard product (elemental product), D, E, and F represent the pigment information (spectral reference) of pigments D, E, and F respectively, and F3 represents the transmission spectrum of filter F3 included in the reference information.

[0171] For example, if the value of a pixel in an image acquired with an F3 filter is z, the corrected pixel value (fluorescence intensity corresponding to dye D) can be calculated as z·j.

[0172] This simplified method eliminates the need to calculate the leak matrix and its inverse, thereby reducing the computational load.

[0173] The above description of the fluorescence image information processing system according to this embodiment can be applied to other embodiments of this technology, unless there are any particular technical inconsistencies.

[0174] [6. A fourth embodiment of this technology (Example 4 of a fluorescence image information processing system)] This embodiment describes a method for removing the effects of autofluorescence. The multi-wavelength fluorescence information processing unit 132 in this embodiment treats autofluorescence as one of the dyes and removes the effects of autofluorescence by including the autofluorescence spectrum in the reference information.

[0175] [(1) System Configuration] The same system configuration as in the first embodiment is used.

[0176] [(2) Operation (Processing Flow)] Figure 12 is a flowchart showing an example of the processing flow of the fluorescence image information processing system 100 according to this embodiment. The operation of the fluorescence image information processing system 100 will be described in detail with reference to Figure 12.

[0177] [Step S201: Obtain SR of staining dye] This is the same as the first embodiment.

[0178] [Step S202: Acquisition of multi-wavelength fluorescence information of unstained cells] The multi-wavelength fluorescence information acquisition unit 120 acquires multi-wavelength fluorescence information from cells that do not contain staining dyes (unstained cells).

[0179] [Step S203: Autofluorescence component spectrum extraction, SR setting] The multi-wavelength fluorescence information processing unit 132 performs multivariate analysis, such as NMF (Non-negative Matrix Factorization), on the multi-wavelength fluorescence information of unstained cells and extracts autofluorescence component spectra. The extracted autofluorescence spectra, along with the SR of the staining dye included in the reference information, are set as SR information (part of the reference information) to be used in subsequent processing. Multiple different spectra may be extracted and set as autofluorescence components. Alternatively, autofluorescence spectrum information pre-entered by the user may be used.

[0180] Multivariate analysis is a general term for statistical methods used to simultaneously analyze data with multiple variables and clarify the relationships and structure between them, and NMF is just one example. In addition to NMF, multivariate analysis also includes methods such as principal component analysis (PCA), independent component analysis (ICA), factor analysis (FA), discriminant analysis (DA), canonical correlation analysis (CCA), cluster analysis, and multidimensional scaling (MDS).

[0181] [When using NMF] The multi-wavelength fluorescence information (matrix) of unstained cells is decomposed into the product of two non-negative matrices. Each column of the decomposed matrix corresponds to the autofluorescence component spectrum. As a specific algorithm, for example, a matrix decomposition is performed that minimizes the following loss function (14).

[0182] ||V-WH|| 2 ...(14)

[0183] Here, V is the multi-wavelength fluorescence information of unstained cells (m×n matrix, m is the number of wavelengths, n is the number of cells). W is the basis matrix (m×k matrix, k is the number of autofluorescent components). H is the coefficient matrix (k×n matrix). ||.|| is the Frobenius norm.

[0184] W and H are optimized by iterative calculations using the following equations (15) and (16). Initial values ​​are often set randomly. Multiplicative update algorithms are used as iterative algorithms.

[0185] H = H * (W T V) / (W T VH) ···(15)

[0186] W = W * (WH T ) / (WHH T ) ···(16)

[0187] Here, "*" represents element-wise multiplication, and " / " represents element-wise division.

[0188] [When using autofluorescence spectrum information pre-entered by the user] The user inputs separately acquired autofluorescence spectrum information (for example, autofluorescence spectrum information specific to a particular cell type) in the form of a text file or spreadsheet. The calculation unit 130 reads the input data and sets it as SR as part of the reference information.

[0189] [When using clustering methods that utilize cosine similarity] Autofluorescence spectra can also be extracted from multi-wavelength fluorescence information of unstained cells by applying clustering based on spectral similarity.

[0190] Specifically, first, the similarity between each spectrum is calculated for the numerous spectral data acquired by the multi-wavelength fluorescence information acquisition unit. As an index of similarity, for example, cosine similarity can be used. The cosine similarity of multi-wavelength fluorescence information A and B is expressed by the following equation (17).

[0191] Similarity=cos(θ)=(A·B) / (||A|| ||B||) ···(17)

[0192] Here, "·" represents the inner product, and ||A|| and ||B|| are the norms of the spectrum.

[0193] Next, based on cosine similarity, multi-wavelength fluorescence information is divided into groups using clustering methods such as hierarchical clustering, k-means clustering, and SOM (Self-Organizing Map).

[0194] Finally, the representative spectrum of each group (e.g., mean or median) is used as the autofluorescence spectrum, which is part of the reference information.

[0195] [Step S204: Image and spectrum acquisition] This is the same as the first embodiment.

[0196] [Step S205: Unmixing using SR (dye + autofluorescence)] The multi-wavelength fluorescence information processing unit 132 performs an unmixing process on the multi-wavelength fluorescence information using the SR of the staining dye and the SR including the autofluorescence spectrum included in the reference information. This separates the contributions of the staining dye and autofluorescence at each pixel, generating multi-color fluorescence separation information.

[0197] [Step S206: Comparison of the number of image filters and the number of staining dyes] This is the same as in the first embodiment. However, the "number of dyes" here refers to the number of staining dyes plus the number of autofluorescent components.

[0198] If the number of dyes > the number of image filters (step S206: YES), the process proceeds to step S207 (virtual filter generation). If the number of dyes ≤ the number of image filters (step S206: NO), the process proceeds to step S210 (image compensation).

[0199] [Step S207: Virtual filter generation] This is the same as the first embodiment.

[0200] [Step S208: Generate leakage matrix] This is the same as in the first embodiment, except that the leakage matrix is ​​generated based on reference information, and rows (or columns) related to autofluorescence are also added.

[0201] [Step S209: Simulation of measurement results using a virtual filter] The same applies as in the first embodiment. However, the calculation of the coefficient k also takes into account the autofluorescence spectrum included in the multi-wavelength fluorescence information and the reference information.

[0202] [Step S210: Image Compensation]

[0203] This is the same as in the first embodiment. However, an image corresponding to the autofluorescence component is also generated, and the effect of autofluorescence is removed using this image.

[0204] [Step S211: Output Result] This is the same as in the first embodiment. An image from which the autofluorescence component has been removed is output.

[0205] Furthermore, the spectral similarity index is not limited to cosine similarity; other known indices such as Euclidean distance, Manhattan distance, Mahalanobis distance, and correlation coefficient can also be used. Multiple indices can also be used in combination. The selection of the similarity index should preferably be made appropriately, taking into account the characteristics of the autofluorescence spectrum and the properties of multi-wavelength fluorescence information.

[0206] [(3) Effects] According to this embodiment, in addition to the effects of the above embodiment, the following effects can be obtained.

[0207] [(a) High-precision autofluorescence removal] By using the cell-specific autofluorescence spectrum as a spectral reference (SR), the influence of autofluorescence can be removed with high precision. This makes it possible to obtain fluorescence images in which only the target dye fluorescence is accurately extracted, even in situations where autofluorescence and dye fluorescence are present, which was difficult with conventional methods.

[0208] [(b) Improved detection sensitivity of weak fluorescence signals] Unmixing that takes autofluorescence spectra into account improves the signal-to-noise ratio when detecting weak fluorescent signals. This makes it possible to detect weak signals that were missed by conventional methods, resulting in more detailed intracellular information.

[0209] [(c) Support for diverse autofluorescence] [Extraction using NMF] By using NMF (Non-negative Matrix Factorization) on multi-wavelength fluorescence information from unstained cells, unknown autofluorescence spectra can be automatically extracted.

[0210] [Pre-enter] It is also possible to utilize autofluorescence spectrum information acquired by the user in advance, and it can handle autofluorescence specific to particular cell types.

[0211] [Clustering] By using a clustering method that employs cosine similarity or the like, even when multiple different autofluorescence spectra are present within a cell population, autofluorescence corresponding to each of these spectra can be removed.

[0212] [(d) Improvement of Quantitativeness] By eliminating the effect of autofluorescence, the quantitativeness of fluorescence intensity is improved. This allows for more accurate evaluation of comparisons of fluorescence intensity between cells, changes in fluorescence intensity over time, and the like.

[0213] [(e) Wide Range of Applications] In various fields such as cell biology, immunology, cancer research, and drug discovery research, high-precision fluorescence imaging unaffected by autofluorescence becomes possible, contributing to the advancement of research. The effect is particularly significant in the analysis of cells (e.g., macrophages, hepatocytes) and tissues with strong autofluorescence.

[0214] The above description of the fluorescence image information processing system according to the present embodiment can be applied to other embodiments of the present technology, as long as there is no particular technical contradiction.

[0215] [7. Fifth Embodiment of the Present Technology (Example 5 of Fluorescence Image Information Processing System)] Autofluorescent substances are present relatively uniformly throughout the entire cell, compared to staining dyes. Therefore, assuming that the shape of a cell is approximately spherical, the luminance distribution of the fluorescence image due to autofluorescence can be estimated by simulation. It is also possible to actually measure the luminance distribution of autofluorescence by imaging unstained cells in advance.

[0216] In the present embodiment, the image information processing unit performs autofluorescence correction using the luminance distribution pattern of autofluorescence estimated based on the outer shape of a cell, the ratio of autofluorescence calculated from multicolor fluorescence separation information, and the luminance value of the fluorescence image information acquired by the image information acquisition unit.

[0217] FIGS. 13 to 15 are schematic diagrams for explaining an autofluorescence correction method according to the present technology. FIG. 13 shows a fluorescence image (autofluorescence + staining dye fluorescence) of a cell stained with dye A. This image includes the luminance signal S of the fluorescence from dye A a as well as the luminance (background) signal S due to autofluorescence g .

[0218] The outer shape of a cell can be extracted using, for example, a membrane-stained fluorescence image or a bright-field image. If the outer shape of a cell can be extracted, it is possible to simulate what kind of autofluorescence distribution occurs for that outer shape. For example, assuming a cell is spherical, the luminance distribution pattern g(r) of autofluorescence can be modeled as a projection of the sphere's thickness onto a plane, as shown in FIG. 14, and can be expressed using the diameter D and area S as variables.

[0219] However, since the absolute luminance (area luminance) of autofluorescence differs from cell to cell, accurate correction is difficult by simply uniformly subtracting the simulated luminance distribution.

[0220] Therefore, in the present embodiment, the color separation result obtained by the multi-wavelength fluorescence information acquisition unit (spectral detector) is used to correct the luminance of autofluorescence. Specifically, when subtracting autofluorescence G from a cell stained with dye A, the ratio (G / (A+G)) of dye A to autofluorescence G in the total luminance of the entire cell is calculated from the multicolor fluorescence separation information from the spectral detector.

[0221] On the other hand, the image information acquisition unit (image detector) can calculate the total luminance (A+G) of the entire cell by summing the fluorescence intensity of each pixel.

[0222] Therefore, by multiplying the combined luminance (A+G) obtained from the image detector by the ratio G / (A+G) (calculated from the multicolor fluorescence separation information) obtained from the spectral detector, the luminance of the autofluorescence to be subtracted can be calculated. Then, based on this autofluorescence luminance and the simulated (or measured) autofluorescence luminance distribution pattern (g(r)), a simulated image of autofluorescence (Figure 14) is generated, and by subtracting this from the original image (autofluorescence + staining dye fluorescence) (Figure 13), an image with corrected autofluorescence (fluorescence image of dye A) (Figure 15) can be obtained.

[0223] The above description of the fluorescence image information processing system according to this embodiment can be applied to other embodiments of this technology, unless there are any particular technical inconsistencies.

[0224] [8. Sixth Embodiment of the Technology (Example 6 of a Fluorescence Image Information Processing System)] Autofluorescence generally has a weaker signal intensity compared to fluorescent dyes. Therefore, when an autofluorescence spectrum is added as a spectral reference (SR), color separation of the autofluorescence spectrum may not be performed properly, especially if the number of detection channels (filters) in the image information acquisition unit is small. This problem becomes particularly pronounced when multiple autofluorescence spectra are used as SRs, especially if each cell type has a different autofluorescence spectrum.

[0225] On the other hand, the multi-wavelength fluorescence information acquisition unit (spectral detector) has numerous detection channels, allowing it to acquire the entire spectral shape with high accuracy. Therefore, in color separation using multi-wavelength fluorescence information (spectral unmixing), it is expected that autofluorescence can also be appropriately separated.

[0226] Therefore, in this embodiment, the image information processing unit substitutes the autofluorescence intensity coefficient (part of the multicolor fluorescence separation information) calculated using multiwavelength fluorescence information and reference information into the color separation of the fluorescence image information acquired by the image information acquisition unit, and performs color separation for dyes other than autofluorescence.

[0227] Specifically, as shown in Figures 16 and 17, first, color separation is performed using multi-wavelength fluorescence information and reference information to calculate the autofluorescence signal intensity (part of the multi-color fluorescence separation information). Figure 16 is a graph showing an example of multi-wavelength fluorescence information. Figure 17 is a graph showing the intensity coefficients of each component. These figures are intended to explain the color separation method using autofluorescence spectra for each cell type in this technology. Then, this autofluorescence signal intensity is substituted as the color separation result of the image data, and subsequently, color separation of the remaining dyes (fluorescent dyes other than autofluorescence) is performed.

[0228] Specifically, the process will be carried out using the following steps.

[0229] [Preparing spectral references] As shown in Figure 16, autofluorescence spectra (AF1, AF2, AF3, ...) are prepared for each cell type. Figure 16A shows the autofluorescence spectrum AF1 of the first cell tumor. Figure 16B shows the autofluorescence spectrum AF2 of the second cell tumor. Figure 16C shows the autofluorescence spectrum AF3 of the third cell tumor. In addition, spectral references for each dye are also prepared.

[0230] [Color separation of multi-wavelength fluorescence information] Using multi-wavelength fluorescence information and the spectral reference included in the above reference information, spectral unmixing is performed to calculate the intensity coefficient (multicolor fluorescence separation information) of each component (autofluorescence of each cell type and each dye) (bar graphs corresponding to each filter shown in Figure 17).

[0231] [Substitution of autofluorescence intensity] When performing color separation of image data, the calculated autofluorescence intensity coefficient (using the following formula (18) Substitute d, e, and f) in the corresponding elements.

[0232]

number

[0233] [Color separation of remaining pigments] With the intensity coefficient of autofluorescence substituted, compensation or spectral unmixing is performed for the remaining dyes (a, b, c in the example of the above formula (18)). In this case, a constrained least-squares method in which an absolute value constraint (non-negative value constraint) is applied to part of the solutions (the intensity coefficient of autofluorescence) is solved.

[0234] This method enables highly accurate color separation of image data even when an autofluorescence spectrum is included.

[0235] The above description of the fluorescence image information processing system according to the present embodiment can be applied to other embodiments of the present technology, as long as no particular technical contradiction arises.

[0236] [9. Seventh Embodiment of Present Technology (Example 7 of Fluorescence Image Information Processing System)] In the method described in the sixth embodiment, the autofluorescence intensity obtained from the multi-wavelength fluorescence information is substituted in the color separation of image data, while in the present embodiment, autofluorescence spectra that differ for each cell type are further taken into consideration. The multi-wavelength fluorescence information processing unit 132 according to the present embodiment acquires in advance autofluorescence spectrum information specific to a cell type as part of reference information, and when the cell type can be identified, the background is removed using that autofluorescence spectrum information.

[0237] When a plurality of autofluorescence spectra that differ for each cell type are added as spectral references (SR), it is assumed that the autofluorescence intensity specific to any one of the cell types will be dominant as the color separation result for each individual cell that is a detection target.

[0238] Since the multi-wavelength fluorescence information acquisition unit (spectral detector) has a large number of detection channels, it is expected that color separation (spectral unmixing) using multi-wavelength fluorescence information can appropriately calculate the respective signal intensities even for a plurality of autofluorescence components.

[0239] Therefore, in this embodiment, the background (BG) due to autofluorescence is removed from the image data by one of the following methods. Figure 18 is a graph showing an example of multi-wavelength fluorescence information. Figure 19 is a graph showing the intensity coefficients of each component. Figures 20 and 21 are flowcharts showing an example of the processing flow of the color separation method of this technology.

[0240] [(1) Method 1: Method for selecting the dominant autofluorescence signal] The method for selecting the dominant autofluorescence signal will be explained with reference to Figure 20.

[0241] [Step S301: Preparing for SR] As shown in Figure 18, autofluorescence spectra (AF1, AF2, AF3, ...) for each cell type and spectral references (SRs) for each dye are prepared. Figure 18A shows the autofluorescence spectrum AF1 of the first cell tumor. Figure 18B shows the autofluorescence spectrum AF2 of the second cell tumor. Figure 18C shows the autofluorescence spectrum AF3 of the third cell tumor.

[0242] [Step S302: Color separation of multi-wavelength fluorescence information] Using multi-wavelength fluorescence information and the spectral reference included in the above reference information, spectral unmixing is performed to calculate the intensity coefficient (part of the multi-color fluorescence separation information) for each component (autofluorescence of each cell type and each dye).

[0243] [Step S303: Selection of the dominant autofluorescence signal] In the multicolor fluorescence separation information for each cell, the autofluorescence component with the highest intensity (e.g., AF1 of cell type 1) is selected.

[0244] [Step S304: Calculation of autofluorescence intensity] The autofluorescence intensity in each wavelength band (F1, F2, F3) of the image detector is calculated from the autofluorescence spectrum (e.g., AF1) included in the selected reference information and its intensity coefficient (multicolor fluorescence separation information) (bar graphs corresponding to each filter shown in Figure 19).

[0245] [Step S305: BG Removal] The autofluorescence intensity (background (BG)) is subtracted from the pixel value of the image data.

[0246] [Step S306: Color separation of the remaining pigments] Compensation or spectral unmixing is performed on the image data after background green removal, and the fluorescence intensity of the remaining dyes (fluorescent dyes) is calculated.

[0247] [(2) Method 2: Method using aggregated autofluorescence components] The method using the aggregated autofluorescence component will be explained with reference to Figure 21.

[0248] [Step S401: Preparing for SR] This is the same as Method 1.

[0249] [Step S402: Color separation of multi-wavelength fluorescence information] This is the same as Method 1.

[0250] [Step S403: Calculation of autofluorescence intensity] The sum of autofluorescence in each wavelength band (F1, F2, F3) of the image detector is calculated from the intensities of each autofluorescence component (AF1, AF2, AF3, ...) in the multicolor fluorescence separation information and the respective spectral references included in the reference information (bar graphs corresponding to each filter shown in Figure 19. All components corresponding to each cell type are added together).

[0251] [Step S404: BG removal] The autofluorescence intensity (background (BG)) is subtracted from the pixel value of the image data.

[0252] [Step S405: Color separation of the remaining pigments] This is the same as Method 1.

[0253] By using one of the methods described above, it becomes possible to more precisely remove the effects of autofluorescence from image data and perform high-precision color separation, even when different autofluorescence spectra exist for each cell type.

[0254] The above description of the fluorescence image information processing system according to this embodiment can be applied to other embodiments of this technology, unless there are any particular technical inconsistencies.

[0255] [10. Eighth Embodiment of the Technology (Example 8 of a Fluorescence Image Information Processing System)] In this embodiment, the fluorescence image information processing system of the first embodiment (or other embodiments) is extended to add a function for acquiring a three-dimensional image (Z-stack image).

[0256] [(1) System Configuration] The fluorescence image information processing system of this embodiment differs from the system configuration of the first embodiment mainly in that the image information acquisition unit 110 is a multi-line sensor type and also includes an optical path branching unit.

[0257] In this embodiment, the image information acquisition unit 110 includes a plurality of line sensors having different focal positions in the optical axis direction. Each line sensor is focused on a different Z-axis position (depth position) within the flow cell. The Z-axis refers to the direction parallel to the optical axis.

[0258] An optical path splitter (e.g., a beam splitter, a dichroic mirror) is placed between the excitation light irradiation system and the image information acquisition unit. The optical path splitter divides the fluorescence from the minute particles into N optical paths and guides each to a different line sensor.

[0259] The multi-wavelength fluorescence information acquisition unit 120 may be configured to include multiple detectors, similar to the image information acquisition unit 110. Each detector can be a multi-channel detector that focuses on different Z-axis positions within the flow cell and detects fluorescence in different wavelength bands (for example, by placing different bandpass filters in front of each channel, or by placing a spectroscopic optical system such as a diffraction grating).

[0260] [(2) Operation (Processing Flow)] The operation of the fluorescence image information processing system 100 will be described in detail with reference to Figure 22. Figure 22 is a flowchart showing an example of the processing flow of the fluorescence image information processing system 100 according to this embodiment.

[0261] [Step S501: Acquire fluorescence image information] Each line sensor in the image information acquisition unit 110 simultaneously acquires fluorescence from minute particles flowing within the flow cell at their respective focal positions. This allows fluorescence image information from different Z-axis positions of the minute particles to be acquired at once.

[0262] [Step S502: Acquisition of multi-wavelength fluorescence information] Similarly, the multi-wavelength fluorescence information acquisition unit 120 acquires fluorescence information at different Z-axis positions and / or different wavelength bands of the minute particles.

[0263] [Step S503: Image Processing] The calculation unit 130 uses the fluorescence image information and multi-wavelength fluorescence information obtained from each line sensor to perform the color separation process (compensation or spectral unmixing) described in the first embodiment. Furthermore, it integrates image information from different Z-axis positions to reconstruct the three-dimensional fluorescence image information of the minute particles. As a reconstruction algorithm, for example, a deconvolution process that takes into account the focal position and PSF (point spread function) of each line sensor can be used.

[0264] [(3) Effects] According to this embodiment, the following effects can be obtained.

[0265] [(a) Visualization of the three-dimensional structure of cells] Because fluorescence image information from multiple different focal positions along the Z-axis can be simultaneously acquired while minute particles pass through the flow cell, high-speed 3D fluorescence imaging becomes possible. This allows for detailed analysis of the 3D structure of cells and the distribution of intracellular organelles, even in high-speed flow cytometry.

[0266] [(b) Combination with multiplex fluorescence imaging] By combining this with the color separation technology of the first embodiment, the three-dimensional structure of cells labeled with multiple fluorescent dyes can be observed with high precision.

[0267] [(c) Quantitative 3D analysis] From the acquired 3D fluorescence image information, quantitative information such as the total volume of the cell, surface area, volume ratio of specific structures, and 3D distribution of fluorescence intensity can be extracted.

[0268] [(d) Wide range of applications] It can be used for three-dimensional structural analysis of cells and tissues in various fields, including cell biology, developmental biology, histology, and pathology.

[0269] The above description of the fluorescence image information processing system according to this embodiment can be applied to other embodiments of this technology, unless there are any particular technical inconsistencies.

[0270] [11. Ninth Embodiment of the Technology (Example of Fluorescence Image Information Processing Method)] This technology provides a fluorescence image information processing method that includes: acquiring fluorescence image information of minute particles; acquiring multi-wavelength fluorescence information of the minute particles; performing multi-wavelength fluorescence information processing, including spectral unmixing, based on the multi-wavelength fluorescence information and reference information to generate multi-color fluorescence separation information; and correcting or supplementing the fluorescence image information based on the fluorescence image information and the multi-color fluorescence separation information when the number of channels used to acquire the fluorescence image information is less than the number of channels required to discriminate the fluorescence of the minute particles.

[0271] The fluorescence image information processing method according to this embodiment will be described with reference to Figure 23 and, if necessary, other drawings. Figure 23 is a flowchart showing an example of the flow of a fluorescence image information processing method according to one embodiment of this technology. As shown in Figure 23, the fluorescence image information processing method of this embodiment includes the following steps.

[0272] [Step S1: Acquire fluorescence image information] The image information acquisition unit 110 (Figures 2 and 3) acquires fluorescence image information of the minute particles 102. In this step, multiple detectors (e.g., line sensors) that detect fluorescence in multiple different wavelength bands are used to detect fluorescence from the minute particles 102 in a line-like manner and generate two-dimensional fluorescence image information.

[0273] [Step S2: Acquisition of multi-wavelength fluorescence information] The multi-wavelength fluorescence information acquisition unit 120 (Figures 2 and 3) acquires multi-wavelength fluorescence information (fluorescence spectrum information) of the fine particles 102. In this step, for example, wavelength-resolved fluorescence intensity data is acquired using a diffraction grating 126 and a multi-channel detector 128.

[0274] [Step S3: Multi-wavelength fluorescence information processing] The multi-wavelength fluorescence information processing unit 132 (see Figure 1) performs multi-wavelength fluorescence information processing, including spectral unmixing, based on the multi-wavelength fluorescence information and reference information to generate multi-color fluorescence separation information. The multi-color fluorescence separation information includes, for example, the relative abundance (or concentration) and intensity coefficient of each dye obtained by spectral unmixing.

[0275] [Step S4: Correction or Complementation] The image information processing unit 131 (see Figure 1) corrects or complements the fluorescence image information based on the fluorescence image information acquired by the image information acquisition unit 110 and the multi-color fluorescence separation information generated by the multi-wavelength fluorescence information processing unit 132. This step is performed when the number of channels used by the image information acquisition unit (e.g., the number of filters) is less than the number of channels required to discriminate the fluorescence of the minute particles (e.g., the number of staining dyes + the number of autofluorescent components).

[0276] Steps S1 to S4 above (and any processing performed as necessary) correct the crosstalk between dyes and obtain fluorescence images corresponding to each dye (and autofluorescence), or images corresponding to the virtual filter.

[0277] The above description of the fluorescence image information processing method according to this embodiment can be applied to other embodiments of this technology, unless there are any particular technical inconsistencies.

[0278] [12. Tenth Embodiment of the Technology (Program and Recording Medium)] This technology can also be realized as a program for causing a computer to execute the above-described fluorescence image information processing method, and as a computer-readable recording medium on which the program is stored.

[0279] The program according to this embodiment includes instructions for causing the computer to perform each of the steps shown in Figure 23.

[0280] The program according to this embodiment may be provided by being recorded on a computer-readable recording medium. Examples of recording media include, but are not limited to, magnetic recording media (e.g., flexible disks, hard disk drives (HDDs)), optical discs (e.g., CD-ROMs, DVDs, Blu-ray® Discs), magneto-optical recording media (e.g., MOs), and semiconductor memory (e.g., USB memory, SSDs).

[0281] The program according to this embodiment is installed and executed on a general-purpose computer, a dedicated computer, or the calculation unit 130 (Figure 1) of the fluorescence image information processing system 100. The program may be downloaded via a network or read from a recording medium.

[0282] By having a computer execute the program according to this embodiment, each function of the fluorescence image information processing system according to the above-described embodiment, in particular each function of the calculation unit 130, can be realized on the computer.

[0283] Although the above embodiment was described using a flow cytometer as an example, this technology can also be applied to fluorescence image analysis of stationary cells.

[0284] While the above embodiments illustrate specific optical systems, detectors, and algorithms, this technology is not limited to these. For example, different optical systems, detectors with different wavelength resolutions, and different unmixing algorithms may be used.

[0285] Furthermore, the embodiments relating to this technology are not limited to the embodiments described above, and various modifications are possible without departing from the gist of this technology. The specific numerical values, shapes, materials (including composition), etc. described in each embodiment are examples only and are not limited to these.

[0286] Furthermore, this technology can also be configured as follows. (1) An image information acquisition unit that acquires fluorescence image information of minute particles, A multi-wavelength fluorescence information acquisition unit that acquires multi-wavelength fluorescence information of the aforementioned fine particles, A multi-wavelength fluorescence information processing unit that performs multi-wavelength fluorescence information processing, including spectral unmixing, based on the multi-wavelength fluorescence information and reference information, and generates multi-color fluorescence separation information. A fluorescence image information processing system comprising: an image information processing unit that corrects or supplements the fluorescence image information based on the fluorescence image information and the multicolor fluorescence separation information when the number of channels used in the image information acquisition unit is less than the number of channels required to discriminate the fluorescence of the minute particles. (2) The image information processing unit generates a virtual detection filter in the image information acquisition unit based on the dye information and device channel information included in the reference information. The fluorescence image information processing system described in (1) above. (3) The image information processing unit generates a leak matrix based on the reference information and the virtual detection filter. The fluorescence image information processing system described in (1) or (2) above. (4) The multi-wavelength fluorescence information processing unit calculates a coefficient for converting the multi-wavelength fluorescence information acquired by the multi-wavelength fluorescence information acquisition unit into a measurement value at the virtual detection filter. A fluorescence image information processing system according to any one of (1) to (3) above. (5) The image information processing unit uses the leakage matrix and the coefficients to perform a color separation operation on the fluorescence image information and generates an image corresponding to each dye. A fluorescence image information processing system according to any one of (2) to (4) above. (6) The multi-wavelength fluorescence information processing unit calculates the coefficient by dividing the multi-wavelength fluorescence information acquired by the multi-wavelength fluorescence information acquisition unit into sections. The fluorescence image information processing system described in (4) or (5) above. (7) The image information processing unit generates the virtual detection filter based on user filter settings or automatic settings based on dye information. A fluorescence image information processing system according to any one of (2) to (6) above. (8) The image information acquisition unit acquires fluorescence image information in at least one wavelength band, The image information processing unit calculates the ratio of each pigment in the fluorescence image information based on the multicolor fluorescence separation information, the pigment information of each pigment included in the reference information, and the transmission spectrum of the filter in the image information acquisition unit included in the reference information, and corrects the fluorescence image information based on the ratio and the pixel values ​​of the fluorescence image information. A fluorescence image information processing system according to any one of (1) to (7) above. (9) The image information processing unit calculates the corrected pixel value as z·j, where z is the pixel value of the fluorescence image information acquired by the image information acquisition unit and j is the ratio. The fluorescence image information processing system described in (8) above. (10) The multi-wavelength fluorescence information processing unit treats autofluorescence as one of the dyes and includes the autofluorescence spectrum in the reference information to remove the influence of autofluorescence. A fluorescence image information processing system according to any one of (1) to (9) above. (11) The multi-wavelength fluorescence information processing unit extracts an autofluorescence spectrum using non-negative matrix factorization or autofluorescence spectrum information previously entered by the user. The fluorescence image information processing system described in (10) above. (12) The multi-wavelength fluorescence information processing unit acquires autofluorescence spectral information specific to the cell type in advance, and when the cell type can be identified, it uses that autofluorescence spectral information to remove the background. The fluorescence image information processing system described in (10) or (11) above. (13) The image information processing unit performs autofluorescence correction using the autofluorescence luminance distribution pattern estimated based on the cell's external shape, the autofluorescence ratio calculated from the multicolor fluorescence separation information, and the luminance value of the fluorescence image information acquired by the image information acquisition unit. A fluorescence image information processing system according to any one of (10) to (12) above. (14) The image information processing unit substitutes the autofluorescence intensity coefficient calculated using the multi-wavelength fluorescence information and the reference information into the color separation of the fluorescence image information acquired by the image information acquisition unit, and performs color separation for dyes other than autofluorescence. A fluorescence image information processing system according to any of (10) to (13) above. (15) The multi-wavelength fluorescence information acquisition unit acquires multi-wavelength fluorescence information of the minute particles using an optical system that employs a diffraction grating and a multi-channel detector. A fluorescence image information processing system according to any one of (1) to (14) above. (16) The multi-wavelength fluorescence information acquisition unit acquires multi-wavelength fluorescence information of the minute particles using an optical system for separating wavelengths and a plurality of detectors provided for each wavelength. A fluorescence image information processing system according to any of (1) to (15) above. (17) The image information acquisition unit has multiple detectors that detect fluorescence in multiple different wavelength regions, and each detector includes a line sensor that detects fluorescence from the minute particles passing through the flow cell in a line. A fluorescence image information processing system according to any one of (1) to (16) above. (18) The multi-wavelength fluorescence information acquisition unit acquires multi-wavelength fluorescence information from the minute particles moving within the flow cell at a speed of 0.1 to 10 m / sec. A fluorescence image information processing system according to any one of (1) to (17) above. (19) To acquire fluorescence image information of minute particles, To acquire multi-wavelength fluorescence information of the aforementioned fine particles, Based on the aforementioned multi-wavelength fluorescence information and reference information, multi-wavelength fluorescence information processing including spectral unmixing is performed to generate multi-color fluorescence separation information. A fluorescence image information processing method, comprising correcting or supplementing the fluorescence image information based on the fluorescence image information and the multicolor fluorescence separation information when the number of channels used to acquire the fluorescence image information is less than the number of channels required to discriminate the fluorescence of the minute particles. (20) Performing the aforementioned correction or supplementation includes generating a virtual detection filter based on the dye information and device channel information included in the reference information. The fluorescence image information processing method described in (19) above. (twenty one) The aforementioned correction or supplementation includes treating autofluorescence as one of the dyes and including the autofluorescence spectrum in the reference information to remove the effect of autofluorescence. The fluorescence image information processing method described in (19) or (20) above. (twenty two) On the computer, To acquire fluorescence image information of minute particles, To acquire multi-wavelength fluorescence information of the aforementioned fine particles, Based on the aforementioned multi-wavelength fluorescence information and reference information, multi-wavelength fluorescence information processing including spectral unmixing is performed to generate multi-color fluorescence separation information. When the number of channels used to acquire the fluorescence image information is less than the number of channels required to discriminate the fluorescence of the minute particles, the fluorescence image information is corrected or supplemented based on the fluorescence image information and the multicolor fluorescence separation information. A program to execute. (twenty three) A computer-readable recording medium on which the program described in (22) above is recorded. [Explanation of Symbols]

[0287] 100 Fluorescence Image Information Processing System 101 Flow Cell 102 Microparticles 110 Image information acquisition unit 120 Multi-wavelength fluorescence information acquisition unit 130 Arithmetic section 131 Image Information Processing Unit 132 Multi-wavelength fluorescence information processing unit S1 Acquire fluorescence image information. S2 Acquisition of multi-wavelength fluorescence information S3 Correction or enhancement of fluorescence image information. S4 Perform multi-wavelength fluorescence information processing.

Claims

1. An image information acquisition unit that acquires fluorescence image information of minute particles, A multi-wavelength fluorescence information acquisition unit that acquires multi-wavelength fluorescence information of the aforementioned fine particles, A multi-wavelength fluorescence information processing unit that performs multi-wavelength fluorescence information processing, including spectral unmixing, based on the multi-wavelength fluorescence information and reference information, and generates multi-color fluorescence separation information. A fluorescence image information processing system comprising: an image information processing unit that corrects or supplements the fluorescence image information based on the fluorescence image information and the multicolor fluorescence separation information when the number of channels used in the image information acquisition unit is less than the number of channels required to discriminate the fluorescence of the minute particles.

2. The image information processing unit generates a virtual detection filter in the image information acquisition unit based on the dye information and device channel information included in the reference information. The fluorescence image information processing system according to claim 1.

3. The image information processing unit generates a leak matrix based on the reference information and the virtual detection filter. The fluorescence image information processing system according to claim 2.

4. The multi-wavelength fluorescence information processing unit calculates a coefficient for converting the multi-wavelength fluorescence information acquired by the multi-wavelength fluorescence information acquisition unit into a measurement value at the virtual detection filter. The fluorescence image information processing system according to claim 3.

5. The image information processing unit uses the leakage matrix and the coefficients to perform a color separation operation on the fluorescence image information and generates an image corresponding to each dye. The fluorescence image information processing system according to claim 4.

6. The multi-wavelength fluorescence information processing unit calculates the coefficient by dividing the multi-wavelength fluorescence information acquired by the multi-wavelength fluorescence information acquisition unit into sections. The fluorescence image information processing system according to claim 4.

7. The image information processing unit generates the virtual detection filter based on user filter settings or automatic settings based on dye information. The fluorescence image information processing system according to claim 2.

8. The image information acquisition unit acquires fluorescence image information in at least one wavelength band, The image information processing unit calculates the ratio of each pigment in the fluorescence image information based on the multicolor fluorescence separation information, the pigment information of each pigment included in the reference information, and the transmission spectrum of the filter in the image information acquisition unit included in the reference information, and corrects the fluorescence image information based on the ratio and the pixel values ​​of the fluorescence image information. The fluorescence image information processing system according to claim 1.

9. The image information processing unit calculates the corrected pixel value as z・j, where z is the pixel value of the fluorescence image information acquired by the image information acquisition unit and j is the ratio. The fluorescence image information processing system according to claim 8.

10. The multi-wavelength fluorescence information processing unit treats autofluorescence as one of the dyes and includes the autofluorescence spectrum in the reference information to remove the influence of autofluorescence. The fluorescence image information processing system according to claim 1.

11. The multi-wavelength fluorescence information processing unit extracts autofluorescence spectra using multivariate analysis or autofluorescence spectrum information previously entered by the user. The fluorescence image information processing system according to claim 10.

12. The multi-wavelength fluorescence information processing unit acquires autofluorescence spectral information specific to the cell type in advance, and when the cell type can be identified, it uses that autofluorescence spectral information to remove the background. The fluorescence image information processing system according to claim 10.

13. The image information processing unit performs autofluorescence correction using the autofluorescence luminance distribution pattern estimated based on the cell's external shape, the autofluorescence ratio calculated from the multicolor fluorescence separation information, and the luminance value of the fluorescence image information acquired by the image information acquisition unit. The fluorescence image information processing system according to claim 10.

14. The image information processing unit substitutes the autofluorescence intensity coefficient calculated using the multi-wavelength fluorescence information and the reference information into the color separation of the fluorescence image information acquired by the image information acquisition unit, and performs color separation for dyes other than autofluorescence. The fluorescence image information processing system according to claim 10.

15. The multi-wavelength fluorescence information acquisition unit acquires multi-wavelength fluorescence information of the minute particles using an optical system that employs a diffraction grating and a multi-channel detector. The fluorescence image information processing system according to claim 1.

16. The multi-wavelength fluorescence information acquisition unit acquires multi-wavelength fluorescence information of the minute particles using an optical system for separating wavelengths and a plurality of detectors provided for each wavelength. The fluorescence image information processing system according to claim 1.

17. The image information acquisition unit has multiple detectors that detect fluorescence in multiple different wavelength regions, and each detector includes a line sensor that detects fluorescence from the minute particles passing through the flow cell in a line. The fluorescence image information processing system according to claim 1.

18. The multi-wavelength fluorescence information acquisition unit acquires multi-wavelength fluorescence information from the minute particles moving within the flow cell at a speed of 0.1 to 10 m / sec. The fluorescence image information processing system according to claim 1.

19. To acquire fluorescence image information of minute particles, To acquire multi-wavelength fluorescence information of the aforementioned fine particles, Based on the aforementioned multi-wavelength fluorescence information and reference information, multi-wavelength fluorescence information processing including spectral unmixing is performed to generate multi-color fluorescence separation information. A fluorescence image information processing method, comprising correcting or supplementing the fluorescence image information based on the fluorescence image information and the multicolor fluorescence separation information when the number of channels used to acquire the fluorescence image information is less than the number of channels required to discriminate the fluorescence of the minute particles.

20. On the computer, To acquire fluorescence image information of minute particles, To acquire multi-wavelength fluorescence information of the aforementioned fine particles, Based on the aforementioned multi-wavelength fluorescence information and reference information, multi-wavelength fluorescence information processing including spectral unmixing is performed to generate multi-color fluorescence separation information. When the number of channels used to acquire the fluorescence image information is less than the number of channels required to discriminate the fluorescence of the minute particles, the fluorescence image information is corrected or supplemented based on the fluorescence image information and the multicolor fluorescence separation information. A program to execute.

Images