Particle observation device and observation method

Abstract

To provide a device capable of more accurately observing a particle relatively moving.SOLUTION: A particle observation device 1A is a device for observing a fine particle P and includes a light source 10, an image-formation optical system 20A, an imaging part 30, an analysis part 40, a display part 50, etc. Both or one of the image-formation system 20A and the imaging part 30 changes a relative positional relation to the particle P during an exposure period of an image pick-up device 31, and moves a formation position of an image of the particle P on an imaging surface of the image pick-up device 31. The analysis part 40 extracts a linear image in the image, and analyzes the particle P according to the extracted linear image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an apparatus and method for observing particles. [Background technology]

[0002] For example, in devices for observing microparticles with sizes ranging from tens to hundreds of nanometers, imaging elements are used that can capture two-dimensional images containing particle images with high sensitivity by receiving weak light from the particles. Examples of such devices include flow cytometers that observe microparticles such as DNA or RNA particles or vesicles containing DNA or RNA, and particle counters that count microscopic particles floating in clean rooms. The imaging elements preferably used here are electron multiplying or electron bombardment types, such as multichannel photomultiplier tubes, EM-CCD sensors, EM-CMOS sensors, EB-CCD sensors, EB-CMOS sensors, and SPAD sensors.

[0003] Images captured using an imaging element may contain not only images of the particles being imaged (true images) but also spot-like noise images (false images). Spot noise images can be caused by the incidence of cosmic rays or by the imaging element itself. Distinguishing between particle images and spot noise images is possible if it is possible to use the size, brightness, or shape of the image. However, this is difficult when the particles are small and the light from them is weak. If it is not possible to distinguish between the two, particle images cannot be extracted from the image, making it impossible to accurately observe particles. For example, the number of particles will be overestimated, and the accuracy of measuring particle shape and size will be reduced.

[0004] Patent Document 1 discloses an invention for acquiring an image in which the influence of spot noise images is reduced. The invention disclosed in this document reduces the influence of spot noise images by acquiring multiple images and averaging these multiple images. This takes advantage of the fact that, while the image of the object to be imaged appears in the same position in all multiple images, spot noise images rarely appear in the same position in multiple images. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2002-286843 [Non-patent literature]

[0006] [Non-Patent Document 1] Kota Tajima, Yuma Mori, and Hiroshi Masuda, "Linear Object Detection Using Point Clouds and Images by Moving Measurement (2nd Report)," Proceedings of the JSPE Annual Conference Academic Lecture Series A68 (2019). Summary of the Invention [Problem to be solved by the invention]

[0007] Although the invention disclosed in Patent Document 1 may be applicable when the object to be imaged is stationary, it is difficult to apply it when the object to be imaged is moving. For example, when particles to be imaged are moving as in the example of the flow cytometer described above, or when minute debris to be imaged is floating in the air as in the example of the particle counter, particle images appear at different positions in multiple images, making it difficult to apply the invention disclosed in this document.

[0008] The present invention has been made to solve the above problems, and has as its object to provide an apparatus and method that can more accurately observe particles that are moving relatively. [Means for solving the problem]

[0009] A first aspect of the particle observation device of the present invention comprises: (1) an imaging optical system that inputs light from particles and forms an image of the particles; (2) an imaging unit that includes an imaging element having an imaging surface at the position where the image is formed by the imaging optical system and outputs image data based on an image formed on the imaging surface during an exposure period of the imaging element; and (3) an analysis unit that inputs the image data output from the imaging unit and analyzes the particles based on the image represented by this image data. Both or one of the imaging optical system and the imaging unit change their relative positional relationship with the particles during the exposure period, thereby moving the position where the particle image is formed on the imaging surface. The analysis unit extracts a linear image from the image and analyzes the particles based on the extracted linear image.

[0010] In a second aspect of the particle observation device of the present invention, in addition to the first aspect, the analysis unit extracts a linear image after reducing noise in the image.

[0011] In a third aspect of the particle observation device of the present invention, in addition to the first or second aspect, the analysis unit applies a machine learning model to extract a linear image in the image.

[0012] In a fourth aspect of the particle observation device of the present invention, in addition to the third aspect, the analysis unit applies any one of a principal component analysis model, an independent component analysis model, a non-negative factorization model, and a deep learning model as a machine learning model to extract linear images in an image.

[0013] In a fifth aspect of the particle observation device of the present invention, in addition to the third or fourth aspect, the analysis unit defines the direction in which the linear image in the image extends as a first direction, defines the direction perpendicular to this first direction as a second direction, and applies a machine learning model to each of the first direction and the second direction to extract the linear image in the image.

[0014] In a sixth aspect of the particle observation device of the present invention, in addition to any one of the first to fifth aspects, the analysis section determines the number of linear images in the image as the number of particles.

[0015] In a seventh aspect of the particle observation device of the present invention, in addition to any one of the first to sixth aspects, the analysis section obtains an image of the particle based on the luminance value distribution of the linear image in the image.

[0016] In an eighth aspect of the particle observation device of the present invention, in addition to any one of the first to seventh aspects, the analysis section determines the size of the particle based on the brightness value of the linear image in the image.

[0017] In a ninth aspect of the particle observation apparatus of the present invention, in addition to any one of the first to eighth aspects, the imaging section includes an electron multiplying type or electron bombardment type imaging element.

[0018] A tenth aspect of the particle observation apparatus of the present invention is, in addition to any one of the first to ninth aspects, further provided with a light source that outputs light to irradiate the particles.

[0019] A first aspect of the particle observation method of the present invention includes: (1) an imaging step using an imaging optical system that inputs light from a particle and forms an image of the particle, and an imaging unit including an imaging element having an imaging surface at the position where the image is formed by the imaging optical system, and outputting image data from the imaging unit based on an image formed on the imaging surface during an exposure period of the imaging element; and (2) an analysis step inputting the image data output from the imaging unit and analyzing the particle based on the image represented by this image data. In the imaging step, the relative positional relationship between the particle and either or both of the imaging optical system and the imaging unit is changed during the exposure period to move the position where the particle image is formed on the imaging surface. In the analysis step, a linear image is extracted from the image, and the particle is analyzed based on the extracted linear image.

[0020] In a second aspect of the particle observation method of the present invention, in addition to the first aspect, in the analyzing step, noise in the image is reduced and then a linear image is extracted.

[0021] In a third aspect of the particle observation method of the present invention, in addition to the first or second aspect, a machine learning model is applied to extract linear images in the image in the analyzing step.

[0022] In a fourth aspect of the particle observation method of the present invention, in addition to the third aspect, in the analysis step, a linear image in the image is extracted by applying one of a principal component analysis model, an independent component analysis model, a non-negative factorization model, and a deep learning model as a machine learning model.

[0023] In a fifth aspect of the particle observation method of the present invention, in addition to the third or fourth aspect, in the analysis step, the direction in which the linear image in the image extends is defined as a first direction, and the direction perpendicular to this first direction is defined as a second direction, and a machine learning model is applied to each of the first direction and the second direction to extract the linear image in the image.

[0024] In a sixth aspect of the particle observation method of the present invention, in addition to any one of the first to fifth aspects, in the analyzing step, the number of linear images in the image is found as the number of particles.

[0025] In a seventh aspect of the particle observation method of the present invention, in addition to any one of the first to sixth aspects, in the analyzing step, an image of the particle is obtained based on the luminance value distribution of the linear image in the image.

[0026] In an eighth aspect of the particle observation method of the present invention, in addition to any one of the first to seventh aspects, in the analyzing step, the size of the particle is determined based on the brightness value of the linear image in the image.

[0027] In a ninth aspect of the particle observation method of the present invention, in addition to any one of the first to eighth aspects, an imaging unit including an electron multiplying or electron bombardment type imaging element is used in the imaging step.

[0028] In a tenth aspect of the particle observation method of the present invention, in addition to any one of the first to ninth aspects, a light source that outputs light to irradiate the particles is further used in the imaging step. [Effects of the Invention]

[0029] According to the present invention, particles that are moving relatively can be observed more accurately. [Brief explanation of the drawings]

[0030] [Figure 1] FIG. 1 is a diagram showing the configuration of a particle observation apparatus 1A. [Figure 2] FIG. 2 is a diagram showing the configuration of the particle observation device 1B. [Figure 3] FIG. 3 is a diagram showing an example of an image acquired by the imaging unit 30. As shown in FIG. [Figure 4] 4(a) and 4(b) are diagrams illustrating an example of noise reduction processing by the analysis unit 40. FIG. [Figure 5] 5(a) and 5(b) are diagrams illustrating an example of noise reduction processing by the analysis unit 40. FIG. [Figure 6] FIG. 6 is a diagram illustrating an example of noise reduction processing by the analysis unit 40. In FIG. [Figure 7] 7(a) and (b) are diagrams illustrating an example of noise reduction processing by the analysis unit 40. FIG. [Figure 8] FIG. 8 is a diagram showing an example of an image acquired by the imaging unit 30. As shown in FIG. [Figure 9] 9(a) and 9(b) are diagrams showing examples of images created by the noise reduction process performed by the analysis unit 40. FIG. [Figure 10] 10(a) and 10(b) are diagrams showing examples of images created by the noise reduction process performed by the analysis unit 40. FIG. [Figure 11] FIG. 11 is a diagram showing an example of an image acquired by the imaging unit 30. As shown in FIG. [Figure 12] 12(a) and 12(b) are diagrams illustrating an example of noise reduction processing by the analysis unit 40. FIG. [Figure 13] FIG. 13 is a diagram showing an example of an image including a still particle image. [Figure 14] Fig. 14(a) shows an image obtained by creating a linear image due to the relative movement of particles based on the stationary particle image shown in Fig. 13 and then adding a spot noise image to it. Fig. 14(b) shows an image obtained by performing noise reduction processing on the image shown in Fig. 14(a). [Figure 15] Fig. 15(a) is a diagram showing an image in which a still particle image is obtained by using a point spread function based on a linear image from the image shown in Fig. 14(b), and Fig. 15(b) is a diagram showing an image in which a still particle image is obtained by using a point spread function based on the movement trajectory of the particle image formation position from the image shown in Fig. 14(b). [Figure 16] FIG. 16 is a graph showing the relationship between the number of distinguishable linear images and the SNR (the ratio of the luminance of a linear image to the luminance of a spot noise image). [Figure 17] FIG. 17 is a table summarizing the parameter values ​​obtained for each of the identifiable linear images. DETAILED DESCRIPTION OF THE INVENTION

[0031] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the description of the drawings, the same elements are designated by the same reference numerals, and duplicate explanations will be omitted. The present invention is not limited to these examples, but is defined by the claims, and is intended to include all modifications within the meaning and scope equivalent to the claims.

[0032] FIG. 1 is a diagram showing the configuration of a particle observation apparatus 1A. The particle observation apparatus 1A is an apparatus for observing minute particles P having a size of, for example, approximately tens to hundreds of nanometers, and includes a light source 10, an imaging optical system 20A, an imaging unit 30, an analysis unit 40, and a display unit 50. For ease of explanation, an xyz Cartesian coordinate system is shown in this figure. The particle observation apparatus 1A may be, for example, a flow cytometer. The particles P include minute particles such as DNA or RNA particles, or vesicles containing DNA, RNA, or the like.

[0033] The light source 10 outputs light to be irradiated onto the particle P. The light source 10 may be any light source, for example, a continuous wave laser light source. The wavelength of the light output from the light source 10 is arbitrary as long as it is included in the wavelength range to which the imaging element 31 of the imaging unit 30 has sensitivity. The lens 11 is optically connected to the light source 10 and is an optical system (for example, a beam expander or collimator) that adjusts the beam diameter of the light to be irradiated onto the particle P.

[0034] The imaging optical system 20A receives scattered light from the particle P, which is generated when the particle P is irradiated with light from the light source 10, and forms an image of the particle P. The imaging optical system 20A includes an objective lens 21 and a tube lens 22.

[0035] The imaging unit 30 is optically connected to the imaging optical system 20A and includes a highly sensitive two-dimensional imaging element 31 having an imaging surface at the position where an image is formed by the imaging optical system 20A. The imaging element 31 is preferably an electron multiplier or electron bombardment type. The imaging element 31 is, for example, a multichannel photomultiplier tube, an EM-CCD sensor, an EM-CMOS sensor, an EB-CCD sensor, an EB-CMOS sensor, or a SPAD sensor. The imaging unit 30 outputs image data based on an image formed on the imaging surface of the imaging element 31 during an exposure period of the imaging element 31.

[0036] The filter 32 is inserted in the optical path between the imaging optical system 20A and the imaging unit 30. The filter 32 selectively transmits scattered light from the particles P and selectively blocks light of wavelengths different from that of the scattered light.

[0037] The analysis unit 40 is electrically connected to the imaging unit 30. The analysis unit 40 receives image data output from the imaging unit 30 and analyzes the particles P based on the image represented by the image data. The analysis unit 40 includes a calculation unit (e.g., a CPU or FPGA) that performs calculation processing, a storage unit (e.g., a hard disk drive, a ROM, a RAM) that stores image data, analysis programs, etc., an input unit (e.g., a keyboard or a mouse) that accepts input of analysis conditions, etc. The analysis unit 40 is, for example, a computer.

[0038] The display unit 50 is electrically connected to the analysis unit 40. The display unit 50 displays various images, for example, images represented by image data input from the imaging unit 30, and images obtained during or at the end of analysis by the analysis unit 40. The display unit 50 is, for example, a liquid crystal display.

[0039] The imaging optical system 20A and / or the imaging unit 30 change their relative positional relationship with the particle P during the exposure period of the imaging element 31. For example, assume that the direction of light irradiation from the light source 10 to the particle P is parallel to the x-axis, and the optical axis of the imaging optical system 20A is parallel to the z-axis. In this case, if the particle P moves parallel to the y-axis along the flow path 91, as in a flow cytometer, for example, both the imaging optical system 20A and the imaging unit 30 can change their relative positional relationship with the particle P during the exposure period of the imaging element 31, even if both the imaging optical system 20A and the imaging unit 30 remain stationary. Conversely, if the particle P is almost stationary during the exposure period, the imaging optical system 20A and / or the imaging unit 30 can change their relative positional relationship with the particle P during the exposure period of the imaging element 31 by moving themselves in a direction perpendicular to the z-axis. At this time, either the objective lens 21 or the tube lens 22 of the imaging optical system 20A may move. Alternatively, the optical system from the light source 10 to the imaging unit 30 may move as a single unit. Other modes of movement are also possible.

[0040] In this way, the relative positional relationship between the particle P and either or both of the imaging optical system 20A and the imaging unit 30 changes during the exposure period, and the position at which the particle image is formed moves on the imaging surface of the imaging element 31 during that exposure period.

[0041] The analysis unit 40 extracts a linear image from the image represented by the image data output from the imaging unit 30. The linear image in this image represents the trajectory of the particle image formation position moving on the imaging surface of the imaging element 31 during the exposure period. The shape of the linear image corresponds to the movement trajectory of the particle image formation position on the imaging surface during the exposure period. The movement trajectory of the particle image formation position on the imaging surface (i.e., the shape of the linear image) may be linear or curved. The analysis unit 40 analyzes the particle P based on this extracted linear image.

[0042] Fig. 2 is a diagram showing the configuration of particle observation device 1B. Compared to the configuration of particle observation device 1A (Fig. 1), particle observation device 1B (Fig. 2) differs in that it includes imaging optical system 20B instead of imaging optical system 20A, and also differs in the direction of light irradiation to particle P and the direction of scattered light detection.

[0043] The imaging optical system 20B includes an objective lens 21 and a tube lens 22, and a beam splitter 23 is inserted on the optical path between the objective lens 21 and the tube lens 22. The beam splitter 23 reflects light that is output from the light source 10 and reaches the objective lens 21 after passing through the lens 11. The beam splitter 23 also transmits scattered light that is generated by the particle P and reaches the objective lens 21 after passing through the objective lens 21 to the tube lens 22.

[0044] In this configuration, light output from the light source 10 is irradiated onto a particle P via a lens 11, a beam splitter 23, and an objective lens 21. Scattered light generated by the particle P is imaged on the imaging plane of the imaging element 31 via the objective lens 21, the beam splitter 23, a tube lens 22, and a filter 32. The direction of light irradiation onto the particle P is parallel to the optical axis of the imaging optical system 20B.

[0045] Even in this configuration, both or either one of the imaging optical system 20B and the imaging unit 30 change their relative positional relationship with the particle P during the exposure period of the imaging element 31. For example, suppose the direction of light irradiation from the light source 10 to the particle P is parallel to the z-axis and the optical axis of the imaging optical system 20B is parallel to the z-axis. In this case, if the particle P moves parallel to the y-axis along a flow path, as in a flow cytometer, or if the stage 92 supporting the particle P moves parallel to the y-axis or x-axis as shown in FIG. 2, both or either one of the imaging optical system 20B and the imaging unit 30 can change their relative positional relationship with the particle P during the exposure period of the imaging element 31, even if both the imaging optical system 20B and the imaging unit 30 remain stationary. Conversely, if the particle P is almost stationary during the exposure period, both or either one of the imaging optical system 20B and the imaging unit 30 can change their relative positional relationship with the particle P during the exposure period of the imaging element 31 by moving themselves in a direction perpendicular to the z-axis. At this time, either the objective lens 21 or the tube lens 22 of the imaging optical system 20B may move. Alternatively, the optical system from the light source 10 to the imaging unit 30 may move as a single unit. Other modes of movement are also possible.

[0046] The particle observation apparatus 1A (FIG. 1) and particle observation apparatus 1B (FIG. 2) described above may be replaced with other configurations. The imaging optical systems 20A, 20B and / or the imaging unit 30 may change the relative positional relationship with the particles during the exposure period of the imaging element 31, thereby moving the position where the particle image is formed on the imaging surface of the imaging element 31. The length of the exposure period of the imaging element 31 is set to a length such that a linear image due to the relative movement of the particles is obtained in the image obtained by imaging by the imaging unit 30, and the linear image can be distinguished from a spot noise image.

[0047] The light generated by the particles does not have to be scattered light, but may be fluorescence or chemiluminescence. If the light generated by the particles is fluorescence, the light source 10 outputs excitation light, and the filter 32 blocks the excitation light. If the light generated by the particles is chemiluminescence, the light source 10 is not necessary. Also, if the particles generate scattered light by being irradiated with natural light or room lighting, the light source 10 is not necessary.

[0048] The particle observation method using the light source 10, the imaging optical systems 20A and 20B, and the image capturing unit 30 includes an imaging step and an analysis step. In the imaging step, during an exposure period of the image capturing element 31, the relative positional relationship between the particle P and either or both of the imaging optical systems 20A and 20B and the image capturing unit 30 is changed to move the position where the particle image is formed on the image capturing surface of the image capturing element 31. Then, in the imaging step, image data is output from the image capturing unit 30 based on the image formed on the image capturing surface of the image capturing element 31 during the exposure period of the image capturing element 31. In the analysis step, a linear image is extracted from the image represented by the image data output from the image capturing unit 30, and the particle P is analyzed based on the extracted linear image.

[0049] FIG. 3 shows an example of an image acquired by the imaging unit 30. This image shows numerous spot noise images (false images) and two linear images (true images). The two linear images extend in the y direction in the image, corresponding to the relative movement of two particles in the y direction within the field of view of the imaging unit 30 during the exposure period of the imaging element 31. As shown in this figure, the linear images corresponding to the movement trajectories of the particle image formation positions on the imaging surface have a different shape from the spot noise images. The analysis unit 40 can extract the linear images based on the difference in shape between the linear images and the spot noise images.

[0050] The linear image in the image corresponding to the movement trajectory of the particle image formation position on the imaging plane may be an image of a single continuous region, or may be an image of multiple small region images connected together if the light from the particle is weak. In the latter case, the analysis unit 40 can convert the image of multiple small region images connected together into an image of a single continuous region by image processing including the technology described in Non-Patent Document 1 and machine learning, and can extract the linear image.

[0051] Furthermore, it is preferable that the analysis unit 40 extracts linear images after reducing noise in the image. The noise to be reduced here includes not only spot noise but also background noise. The analysis unit 40 may simultaneously reduce spot noise and background noise, or may reduce spot noise after reducing background noise. Alternatively, the analysis unit 40 may selectively reduce either spot noise or background noise.

[0052] Various methods are available for reducing noise in an image by applying a machine learning model. The analysis unit 40 can reduce noise using, for example, a principal component analysis (PCA) model, an independent component analysis (ICA) model, or a nonnegative matrix factorization (NMF) model. These reduce noise by processing data on a column-by-column and row-by-row basis. The analysis unit 40 can also reduce noise using an unsupervised deep learning model such as Noise2Self (N2S).

[0053] Furthermore, it is preferable that the analysis unit 40 extracts linear images in the image by applying a machine learning model to each of the first and second directions, with the y direction in which the linear images in the image extend being the first direction and the x direction perpendicular to this first direction being the second direction.

[0054] 4 to 7 are diagrams illustrating an example of noise reduction processing by the analysis unit 40. Here, an example will be described in which the number of pixels of the image shown in Fig. 3 is 256 x 256, and noise reduction processing is performed on this image using PCA. In noise reduction processing using PCA, data processing is performed on each column and row of the image.

[0055] In column-wise PCA, an image is considered as a set of 256 column vectors (Figure 4(a)). The number of dimensions of each column vector is 256. Then, PCA is used to reduce the number of dimensions of the column vectors. This data reduction process leaves 50% of the variance of the column vectors, creating an image (Figure 4(b)) that retains a certain amount of variation between columns.

[0056] In row-wise PCA, an image is considered as a set of 256 row vectors (Figure 5(a)). Each row vector has 256 dimensions. PCA then reduces the dimension of the row vectors. This dimensionality reduction process leaves 10% of the variance in the row vectors, creating an image with reduced row-by-row differences (Figure 5(b)).

[0057] The image after noise reduction is the average image of the image after data processing by column-wise PCA (FIG. 4(b)) and the image after data processing by row-wise PCA (FIG. 5(b)). Here, the average image may be the arithmetic mean image (FIG. 6) or the geometric mean image (FIG. 7(b)). FIG. 6 shows the arithmetic mean image of the image after data processing by column-wise PCA (FIG. 4(b)) and the image after data processing by row-wise PCA (FIG. 5(b)). FIG. 7(a) shows the product image of the image after data processing by column-wise PCA (FIG. 4(b)) and the image after data processing by row-wise PCA (FIG. 5(b)). FIG. 7(b) shows the square root image (i.e., the geometric mean image) of the product image (FIG. 7(a)). Looking at these images, it can be seen that noise has been reduced in both the arithmetic mean image (FIG. 6) and the geometric mean image (FIG. 7(b)).

[0058] 8 to 10 are diagrams comparing the results of noise reduction processing by each method performed by the analysis unit 40. FIG. 8 is a diagram showing an example of an image acquired by the imaging unit 30. FIGS. 9 and 10 are diagrams showing examples of images created by noise reduction processing by the analysis unit 40 for the image shown in FIG. 8. FIG. 9(a) shows an image created by noise reduction processing using PCA. FIG. 9(b) shows an image created by noise reduction processing using ICA. FIG. 10(a) shows an image created by noise reduction processing using NMF. FIG. 10(b) shows an image created by noise reduction processing using N2S. Images with reduced noise can be obtained by either method. In the image created by noise reduction processing using NMF (FIG. 10(a)), noise is significantly suppressed due to the regularization effect caused by sparsity. In the image created by noise reduction processing using N2S (FIG. 10(b)), streak-like artifacts extending in the y direction are suppressed compared to the other images.

[0059] In each of the images shown in FIGS. 3 to 10 , the linear image extends in the y direction in the image, corresponding to the relative movement of particles in the y direction during the exposure period of the image sensor 31. When the linear image in the image does not extend in either the x or y direction, as shown in FIG. 11 , the analysis unit 40 converts the axial direction of the image so that the linear image extends in the x or y direction in the converted image, and then performs noise reduction processing using the method described with reference to FIGS. 4 to 7 . Alternatively, as shown in FIG. 12 , the analysis unit 40 may define the direction in which the linear image in the image extends as a first direction and the direction perpendicular to the first direction as a second direction, and then perform noise reduction processing using the method described with reference to FIGS. 4 to 7 . FIG. 12( a ) illustrates a case where a vector is set in the first direction (the direction in which the linear image extends) for the image shown in FIG. 11 . FIG. 12(b) shows a case where a vector is set in the second direction (direction perpendicular to the first direction) for the image shown in FIG.

[0060] The analysis unit 40 extracts linear images from the image acquired by the imaging unit 30 (or the image after noise reduction processing) and analyzes particles based on the extracted linear images. For example, the analysis unit 40 can determine the number of linear images in the image as the number of particles. Furthermore, in the case where particles move along a flow path, such as in a flow cytometer, the analysis unit 40 can determine the number of particles moving per unit time. In the case where particles are almost stationary during the exposure period, the analysis unit 40 can determine the number of particles present per unit volume or unit area. The analysis unit 40 can also determine the relative movement speed and movement direction of the particles.

[0061] The analysis unit 40 can also obtain particle images based on the brightness value distribution of the linear image in the image, and can also obtain the shape and size of the particles from this. If the movement trajectory of the particle image formation position on the imaging surface of the imaging element 31 during the exposure period of the imaging element 31 is known, this can be used to obtain a still particle image based on the brightness value distribution of the linear image.

[0062] Furthermore, a still particle image can be obtained by performing the following image processing. The brightness value distribution of one linear image in the image can be regarded as a function (point spread function) that represents the blur of the particle image due to the movement of the particle image on the imaging surface of the image sensor 31. Therefore, by obtaining this point spread function, a still particle image can be obtained from the linear image using this function.

[0063] The point spread function can be calculated by normalizing the brightness distribution within a region set to surround a single linear image in the image so that the total brightness is 1. At this time, it is preferable to reduce noise within the region, for example by threshold processing. Calculation of stationary particle images using the point spread function is possible for each linear image in the image using the Lucy-Richardson method, etc.

[0064] FIG. 13 is a diagram showing an example of an image including a still particle image. FIG. 14(a) is a diagram showing an image obtained by creating a linear image due to the relative movement of particles based on the still particle image shown in FIG. 13 and then adding a spot noise image. FIG. 14(b) is a diagram showing an image obtained by performing noise reduction processing on the image shown in FIG. 14(a). FIG. 15(a) is a diagram showing an image obtained by calculating a still particle image based on the image shown in FIG. 14(b) using a point spread function based on the linear image. FIG. 15(b) is a diagram showing an image obtained by calculating a still particle image based on the image shown in FIG. 14(b) using a point spread function based on the movement trajectory of the particle image formation position. In both FIG. 15(a) and FIG. 15(b), the blurring of the linear image has been removed, resulting in a blur-free still particle image. In Figure 15(a), the position of the obtained stationary particle image does not coincide with the position of the linear image, because the center position of the linear image and the center of gravity position of the point spread function do not coincide with each other.

[0065] The analysis unit 40 can also determine the particle size based on the brightness value of the linear image in the image. The method for determining the particle size is as follows: It is known that the intensity of scattered light from a particle strongly depends on the particle size. In the Rayleigh scattering region, which occurs when the particle size is smaller than the wavelength of the light irradiating the particle, the scattered light intensity is proportional to the sixth power of the particle size. Therefore, the scattered light intensity can be determined from the linear image in the image, and the particle size can be estimated based on this scattered light intensity. In this case, it is preferable to prepare in advance a lookup table that stores the relationship between scattered light intensity and particle size, and to determine the particle size by referring to this lookup table.

[0066] There are several methods for determining scattered light intensity, including the following: In the first method, the total brightness value within an area set to surround one linear image in the image is determined, and this total brightness value is used as the scattered light intensity; in the second method, a particle image is determined from one linear image in the image using the method described above, and the maximum brightness value of that particle image is determined, and this maximum brightness value is used as the scattered light intensity; in the third method, the total brightness value of the image is determined, and this total brightness value is divided by the number of linear images to determine the average brightness value, and this average brightness value is used as the scattered light intensity.

[0067] Next, the simulation results will be explained. The conditions assumed in this simulation are as follows: A laser light source outputting CW laser light with a wavelength of 488 nm was used as light source 10, and the laser light output from this light source 10 was collimated and irradiated onto particles. The beam diameter of the laser light when irradiating particles was set to 3 mm so that it was several times larger than the particle size. Nanoparticles suspended in a fluid (liquid or gas) were flowed through a glass flow channel. The cross-sectional size of the flow channel was set to 250 μm x 250 μm. The direction of light irradiation from light source 10 to the particles was assumed to be parallel to the x-axis, the optical axis of the imaging optical system was parallel to the z-axis, and the direction of particle movement along the flow channel was assumed to be parallel to the y-axis.

[0068] The imaging unit 30 was a camera including an electron bombardment CMOS sensor (EB-CMOS sensor) as the imaging element 31. Ten images were created using the imaging unit 30. For each of these 10 images, the luminance values ​​of the spot noise image were left unchanged, while the luminance values ​​of the linear image region were multiplied by a fixed value to set the SNR of the image to four different values ​​(approximately 0.55, approximately 0.75, approximately 0.9, and approximately 1.1). Here, the SNR (Signal-to-Noise Ratio) was defined as the ratio of the luminance values ​​of the linear image to the luminance values ​​of the spot noise image. This resulted in a total of 40 pre-noise reduction images. Furthermore, for each of these 40 pre-noise reduction images, images after noise reduction using PCA were created, as well as images without spot noise.

[0069] Figure 16 is a graph showing the relationship between the number of distinguishable linear images and SNR. This graph shows the results after noise reduction (after denoising), before noise reduction (before denoising), and without spot noise. Without spot noise, all 10 particles could be distinguished at all SNRs. Before noise reduction (before denoising), particles could not be distinguished at low SNRs, but 9 out of 10 particles could be distinguished at an SNR of approximately 1.1 dB. After noise reduction (after denoising), 3 out of 10 particles could be distinguished at an SNR of approximately 0.55, 6 out of 10 particles could be distinguished at an SNR of approximately 0.75, 9 out of 10 particles could be distinguished at an SNR of approximately 0.9, and all 10 particles could be distinguished at an SNR of approximately 1.1. Thus, by performing noise reduction on the image, it became easier to distinguish particles.

[0070] Fig. 17 is a table summarizing the parameter values ​​obtained for each identifiable linear image, including the linear image area, average brightness, maximum brightness, circumference, circularity, aspect ratio, and roundness, as well as the width and height of the smallest rectangle enclosing the linear image.

[0071] As described above, according to this embodiment, particles that are moving relatively can be observed more accurately. For example, even when the particles to be imaged are moving as in the example of the flow cytometer described above, or when minute particles to be imaged are floating in the air as in the example of the particle counter, it is possible to distinguish between particle images and spot noise light.

[0072] The present invention is not limited to the above-described embodiment, and various modifications are possible. For example, the particle observation device may not be provided with the light source 10, and the image sensor 31 may receive scattered light generated when natural light or room light is irradiated onto the particles P. Furthermore, the light from the particles P does not have to be scattered light, and may be, for example, fluorescence or chemiluminescence. [Explanation of symbols]

[0073] 1A, 1B... particle observation device, 10... light source, 11... lens, 20A, 20B... imaging optical system, 21... objective lens, 22... tube lens, 23... beam splitter, 30... imaging unit, 31... imaging element, 32... filter, 40... analysis unit, 50... display unit.

Claims

1. an imaging optical system that inputs light from the particles and forms an image of the particles; an imaging unit including an imaging element having an imaging surface at a position where an image is formed by the imaging optical system, and outputting image data based on an image formed on the imaging surface during an exposure period of the imaging element; an analysis unit that receives image data output from the imaging unit and analyzes the particles based on the image represented by the image data; Equipped with and both or either one of the imaging optical system and the imaging unit changes a relative positional relationship with the particle during the exposure period to move a formation position of an image of the particle on the imaging plane; the analysis unit extracts a linear image from the image and analyzes the particle based on the extracted linear image. Particle observation device.

2. the analysis unit extracts a linear image after reducing noise in the image; The particle observation device according to claim 1 .

3. the analysis unit applies a machine learning model to extract a linear image in the image; The particle observation device according to claim 1 .

4. the analysis unit applies, as the machine learning model, any one of a principal component analysis model, an independent component analysis model, a non-negative factorization model, and a deep learning model to extract a linear image in the image; The particle observation device according to claim 3 .

5. the analysis unit defines a direction in which the linear image in the image extends as a first direction, defines a direction perpendicular to the first direction as a second direction, and applies a machine learning model to each of the first direction and the second direction to extract the linear image in the image. The particle observation device according to claim 3 .

6. the analysis unit determines the number of linear images in the image as the number of particles. The particle observation device according to claim 1 .

7. the analysis unit obtains an image of the particle based on a luminance value distribution of a linear image in the image. The particle observation device according to claim 1 .

8. the analysis unit determines the size of the particle based on the brightness value of the linear image in the image. The particle observation device according to claim 1 .

9. The imaging unit includes an electron multiplying type or an electron bombardment type imaging element. The particle observation device according to claim 1 .

10. Further provided is a light source that outputs light to be irradiated onto the particles. The particle observation device according to claim 1 .

11. an imaging step using an imaging optical system that inputs light from a particle and forms an image of the particle, and an imaging unit including an imaging element having an imaging surface at a position where an image is formed by the imaging optical system, and outputting image data from the imaging unit based on an image formed on the imaging surface during an exposure period of the imaging element; an analysis step of inputting image data output from the imaging unit and analyzing the particles based on the image represented by the image data; Equipped with In the imaging step, during the exposure period, a relative positional relationship between the particle and either one of the imaging optical system and the imaging unit is changed to move a position where an image of the particle is formed on the imaging plane; In the analyzing step, a linear image is extracted from the image, and the particle is analyzed based on the extracted linear image. Particle observation methods.

Images