Particle observation device and observation method

Abstract

A particle observation device 1A is a device for observing minute particles P, and comprises a light source 10, an imaging optical system 20A, an imaging unit 30, an analyzing unit 40, and a display unit 50. Either or both of the imaging optical system 20A and the imaging unit 30 change a relative positional relationship with the particles P during an exposure period of the imaging element 31, to move the formation position of an image of the particles P on an imaging surface of the imaging element 31. The analyzing unit 40 extracts a linear image from the image, and analyzes the particle P on the basis of the extracted linear image. As a result, realized is a particle observation device capable of more accurately observing particles moving relatively.

Description

Particle observation device and observation method

[0001] The present disclosure relates to an apparatus and method for observing particles.

[0002] For example, in an apparatus for observing minute particles having a size of about several tens to several hundreds of nanometers, an imaging element capable of receiving weak light from the particles and capturing a two-dimensional image including the particle image with high sensitivity is used. Examples of such an apparatus include a flow cytometer for observing minute particles such as DNA or RNA particles or vesicles containing DNA or RNA, and a particle counter for counting minute dust particles floating in a clean room.

[0003] Furthermore, the imaging element preferably used here is an electron multiplying type or an electron bombardment type, such as a multi-channel photomultiplier tube, an EM-CCD sensor, an EM-CMOS sensor, an EB-CCD sensor, an EB-CMOS sensor, or a SPAD sensor.

[0004] In an image acquired by imaging using an imaging element, not only the image of the particle being imaged (true image) but also spot-like noise images (false images) may appear. Spot noise images can be caused by, for example, the incidence of cosmic rays or originating from the imaging element.

[0005] It is sufficient to be able to distinguish between particle images and spot noise images by using one of the following criteria: size, brightness, or shape on the image. However, this is difficult when the particles are small and the light from them is weak. If it is not possible to distinguish between the two, particle images cannot be extracted from the image, and particles cannot be observed accurately. For example, the number of particles will be overestimated, and the accuracy of measuring the shape and size of particles will be reduced.

[0006] Patent Document 1 discloses an invention for acquiring an image in which the influence of spot noise images is reduced. The invention disclosed in this document reduces the influence of spot noise images by acquiring multiple images and averaging these multiple images. This takes advantage of the fact that, while the image of the object to be imaged appears at a common position in all of the multiple images, spot noise images rarely appear at a common position in multiple images.

[0007] Japanese Patent Application Laid-Open No. 2002-286843

[0008] Kota Tajima, Yuma Mori, Hiroshi Masuda, "Linear object detection using point clouds and images by moving measurement (2nd report)", Proceedings of the 2019 Japan Society for Precision Engineering Spring Meeting A68 (2019)

[0009] Although the invention disclosed in Patent Document 1 may be applicable when the object to be imaged is stationary, it is difficult to apply it when the object to be imaged is moving. For example, when particles to be imaged are moving as in the example of the flow cytometer described above, or when minute debris to be imaged is floating in the air as in the example of the particle counter, particle images appear at different positions in multiple images, making it difficult to apply the invention disclosed in this document.

[0010] The embodiments aim to provide an apparatus and method that can more accurately observe particles that are moving relative to one another.

[0011] An embodiment is a particle observation device that includes: (1) an imaging optical system that inputs light from particles and forms an image of the particles; (2) an imaging unit that includes an imaging element having an imaging surface at a position where an image is formed by the imaging optical system and outputs image data based on an image formed on the imaging surface during an exposure period of the imaging element; (3) an analysis unit that inputs the image data output from the imaging unit and analyzes the particles based on the image represented by the image data; (4) one or both of the imaging optical system and the imaging element change their relative positional relationship with the particles during the exposure period to move the position where the particle image is formed on the imaging surface; and (5) the analysis unit extracts a linear image from the image and analyzes the particles based on the extracted linear image.

[0012] An embodiment is a particle observation method, which includes: (1) an imaging step of using an imaging optical system that inputs light from a particle and forms an image of the particle, and an imaging unit including an imaging element having an imaging surface at a position where the image is formed by the imaging optical system, and outputting image data from the imaging unit based on an image formed on the imaging surface during an exposure period of the imaging element; (2) an analysis step of inputting the image data output from the imaging unit and analyzing the particle based on the image represented by the image data; (3) in the imaging step, during the exposure period, changing the relative positional relationship between the particle and either or both of the imaging optical system and the imaging unit to move the position where the particle image is formed on the imaging surface; and (4) in the analysis step, extracting a linear image from the image, and analyzing the particle based on the extracted linear image.

[0013] According to the particle observation device and particle observation method of the embodiment, particles that are moving relatively can be observed more accurately.

[0014] FIG. 1 is a diagram showing the configuration of a particle observation apparatus 1A. FIG. 2 is a diagram showing the configuration of a particle observation apparatus 1B. FIG. 3 is a diagram showing an example of an image acquired by the imaging unit 30. FIGS. 4A and 4B are diagrams explaining an example of noise reduction processing by the analysis unit 40. FIGS. 5A and 5B are diagrams explaining an example of noise reduction processing by the analysis unit 40. FIG. 6 is a diagram explaining an example of noise reduction processing by the analysis unit 40. FIGS. 7A and 7B are diagrams explaining an example of noise reduction processing by the analysis unit 40. FIG. 8 is a diagram showing an example of an image acquired by the imaging unit 30. FIGS. 9A and 9B are diagrams showing an example of an image created by the noise reduction processing by the analysis unit 40. FIGS. 10A and 10B are diagrams showing an example of an image created by the noise reduction processing by the analysis unit 40. FIG. 11 is a diagram showing an example of an image acquired by the imaging unit 30. FIGS. 12A and 12B are diagrams explaining an example of noise reduction processing by the analysis unit 40. FIG. 13 is a diagram showing an example of an image including a still particle image. Fig. 14(a) shows an image obtained by creating a linear image due to the relative movement of particles based on the still particle image shown in Fig. 13 and then adding a spot noise image, and Fig. 14(b) shows an image obtained by performing noise reduction processing on the image shown in Fig. 14(a). Fig. 15(a) shows an image obtained by calculating a still particle image based on the image shown in Fig. 14(b) using a point spread function based on the linear image, and Fig. 15(b) shows an image obtained by calculating a still particle image based on the image shown in Fig. 14(b) using a point spread function based on the movement trajectory of the particle image formation position. Fig. 16 is a graph showing the relationship between the number of identifiable linear images and the SNR (the ratio of the brightness of the linear image to the brightness of the spot noise image). Fig. 17 is a table summarizing the parameter values ​​obtained for each identifiable linear image.

[0015] Hereinafter, embodiments of a particle observation device and a particle observation method will be described in detail with reference to the accompanying drawings. In the description of the drawings, the same elements are designated by the same reference numerals, and duplicate explanations will be omitted. The present invention is not limited to these examples, but is defined by the claims, and is intended to include all modifications within the meaning and scope equivalent to the claims.

[0016] 1 is a diagram showing the configuration of a particle observation device 1A. The particle observation device 1A is a device for observing minute particles P having a size of, for example, about several tens to several hundreds of nanometers, and includes a light source 10, an imaging optical system 20A, an imaging unit 30, an analysis unit 40, a display unit 50, etc.

[0017] For ease of explanation, an xyz Cartesian coordinate system is shown in this figure. The particle observation device 1A may be, for example, a flow cytometer. The particles P include fine particles such as DNA or RNA, or microparticles such as vesicles containing DNA, RNA, or the like.

[0018] The light source 10 outputs light to be irradiated onto the particles P. The light source 10 may be any light source, for example, a continuous wave laser light source. The wavelength of the light output from the light source 10 is arbitrary as long as it is included in the wavelength range to which the imaging element 31 of the imaging unit 30 is sensitive. The lens 11 is optically connected to the light source 10 and is an optical system (for example, a beam expander or collimator) that adjusts the beam diameter of the light to be irradiated onto the particles P.

[0019] The imaging optical system 20A receives scattered light from the particle P, which is generated when the particle P is irradiated with light from the light source 10, and forms an image of the particle P. The imaging optical system 20A includes an objective lens 21 and a tube lens 22.

[0020] The imaging unit 30 is optically connected to the imaging optical system 20A and includes a highly sensitive two-dimensional imaging element 31 having an imaging surface at the position where the image is formed by the imaging optical system 20A. The imaging element 31 is preferably an electron multiplier or electron bombardment type. The imaging element 31 is, for example, a multichannel photomultiplier tube, an EM-CCD sensor, an EM-CMOS sensor, an EB-CCD sensor, an EB-CMOS sensor, or a SPAD sensor. The imaging unit 30 outputs image data based on an image formed on the imaging surface of the imaging element 31 during an exposure period of the imaging element 31.

[0021] The filter 32 is inserted in the optical path between the imaging optical system 20A and the image capturing unit 30. The filter 32 selectively transmits scattered light from the particles P and selectively blocks light of wavelengths different from that of the scattered light.

[0022] The analysis unit 40 is electrically connected to the imaging unit 30. The analysis unit 40 receives image data output from the imaging unit 30 and analyzes the particles P based on the image represented by the image data. The analysis unit 40 includes a calculation unit (e.g., a CPU or FPGA) that performs calculation processing, a storage unit (e.g., a hard disk drive, ROM, RAM, etc.) that stores image data, analysis programs, etc., an input unit (e.g., a keyboard or mouse, etc.) that accepts input of analysis conditions, etc. The analysis unit 40 is, for example, a computer.

[0023] The display unit 50 is electrically connected to the analysis unit 40. The display unit 50 displays various images, for example, images represented by image data input from the imaging unit 30, and images obtained during or at the end of analysis by the analysis unit 40. The display unit 50 is, for example, a liquid crystal display.

[0024] The imaging optical system 20A and / or the imaging unit 30 change the relative positional relationship between the particle P and the imaging optical system 20A and / or the particle P during the exposure period of the imaging element 31.

[0025] For example, suppose that the direction of light irradiation from the light source 10 to the particle P is parallel to the x-axis and the optical axis of the imaging optical system 20A is parallel to the z-axis. In this case, if the particle P moves parallel to the y-axis along the flow path 91, as in a flow cytometer, for example, both the imaging optical system 20A and the image capturing unit 30 can change the relative positional relationship between the particle P and the imaging optical system 20A or the image capturing unit 30 during the exposure period of the image capturing element 31, even if both the imaging optical system 20A and the image capturing unit 30 remain stationary.

[0026] Conversely, if the particle P is almost stationary during the exposure period, both or either one of the imaging optical system 20A and the image capturing unit 30 can move in a direction perpendicular to the z-axis to change the relative positional relationship with the particle P during the exposure period of the image capturing element 31. In this case, either the objective lens 21 or the tube lens 22 of the imaging optical system 20A may move. Alternatively, the optical system from the light source 10 to the image capturing unit 30 may move as a unit. Other modes of movement are also possible.

[0027] In this way, the relative positional relationship between the particle P and either or both of the imaging optical system 20A and the imaging unit 30 changes during the exposure period, and the position at which the particle image is formed moves on the imaging surface of the imaging element 31 during that exposure period.

[0028] The analysis unit 40 extracts a linear image from the image represented by the image data output from the imaging unit 30. The linear image in this image represents the trajectory of the movement of the particle image formation position on the imaging surface of the imaging element 31 during the exposure period. The shape of the linear image corresponds to the movement trajectory of the particle image formation position on the imaging surface during the exposure period. The movement trajectory of the particle image formation position on the imaging surface (i.e., the shape of the linear image) may be linear or curved. The analysis unit 40 analyzes the particle P based on this extracted linear image.

[0029] Fig. 2 is a diagram showing the configuration of particle observation device 1B. Compared to the configuration of particle observation device 1A (Fig. 1), particle observation device 1B (Fig. 2) differs in that it includes imaging optical system 20B instead of imaging optical system 20A, and also differs in the direction of light irradiation to particles P and the direction of scattered light detection.

[0030] The imaging optical system 20B includes an objective lens 21 and a tube lens 22, and a beam splitter 23 is inserted in the optical path between the objective lens 21 and the tube lens 22. The beam splitter 23 reflects light that is output from the light source 10 and reaches the objective lens 21 via the lens 11. The beam splitter 23 also transmits scattered light that is generated by the particle P and reaches the objective lens 21 via the objective lens 21 to the tube lens 22.

[0031] In this configuration, light output from the light source 10 is irradiated onto a particle P via the lens 11, the beam splitter 23, and the objective lens 21. Scattered light generated by the particle P is imaged on the imaging plane of the imaging element 31 via the objective lens 21, the beam splitter 23, the tube lens 22, and the filter 32. The direction of light irradiation onto the particle P is parallel to the optical axis of the imaging optical system 20B.

[0032] In this configuration, the relative positional relationship between the particle P and the imaging optical system 20B and / or the imaging unit 30 changes during the exposure period of the imaging element 31.

[0033] For example, suppose that the direction of light irradiation from the light source 10 to the particle P is parallel to the z-axis, and the optical axis of the imaging optical system 20B is parallel to the z-axis. In this case, if the particle P moves parallel to the y-axis along a flow path, as in a flow cytometer, or if the stage 92 supporting the particle P moves parallel to the y-axis or x-axis as shown in Fig. 2, then even if both the imaging optical system 20B and the image capturing unit 30 remain stationary, both or either one of the imaging optical system 20B and the image capturing unit 30 can change the relative positional relationship with the particle P during the exposure period of the image capturing element 31.

[0034] Conversely, if the particle P is almost stationary during the exposure period, both or either one of the imaging optical system 20B and the image capturing unit 30 can move in a direction perpendicular to the z-axis to change the relative positional relationship with the particle P during the exposure period of the image capturing element 31. In this case, either the objective lens 21 or the tube lens 22 of the imaging optical system 20B may move. Alternatively, the optical system from the light source 10 to the image capturing unit 30 may move as a unit. Other modes of movement are also possible.

[0035] The particle observation apparatus 1A ( FIG. 1 ) and particle observation apparatus 1B ( FIG. 2 ) described above may be replaced with other configurations. The imaging optical systems 20A, 20B and / or the imaging unit 30 may change the relative positional relationship with the particles during the exposure period of the imaging element 31, thereby moving the position where the particle image is formed on the imaging surface of the imaging element 31. The length of the exposure period of the imaging element 31 is set to a length such that a linear image due to the relative movement of the particles is obtained in the image obtained by imaging by the imaging unit 30, and the linear image can be distinguished from a spot noise image.

[0036] The light generated by the particles does not have to be scattered light, and may be fluorescence or chemiluminescence. When the light generated by the particles is fluorescence, the light source 10 outputs excitation light, and the filter 32 blocks the excitation light. When the light generated by the particles is chemiluminescence, the light source 10 is not required. Also, when the particles generate scattered light by being irradiated with natural light or room lighting, the light source 10 is not required.

[0037] The particle observation method using the light source 10, the imaging optical systems 20A and 20B, and the image capturing unit 30 includes an imaging step and an analysis step. In the imaging step, during an exposure period of the image capturing element 31, the relative positional relationship between the particle P and either or both of the imaging optical systems 20A and 20B and the image capturing unit 30 is changed to move the position where the particle image is formed on the image capturing surface of the image capturing element 31.

[0038] Then, in the imaging step, image data is output from the imaging unit 30 based on the image formed on the imaging surface of the imaging element 31 during the exposure period of the imaging element 31. In the analysis step, a linear image is extracted from the image represented by the image data output from the imaging unit 30, and the particle P is analyzed based on this extracted linear image.

[0039] 3 is a diagram showing an example of an image acquired by the imaging unit 30. This image shows a number of spot noise images (false images) and two linear images (true images). The two linear images extend in the y direction in the image, corresponding to the relative movement of two particles in the y direction within the field of view of the imaging unit 30 during the exposure period of the imaging element 31.

[0040] As shown in this figure, the linear image corresponding to the movement trajectory of the particle image formation position on the imaging plane has a shape different from that of the spot noise image. The analysis unit 40 can extract the linear image based on the difference in shape between the linear image and the spot noise image.

[0041] The linear image in the image corresponding to the movement trajectory of the particle image formation position on the imaging plane may be an image of a single continuous region, or may be an image of a series of images of multiple small regions if the light from the particles is weak. In the latter case, the analysis unit 40 can convert the image of a series of images of multiple small regions into an image of a single continuous region by image processing including the technology described in Non-Patent Document 1 and machine learning, and can extract the linear image.

[0042] Furthermore, it is preferable that the analysis unit 40 extracts linear images after reducing noise in the image. The noise to be reduced here includes not only spot noise but also background noise. The analysis unit 40 may simultaneously reduce spot noise and background noise, or may reduce spot noise after reducing background noise. Alternatively, the analysis unit 40 may selectively reduce either spot noise or background noise.

[0043] Various methods are available for reducing noise in an image by applying a machine learning model. The analysis unit 40 can reduce noise using, for example, a principal component analysis (PCA) model, an independent component analysis (ICA) model, or a nonnegative matrix factorization (NMF) model. These reduce noise by processing data on a column-by-column and row-by-row basis. The analysis unit 40 can also reduce noise using an unsupervised deep learning model such as Noise2Self (N2S).

[0044] Furthermore, it is preferable that the analysis unit 40 extracts linear images in the image by applying a machine learning model to each of the first and second directions, with the y direction in which the linear images in the image extend being the first direction and the x direction perpendicular to this first direction being the second direction.

[0045] 4 to 7 are diagrams illustrating an example of noise reduction processing by the analysis unit 40. Here, an example will be described in which the image shown in Fig. 3 has 256 x 256 pixels and noise reduction processing is performed on this image using PCA. In noise reduction processing using PCA, data processing is performed on each column and row of the image.

[0046] In column-wise PCA, an image is considered as a set of 256 column vectors (Fig. 4(a)). Each column vector has 256 dimensions. Then, PCA is used to reduce the dimension of the column vectors. This data processing for dimensionality reduction leaves 50% of the variance of the column vectors, creating an image (Fig. 4(b)) that retains a certain amount of variation between columns.

[0047] In row-wise PCA, an image is viewed as a set of 256 row vectors (Fig. 5(a)). Each row vector has 256 dimensions. PCA then reduces the dimension of the row vectors. This dimensionality reduction process retains 10% of the variance in the row vectors, creating an image with reduced row-to-row variance (Fig. 5(b)).

[0048] The image after noise reduction processing is the average image of the image after data processing by column-wise PCA (FIG. 4(b)) and the image after data processing by row-wise PCA (FIG. 5(b)). Here, the average image may be the arithmetic mean image (FIG. 6) or the geometric mean image (FIG. 7(b)).

[0049] FIG. 6 shows an arithmetic mean image of the image after data processing by column-wise PCA (FIG. 4(b)) and the image after data processing by row-wise PCA (FIG. 5(b)). FIG. 7(a) shows a product image of the image after data processing by column-wise PCA (FIG. 4(b)) and the image after data processing by row-wise PCA (FIG. 5(b)). FIG. 7(b) shows an image of the square root of the product image (FIG. 7(a)) (i.e., the geometric mean image). Looking at these images, it can be seen that noise has been reduced in both the arithmetic mean image (FIG. 6) and the geometric mean image (FIG. 7(b)).

[0050] 8 to 10 are diagrams showing a comparison of the results of noise reduction processing by each method performed by the analysis unit 40. Fig. 8 is a diagram showing an example of an image acquired by the imaging unit 30. Figs. 9 and 10 are diagrams showing examples of images created by performing noise reduction processing by the analysis unit 40 on the image shown in Fig. 8.

[0051] Fig. 9(a) shows an image created by noise reduction processing using PCA. Fig. 9(b) shows an image created by noise reduction processing using ICA. Fig. 10(a) shows an image created by noise reduction processing using NMF. Fig. 10(b) shows an image created by noise reduction processing using N2S. Either method can produce an image with reduced noise.

[0052] In the image created by noise reduction processing using NMF (Fig. 10(a)), noise is significantly suppressed due to the regularization effect of sparsity. In the image created by noise reduction processing using N2S (Fig. 10(b)), streak-like artifacts extending in the y direction are suppressed compared to the other images.

[0053] 3 to 10 shown so far, the linear image in the image extends in the y direction, corresponding to the relative movement of the particles in the y direction during the exposure period of the image sensor 31. When the linear image in the image does not extend in either the x direction or the y direction, as shown in Fig. 11, the analysis unit 40 converts the axial direction of the image so that the linear image in the converted image extends in the x direction or the y direction, and then performs noise reduction processing using the method described with reference to Figs.

[0054] Alternatively, as shown in Fig. 12, the analysis unit 40 may define the direction in which a linear image in an image extends as a first direction, and the direction perpendicular to the first direction as a second direction, and perform noise reduction processing using the method described with reference to Figs. 4 to 7. Fig. 12(a) shows a case in which a vector is set in the first direction (the direction in which the linear image extends) for the image shown in Fig. 11. Fig. 12(b) shows a case in which a vector is set in the second direction (the direction perpendicular to the first direction) for the image shown in Fig. 11.

[0055] The analysis unit 40 extracts linear images from the image acquired by the imaging unit 30 (or the image after noise reduction processing), and analyzes particles based on the extracted linear images. For example, the analysis unit 40 can determine the number of linear images in the image as the number of particles.

[0056] Furthermore, in the case where particles move along a flow path, such as in a flow cytometer, the analysis unit 40 can determine the number of particles moving per unit time. In the case where particles are almost stationary during the exposure period, the analysis unit 40 can determine the number of particles present per unit volume or unit area. The analysis unit 40 can also determine the relative movement speed and movement direction of the particles.

[0057] The analysis unit 40 can also obtain particle images based on the brightness value distribution of the linear images in the image, and can also obtain the shape and size of the particles from this. If the movement trajectory of the particle image formation position on the imaging surface of the imaging element 31 during the exposure period of the imaging element 31 is known, this can be used to obtain static particle images based on the brightness value distribution of the linear images.

[0058] Furthermore, a still particle image can be obtained by performing the following image processing. The brightness value distribution of one linear image in the image can be regarded as a function (point spread function) that represents the blur of the particle image due to the movement of the particle image on the imaging surface of the imaging element 31. Therefore, by obtaining this point spread function, a still particle image can be obtained from the linear image using this function.

[0059] The point spread function can be obtained by normalizing the brightness value distribution within a region set to surround one linear image in the image so that the total brightness value is 1. At this time, it is preferable to reduce noise within the region, for example, by performing threshold processing. Calculation of a stationary particle image using the point spread function is possible for each linear image in the image by the Lucy-Richardson method or the like.

[0060] Fig. 13 is a diagram showing an example of an image including a stationary particle image. Fig. 14(a) is a diagram showing an image obtained by creating a linear image due to the relative movement of particles based on the stationary particle image shown in Fig. 13 and then adding a spot noise image. Fig. 14(b) is a diagram showing an image obtained by performing noise reduction processing on the image shown in Fig. 14(a).

[0061] Fig. 15(a) is a diagram showing an image in which a still particle image is obtained by using a point spread function based on a linear image from the image shown in Fig. 14(b), and Fig. 15(b) is a diagram showing an image in which a still particle image is obtained by using a point spread function based on the movement trajectory of the particle image formation position from the image shown in Fig. 14(b).

[0062] In both Figures 15(a) and 15(b), the blur of the linear image has been removed, resulting in a blur-free static particle image. Note that in Figure 15(a), the position of the obtained static particle image does not coincide with the position of the linear image, but this is because the center position of the linear image and the center of gravity position of the point spread function do not coincide with each other.

[0063] The analysis unit 40 can also determine the particle size based on the brightness value of the linear image in the image. The method for determining the particle size is as follows: It is known that the intensity of scattered light from a particle strongly depends on the particle size. In the region of Rayleigh scattering, which occurs when the particle size is smaller than the wavelength of the light irradiating the particle, the scattered light intensity is proportional to the sixth power of the particle size.

[0064] Therefore, the scattered light intensity can be determined from the linear image in the image, and the particle size can be estimated based on this scattered light intensity. In this case, it is preferable to prepare a lookup table in advance that stores the relationship between scattered light intensity and particle size, and to determine the particle size by referring to this lookup table.

[0065] Methods for determining scattered light intensity include, for example, the following: In a first method, the total luminance value within an area set to surround one linear image in the image is determined, and this total luminance value is used as the scattered light intensity; in a second method, a particle image is determined from one linear image in the image using the method described above, and the maximum luminance value of that particle image is determined, and this maximum luminance value is used as the scattered light intensity; and in a third method, the total luminance value of the image is determined, and this total luminance value is divided by the number of linear images to determine the average luminance value, and this average luminance value is used as the scattered light intensity.

[0066] Next, the results of the simulation will be described. The conditions assumed in this simulation are as follows: A laser light source outputting a CW laser beam with a wavelength of 488 nm was used as light source 10, and the laser beam output from this light source 10 was collimated and irradiated onto the particles. The beam diameter of the laser beam when irradiating the particles was set to a diameter of 3 mm so that it was several times larger than the size of the particles.

[0067] Nanoparticles suspended in a fluid (liquid or gas) were flowed through a glass channel with a cross-sectional size of 250 μm × 250 μm. The direction of light irradiation from the light source 10 to the particles was parallel to the x-axis, the optical axis of the imaging optical system was parallel to the z-axis, and the direction of particle movement along the channel was parallel to the y-axis.

[0068] A camera including an electron bombardment type CMOS sensor (EB-CMOS sensor) as the imaging element 31 was used as the imaging unit 30. Ten images were created by imaging using this imaging unit 30. In each of these ten images, the luminance value of the spot noise image was left unchanged, while the luminance value of the linear image region was multiplied by a constant value, thereby setting the SNR of the image to four different values ​​(approximately 0.55, approximately 0.75, approximately 0.9, and approximately 1.1).

[0069] Here, the SNR (Signal to Noise Ratio) was defined as the ratio of the luminance value of the linear image to the luminance value of the spot noise image. A total of 40 pre-noise reduction images were created using this method. For each of these 40 pre-noise reduction images, an image was created after noise reduction using PCA, and an image without spot noise was also created.

[0070] Figure 16 is a graph showing the relationship between the number of distinguishable linear images and SNR. This graph shows graphs for the cases after noise reduction (after denoising), before noise reduction (before denoising), and without spot noise. Without spot noise, all 10 particles could be distinguished at all SNRs. Before noise reduction (before denoising), particles could not be distinguished when the SNR was low, but 9 out of 10 particles could be distinguished when the SNR was approximately 1.1 dB.

[0071] After noise reduction processing (denoising), when the SNR was about 0.55, 3 out of 10 particles could be identified, when the SNR was about 0.75, 6 out of 10 particles could be identified, when the SNR was about 0.9, 9 out of 10 particles could be identified, and when the SNR was about 1.1, all 10 particles could be identified. Thus, by performing noise reduction processing on the image, it became easier to identify particles.

[0072] 17 is a table summarizing the parameter values ​​obtained for each identifiable linear image, including the linear image area, average brightness, maximum brightness, perimeter, circularity, aspect ratio, and roundness, as well as the width and height of the smallest rectangle enclosing the linear image.

[0073] As described above, according to this embodiment, particles that are moving relatively can be observed more accurately. For example, even when the particles to be imaged are moving as in the example of the flow cytometer described above, or when minute particles to be imaged are floating in the air as in the example of the particle counter, it is possible to distinguish between particle images and spot noise light.

[0074] The particle observation device and particle observation method are not limited to the above-described embodiment and configuration example, and various modifications are possible. For example, the particle observation device does not need to be equipped with the light source 10, and the image sensor 31 may receive scattered light generated when natural light or room lighting is irradiated onto the particles P. Furthermore, the light from the particles P does not need to be scattered light, and may be, for example, fluorescence or chemiluminescence.

[0075] The particle observation device of the first aspect according to the above embodiment comprises: (1) an imaging optical system that inputs light from a particle and forms an image of the particle; (2) an imaging unit that includes an imaging element having an imaging surface at the position where the image is formed by the imaging optical system and outputs image data based on the image formed on the imaging surface during the exposure period of the imaging element; and (3) an analysis unit that inputs the image data output from the imaging unit and performs analysis of the particle based on the image represented by this image data; (4) both or one of the imaging optical system and the imaging unit changes the relative positional relationship with the particle during the exposure period, thereby moving the position where the particle image is formed on the imaging surface; and (5) the analysis unit extracts a linear image in the image and performs analysis of the particle based on the extracted linear image.

[0076] In the particle observation device of the second aspect, in the configuration of the first aspect, the analysis unit may be configured to extract the linear image after reducing noise in the image.

[0077] In the particle observation device of the third aspect, in the configuration of the first or second aspect, the analysis unit may be configured to extract a linear image in the image by applying a machine learning model.

[0078] In the particle observation device of the fourth aspect, in the configuration of the third aspect, the analysis unit may be configured to extract linear images in the image by applying any one of a principal component analysis model, an independent component analysis model, a non-negative factorization model, and a deep learning model as a machine learning model.

[0079] In the particle observation device of the fifth aspect, in the configuration of the third or fourth aspect, the analysis unit may be configured to define the direction in which the linear image in the image extends as a first direction, define the direction perpendicular to this first direction as a second direction, and apply a machine learning model to each of the first direction and the second direction to extract the linear image in the image.

[0080] In the particle observation device of the sixth aspect, in the configuration of any one of the first to fifth aspects, the analysis section may be configured to obtain the number of linear images in the image as the number of particles.

[0081] In the particle observation device of the seventh aspect, in the configuration of any one of the first to sixth aspects, the analysis section may be configured to obtain an image of the particle based on the luminance value distribution of the linear image in the image.

[0082] In the particle observation device of the eighth aspect, in the configuration of any one of the first to seventh aspects, the analysis section may be configured to find the size of the particle based on the brightness value of the linear image in the image.

[0083] In the particle observation device of the ninth aspect, in the configuration of any one of the first to eighth aspects, the imaging section may be configured to include an electron multiplying type or an electron bombardment type imaging element.

[0084] The particle observation device of the tenth aspect may be configured in any one of the first to ninth aspects, further comprising a light source that outputs light to irradiate the particles.

[0085] The particle observation method of the first aspect according to the above embodiment includes: (1) an imaging step using an imaging optical system that inputs light from a particle and forms an image of the particle, and an imaging unit including an imaging element having an imaging surface at the position of image formation by the imaging optical system, and outputting image data from the imaging unit based on the image formed on the imaging surface during an exposure period of the imaging element; (2) an analysis step that inputs the image data output from the imaging unit and analyzes the particle based on the image represented by this image data; (3) in the imaging step, during the exposure period, the relative positional relationship between the particle and either or both of the imaging optical system and the imaging unit is changed to move the position where the particle image is formed on the imaging surface; and (4) in the analysis step, a linear image in the image is extracted, and the particle is analyzed based on the extracted linear image.

[0086] In the particle observation method of the second aspect, in the configuration of the first aspect, the analyzing step may be configured to extract a linear image after reducing noise in the image.

[0087] In the particle observation method of the third aspect, in the configuration of the first or second aspect, the analyzing step may be configured to apply a machine learning model to extract linear images in the image.

[0088] In the particle observation method of the fourth aspect, in the configuration of the third aspect, in the analysis step, any one of a principal component analysis model, an independent component analysis model, a non-negative factorization model, and a deep learning model may be applied as a machine learning model to extract linear images in the image.

[0089] In the particle observation method of the fifth aspect, in the configuration of the third or fourth aspect, in the analysis step, the direction in which the linear image in the image extends may be defined as a first direction, and the direction perpendicular to this first direction may be defined as a second direction, and a machine learning model may be applied to each of the first direction and the second direction to extract the linear image in the image.

[0090] In the particle observation method of the sixth aspect, in the configuration of any one of the first to fifth aspects, the analyzing step may be configured to find the number of linear images in the image as the number of particles.

[0091] In the particle observation method of the seventh aspect, in the configuration of any one of the first to sixth aspects, the analysis step may be configured to obtain an image of the particle based on the luminance value distribution of a linear image in the image.

[0092] In the particle observation method of the eighth aspect, in the configuration of any one of the first to seventh aspects, the analysis step may be configured to determine the particle size based on the luminance value of a linear image in the image.

[0093] In the particle observation method of the ninth aspect, in the configuration of any one of the first to eighth aspects, the imaging step may be configured to use an imaging unit including an electron multiplying or electron bombardment type imaging element.

[0094] In the particle observation method of the tenth aspect, in the configuration of any one of the first to ninth aspects, the imaging step may further include a light source that outputs light to irradiate the particles.

[0095] The embodiments can be used as a particle observation device and a particle observation method that can more accurately observe particles that are moving relatively.

[0096] 1A, 1B...particle observation device, 10...light source, 11...lens, 20A, 20B...imaging optical system, 21...objective lens, 22...tube lens, 23...beam splitter, 30...imaging unit, 31...imaging element, 32...filter, 40...analysis unit, 50...display unit, 91...flow path, 92...stage.

Claims

1. A particle observation device comprising: an imaging optical system that inputs light from a particle and forms an image of the particle; an imaging unit that includes an imaging element having an imaging surface at the position of image formation by the imaging optical system and outputs image data based on the image formed on the imaging surface during an exposure period of the imaging element; and an analysis unit that inputs image data output from the imaging unit and analyzes the particle based on the image represented by this image data, wherein both or one of the imaging optical system and the imaging unit change the relative positional relationship with the particle during the exposure period to move the position where the particle image is formed on the imaging surface, and the analysis unit extracts a linear image from the image and analyzes the particle based on the extracted linear image.

2. The particle observation device according to claim 1, wherein the analysis unit extracts the linear image after reducing noise in the image.

3. The particle observation device according to claim 1 or 2, wherein the analysis unit applies a machine learning model to extract linear images in the image.

4. The particle observation device according to claim 3, wherein the analysis unit applies any one of a principal component analysis model, an independent component analysis model, a non-negative factorization model, and a deep learning model as the machine learning model to extract linear images in the image.

5. A particle observation device as described in claim 3 or 4, wherein the analysis unit defines a direction in which the linear image in the image extends as a first direction and a direction perpendicular to the first direction as a second direction, and applies a machine learning model to each of the first direction and the second direction to extract the linear image in the image.

6. A particle observation device according to any one of claims 1 to 5, wherein the analysis section determines the number of linear images in the image as the number of particles.

7. A particle observation device according to any one of claims 1 to 6, wherein the analysis section determines the image of the particle based on the brightness value distribution of a linear image in the image.

8. A particle observation device according to any one of claims 1 to 7, wherein the analysis unit determines the size of the particle based on the brightness value of a linear image in the image.

9. A particle observation device according to any one of claims 1 to 8, wherein the imaging unit includes an electron multiplying type or electron bombardment type imaging element.

10. The particle observation device according to any one of claims 1 to 9, further comprising a light source that outputs light to irradiate the particles.

11. A particle observation method comprising: an imaging step of using an imaging optical system that inputs light from a particle and forms an image of the particle, and an imaging unit including an imaging element having an imaging surface at the position where an image is formed by the imaging optical system, and outputting image data from the imaging unit based on an image formed on the imaging surface during an exposure period of the imaging element; and an analysis step of inputting the image data output from the imaging unit and analyzing the particle based on the image represented by this image data, wherein in the imaging step, the relative positional relationship between the particle and either or both of the imaging optical system and the imaging unit is changed during the exposure period to move the position where the image of the particle is formed on the imaging surface, and in the analysis step, a linear image is extracted from the image, and the particle is analyzed based on the extracted linear image.

Images