Measuring device and imaging control method

Abstract

The measurement device according to the present invention is provided with: a light emitting unit for emitting light to a fluid; a light receiving unit for performing, on incoming light, a photoelectric conversion utilizing an electronic avalanche effect in a plurality of pixels to obtain a light reception signal; and a control unit for performing a detection process on an object in the fluid on the basis of the light reception signal and, on condition that the object has been detected, for causing execution of an operation to image the object.

Description

[Technical field]

[0001] The present technology relates to a technical field of a measuring device for performing measurements of objects contained in a fluid, such as plankton contained in water, based on captured images, and an imaging control method thereof. [Background technology]

[0002] Conventionally, the measurement of microorganisms in water has been carried out by collecting samples at different depths with a water sampler and testing them on land, which is time-consuming and not very time-saving.

[0003] There is technology to solve this problem by installing a microorganism measurement device with an automatic identification function on an autonomous underwater vehicle (AUV) or underwater drone, as described in Patent Document 1 below, for example.

[0004] However, in the method of Patent Document 1, after passing the sample through a flow cell, imaging and identification are performed regardless of the presence or absence of microorganisms in the sample, which poses the problem of high power consumption for imaging. Due to space restrictions on the exploration vessel and the limited capacity of the onboard batteries, the measuring equipment must be as small as possible and must operate with as little power as possible.

[0005] The following Patent Document 2 discloses a method for detecting the presence or absence of microorganisms in a sample by detecting weak light excited by the microorganisms when a specific light such as a laser light is irradiated, and capturing an image of the sample when the microorganism is detected as a trigger. According to the method of Patent Document 2, it is not necessary to constantly capture an image for measurement regardless of the presence or absence of microorganisms in the sample, and power consumption can be reduced. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2016-95259 A [Patent Document 2] US Patent Application Publication No. 2017-82530 Summary of the Invention [Problem to be solved by the invention]

[0007] However, in the method of Patent Document 2, a single (single pixel) photomultiplier tube is used as a light receiving sensor for detecting the presence or absence of microorganisms, and imaging is performed in response to detection of return light from the microorganism in the single pixel. Specifically, imaging is performed in response to detection of return light of a specific wavelength in the single pixel. Since the start condition for imaging is simply detection of return light of a specific wavelength, there is a high possibility of reacting to something other than microorganisms, and there is a problem with the accuracy of detection of the presence or absence of microorganisms. If the accuracy of detection of the presence or absence of microorganisms is low, imaging may be performed even when there are no microorganisms in the sample, making it difficult to achieve power saving.

[0008] The present technology has been made in consideration of the above circumstances, and aims to reduce power consumption in a measuring device that measures targets in a fluid, such as a measuring device for microorganisms in water. [Means for solving the problem]

[0009] A first measuring device relating to the present technology includes a light emitting unit that emits light toward a fluid, a light receiving unit that performs photoelectric conversion of the incident light using an electron avalanche phenomenon using a plurality of pixels to obtain a received light signal, and a control unit that performs a detection process for an object in the fluid based on the received light signal, and executes an imaging operation of the object on the condition that the object is detected. According to the above configuration, rather than constantly capturing images of the object, the detection of the object based on the light receiving signal from the light receiving unit is used as a trigger to reduce power consumption related to imaging, making it possible to detect the presence or absence of an object based on the light receiving signals of multiple pixels.

[0010] In the first measuring device according to the present technology described above, the light receiving section may have a SPAD element as a photoelectric conversion element. This eliminates the need to use a large-sized, high-power-consumption photoelectric conversion element such as a photomultiplier tube in the light receiving section.

[0011] In the above-described first measuring device according to the present technology, it is considered that the control unit is configured to perform the detection process of the object based on image features of a light-receiving reaction portion in the light-receiving unit. The "image characteristics of the light-receiving reaction part" referred to here means characteristics of an image composed of at least one pixel that has reacted to light, such as the image size and position of the light-receiving reaction part, the wavelength of the received light, and the value of the received light signal, as the light-receiving reaction part.

[0012] In the first measuring device related to the present technology described above, it is considered that the control unit is configured to prevent imaging of the imaging range corresponding to the light-receiving reaction part if the image features of the light-receiving reaction part do not match the specified image features. This makes it possible to prevent objects other than those having the designated image characteristics from being captured indiscriminately.

[0013] In the first measuring device according to the present technology described above, it is considered that the control unit is configured to detect a pixel position and an image size of the light receiving reaction portion as the image feature. This makes it possible to specify the pixel range in which the object is captured, that is, the pixel range in which imaging should be performed, for the image sensor that captures the object.

[0014] In the first measuring device related to the present technology described above, it is considered that the control unit is configured to control an imaging sensor that images the object so that an imaging operation is performed only for a portion of the pixel range in which the object is captured. This makes it possible to reduce power consumption relating to imaging, compared to the case where imaging is performed over the entire pixel range of the image sensor.

[0015] In the first measuring device according to the present technology described above, the control unit may be configured to match the captured image with a template image for the partial pixel range. By performing matching based on the captured image, it becomes possible to appropriately identify the type of object.

[0016] In the first measuring device related to the present technology described above, it is possible to configure the control unit to perform class identification of an object captured in a captured image for the portion of the pixel range, and to perform the matching using a template image of the identified class from among the template images prepared for each class. By narrowing down the classes in this way before performing image matching, it is possible to improve the efficiency of the image matching process.

[0017] In the first measuring device related to the present technology described above, the control unit may be configured to set a bounding box as a range surrounding the object from an captured image of the portion of the pixel range in a reference frame, which is a predetermined frame after the object is detected based on the light receiving signal, and to set an ROI that is an area that contains the bounding box and is larger than the bounding box, and in a frame after the reference frame, to set the bounding box of the object within the ROI set in the immediately preceding frame, and to set the ROI based on the bounding box. This makes it possible to track the target even if the target moves within the captured image.

[0018] In the first measuring device related to the present technology described above, it is possible to configure the sensor that functions as the light receiving unit and the imaging sensor that images the object based on the control of the control unit as separate entities. This makes it possible to use existing sensors as both the sensor that functions as the light receiving unit (a sensor that performs photoelectric conversion using the electron avalanche phenomenon) and the imaging sensor that captures an image of the target object.

[0019] In the first measuring device related to the present technology described above, it is possible to configure the device with a single sensor that has a function as the light receiving unit and a function of capturing an image of the object based on the control of the control unit. When the sensors are separate, it is necessary to provide a spectroscopic means for distributing the light from the fluid to each sensor. However, by providing an integrated sensor, there is no need to provide such a spectroscopic means.

[0020] The first measuring device according to the present technology described above may be configured to include a flow cell in which the fluid is sampled from an internal flow path, and the control unit may be configured to, after completion of the imaging operation, cause a fluid other than the fluid serving as a sample to flow into the flow path to clean the flow cell. This makes it possible to prevent erroneous measurements, such as measuring an object that has already been measured again.

[0021] In the first measuring device related to the present technology described above, it is considered that the control unit is configured to perform a detection process of the object based on the light receiving signal after the other fluid flows into the flow path. This makes it possible to check whether any objects remain after cleaning.

[0022] The imaging control method relating to the present technology is an imaging control method for a measuring device that has at least an emission unit that emits light toward a fluid and a light receiving unit that performs photoelectric conversion of incident light using an electron avalanche phenomenon with a plurality of pixels to obtain a received light signal, the imaging control method performing a detection process for an object in the fluid based on the received light signal, and executing an imaging operation of the object on the condition that the object is detected. With such an imaging control method, the same effect as that of the first measuring device according to the present technology described above can be obtained.

[0023] A second measuring device related to the present technology includes a light-emitting unit that emits light toward a fluid, an imaging sensor that performs photoelectric conversion on the incident light using a plurality of pixels to obtain a received light signal, and a control unit that performs a detection process for an object in the fluid based on the received light signal, and, on condition that the object is detected, causes the imaging sensor to perform an imaging operation of the object, wherein the control unit controls the imaging operation of the object so that an imaging operation is performed only for a portion of the pixel range in which the object is captured. According to the above configuration, it is possible to reduce power consumption related to imaging by not capturing an image of an object all the time, but by using the detection of an object based on a light receiving signal as a trigger to capture the image, and this also reduces power consumption related to imaging more than when imaging is performed for the entire pixel range of the imaging sensor.

[0024] In the second measuring device according to the present technology described above, the control unit can be configured to match the captured image with a template image for the partial pixel range. By performing matching based on the captured image, it becomes possible to appropriately identify the type of object.

[0025] Furthermore, in the second measuring device related to the present technology described above, the control unit can be configured to perform class identification of an object captured in a captured image for the partial pixel range, and to perform the matching using a template image of the identified class from among the template images prepared for each class. By narrowing down the classes in this way before performing image matching, it is possible to improve the efficiency of the image matching process. [Brief description of the drawings]

[0026] [Figure 1] 1A and 1B are diagrams for explaining an example of a device configuration of a measurement device according to an embodiment. [Diagram 2]1 is a block diagram showing an example of the internal configuration of a measurement device according to an embodiment; [Diagram 3] FIG. 13 is a diagram showing an example of an image captured when illuminated by a backlight source. [Figure 4] FIG. 13 is a diagram showing an example of an image captured when illuminated by a front light source. [Diagram 5] 1 is an explanatory diagram of a target object detection process according to an embodiment; [Figure 6] FIG. 13 is a diagram showing an example of noise caused by dark current. [Figure 7] 11A and 11B are diagrams illustrating a noise countermeasure in the object detection process. [Figure 8] FIG. 11 is a diagram showing an example of definition information. [Figure 9] FIG. 11 is a diagram showing another example of definition information. [Figure 10] FIG. 13 is a diagram showing an example of a SPAD sensor in which a different wavelength filter is provided for each pixel. [Figure 11] FIG. 13 is an explanatory diagram of an example in which a SPAD sensor is provided for each wavelength. [Figure 12] FIG. 2 is a diagram showing a schematic example of an image received by a SPAD sensor. [Figure 13] 11 is an explanatory diagram of an example of a detection process when the sensor in FIG. 10 is used. [Figure 14] 12 is an explanatory diagram of an example of a detection process when the sensor of FIG. 11 is used. [Figure 15] 10A to 10C are diagrams for explaining a process of determining an object based on a captured image in an embodiment. [Figure 16] FIG. 2 is an explanatory diagram of tracking of an object in the embodiment. [Figure 17] 1 is a flowchart showing a process flow from the start of measurement to the end of measurement in an embodiment. [Figure 18] 11 is a flowchart illustrating a processing procedure for realizing tracking of an object. [Figure 19] 18 is a flowchart of the cleaning process (S117) in FIG. 17. [Figure 20]FIG. 1 is an explanatory diagram of an example in which a SPAD sensor and an image sensor are formed on separate substrates. [Figure 21] FIG. 1 is an explanatory diagram of an example in which a SPAD sensor and an image sensor are formed on a common substrate. [Figure 22] FIG. 1 is an explanatory diagram of an example in which a functional section serving as a SPAD sensor and a functional section serving as an image sensor are formed within a common semiconductor chip. [Diagram 23] 21 is a diagram showing an example of the configuration of a measurement device in which a mirror is omitted on the optical path of a SPAD sensor in correspondence with the case where the configuration shown in FIG. 20 is adopted. [Figure 24] FIG. 2 is an explanatory diagram of an example of a single sensor that functions as a SPAD sensor and as an imaging sensor. [Diagram 25] 11 is an explanatory diagram of another example of a single sensor having a function as a SPAD sensor and a function as an imaging sensor. FIG. [Figure 26] FIG. 13 is a block diagram showing an example of the internal configuration of an image sensor as a modified example. [Figure 27] FIG. 13 is a diagram showing an example of the internal configuration of a measurement device as a first modified example. [Figure 28] 1 is a flowchart showing a process for measuring microplastics in a first modified example. [Figure 29] FIG. 11 is a block diagram showing an example of the internal configuration of a measurement device as a second modified example. [Diagram 30] 13 is a flowchart showing a process flow from the start of measurement to the end of measurement in a second modified example. [Diagram 31] FIG. 13 is a block diagram showing an example of the internal configuration of a measurement device as a third modified example. [Diagram 32] FIG. 13 is a block diagram showing an example of the internal configuration of a measurement device as a fourth modified example. [Diagram 33] FIG. 1 is a diagram showing an example of a captured image in which zooplankton and phytoplankton are captured. [Diagram 34] FIG. 13 is a diagram showing an example of an ROI calculated for zooplankton and an ROI calculated for phytoplankton. [Diagram 35]FIG. 13 is a block diagram showing an example of the internal configuration of a measurement device as a sixth modified example. [Diagram 36] 13 is a flowchart showing a process flow from the start to the end of measurement in a sixth modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] The embodiments will be described below in the following order. <1. Configuration of the measuring device> <2. Measurement method as an embodiment> <3. Processing Procedure> <4. Sensor structure> <5. Image Sensor> <6. Variations> [6-1. First modified example] [6-2. Second modified example] [6-3.Third modified example] [6-4. Fourth modified example] [6-5. Fifth Modification] [6-6. Sixth Modification] <7. Summary of the embodiment> <8. This Technology>

[0028] <1. Configuration of the measuring device> First, a configuration of a measurement device 1 according to an embodiment of the present technology will be described. The measuring device 1 is a device that measures objects contained in a fluid taken in as a sample, such as microorganisms contained in seawater. Specifically, the measuring device 1 of this example takes in seawater or lake water as a sample and measures objects such as plankton contained in the sample. Here, measurement is a concept that includes at least one of identifying the number, type, or characteristics of objects, or recording or storing captured images of the objects.

[0029] FIG. 1 is a diagram for explaining an example of the device configuration of a measurement device 1. As shown in FIG. As shown in the figure, the measuring device 1 may take the form of a towed type towed by a ship sailing on the sea or a lake, a stationary type installed in the sea or in a lake, or a submersible type installed in a submersible sailing in the sea or in a lake.

[0030] FIG. 2 is a block diagram showing an example of the internal configuration of the measurement device 1. As shown in the figure, the measuring device 1 includes a sample container 2, a cleaning liquid container 3, a sample switching unit 4, a flow cell 5, a sample discharge unit 6, a front light source 7, a rear light source 8, a detection light source 9, a SPAD (Single Photon Avalanche Diode) sensor 10, an imaging sensor 11, a half mirror 12, a mirror 13, a lens 14, a lens 15, a control unit 16, a memory unit 17, and a communication unit 18.

[0031] The sample container 2 is a container for containing a fluid (seawater or lake water in this example) as a sample, and contains the sample taken in from outside the device via a sample inlet Mi. The cleaning liquid container 3 is a container that contains a cleaning liquid for cleaning the flow path in the flow cell 5. The sample switching unit 4 switches the fluid to be flowed into the flow path in the flow cell 5 between the sample from the sample container 2 and the washing liquid from the washing liquid container 3 .

[0032] The flow cell 5 functions as a sample container, and a fluid as a sample is sampled into a flow path formed inside the flow cell 5. As will be described later, when the sample switching unit 4 is switched to the cleaning liquid container 3 side, cleaning liquid flows into the flow path of the flow cell 5.

[0033] The sample discharge section 6 has a pump for discharging fluid, and when the pump is driven, the fluid in the flow channel of the flow cell 5 is discharged through a sample discharge port Mo located outside the device. In this example, the flow path from the sample container 2 via the sample switching unit 4 → flow cell 5 to the sample discharge unit 6, and the flow path from the cleaning liquid container 3 via the sample switching unit 4 → flow cell 5 to the sample discharge unit 6 are each made into a continuous flow path, and the flow of sample from the sample container 2 to the flow cell 5, and the flow of cleaning liquid from the cleaning liquid container 3 to the flow cell 5 are achieved by driving the pump in the sample discharge unit 6.

[0034] The front light source 7 is a light source for illuminating the fluid in the flow cell 5 in response to imaging by the imaging sensor 11. Here, the "front" refers to the surface on the imaging sensor 11 side with respect to the position of the flow cell 5. In this example, the front light source 7 is an annular light source, which prevents interference with imaging by the imaging sensor 11 and illuminates the sample from the front side of the flow cell 5 at an oblique angle. The rear light source 8, like the front light source 7, is a light source for illuminating the fluid in the flow cell 5 in response to imaging by the imaging sensor 11, and is positioned on the opposite side of the flow cell 5 from the front light source 7.

[0035] Here, the roles of the front light source 7 and the rear light source 8 will be explained. FIG. 3 shows an example of an image captured when illuminated by a back light source 8, and FIG. 4 shows an example of an image captured when illuminated by a front light source 7. The backlight source 8 is used for bright-field imaging. Light transmitted through the sample is received by the image sensor 11, which is similar to the method used in general microscopes. The illumination light is directly incident on the lens 15, so the background becomes bright. On the other hand, the front light source 7 is used for dark-field imaging. Light is applied to the sample from an oblique side, and the scattered light and reflected light from the object are received by the image sensor 11. Even transparent objects can be measured with high contrast and in detail. In this case, the illumination light does not directly enter the lens 15, so the background is dark.

[0036] 2, a detection light source 9 emits light for detecting an object in a sample sampled in a flow cell 5. For example, a semiconductor laser or the like is used as the detection light source 9. As shown in the figure, the light emitted from the detection light source 9 is reflected by a half mirror 12 and irradiated onto the fluid sampled in the flow path in the flow cell 5.

[0037] The SPAD sensor 10 functions as a sensor for detecting an object in the fluid in the flow cell 5. In the measurement device 1 of the embodiment, a pixel array in which a plurality of light detection pixels are arranged is used to detect the weak light of microorganisms and particles. SPAD is considered as one of the technologies of such light detection pixels. In the SPAD, avalanche amplification occurs when one photon enters a PN junction region of a high electric field while a voltage larger than the breakdown voltage is applied. At that time, the presence, position, size, etc. of microorganisms or particles in the flow cell 5 can be identified by detecting the position and timing of the pixel through which a current flows instantaneously. The SPAD sensor 10 has a SPAD element that performs photoelectric conversion of incident light using the electron avalanche phenomenon. The electron avalanche phenomenon in the SPAD element is a type of phenomenon known as the internal photoelectric effect. The internal photoelectric effect is a phenomenon in which the number of conduction electrons inside a semiconductor or insulator increases when the semiconductor or insulator is irradiated with light. As is well known, a SPAD element is an element that has a light receiving resolution in photon units, in other words, an element that can distinguish the presence or absence of received light in photon units.

[0038] The SPAD sensor 10 in this example has a configuration in which a plurality of pixels, each having a SPAD element, are arranged two-dimensionally. Light emitted from an object in the fluid within the flow cell 5 enters the SPAD sensor 10 via a half mirror 12, a mirror 13, and a lens 14.

[0039] The image sensor 11 is configured as an image sensor of, for example, a CCD (Charge Coupled Device) type or a CMOS (Complementary Metal Oxide Semiconductor) type, and has a plurality of pixels, each having a photoelectric conversion element, arranged two-dimensionally. The photoelectric conversion element of each pixel of the image sensor 11 does not perform photoelectric conversion using the electron avalanche phenomenon, but employs a photoelectric conversion element used in general imaging, such as a photodiode. In other words, it is a photoelectric conversion element with a lower light receiving resolution than a SPAD element. The imaging sensor 11 captures an image of the flow path in the flow cell 5 (an image including at least the flow path within an imaging field of view). Light (image light) from the flow cell 5 passes through a half mirror 12 and enters the imaging sensor 11 via a lens 15.

[0040] The control unit 16 is configured with, for example, a microcomputer having a CPU (Central Processing Unit), ROM (Read Only Memory), and RAM (Random Access Memory), and performs overall control of the measurement device 1. For example, the control unit 16 performs switching control of the sample switching unit 4, light emission drive control of the front light source 7 and the back light source 8, drive control of the pump in the sample discharge unit 6, light emission drive control of the detection light source 9, etc. The control unit 16 also reads data stored in the storage unit 17, stores data in the storage unit 17, and exchanges various data with external devices via the communication unit 18. For example, the storage unit 17 is configured with a non-volatile memory. The communication unit 18 performs wired or wireless data communication with the external devices. In addition, the control unit 16 in this example performs target object detection processing based on the light reception signal from the SPAD sensor 10, various image analysis processing based on the image captured by the imaging sensor 11, and the like, but these processing will be described later.

[0041] <2. Measurement method as an embodiment> A measurement method according to an embodiment will be described. FIG. 5 is an explanatory diagram of the detection process of a target substance in a sample. First, as shown in FIG. 5A, in a state where a fluid serving as a sample is sampled in the flow cell 5, the detection light source 9 is caused to emit light to irradiate the sample with the detection light. During measurement, the fluid as a sample is moved in the discharge direction (toward the sample outlet Mo) in the flow path of the flow cell 5. That is, the pump of the sample outlet 6 is driven to gradually move the fluid in the discharge direction.

[0042] As shown in FIG. 5A, when an object in a sample does not appear within the field of view, the SPAD sensor 10 does not receive the return light from the object. On the other hand, when an object appears within the field of view as shown in FIG. 5B, the SPAD sensor 10 receives return light from the object based on the light emitted from the detection light source 9. Here, if the object is phytoplankton, the returning light is light excited from the phytoplankton by a fluorescent reaction based on the illumination light from the detection light source 9. If the object is zooplankton, the returning light is scattered light generated by the zooplankton based on the illumination light from the detection light source 9.

[0043] In the measurement method of this embodiment, as a result of irradiating the sample with detection light in this manner, if returning light from the sample side is received by the SPAD sensor 10, imaging operation is performed by the image sensor 11. In other words, if no returning light is received, imaging operation by the image sensor 11 is not performed, thereby reducing power consumption related to imaging. In this case, by using the SPAD sensor 10 as the light receiving section for receiving the returning light, the light receiving section can be made smaller and consume less power than when a conventional light receiving section using a photomultiplier tube is used.

[0044] Here, in order to appropriately determine whether or not return light from an object has been received, the effect of noise caused by dark current must be taken into consideration. In order to appropriately eliminate the influence of such noise, the present embodiment employs the following method for the process of detecting an object based on the light reception signal of the SPAD sensor 10.

[0045] FIG. 6 shows an example of noise caused by dark current. As shown in the figure, the noise occurs randomly in individual pixels. For this reason, in this example, as shown in FIG. 7, when light reception is recognized in an area of ​​a plurality of adjacent pixels, the area is determined to be a light receiving area of ​​return light from an object. FIG. 7A shows an example of a light receiving area (black pixels in the figure) of the light returning from phytoplankton, and FIG. 7B shows an example of a light receiving area of ​​the light returning from zooplankton. When reception of return light from an object is confirmed as shown in these figures, an image is captured using the image sensor 11. In addition, in order to eliminate the influence of noise, a noise countermeasure can be taken by making the detection light source 9 emit pulsed light (blinking light) and synchronizing the light receiving timing of the SPAD sensor 10 with this.

[0046] Here, for the measurement device 1, information defining an object to be measured (hereinafter referred to as "definition information I1") is set in advance. 8 and 9 show examples of the definition information I1. The definition information I1 may be information including the class particle name, size, wavelength component, and image data of the object, as exemplified in FIG. 8. For example, if the object is plankton, the class particle name is determined as the specific name information. The size information is information indicating the size classification of the object, and is information having a range, such as "20 μm to 40 μm" as shown in the figure. The wavelength component information is information defining the wavelength component of the return light corresponding to the irradiation light of the detection light source 9. The image data is image data (image data) of the object. For this image data, data actually captured of one individual object may be used, or representative image data obtained by machine learning from captured images of multiple individuals may be used.

[0047] Furthermore, the definition information I1 may be information that specifies a specific object as in FIG. 8, or may be information that specifies only some features as in the example shown in FIG. FIG. 9 shows an example of definition information I1 that specifies only the size and wavelength components of an object to be measured.

[0048] In the measurement device 1, the definition information I1 is stored in, for example, the storage unit 17 shown in FIG.

[0049] In order to detect the presence or absence of a target object in a sample in accordance with the definition information I1 as shown in FIGS. 8 and 9, the SPAD sensor 10 must be able to distinguish the wavelength of the received light. The configuration of a SPAD sensor 10 for realizing such a wavelength discrimination function will be described with reference to FIGS. FIG. 10 shows an example in which a different wavelength filter is provided for each pixel of the SPAD sensor 10. In the figure, the pixel marked with "A" is a pixel provided with a wavelength filter of 550 nm (hereinafter referred to as "pixel A"), the pixel marked with "B" is a pixel provided with a wavelength filter of 600 nm (hereinafter referred to as "pixel B"), the pixel marked with "C" is a pixel provided with a wavelength filter of 650 nm (hereinafter referred to as "pixel C"), and the pixel marked with "D" is a pixel provided with a wavelength filter of 700 nm (hereinafter referred to as "pixel D"). In this case, in the SPAD sensor 10, pixels A, B, C, and D are arranged so that they exist at every other pixel in both the horizontal direction (row direction) and the vertical direction (column direction). With this configuration, it is possible to identify which wavelength of light is received at which pixel position.

[0050] FIG. 11 shows a configuration example corresponding to the case where a plurality of SPAD sensors 10 are used. 11A, 11B, 11C, and 11D, a SPAD sensor 10 consisting of only A pixels (550 nm), a SPAD sensor 10 consisting of only B pixels (600 nm), a SPAD sensor 10 consisting of only C pixels (650 nm), and a SPAD sensor 10 consisting of only D pixels (700 nm) are used. In this case, an optical system is configured to split the return light from the flow cell 5 side and guide it to the light receiving surface of each SPAD sensor 10.

[0051] Fig. 12 shows a schematic example of a light receiving image by the SPAD sensor 10. In the figure, the white hazy, roughly circular parts (three parts in the illustrated example) show the light receiving parts. Also, in the figure, two white circles show an example of the correspondence between the size of the light receiving surface of the SPAD sensor 10 and the actual size of the subject. Specifically, the small white circle shows a size of 20 μm, and the large white circle shows a size of 40 μm. Here, the received light image shown in FIG. 12 is an example of an image received by the SPAD sensor 10 without a wavelength filter.

[0052] Figure 13 shows an example of light-receiving response pixels (shown in black in the figure) when a light-receiving operation is performed on the same subject as in Figure 12 for a SPAD sensor 10 in which pixels A, B, C, and D are arranged in a mixed manner as shown in Figure 10 above. FIG. 13 illustrates a case where a light receiving action is performed on the same subject as in FIG. 12, and light receiving reactions occur in three places similar to those in FIG. 8 is set as the definition information I1 of the object, the size of the object is 20 nm to 40 nm, and the wavelength component is 650 nm±10 nm. In the figure, a single B pixel and a single D pixel each have a light-receiving response, but these light-receiving response regions differ from the wavelength condition in the definition information I1 and do not satisfy the condition with multiple adjacent pixels, so they are not determined to be light-receiving regions of the object. On the other hand, in the figure, there is an area where a light-receiving reaction is obtained from multiple adjacent C pixels (wavelength 650 nm). Specifically, the light-receiving reaction area of ​​this C pixel is an area of ​​3 x 3 = 9 pixels. For the sake of explanation, this area of ​​3 x 3 = 9 pixels is assumed to be an area of ​​20 μm or more and 40 μm or less in size when converted into the actual subject size. In this way, if a light-receiving reaction area satisfies the condition of being an area of ​​multiple adjacent pixels and also satisfies the size and wavelength component conditions specified in the definition information, the light-receiving reaction area is determined to be a light-receiving area of ​​the object.

[0053] Such determination of the light receiving region of the object can also be performed in the same manner when a plurality of SPAD sensors 10 are used as shown in FIG. Specifically, when a light receiving operation is performed on the same subject as in the case of Figure 12, each SPAD sensor 10 will obtain a light receiving response as illustrated in Figure 14, but the light receiving response area in the SPAD sensor 10 with only B pixel and the light receiving response area in the SPAD sensor 10 with only D pixel each differ from the wavelength conditions in the definition information I1 and do not satisfy the conditions with multiple adjacent pixels, and are therefore not determined to be a light receiving area of ​​the object. For the light receiving reaction area in a SPAD sensor 10 having only C pixels, if the area size of the 2 x 2 = 4 pixels shown in the figure is 20 μm or more and 40 μm or less when converted into the actual subject size, then it satisfies the wavelength and size conditions in the definition information I1, and also satisfies the conditions with multiple adjacent pixels, so that the light receiving reaction area can be determined to be the light receiving area of ​​the object.

[0054] Hereinafter, the light receiving region (light receiving reaction region) of the object determined according to the conditions based on the definition information I1 as described above will be referred to as the "light receiving region Ats."

[0055] Although an example in which wavelength filters are provided for all pixels of the SPAD sensor 10 is shown in FIG. 10 and FIG. 11, some pixels may be mixed in which no wavelength filter is provided.

[0056] In this example, when the light receiving area Ats is identified based on the light receiving image by the SPAD sensor 10 as described above (i.e., the presence of an object matching the size and wavelength conditions of the target object is confirmed), an imaging operation is performed by the imaging sensor 11.

[0057] At this time, if the definition information I1 includes image data of the object as shown in FIG. 8, a determination is made as to whether or not it is an object based on the image captured by the image sensor 11 and the image data.

[0058] FIG. 15 is a diagram for explaining such a process of determining an object based on a captured image. 15A illustrates an example of the relationship between the light receiving area Ats on the SPAD sensor 10 and the imaging area Ati on the image sensor 11. Here, the imaging area Ati refers to a pixel area of ​​the image sensor 11 that can capture the same subject as the subject captured by the light receiving area Ats.

[0059] In the process of determining an object based on a captured image, the image of the imaging area Ati specified from the light receiving area Ats is compared with the target image data (i.e., the image data included in the definition information) as shown in Fig. 15B. If the image comparison results in a match with the image data, a final determination result is obtained that the object captured in the imaging area Ati (i.e., the object captured in the light receiving area Ats) is the object.

[0060] In this example, for a subject determined to be a target object, an imaging operation based on predetermined measurement setting information I2 is performed. The measurement setting information I2 is information that defines various conditions related to the measurement of the object. Specifically, the measurement setting information I2 in this example includes the following information: "measurement execution conditions," "sample injection speed," "imaging rules," and "illumination settings."

[0061] "Measurement execution conditions" is information that defines the conditions for performing measurements, such as "measurement for 10 minutes every 200 m depth" or "measurement for 5 minutes at electrical conductivity of 80 mS / cm or more." Here, electrical conductivity is an index of the mineral content in water. Electrical conductivity can be used when investigating the relationship between the mineral content and the microorganisms that inhabit it. For example, when measuring microorganisms in a part with a large amount of minerals, the electrical conductivity conditions as in the above example are set. When electrical conductivity is used as a measurement condition, a sensor for detecting the electrical conductivity of seawater or lake water is externally attached to the measurement device 1. Information detected by the external sensor is input to the control unit 16 via the communication unit 18 shown in FIG.

[0062] "Sample injection rate" is information that defines the injection rate of the sample into the flow cell 5, and is, for example, information such as "0.5 ml / min." The "imaging rule" is information that defines a rule for imaging an object using the imaging sensor 11, and is, for example, information such as "still image imaging" or "video imaging." Here, the video imaging rule can be information that specifies the end condition and frame rate of video imaging, such as "image the object at 20 fps until it leaves the flow cell 5." The "illumination setting" is information that defines the illumination used when capturing an image of an object using the imaging sensor 11, and in this example, is definition information for the front light source 7 and back light source 8 described above. For example, it is information such as "dark field imaging (front light source 7)" and "bright field imaging (back light source 8)". Note that both the front light source 7 and back light source 8 can be used for illumination when capturing an image.

[0063] Such measurement setting information I2 is stored in, for example, the storage unit 17, and the measurement device 1 measures the object in accordance with this measurement setting information I2.

[0064] Here, in the measuring device 1, detection of an object based on a light receiving signal by the SPAD sensor 10 is used as a trigger to capture an image using the imaging sensor 11, thereby reducing power consumption related to imaging. However, in this example, in order to further reduce power consumption, the imaging operation of the object is performed only on the imaging area Ati. Specifically, in the example of Figure 15, when an imaging operation is started in response to the light receiving area Ats being identified based on the light receiving image by the SPAD sensor 10, rather than performing an imaging operation using all the pixels of the imaging sensor 11, an imaging operation is performed using only the imaging area Ati determined from the light receiving area Ats. As a result, the imaging operation for measurement is performed only on a necessary portion of the pixel range, making it possible to reduce power consumption.

[0065] Incidentally, measuring objects is a concept that includes identifying the number, type, and characteristics of objects, as mentioned above. However, in order to properly identify (count) the number of each type of object, it is essential to properly manage the objects detected within the imaging field of view, dividing them into counted objects and uncounted objects. Therefore, in this example, once an object has been recognized, tracking is continued until it goes out of the imaging field of view, so that duplicate counting is not performed.

[0066] FIG. 16 is an explanatory diagram of the tracking of an object in the embodiment. First, the frame F1 shown in Fig. 16A means a frame at the stage where the imaging area Ati is specified from the light receiving area Ats as described above in Fig. 15A. As can be understood from the above description, in the frame F1, an imaging operation is performed only on the imaging area Ati. Then, as described in Fig. 15B, the image of the imaging area Ati is compared (image matching) with the image data in the definition information I1. When an object is recognized by this image matching, a bounding box 20 is calculated as a range surrounding the area of ​​the object, as shown in FIG. 16A.

[0067] After the bounding box is calculated, a ROI (Region Of Interest) 21 is calculated based on the bounding box, as shown in FIG. 16B. This ROI 21 is calculated, for example, by enlarging (ax×by) the vertical and horizontal sizes (x×y) of the bounding box 20. The enlargement scales a and b can be set separately for the vertical and horizontal dimensions, and the enlargement ratio may be fixed or variable.

[0068] Frame F2 shown in FIG. 16C is the frame following frame F1. For frame F2, imaging is performed only on the ROI 21 calculated in the previous frame, frame F1. If the object gradually moves to the right on the paper, the position of the object in frame F2 will be shifted to the right on the paper from the position in frame F1. However, since the ROI 21 is calculated as a range that expands the vertical and horizontal sizes of the bounding box 20, it becomes possible to capture the object within the ROI 21 in frame F2 as shown in the figure.

[0069] In frame F2, the captured image of ROI21 calculated in the previous frame is subjected to recognition processing of the object in the image, for example by performing image analysis based on the image data of definition information I1, and the bounding box 20 of the object is calculated. Then, in frame F2, an ROI 21 is calculated for the bounding box 20 thus newly calculated (FIG. 16D).

[0070] 16E, an imaging operation is performed targeting only the ROI 21 calculated in frame F2 in this manner. In this case, too, the ROI 21 is set to a range obtained by enlarging the vertical and horizontal sizes of the bounding box 20, so that the target can be captured within the ROI 21 even if the target moves in a certain direction. Although not shown in the figure, even in frames F3 and after, image analysis is performed on the captured image of the ROI21 calculated in the previous frame to perform object recognition processing, and the bounding box 20 for the recognized object is calculated, and the ROI21 is calculated based on the calculated bounding box 20.

[0071] The above tracking method can be said as the following method: In other words, in a reference frame (frame F1 in this example) which is a predetermined frame after the target object is detected based on the light reception signal by the SPAD sensor 10, a bounding box 20 is set as a range surrounding the target object, and an ROI 21 is set as an area that includes the bounding box 20 and is larger than the bounding box 20. Furthermore, in a frame after the reference frame, a bounding box 20 of the target object is set within the ROI 21 set in the immediately preceding frame, and an ROI 21 based on the bounding box 20 is set. By using such a method, it is possible to track the target object even if the target object moves within the captured image. At this time, the captured image required in each frame for tracking the target is only the captured image of the ROI 21. For this reason, in this example, as described above, imaging is performed in each frame only for the ROI 21 calculated in the immediately preceding frame. This makes it possible to reduce power consumption related to imaging for tracking when tracking a target to prevent erroneous counting.

[0072] In the above description, an example has been given in which the bounding box 20 is enlarged to form a rectangular region as the ROI 21, but the ROI 21 is not limited to a rectangular region. For example, the ROI 21 may be calculated in a shape other than a rectangle using semantic segmentation, that is, object area detection results at the pixel level.

[0073] Here, in the measuring device 1, a process for cleaning the flow cell 5 with the cleaning liquid contained in the cleaning liquid container 3 shown in FIG. 2 is also performed, which will be described later with reference to the flowchart of FIG.

[0074] <3. Processing Procedure> Next, an example of a specific process to be executed to realize the measurement method according to the embodiment described above will be described with reference to the flowcharts of FIGS. FIG. 17 is a flowchart showing the flow of processing from the start of measurement to the end of measurement. The processes shown in FIGS. 17 to 19 are executed by the control unit 16 shown in FIG. 2 based on a program stored in a predetermined storage device such as an internal ROM.

[0075] 17, in step S101, the control unit 16 waits for the measurement start condition to be satisfied. That is, the control unit 16 waits until the condition specified in the "measurement execution condition" in the measurement setting information I2 described above is satisfied. Note that in the above explanation, the depth and electrical conductivity conditions are specified as examples of the "measurement execution condition", but these depth and electrical conductivity are input to the measurement device 1 from an external sensor. Specifically, they are input via the communication unit 18.

[0076] When the measurement start condition is met, the control unit 16 proceeds to step S102 to perform sample injection start processing. That is, the control unit 16 controls the sample switching unit 4 shown in Fig. 2 to switch to the sample container 2 side, and instructs the sample discharge unit 6 to drive the pump, thereby starting the injection of the sample into the flow cell 5. At this time, the pump is driven according to the information on "sample injection speed" in the measurement setting information I2 described above.

[0077] In step S103 following step S102, the control unit 16 performs processing to turn on the detection light source 9, and in the next step S104, issues a light reception instruction to the SPAD sensor 10. That is, the control unit 16 causes the SPAD sensor 10 to perform a light reception operation to obtain one received-light image. Then, in step S105 following step S104, the control unit 16 performs processing to turn off the detection light source 9, and in the next step S106, obtains the received-light image.

[0078] In step S107 following step S106, the control unit 16 determines whether or not there is a light-receiving region that meets the conditions (i.e., the light-receiving region Ats described above). As will be understood from the above explanation, in this example, the light-receiving region is determined to be the light-receiving region Ats when it meets not only the condition that a light-receiving reaction region is obtained from a plurality of adjacent pixels, but also the conditions of wavelength and size defined in the definition information I1.

[0079] If it is determined in step S107 that there is no light receiving region that meets the condition, the control unit 16 returns to step S103. As a result, the light irradiation by the detection light source 9 and the light receiving operation by the SPAD sensor 10 are repeatedly executed until the light receiving region Ats is determined.

[0080] In step S107, when it is determined that there is a light receiving region that meets the condition, the control unit 16 proceeds to step S108 and calculates the imaging region Ati. That is, the imaging region Ati is calculated based on the light receiving region Ats. Then, in step S109 following step S108, the control unit 16 causes the imaging sensor 11 to perform partial imaging of the imaging region Ati. This partial imaging, i.e., imaging operation targeting only the imaging region Ati, can be, for example, imaging operation in which the readout of accumulated charge signals is performed only for a partial pixel range of the imaging region Ati. Alternatively, partial imaging can be imaging operation in which A / D conversion of charge signals read out from each pixel is performed only for a partial pixel range. In this embodiment, imaging operation targeting a portion of the pixel range means that for at least a portion of the processing from the start of light reception to the acquisition of an imaging image signal in the form of a digital signal, processing is limited to a portion of the pixel range rather than the entire pixel range.

[0081] In addition, with respect to the partial imaging in step S109, it is considered that the lighting control is performed according to the information of "lighting setting" in the measurement setting information I2 described above. Alternatively, the lighting control in the partial imaging in step S109 can be performed according to designation information separate from the measurement setting information I2.

[0082] In step S110 following step S109, the control unit 16 performs a process of acquiring a partial captured image from the imaging sensor 11, and in the next step S111, performs matching with a target template image. That is, as described above in Fig. 15B, an image comparison is performed between the partial captured image for the imaging region Ati and the image data in the definition information I1, and a process of determining whether or not the subject captured in the imaging region Ati is the object defined in the definition information I1 is performed.

[0083] In step S112 following step S111, the control unit 16 determines whether or not the particle is a target particle. That is, based on the result of the matching process in step S111, it determines whether or not the subject captured in the imaging area Ati is an object. In step S112, if it is determined that the particle is not a target particle (i.e., not an object of interest), the control unit 16 returns to step S103. In other words, if it is determined that the subject captured in the imaging area Ati is not an object of interest, the light receiving operation by the SPAD sensor 10 is performed again.

[0084] On the other hand, if it is determined in step S112 that the subject captured in the imaging area Ati is the target object, the control unit 16 proceeds to step S113 and performs imaging control according to the imaging rule specified in the measurement settings. That is, the control of the imaging sensor 11 is performed according to the information on the "imaging rule" in the measurement setting information I2. As described above, the information on the "imaging rule" may be, for example, information such as "still image capture" or "video capture" or information such as "image capture at 20 fps until the target object leaves the flow cell 5". Here, the illumination control during the imaging operation executed in step S113 is performed according to the information on "illumination setting" in the measurement setting information I2.

[0085] In step S114 following step S113, the control unit 16 determines whether or not an imaging end condition has been established. The imaging end condition here is a condition specified from the information specified as the above-mentioned "imaging rule." For example, in the case of "still image imaging," the imaging end condition is that a still image has been captured, and in the case of video imaging, in the case of "imaging at 20 fps until the target object leaves the flow cell 5," the imaging end condition is that the target object frames out of the field of view (imageable range) of the imaging sensor 11. If the imaging end condition is not satisfied, the control unit 16 executes the process of step S113 again.

[0086] On the other hand, if the imaging end condition is satisfied, the control unit 16 proceeds to step S115 to determine whether or not the measurement end condition is satisfied. The measurement end condition is a condition specified from information specified as the "measurement execution condition" in the measurement setting information I2. For example, if "measurement for 10 minutes every 200 m depth" is specified as the "measurement execution condition", the measurement end condition is the passage of 10 minutes from the satisfaction of the measurement start condition. If the measurement end condition is not met, the control unit 16 returns to step S103.

[0087] On the other hand, if the measurement end condition is met, the control unit 16 proceeds to step S116 to execute an injection stop process. That is, the pump of the sample discharge unit 6 is stopped to stop the injection of the sample into the flow cell 5. Then, the control unit 16 executes a cleaning process in the following step S117, and ends the series of processes shown in FIG. The cleaning process in step S117 will be described later.

[0088] FIG. 18 is a flowchart illustrating a processing procedure for realizing tracking of an object. Here, the tracking process shown in Figure 18 can be considered to be performed as a process for tracking an object that has been recognized once until it goes out of frame, so as to prevent objects that have already been counted from being counted twice when counting the number of objects in flow cell 5 as a measurement of the object. Alternatively, from the viewpoint of reducing power consumption related to imaging, the process shown in FIG. 18 may be executed as a process for imaging when moving image imaging is designated as the "imaging rule".

[0089] First, in step S201, the control unit 16 recognizes an object area in a partial captured image of the imaging area Ati. The partial captured image of the imaging area Ati is the one acquired in step S110 of Fig. 17. In the process of step S201, the area of ​​the object captured in this partial captured image is recognized.

[0090] In step S202 following step S201, the control unit 16 calculates the bounding box 20. That is, the control unit 16 calculates the bounding box 20 described in FIG. 16 based on the area of ​​the object recognized in the partially captured image.

[0091] In step S203 following step S202, the control unit 16 calculates the ROI 21, and in step S204 waits for the next frame.

[0092] After waiting for the next frame in step S204, the control unit 16 causes the image sensor 11 to perform partial imaging of the ROI 21 in step S205, that is, to perform partial imaging of the ROI 21 calculated in step S203.

[0093] In step S206 following step S205, the control unit 16 performs processing for recognizing an object in the ROI 21. That is, the control unit 16 performs processing for recognizing an object in the partial captured image of the ROI 21 by performing image analysis based on the image data of the definition information I1, etc.

[0094] In step S207 following step S206, the control unit 16 determines whether or not an object has been recognized. If an object has been recognized in step S207, the control unit 16 returns to step S202. As a result, if an object is recognized within the ROI 21 calculated in the previous frame, a new bounding box 20 and ROI 21 for the object are calculated in the next frame.

[0095] On the other hand, if the object is not recognized in step S207, the control unit 16 ends the series of processes shown in FIG. Note that, although the example given here is one in which tracking of an object is terminated when the object is lost, it is also possible to resume tracking if the object is recognized again within a specified number of frames after being lost.

[0096] Note that while FIG. 18 illustrates processing for one object, when processing for multiple objects, the processing from steps S201 to S207 can be executed for each object for which the light receiving area Ats has been identified.

[0097] 17, since the sample moves in a fixed direction in the flow cell 5, an object that is not captured in the imaging field of view at a certain time may be captured in the imaging field of view at another time thereafter, and it is also possible to make the process capable of dealing with such a situation. Specifically, in such a case, it is possible to execute the processes from steps S103 to S107 at fixed time intervals, for example, after the start of sample injection in step S102. When a new light-receiving region Ats is identified, an imaging region Ati corresponding to the light-receiving region Ats is identified, and partial imaging is performed for the imaging region Ati.

[0098] FIG. 19 is a flowchart of the cleaning process (S117) shown in FIG. First, in step S121, the control unit 16 performs a process of switching to the cleaning liquid container 3. That is, the control unit 16 instructs the sample switching unit 4 to switch from the sample container 2 to the cleaning liquid container 3.

[0099] In step S122 following step S121, the control unit 16 drives the pump of the sample discharge unit 6 to start injecting the cleaning liquid from the cleaning liquid container 3 into the flow cell 5 as an injection start process, and waits for the establishment of an injection stop condition in the next step S123. The injection stop condition here may be, for example, a certain time has elapsed since the start of injection, or a predetermined amount or more of cleaning liquid has been injected into the flow cell 5. Note that the injection stop condition and other conditions related to washing may be variably set by setting information such as the measurement setting information I2.

[0100] If the injection stop condition is met, the control unit 16 performs an injection stop process in step S124, which is a process of stopping the driving of the pump of the sample discharge unit 6, and the process proceeds to step S125.

[0101] Steps S125 to S128 are processes for obtaining an image of light received by the SPAD sensor 10 for the cleaned flow cell 5. First, in step S125, the control unit 16 turns on the detection light source 9, instructs the SPAD sensor 10 to receive light in step S126, and turns off the detection light source 9 in step S127. Then, in the next step S128, the control unit 16 obtains an image of light received by the SPAD sensor 10.

[0102] In step S129 following step S128, the control unit 16 determines whether or not there is a light receiving area that meets the conditions. That is, it determines whether or not there is a light receiving area Ats in the received light image that meets the conditions of wavelength and size specified in the definition information I1. This corresponds to determining whether or not an object corresponding to the target remains in the flow cell 5. In step S129, when it is determined that there is a light receiving region that meets the condition, the control unit 16 returns to step S122. As a result, if an object remains in the flow cell 5 after cleaning, the flow cell 5 is cleaned again.

[0103] On the other hand, if it is determined in step S129 that there is no light-receiving region that meets the conditions, the control unit 16 proceeds to step S130 to execute a process of switching to the sample container 2, and then ends the series of processes shown in FIG.

[0104] Although not shown in the figures, the measurement of the object has been described as specifying the number and size, but the measurement can also be a process of specifying the characteristics of the object. For example, if the object is plankton, it is possible to specify shape characteristics (presence and number of flagella, cilia, antennae, legs, eyes, body segments, etc.) and structural characteristics (presence or absence of cells, whether it is unicellular or multicellular, presence or absence of movement, presence or absence of chloroplasts, etc.) by image analysis of the captured image. In addition, in the measurement, the specified information can be stored as measurement result information in the storage unit 17 or the like. For example, it is conceivable to store characteristic information on the shape and structure as described above. In addition, in the light reception image by the SPAD sensor 10, return light (fluorescence, reflected light, scattered light) from the object is detected, and it is conceivable to store information indicating the wavelength components of this return light. Furthermore, the measurement results can be stored for each type of identified object. Furthermore, together with the information on these measurement results, information detected by an external sensor can also be stored. For example, when the above-mentioned information on depth and electrical conductivity is used, it is conceivable to store the information on the external sensor together with the information on the measurement results.

[0105] <4. Sensor structure> The SPAD sensor 10 and the image sensor 11 can have the structures shown in FIGS. In FIG. 20, a SPAD sensor 10 and an image sensor 11 are formed on separate substrates. FIG. 21 shows a SPAD sensor 10 and an image sensor 11 formed on a common substrate. FIG. 22 shows a common semiconductor chip in which a functional section serving as a SPAD sensor 10 and a functional section serving as an image sensor 11 are formed.

[0106] Here, if the SPAD sensor 10 and the image sensor 11 are formed on separate substrates as shown in Figure 20, there is no need to arrange the SPAD sensor 10 and the image sensor 11 in parallel (with their light receiving surfaces parallel to each other) in the measurement device 1. For this reason, a configuration in which the mirror 13 is omitted as shown in FIG. 23 can be adopted.

[0107] Also, a single sensor having both the function as the SPAD sensor 10 and the function as the image sensor 11 (that is, the function of capturing an image of an object under the control of the control unit 16) can be used. 24 and 25 show examples of such a single sensor. 24, a pixel G10 having a SPAD element as a photoelectric conversion element and a pixel G11 used in an image sensor 11 are mixed in the same pixel array section. Here, the pixel G11 can be said to be a pixel having a photoelectric conversion element with a lower light receiving resolution than the SPAD element.

[0108] 25 is an example using a pixel Gmx that has both the functions of the pixel G10 and the pixel G11 described above. In this case, in the sensor, a plurality of pixels Gmx are arranged two-dimensionally as shown in the figure. Here, pixel Gmx can be said to be a pixel having a photoelectric conversion element that is capable of both detecting the amount of light with a resolution in photons and with a resolution for normal imaging.

[0109] When the SPAD sensor 10 and the image sensor 11 are separate as in the examples of Figures 22 and 21, it is possible to reuse existing sensors as the sensor that functions as the light receiving unit (a sensor that performs photoelectric conversion using the electron avalanche phenomenon) and the image sensor that captures the image of the target object, so there is no need to develop and use a new sensor, which makes it possible to reduce the cost of the measuring device 1.

[0110] On the other hand, when configured as a single sensor as in the examples of Figures 24 and 25, there is no need to provide a spectroscopic means (half mirror 12) that is required when the sensor is separate, so the number of optical components can be reduced and the measuring device 1 can be made smaller.

[0111] <5. Image Sensor> FIG. 26 is a block diagram showing an example of the internal configuration of an image sensor 11A as a modified example. The imaging sensor 11A has a function of generating captured image data and a function of image processing on the captured image data. Specifically, the imaging sensor 11A has a function of detecting an object by image analysis, and is a device that can be called an intelligent array sensor.

[0112] As shown in the figure, the image sensor 11A includes a pixel array section 31, an ADC (Analog to Digital Converter) / pixel selector 32, a buffer 33, a logic section 34, a memory 35, an interface (I / F) section 36, and a calculation section 37. The ADC / pixel selector 32 , the buffer 33 , the logic unit 34 , the memory 35 , the interface (I / F) unit 36 ​​, and the calculation unit 37 are capable of performing data communication with one another via a bus 30 .

[0113] The pixel array section 31 is configured by two-dimensionally arranging a plurality of pixels, each having a photoelectric conversion element, such as the pixel G11 described above. The ADC / pixel selector 32 receives an electrical signal photoelectrically converted by the pixel array unit 31. The ADC / pixel selector 32 converts the input electrical signal as an analog signal into digital data, and outputs an image signal (image data) as digital data. Furthermore, the ADC / pixel selector 32 has a pixel selection function for pixels (photoelectric conversion elements) in the pixel array section 31. This makes it possible to acquire, convert, and output photoelectric conversion signals only from selected pixels in the pixel array section 31. In other words, the ADC / pixel selector 32 normally converts photoelectric conversion signals into digital data for all valid pixels that make up an image of one frame, but it is also possible to convert photoelectric conversion signals only from selected pixels into digital data for output. For example, such an ADC / pixel selector 32 can realize partial imaging of the imaging region Ati described above.

[0114] Image data is acquired on a frame-by-frame basis by the ADC / pixel selector 32, and the image data for each frame is temporarily stored in a buffer 33 and is read out at an appropriate timing and provided for processing by a logic unit . The logic section 34 is capable of carrying out various necessary signal processing (image signal processing) on ​​each input frame image signal. For example, the logic unit 34 can adjust image quality by performing processes such as color correction, gamma correction, color gradation processing, gain processing, edge emphasis processing, etc. The logic unit 34 can also perform processes to change data size, such as data compression processing, resolution conversion, and frame rate conversion. Parameters used for each process performed by the logic unit 34 are set. For example, there are set values ​​for color and brightness correction coefficients, gain values, compression ratios, frame rates, etc. The logic unit 34 performs necessary processing using the parameters set for each process. In this embodiment, these parameters may be set by the calculation unit 37.

[0115] The image data processed by the logic unit 34 is stored in a memory 35 configured, for example, by a dynamic random access memory (DRAM) or the like. The image data stored in the memory 35 is transmitted and output to the outside (for example, the control unit 16) by the interface unit 36 ​​at the required timing.

[0116] The calculation unit 37 is configured with a microcomputer having, for example, a CPU, a ROM, a RAM, etc. The calculation unit 37 issues instructions and exchanges data with each of the ADC / pixel selector 32, the buffer 33, the logic unit 34, the memory 35, and the interface (I / F) unit 36 ​​via the bus 30. For example, the calculation unit 37 performs processing to instruct the ADC / pixel selector 32 on the pixel range for converting the photoelectric conversion signal into digital data and outputting it. In addition, the calculation unit 37 also instructs the logic unit 34 on various parameters as necessary.

[0117] The calculation unit 37 also has a function as an image processing unit 37a. The image processing unit 37a is, for example, a processing unit having AI (Artificial Intelligence), and is capable of detecting an object in captured image data and recognizing the detected object. The term "object" as used here refers to an object that can be a detection target for the purpose of recognition from an image. The type of object that is a detection target varies depending on the purpose of the measuring device 1, but any object can be considered as an object as used here. To give only a partial list of examples, any object can be considered as an object, including animals including humans, moving objects (cars, bicycles, airplanes, etc.), natural objects (vegetables, plants, etc.), industrial products / parts, buildings, facilities, mountains, oceans, rivers, stars, the sun, clouds, etc. In addition, in the object recognition process by the image processing unit 37a, it is possible to classify the detected objects. Here, a class is information that indicates a category of an object, and is a classification of objects to be identified, such as "person", "car", "airplane", "ship", "truck", "bird", "cat", "dog", "deer", "frog", "horse", etc.

[0118] The image processing by calculation unit 37 as exemplified above is processing that is not normally performed within an image sensor. Therefore, it can be said that image sensor 11A performs more intelligent processing than a normal image sensor, and in that sense, it can be called an intelligent array sensor.

[0119] When the imaging sensor 11A as described above is used, at least the processing related to image recognition among the processing of the control unit 16 described above can be carried out on the imaging sensor 11A side. Specifically, among the processes shown in Fig. 17, the matching process in step S111 and the determination process in step S112 based on the matching process can be performed using the image processing unit 37a in the imaging sensor 11A. Also, the object tracking process shown in Fig. 18 can be performed using the image processing unit 37a.

[0120] Here, when these processes are performed using the image processing unit 37a, the image data used in the object recognition process is stored in a storage device (for example, the memory 35) in the imaging sensor 11A. Furthermore, when performing the process of counting the number of objects by type and the process of extracting feature information as described above for measuring objects based on captured images, it is possible to use the image processing unit 37a to perform these processes as well. In this case, the information on the measurement results may be stored in a storage device within the imaging sensor 11A, such as the memory 35, and the calculation unit 37 may output the information via the interface unit 36 ​​in response to a request from the outside (for example, the control unit 16).

[0121] In addition, when the matching process of step S111 is performed using the image processing unit 37a, the matching process can be performed using a class identification function by AI. In this case, the AI ​​is configured to be able to identify a plurality of classes, such as "phytoplankton" and "zooplankton", as object classes. In addition, a template image to be used in the matching process is prepared for each of the classes (for example, stored in the memory 35, etc.). Then, the image processing unit 37a performs class identification process using AI on the captured image of the imaging area Ati as the matching process of step S111 to identify the class of the object captured in the imaging area Ati. Then, a template image according to the identified class is selected, image matching is performed using the selected template image, and it is determined whether or not the object in the imaging area Ati is a target object. By narrowing down the classes in this way before performing image matching, it is possible to improve the efficiency of the image matching process. Incidentally, such matching processing using class identification in combination can also be performed by the control unit 16.

[0122] <6. Variations> [6-1. First modified example] Here, the embodiment is not limited to the specific example described above, and various modified configurations may be adopted. For example, in the above, living organisms such as plankton are mainly given as examples of objects to be measured, but the objects may be non-living objects. Below, as an example, a lighting device 1B that can be used to detect microplastics floating in seawater or the like as an object to be measured will be described. FIG. 27 shows an example of the internal configuration of a lighting device 1B as a modified example.

[0123] First, as a premise, microplastics floating in seawater etc. can be roughly classified into chip type and fiber type based on their shape. Then, each of these chip type and fiber type microplastics can be further classified based on their material. Specifically, examples of the material types of microplastics include polyethylene, phenol, polycarbonate, polystyrene, polypropylene, etc. For example, it is possible to distinguish between polyethylene material and phenol material in chip type, and to distinguish between chip type and fiber type, which are the same polystyrene material.

[0124] These microplastics react (i.e., produce reflected light) to near-infrared light (wavelengths of about 780 nm to 2000 nm). For this reason, when detecting microplastics, a detection light source 9B capable of emitting light containing wavelength components of near-infrared light is used instead of the detection light source 9. In addition, a SPAD sensor 10B having sensitivity to near-infrared light is used as the SPAD sensor 10. Here, the detection light source 9B can be configured, for example, with a tungsten halogen lamp or a semiconductor laser. As for the image sensor 11, an image sensor 11B having sensitivity to near-infrared light is used.

[0125] In addition, in the measuring device 1B, a control unit 16B is provided instead of the control unit 16 in order to detect microplastics and recognize objects.

[0126] Here, microplastics have a unique distribution in the power spectrum distribution of reflected light (distribution of reflected light intensity versus wavelength) in the near-infrared region. This unique power spectrum distribution is referred to as the "characteristic power spectrum." Therefore, by determining whether or not the power spectrum distribution of reflected light in the near-infrared region of the light-receiving reaction portion of the SPAD sensor 10B has a distribution that is a characteristic power spectrum of microplastics, it is possible to determine whether or not the light-receiving reaction portion is the light-receiving reaction portion (light-receiving region) of microplastics.

[0127] In this case, in order to enable detection of the spectral distribution, the SPAD sensor 10B is provided with wavelength filters for different wavelengths in the near-infrared region. For example, the SPAD sensor 10B is used in which wavelength filters for different wavelengths are arranged alternately as shown in FIG.

[0128] In addition, the power spectrum distribution pattern of the reflected light of microplastics varies depending on the material type. In this example, the material type is determined based on the power spectrum distribution pattern based on the image captured by the image sensor 11B. For this reason, the image sensor 11B is also configured to be able to distinguish between differences in wavelength of reflected light of near-infrared light, similarly to the SPAD sensor 10B. Specifically, wavelength filters of different wavelengths in the near-infrared region are arranged alternately as in the example of FIG.

[0129] Based on the above assumptions, the process for measuring microplastics will be explained with reference to the flowchart in Figure 28. In addition, in this figure, it is assumed that the light receiving operation by the SPAD sensor 10B has already been performed with the detection light source 9B turned on, and the received light image has already been acquired by the control unit 16.

[0130] In FIG. 28, in step S301, the control unit 16B performs a process of excluding from the target, among the near-infrared light receiving regions Ats, those that do not have the characteristic power spectrum of plastic. As described above, in the SPAD sensor 10B, wavelength filters of different wavelengths are provided for each pixel in the near-infrared region, so that the light receiving region Ats in this case can detect the reflected light power for each different wavelength in the near-infrared region. In step S301, it is possible to determine whether or not the object has a characteristic power spectrum of plastic based on the reflected light power for each wavelength. By excluding from the target the near-infrared light receiving region Ats that does not have the characteristic power spectrum of plastic, imaging is not performed in the imaging range corresponding to that light receiving region Ats.

[0131] In step S302 following step S301, the control unit 16B calculates the corresponding imaging area Ati for the target light receiving area Ats. That is, for the light receiving area Ats determined to have the characteristic power spectrum of plastic in step S301, the control unit 16B calculates the corresponding imaging area Ati. Then, in step S303 following step S302, the control unit 16B determines the shape type of the plastic based on the partial captured image of the imaging area Ati. That is, the above-mentioned chip type or fiber type is determined. It goes without saying that, in executing the process of step S303, the control unit 16B instructs the imaging unit 11B to perform partial imaging of the imaging area Ati. The shape type can be determined in step S303 by image analysis of the partially captured image. For example, the shape type can be determined by matching with image data in definition information I1 that is set in advance for each target plastic.

[0132] In step S304 following step S303, the control unit 16B determines the type of plastic material by power spectrum analysis. As described above, the imaging sensor 11B is provided with wavelength filters of different wavelengths for each pixel in the near-infrared region, so that the reflected light power for each different wavelength in the near-infrared region can be detected in the imaging region Ati in this case. In step S304, the type of plastic material is determined based on the reflected light power for each wavelength and a characteristic power spectrum for each target plastic material that has been set in advance.

[0133] In step S305 following step S304, the control unit 16B determines the size of the plastic by image analysis. This size determination may be performed for a range of sizes, for example, from 20 μm to 40 μm.

[0134] In response to executing the process of step S305, the control unit 16B ends the series of processes shown in FIG.

[0135] In addition, while Fig. 28 shows an example of a process for measuring microplastics, a process for measuring microorganisms such as plankton can be carried out in conjunction with the process in Fig. 28. In other words, the process for measuring microorganisms as shown in Figs. 17 and 18 can be carried out in conjunction with the process in Fig. 28.

[0136] Here, when measuring both microorganisms and microplastics, depending on the type of microorganism being measured, the wavelength band of the return light may be close to the wavelength band of the return light from microplastics. In such a case, in order to improve the detection accuracy of the target object, the light receiving area Ats specified as the light receiving area for microorganisms can be excluded from the detection target for microplastics. In addition, in measurements based on captured images, imaged areas Ati in which characteristics of microorganisms, such as the presence of cilia or flagella, are observed can be excluded from the measurement of microplastics.

[0137] In addition, even in the measurement device 1B as the above-mentioned modified example, it is possible to perform the tracking process of the target as described in Fig. 18. That is, it is possible to perform the tracking process on the target as microplastics.

[0138] The imaging sensor 11A described in Fig. 26 can also be applied to the measuring device 1B. In that case, the shape type determination process in step S303, the material type determination process in step S304, and the size determination process in step S305 can be executed by the image processing unit 37a. In addition, when a tracking process is performed on microplastics, the tracking process can also be executed by the image processing unit 37a.

[0139] [6-2. Second modified example] In the above, an example has been given in which seawater is sampled in the flow cell 5 as a sample for measuring an object, but it is not essential to use the flow cell 5 for measuring an object. FIG. 29 is a block diagram showing an example of the internal configuration of a measurement device 1C as a second modified example that allows measurement of an object without using a flow cell 5. 2 in that the flow cell 5 is omitted, and that the configurations relating to the intake and discharge of the sample into and from the flow cell 5, specifically the sample container 2, the cleaning liquid container 3 (including the sample intake port Mi), the sample switching unit 4, and the sample discharge unit 6 (including the sample discharge port Mo) are omitted, and in that the backlight source 8 is omitted. Also, it differs from the measuring device 1 in that a distance calculation unit 25 that calculates the distance to the light-receiving reaction part based on the light-receiving signal of the SPAD sensor 10 is added, and that a control unit 16C is provided instead of the control unit 16. Here, the distance calculation unit 25 calculates the distance based on the light reception signal of the SPAD sensor 10 directly using, for example, a ToF (Time Of Flight) method.

[0140] As shown in the figure, in the measurement device 1C, light emitted from a detection light source 9 and reflected by a half mirror 12 is irradiated through a light transmission window Mt onto a sample, such as seawater, present outside the measurement device 1C. In the figure, the range shown as "imaging distance range" shows a schematic representation of the distance range in which image capture by the imaging sensor 11 is possible. The imaging distance range is defined as at least the range in which the imaging sensor 11 is in focus when capturing an image (depth of field range).

[0141] Even if an object is detected based on the light receiving signal of the SPAD sensor 10, if the position of the object is outside the imaging distance range, an appropriate image of the object cannot be obtained by the imaging sensor 11, making it difficult to perform appropriate measurements. Therefore, in the second modified example, a distance calculation unit 25 is provided to calculate the distance to the object, and imaging by the imaging sensor 11 is performed when the object is located within an imaging distance range, which is a trigger condition.

[0142] Fig. 30 is a flowchart showing the flow of processing from the start of measurement to the end of measurement in the second modified example. Note that the processing in Fig. 30 is executed by the control unit 16C based on a program stored in a predetermined storage device such as an internal ROM.

[0143] The differences from the process shown in FIG. 17 are that the sample injection start process in step S102 is omitted, the judgment process in step S151 is inserted between steps S107 and S108, and the injection stop process in step S116 and the cleaning process in step S117 are omitted.

[0144] In step S151, the control unit 16C determines whether or not there is a light receiving area within the imageable distance range. That is, it determines whether or not there is a light receiving area within the imageable distance range among the light receiving areas Ats identified in step S107. Specifically, the control unit 16C acquires information on the distance to the light receiving area Ats identified in step S107 based on the distance information (depth image) obtained by the distance calculation unit 25, and determines whether or not the distance is within the distance range defined as the imageable distance range for all the identified light receiving areas Ats. If there is at least one light receiving area Ats whose distance is within the imageable distance range, the control unit 16C obtains a determination result that there is a light receiving area within the imageable distance range, and otherwise obtains a determination result that there is no light receiving area within the imageable distance range.

[0145] When it is determined that there is no light receiving area within the image capturing distance range, the control unit 16C returns to step S103. That is, when there is no light receiving area Ats within the image capturing distance range, the image sensor 11 does not capture an image. On the other hand, if it is determined that there is a light receiving area within the imageable distance range, the control unit 16C advances the process to step S108. As a result, imaging is performed by the imaging sensor 11 on the condition that there is a light receiving area Ats within the imageable distance range, and appropriate measurement can be performed in a configuration in which the flow cell 5 is omitted.

[0146] In the second modified example, it goes without saying that the processes in and after step S108 are performed for the light-receiving region Ats within the image capturing distance range.

[0147] [6-3.Third modified example] In the third modified example, when the configuration is such that the flow cell 5 is omitted like in the second modified example, a slit light is used as the light for detecting the object. FIG. 31 is a block diagram showing an example of the internal configuration of a measurement device 1D serving as a third modified example. The difference from the measuring apparatus 1C shown in FIG. 29 is that a slit light source 26 is provided instead of the detection light source 9, and that the distance calculation unit 25 is omitted.

[0148] As shown in the figure, the slit light source 26 emits a slit light Ls that illuminates the imageable distance range. Note that the slit light source 26 may be, for example, a semiconductor laser or an LED (Light Emitting Diode).

[0149] By using the slit light Ls as described above, reflected light is detected only from an object located within the image capture distance range, which eliminates the need to calculate the distance to the object in order to determine whether the object is located within the image capture distance range as in the second modified example, and therefore distance calculation unit 25 can be omitted.

[0150] Compared to control unit 16C in the second modified example, control unit 16D differs in that, among the series of processes shown in FIG. 30, the control process for the detection light source (steps S103 and S105) is performed on slit light source 26 rather than detection light source 9, and in that it does not execute the determination process of step S151, i.e., the process of determining whether or not there is a light receiving area within the image capturing distance range.

[0151] [6-4. Fourth modified example] In the fourth modified example, a digital holographic microscope is applied to the imaging system using the imaging sensor 11. FIG. 32 is a block diagram showing an example of the internal configuration of a measurement device 1E serving as a fourth modified example. 2, in the measuring device 1E, the flow cell 5 and components related to the intake and discharge of the sample into and from the flow cell 5 (sample container 2, cleaning liquid container 3, sample switching unit 4, sample discharge unit 6), as well as the front light source 7, back light source 8, and detection light source 9 are omitted. Also, a control unit 16E is provided in place of the control unit 16.

[0152] In the measurement apparatus 1E, a light source 27, a collimation lens 40, a beam splitter 41, a beam combining element 42, a mirror 43, and a mirror 44 are provided as an optical system for realizing a digital holographic microscope. The light source 27 may be, for example, a semiconductor laser. Coherent light emitted from the light source 27 passes through a collimation lens 40, with a portion of the light passing through a beam splitter 41 and entering a beam combining element 42 as object light, and another portion of the light being reflected by the beam splitter 41 and then entering the beam combining element 42 as reference light via mirrors 43 and 44 as shown in the figure. The beam combining element 42 transmits the incident object light, and combines the reference light incident via a mirror 44 onto the same optical axis as the object light, and outputs the combined light to the half mirror 12 . As shown in the figure, part of the combined light incident on the half mirror 12 is transmitted and directed to the image sensor 11 side, and the other part is reflected and directed to the SPAD sensor 10 side.

[0153] Digital holographic technology is a technology that obtains three-dimensional information about an object by capturing an interference fringe pattern between an object light and a reference light using an image sensor (imaging sensor 11) and calculating the diffraction phenomenon of light from the captured interference fringe pattern.

[0154] In a typical microscope, the depth of field is relatively shallow, for example, the depth of field of an objective lens for imaging fine particles such as plankton is about 1 mm. Therefore, if you try to directly image seawater in the vertical direction while diving, you need to change the depth and take images many times. On the other hand, digital holographic microscopes can achieve a depth of field about 100 times deeper than lens imaging methods using objective lenses. Therefore, when imaging while moving a certain volume, the number of times of imaging can be significantly reduced compared to conventional microscopes.

[0155] 32, light emitted by the light source 27 is used to detect an object using the SPAD sensor 10. In this case, the control unit 16E turns on the light source 27 and controls the imaging sensor 11 to perform an imaging operation in response to determining that there is a light receiving area Ats that meets the conditions based on the light receiving signal of the SPAD sensor 10. Regarding the light for detecting an object using the SPAD sensor 10, it is also possible to use light from a light source provided separately from the light source 27, rather than the light emitted by the light source 27.

[0156] [6-5. Fifth Modification] The fifth modified example is a modified example relating to object measurement based on an image captured by the imaging sensor 11. In the previous Figure 18, it was mentioned that the object area is recognized based on a partial captured image of the imaging area Ati (S201), a bounding box 20 is calculated based on the recognized object area (S202: see Figure 16), an ROI 21 is calculated based on the calculated bounding box 20 (S203), partial imaging of the ROI 21 is performed in the next frame (S205), and object recognition processing within the ROI 21 is performed (S206). In such a series of processes, the image resolution for the partial imaging of the ROI 21 performed in step S205 can be varied depending on the size of the ROI 21 calculated in step S203 (that is, the ROI 21 calculated in the previous frame).

[0157] A specific example will be described with reference to FIG. 33 and FIG. Figure 33 shows an example of an image captured by the imaging sensor 11, in which zooplankton Pm and phytoplankton Pp are captured, and Figure 34 shows examples of ROI-1, which is ROI 21 calculated for zooplankton Pm, and ROI-2, which is ROI 21 calculated for phytoplankton Pp.

[0158] In general, zooplankton Pm is larger than phytoplankton Pp. For example, zooplankton Pm such as Daphnia magna have a body length of about 2 to 3.5 mm, while phytoplankton Pp such as Diatomella cepa have a body length of about 0.06 mm.

[0159] In the fifth modified example, when the calculated size of ROI 21 is small, the image resolution in the partial image capture of ROI 21 in the next frame is controlled to be higher than when the calculated size of ROI 21 is large. Specifically, in the example of Fig. 34, the image resolution is maximized (i.e., no thinning) for ROI-2 of small-sized phytoplankton Pp, and the image resolution is thinned to 1 / 9 (only one representative pixel is extracted from 3 x 3 = 9 pixels) for ROI-1 of large-sized zooplankton Pm.

[0160] Here, in the object recognition process of step S206 (Figure 18), for objects with a large size of ROI21, it is possible to perform recognition even if the image resolution is somewhat low. However, for objects with a small size of ROI21, there is a risk that recognition will not be possible if the image resolution is reduced. For this reason, in this example, when the size of ROI21 is small, the image resolution in the partial image capture of ROI21 in the next frame is set higher than when the size of ROI21 is large. This makes it possible to reduce the recognition processing load for objects with a large size in ROI21 while preventing the accuracy of the recognition processing from decreasing for objects with a small size in ROI21. In other words, it is possible to achieve both reduction in the recognition processing load and prevention of a decrease in the recognition processing accuracy.

[0161] As can be seen by referring to Figures 33 and 34, according to the measurement technique of the embodiment described thus far, measurements of objects of different sizes can be performed simultaneously without changing the objective lens for each size of the object.

[0162] [6-6. Sixth Modification] In the sixth modified example, detection of an object, which has been performed using the SPAD sensor 10 up until now, is performed using the image sensor 11. FIG. 35 is a block diagram showing an example of the internal configuration of a measurement apparatus 1F serving as a sixth modified example. The difference from the measuring device 1 shown in FIG. 2 is that the SPAD sensor 10, the mirror 13, and the lens 14 are omitted, and that a control unit 16F is provided instead of the control unit 16.

[0163] When it is possible to detect an object based on weak return light from the object, such as excitation light caused by the fluorescent reaction of phytoplankton, it is necessary to use a SPAD sensor 10 in order to enable detection of the weak return light. However, when returning light having a sufficiently higher light intensity than the fluorescent reaction is obtained, such as the scattered light of zooplankton, the SPAD sensor 10 is not necessary, and the returning light can be detected by the imaging sensor 11. Therefore, in the sixth modified example, the SPAD sensor 10 is omitted, and the target detection process is performed using the imaging sensor 11.

[0164] Fig. 36 is a flowchart showing the flow of processing from the start to the end of measurement in the sixth modified example. The processing in Fig. 36 is executed by the control unit 16F based on a program stored in a predetermined storage device such as an internal ROM.

[0165] The difference from the process shown in FIG. 17 is that the light reception instruction process of step S161 is carried out instead of the light reception instruction process of step S104. Specifically, in step S161, the control unit 16F issues a light receiving instruction to the imaging sensor 11. As a result, based on a light receiving image (captured image) by the imaging sensor 11, the determination process of step S107 is performed, that is, the determination process of whether or not there is a light receiving region that meets the conditions.

[0166] The imaging sensor 11 may be configured as a vision sensor that reads out a light receiving signal from a pixel where an event occurs only when an event occurs. The vision sensor is a sensor called a DVS (Dynamic Vision Sensor) or an EVS (Event-based Vision Sensor), and is an asynchronous image sensor in which a plurality of pixels having photoelectric conversion elements are arranged two-dimensionally, and a detection circuit that detects address events in real time is provided for each pixel. An address event is an event that occurs for each address assigned to each of a plurality of pixels arranged two-dimensionally. The event here is, for example, a current value based on a charge generated in a photoelectric conversion element, or a change amount thereof, exceeding a certain threshold value. The vision sensor detects whether an address event occurs for each pixel, and when the occurrence of an address event is detected, reads out a pixel signal from the pixel at the corresponding address as pixel data. In the vision sensor described above, pixel data is read from the pixel where the occurrence of an address event is detected, so it is possible to read data much faster than a synchronous image sensor that reads data from all pixels at a certain frame rate, and the amount of data read for one frame is small. Therefore, by using a vision sensor, it is possible to detect the movement of an object more quickly, and also to reduce the frequency of read operations, thereby reducing power consumption.

[0167] As can be seen by referring to Fig. 36, the control unit 16F in the sixth modified example also performs matching between the captured image for a part of the pixel range in which the target object is captured and the template image (see steps S108 to S111). At this time, as described in the modified example of Fig. 26, the control unit 16F can also perform class identification of the object captured in the captured image for the part of the pixel range, and perform matching using the template image of the identified class among the template images prepared for each class.

[0168] Although not illustrated, the control unit 16F in the sixth modified example also performs processing for realizing the tracking of the object as described in Fig. 18 etc. Specifically, it calculates the bounding box 20 and the ROI 21 based on the bounding box 20, and performs processing for recognizing the object within the ROI 21.

[0169] Although the sixth modified example has been described above as an example of application to a type that uses a flow cell 5, it is also possible to adopt a configuration in which the flow cell 5 is omitted, as in the second and third modified examples described above, or a type that applies a digital holographic microscope, as in the fourth modified example.

[0170] In the above description, the SPAD sensor (10 or 10B) has a two-dimensional array of pixels, but the SPAD sensor may have a one-dimensional array of pixels. Alternatively, the SPAD sensor may be a single pixel sensor.

[0171] <7. Summary of the embodiment> As described above, the first measuring device of the embodiment (1, 1B, 16C, 16D, 16E) includes a light emitting unit (detection light source 9, 9B, slit light source 26) that emits light to the fluid, a light receiving unit (SPAD sensor 10, 10B) that performs photoelectric conversion of the incident light using the electron avalanche phenomenon with multiple pixels to obtain a received light signal, and a control unit (16, 16B, 16C, 16D, 16E) that performs detection processing of an object in the fluid based on the received light signal and executes an imaging operation of the object on the condition that the object is detected. According to the above configuration, rather than constantly capturing images of the object, the detection of the object based on the light receiving signal from the light receiving unit is used as a trigger to reduce power consumption related to imaging, making it possible to detect the presence or absence of an object based on the light receiving signals of multiple pixels. Therefore, it is possible to improve the accuracy of detecting the presence or absence of an object, and to reduce the power consumption of the measuring device. By reducing the power consumption of the measuring device, it becomes possible to reduce the size of the battery used as the power source, which in turn makes it possible to reduce the size of the measuring device.

[0172] In the first measuring device as the embodiment, the light receiving section has a SPAD element as a photoelectric conversion element. This eliminates the need to use a large-sized, high-power-consumption photoelectric conversion element such as a photomultiplier tube in the light receiving section. Therefore, the light receiving section can be made smaller and consume less power, and the measuring device can be made smaller and consume less power.

[0173] Furthermore, in the first measuring device as an embodiment, the control unit performs detection processing of the object based on the image features of the light-receiving reaction part in the light-receiving unit (see S107 in FIG. 17 and S301 in FIG. 28). The "image characteristics of the light-receiving reaction part" referred to here means characteristics of an image composed of at least one pixel that has reacted to light, such as the image size and position of the light-receiving reaction part, the wavelength of the received light, and the value of the received light signal, as the light-receiving reaction part. Based on the image features of such a light-reacting portion, it is possible to appropriately estimate whether or not the light-reacting portion has captured an object.

[0174] Furthermore, in the first measuring device as an embodiment, the control unit prevents imaging of the imaging range corresponding to the light-receiving reaction part if the image characteristics of the light-receiving reaction part do not match the specified image characteristics. This makes it possible to prevent objects other than those having the designated image characteristics from being captured indiscriminately. Therefore, it is possible to reduce the power consumption related to imaging, and it is possible to achieve power saving of the measuring device.

[0175] Furthermore, in the first measuring device as the embodiment, the control unit detects the pixel position and image size of the light-receiving reaction portion as the image features (see FIGS. 12 to 14, etc.). This makes it possible to specify the pixel range in which the object is captured, that is, the pixel range in which imaging should be performed, for the image sensor that captures the object. Therefore, it is possible to prevent objects other than those having the specified image characteristics from being captured indiscriminately, and it is possible to reduce power consumption related to imaging.

[0176] Furthermore, in the first measuring device as the embodiment, the control unit controls the imaging sensor (11, 11B) that captures an image of the object so that the imaging operation is performed only for a part of the pixel range in which the object is captured. This makes it possible to reduce power consumption relating to imaging, compared to the case where imaging is performed over the entire pixel range of the image sensor. Therefore, the power consumption of the measuring device can be reduced.

[0177] Furthermore, in the first measurement device as the embodiment, the control unit performs matching between the captured image and the template image for a portion of the pixel range (see S111 in FIG. 17). By performing matching based on the captured image, it becomes possible to appropriately identify the type of object. Therefore, the accuracy of object measurement can be improved by improving the accuracy of object recognition.

[0178] Moreover, in the first measuring device as an embodiment, the control unit performs class identification of an object captured in a captured image for a portion of a pixel range, and performs matching using a template image of the identified class from among template images prepared for each class. By narrowing down the classes in this way before performing image matching, it is possible to improve the efficiency of the image matching process.

[0179] Furthermore, in the first measuring device as an embodiment, the control unit sets a bounding box (ibid. 20) as a range surrounding the object from an image of a portion of a pixel range in a reference frame, which is a predetermined frame after the object is detected based on the received light signal, and sets an ROI (ibid. 21) that is an area that contains the bounding box and is larger than the bounding box, and in frames after the reference frame, sets a bounding box of the object within the ROI set in the immediately preceding frame, and sets an ROI based on the bounding box (see Figure 16). This makes it possible to track the target even if the target moves within the captured image. In this case, the only captured image required in each frame for tracking the target is the image of the ROI. Therefore, when tracking the target to prevent miscounting, it is possible to narrow the imaging range for tracking to only the ROI, and reduce the power consumption related to imaging for tracking.

[0180] Furthermore, in the first measuring device as the embodiment, the sensor that functions as the light receiving unit and the imaging sensor that captures an image of the object based on the control of the control unit are separate entities (see FIGS. 20 to 22). This makes it possible to use existing sensors as both the sensor that functions as the light receiving unit (a sensor that performs photoelectric conversion using the electron avalanche phenomenon) and the imaging sensor that captures an image of the target object. Therefore, there is no need to develop and use a new sensor, and the cost of the measurement device can be reduced.

[0181] Furthermore, the first measuring device as an embodiment includes a single sensor having a function as a light receiving unit and a function of capturing an image of an object based on the control of the control unit (see FIGS. 24 and 25). When the sensors are separate, it is necessary to provide a spectroscopic means for distributing the light from the fluid to each sensor. However, by providing an integrated sensor, there is no need to provide such a spectroscopic means. Therefore, the number of optical components can be reduced, and the measuring device can be made more compact.

[0182] In addition, the first measuring device as an embodiment is equipped with a flow cell (same as 5) in which a fluid is sampled from an internal flow path, and after the imaging operation is completed, the control unit causes a fluid other than the sample fluid to flow into the flow path to wash the flow cell (see Figure 19). This makes it possible to prevent erroneous measurements, such as measuring an object that has already been measured again. Therefore, the measurement accuracy of the object can be improved.

[0183] Furthermore, in the first measurement device as the embodiment, the control unit performs a detection process for the object based on the light receiving signal after another fluid flows into the flow path (see S123 to S129 in FIG. 19). This makes it possible to check whether any objects remain after cleaning. Therefore, if any target substance remains, it becomes possible to take appropriate measures to prevent erroneous measurement, such as re-cleaning the flow cell, thereby making it possible to enhance the effect of preventing erroneous measurement.

[0184] In addition, the imaging control method of the embodiment is an imaging control method for a measuring device that has at least an light-emitting unit that emits light toward a fluid and a light-receiving unit that performs photoelectric conversion of incident light using the electron avalanche phenomenon with multiple pixels to obtain a received light signal, and is an imaging control method that performs a detection process for an object in the fluid based on the received light signal, and executes an imaging operation of the object on the condition that the object is detected. With this imaging control method as well, it is possible to obtain the same functions and effects as those of the first measuring device as the above-mentioned embodiment.

[0185] A second measuring device (1F) as an embodiment includes a light emitting unit (e.g., detection light source 9) that emits light to the fluid, an imaging sensor (e.g., 11) that performs photoelectric conversion on the incident light using multiple pixels to obtain a received light signal, and a control unit (16F) that performs a detection process for an object in the fluid based on the received light signal and, on condition that the object is detected, causes the imaging sensor to perform an imaging operation of the object, and the control unit controls the imaging operation of the object so that an imaging operation is performed only for a portion of the pixel range in which the object is captured. According to the above configuration, it is possible to reduce power consumption related to imaging by not capturing an image of an object all the time, but by using the detection of an object based on a light receiving signal as a trigger to capture the image, and this also reduces power consumption related to imaging more than when imaging is performed for the entire pixel range of the imaging sensor. Therefore, the power consumption of the measuring device can be reduced.

[0186] Furthermore, in the second measuring device as the above-described embodiment, the control unit performs matching between the captured image and the template image for a portion of the pixel range. By performing matching based on the captured image, it becomes possible to appropriately identify the type of object. Therefore, the accuracy of object measurement can be improved by improving the accuracy of object recognition.

[0187] Furthermore, in the second measuring device as the above-described embodiment, the control unit performs class identification of an object captured in a captured image for a portion of the pixel range, and performs matching using a template image of the identified class from among template images prepared for each class. By narrowing down the classes in this way before performing image matching, it is possible to improve the efficiency of the image matching process.

[0188] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0189] <8. This Technology> The present technology can also be configured as follows. (1) a light emitting unit that emits light toward the fluid; a light receiving section that performs photoelectric conversion of incident light using an electron avalanche phenomenon to obtain a received light signal; a control unit that performs a process of detecting an object in the fluid based on the light receiving signal, and executes an image capturing operation of the object on the condition that the object is detected. Measuring equipment. (2) The light receiving section has a SPAD element as a photoelectric conversion element. The measuring device according to (1) above. (3) The control unit is The detection process of the object is carried out based on the image characteristics of the light-receiving reaction part in the light-receiving part. The measuring device according to (1) or (2) above. (4) The control unit is When the image characteristics of the light-receiving reaction part do not match the designated image characteristics, imaging of the imaging range corresponding to the light-receiving reaction part is not performed. The measuring device according to (3) above. (5) The control unit is Detecting the pixel position and image size of the light receiving reaction portion as the image feature. A measuring device according to any one of (3) or (4) above. (6) The control unit is In an image sensor for capturing an image of the object, control is performed so that an image capturing operation is performed only for a part of a pixel range in which the object is captured. A measuring device according to any one of (1) to (5) above. (7) The control unit is Matching the captured image with the template image for the partial pixel range is performed. The measuring device according to (6) above. (8) The control unit is A class of an object captured in the captured image for the partial pixel range is identified, and the matching is performed using a template image of the identified class among the template images prepared for each class. The measuring device according to (7) above. (9) The control unit is In a reference frame, which is a predetermined frame after the object is detected based on the light receiving signal, a bounding box is set as a range surrounding the object from the captured image of the partial pixel range, and an ROI is set as an area that includes the bounding box and is larger than the bounding box; In a frame subsequent to the reference frame, the bounding box of the object is set within the ROI set in the immediately preceding frame, and the ROI is set based on the bounding box. The measuring device according to any one of (6) to (8). (10) The sensor that functions as the light receiving unit and the image sensor that captures the image of the object based on the control of the control unit are separate. A measuring device according to any one of (1) to (9) above. (11) A single sensor having a function as the light receiving unit and a function of capturing an image of the object based on the control of the control unit. A measuring device according to any one of (1) to (9) above. (12) a flow cell through which the fluid is sampled; The control unit is After the imaging operation is completed, A fluid other than the fluid as a sample is caused to flow into the flow channel to wash the flow cell. The measuring device according to any one of (1) to (11) above. (13) The control unit is After the other fluid flows into the flow path, a detection process for the object is performed based on the light receiving signal. The measuring device according to (12) above. (14) An imaging control method for a measuring device including at least a light emitting unit that emits light toward a fluid, and a light receiving unit that performs photoelectric conversion of incident light using an electron avalanche phenomenon by a plurality of pixels to obtain a light receiving signal, comprising: A process of detecting an object in the fluid is performed based on the light receiving signal, and an image capturing operation of the object is performed on the condition that the object is detected. Imaging control method. (15) a light emitting unit that emits light toward the fluid; an image sensor that performs photoelectric conversion on incident light using a plurality of pixels to obtain a light receiving signal; a control unit that performs a process of detecting an object in the fluid based on the light receiving signal, and causes the image sensor to capture an image of the object on the condition that the object is detected; The control unit is As an image capturing operation of the object, control is performed so that an image capturing operation is performed only for a part of a pixel range in which the object is captured. Measuring equipment. (16) The control unit is Matching the captured image with the template image for the partial pixel range is performed. The measuring device according to (15) above. (17) The control unit is A class of an object captured in the captured image for the partial pixel range is identified, and the matching is performed using a template image of the identified class among the template images prepared for each class. The measuring device according to (16) above. [Explanation of symbols]

[0190] 1,1B,1C,1D,1E,1F Measuring equipment 2. Sample container 3 Cleaning solution container 4. Sample switching section 5 Flow Cell 6 Sample ejection section 7 Front light source 8 Rear light source 9,9B Detection light source 10,10B SPAD sensor 11, 11A, 10B Image sensor 12 Half Mirror 13. Mirror 14,15 Lens 16, 16B, 16C, 16D, 16E, 16F Control section 17 Memory section 18 Communications Department Mi sample inlet Mo sample outlet 20 Bounding Box 21 ROI G10, G11, Gmx pixels 30 Bus 31 Pixel array section 32 ADC / Pixel Selector 33 Buffer 34 Logic Section 35 Memory 36 Interface section 37 Arithmetic section 37a Image processing section

Claims

1. a light emitting unit that emits light toward the fluid; a light receiving section that performs photoelectric conversion of incident light by using a plurality of pixels to obtain a light receiving signal by utilizing an electron avalanche phenomenon; a control unit that performs a process of detecting an object in the fluid based on the light receiving signal, and executes an image capturing operation of the object on the condition that the object is detected; The control unit is The detection process of the object is performed based on whether or not the image characteristics of the light-receiving reaction portion of the light-receiving unit match the designated image characteristics. Measuring equipment.

2. The light receiving section has a SPAD element as a photoelectric conversion element.

2. The measuring device of claim 1.

3. The control unit is When the image characteristics of the light-receiving reaction part do not match the designated image characteristics, imaging of the imaging range corresponding to the light-receiving reaction part is not performed.

2. The measuring device of claim 1.

4. The control unit is Detecting the pixel position and image size of the light receiving reaction portion as the image feature.

2. The measuring device of claim 1.

5. The control unit is In an image sensor for capturing an image of the object, control is performed so that an image capturing operation is performed only for a part of a pixel range in which the object is captured.

2. The measuring device of claim 1.

6. The control unit is Matching the captured image with the template image for the partial pixel range is performed.

6. The measuring device according to claim 5.

7. The control unit is A class of an object captured in the captured image for the partial pixel range is identified, and the matching is performed using a template image of the identified class among the template images prepared for each class.

7. The measuring device according to claim 6.

8. The control unit is In a reference frame, which is a predetermined frame after the object is detected based on the light receiving signal, a bounding box is set as a range surrounding the object from the captured image of the partial pixel range, and an ROI is set as a region that includes the bounding box and is larger than the bounding box; In a frame subsequent to the reference frame, the bounding box of the object is set within the ROI set in the immediately preceding frame, and the ROI is set based on the bounding box.

6. The measuring device according to claim 5.

9. The sensor that functions as the light receiving unit and the image sensor that captures the image of the object based on the control of the control unit are separate.

2. The measuring device of claim 1.

10. A single sensor having a function as the light receiving unit and a function of capturing an image of the object based on the control of the control unit.

2. The measuring device of claim 1.

11. a flow cell through which the fluid is sampled; The control unit is After the imaging operation is completed, A fluid other than the fluid as a sample is caused to flow into the flow channel to wash the flow cell.

2. The measuring device of claim 1.

12. The control unit is After the other fluid flows into the flow path, a detection process for the object is performed based on the light receiving signal.

12. The measuring device according to claim 11.

13. An imaging control method for a measuring device including at least a light emitting unit that emits light toward a fluid, and a light receiving unit that performs photoelectric conversion of incident light using an electron avalanche phenomenon by a plurality of pixels to obtain a light receiving signal, comprising: performing a process of detecting an object in the fluid based on the light receiving signal, and executing an image capturing operation of the object on the condition that the object is detected; The detection process of the object is performed based on whether or not the image characteristics of the light-receiving reaction portion of the light-receiving unit match the designated image characteristics. Imaging control method.

Images