Measuring device, measurement method, program

Abstract

According to the present invention, a measurement device comprises an imaging control unit that makes an imaging unit capture an image of a prescribed underwater imaging area and a measurement unit that measures information about the position of a target object in the imaging direction on the basis of the image captured by the imaging unit.

Description

Technical Field

[0006] , , ,

[0005] , , The imaging unit includes a vision sensor that asynchronously acquires pixel data according to the amount of light incident on each of the multiple pixels arranged in two dimensions, and the imaging control unit temporarily stops the irradiation of light from the illumination unit when the target object can no longer be detected within the imaging range. , ,

[0001] This technology relates to a measuring device, a measuring method, and a program, and particularly to a technology for measuring a target object in water.

Background Art

[0002] There has been proposed a measuring device that measures the abundance of phytoplankton by irradiating excitation light of a predetermined wavelength to excite the phytoplankton and measuring the intensity of fluorescence emitted from the excited phytoplankton (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the above-described measuring device, it is only possible to measure phytoplankton excited by excitation light. Further, although the measuring device can measure the abundance of phytoplankton, it cannot measure information regarding the position.

[0005] Therefore, an object of the present technology is to efficiently measure information regarding the position of a target object.

Means for Solving the Problems

[0006] The measuring device according to the present technology includes an imaging control unit that causes an imaging unit to image a predetermined imaging range in water An illumination unit that irradiates light of a predetermined wavelength onto the object, and the and a measuring unit that measures information regarding the position of a target object in the imaging direction based on an image imaged by the imaging unit. The imaging unit includes a vision sensor that asynchronously acquires pixel data according to the amount of light incident on each of the multiple pixels arranged in two dimensions, and the imaging control unit temporarily stops the irradiation of light from the illumination unit when the target object can no longer be detected within the imaging range. . This allows the measuring device to measure information regarding the position of the target object in the imaging direction without requiring a complex configuration. [Brief explanation of the drawing]

[0007] [Figure 1] This is a diagram illustrating the configuration of a measuring device as the first embodiment. [Figure 2] This diagram illustrates the imaging range and measurement direction. [Figure 3] This diagram illustrates the target object and its movement. [Figure 4] This diagram illustrates an example of measurement settings. [Figure 5] This is a diagram illustrating an example of an operation time sheet. [Figure 6] This is a flowchart showing the measurement process steps. [Figure 7] This diagram illustrates a rule-based distance and speed measurement process. [Figure 8] This is a diagram illustrating the images used as training data. [Figure 9] This is a diagram of a deep learning model. [Figure 10] This diagram illustrates the configuration of a measuring device as a second embodiment of this technology. [Figure 11] This diagram illustrates an example of measurement settings. [Figure 12] This is a flowchart showing the measurement process steps. [Figure 13] This diagram illustrates the configuration of the measuring device for modified examples. [Figure 14] This diagram illustrates the configuration of the measuring device for modified examples. [Figure 15] This diagram illustrates the lighting control in the first modified example. [Figure 16] This figure illustrates the image captured by the vision sensor during lighting control in the modified example 1. [Figure 17] This diagram illustrates the lighting control in Modification 2. In Modification 2, multiple lighting units 3 are provided.

Mode for Carrying Out the Invention

[0008] Hereinafter, embodiments will be described in the following order. <1. First Embodiment> [1.1 Configuration of the Measuring Device] [1.2 Regarding the Target Object] [1.3 Measuring Method of the First Embodiment] [1.4 Measurement Process] [1.5 Distance and Velocity Measurement Process] <2. Second Embodiment> [2.1 Configuration of the Measuring Device] [2.2 Measurement Process] [2.3 Machine Learning Distance and Velocity Measurement Process] <3. Other Configuration Examples of the Measuring Device> <4. Summary of Embodiments> <5. This Technology>

[0009] <1. First Embodiment> [1.1 Configuration of the Measuring Device] First, the configuration of the measuring device 1 as the first embodiment according to this technology will be described. The measuring device 1 is a device that measures information regarding the position of a target object in the imaging direction, with, for example, microorganisms or fine particles existing in water such as in the sea as the target object.

[0010] Here, the microorganisms that are the target objects are aquatic microorganisms such as phytoplankton, zooplankton, and larvae of aquatic organisms existing in water. Also, the fine particles that are the target objects are microplastics, dust, sand, marine snow, air bubbles, etc. However, these are just examples, and the target object may be other than these.

[0011] Also, the information regarding the position of the target object in the imaging direction is, for example, the distance to the target object in the imaging direction (Z-axis direction in FIG. 2) of the imaging unit 14, or the velocity of the target object.

[0012] Figure 1 is a diagram illustrating the configuration of the measuring device 1 as a first embodiment. Figure 2 is a diagram illustrating the imaging range IR and the measurement direction. As shown in Figure 1, the measuring device 1 comprises a main body 2 and an illumination unit 3. The illumination unit 3 may be located within the main body 2.

[0013] The main unit 2 includes a control unit 10, a memory 11, a communication unit 12, a gravity sensor 13, an imaging unit 14, and a lens 15.

[0014] The control unit 10 is configured with, for example, a microcomputer having a CPU (Central Processing Unit), ROM (Read Only Memory), and RAM (Random Access Memory), and performs overall control of the measuring device 1. In the first embodiment, the control unit 10 functions as an imaging control unit 21, a class identification unit 22, and a distance-velocity measurement unit 23. The imaging control unit 21, the class identification unit 22, and the distance-velocity measurement unit 23 will be described in detail later. Furthermore, the control unit 10 performs data reading processing from the memory 11, processing to store data in the memory 11, and sending and receiving various types of data with external devices via the communication unit 12.

[0015] Memory 11 is composed of non-volatile memory. The communication unit 12 performs wired or wireless data communication with external devices. The gravity sensor 13 detects gravitational acceleration (direction of gravity) and outputs the detection result to the control unit 10. Note that the measuring device 1 does not necessarily have to be equipped with the gravity sensor 13.

[0016] The imaging unit 14 includes either or both of the vision sensor 14a and the imaging sensor 14b. The vision sensor 14a is a sensor called a DVS (Dynamic Vision Sensor) or EVS (Event-Based Vision Sensor). The vision sensor 14a images a predetermined imaging range IR underwater through the lens 15. In the following, as shown in Figure 2, the left-right direction of the imaging range IR may be referred to as the X-axis direction, the up-down direction of the imaging range IR as the Y-axis direction, and the imaging direction of the imaging unit 14 (optical axis direction) as the Z-axis direction.

[0017] The vision sensor 14a is an asynchronous image sensor in which multiple pixels having photoelectric conversion elements are arranged in two dimensions, and a detection circuit for detecting address events in real time is provided for each pixel. An address event is an event that occurs in accordance with the amount of incident light for each address assigned to each of the multiple pixels arranged in two dimensions. For example, this could be the current value of the current based on the charge generated by the photoelectric conversion element, or the amount of change thereof, exceeding a certain threshold.

[0018] The vision sensor 14a detects whether or not an address event has occurred for each pixel, and if an address event is detected, it reads the pixel signal as pixel data from the pixel where the address event occurred. In other words, the vision sensor 14a acquires pixel data asynchronously according to the amount of light incident on each of the multiple pixels arranged in two dimensions.

[0019] In the vision sensor 14a, the pixel signal readout operation is performed for pixels where an address event has been detected. This allows for significantly faster readout than a synchronous image sensor, where the readout operation is performed for all pixels at a predetermined frame rate, and also results in a smaller amount of data being read out per frame.

[0020] Therefore, by using the vision sensor 14a in the measuring device 1, it becomes possible to detect the movement of the target object more quickly. In addition, the vision sensor 14a can reduce the amount of data and also reduce power consumption.

[0021] The image sensor 14b is, for example, a CCD (Charge Coupled Device) or CMOS (Complementary Metal-Oxide-Semiconductor) type image sensor, in which multiple pixels having photoelectric conversion elements are arranged in two dimensions. The image sensor 14b generates image data by capturing a predetermined imaging range IR through the lens 15 at regular intervals according to the frame rate. In the measuring device 1, a zone plate, pinhole plate, or transparent plate can be used instead of the lens 15.

[0022] The vision sensor 14a and the imaging sensor 14b are arranged to capture substantially the same imaging range IR through the lens 15. For example, a half-mirror (not shown) can be placed between the vision sensor 14a and the imaging sensor 14b and the lens 15, so that one portion of the light split by the half-mirror is incident on the vision sensor 14a and the other portion is incident on the imaging sensor 14b.

[0023] The illumination unit 3 is driven based on the control of the control unit 10 and irradiates the imaging range IR of the imaging unit 14 with light. The illumination unit 3 can switch between irradiating light of different wavelengths, for example, by irradiating light of different wavelengths at 10 nm intervals.

[0024] [1.2 Regarding the target object] Figure 3 illustrates the target object and its movement. In Figure 3, the upper section shows an image of the target object, and the lower section shows the direction of the target object's movement with arrows.

[0025] As shown in Figure 3, the target objects include microorganisms, marine snow, seabed sand, smoke, and bubbles. Furthermore, it is known that some microorganisms exhibit phacoemulsion when exposed to light of a specific wavelength. Here, phacoemulsion is an innate behavior in which an organism responds to light (an external stimulus). Therefore, when a phacoemulsifying microorganism is exposed to light of a specific wavelength, it will move according to its phacoemulsion properties.

[0026] Marine snow consists of particles such as plankton excrement, dead plankton, or their decomposition, which move in the ocean towards the bottom (in the direction of gravity). Seabed sand, for example, consists of particles such as sand that have settled on the seabed, and it moves in a swirling motion due to seabed currents. Smoke is a phenomenon where, for example, hot water heated by geothermal energy erupts from hydrothermal vents on the seabed. The hot water erupting from these vents can reach temperatures of several hundred degrees Celsius and contains abundant dissolved heavy metals and hydrogen sulfide. These react with seawater, causing black or white smoke to rise in a swirling pattern. The bubbles are natural gases such as methane and carbon dioxide leaking (erupting) from the seabed, or carbon dioxide leaking from reservoirs artificially injected in CCS (carbon capture and storage), and they move upward from the seabed.

[0027] Thus, the target objects are not limited to microorganisms; even fine particles can move in a specific direction. In the first embodiment of the measuring device 1, microorganisms and fine particles whose direction of movement is known are identified as target objects.

[0028] [1.3 Measurement Method of the First Embodiment] Next, a measurement method (measurement process) for the target object as the first embodiment will be described. The ocean becomes a photic zone at a depth of approximately 150m where sunlight does not penetrate. This photic zone makes up most of the open ocean, and many of the objects mentioned above exist there. On the other hand, these objects are known to reflect or emit light of different wavelengths or intensities depending on the wavelength of light they are exposed to.

[0029] Therefore, assuming that the measurement will be performed in a light-free layer where sunlight does not reach, the measuring device 1 irradiates the target object with light of different wavelengths and identifies the type of target object by capturing an image of the reflected light (or excitation light). Then, the measuring device 1 measures the distance and velocity of the identified target object in the imaging direction.

[0030] Figure 4 illustrates an example of measurement settings. Figure 5 illustrates an example of an operation time sheet.

[0031] The control unit 10 performs measurements according to the pre-specified measurement settings shown in Figure 4. The measurement settings specify the measurement start conditions, the operation time sheet for the illumination unit 3, the identification program (identification method), the distance-speed measurement program (distance-speed measurement method), and the measurement end conditions.

[0032] The measurement start conditions specify the conditions for starting the measurement, such as the time to start the measurement, or the receipt of a measurement start command input via the communication unit 12.

[0033] The operation time sheet specifies the time sheet for operating the illumination unit 3. For example, the operation time sheet shown in Figure 5 specifies that the wavelengths should be varied in 10nm increments within the range of 400nm to 700nm (400nm, 410nm, ..., 690nm, 700nm), and that the light should be irradiated with an off period in between each wavelength.

[0034] As described above, the operation time sheet specifies what wavelength of light the illumination unit 3 should emit and at what timing for the imaging range IR. The reason why there is a period when the illumination unit 3 is turned off, i.e., when no light is emitted, is to capture the light when the target object is emitting light (excited). In addition, by inserting an off period between each wavelength, the asynchronous vision sensor 14a has the effect of making it easier to detect events of each wavelength.

[0035] The identification program specifies a program (method) for identifying the type of target object, such as machine learning-based identification or rule-based identification.

[0036] The distance-velocity measurement program specifies a program (method) for measuring information regarding the position of the target object in the imaging direction, such as machine learning-based measurement or rule-based measurement.

[0037] The measurement termination conditions specify the conditions for ending the measurement, such as the time at which the measurement will end, or the receipt of a measurement termination command input via the communication unit 12.

[0038] [1.4 Measurement Process] Figure 6 is a flowchart showing the measurement process procedure. The control unit 10 executes the measurement process shown in Figure 6 by running the software (including the identification program and the distance-speed measurement program) stored in the memory 11.

[0039] In step S1, the control unit 10 reads external environmental information, which will be described in more detail later. Then, in step S2, the control unit 10 determines whether the measurement start conditions specified in the measurement settings have been met. The control unit 10 then repeats steps S1 and S2 until the measurement start conditions are met.

[0040] On the other hand, if the measurement start condition is met (Yes in step S2), in step S3 the imaging control unit 21 switches and irradiates the illumination unit 3 with light of different wavelengths according to the operation time sheet specified in the measurement settings. The imaging control unit 21 also causes the imaging unit 14 to capture the imaging range IR each time the wavelength and on / off state of the light irradiated from the illumination unit 3 are switched, and acquires pixel data and image data. Subsequently, in step S4 the class identification unit 22 performs class identification processing.

[0041] In the class identification process, the class identification unit 22 identifies (specifies) the type of target object based on the image (pixel data and image data) captured by the imaging unit 14. The class identification unit 22 derives identification information from the image captured by the imaging unit 14 and detects the target object by comparing it with the definition information stored in the memory 11.

[0042] Definition information is provided for each target object and stored in memory 11. The definition information includes the type of target object, movement information, and image information.

[0043] Movement information is information detected primarily based on images captured by the vision sensor 14a, and is information based on the movement of an object, as shown in the lower part of Figure 3. When the object is a microorganism, the movement information includes information such as the direction of movement (positive or negative), velocity, and trajectory relative to the light source. When the object is a microparticle, the movement information includes information such as the direction of movement, velocity, and trajectory.

[0044] Image information is information detected mainly based on images captured by the imaging sensor 14b, and is information about the external shape of the target object. Alternatively, image information may also be information detected based on images captured by the vision sensor 14a.

[0045] Furthermore, the definition information may include the direction of gravity detected by the gravity sensor 13 and external environmental information acquired via the communication unit 12. Examples of external environmental information include depth, position coordinates (latitude and longitude of the measurement point, plane rectangular coordinates), electrical conductivity, temperature, pH, gas concentration (e.g., methane, hydrogen, helium), and metal concentration (e.g., manganese, iron).

[0046] The class identification unit 22 detects objects present in the imaging range IR based on the image (pixel data) captured by the vision sensor 14a. For example, the class identification unit 22 creates one image (frame data) based on the pixel data input within a predetermined period, and detects a group of pixels within a predetermined range in that image where motion is detected as a single object.

[0047] Furthermore, the class identification unit 22 tracks the object across multiple frames using pattern matching or the like. Based on the object tracking results, the class identification unit 22 derives the direction of movement, velocity, and trajectory as identification information.

[0048] Furthermore, the period during which the class identification unit 22 generates an image from the pixel data may be the same as, or shorter than, the period (frame rate) during which the imaging sensor 14b acquires image data.

[0049] Furthermore, the class identification unit 22 extracts the image portion corresponding to the object from the image data input from the imaging sensor 14b for the object for which identification information has been derived. Then, based on the extracted image portion, the class identification unit 22 derives the external features as identification information through image analysis. Note that since known methods can be used for image analysis, their explanation is omitted here.

[0050] The class identification unit 22 identifies whether or not an object is a target object by comparing the wavelength of light irradiated by the illumination unit 3 and the identification information derived for the detected object (direction of movement, trajectory, velocity, external characteristics) with definition information according to a specified identification program. Here, for example, if the derived identification information of an object falls within the range indicated in the definition information of the target object, the class identification unit 22 identifies the derived object as belonging to the type indicated in that definition information.

[0051] This definition information will be stored in memory 11 in a different way for each identification program. For example, in a rule-based identification program, the definition information is pre-set by the user and stored in memory 11. In a machine learning identification program, the definition information is generated and updated by machine learning in learning mode and stored in memory 11.

[0052] Subsequently, the class identification unit 22 stores the identification result of the detected target object and the image captured by the imaging sensor 14b in the memory 11, or transmits them to an external device via the communication unit 12.

[0053] In step S5, the distance-velocity measurement unit 23 performs a distance-velocity measurement process to measure the distance and velocity (information regarding the position of the object) of the object in the imaging direction based on the type of object identified by the class identification unit 22. The distance-velocity measurement process in step S5 will be described in more detail later.

[0054] Subsequently, in step S6, the control unit 10 determines whether the measurement termination condition has been met. The control unit 10 then repeats steps S1 to S6 until the measurement termination condition is met, and if the termination condition is met (Yes in step S6), it terminates the determination process.

[0055] [1.5 Distance and Speed ​​Measurement Process] Next, the distance-speed measurement process will be described. As described above, in step S5, the distance-speed measurement unit 23 executes the distance-speed measurement process based on a rule-based or machine learning distance-speed measurement program. This section will explain rule-based distance and speed measurement processes and machine learning-based distance and speed measurement processes with specific examples.

[0056] [1.5.1 Rule-based distance and speed measurement process] Figure 7 illustrates the rule-based distance-velocity measurement process. In the rule-based distance-velocity measurement process, the focal length f of the vision sensor 14a is stored in memory 11 as known information.

[0057] Furthermore, memory 11 stores statistical information (average size H) for each target object. This information is pre-registered by the user as part of a database.

[0058] Then, when the distance-velocity measurement unit 23 identifies the target object from the image based on the pixel data, it reads the average size H of the target object and the focal length f of the vision sensor 14a from the memory 11. After that, the distance-velocity measurement unit 23 calculates the length s of the longitudinal direction of the image 42 of the target object captured on the imaging surface 40, for example, based on the number of pixels in which the image 42 is captured.

[0059] Furthermore, the distance-velocity measuring unit 23 calculates the distance D from the measuring device 1 to the target object 41 in the imaging direction (Z direction) using equation (1). D = fH / s ... (1)

[0060] In this way, the distance-velocity measuring unit 23 calculates (measures) the distance D from the measuring device 1 to the actual target object 41 each time an image based on pixel data is acquired (each time the target object is detected from the image). Furthermore, the distance-velocity measurement unit 23 calculates (measures) the velocity in the imaging direction (Z-axis direction) of the target object 41 being tracked between consecutive images, based on the interval at which the images are acquired and the distance D in each image.

[0061] As described above, in the rule-based distance-speed measurement process, the distance-speed measurement unit 23 measures information regarding the position of the target object based on statistical information (average size) for each target object.

[0062] [1.5.2 Distance and Speed ​​Measurement Processing in Machine Learning] Figure 8 is a diagram illustrating the images used as training data. Figure 9 is a diagram of the deep learning model.

[0063] In machine learning-based distance and speed measurement processing, machine learning is performed using images as training data, such as those shown in Figure 8, to generate a model (architecture) for distance and speed measurement processing.

[0064] Specifically, images of known target objects captured by the vision sensor 14a are pre-prepared for a total of 153 patterns, with five patterns of distances from the measuring device 1 to the target object in the imaging direction (1 mm, 5 mm, 10 mm, 100 mm, and 200 mm) and 31 patterns of wavelengths of irradiated light varying in 10 nm increments from 400 nm to 700 nm.

[0065] The distance and speed measuring unit 23 then detects a group of pixels within a predetermined range where motion is detected as the target object for each prepared image, and resizes that group of pixels to 32 pixels × 32 pixels to generate a training data image as shown in Figure 8.

[0066] Figure 8 shows a portion of the training data images. Here, in the ocean, the attenuation rate of light at approximately 500 nm is low, while the attenuation rates of light with wavelengths smaller than approximately 500 nm and light with wavelengths larger than approximately 500 nm increase as you move away from approximately 500 nm. Furthermore, the greater the distance from measuring device 1 to the target object, the lower the light penetration rate becomes.

[0067] Therefore, as shown in Figure 8, in the image of the target object, the closer the target object is to the measuring device 1 and the closer the wavelength of the irradiated light is to 500 nm, the clearer the image of the target object appears. Conversely, the further the target object is from the measuring device 1 and the further the wavelength of the irradiated light is from 500 nm, the blurrier or completely absent the image of the target object appears.

[0068] When the training data images are resized, the distance-velocity measurement unit 23 uses a deep neural network to machine-learn using these training data images, as shown in Figure 9. This model consists of, for example, five convolutional layers (Conv1 to Conv5), three pooling layers (Max Pooling), and two fully connected layers (FC). Through machine learning, a model is generated that ultimately outputs a one-dimensional classification vector with five elements ranging from Distance 1 mm to Distance 200 mm, and this model is stored in memory 11.

[0069] This type of machine learning using deep neural networks is performed for each target object, and a model is generated for each target object and stored in memory 11.

[0070] Then, once the type of object is identified by the class identification unit 22 (step S4), the distance-velocity measurement unit 23 reads the model of the identified type from the memory 11. The distance-velocity measurement unit 23 also resizes the portion of the object in the image captured by the vision sensor 14a to 32 pixels × 32 pixels and inputs the resized image into the read model. This outputs a one-dimensional classification vector value with five elements ranging from Distance 1 mm to Distance 200 mm. The distance-velocity measurement unit 23 then outputs (measures) the element with the highest value among the five elements (any of Distance 1 mm to Distance 200 mm) as the distance in the imaging direction of the object. Furthermore, the distance-velocity measurement unit 23 calculates (measures) the velocity in the imaging direction (Z-axis direction) for the target object being tracked between consecutive images, based on the interval at which the images are acquired and the distance in the imaging direction for each image.

[0071] As described above, in the machine learning distance and speed measurement process, the distance and speed measurement unit 23 measures information regarding the position of the target object based on the learning results of position information that has been learned in advance for each type of target object.

[0072] <2. Second Embodiment> [2.1 Configuration of the measuring device] Figure 10 is a diagram illustrating the configuration of a measuring device 100 as a second embodiment of the present technology. As shown in Figure 10, the measuring device 100 differs from the measuring device 1 as the first embodiment in that the control unit 110 does not function as a class identification unit 22, but the other configurations are the same as those of the measuring device 1.

[0073] The measuring device 100 then measures the distance and speed to the target object in the imaging direction, without identifying the type of target object, based on the image captured by the vision sensor 14a.

[0074] Figure 11 illustrates an example of measurement settings. The control unit 110 performs measurements according to the pre-specified measurement settings shown in Figure 11. The measurement settings specify the measurement start conditions, the operation time sheet for the illumination unit 3, the distance speed measurement program (measurement method), and the measurement end conditions.

[0075] The measurement start conditions specify the conditions for starting the measurement, such as the time to start the measurement, or the receipt of a measurement start command input via the communication unit 12.

[0076] The operation time sheet specifies the time sheet for operating the illumination unit 3. For example, the operation time sheet shown in Figure 5 specifies that the wavelength should be varied in 10nm increments within the range of 400nm to 700nm (400nm, 410nm, ..., 690nm, 700nm) and that the light should be irradiated while repeatedly switching it on and off.

[0077] The distance-velocity measurement program specifies a program (method) for measuring information regarding the position of the target object in the imaging direction, such as machine learning-based measurement or rule-based measurement.

[0078] The measurement termination conditions specify the conditions for ending the measurement, such as the time at which the measurement will end, or the receipt of a measurement termination command input via the communication unit 12.

[0079] Thus, the measurement settings in the second embodiment differ from those in the first embodiment in that an identification program is not provided.

[0080] [2.2 Measurement Process] Figure 12 is a flowchart showing the measurement process. The control unit 110 executes the measurement process shown in Figure 12 by running the software (distance speed measurement program) stored in the memory 11.

[0081] In step S1, the control unit 110 reads external environmental information. Then, in step S2, the control unit 10 determines whether the measurement start conditions specified in the measurement settings have been met. The control unit 110 then repeats steps S1 and S2 until the measurement start conditions are met.

[0082] On the other hand, if the measurement start condition is met (Yes in step S2), in step S3, the imaging control unit 21 switches and irradiates the illumination unit 3 with light of different wavelengths according to the operation time sheet specified in the measurement settings. In addition, each time the wavelength and on / off state of the light irradiated from the illumination unit 3 are switched, the imaging control unit 21 causes the imaging unit 14 to capture the imaging range IR, and acquires pixel data and image data.

[0083] Subsequently, in step S11, the distance and velocity measuring unit 23 detects objects within the imaging range as target objects based on the image derived from the pixel data, and performs a distance and velocity measurement process to measure the distance and velocity of those target objects in the imaging direction. The distance and velocity measurement process in step S11 will be described in more detail later.

[0084] Subsequently, in step S6, the control unit 10 determines whether the termination condition for ending the judgment process has been met. The control unit 10 then repeats steps S1 to S6 until the termination condition for ending the judgment process is met, and if the termination condition for ending the purpose-specific measurement operation process is met (Yes in step S6), the judgment process is terminated.

[0085] [2.3 Machine Learning Distance and Speed ​​Measurement Process] As described above, in step S11, the distance-speed measurement unit 23 performs distance-speed measurement processing based on a rule-based or machine learning distance-speed measurement program. This section explains distance and speed measurement processing in machine learning with specific examples.

[0086] In the measurement device 100, a deep learning model is created as shown in Figure 9, similar to the measurement device 1. In the first embodiment, a model is generated for each target object, but in the second embodiment, instead of generating a model for each target object, only one pre-trained model is generated regardless of the type of target object.

[0087] Specifically, images are pre-prepared using a vision sensor 14a with a total of 153 patterns, where the distance from the measuring device 1 to the target object in the imaging direction is 1 mm, 5 mm, 10 mm, 100 mm, and 200 mm in five patterns, and the wavelength of the irradiated light is varied in 10 nm increments from 400 nm to 700 nm, for a total of 31 patterns, and the target object is also varied, for a total of 153 patterns × number of types of target objects.

[0088] The distance and speed measuring unit 23 then detects a group of pixels within a predetermined range where motion is detected as the target object for each prepared image, and resizes that group of pixels to 32 pixels × 32 pixels to generate a training data image as shown in Figure 8.

[0089] When the training data images are resized, the distance-velocity measurement unit 23 uses a deep neural network to machine-learn the training data consisting of these images, as shown in Figure 9, and stores the generated model in memory 11.

[0090] The distance-velocity measurement unit 23 then resizes the portion of the target object in the image captured by the vision sensor 14a to 32 pixels x 32 pixels and inputs the resized image into the model read from memory 11. This outputs a one-dimensional classification vector value with five elements ranging from Distance 1 mm to Distance 200 mm. The distance-velocity measurement unit 23 then outputs (measures) the element with the highest value among the five elements (Distance from 1 mm to Distance 200 mm) as the distance in the imaging direction of the target object. Furthermore, the distance-velocity measurement unit 23 calculates (measures) the velocity in the imaging direction (Z-axis direction) for the target object being tracked between consecutive images, based on the interval at which the images are acquired and the distance in the imaging direction for each image.

[0091] As described above, in the machine learning distance and speed measurement process, the distance and speed measurement unit 23 measures information about the position of the target object based on the learning results of pre-learned position information, regardless of the type of target object.

[0092] Therefore, in the second embodiment, the data volume can be reduced compared to the first embodiment because the number of models is smaller. Also, in the second embodiment, although the distance measurement accuracy is lower, the calculation time is also shorter.

[0093] <3. Other configuration examples of measuring devices> It should be noted that the embodiments are not limited to the specific examples described above, and various other modified configurations are possible.

[0094] In the above embodiment, the measuring device 1 is equipped with one illumination unit 3. However, the number of illumination units 3 is not limited to one, and multiple units may be provided.

[0095] Figure 13 illustrates the configuration of a modified measurement device 200. As shown in Figure 13, the modified measurement device 200 comprises one main body 2 and two illumination units 3. The two illumination units 3 are arranged so that light can be emitted in directions perpendicular to each other, and light of different wavelengths can be emitted onto the imaging range. In such a measuring device 200, since light of different wavelengths can be irradiated from two illumination units 3, identification information of target objects (microorganisms) that exhibit catataxis to light of different wavelengths can be derived in a single measurement, enabling efficient measurement.

[0096] Figure 14 is a diagram illustrating the configuration of a modified measurement device 300. As shown in Figure 14, the modified measurement device 300 comprises two main body units 2 and one illumination unit 3. The two main body units 2 are arranged to enable the capture of images in mutually orthogonal directions. In such a measuring device 300, images can be captured by two main body units 2 (imaging units 14), making it possible to detect the three-dimensional movement of the target object and perform measurements more efficiently. Furthermore, if two main units 2 are provided, one of the main units 2 may be equipped only with the imaging unit 14.

[0097] Furthermore, in the embodiment described above, the imaging unit 14 is equipped with a vision sensor 14a and an imaging sensor 14b. However, the imaging unit 14 may be equipped with only one of the vision sensor 14a or the imaging sensor 14b, as long as it can capture an image that can measure information regarding the position of the target object in the imaging direction. Alternatively, the imaging unit 14 may be equipped with a SPAD (Single Photon Avalanche Diode) sensor instead of the vision sensor 14a and the imaging sensor 14b.

[0098] Furthermore, in the above embodiment, identification information is derived based on the pixel data acquired by the vision sensor 14a and the image data acquired by the imaging sensor 14b to identify the type of target object. However, if the type of target object can be identified based on at least one of the pixel data acquired by the vision sensor 14a and the image data acquired by the imaging sensor 14b, it may be identified by other methods.

[0099] Furthermore, in the above embodiment, machine learning was performed using deep learning. However, the method of machine learning is not limited to this, and other methods may be used. Also, the model generated by machine learning may be created by an external device rather than the measurement device 1.

[0100] [3-1 Variation 1] Figure 15 illustrates the lighting control in Modification Example 1. Figure 16 illustrates the image captured by the vision sensor 14a during lighting control in Modification Example 1.

[0101] Incidentally, in the vision sensor 14a, an address event occurs in each pixel when the brightness changes and the current value exceeds a certain threshold. Therefore, if the target object TO is moving at an extremely slow speed or not moving at all within the imaging range (hereinafter collectively referred to as being stationary), no address event occurs in each pixel. Consequently, in such cases, the vision sensor 14a cannot image the target object TO.

[0102] Therefore, when the target object TO is stationary, the illumination from the illumination unit 3 is temporarily stopped. Specifically, as shown in the upper part of Figure 15, when the imaging control unit 21 is emitting light from the illumination unit 3, if the target object TO moves within the imaging range, the vision sensor 14a captures the target object TO as shown in Figures 16(a) and 16(b).

[0103] Subsequently, when the target object TO stops, address events cease to occur in the vision sensor 14a, and as shown in Figure 16(c), the target object TO is no longer imaged (indicated by a dashed line in the figure).

[0104] The imaging control unit 21 determines that if the target object TO can no longer be detected within the imaging range (if the target object TO disappears within the imaging range), the target object TO has either stopped, moved at high speed outside the imaging range, or disappeared. Then, as shown in the middle of Figure 15, the imaging control unit 21 temporarily stops the illumination of light from the illumination unit 3. When the illumination of light from the illumination unit 3 is stopped, if the target object TO is present within the imaging range, a change in brightness occurs around the target object TO, and as shown in Figure 16(d), the vision sensor 14a captures the target object TO.

[0105] Furthermore, the imaging control unit 21 restarts the illumination of light from the illumination unit 3, as shown in the lower part of Figure 15. When the illumination of light from the illumination unit 3 is restarted, a change in brightness occurs around the target object TO, and as shown in Figure 16(e), the vision sensor 14a captures an image of the target object TO.

[0106] In this manner, if the target object TO can no longer be detected within the imaging range, the illumination from the lighting unit 3 is temporarily stopped. By doing so, if the target object TO is within the imaging range, the target object TO will be captured in the image taken by the vision sensor 14a, making it possible to continue measuring the target object TO. Furthermore, if the target object TO can no longer be detected within the imaging range, the imaging control unit 21 may change the wavelength of the light emitted from the illumination unit 3. By changing the wavelength of the light emitted from the illumination unit 3, it becomes possible to image the target object TO, which has stopped within the imaging range, with the vision sensor 14a.

[0107] [3-2 Variation 2] Figure 17 illustrates the lighting control in Modification 2. In Modification 2, multiple lighting units 3 are provided. Here, we will explain the case where two lighting units 3 are provided. The two lighting units 3 are located in different positions.

[0108] For example, as shown in the upper part of Figure 17, when the imaging control unit 21 is irradiating light from one of the illumination units 3 and the target object TO moves within the imaging range, the target object TO is captured in the image taken by the vision sensor 14a.

[0109] On the other hand, when the target object TO stops within the imaging range, no address events are generated in the vision sensor 14a, and therefore the target object TO will no longer be visible in the image captured by the vision sensor 14a.

[0110] If the imaging control unit 21 cannot detect the target object TO within the imaging range, it stops the illumination of light from one illumination unit 3 and starts the illumination of light from the other illumination unit 3, as shown in the lower part of Figure 17. When the illumination of light from the other illumination unit 3 begins, a change in brightness occurs for the target object TO in the vision sensor 14a, and the target object TO is captured.

[0111] Thus, in Modification 2, just like in Modification 1, it is possible to continuously measure the target object TO. Furthermore, if multiple lighting units 3 are not provided, the target object TO can be measured even when the target object TO is stationary by moving the lighting unit 3, similar to the case where multiple lighting units 3 are switched to emit light.

[0112] <4. Summary of Embodiments> As described above, the measuring device 1 of the embodiment includes an imaging control unit 21 that causes an imaging unit 14 to image a predetermined imaging range underwater, and a measuring unit (distance-velocity measuring unit 23) that measures information regarding the position of a target object in the imaging direction based on the image captured by the imaging unit 14. This allows the measuring device 1 to measure information regarding the position of the target object in the imaging direction without requiring a complex configuration. For example, it is conceivable to measure information regarding the position of a target object in the imaging direction by arranging two imaging units 14 in parallel and using them as a stereo camera. However, this method complicates the device, and calibrating the two imaging units 14 becomes difficult. In contrast, measuring device 1 can efficiently measure information regarding the position of the target object.

[0113] In the measuring device 1 related to the present technology described above, the imaging unit 14 may be equipped with a vision sensor 14a that acquires pixel data asynchronously according to the amount of light incident on each of the multiple pixels arranged in two dimensions. This makes it possible to read only the pixel data of the pixel where the event occurred and to measure the target object based on that pixel data. Therefore, the measuring device 1 can achieve high-speed imaging, reduced power consumption, and lower computational costs for image processing through automatic separation from the background.

[0114] In the measuring device 1 related to the present technology described above, an illumination unit 3 is provided that irradiates the imaging range with light of a predetermined wavelength, and the imaging unit 14 is thought to image the imaging range that has been irradiated with light of a predetermined wavelength by the illumination unit 3. This makes it possible to image only the reflected or excited light from the target object at depths where sunlight does not reach. Therefore, the measuring device 1 can efficiently measure the target object.

[0115] In the measuring device 1 related to the present technology described above, the illumination unit 3 is capable of switching between illuminating light of different wavelengths, and the imaging unit 14 is capable of imaging the imaging ranges irradiated with light of different wavelengths. This allows for the acquisition of images of reflected or excited light that differ depending on the wavelength, for each type of target object. Therefore, the measuring device 1 can acquire characteristic images for each target object.

[0116] In the measuring device 1 related to the above-described technology, the measuring unit is thought to measure the distance to the target object in the imaging direction. This allows for the measurement of the distance to an object in the imaging direction using a simple configuration, without the need for complex setups such as stereo cameras.

[0117] In the measuring device 1 related to the above-described technology, the measuring unit is thought to measure the velocity of the target object in the imaging direction. This allows for the measurement of the velocity of an object in the imaging direction using a simple configuration, without the need for complex setups such as stereo cameras.

[0118] In the measuring device 1 related to the present technology described above, an identification unit (class identification unit 22) is provided that identifies the type of target object based on the image captured by the imaging unit 14, and the measuring unit measures information regarding the position of the target object based on the type of target object identified by the identification unit. This makes it possible to measure positional information using methods (models) tailored to each type of object. Therefore, the measuring device 1 can accurately measure information regarding the position of the target object in the imaging direction.

[0119] In the measuring device 1 related to the technology described above, the measuring unit may measure information regarding the position of the target object based on statistical information for each type of target object. This makes it possible to measure information about the position of the target object in the imaging direction using a simple method.

[0120] In the measuring device 1 related to the above-described technology, the measuring unit may derive information regarding the position of a target object based on the learning results of position information that has been learned in advance for each type of target object. This makes it possible to accurately measure information regarding the position of the target object in the imaging direction.

[0121] In the measuring device 1 related to the above-described technology, the measuring unit may derive information regarding the position of a target object based on the learning results of pre-learned position information, regardless of the type of target object. This allows for a reduction in data volume and a decrease in computation time.

[0122] In the measuring device 1 related to the above-described technology, the imaging control unit 21 may temporarily stop the irradiation of light from the illumination unit 3 if the target object can no longer be detected within the imaging range. This allows for continuous measurement of a stationary target object within the imaging range.

[0123] In the measuring device 1 related to the above-described technology, the imaging control unit 21 may change the wavelength of the light emitted from the illumination unit 3 if the target object can no longer be detected within the imaging range. This allows for continuous measurement of a stationary target object within the imaging range.

[0124] In the measuring device 1 related to the technology described above, the imaging control unit 21 may move the illumination unit 3 if the target object can no longer be detected within the imaging range. This allows for continuous measurement of a stationary target object within the imaging range.

[0125] In the measuring device 1 related to the present technology described above, multiple illumination units 3 are provided, and the imaging control unit 21 may irradiate light from a different illumination unit 3 if the target object can no longer be detected within the imaging range. This allows for continuous measurement of a stationary target object within the imaging range.

[0126] In the measurement method related to the technology described above, a predetermined imaging range underwater is imaged by the imaging unit, and information regarding the position of the target object in the imaging direction is measured based on the captured image. In the program relating to the technology described above, the imaging unit captures an image of a predetermined imaging range underwater, and the information processing device is instructed to perform a process to measure information regarding the position of a target object in the imaging direction based on the captured image.

[0127] Such programs can be pre-recorded in storage media such as HDDs (hard disk drives) built into computer devices, or in ROM (remote data) within microcomputers with CPUs. Alternatively, the data can be temporarily or permanently stored (recorded) on removable recording media such as flexible disks, CD-ROMs (Compact Disc Read Only Memory), MO (Magneto Optical) disks, DVDs (Digital Versatile Discs), Blu-ray Discs (registered trademark), magnetic disks, semiconductor memory, and memory cards. Such removable recording media can be provided as so-called packaged software. In addition to installing such programs from removable storage media to personal computers, they can also be downloaded from download sites via networks such as LANs (Local Area Networks) and the Internet.

[0128] Furthermore, such a program is suitable for providing information processing devices of the embodiment to a wide range of users. For example, by downloading the program to mobile terminal devices such as smartphones and tablets, mobile phones, personal computers, game consoles, video equipment, PDAs (Personal Digital Assistants), etc., the device can function as an information processing device according to the disclosure.

[0129] Furthermore, the effects described herein are merely illustrative and not limited to those described herein, and other effects may also occur.

[0130] <5. This Technology> This technology can also be configured as follows: (1) An imaging control unit that causes an imaging unit to capture an image of a predetermined imaging range underwater, A measuring unit measures information regarding the position of a target object in the imaging direction based on the image captured by the imaging unit, A measuring device equipped with this device. (2) The imaging unit is The vision sensor is equipped with a two-dimensional arrangement of multiple pixels and asynchronously acquires pixel data according to the amount of light incident on each pixel. (1) The measuring device described above. (3) The imaging area is equipped with an illumination unit that irradiates light of a predetermined wavelength, The imaging unit is The imaging area illuminated by the illumination unit with light of a predetermined wavelength is imaged. The measuring device described in (1) or (2). (4) The illumination unit is capable of emitting light of different wavelengths by switching between them. The imaging unit captures images of imaging areas irradiated with light of different wavelengths. (3) The measuring device described above. (5) The aforementioned measuring unit is The distance to the target object in the imaging direction is measured. A measuring device as described in any of (1) to (4). (6) The aforementioned measuring unit is The velocity of the target object in the imaging direction is measured. A measuring device as described in any of (1) to (5). (7) The system includes an identification unit that identifies the type of target object based on the image captured by the imaging unit, The measuring unit measures information regarding the position of the target object based on the type of target object identified by the identification unit. A measuring device as described in any of (1) to (6). (8) The aforementioned measuring unit is Based on statistical information for each type of object, information regarding the position of the object is measured. (7) The measuring device described above. (9) The aforementioned measuring unit is Based on the learning results of the position information that has been pre-learned for each type of target object, information regarding the position of the target object is measured. A measuring device as described in any of (1) to (6). (10) The aforementioned measuring unit is Regardless of the type of object, information regarding the position of the object is measured based on the learning results of previously learned position information. A measuring device as described in any of (1) to (6). (11) The imaging control unit, If the target object can no longer be detected within the imaging range, the illumination from the lighting unit is temporarily stopped. (3) or (4) the measuring device described above. (12) The imaging control unit, If the target object can no longer be detected within the imaging range, the wavelength of the light emitted from the illumination unit is changed. (3) The measuring device described above. (13) The imaging control unit, If the target object can no longer be detected within the imaging range, the illumination unit is moved. (3) or (4) the measuring device described above. (14) Multiple lighting units are provided, The imaging control unit, If the target object can no longer be detected within the imaging range, light will be emitted from a different illumination unit. (3) or (4) the measuring device described above. (15) The imaging unit captures an image of a predetermined area underwater. Based on the captured image, information regarding the position of the target object in the imaging direction is measured. Measurement method. (16) The imaging unit captures an image of a predetermined area underwater. Based on the captured image, information regarding the position of the target object in the imaging direction is measured. A program that instructs a measuring device to perform a process. [Explanation of Symbols]

[0131] 1. Measuring device 3. Lighting section 10 Control Unit 14 Imaging Unit 14a Vision Sensor 14b Imaging sensor 21 Imaging Control Unit 22 Class Identification Unit 23 Distance speed measuring section

Claims

1. An illumination unit that irradiates a predetermined imaging area underwater with light of a predetermined wavelength, An imaging control unit that causes the imaging range to be imaged by the imaging unit, A measuring unit measures information regarding the position of a target object in the imaging direction based on the image captured by the imaging unit, Equipped with, The imaging unit includes a vision sensor that asynchronously acquires pixel data according to the amount of light incident on each of the multiple pixels arranged in two dimensions. The imaging control unit temporarily stops the illumination of light from the illumination unit when the target object can no longer be detected within the imaging range. Measuring device.

2. The illumination unit is capable of emitting light of different wavelengths by switching between them. The imaging unit captures images of the imaging ranges irradiated with light of different wavelengths. The measuring device according to claim 1.

3. The aforementioned measuring unit is The distance to the target object in the imaging direction is measured. The measuring device according to claim 1.

4. The aforementioned measuring unit is The velocity of the target object in the imaging direction is measured. The measuring device according to claim 1.

5. The system includes an identification unit that identifies the type of the target object based on the image captured by the imaging unit, The measuring unit measures information regarding the position of the target object based on the type of target object identified by the identification unit. The measuring device according to claim 1.

6. The aforementioned measuring unit is Based on statistical information for each type of object, information regarding the position of the object is measured. The measuring device according to claim 5.

7. The aforementioned measuring unit is Based on the learning results of the position information that has been pre-learned for each type of target object, information regarding the position of the target object is measured. The measuring device according to claim 1.

8. The aforementioned measuring unit is Regardless of the type of object, information regarding the position of the object is measured based on the learning results of previously learned position information. The measuring device according to claim 1.

9. By illuminating a predetermined imaging area underwater with light of a predetermined wavelength, The imaging range is imaged by a vision sensor that asynchronously acquires pixel data according to the amount of light incident on each of the multiple pixels arranged in two dimensions. Based on the captured image, information regarding the position of the target object in the imaging direction is measured. If the target object can no longer be detected within the imaging range, the illumination from the lighting unit is temporarily stopped. Measurement method.

10. By illuminating a predetermined imaging area underwater with light of a predetermined wavelength, The imaging range is imaged by a vision sensor that asynchronously acquires pixel data according to the amount of light incident on each of the multiple pixels arranged in two dimensions. Based on the captured image, information regarding the position of the target object in the imaging direction is measured. If the target object can no longer be detected within the imaging range, the illumination from the lighting unit is temporarily stopped. A program that instructs a measuring device to perform a process.

11. An illumination unit capable of switching and irradiating a predetermined imaging range in water with light of different wavelengths, An imaging control unit that causes the imaging range to be imaged by the imaging unit, A measuring unit measures information regarding the position of a target object in the imaging direction based on the image captured by the imaging unit, Equipped with, The imaging unit includes a vision sensor that asynchronously acquires pixel data according to the amount of light incident on each of the multiple pixels arranged in two dimensions. The imaging control unit changes the wavelength of the light emitted from the illumination unit when the target object can no longer be detected within the imaging range. Measuring device.

12. An illumination unit that irradiates a predetermined imaging area in water with light of a predetermined wavelength, An imaging control unit that causes the imaging range to be imaged by the imaging unit, A measuring unit measures information regarding the position of a target object in the imaging direction based on the image captured by the imaging unit, Equipped with, The imaging unit includes a vision sensor that asynchronously acquires pixel data according to the amount of light incident on each of the multiple pixels arranged in two dimensions. The imaging control unit moves the illumination unit when the target object can no longer be detected within the imaging range. Measuring device.

13. An illumination unit that irradiates a predetermined imaging range in water with light of a predetermined wavelength, An imaging control unit that causes the imaging range to be imaged by the imaging unit, A measuring unit measures information regarding the position of a target object in the imaging direction based on the image captured by the imaging unit, Equipped with, Multiple lighting units are provided, The imaging unit includes a vision sensor that asynchronously acquires pixel data according to the amount of light incident on each of the multiple pixels arranged in two dimensions. If the target object can no longer be detected within the imaging range, the imaging control unit will irradiate light from a different illumination unit. Measuring device.

Images