Measurement device, measurement program, and measurement method

Abstract

To track particles with high accuracy.SOLUTION: An imaging unit periodically images a fluid containing particles in a mixed manner from one direction. An irradiation unit irradiates the fluid with light whose color changes with respect to the one direction from an intersection direction intersecting the one direction. An identification unit identifies the particles from each image captured by the imaging unit. A position specifying unit specifies the positions of the particles identified by the identification unit within the image. A feature specifying unit specifies features including the colors of the particles identified by the identification unit. A two-dimensional tracking unit tracks the movement of the particles in two dimensions on the basis of the positions of the particles within each image specified by the position specifying unit and the features of the particles within each image specified by the feature specifying unit. A three-dimensional tracking unit tracks the movement of the particles in three dimensions on the basis of the tracking result of the particles in the two dimensions by the two-dimensional tracking unit and the colors of the particles within each image specified by the feature specifying unit.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a measurement apparatus, a measurement program, and a measurement method. [Background technology]

[0002] Conventionally, PTV (Particle Tracking Velocimetry) has been known, which measures the flow of a fluid such as liquid or gas by flowing particles in the fluid and tracking the trajectories of the particles. Non-Patent Documents 1 to 3 describe techniques for tracking the trajectories of particles in a fluid using PTV. In these techniques, the two-dimensional position of a particle is determined from the position of the particle in an image of the fluid captured by a camera, and the three-dimensional position of the particle is determined by determining the position of the particle in the depth direction of the image from the color of the particle in the image, and the particle is tracked based on the determined three-dimensional position. [Prior art documents] [Patent documents]

[0003] [Non-Patent Document 1] “Colour-coded tomography in fluid mechanics”, Ruck, Bodo, Optics and Laser Technology, Volume 43, Issue 2, p. 375-380., March 2011 [Non-Patent Document 2] “LCD-projector-based 3D color PTV”, Watamura T, Tasaka Y, Murai Y, Experimental Thermal and Fluid Science Volume 47 Pages 68-80, May 2013 [Non-Patent Document 3] “Low-cost 3D color particle tracking velocimetry: application to thermal turbulence in water”, Noto D, Tasaka Y, Murai Y, Experiments in Fluids 64:92, April 2023 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the position of a particle in the depth direction of an image, which is determined from the color of the particle, is not very accurate. Therefore, when the three-dimensional position of a particle is identified and the particle is tracked based on the identified three-dimensional position, the accuracy of the particle tracking is low.

[0005] The present invention has been made in view of the above, and an object of the present invention is to provide a measurement device, a measurement program, and a measurement method that are capable of tracking particles with high accuracy. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems and achieve the object, the measurement device of the present invention has an imaging unit, an irradiation unit, an identification unit, a position identification unit, a feature identification unit, a two-dimensional tracking unit, and a three-dimensional tracking unit. The imaging unit periodically images a fluid containing particles from one direction. The irradiation unit irradiates the fluid with light whose color has been changed from a crossing direction crossing the one direction to the one direction. The identification unit identifies particles from each image captured by the imaging unit. The position identification unit identifies the position of the particle identified by the identification unit in the image. The feature identification unit identifies the feature including the color of the particle identified by the identification unit. The two-dimensional tracking unit tracks the movement of the particle in two dimensions based on the position of the particle in each image identified by the position identification unit and the feature of the particle in each image identified by the feature identification unit. The three-dimensional tracking unit tracks the movement of the particle in three dimensions based on the results of tracking the particle in two dimensions by the two-dimensional tracking unit and the color of the particle in each image identified by the feature identification unit. Effect of the Invention

[0007] The present invention has the advantage that particles can be tracked with high accuracy. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of a measurement device according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a functional configuration of the control device according to the embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an example of a captured image according to the embodiment. [Figure 4A] FIG. 4A is a diagram showing an example of a pattern of overlapping particles according to an embodiment. [Figure 4B] FIG. 4B is a diagram showing an example of a pattern of overlapping particles according to an embodiment. [Diagram 5] FIG. 5 is a diagram illustrating an example of particle tracking according to the embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of particle tracking according to the embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of particle tracking according to the embodiment. [Figure 8] FIG. 8 is a diagram showing an example of the relationship between the hue angle and the position in the x direction according to the embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of a display according to the embodiment. [Figure 10] FIG. 10 is a flowchart illustrating an example of a procedure of the measurement process according to the embodiment. [Figure 11] FIG. 11 is a diagram illustrating a computer that executes a measurement processing program. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of the measurement device, the measurement program, and the measurement method according to the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to these embodiments. Each embodiment can be appropriately combined as long as the processing contents are not contradictory. EXAMPLES

[0010] [Typical configuration of the measurement device 10] An example of the configuration of a measuring device 10 according to the present embodiment will be described. Fig. 1 is a diagram showing an example of the configuration of a measuring device 10 according to the present embodiment. Hereinafter, in this embodiment, two orthogonal horizontal directions are defined as the x direction and the y direction, and a vertical direction orthogonal to the x direction and the y direction is defined as the z direction. In this embodiment, colors are indicated by black and white patterns.

[0011] The measuring device 10 is a device that measures the flow of a fluid by tracking the trajectory of particles in the fluid. In the following, an example will be described in which the fluid to be measured is a liquid, and the measuring device 10 measures the flow of the liquid by tracking the trajectory of particles in the liquid. FIG. 1 shows a rectangular container 1. The container 1 has a side wall formed of a transparent material such as glass, and is configured so that the inside can be seen through the side wall. In FIG. 1, the container 1 is arranged so that the side wall 1a is perpendicular to the x direction and the side wall 1b is perpendicular to the y direction. The container 1 stores a transparent liquid such as water. A large number of particles are placed in the liquid in the container 1, and the particles are mixed. The particles are formed into a sphere having a diameter of, for example, several hundred micrometers to several millimeters, and float in the liquid. A rotating flow is generated in the liquid in the container 1 by a flade (not shown) provided in the container 1. In FIG. 1, the liquid in the container 1 rotates around the z direction as the rotation axis.

[0012] The measurement device 10 includes an imaging unit 20, an irradiation unit 30, and a control device 40.

[0013] The photographing unit 20 periodically photographs the liquid in the container 1 from one direction. For example, the photographing unit 20 is configured as a camera capable of photographing images at a predetermined frame rate. The photographing unit 20 is disposed in the x direction with respect to the container 1, and photographs the liquid in the container 1 from the x direction through the side wall 1a.

[0014] The irradiation unit 30 irradiates the liquid in the container 1 with light whose color has changed in one direction from a crossing direction crossing the one direction. For example, the irradiation unit 30 is composed of a projector 31 and a Fresnel lens 32. The projector 31 is arranged in the y direction with respect to the container 1, with a projection lens 31a facing the container 1 side. The projector 31 irradiates light whose color has changed in the x direction from the projection lens 31a. For example, a color pattern is attached to the projection lens 31a of the projector 31. The color pattern is arranged so that the color changes in the x direction and is the same color in the z direction at each position in the x direction. The color of the color pattern in the x direction is arranged so that the hue angle gradually changes.

[0015] The projector 31 irradiates light through a color pattern, thereby irradiating light whose color changes in the x direction in response to the color pattern. The Fresnel lens 32 is provided on the optical axis of the light irradiated by the projector 31. In FIG. 1, the change in color of the light irradiated from the projector 31 to the Fresnel lens 32 is shown by the change in the black and white pattern. The refraction of each part of the Fresnel lens 32 is adjusted so that the light irradiated from the projector 31 becomes light parallel to the y direction. The light irradiated from the projector 31 becomes light parallel to the y direction by passing through the Fresnel lens 32. The light transmitted through the Fresnel lens 32 is irradiated from the y direction to the liquid in the container 1 through the side wall 1b. The light irradiated from the projector 31 changes color in the x direction by passing through the Fresnel lens 32, and is irradiated to the liquid in the container 1 as light parallel to the y direction for each color. As a result, the color of the light irradiated to the particles in the liquid changes depending on the position in the x direction. In Figure 1, the change in color of the light shining into the liquid is shown as a changing black and white pattern, causing particles in the liquid to change color depending on their position in the x-direction.

[0016] The photographing unit 20 is connected to the control device 40. The photographing unit 20 transmits data of the photographed image to the control device 40.

[0017] The control device 40 is a device that controls the measuring device 10. The control device 40 is, for example, a computer such as a personal computer or a server computer. The control device 40 may be implemented as a single computer, or may be implemented as a computer system using multiple computers. In this embodiment, the control device 40 is described as a single computer.

[0018] [Configuration of control device 40] Next, a description will be given of the configuration of the control device 40. Fig. 2 is a diagram showing an example of a functional configuration of the control device 40 according to the embodiment. The control device 40 has an external IF (interface) unit 41, an operation unit 42, a display unit 43, a storage unit 50, and a control unit 60.

[0019] The external IF unit 41 is an interface for transmitting and receiving various information to and from the outside. Examples of the external IF unit 41 include a USB (Universal Serial Bus) port and a communication interface. The external IF unit 41 is connected to the imaging unit 20 and receives data of an image captured by the imaging unit 20.

[0020] The operation unit 42 is an input device that accepts various operation inputs. Examples of the operation unit 42 include input devices that accept operation inputs such as a mouse and a keyboard. The operation unit 42 accepts input of various information. The operation unit 42 accepts operation inputs from a user, and inputs operation information indicating the content of the accepted operation to the control unit 60.

[0021] The display unit 43 is a display device that displays various information. Examples of the display unit 43 include a display device such as an LCD (Liquid Crystal Display) or a CRT (Cathode Ray Tube). The display unit 43 displays various information. For example, the display unit 43 displays various screens such as an operation screen. The control device 40 may receive access from an external terminal device used by a user or the like, display various screens such as an operation screen on the terminal device, and receive operation input from the various screens.

[0022] The storage unit 50 is a storage device that stores various data. For example, the storage unit 50 is a storage device such as a hard disk, a solid state drive (SSD), an optical disk, etc. The storage unit 50 may be a semiconductor memory in which data can be rewritten, such as a random access memory (RAM), a flash memory, or a non-volatile static random access memory (NVSRAM).

[0023] The storage unit 50 stores an OS (Operating System) and various programs executed by the control unit 60. For example, the storage unit 50 stores various programs including a measurement program that executes a measurement process described later. Furthermore, the storage unit 50 stores various data used in the programs executed by the control unit 60. For example, the storage unit 50 stores learning data 51, model data 52, particle data 53, tracking candidate data 54, two-dimensional trajectory data 55, and three-dimensional trajectory data 56.

[0024] The training data 51 is data for learning used to generate an identification model that identifies particles. Details of the training data 51 will be described later. The model data 52 is data for the identification model. The particle data 53 is data that stores information about identified particles. The tracking candidate data 54 is data that stores candidate particles that are candidates for tracking when tracking particles. The two-dimensional trajectory data 55 is data that stores, for each particle, the two-dimensional trajectory of the tracked particle. The three-dimensional trajectory data 56 is data that stores, for each particle, the three-dimensional trajectory of the tracked particle.

[0025] The control unit 60 is a device that controls the control device 40. As the control unit 60, electronic circuits such as a central processing unit (CPU) and a micro processing unit (MPU), and integrated circuits such as an application specific integrated circuit (ASIC) and a field programmable gate array (FPGA) can be adopted. The control unit 60 has an internal memory for storing programs and control data that define various processing procedures, and executes various processes using these. The control unit 60 functions as various processing units by the operation of various programs. For example, the control unit 60 has an identification unit 61, a generation unit 62, a position identification unit 63, a feature identification unit 64, a two-dimensional tracking unit 65, a three-dimensional tracking unit 66, and a display control unit 67.

[0026] The identification unit 61 identifies particles from each image captured by the imaging unit 20. For example, the identification unit 61 identifies particles based on shape and color from the image of the data received by the external IF unit 41 from the imaging unit 20. For example, the identification unit 61 extracts the outline of an object appearing in the image from the image and identifies particles from the shape of the outline. Alternatively, for example, the identification unit 61 extracts an area in which a color appears from the image and identifies particles from the shape of the extracted area.

[0027] 3 is a diagram illustrating an example of a captured image according to an embodiment. For example, the identification unit 61 identifies an area in the image that shows a circular object or color as a particle 2. In FIG. 3, since many particles 2 are captured, only some of the particles 2 are labeled with symbols.

[0028] Here, a plurality of particles 2 may appear overlapping in the image.

[0029] Therefore, in this embodiment, an identification model is used to identify overlapping particles 2.

[0030] The control device 40 according to this embodiment prepares learning data 51 in advance and stores it in the storage unit 50. The learning data 51 stores, for each of a plurality of patterns in which a plurality of particles 2 are overlapped, an image of the overlapping portion of the plurality of particles 2 and the color of each of the overlapping particles 2. For example, the learning data 51 is generated by a user identifying each of the overlapping particles 2 and recording the color of each of the particles 2 for each of a plurality of patterns in which a plurality of particles 2 are overlapped in various states.

[0031] 4A and 4B are diagrams showing an example of a pattern of overlapping particles 2 according to the embodiment. In FIG. 4A, two particles 2 (2a, 2b) overlap. In FIG. 4B, two particles 2 (2c, 2d) overlap. For example, in the learning data 51, images of the two particles 2 and the colors of the particles 2 are stored in association with each other for each pattern of overlapping particles 2 shown in FIG. 4A and FIG. 4B.

[0032] The generation unit 62 performs machine learning on the learning data 51 to generate an identification model that identifies overlapping particles 2. For example, an object detection model such as YOLO can be used as a machine learning method, but the method is not limited to this. The identification model may be generated using any method as long as it can identify overlapping particles 2. The generation unit 62 stores data of the generated identification model in the storage unit 50 as model data 52.

[0033] When there are overlapping particles 2 in each image captured by the imaging unit 20, the identification unit 61 identifies each particle 2 using the identification model of the model data 52. For example, for an area containing contours or colors not identified as particles 2, the identification unit 61 identifies each overlapping particle 2 and also identifies the color of each particle 2 using the identification model of the model data 52.

[0034] In addition, if the identification model is capable of identifying the particle 2 and the color of the particle 2 for both a single particle 2 and overlapping particles 2, the identification unit 61 may use the identification model to identify the particle 2 and the color of the particle 2 for both a single particle 2 and overlapping particles 2.

[0035] The position identifying unit 63 identifies the position within the image of the particle 2 identified by the identification unit 61. For example, the position identifying unit 63 identifies the position within the plane of the image for the particle 2 identified by the identification unit 61. In this embodiment, the position identifying unit 63 identifies the y-direction and z-direction coordinates within the image for the particle 2 identified by the identification unit 61. The position identifying unit 63 stores the identified position within the image of each particle 2 in the particle data 53 for each image.

[0036] The feature identifying unit 64 identifies the features of the particle 2 identified by the identifying unit 61. For example, the feature identifying unit 64 identifies color, brightness distribution, shape, and size as the features of the particle 2 identified by the identifying unit 61. The feature identifying unit 64 stores the identified features of each particle 2 for each image in the particle data 53. Note that, for overlapping particles 2, the feature identifying unit 64 identifies the color of each particle 2 from the color identification result of each particle 2 by the identification model of the model data 52. Furthermore, if the identification model is capable of identifying the color of one particle 2 and overlapping particles 2, the feature identifying unit 64 identifies the color of each particle 2 from the color identification result of each particle 2 by the identification model of the model data 52 for one particle 2 and overlapping particles 2.

[0037] The two-dimensional tracking unit 65 tracks each particle 2 in the two-dimensional image using a Kalman filter. For example, the two-dimensional tracking unit 65 tracks the movement of the particle 2 in two dimensions based on the position of the particle 2 in each image specified by the position specifying unit 63 and the features of the particle 2 in each image specified by the feature specifying unit 64. For example, the two-dimensional tracking unit 65 tracks the movement of the particle 2 in two dimensions based on the position of the particle 2 in each image stored in the particle data 53 and the features of the particle 2. Specifically, the two-dimensional tracking unit 65 predicts the position of the target particle in the second image taken after the first image from the position of the target particle in the first image taken and the moving speed and moving direction of the target particle as the target particle to be tracked for each particle 2. Then, the two-dimensional tracking unit 65 specifies the particle 2 having similar features to the target particle among the particles 2 within a predetermined range from the predicted position in the second image as a candidate particle for tracking. For example, the two-dimensional tracking unit 65 identifies particles 2 within a predetermined range from the predicted position in the second image that are similar in color, brightness distribution, shape, and size to the target particle as candidate particles for tracking. For example, the two-dimensional tracking unit 65 obtains the similarity in color, brightness distribution, shape, and size between the target particle and each particle 2 within the predetermined range, and identifies particle 2 with the highest similarity as a candidate particle for tracking. In addition, the two-dimensional tracking unit 65 further identifies the positional relationship of particles 2 that move in the same way as the target particle in the captured first image, and identifies particle 2 that corresponds to the position of the target particle in the identified positional relationship as a candidate particle for tracking among particles 2 within the predetermined range.

[0038] The two-dimensional tracking unit 65 predicts the position of the particle 2 in the third image taken after the second image from the position of the candidate particle in the second image and the moving speed and moving direction of the target particle or the candidate particle. Then, the two-dimensional tracking unit 65 specifies the particle 2 having similar characteristics to the target particle among the particles 2 within a predetermined range from the predicted position in the third image as a candidate particle for tracking. The two-dimensional tracking unit 65 repeats the process of specifying the candidate particle for the images taken in this way, and specifies the movement of the candidate particle as the movement of the target particle for the target particle for which the candidate particle is specified in three or more consecutive images. For example, the two-dimensional tracking unit 65 stores the position of the candidate particle in the image in the tracking candidate data 54 for each candidate particle specified for the target particle for each target particle in the order of taking the images. Then, the two-dimensional tracking unit 65 stores the position data of the candidate particle in three or more consecutive images stored in the tracking candidate data 54 as the position data of the target particle in the three or more consecutive images in the two-dimensional trajectory data 55 as the position data of the target particle in the three or more consecutive images. As a result, the two-dimensional position data of the particle 2 whose trajectory can be tracked is stored in the two-dimensional trajectory data 55. The two-dimensional tracking unit 65 deletes the data of the target particle in which the candidate particle is not identified in three or more consecutive images from the tracking candidate data 54. As a result, the target particle in which the candidate particle is not identified is excluded from the target of tracking. Note that, when the two-dimensional tracking unit 65 identifies a candidate particle in a captured image, it may store information indicating the image capture time or the image capture order in the tracking candidate data 54 together with the position in the image. Then, the two-dimensional tracking unit 65 may store information indicating the image capture time or the image capture order in the two-dimensional trajectory data 55 together with the position data of the candidate particle.

[0039] Fig. 5 is a diagram showing an example of tracking of a particle 2 according to an embodiment. Fig. 5 shows three particles 2 (2d to 2f) photographed at three consecutive times t, t+1, and t+2. Two particles 2 (2d, 2e) are moving in parallel in the upper right direction. One particle 2 (2f) is moving in the upper left direction.

[0040] In the following, a case will be described in which each particle 2 can be tracked at times t and t+1, and the movement of two particles 2d is tracked at time t+2. The two-dimensional tracking unit 65 assumes the particle 2d to be a target particle to be tracked, and predicts the position at time t+2 based on the positions of the particle 2d at times t and t+1. For example, the two-dimensional tracking unit 65 calculates the moving speed and moving direction of the particle 2d from the positions of the particle 2d at times t and t+1. In FIG. 5, the moving speed and moving direction of each particle 2 are shown as a velocity vector 5 at time t. For example, the two-dimensional tracking unit 65 predicts the position of the particle 2d at time t+2 using the position of the particle 2d at time t+1 and the velocity vector 5 of the particle 2d. In FIG. 5, a predetermined range from the predicted position of the particle 2d at time t+2 is shown as a prediction range 6. The prediction range 6 is a range of a predetermined distance from the predicted position. The predetermined distance may be fixed, or may be set from the operation unit 42 or the like. The predetermined distance is preferably several times (for example, 5 to 10 times) the diameter of the particle 2. The predetermined distance may be variable depending on the speed of the target particle. For example, the predetermined distance may be increased as the speed of the target particle increases.

[0041] The two-dimensional tracking unit 65 identifies, among the particles 2 in the prediction range 6 at time t+2, a particle 2 having similar characteristics to the target particle as a candidate particle for tracking. For example, among the particles 2 in the prediction range 6, the two-dimensional tracking unit 65 identifies, as a candidate particle for tracking, a particle 2d having similar color, brightness distribution, shape, and size to a particle 2d at time t+1. In addition, the two-dimensional tracking unit 65 further identifies the positional relationship of a particle 2e that moves in the same manner as the particle 2d in the image at time t+1, and identifies, among the particles 2 in the prediction range 6, a particle 2 that corresponds to the particle 2d in the identified positional relationship at time t+1 as a candidate particle for tracking. In the image at time t+2, the particle 2d corresponds to the position of the particle 2d in the positional relationship at time t+1, and is therefore identified as a candidate particle.

[0042] In this embodiment, for a target particle for which a candidate particle has been identified in four consecutive images, the two-dimensional tracking unit 65 identifies the movement of the candidate particle as the movement of the target particle.

[0043] Fig. 6 is a diagram showing an example of tracking of a particle 2 according to the embodiment. In Fig. 6, for a particle 2g in an image at time t, particles 2t1 to 2t4 are specified as candidate particles for tracking in images from time t+1 to t+4. In this case, the two-dimensional tracking unit 65 stores the position data of the particles 2t1 to 2t4 in the two-dimensional trajectory data 55 as the positions of the particle 2g from time t+1 to t+4.

[0044] The two-dimensional tracking unit 65 tracks the movement of each particle 2 in two dimensions for each particle 2 whose trajectory has been tracked and stored in the two-dimensional trajectory data 55, treating each particle 2 as a target particle.

[0045] For the particle 2 in the first image or the particle 2 that has not been tracked from the image captured previously, the two-dimensional tracking unit 65 tracks the particle 2 with the moving speed and moving direction of the particle 2 set as initial conditions determined in advance. For example, when there is no image at time t, or for the particle 2 that has not been tracked from the image at time t, the two-dimensional tracking unit 65 predicts the position of the particle 2d at time t+2 using the position of the particle 2d at time t+1 and the moving speed and moving direction of the initial conditions. Then, the two-dimensional tracking unit 65 specifies the particle 2 that has similar characteristics to the target particle among the particles 2 within a predetermined range from the predicted position as a candidate particle for tracking. The initial conditions of the moving speed and moving direction are set according to the flow of the liquid in the container 1 and the movement of the particle 2. For example, the liquid in the container 1 shown in FIG. 1 rotates with the z direction as the rotation axis. For this reason, on the left side of the image captured by the imaging unit 20, the particle 2 moves from right to left or left to right, and on the right side of the image captured by the imaging unit 20, the particle 2 moves from left to right or right to left. For example, the movement direction of the initial condition is set to be from left to right on the left side of the image, and from right to left on the right side of the image. The movement speed of the initial condition is set to the standard movement speed of the particle 2 in the container 1. After the two-dimensional tracking unit 65 identifies the candidate particle with the movement speed and movement direction of the initial condition, it subsequently obtains the movement speed and movement direction of the candidate particle using each image, and identifies the movement of the candidate particle from the next captured image. For a target particle for which a candidate particle has been identified in three or more consecutive images, the two-dimensional tracking unit 65 identifies the movement of the candidate particle as the movement of the target particle. Although the two-dimensional tracking unit 65 can track a small number of particles 2 at first, the number of particles 2 that can be tracked increases over time, and therefore it can track many particles 2 over time.

[0046] Incidentally, when multiple particles 2 overlap, particles 2 with a large overlapping area or particles 2 that are completely hidden behind others are thought to be temporarily lost. In particular, to observe the flow of a liquid in detail, it is better to have a large number of particles 2 in the liquid, but as the number of particles 2 increases, the particles 2 tend to overlap more frequently and more particles 2 are temporarily lost.

[0047] Therefore, the two-dimensional tracking unit 65 specifies a virtual candidate particle when the particle 2, whose movement is being tracked in two dimensions, is temporarily lost. For example, when the two-dimensional tracking unit 65 cannot specify a candidate particle in the second image taken after the first image for the particle 2, whose movement is being tracked in two dimensions in the first image, the two-dimensional tracking unit 65 specifies a virtual candidate particle at a predicted position in the second image based on the moving speed and moving direction of the target particle. Then, for the specified virtual candidate particle, the two-dimensional tracking unit 65 predicts the position of the particle 2 in the third image taken after the second image from the position of the virtual candidate particle in the second image and the moving speed and moving direction of the target particle or the virtual candidate particle, and when the particle 2, whose characteristics are similar to the target particle, among the particles 2 within a predetermined range from the predicted position in the third image, can be specified as a candidate particle for tracking, the position of the virtual candidate particle in the second image is specified as the position of the target particle in the second image.

[0048] FIG. 7 is a diagram showing an example of tracking of a particle 2 according to an embodiment. FIG. 7 shows the movement of two particles 2 (2h, 2i) in images from time t to time t+2. For example, the particle 2h flows on the back side of the container 1 in the x direction, and the particle 2i flows on the front side of the container 1 in the x direction, and the particles 2h and 2i move so as to approach each other on the screen. In FIG. 7, the particles 2h and 2i are being tracked in two-dimensional movements in the image at time t. In the image at time t+1, the particle 2h completely overlaps with the particle 2i, so that the particle 2h is not captured. When the two-dimensional tracking unit 65 cannot identify a candidate particle in the image at time t+1 for the particle 2h whose movement is being tracked in two dimensions in the image at time t, the two-dimensional tracking unit 65 identifies a virtual candidate particle 2ha at the predicted position of the particle 2h in the image at time t+1. The virtual candidate particle 2ha is identified as moving at the moving speed and moving direction of the particle 2h. Then, the two-dimensional tracking unit 65 predicts the position of particle 2 in the image at time t+2 from the virtual position of the virtual candidate particle 2ha in the image at time t+1 and the moving speed and moving direction of particle 2h or virtual candidate particle 2ha, and if it can identify the candidate particle 2hb to be tracked in the image at time t+2, it identifies the position of the virtual candidate particle 2ha in the image at time t+1 as the position of particle 2h in the image at time t+1.

[0049] The two-dimensional tracking unit 65 stores the position of the virtual candidate particle 2ha in the image at time t+1 and the position of the candidate particle 2hb in the image at time t+2 in the two-dimensional trajectory data 55 as data on the position of the particle 2h at times t+1 and t+2.

[0050] The three-dimensional tracking unit 66 tracks the movement of the particle 2 in three dimensions based on the two-dimensional tracking result of the particle 2 by the two-dimensional tracking unit 65 and the color of the particle 2 in each image identified by the feature identification unit 64. For example, the three-dimensional tracking unit 66 obtains the position of the particle 2 in the depth direction of the image from the color in each image for each particle 2 stored in the two-dimensional trajectory data 55. For example, the three-dimensional tracking unit 66 converts the RGB value indicating the color of the particle 2 in the image into a hue angle, and identifies the position of the particle 2 in the x direction from the hue angle of the color of the particle 2. The hue angle of the color of the particle 2 in the liquid changes depending on the position in the x direction. Therefore, by determining the relationship between the hue angle and the position in the x direction in advance, the position of the particle 2 in the x direction can be identified from the hue angle of the color of the particle 2. FIG. 8 is a diagram showing an example of the relationship between the hue angle and the position in the x direction according to the embodiment. In FIG. 8, a hue circle is shown, and the position in the x direction is shown around the hue circle. For example, the three-dimensional tracking unit 66 specifies the x-direction position of the particle 2 from the hue angle of the color of the particle 2 in each image using conversion data or a conversion formula that defines the relationship between the hue angle and the position in the x direction for each particle 2 stored in the two-dimensional trajectory data 55. Then, the three-dimensional tracking unit 66 stores the y-direction and z-direction positions of each particle 2 stored in the two-dimensional trajectory data 55 and the specified x-direction position in the three-dimensional trajectory data 56. As a result, three-dimensional position data for the particle 2 whose trajectory has been tracked is stored in the three-dimensional trajectory data 56.

[0051] Here, the position of particle 2 in two dimensions (y direction and z direction) from each image is determined with high accuracy. On the other hand, the position in the depth direction (x direction) of the image determined from the color of particle 2 is less accurate. For this reason, when the three-dimensional position of particle 2 is determined by determining the position of particle 2 in the depth direction of the image from the color of particle 2 shown in the image and particle 2 is tracked based on the determined three-dimensional position, the accuracy of tracking particle 2 is poor because of the poor accuracy of the position in the x direction. Furthermore, tracking particle 2 in three dimensions imposes a large computational load.

[0052] Therefore, the measuring device 10 according to this embodiment tracks the movement of the particle 2 in two dimensions from each captured image, and tracks the movement of the particle 2 in three dimensions from the results of tracking the particle 2 in two dimensions and the color of the particle 2 in each image. Since the two-dimensional position of the particle 2 can be determined with high accuracy from each image, tracking the movement of the particle 2 in two dimensions can improve the accuracy of tracking the particle 2. Furthermore, since the measuring device 10 according to this embodiment tracks the particle 2 in two dimensions, the calculation load can be reduced.

[0053] The display control unit 67 performs various display controls. For example, the display control unit 67 performs display control to display various screens such as an operation screen on the display unit 43. In addition, the display control unit 67 displays various information in response to an operation from the operation unit 42 on the operation screen. For example, the display control unit 67 displays the flow in the liquid in the container 1. For example, the display control unit 67 displays the moving state of the particles 2 based on the three-dimensional trajectory data 56. FIG. 9 is a diagram showing an example of a display according to the embodiment. FIG. 9 shows a case where a velocity vector 5 indicating the moving speed and moving direction of each particle 2 is displayed at the position of each particle 2 when the container 1 is viewed from the z direction. In FIG. 9, since many velocity vectors 5 are shown, only some of the velocity vectors 5 are labeled. The measuring device 10 according to the embodiment can track each particle 2 even when many particles 2 are mixed in the liquid in the container 1, so that the flow of the liquid can be measured in detail.

[0054] [Processing flow] The flow of the measurement process in which the measurement device 10 tracks the trajectory of the particle 2 in the liquid will be described below. Fig. 10 is a flowchart showing an example of the procedure of the measurement process according to the embodiment.

[0055] The identification unit 61 identifies the particles 2 from each image captured by the imaging unit 20 (step S10). The position identification unit 63 identifies the position of the identified particle 2 in the image, and stores the identified position of each particle 2 in the image for each image in the particle data 53 (step S11). The feature identification unit 64 identifies features including the color of the identified particle 2, and stores the identified feature of each particle 2 for each image in the particle data 53 (step S12).

[0056] The two-dimensional tracking unit 65 tracks the movement of the particle 2 in two dimensions based on the position of the particle 2 in each image and the characteristics of the particle 2 in each image, and stores the two-dimensional position data for the particle 2 whose trajectory has been tracked in the two-dimensional trajectory data 55 (step S13).

[0057] The three-dimensional tracking unit 66 tracks the movement of the particle 2 in three dimensions based on the two-dimensional tracking results of the particle 2 and the color of the particle 2 in each image, and stores the three-dimensional position data for the particle 2 whose trajectory has been tracked in the three-dimensional trajectory data 56 (step S14), and ends the processing.

[0058] In the above embodiment, the fluid to be measured is a liquid. However, the disclosed technology is not limited to this. The fluid to be measured may be a gas. For example, the measuring device 10 can measure the flow of the gas by tracking the trajectory of particles flowing through the gas, such as air.

[0059] [effect] As described above, the measuring device 10 according to the present embodiment includes the photographing unit 20, the irradiating unit 30, the identifying unit 61, the position identifying unit 63, the feature identifying unit 64, the two-dimensional tracking unit 65, and the three-dimensional tracking unit 66. The photographing unit 20 periodically photographs the fluid in which the particles 2 are mixed from one direction (x direction). The irradiating unit 30 irradiates the fluid with light whose color has changed from a cross direction (y direction) that crosses the one direction. The identifying unit 61 identifies the particle 2 from each image photographed by the photographing unit 20. The position identifying unit 63 identifies the position of the particle 2 identified by the identifying unit 61 in the image. The feature identifying unit 64 identifies the feature of the particle 2 identified by the identifying unit 61, including the color. The two-dimensional tracking unit 65 tracks the movement of the particle 2 in two dimensions based on the position of the particle 2 in each image identified by the position identifying unit 63 and the feature of the particle 2 in each image identified by the feature identifying unit 64. The three-dimensional tracking unit 66 tracks the movement of the particle 2 in three dimensions based on the two-dimensional tracking result of the particle 2 by the two-dimensional tracking unit 65 and the color of the particle 2 in each image identified by the feature identifying unit 64. This enables the measuring device 10 to track the particle 2 with high accuracy.

[0060] Furthermore, the two-dimensional tracking unit 65 predicts the position of the target particle in a second image captured after the first image from the position of the target particle in the captured first image and the moving speed and moving direction of the target particle for each particle 2 as a target particle to be tracked, and identifies particles 2 having characteristics similar to the target particle among particles 2 within a predetermined range from the predicted position in the second image as candidate particles to be tracked, and tracks the movement of the particles 2 in two dimensions. This allows the measuring device 10 to track the movement of each particle 2 in two dimensions with high accuracy.

[0061] Furthermore, the feature identifying unit 64 further identifies the luminance distribution, shape, and size of the particle 2 as features of the particle 2. The two-dimensional tracking unit 65 identifies, among the particles 2 within a predetermined range, particles 2 that are similar in color, luminance distribution, shape, and size to the target particle as candidate particles for tracking. This enables the measuring device 10 to track the movement of the target particle with high accuracy, and to track the movement of each particle 2 in two dimensions with high accuracy.

[0062] In addition, the two-dimensional tracking unit 65 further specifies the positional relationship of the particle 2 that moves in the same manner as the target particle in the captured first image, and specifies, among the particles 2 within the predetermined range, the particle 2 that corresponds to the position of the target particle in the specified positional relationship as a candidate particle for tracking. This allows the measuring device 10 to track the movement of the target particle with high accuracy even when there are multiple particles 2 within the predetermined range.

[0063] Furthermore, the two-dimensional tracking unit 65 predicts the position of the particle 2 in the third image taken after the second image from the position of the candidate particle in the second image and the moving speed and moving direction of the target particle or the candidate particle, and repeats the process of identifying particles 2 having similar characteristics to the target particle among particles 2 within a predetermined range from the predicted position in the third image as candidate particles for tracking, and identifies the movement of the candidate particle as the movement of the target particle for the target particle identified as the candidate particle in three or more consecutive images. This allows the measuring device 10 to track the movement of the target particle with high accuracy even when multiple candidate particles for the target particle are identified in the second image and the third image.

[0064] In addition, the two-dimensional tracking unit 65 determines the moving speed and moving direction of the target particle based on the movement of the target particle between the image captured before the first image and the first image, or uses a predetermined initial condition, thereby enabling the measuring device 10 to track the particle 2 with high accuracy.

[0065] The measuring device 10 according to the present embodiment further includes a storage unit 50. The storage unit 50 stores model data 52 of a discrimination model obtained by machine learning of learning data 51 that determines the color of each particle 2 when the particles 2 appearing in an image overlap. The identification unit 61 uses the discrimination model of the model data 52 stored in the storage unit 50 to identify the color of the overlapping particles 2 from each image captured by the image capture unit 20. This allows the measuring device 10 to track the overlapping particles 2 even when the particles 2 appearing in the image overlap. For example, the measuring device 10 can track the overlapping particles 2 even when the particles 2 are mixed in the fluid and many particles 2 are overlapped in the image. The measuring device 10 can track each particle 2 by mixing many particles 2 in the fluid, so that the flow of the fluid can be measured in detail.

[0066] Furthermore, when a candidate particle cannot be identified in the second image for particle 2 whose movement is being tracked in two dimensions in the first image, the two-dimensional tracking unit 65 identifies a virtual candidate particle at a predicted position in the second image based on the moving speed and moving direction of the target particle. This allows the measuring device 10 to track the movement of the target particle by identifying the virtual candidate particle even when a candidate particle for the target particle cannot be identified in the second image.

[0067] Furthermore, the two-dimensional tracking unit 65 predicts the position of the particle 2 in the third image taken after the second image from the position of the virtual candidate particle in the second image and the moving speed and moving direction of the target particle or the virtual candidate particle for the identified virtual candidate particle, and when a particle having characteristics similar to the target particle among the particles 2 within a predetermined range from the predicted position in the third image can be identified as a candidate particle for tracking, the position of the virtual candidate particle in the second image is identified as the position of the target particle in the second image. As a result, even if the measuring device 10 cannot identify a candidate particle for the target particle in the second image, it can identify the position of the target particle in the second image and track the movement of the target particle.

[0068] Moreover, the irradiation unit 30 irradiates the fluid with light whose hue changes sequentially in one direction. The three-dimensional tracking unit 66 obtains the hue from the color of the particle 2 in each image, obtains the position of the particle 2 in one direction in each image from the hue, and tracks the movement of the particle 2 in three dimensions from the position of the particle 2 in each image tracked by the two-dimensional tracking unit 65 and the position of the particle 2 in one direction in each image. This allows the measuring device 10 to track the movement of the particle 2 in three dimensions with high accuracy. EXAMPLES

[0069] Although the embodiments of the disclosed device have been described above, the disclosed technology may be embodied in various different forms other than the above-described embodiments.

[0070] Moreover, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific state of distribution and integration of each device is not limited to that shown in the figure, and all or a part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions. For example, each processing unit of the identification unit 61, the generation unit 62, the position identification unit 63, the feature identification unit 64, the two-dimensional tracking unit 65, the three-dimensional tracking unit 66, and the display control unit 67 may be appropriately integrated. Furthermore, the processing of each processing unit may be appropriately separated into processing of multiple processing units. Furthermore, each processing function performed by each processing unit may be realized in whole or in any part by a CPU and a program analyzed and executed by the CPU, or may be realized as hardware by wired logic.

[0071] [Measurement processing program] The various processes described in the above embodiments can also be realized by executing a prepared program on a computer system such as a personal computer or a workstation. Therefore, an example of a computer system that executes a program having the same functions as the above embodiments will be described below. Figure 11 is a diagram showing a computer that executes a measurement processing program.

[0072] 11, a computer 300 includes a central processing unit (CPU) 310, a hard disk drive (HDD) 320, and a random access memory (RAM) 340. These components 300 to 340 are connected to each other via a bus 400.

[0073] The HDD 320 stores in advance a measurement processing program 320a that performs the same functions as the above-mentioned identification unit 61, generation unit 62, position identification unit 63, feature identification unit 64, two-dimensional tracking unit 65, three-dimensional tracking unit 66, and display control unit 67. The measurement processing program 320a may be separated as appropriate.

[0074] In addition, the HDD 320 stores various types of information, such as the above-mentioned learning data 51, model data 52, particle data 53, tracking candidate data 54, two-dimensional trajectory data 55, and three-dimensional trajectory data 56.

[0075] Then, the CPU 310 reads out the measurement processing program 320a from the HDD 320 and executes it to perform the same operations as those of the respective processing units in the embodiment. That is, the measurement processing program 320a performs the same operations as those of the recognition unit 61, the generation unit 62, the position identification unit 63, the feature identification unit 64, the two-dimensional tracking unit 65, the three-dimensional tracking unit 66, and the display control unit 67.

[0076] It should be noted that the above-mentioned measurement processing program 320a does not necessarily need to be stored in the HDD 320 from the beginning.

[0077] For example, the program may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD disk, magneto-optical disk, or IC card that is inserted into computer 300. Computer 300 may then read and execute the program from these.

[0078] Furthermore, the program may be stored in "another computer (or server)" connected to computer 300 via a public line, the Internet, a LAN, a WAN, etc. Then, computer 300 may read and execute the program from the other computer (or server). [Explanation of symbols]

[0079] 1 container 1a, 1b side wall 2, 2a~2i, 2t1~2t4 particles 2ha, 2hb candidate particles 5. Velocity Vector 6 Prediction range 10. Measurement Equipment 20. Photography Department 30 Irradiation unit 31 Projector 31a Projection lens 32 Fresnel Lens 40 Control device 41 External IF section 42 Operation section 43 Display section 50 Storage section 51 Training Data 52 Model Data 53 Particle Data 54 Tracking candidate data 55 2D trajectory data 56 3D trajectory data 60 Control section 61 Identification section 62 Generation part 63 Location identification part 65 2D Tracking Unit 66 3D Tracking Unit 64 Feature Identification Unit 67 Display control section 300 Computers 310 CPU 320 HDD 320a Measurement and processing program 400 Bus

Claims

1. an imaging unit that periodically images a fluid containing particles from one direction; an irradiation unit that irradiates the fluid with light whose color has been changed from a cross direction crossing the one direction to the one direction; an identification unit that identifies the particles from each image captured by the imaging unit; a position identification unit that identifies a position within an image of the particle identified by the identification unit; a feature identifying unit that identifies features including a color of the particle identified by the identifying unit; a two-dimensional tracking unit that tracks the movement of the particle in two dimensions based on the position of the particle in each image identified by the position identifying unit and the feature of the particle in each image identified by the feature identifying unit; a three-dimensional tracking unit that tracks movement of the particle in three dimensions based on a result of tracking the particle in two dimensions by the two-dimensional tracking unit and a color of the particle in each image identified by the feature identifying unit; A measuring device having the above structure.

2. The two-dimensional tracking unit predicts, for each particle, a position of the target particle in a second image captured after the first image from the position of the target particle in the captured first image and the moving speed and moving direction of the target particle, and identifies particles having characteristics similar to those of the target particle among particles within a predetermined range from the predicted position in the second image as candidate particles for tracking, and tracks the movement of the particles in two dimensions. The measurement device according to claim 1 .

3. the characteristic specifying unit further specifies a luminance distribution, a shape, and a size of the particle as the characteristics of the particle; The two-dimensional tracking unit identifies particles within the predetermined range that are similar in color, brightness distribution, shape, and size to the target particle as candidate particles for tracking. The measurement device according to claim 2 .

4. The two-dimensional tracking unit further specifies a positional relationship of particles that move in the same manner as the target particle in the captured first image, and specifies, among the particles within the predetermined range, a particle that corresponds to the position of the target particle in the specified positional relationship as a candidate particle for tracking. The measurement device according to claim 2 .

5. The two-dimensional tracking unit predicts, for the identified candidate particle, a position of the particle in a third image taken after the second image from the position of the candidate particle in the second image and the moving speed and moving direction of the target particle or the candidate particle, and repeats the process of identifying particles having characteristics similar to those of the target particle among particles within a predetermined range from the predicted position in the third image as candidate particles for tracking, and for the target particle for which a candidate particle has been identified in three or more consecutive images, identifies the movement of the candidate particle as the movement of the target particle. The measurement device according to claim 2 .

6. The two-dimensional tracking unit determines a moving speed and a moving direction of the target particle based on a movement of the target particle between an image captured before the first image and the first image, or uses a predetermined initial condition. The measurement device according to claim 2 .

7. The method further includes a storage unit that stores model data of a discrimination model that is machine-learned from learning data that determines the color of each particle when particles appearing in an image overlap, The identification unit identifies the color of overlapping particles from each image captured by the image capture unit using a discrimination model of the model data stored in the storage unit. The measurement device according to claim 1 .

8. When a candidate particle cannot be identified in the second image for a particle whose movement in two dimensions is being tracked in the first image, the two-dimensional tracking unit identifies a virtual candidate particle at a predicted position in the second image based on the moving speed and moving direction of the target particle. The measurement device according to claim 2 .

9. The two-dimensional tracking unit predicts the position of the identified virtual candidate particle in a third image taken after the second image from the position of the virtual candidate particle in the second image and the moving speed and moving direction of the target particle or the virtual candidate particle, and when a particle having characteristics similar to the target particle among particles within a predetermined range from the predicted position in the third image can be identified as a candidate particle for tracking, the two-dimensional tracking unit specifies the position of the virtual candidate particle in the second image as the position of the target particle in the second image. The measurement device according to claim 8.

10. The irradiation unit irradiates the fluid with light whose hue changes sequentially in the one direction, The three-dimensional tracking unit obtains a hue from the color of the particle in each image, obtains a position of the particle in each image in the one direction from the hue, and tracks the movement of the particle in three dimensions from the position of the particle in each image tracked by the two-dimensional tracking unit and the position of the particle in each image in the one direction. The measurement device according to claim 1 .

11. A fluid containing particles is irradiated with light whose color is changed from a cross direction intersecting with one direction from an irradiation unit, and the particles are identified from each image of the fluid periodically captured from one direction by an imaging unit; determining a location within the image of the identified particle; Identifying characteristics of the identified particles, including color; tracking the movement of the particle in two dimensions based on the position of the particle in each identified image and the characteristics of the particle in each identified image; tracking the movement of the particle in three dimensions from the position of the particle in each pre-tracked image and the color of the particle in each identified image; A measurement program that causes a computer to carry out processing.

12. A fluid containing particles is irradiated with light whose color is changed from a cross direction intersecting with one direction from an irradiation unit, and the particles are identified from each image of the fluid periodically captured from one direction by an imaging unit; determining a location within the image of the identified particle; Identifying characteristics of the identified particles, including color; tracking the movement of the particle in two dimensions based on the position of the particle in each identified image and the characteristics of the particle in each identified image; tracking the movement of the particle in three dimensions from the position of the particle in each pre-tracked image and the color of the particle in each identified image; A measurement method for how a process is carried out by a computer.

Images