Information processing device

Abstract

This information processing device for detecting and tracking an arbitrary number of foreground objects in an image as separate individuals by a binarization process comprises: a storage means for storing, for each individual, tracking information that includes a threshold value which has been used in the binarization process and information pertaining to a detected individual; a threshold value setting means for setting a prescribed number of threshold values on the basis of the threshold value that is included in the tracking information; a binarization means for using the set prescribed number of threshold values to binarize the image so that a prescribed number of binarized images are generated; a collation means for detecting, from the generated binarized images, an object that matches the information pertaining to the individual which is included in the tracking information; and an update means for updating the tracking information with information pertaining to the matching object and the threshold value that has been used to generate the binarized image of the object which has been successfully detected.

Description

[Technical Field]

[0001] The present invention relates to an information processing apparatus, an information processing method, and a recording medium. [Background technology]

[0002] Objects captured in images obtained from imaging devices are detected and tracked using binarization processing.

[0003] For example, Patent Document 1 describes detecting and tracking any number of cells from frame images, which are captured at regular time intervals within a predetermined region, by binarization. It also describes setting multiple thresholds based on the range of brightness values ​​of the frame images during the binarization process.

[0004] Furthermore, Patent Document 2 describes detecting and tracking various objects from frame images taken of the sea surface at regular time intervals using binarization processing. It also describes setting the threshold used in the binarization processing based on the difference image between a short-term background image and a long-term background image.

[0005] Furthermore, Patent Document 3 describes identifying a region in the captured image that is highly likely to contain a target for detection as a prediction region, and setting the threshold used for detection processing in the identified prediction region to be smaller than the threshold used for detection processing in regions other than the prediction region. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Patent No. 6090770 [Patent Document 2] Japanese Patent Publication No. 2018-152106 [Patent Document 3] Japanese Patent Publication No. 2014-197353 [Patent Document 4] WO2021 / 214994 [Overview of the project] [Problems that the invention aims to solve]

[0007] As described in Patent Document 1, a method for setting multiple thresholds based on the range of brightness values ​​of a frame image; as described in Patent Document 2, a method for setting thresholds based on the difference image between a short-term background image and a long-term background image; and as described in Patent Document 3, a method in which the threshold used in the detection process for the predicted region is smaller than the threshold used in the detection process for regions other than the predicted region, all of these methods result in setting a common threshold for all tracked individuals. However, the lighting, background, and appearance are not necessarily the same for all tracked individuals. Therefore, in binarization processing using a common threshold, the extraction results for each individual may not be stable.

[0008] The object of the present invention is to provide an information processing device that solves the above-mentioned problem, namely, the problem that the extraction results for each individual are not stable in binarization processing using a common threshold. [Means for solving the problem]

[0009] An information processing device according to one embodiment of the present invention is An information processing device that detects and tracks any number of objects in the foreground of an image as separate individuals by binarization, A storage means for storing tracking information for each individual, which includes the threshold used in the binarization process and information about the detected individual, A threshold setting means that sets a predetermined number of thresholds based on the thresholds included in the tracking information, A binarization means that generates a predetermined number of binarized images by binarizing the image using the predetermined number of threshold values ​​set above, A matching means for detecting objects from the generated binarized image that are consistent with the individual information included in the tracking information, Updating means for updating the tracking information with the threshold value used for generating the binary image in which the detection of the object has succeeded and the information of the object that matches the threshold value; is configured to include.

[0010] An information processing method according to another aspect of the present invention is An information processing method for detecting and tracking any number of objects that are foregrounds in an image as separate individuals by binarization processing, tracking information including the threshold value used in the binarization processing and the information of the detected individual is stored for each individual, setting a predetermined number of threshold values based on the threshold values included in the tracking information, binarizing the image using the set predetermined number of threshold values to generate a predetermined number of binary images, detecting an object that matches the information of the individual included in the tracking information from the generated binary image, updating the tracking information with the threshold value used for generating the binary image in which the detection of the object has succeeded and the information of the object that matches the threshold value, is configured as described above.

[0011] A computer-readable recording medium according to another aspect of the present invention is for a computer that detects and tracks any number of objects that are foregrounds in an image as separate individuals by binarization processing, a process of storing, for each individual, tracking information including the threshold value used in the binarization process and the information of the detected individual; a process of setting a predetermined number of threshold values based on the threshold values included in the tracking information; a process of binarizing the image using the set predetermined number of threshold values to generate a predetermined number of binary images; a process of detecting an object that matches the information of the individual included in the tracking information from the generated binary image; a process of updating the tracking information with the threshold value used for generating the binary image in which the detection of the object has succeeded and the information of the object that matches the threshold value; configured to record a program for causing the

Effect of the Invention

[0012] By having the configuration as described above, the present invention can stably extract the individual to be tracked.

Brief Description of the Drawings

[0013] [Figure 1] It is a block diagram of an inspection system to which the information processing apparatus according to the first embodiment of the present invention is applied. [Figure 2] It is a block diagram showing an example of the information processing apparatus according to the first embodiment of the present invention. [Figure 3] It is a diagram showing a configuration example of image information input to the information processing apparatus according to the first embodiment of the present invention. [Figure 4] It is a diagram showing a configuration example of a tracking information group generated by the information processing apparatus according to the first embodiment of the present invention. [Figure 5] It is a diagram showing a configuration example of an inspection result generated by the information processing apparatus according to the first embodiment of the present invention. [Figure 6] It is a flowchart showing an example of the operation of an inspection system to which the information processing apparatus according to the first embodiment of the present invention is applied. [Figure 7] It is a block diagram showing an example of a tracking unit in the information processing apparatus according to the first embodiment of the present invention. [Figure 8] It is a schematic diagram showing an example of an image area predicted by a prediction unit in the information processing apparatus according to the first embodiment of the present invention. [Figure 9] It is a flowchart showing an example of the processing of a tracking unit in the information processing apparatus according to the first embodiment of the present invention. [Figure 10] It is a block diagram of an information processing apparatus in the second embodiment of the present invention.

Mode for Carrying Out the Invention

[0014] [First Embodiment] First, to facilitate understanding of the first embodiment of the present invention, the problems assumed by the first embodiment will be described.

[0015] One method for inspecting for foreign matter in liquids sealed in containers involves shaking the container, then continuously photographing the liquid flowing inside with a camera to obtain multiple images. These images are then binarized to detect and track foreground floating objects, and the method determines whether the floating objects are bubbles or foreign matter based on the characteristics of their movement trajectories (see, for example, Patent Document 4). However, lighting, background, and appearance are not necessarily the same for all floating objects. Therefore, binarization using a common threshold for all floating objects may result in unstable extraction results. When the extraction results of floating objects by binarization are unstable, tracking failures or changes in the tracked individual may occur, making accurate foreign matter inspection difficult. This embodiment aims to provide an inspection system that solves the above-mentioned problem, namely the problem of unstable extraction results of floating objects by binarization.

[0016] Next, the configuration and operation of the first embodiment of the present invention will be described in detail with reference to the drawings.

[0017] Figure 1 is a block diagram of an inspection system 1 to which an information processing device according to the first embodiment of the present invention is applied. Referring to Figure 1, the inspection system 1 is a system for inspecting whether or not there are foreign substances in a liquid sealed in a container 2. The inspection system 1 mainly comprises a flow induction device 3, a lighting device 4, a camera device 5, and an information processing device 6.

[0018] Container 2 is a transparent or translucent container such as a glass bottle or a plastic bottle. Container 2 contains a liquid such as medicine or water. Furthermore, the liquid contained in container 2 may contain foreign matter. Examples of such foreign matter include glass fragments, plastic fragments, rubber fragments, hair, fiber fragments, and soot.

[0019] The fluid induction device 3 is configured to grip the container 2 in a predetermined position. The predetermined position is arbitrary. For example, the position in which the container 2 is upright may be considered the predetermined position. The fluid induction device 3 is configured to tilt, swing, or rotate the container 2 in a predetermined direction from its upright position while gripping it. The fluid induction device 3 is also connected to the information processing device 6 by wire or wireless. When activated by an instruction from the information processing device 6, the fluid induction device 3, while gripping the container 2, tilts, swings, or rotates the container 2 in a predetermined direction from its upright position. When stopped by an instruction from the information processing device 6, the fluid induction device 3 stops tilting, swinging, and rotating the container 2 and returns to gripping the container 2 in an upright position.

[0020] As described above, when container 2 is tilted, oscillated, and rotated, and then brought to a standstill, a state is obtained in which the liquid flows within the stationary container 2 due to inertia. When the liquid flows, a state is obtained in which foreign matter mixed into the liquid becomes suspended. In addition, when the liquid flows, air bubbles that were attached to the inner wall of container 2 or that were mixed in during the flow of the liquid may become suspended in the liquid. Therefore, the information processing device 6 needs to distinguish whether the suspended matter is foreign matter or air bubbles.

[0021] The illumination device 4 is configured to irradiate the liquid sealed in the container 2 with illumination light. The illumination device 4 is, for example, a spot light source large enough to illuminate the entire liquid in the container 2 being inspected. The illumination device 4 is installed on the same side as the camera device 5, or on the opposite side, relative to the container 2. That is, the illumination by the illumination device 4 is transmitted illumination or reflected illumination.

[0022] The camera device 5 is an imaging device that continuously captures images of the liquid in the container 2 at a predetermined frame rate. The camera device 5 may consist of a color camera or a monochrome camera equipped with a CCD (Charge-Coupled Device) image sensor or a CMOS (Complementary MOS) image sensor having a pixel capacity of several million pixels, for example. The camera device 5 is connected to the information processing device 6 by wire or wireless connection. The camera device 5 is configured to transmit the time-series images obtained by the camera, along with information indicating the time of capture, to the information processing device 6.

[0023] The information processing device 6 is configured to perform image processing on time-series images captured by the camera device 5 to inspect for the presence or absence of foreign matter in the liquid sealed in the container 2. The information processing device 6 is connected to the flow induction device 3, the lighting device 4, and the camera device 5 by wire or wireless connection.

[0024] Figure 2 is a block diagram showing an example of an information processing device 6. Referring to Figure 2, the information processing device 6 includes a communication I / F unit 61, an operation input unit 62, a screen display unit 63, a storage unit 64, and an arithmetic processing unit 65.

[0025] The communication interface unit 61 consists of a data communication circuit and is configured to communicate data with the flow induction device 3, lighting device 4, camera device 5, and other external devices (not shown) via wired or wireless connection. The operation input unit 62 consists of an operation input device such as a keyboard or mouse and is configured to detect operator operations and output them to the calculation processing unit 65. The screen display unit 63 consists of a display device such as an LCD (Liquid Crystal Display) and is configured to display the inspection results of the container 2, etc., in response to instructions from the calculation processing unit 65.

[0026] The storage unit 64 consists of one or more storage devices of one or more types, such as a hard disk or memory, and is configured to store processing information and programs 641 necessary for various processes in the arithmetic processing unit 65. The programs 641 are programs that realize various processing processes when read and executed by the arithmetic processing unit 65, and are pre-read from external devices or recording media (not shown) via data input / output functions such as the communication I / F unit 61 and stored in the storage unit 64. The main processing information stored in the storage unit 64 includes image information 642, tracking information group 643, and inspection results 644.

[0027] Image information 642 includes a series of images obtained by continuously photographing the liquid in container 2 with the camera device 5. If suspended particles are present in the liquid in container 2, the image information 642 will show images of the suspended particles.

[0028] Figure 3 shows an example of the configuration of image information 642. In this example, image information 642 consists of entries comprising a container ID 6421, a shooting time 6422, and a frame image 6423. The container ID 6421 field is set to an ID that uniquely identifies container 2. Possible container IDs for container 2 include a serial number assigned to container 2, a barcode attached to container 2, or object fingerprint information taken from the cap of container 2. The shooting time 6422 field is set to the time when container 2 was photographed by the camera device 5. The shooting time 6422 may be, for example, the elapsed time from the point when the tilting, shaking, and rotation of container 2 stopped. The frame image 6423 field is set to the frame image of container 2 photographed by the camera device 5 at the shooting time 6422. In this embodiment, the frame image is a grayscale image with 256 gradations from 0 (black) to 255 (white). However, the frame image is not limited to a grayscale image with the above gradations. Furthermore, the frame images may be RGB color images. In the example in Figure 3, a container ID 6421 is associated with each frame image 6423, but a container ID 6421 may also be associated with each group of multiple frame images 6423.

[0029] The tracking information group 643 contains tracking information for each floating object present in the liquid inside container 2 as shown in the image information 642. Figure 4 shows an example of the configuration of the tracking information group 643. In this example, the tracking information group 643 consists of entries for container ID 6431, and a pair of tracking ID 6432 and pointer 6433. The entry for container ID 6431 is set with an ID that uniquely identifies container 2. An entry consisting of a pair of tracking ID 6432 and pointer 6433 is provided for each floating object to be tracked. The item for tracking ID 6432 is set with an ID to distinguish the floating object to be tracked from other floating objects in the same container 2. The item for pointer 6433 is set with a pointer to the tracking information 6434 of the floating object to be tracked.

[0030] The tracking information 6434 for the floating object identified by tracking ID 6432 consists of one or more entries, each entry consisting of a set of time 64341, threshold 64342, location 64343, region 64344, and image features 64345. Each entry in the tracking information 6434 corresponds one-to-one with the frame image 6423. The time 64341 field is set to the time 6422 when the corresponding frame image was captured. The threshold 64342 field is set to the threshold at which detection of the floating object identified by tracking ID 6432 was successful. The location 64343, region 64344, and image features 64345 fields are set to the coordinate values ​​indicating the center position of the floating object with tracking ID 6432, data indicating the region of the floating object, and data indicating the image features of the floating object, which are detected from the binarized image obtained by binarizing the frame image 6423 at time 64341 using the threshold 64342. The coordinate values ​​indicating the center position of the floating object may, for example, be coordinate values ​​in a predetermined coordinate system. The predetermined coordinate system may be a camera coordinate system centered on the camera, or a world coordinate system centered on a certain position in space. The region 64344 of the floating object may be data indicating, for example, the height, width, and length of the floating object. The image features 64345 may be data indicating the brightness distribution of the floating object, average brightness, median brightness, minimum brightness, and maximum brightness. Position 64343, region 64344, and image features 64345 are examples of information about an individual floating object. However, the information about an individual floating object is not limited to the above. In addition to the above, or instead of the above, the information about an individual floating object may include the size, shape, movement speed, acceleration, etc.

[0031] The multiple entries in tracking information 6434 are arranged in order of time 64341. The time 64341 of the first entry is the start time of tracking. The time 64341 of the last entry is the end time of tracking. The times 64341 of entries other than the first and last are intermediate tracking times. The position column, which is the position 64343 of each entry arranged in order of time 64341, represents the movement trajectory of the floating object.

[0032] The inspection result 644 is information regarding the results of an inspection to determine whether or not foreign matter is present in the liquid sealed in container 2, which is the object of inspection. Figure 5 shows an example of the structure of inspection result 644. In this example, inspection result 644 consists of a set of container ID 6441 and inspection result 6442. The entry for container ID 6441 is set to an ID that uniquely identifies container 2, which is the object of inspection. The entry for inspection result 6442 is set to an inspection result of either OK (inspection passed) or NG (inspection failed). OK indicates that no foreign matter was detected in the liquid in the container. NG indicates that foreign matter was detected in the liquid in the container.

[0033] Referring again to Figure 2, the arithmetic processing unit 65 has a processor such as a CPU (Central Processing Unit) and its peripheral circuits, and is configured to realize various processing units by having the above hardware and program 641 cooperate by reading and executing the program 641 from the storage unit 64. The main processing units realized by the arithmetic processing unit 65 are the flow induction control unit 651, the image acquisition unit 652, the tracking unit 653, the determination unit 654, and the output unit 655.

[0034] The flow induction control unit 651 is configured to induce the flow of liquid in the container 2 by transmitting a flow start command to the flow induction device 3 via the communication I / F unit 61. The flow induction control unit 651 is also configured to stop the induction of liquid flow by the flow induction device 3 by transmitting a flow stop command to the flow induction device 3 via the communication I / F unit 61. Even after the induction of liquid flow by the flow induction device 3 is stopped, the liquid in the container 2 will continue to flow for a while due to inertia.

[0035] The image acquisition unit 652 is configured to control the illumination device 4 and the camera device 5 via the communication interface unit 61 to acquire image information 642 that captures images of suspended matter present in the liquid sealed in the container 2. For example, the image acquisition unit 652 sends a power-on command to the illumination device 4 via the communication interface unit 61, and then sends a shooting start command to the camera device 5 via the communication interface unit 61, thereby initiating the process of continuously photographing the liquid flowing in the container 2 under the illumination of the illumination device 4 using the camera device 5 at a predetermined frame rate. The image acquisition unit 652 also generates image information 642 as shown in Figure 3 from the time-series images obtained by the shooting and stores it in the storage unit 64. The image acquisition unit 652 also sends a shooting end command to the camera device 5 and a power-off command to the illumination device 4 via the communication interface unit 61, thereby ending the shooting by the camera device 5 under illumination.

[0036] The tracking unit 653 is configured to read image information 642 generated by the image acquisition unit 652 from the storage unit 64, and to detect and track any number of floating objects in the foreground of the image information 642 by binarizing them as separate individuals. The tracking unit 653 is configured to predict the next image region where a floating object will be located, for each tracking ID of the floating object being tracked, using the floating object information contained in the tracking information 6434. The tracking unit 653 is also configured to set a predetermined number of thresholds for the predicted image region, based on the threshold values ​​64342 contained in the tracking information 6434, for each tracking ID of the floating object being tracked. The tracking unit 653 is also configured to generate a predetermined number of binarized images by binarizing the predicted image region using the set predetermined number of thresholds, for each tracking ID of the floating object being tracked. The tracking unit 653 is also configured to detect floating objects from the generated binarized images that match the individual floating object information contained in the tracking information 6434 of the floating object being tracked, for each tracking ID of the floating object being tracked. Furthermore, the tracking unit 653 is configured to add a new entry to the tracking information 6434 for each tracking ID of a floating object being tracked, using the threshold used to generate the binarized image in which a matching floating object was successfully detected and the individual information of the matching floating object.

[0037] Furthermore, the tracking unit 653 is configured to, for each tracking ID of the floating object being tracked, if it fails to detect a floating object consistent with the floating object being tracked from the binarized image generated by binarization using the predetermined number of thresholds set above, it sets a predetermined number of other thresholds again based on the thresholds included in the tracking information 6434. The tracking unit 653 is also configured to, for each tracking ID of the floating object being tracked, binarize the predicted image region using the predetermined number of other thresholds set again above to generate a predetermined number of other binarized images. The tracking unit 653 is also configured to, for each tracking ID of the floating object being tracked, detect a floating object consistent with the individual floating object information included in the tracking information 6434 from the predetermined number of other binarized images generated again.

[0038] Furthermore, the tracking unit 653 is configured to repeat the following processes for each tracking ID of a floating object being tracked: setting a predetermined number of additional thresholds based on the thresholds included in the tracking information 6434 until the total number of thresholds already set reaches the upper limit or a floating object consistent with the individual floating object information included in the tracking information 6434 is detected; binarizing the predicted image region using those thresholds to generate a binarized image; and detecting a floating object from the generated binarized image that is consistent with the individual floating object information included in the tracking information 6434.

[0039] In this embodiment, the tracking unit 653 has a predetermined number of 1. However, as will be described later, the predetermined number is not limited to 1 and may be 2 or more.

[0040] The determination unit 654 reads the tracking information group 643 generated by the tracking unit 653 from the storage unit 64 and is configured to determine the presence or absence of foreign matter based on the tracking information group 643. For example, for each tracking ID 6432 of a floating object included in the tracking information group 643, the determination unit 654 determines whether the floating object is a bubble or a foreign object based on the characteristics of the movement trajectory of the floating object, which is represented by the time-series change of the position 64343 in the tracking information 6434 identified by the corresponding pointer 6433. The reason why it is possible to determine whether a floating object is a foreign object or a bubble based on the movement trajectory of the floating object is that the characteristics of the movement trajectory of a foreign object in a liquid are different from those of a bubble. That is, bubbles, which have a significantly lower specific gravity than liquid, tend to move in the anti-gravity direction in the liquid. In contrast, foreign objects, which have a higher specific gravity than bubbles, do not tend to move in the anti-gravity direction in the liquid, but rather tend to move in the gravitational direction. Based on these findings, floating objects that trace a trajectory in the anti-gravity direction within a liquid can be identified as bubbles, while floating objects that trace a trajectory in the direction of gravity within a liquid can be identified as foreign objects.

[0041] Furthermore, the determination unit 654 is configured to generate an inspection result 644 based on the determination result and store it in the storage unit 64. For example, when the determination unit 654 determines that at least one floating object is a foreign object, it creates an inspection result 644 consisting of container ID 6441 and an inspection result 6442 of NG and stores it in the storage unit 64. Also, when the determination unit 654 determines that all floating objects are air bubbles, it creates an inspection result 644 consisting of container ID 6441 and an inspection result 6442 of OK and stores it in the storage unit 64.

[0042] The output unit 655 is configured to read the test result 644 generated by the determination unit 654 from the storage unit 64, display it on the screen display unit 63, and / or transmit it to an external device (not shown) via the communication I / F unit 61.

[0043] Next, we will explain the overall operation of inspection system 1. Figure 6 is a flowchart showing an example of the operation of inspection system 1.

[0044] Referring to Figure 6, first, the fluid induction control unit 651 induces fluid flow in the liquid inside the container 2 by shaking the container 2 (step S1). Next, the image acquisition unit 652 acquires an image by continuously capturing images of the liquid flowing inside the container 2 for a certain period of time using the camera device 5 under illumination by the lighting device 4, and stores it in the storage unit 64 as image information 642 (step S2). As illustrated in Figure 3, the image information 642 consists of multiple entries, each consisting of a container ID 6421, a shooting time 6422, and a frame image 6423.

[0045] Next, the tracking unit 653 reads image information 642 from the storage unit 64, detects any number of floating objects in the foreground of the multiple frame images constituting the image information 642 as separate individuals by binarization processing and tracks them, thereby generating a group of tracking information 643 and storing it in the storage unit 64 (step S3). As illustrated in Figure 4, the group of tracking information 643 includes tracking information 6434 for each floating object to be tracked. Furthermore, the entries in the tracking information 6434 include a time 64341, a threshold 64342, and information about the floating object, such as a position 64343, a region 64344, and image features 64345.

[0046] Next, the determination unit 654 reads the tracking information group 643 from the storage unit 64 and determines whether each floating object identified by the tracking ID 6432 included in the tracking information group 643 is a foreign object or a bubble based on the movement trajectory, which is the time-series change of the position 64343 of the tracking information 6434 of that floating object (step S4). Next, the determination unit 654 generates an inspection result 644 based on the determination result for each floating object and stores it in the storage unit 64 (step S5). As illustrated in Figure 5, the inspection result 644 consists of a container ID 6441 and an inspection result 6442 representing either OK or NG. Next, the output unit 655 reads the inspection result 644 from the storage unit 64 and displays it on the screen display unit 63, and / or transmits it to an external device (not shown) via the communication I / F unit 61 (step S6).

[0047] Next, we will explain the tracking unit 653 in detail.

[0048] Figure 7 is a block diagram showing an example of the tracking unit 653. Referring to Figure 7, the tracking unit 653 is composed of a prediction unit 6531, a threshold setting unit 6532, a binarization unit 6533, a matching unit 6534, and an update unit 6535.

[0049] The prediction unit 6531 is configured to predict the next image region where a floating object will be located, based on the tracking information 6434 of the tracking information group 643, for each tracking ID of the floating object being tracked. The tracking information group 643 contains tracking information 6434 for each tracking ID 6432 of the floating object to be tracked, and this tracking information 6434 includes the center position 64343 of the floating object being tracked and a region 64344 such as height, width, and length. Based on this information (position and region) of each floating object being tracked, the prediction unit 6531 predicts the next image region where each floating object being tracked will be located. The prediction unit 6531 may predict the next image region where the floating object will be located based only on the floating object's most recent position 64343 and region 64344. Alternatively, the prediction unit 6531 may estimate the direction and speed of movement of the floating object from the time-series changes of multiple positions 64343 and regions 64344 up to the immediate vicinity, and predict the next image region where the floating object will be located based on the estimation result. The prediction unit 6531 may also predict a slightly larger image region, taking variance into consideration. For each floating object being tracked, the prediction unit 6531 sends prediction information to the binarization unit 6533. The prediction information includes the tracking ID of the floating object and coordinate values ​​that identify the predicted image region. For example, if the image region is rectangular, the prediction information may include, for example, the coordinate values ​​of each vertex of the rectangle.

[0050] Figure 8 is a schematic diagram showing examples of image regions 81-83 predicted to exist at time t for each of the three floating objects with tracking IDs 71-73. The image region 81 of the floating object with tracking ID 71 is predicted to move upward in the frame image 6423, based on the position and region information 71(t-2) from two frames ago at time t-2 and the position and region information 71(t-1) from one frame ago at time t-1. The image region 82 of the floating object with tracking ID 72 is predicted to move downward in the frame image 6423, based on the position and region information 72(t-2) from two frames ago at time t-2 and the position and region information 72(t-1) from one frame ago at time t-1. The image region 83 of the floating object with tracking ID 73 is predicted to move to the right in the frame image 6423, based on the position and region information 73(t-2) from two frames ago at time t-2 and the position and region information 73(t-1) from one frame ago at time t-1.

[0051] The threshold setting unit 6532 is configured to set a threshold for detecting floating objects based on the tracking information 6434 for each tracking ID of a floating object being tracked. For example, the threshold setting unit 6532 may read the most recent threshold 64342 used for each floating object being tracked from the tracking information 6434 and initially set a threshold of the same value. Alternatively, the threshold setting unit 6532 may read multiple thresholds 64342 used up to the most recent for the floating object being tracked from the tracking information 6434, calculate their average, minimum, maximum, or median, and initially set a threshold of the same value as the calculation result. The threshold setting unit 6532 sends threshold information to the binarization unit 6533 for each tracking ID of a floating object being tracked. The threshold information includes the tracking ID of the floating object and the threshold value.

[0052] In addition, when the threshold setting unit 6532 receives tracking failure information including a tracking ID from the collation unit 6534, it is configured to set a different threshold for the tracking ID. Each time the threshold setting unit 6532 receives tracking failure information, it is configured to repeat the process of setting a different threshold until the total number of thresholds set for the same tracking ID reaches the upper limit. For the tracking ID for which the total number of set thresholds has reached the upper limit, the threshold setting unit 6532 responds to the collation unit 6534 indicating that the total number of set thresholds has reached the upper limit.

[0053] The method by which the threshold setting unit 6532 sequentially changes the threshold as described above is arbitrary. For example, the threshold setting unit 6532 may sequentially change the threshold according to a certain rule. As a certain rule, for example, the following rules can be considered. <Example of rule> Set the initially used threshold as the most recent threshold 64342 used for the floating matter of the tracking ID. The threshold Th curr to be used subsequently is determined by the following equation from the previously used threshold Th prev . Th curr = Th prev + a ··· (1) Here, a is the quantization step, for example, a = 5. Also, set the upper limit of Th curr as Th max . When reaching the upper limit Th max , the threshold to be used next is a value smaller than the initially used threshold by the quantization step a, and the thresholds to be used thereafter are determined by the following equation from the previously used threshold. Th curr = Th prev - a ··· (2) Here, set the lower limit of Th curr as Th min .

[0054] The value range of the threshold (Th min ~ Th maxThe step size a may be determined from the histogram of image features associated with the tracking information 6434 of the tracking ID, or from changes in background brightness information. In that case, the threshold range and step size may be determined from the maximum value, minimum value, and variance of the histogram.

[0055] According to the above rule, the threshold setting unit 6532 will change the threshold value to gradually increase from the initial value, and once the upper limit is reached, it will change it to gradually decrease from the initial value. For example, if the threshold setting unit 6532 initially uses a threshold value of, say, 50 and the increment a is 5, it will change it sequentially to 50, 55, 60, 65, ... until the upper limit Th is reached. max After changing up to that point, the lower limit Th is then changed in order to 45, 40, 35, ... min This will change the threshold.

[0056] However, conversely, the threshold setting unit 6532 may change the threshold value to gradually decrease from the initial value, and once it reaches the lower limit, change it to gradually increase from the initial value. Alternatively, the threshold setting unit 6532 may change the threshold value alternately in the increasing and decreasing directions. For example, if the threshold setting unit 6532 initially uses a threshold value of, say, 50 and the step size a is 5, it may change the threshold value sequentially to 50, 55, 45, 60, 40, 65, ...

[0057] The binarization unit 6533 is configured to input a frame image 6423 from image information 642, prediction information from prediction unit 6531, and threshold information from threshold setting unit 6532, and for each tracking ID, to binarize the image region of the floating object with the tracking ID in the frame image using the threshold set for that tracking ID. The binarization unit 6533 outputs the binarization information for each tracking ID to the matching unit 6534. The binarization information includes, for example, the tracking ID, the time the frame image was taken, the binarized image of the image region, and the threshold used for binarization. For example, the binarization unit 6533 performs binarization on the predicted image region 81 of the floating object with tracking ID 71 in the frame image 6423 illustrated in Figure 8, using the threshold set for the tracking ID 71 by the threshold setting unit 6532, and outputs the binarization information including the binarized image to the matching unit 6534. Subsequently, if threshold information including the tracking ID is input again from the threshold setting unit 6532, the binarization unit 6533 will use the threshold included in this input threshold information to binarize the image region 81 again, and output the binarized information including the binarized image to the matching unit 6534 again.

[0058] The matching unit 6534 receives binarized information for each tracking ID from the binarization unit 6533 and is configured to detect floating objects from the binarized image contained in the binarized information that match the information of individual floating objects contained in the tracking information 6434 related to the tracking ID. In this embodiment, the information of individual floating objects contained in the tracking information 6434 consists of a region 64344 and image features 64345. As mentioned above, the region 64344 is, for example, the height, width, and length of the floating object, and the image features 64345 are the brightness distribution of the floating object, the average brightness, the median brightness, the minimum brightness, the maximum brightness, etc.

[0059] The matching unit 6534 extracts regions and image features similar to region 64344 and image features 64345 for each floating object detected from the binarized image, and compares them with region 64344 and image features 64345 in the tracking information 6434 of the floating object to determine the floating object that matches the floating object being tracked. For example, for each floating object detected from the binarized image, the matching unit 6534 calculates a score, with a higher value indicating a greater degree of matching. Among the floating objects whose calculated score is equal to or greater than a predetermined lower limit, the floating object with the highest score is determined to be the floating object that matches the floating object being tracked. When the matching unit 6534 determines a floating object that matches the floating object being tracked, it outputs tracking success information to the update unit 6535, including the tracking ID of the determined floating object being tracked, the time the frame image was captured, the threshold used for binarization, the location, region, and image features of the successfully tracked floating object.

[0060] On the other hand, if the matching unit 6534 finds no floating objects with a score equal to or greater than the lower limit in the binarized image for each tracking ID, it determines that the floating object for the tracking ID could not be detected. In this case, the matching unit 6534 outputs tracking failure information, including the tracking ID, to the threshold setting unit 6532. The threshold setting unit 6532 sets a different threshold if the total number of thresholds has not reached the upper limit, but if the upper limit has been reached, it responds to the matching unit 6534 accordingly. The matching unit 6534 terminates tracking for floating objects that it has received a response indicating that the total number of thresholds has reached the upper limit. However, it may update only the prediction results as the tracking results a certain number of times. For example, the matching unit 6534 may, for floating objects whose tracking has been completed, predict the position of the floating object at the time the binarized frame image was taken, based on the position 64343 in the most recent entry of the tracking information 6434 for subsequent frame images, and send tracking success information, including the predicted position and time of capture, to the update unit 6535.

[0061] The update unit 6535 receives tracking success information from the matching unit 6534 for each tracking ID, and is configured to update the tracking information 6434 according to this tracking success information. The tracking success information includes the tracking ID of the floating object, the time the frame image was captured, the threshold used for binarization, the location, region, and image features of the floating object that was successfully tracked. The update unit 6535 reserves a new entry in the tracking information 6434 identified by the ID of the floating object that was successfully tracked, and sets the time 64341, threshold 64342, location 64343, region 64344, and image features 64345 of the reserved entry to the time 64341, threshold 64342, location 64343, region 64344, and image features 64345, respectively, which are the time the frame image was captured, the threshold used for binarization, the location, region, and image features of the floating object that was successfully tracked, as contained in the tracking success information.

[0062] Figure 9 is a flowchart showing an example of the processing performed by the tracking unit 653. An example of the operation of the tracking unit 653 will be described below with reference to Figure 9.

[0063] Currently, the tracking information group 643 is assumed to have tracking information 6434 for each floating object tracked up to the frame image of the tracking start point, which is pre-generated by the tracking unit 653 using any method. Figure 9 shows an example of the processing performed by the tracking unit 653 on frame images after the tracking start point. As an example of the above arbitrary method, for example, multiple binarized images are generated from each of the frame images of the tracking start point using multiple thresholds, a tracking ID 6432 is assigned to all floating objects comprehensively detected from these multiple binarized images, and tracking information 6434 is generated consisting of one entry in which the time of capture of the frame image of the tracking start point, the threshold for successful detection, the position, region, and image features of the floating object are set in the items of time 64341, threshold 64342, position 64343, region 64344, and image features 64345, but is not limited to this.

[0064] First, the tracking unit 653 focuses on the timestamp of the frame image at the tracking start point (step S10). Next, the tracking unit 653 generates a set of tracking IDs that are active at the timestamp being focused on (step S11). A tracking ID that is active at the timestamp being focused on is a tracking ID 6432 that has tracking information 6434 in which the timestamp being focused on is set in the item at time 64341. Next, the tracking unit 653 focuses on the next frame image (step S12). At the start, the frame image immediately following the frame image at the tracking start point becomes the next frame image. If there is no next frame image (YES in step S13), the tracking unit 653 terminates the process shown in Figure 9.

[0065] If there is a next frame image (NO in step S13), the tracking unit 653 focuses on one of the living tracking IDs (step S14). Next, the tracking unit 653 uses the prediction unit 6531 to predict an image region for the tracking ID under focus (step S15). Next, the tracking unit 653 uses the threshold setting unit 6532 to set a threshold for the tracking ID under focus (step S16). Next, the tracking unit 653 uses the binarization unit 6533 to binarize the image region predicted for the tracking ID under focus in the frame image under focus using the threshold set for the tracking ID under focus, and generates a binarized image (step S17). Next, the tracking unit 653 uses the matching unit 6534 to compare the regions and image features of each floating object detected in the generated binarized image with the regions 64344 and image features 64345 of the floating objects recorded in the tracking information 6434 for the tracking ID of interest, and calculates a score representing the degree of matching (step S18). Next, the tracking unit 653 uses the matching unit 6534 to determine whether or not there was a score above a threshold (step S19). Next, if there is a score above a threshold, the tracking unit 653 uses the update unit 6535 to update the tracking information 6434 for the tracking ID of interest with the information of the floating object with the highest score (step S20). Then, the tracking unit 653 proceeds to step S23.

[0066] On the other hand, if there are no scores above the threshold, the tracking unit 653 determines whether the total number of thresholds set for the tracking ID under interest by the threshold setting unit 6532 has reached the upper limit (step S21). Next, if the total number of thresholds set for the tracking ID under interest by the threshold setting unit 6532 has not reached the upper limit, the tracking unit 653 returns to step S16 to change the threshold and continue tracking, and repeats the same process as described above. If the total number of thresholds set for the tracking ID under interest by the threshold setting unit 6532 has reached the upper limit, the tracking unit 653 terminates tracking of the tracking ID under interest (step S22) and proceeds to step S23.

[0067] In step S23, the tracking unit 653 shifts its attention to the next living tracking ID. If there is another living tracking ID (NO in step S24), the tracking unit 653 returns to step S15 and repeats the same process as described above, focusing on that tracking ID. If there is no next living tracking ID (YES in step S24), the tracking unit 653 has finished processing all living tracking IDs at the time of focus and shifts its attention to the time of capture of the frame image at focus (step S25). The tracking unit 653 then repeats the same process as described above, starting from the process in step S11 which generates a set of living tracking IDs at the time of focus. Finally, when the tracking unit 653 has finished processing up to the frame image of the last capture time included in the image information 642 (YES in step S13), it terminates the process shown in Figure 9.

[0068] As described above, the information processing device 6 according to this embodiment records in the tracking information 6434 the threshold value that was effectively used in the binarization process up to the previous frame image for each floating object being tracked, and in the binarization for detecting the floating object from the current frame image, the threshold value determined based on the recorded threshold value is used. Therefore, the information processing device 6 can stably extract the floating objects to be tracked compared to the case where the same threshold value is used for all floating objects.

[0069] Furthermore, the information processing device 6 according to this embodiment is configured to use a different threshold value determined based on the recorded threshold value if it fails to detect a floating object from the binarized image using a set threshold value for each floating object being tracked. Therefore, the information processing device 6 can stably extract the floating objects to be tracked compared to the case where the same multiple threshold values ​​are used for all floating objects.

[0070] Next, a modified example of this embodiment will be described.

[0071] In the above embodiment, the threshold setting unit 6532 of the tracking unit 653 set one threshold value at a time for each tracking ID. However, the threshold setting unit 6532 may set any number M of threshold values ​​at a time for each tracking ID, which may be 2 or more. In this case, in step S16 shown in Figure 9, the threshold setting unit 6532 sets M threshold values ​​that differ from each other. In step S17, the binarization unit 6533 generates M binarized images by binarizing the image region predicted in step S15 using the M threshold values. In step S18, the matching unit 6534 calculates a score indicating the degree of consistency between the information (region and brightness information) of one or more floating objects present in each of the M binarized images and the region and brightness information in the tracking information 6434. In step S19, the matching unit 6534 checks whether there is a score greater than or equal to a threshold value among all the scores calculated in step S18. Furthermore, in step S20, the matching unit 6534 updates the tracking information group with the information of the floating object with the highest score among them.

[0072] The method by which the threshold setting unit 6532 sets M thresholds at once for each tracking ID is arbitrary. For example, it may use a method such as extracting M thresholds at a time from the beginning of a column of thresholds generated sequentially using the method described in the <Example of a Rule> above. The maximum value of M is the total number of thresholds. If M is set to match the total number of thresholds, the threshold setting unit 6532 will set all thresholds at once for each tracking ID at the beginning.

[0073] In the above embodiment, the tracking unit 653 included a prediction unit 6531. However, the prediction unit 6531 may be omitted from the tracking unit 653. In this case, the binarization unit 6533 generates a binarized image for each floating object being tracked by binarizing the entire area of ​​the frame image or the liquid area in the container 2 with a threshold value set by the threshold value setting unit 6532.

[0074] [Second Embodiment] Next, a second embodiment of the present invention will be described with reference to Figure 10. Figure 10 is a block diagram of the information processing device in this embodiment. This embodiment will provide an overview of the information processing device of the present invention.

[0075] Referring to Figure 10, the information processing device 10 in this embodiment is an information processing device that detects and tracks any number of objects in the foreground of an image as separate individuals by binarization processing, and is composed of a storage unit 11, a threshold setting unit 12, a binarization unit 13, a matching unit 14, and an update unit 15.

[0076] The storage unit 11 is configured to store tracking information for each individual, including the threshold used in the binarization process and the information of the detected individual. The threshold setting unit 12 is configured to set a predetermined number of thresholds based on the thresholds included in the tracking information stored in the storage unit 11. The binarization unit 13 is configured to binarize an image using the predetermined number of thresholds set by the threshold setting unit 12 and generate a predetermined number of binarized images. The matching unit 14 is configured to detect objects from the binarized images generated by the binarization unit 13 that are consistent with the information of the individual included in the tracking information. The update unit 15 is configured to update the tracking information with the information of objects that are consistent with the threshold used to generate the binarized images in which the matching unit 14 successfully detected an object.

[0077] The information processing device 10 configured as described above operates as follows: The storage unit 11 stores tracking information for each individual, which includes the threshold used in the binarization process and the information of the detected individual. The threshold setting unit 12 sets a predetermined number of thresholds based on the thresholds included in the tracking information stored in the storage unit 11. Next, the binarization unit 13 binarizes the image using the predetermined number of thresholds set by the threshold setting unit 12 to generate a predetermined number of binarized images. Next, the matching unit 14 detects objects from the binarized images generated by the binarization unit 13 that are consistent with the information of the individual included in the tracking information. Next, the update unit 15 updates the tracking information with the information of the object that is consistent with the threshold used to generate the binarized image in which the matching unit 14 successfully detected the object.

[0078] According to the information processing device 10 configured and operating as described above, the threshold value for successful object detection is recorded in the storage unit 11 for each individual object, and a predetermined number of threshold values ​​are set based on the recorded threshold values ​​for each object being tracked, and binarization is performed. Therefore, compared to the case where the same threshold value is used for all objects, the objects to be tracked can be extracted more stably.

[0079] Although the present invention has been described above with reference to the embodiments described above, the present invention is not limited to the embodiments described above. Various modifications to the configuration and details of the present invention can be made that will be understood by those skilled in the art within the scope of the present invention.

[0080] For example, in the embodiment described above, the present invention was applied to the detection and tracking of suspended matter in a liquid, but the objects to be detected and tracked are not limited to suspended matter in a liquid, but may be any other object. For example, it may be applied to an information processing device that detects and tracks drones flying over a building complex or similar landscape from images captured by a surveillance camera.

[0081] Furthermore, for example, an information processing device may use a GPU (Graphic Processing Unit), DSP (Digital Signal Processor), MPU (Micro Processing Unit), FPU (Floating Number Processing Unit), PPU (Physics Processing Unit), TPU (Tensor Processing Unit), quantum processor, microcontroller, or a combination thereof, instead of the CPU mentioned above. [Industrial applicability]

[0082] This invention can be used in general image processing to detect / track arbitrary objects from an image by utilizing binarization.

[0083] Some or all of the above embodiments may also be described as follows, but are not limited to the following: [Note 1] An information processing device that detects and tracks any number of objects in the foreground of an image as separate individuals by binarization, A storage means for storing tracking information for each individual, which includes the threshold used in the binarization process and information about the detected individual, A threshold setting means that sets a predetermined number of thresholds based on the thresholds included in the tracking information, A binarization means that generates a predetermined number of binarized images by binarizing the image using the predetermined number of threshold values ​​set above, A matching means for detecting objects from the generated binarized image that are consistent with the individual information included in the tracking information, An update means for updating the tracking information with the threshold used to generate a binarized image in which the object was successfully detected and the matching object information, Information processing device including [Note 2] The system further includes a prediction means for predicting the next image region where an individual is located, using information about the individual included in the tracking information. The binarization means binarizes the predicted image region using the set predetermined number of thresholds to generate a predetermined number of binarized images. The information processing device described in Appendix 1. [Note 3] If the matching means fails to detect the object, the threshold setting means sets a predetermined number of other thresholds based on the thresholds included in the tracking information. The binarization means binarizes the image region using a predetermined number of other threshold values ​​to generate a predetermined number of other binarized images. The matching means detects objects from a predetermined number of other binarized images that match the individual information included in the tracking information. The information processing device described in Appendix 2. [Note 4] The threshold setting means repeats the process of setting a predetermined number of other thresholds based on the thresholds included in the tracking information until the total number of set thresholds reaches an upper limit. The information processing device described in Appendix 3. [Note 5] The predetermined number is 1. An information processing device as described in any of the appendices 1 to 4. [Note 6] The aforementioned predetermined number is 2 or more. An information processing device as described in any of the appendices 1 to 4. [Note 7] An information processing method for detecting and tracking any number of objects in the foreground of an image as separate individuals by binarization, Tracking information, including the threshold used in the binarization process and information about the detected individual, is stored for each individual. A predetermined number of thresholds are set based on the thresholds included in the tracking information. The image is binarized using the predetermined number of threshold values ​​set above to generate a predetermined number of binarized images. From the generated binarized image, objects consistent with the individual information included in the tracking information are detected. The tracking information is updated using the threshold used to generate the binarized image in which the object was successfully detected, and the matching object information. Information processing methods. [Note 8] Furthermore, using the individual information included in the tracking information, the next image region in which the individual is located is predicted. In the generation of the binarized image, the predicted image region is binarized using the set predetermined number of thresholds to generate a predetermined number of binarized images. The information processing method described in Appendix 7. [Note 9] moreover, If the detection of the object fails, a predetermined number of other thresholds are set based on the thresholds included in the tracking information. The image region is binarized using the predetermined number of other thresholds set above to generate a predetermined number of other binarized images. From the predetermined number of other binarized images generated, objects consistent with the individual information included in the tracking information are detected. The information processing method described in Appendix 8. [Note 10] Furthermore, the process of setting a predetermined number of other thresholds based on the thresholds included in the tracking information is repeated until the total number of set thresholds reaches the upper limit. The information processing method described in Appendix 9. [Note 11] The predetermined number is 1. The information processing method described in any of the appendices 7 to 10. [Note 12] The aforementioned predetermined number is 2 or more. The information processing method described in any of the appendices 7 to 10. [Note 13] A computer that detects and tracks any number of objects in the foreground of an image as separate entities through a binarization process, A process to store tracking information for each individual, including the threshold used in the binarization process and information about the detected individual, A process of setting a predetermined number of thresholds based on the thresholds included in the aforementioned tracking information, A process to generate a predetermined number of binarized images by binarizing the image using the predetermined number of threshold values ​​set above, The process of detecting objects from the generated binarized image that are consistent with the individual information included in the tracking information, A process to update the tracking information using the threshold used to generate a binarized image in which the object was successfully detected and the matching object information, A computer-readable recording medium containing a program for performing a certain action. [Explanation of Symbols]

[0084] 1. Inspection System 2 containers 3. Flow induction device 4. Lighting equipment 5 Camera equipment 6. Information Processing Device 10 Information Processing Devices 11 Storage section 12. Threshold setting section 13 Binarization section 14. Verification section 15 Update section 61 Communication I / F Section 62 Operation Input Section 63 Screen display section 64 Storage section 65. Processing Unit

Claims

1. An information processing device that detects and tracks any number of objects in the foreground of an image as separate individuals by binarization, A storage means for storing tracking information for each individual, which includes the threshold used in the binarization process and information about the detected individual, A threshold setting means that sets a predetermined number of thresholds based on the thresholds included in the tracking information, A binarization means that generates a predetermined number of binarized images by binarizing the image using the predetermined number of threshold values ​​set above, A matching means for detecting objects from the generated binarized image that are consistent with the individual information included in the tracking information, An update means for updating the tracking information with the threshold used to generate a binarized image in which the object was successfully detected and the matching object information, Information processing device including

2. The system further includes a prediction means for predicting the next image region where an individual is located, using information about the individual included in the tracking information. The binarization means binarizes the predicted image region using the set predetermined number of thresholds to generate a predetermined number of binarized images. The information processing apparatus according to claim 1.

3. If the matching means fails to detect the object, the threshold setting means sets a predetermined number of other thresholds based on the thresholds included in the tracking information. The binarization means binarizes the predicted image region using a predetermined number of other threshold values ​​to generate a predetermined number of other binarized images. The matching means detects objects from a predetermined number of other binarized images that match the individual information included in the tracking information. The information processing apparatus according to claim 2.

4. The threshold setting means repeats the process of setting a predetermined number of other thresholds based on the thresholds included in the tracking information until the total number of set thresholds reaches an upper limit. The information processing apparatus according to claim 3.

5. The predetermined number is 1. The information processing apparatus according to claim 1 or 2.

6. The predetermined number is 2 or more. The information processing apparatus according to claim 1 or 2.

7. An information processing method for detecting and tracking any number of objects in the foreground of an image as separate individuals by binarization, Tracking information, including the threshold used in the binarization process and information about the detected individual, is stored for each individual. A predetermined number of thresholds are set based on the thresholds included in the tracking information. The image is binarized using the predetermined number of threshold values ​​set above to generate a predetermined number of binarized images. From the generated binarized image, objects consistent with the individual information included in the tracking information are detected. The tracking information is updated using the threshold used to generate the binarized image in which the object was successfully detected, and the matching object information. Information processing methods.

8. Furthermore, using the individual information included in the tracking information, the next image region in which the individual is located is predicted. In the generation of the binarized image, the predicted image region is binarized using the set predetermined number of thresholds to generate a predetermined number of binarized images. The information processing method according to claim 7.

9. moreover, If the detection of the object fails, a predetermined number of other thresholds are set based on the thresholds included in the tracking information. The predicted image region is binarized using the predetermined number of other thresholds set above to generate a predetermined number of other binarized images. From the predetermined number of other binarized images generated, objects that are consistent with the individual information included in the tracking information are detected. The information processing method according to claim 8.

10. A computer that detects and tracks any number of objects in the foreground of an image by binarizing them, treating each as a separate individual. A process to store tracking information for each individual, including the threshold used in the binarization process and information about the detected individual, A process of setting a predetermined number of thresholds based on the thresholds included in the aforementioned tracking information, A process to generate a predetermined number of binarized images by binarizing the image using the predetermined number of threshold values ​​set above, The process of detecting objects from the generated binarized image that are consistent with the individual information included in the tracking information, A process to update the tracking information using the threshold used to generate a binarized image in which the object was successfully detected and the matching object information, A program to perform that action.

Images