Information processing device

Abstract

This information processing device for detecting and tracking an arbitrary number of foreground objects in an image as separate individuals by a binarization process comprises: a storage means for storing, for each individual, tracking information that includes a threshold value which has been used in the binarization process and information pertaining to a detected individual; a threshold value setting means for setting a prescribed number of threshold values on the basis of the threshold value that is included in the tracking information; a binarization means for using the set prescribed number of threshold values to binarize the image so that a prescribed number of binarized images are generated; a collation means for detecting, from the generated binarized images, an object that matches the information pertaining to the individual which is included in the tracking information; and an update means for updating the tracking information with information pertaining to the matching object and the threshold value that has been used to generate the binarized image of the object which has been successfully detected.

Description

Information processing device

[0001] The present invention relates to an information processing device, an information processing method, and a recording medium.

[0002] 2. Description of the Related Art Objects captured in images obtained by an imaging device are detected and tracked by binarization processing.

[0003] For example, Patent Document 1 describes a method for detecting and tracking an arbitrary number of cells from frame images obtained by capturing images of cells contained in a predetermined area at regular time intervals, using a binarization process. It also describes that, in the binarization of the frame images, multiple thresholds are set based on the range of brightness values ​​of the frame images.

[0004] Patent Document 2 also describes a method for detecting and tracking various objects from frame images of the sea taken at regular time intervals using a binarization process, and also describes setting a threshold value used in the binarization process based on a difference image between a short-term background image and a long-term background image.

[0005] Patent document 3 also describes identifying an area in a captured image where a detection target is likely to exist as a predicted area, and setting a threshold value used in the detection process for the identified predicted area to be smaller than the threshold value used in the detection process for areas other than the predicted area.

[0006] Japanese Patent No. 6090770 JP 2018-152106 A JP 2014-197353 A WO2021 / 214994

[0007] In the method of setting multiple thresholds based on the range of brightness values ​​of frame images as described in Patent Document 1, the method of setting thresholds based on a difference image between a short-term background image and a long-term background image as described in Patent Document 2, and the method of using a threshold used in detection processing for a predicted region that is smaller than the threshold used in detection processing for regions other than the predicted region as described in Patent Document 3, a common threshold is set for all tracked individuals. However, lighting, background, and appearance may not necessarily be the same for all tracked individuals. Therefore, binarization processing using a common threshold may result in unstable extraction results for each individual.

[0008] An object of the present invention is to provide an information processing device that solves the above-mentioned problem, that is, the problem that the extraction results for each individual are not stable when a binarization process uses a common threshold value.

[0009] An information processing device according to one embodiment of the present invention is an information processing device that detects and tracks any number of objects that appear in the foreground in an image as separate individuals through a binarization process, and is configured to include: a storage means that stores tracking information for each individual, the tracking information including threshold values ​​used in the binarization process and information about the detected individuals; a threshold setting means that sets a predetermined number of threshold values ​​based on the threshold values ​​included in the tracking information; a binarization means that binarizes the image using the set predetermined number of threshold values ​​to generate a predetermined number of binary images; a matching means that detects objects that match information about the individuals included in the tracking information from the generated binary images; and an update means that updates the tracking information with the threshold values ​​used to generate the binary images that successfully detected the objects and the information about the matching objects.

[0010] An information processing method according to another aspect of the present invention is an information processing method for detecting and tracking any number of objects that appear in the foreground in an image as separate individuals through binarization processing, the information including the threshold value used in the binarization processing and information about the detected individuals is stored for each individual, a predetermined number of threshold values ​​are set based on the threshold value included in the tracking information, the image is binarized using the set predetermined number of threshold values ​​to generate a predetermined number of binary images, objects that match the information about the individuals included in the tracking information are detected from the generated binary images, and the tracking information is updated with the threshold value used to generate the binary image in which the object was successfully detected and the information about the matching object.

[0011] A computer-readable recording medium according to another aspect of the present invention is configured to record a program for causing a computer, which detects and tracks any number of objects that appear in the foreground of an image as separate individuals through a binarization process, to perform the following steps: storing tracking information for each individual, the tracking information including threshold values ​​used in the binarization process and information about the detected individuals; setting a predetermined number of threshold values ​​based on the threshold values ​​included in the tracking information; binarizing the image using the set predetermined number of threshold values ​​to generate a predetermined number of binary images; detecting, from the generated binary images, an object that matches information about the individual included in the tracking information; and updating the tracking information with the threshold values ​​used to generate the binary images in which the object was successfully detected and information about the matching object.

[0012] By having the above-described configuration, the present invention can stably extract individuals to be tracked.

[0013] FIG. 1 is a block diagram of an inspection system to which an information processing device according to a first embodiment of the present invention is applied. FIG. 2 is a block diagram of an example of an information processing device according to the first embodiment of the present invention. FIG. 3 is a diagram showing an example of the configuration of image information input to the information processing device according to the first embodiment of the present invention. FIG. 4 is a diagram showing an example of the configuration of a tracking information group generated by the information processing device according to the first embodiment of the present invention. FIG. 5 is a diagram showing an example of the configuration of an inspection result generated by the information processing device according to the first embodiment of the present invention. FIG. 6 is a flowchart showing an example of the operation of an inspection system to which an information processing device according to the first embodiment of the present invention is applied. FIG. 7 is a block diagram of an example of a tracking unit in the information processing device according to the first embodiment of the present invention. FIG. 8 is a schematic diagram showing an example of an image area predicted by a prediction unit in the information processing device according to the first embodiment of the present invention. FIG. 9 is a flowchart showing an example of processing by the tracking unit in the information processing device according to the first embodiment of the present invention. FIG. 10 is a block diagram of an information processing device according to a second embodiment of the present invention.

[0014] First Embodiment First, to facilitate understanding of the first embodiment of the present invention, a problem that the first embodiment of the present invention assumes will be described.

[0015] One method for inspecting for the presence of foreign matter in a liquid sealed in a container involves shaking the container, then continuously capturing images of the liquid flowing in the container with a camera, using a binarization process to detect and track foreground floating matter from multiple images, and determining whether the floating matter is an air bubble or a foreign matter based on the characteristics of the floating matter's movement trajectory (see, for example, Patent Document 4). However, the lighting, background, and appearance are not necessarily the same for all floating matter. Therefore, binarization using a common threshold for all floating matter can result in unstable floating matter extraction results. If the floating matter extraction results from the binarization process become unstable, tracking failures or the tracking individual may be swapped, making accurate foreign matter inspection difficult. The present embodiment aims to provide an inspection system that solves the above-mentioned problem, i.e., the problem of unstable floating matter extraction results from the binarization process.

[0016] Next, the configuration and operation of the first embodiment of the present invention will be described in detail with reference to the drawings.

[0017] Fig. 1 is a block diagram of an inspection system 1 to which an information processing device according to a first embodiment of the present invention is applied. Referring to Fig. 1, the inspection system 1 is a system that inspects the presence or absence of foreign matter in a liquid sealed in a container 2. The inspection system 1 includes, as main components, a flow inducing device 3, a lighting device 4, a camera device 5, and an information processing device 6.

[0018] Container 2 is a transparent or translucent container such as a glass bottle or a plastic bottle. A liquid such as a medicine or water is sealed or filled inside container 2. There is also a possibility that foreign matter may be mixed into the liquid sealed in container 2. Possible foreign matter includes, for example, glass fragments, plastic fragments, rubber fragments, hair, fiber fragments, soot, and the like.

[0019] The flow-inducing device 3 is configured to hold the container 2 in a predetermined position. The predetermined position may be any position. For example, the position of the container 2 when it is upright may be the predetermined position. The flow-inducing device 3 is configured to tilt, rock, or rotate the container 2 in a predetermined direction from the upright position while holding the container 2. The flow-inducing device 3 is also connected to the information processing device 6 via a wire or wirelessly. When activated by an instruction from the information processing device 6, the flow-inducing device 3 tilts, rocks, or rotates the container 2 in a predetermined direction from the upright position while holding the container 2. When stopped by an instruction from the information processing device 6, the flow-inducing device 3 stops tilting, rocking, and rotating the container 2 and returns to a state in which the container 2 is held in an upright position.

[0020] When the container 2 is tilted, swung, and rotated as described above and then brought to a standstill, the liquid in the stationary container 2 flows due to inertia. When the liquid flows, foreign matter mixed in the liquid becomes suspended. Furthermore, when the liquid flows, there is a possibility that air bubbles adhering to the inner wall surface of the container 2 or air bubbles mixed in during the flow of the liquid may become suspended in the liquid. Therefore, the information processing device 6 needs to distinguish whether the floating matter is a foreign matter or an air bubble.

[0021] The illumination device 4 is configured to irradiate illumination light onto the liquid sealed in the container 2. The illumination device 4 is, for example, a spot light source of a size that can illuminate the entire liquid inside the container 2 to be inspected. The illumination device 4 is installed on the same side as the side on which the camera device 5 is installed or on the opposite side as viewed from the container 2. In other words, the illumination provided by the illumination device 4 is transmitted illumination or reflected illumination.

[0022] The camera device 5 is an imaging device that continuously captures images of the liquid in the container 2 at a predetermined frame rate. The camera device 5 may be configured, for example, as a color camera or a black-and-white camera equipped with a CCD (Charge-Coupled Device) image sensor or a CMOS (Complementary MOS) image sensor having a pixel capacity of several million pixels. The camera device 5 is connected to the information processing device 6 via a wired or wireless connection. The camera device 5 is configured to transmit the captured time-series images to the information processing device 6 together with information indicating the time of capture, etc.

[0023] The information processing device 6 is configured to perform image processing on the time-series images captured by the camera device 5, and to inspect the presence or absence of foreign matter in the liquid sealed in the container 2. The information processing device 6 is connected to the flow inducing device 3, the lighting device 4, and the camera device 5 by wire or wirelessly.

[0024] 2 is a block diagram showing an example of the information processing device 6. Referring to FIG. 2, the information processing device 6 includes a communication I / F unit 61, an operation input unit 62, a screen display unit 63, a storage unit 64, and an arithmetic processing unit 65.

[0025] The communication I / F unit 61 is composed of a data communication circuit and is configured to perform data communication via wire or wirelessly with the flow inducing device 3, the lighting device 4, the camera device 5, and other external devices (not shown). The operation input unit 62 is composed of operation input devices such as a keyboard and a mouse and is configured to detect operator operations and output them to the arithmetic processing unit 65. The screen display unit 63 is composed of a display device such as an LCD (Liquid Crystal Display) and is configured to display the inspection results of the container 2, etc., in response to instructions from the arithmetic processing unit 65.

[0026] The storage unit 64 is composed of one or more storage devices of one or more types, such as a hard disk or memory, and is configured to store processing information and a program 641 required for various processes in the arithmetic processing unit 65. The program 641 is a program that is read into the arithmetic processing unit 65 and executed to realize various processing units, and is read in advance from an external device or recording medium (not shown) via a data input / output function such as the communication I / F unit 61 and stored in the storage unit 64. The main processing information stored in the storage unit 64 includes image information 642, a tracking information group 643, and inspection results 644.

[0027] The image information 642 includes a plurality of time-series images obtained by successively photographing the liquid in the container 2 with the camera device 5. If floating matter is present in the liquid in the container 2, the image information 642 will include an image of the floating matter.

[0028] FIG. 3 shows an example of the configuration of image information 642. In this example, the image information 642 is configured with entries each consisting of a set of a container ID 6421, a photographing time 6422, and a frame image 6423. An ID that uniquely identifies the container 2 is set in the container ID 6421 field. The container ID 6421 may be a serial number assigned to the container 2, a barcode affixed to the container 2, or Fingerprint of the Object information collected from the cap of the container 2, etc. The photographing time 6422 field is set with the time at which the container 2 was photographed by the camera device 5. The photographing time 6422 may be, for example, the elapsed time from the point at which the tilting, shaking, or rotation of the container 2 was stopped. The frame image 6423 field is set with a frame image of the container 2 photographed by the camera device 5 at the photographing time 6422. In this embodiment, the frame image is a grayscale image with 256 gradations ranging from 0 (black) to 255 (white). However, the frame image is not limited to a grayscale image with the above gradations. 3, a container ID 6421 is associated with each frame image 6423, but a container ID 6421 may be associated with each group of multiple frame images 6423.

[0029] The tracking information group 643 includes tracking information for each floating object present in the liquid in the container 2 shown in the image information 642. FIG. 4 shows an example of the configuration of the tracking information group 643. In this example, the tracking information group 643 is made up of the following entries: a container ID 6431, and a pair of a tracking ID 6432 and a pointer 6433. An ID that uniquely identifies the container 2 is set in the entry for container ID 6431. An entry consisting of a pair of a tracking ID 6432 and a pointer 6433 is provided for each floating object to be tracked. An ID that distinguishes the floating object to be tracked from other floating objects in the same container 2 is set in the tracking ID 6432 item. A pointer to tracking information 6434 for the floating object to be tracked is set in the pointer 6433 item.

[0030] The tracking information 6434 of the floatable matter identified by the tracking ID 6432 is made up of one or more entries, each of which is made up of a set of a time 64341, a threshold value 64342, a position 64343, an area 64344, and an image feature 64345. Each entry of the tracking information 6434 has a one-to-one correspondence with a frame image 6423. The time 64341 field contains the shooting time 6422 of the corresponding frame image. The threshold value 64342 field contains a threshold value at which the detection of the floatable matter identified by the tracking ID 6432 was successful. The position 64343, area 64344, and image feature 64345 fields contain coordinate values ​​indicating the center position of the floatable matter of the tracking ID 6432 detected from the binarized image obtained by binarizing the frame image 6423 at time 64341 using the threshold value 64342, data indicating the area of ​​the floatable matter, and data indicating the image feature of the floatable matter. The coordinate values ​​indicating the center position of the floatable object may be, for example, coordinate values ​​in a predetermined coordinate system. Furthermore, the predetermined coordinate system may be a camera coordinate system centered on the camera, or a world coordinate system centered on a certain position in space. The floatable object area 64344 may be data indicating, for example, the height, width, length, etc. of the floatable object. The image feature 64345 may be data indicating the brightness distribution, average brightness, median brightness, minimum value, maximum value, etc. of the floatable object. The position 64343, the area 64344, and the image feature 64345 are examples of information about individual floatable objects. However, the information about individual floatable objects is not limited to the above. The information about individual floatable objects may include, in addition to or instead of, the size, shape, movement speed, acceleration, etc. of the floatable object.

[0031] The multiple entries in the tracking information 6434 are arranged in order of time 64341. The time 64341 of the first entry is the tracking start time. The time 64341 of the last entry is the tracking end time. The times 64341 of entries other than the first and last are tracking intermediate times. A position column in which the positions 64343 of each entry are arranged in order of time 64341 represents the movement trajectory of the floating matter.

[0032] The inspection result 644 is information on the results of inspecting the presence or absence of foreign matter in the liquid sealed in the container 2 to be inspected. FIG. 5 shows an example of the configuration of the inspection result 644. In this example, the inspection result 644 is composed of a pair of a container ID 6441 and an inspection result 6442. An ID that uniquely identifies the container 2 to be inspected is set in the entry for the container ID 6441. An inspection result of either OK (inspection passed) or NG (inspection failed) is set in the entry for the inspection result 6442. OK indicates that no foreign matter was detected in the liquid in the container. NG indicates that a foreign matter was detected in the liquid in the container.

[0033] 2 again, the arithmetic processing unit 65 has a processor such as a CPU (Central Processing Unit) and its peripheral circuits, and is configured to read and execute a program 641 from the storage unit 64, thereby realizing various processing units through cooperation between the above hardware and the program 641. The main processing units realized by the arithmetic processing unit 65 include a flow induction control unit 651, an image acquisition unit 652, a tracking unit 653, a determination unit 654, and an output unit 655.

[0034] The flow induction control unit 651 is configured to send a flow start command to the flow induction device 3 via the communication I / F unit 61, thereby inducing a flow of the liquid in the container 2 by the flow induction device 3. The flow induction control unit 651 is also configured to stop the induction of the liquid flow by the flow induction device 3 by sending a flow stop command to the flow induction device 3 via the communication I / F unit 61. Even if the induction of the liquid flow by the flow induction device 3 is stopped, the liquid in the container 2 will continue to flow for a while thereafter due to inertia.

[0035] The image acquisition unit 652 is configured to control the lighting device 4 and the camera device 5 via the communication I / F unit 61 to acquire image information 642 depicting an image of floating matter present in the liquid sealed in the container 2. For example, the image acquisition unit 652 transmits a turn-on command to the lighting device 4 via the communication I / F unit 61, and then transmits a shooting start command to the camera device 5 via the communication I / F unit 61, thereby starting a process of continuously capturing images of the liquid flowing in the container 2 at a predetermined frame rate by the camera device 5 under the illumination of the lighting device 4. The image acquisition unit 652 also generates image information 642 such as that shown in FIG. 3 from the time-series images obtained by the capture, and stores the image information 642 in the storage unit 64. The image acquisition unit 652 also transmits a shooting end command to the camera device 5 and a light-off command to the lighting device 4 via the communication I / F unit 61, thereby ending the capture by the camera device 5 under the illumination.

[0036] The tracking unit 653 is configured to read out the image information 642 generated by the image acquisition unit 652 from the storage unit 64, and to detect and track any number of floaters in the foreground of the image information 642 as separate individuals through a binarization process. The tracking unit 653 is configured to predict, for each tracking ID of the floater being tracked, an image area where the floater will next be located using information about the floater included in the tracking information 6434. The tracking unit 653 is also configured to set a predetermined number of thresholds for the predicted image area based on the threshold value 64342 included in the tracking information 6434 for each tracking ID of the floater being tracked. The tracking unit 653 is also configured to binarize the predicted image area using the set predetermined number of thresholds for each tracking ID of the floater being tracked to generate a predetermined number of binary images. The tracking unit 653 is also configured to detect, for each tracking ID of the floater being tracked, floaters that match information about the individual floaters included in the tracking information 6434 of the floater being tracked from the generated binary images. In addition, the tracking unit 653 is configured to add a new entry to the tracking information 6434 for each tracking ID of the floating object being tracked, using the threshold value used to generate a binary image that successfully detected a matching floating object and information about the individual floating object.

[0037] Furthermore, the tracking unit 653 is configured, for each tracking ID of the floatable matter being tracked, to, if it fails to detect a floatable matter matching the floatable matter being tracked from the binary image generated by binarization using the set predetermined number of thresholds, set a predetermined number of different thresholds again based on the thresholds included in the tracking information 6434. Furthermore, the tracking unit 653 is configured, for each tracking ID of the floatable matter being tracked, to binarize the predicted image area using the set predetermined number of different thresholds to generate a predetermined number of different binary images again. Furthermore, the tracking unit 653 is configured to, for each tracking ID of the floatable matter being tracked, detect a floatable matter matching information about the individual floatable matter included in the tracking information 6434 from the regenerated predetermined number of different binary images.

[0038] In addition, the tracking unit 653 is configured to repeat the following processes for each tracking ID of the floating object being tracked: setting a predetermined number of other threshold values ​​based on the threshold values ​​included in the tracking information 6434, binarizing the predicted image area using those threshold values ​​to generate a binary image, and detecting floating objects that match the information on the individual floating objects included in the tracking information 6434 from the generated binary image, until the total number of threshold values ​​already set reaches an upper limit or a floating object that matches the information on the individual floating objects included in the tracking information 6434 is detected.

[0039] In this embodiment, the tracking unit 653 sets the predetermined number to 1. However, as will be described later, the predetermined number is not limited to 1, and may be 2 or more.

[0040] The determination unit 654 is configured to read the tracking information group 643 generated by the tracking unit 653 from the storage unit 64 and determine the presence or absence of a foreign object based on the tracking information group 643. For example, for each tracking ID 6432 of a floating object included in the tracking information group 643, the determination unit 654 determines whether the floating object is an air bubble or a foreign object based on the characteristics of the floating object's movement trajectory, which is represented by a time-series change in the position 64343 in the tracking information 6434 identified by the corresponding pointer 6433. Whether a floating object is a foreign object or an air bubble can be determined based on the floating object's movement trajectory because the characteristics of the movement trajectory of a floating object in a liquid differ from the characteristics of the movement trajectory of an air bubble. In other words, air bubbles, which have an overwhelmingly lighter specific gravity than the liquid, tend to move in the antigravity direction in the liquid. In contrast, foreign objects, which have a heavier specific gravity than air bubbles, do not tend to move in the antigravity direction in the liquid, but tend to move in the gravity direction. For this reason, floating objects that trace a path through the liquid in the direction against gravity can be determined to be air bubbles, and floating objects that trace a path through the liquid in the direction of gravity can be determined to be foreign matter.

[0041] Furthermore, determination unit 654 is configured to generate inspection result 644 based on the result of the determination and store it in memory unit 64. For example, when determination unit 654 determines that at least one floating matter is a foreign body, it creates inspection result 644 consisting of container ID 6441 and NG inspection result 6442 and stores it in memory unit 64. When determination unit 654 determines that all floating matters are air bubbles, it creates inspection result 644 consisting of container ID 6441 and OK inspection result 6442 and stores it in memory unit 64.

[0042] The output unit 655 is configured to read the inspection results 644 generated by the judgment unit 654 from the memory unit 64, display them on the screen display unit 63, and / or transmit them to an external device (not shown) via the communication I / F unit 61.

[0043] Next, a description will be given of the overall operation of the inspection system 1. Fig. 6 is a flowchart showing an example of the operation of the inspection system 1.

[0044] 6, first, the flow induction control unit 651 induces a flow of the liquid in the container 2 by, for example, shaking the container 2 (step S1). Next, the image acquisition unit 652 acquires images by continuously capturing images of the liquid flowing in the container 2 over a certain period of time using the camera device 5 under illumination by the lighting device 4, and stores the images in the storage unit 64 as image information 642 (step S2). As illustrated in FIG. 3, the image information 642 is composed of multiple entries, each of which is a set of a container ID 6421, a capture time 6422, and a frame image 6423.

[0045] Next, the tracking unit 653 reads the image information 642 from the storage unit 64, and generates a tracking information group 643 by detecting and tracking any number of floating objects in the foreground in the multiple frame images that make up the image information 642 as separate individuals through a binarization process, and stores the generated tracking information group 643 in the storage unit 64 (step S3). As illustrated in Fig. 4, the tracking information group 643 includes tracking information 6434 for each floating object to be tracked. Each entry of the tracking information 6434 includes a time 64341, a threshold value 64342, and information about the floating object, such as a position 64343, an area 64344, and an image feature 64345.

[0046] Next, the determination unit 654 reads the tracking information group 643 from the storage unit 64 and, for each floating object identified by the tracking ID 6432 included in the tracking information group 643, determines whether the floating object is a foreign object or an air bubble based on the movement trajectory, which is a time-series change in the position 64343 of the tracking information 6434 of the floating object (step S4). Next, the determination unit 654 generates an inspection result 644 based on the determination result for each floating object and stores it in the storage unit 64 (step S5). As illustrated in FIG. 5, the inspection result 644 consists of a container ID 6441 and an inspection result 6442 indicating either OK or NG. Next, the output unit 655 reads the inspection result 644 from the storage unit 64 and displays it on the screen display unit 63 and / or transmits it to an external device (not shown) via the communication I / F unit 61 (step S6).

[0047] Next, the tracking unit 653 will be described in detail.

[0048] Fig. 7 is a block diagram showing an example of the tracking unit 653. Referring to Fig. 7, the tracking unit 653 includes a prediction unit 6531, a threshold setting unit 6532, a binarization unit 6533, a matching unit 6534, and an update unit 6535.

[0049] The prediction unit 6531 is configured to predict, for each tracking ID of a floatable object being tracked, an image area where the floatable object will next be located, based on the tracking information 6434 in the tracking information group 643. The tracking information group 643 contains tracking information 6434 for each tracking ID 6432 of the floatable object to be tracked, and the tracking information 6434 includes a center position 64343 of the floatable object being tracked, and an area 64344 such as height, width, and length. The prediction unit 6531 predicts the image area where each floatable object being tracked will next be located, based on this information (position and area) of each floatable object being tracked. The prediction unit 6531 may predict the image area where the floatable object will next be located, based only on the most recent position 64343 and area 64344 of the floatable object. Alternatively, the prediction unit 6531 may estimate the movement direction and movement speed from time-series changes in multiple positions 64343 and areas 64344 immediately adjacent to the floating object, and predict the image area in which the floating object will next be located based on the estimation result. Furthermore, the prediction unit 6531 may predict a slightly larger image area, taking variance into consideration. The prediction unit 6531 sends prediction information to the binarization unit 6533 for each floating object being tracked. The prediction information includes the tracking ID of the floating object and coordinate values ​​identifying the predicted image area. For example, in the case of a rectangular image area, the prediction information may include, for example, the coordinate values ​​of each vertex of the rectangle.

[0050] 8 is a schematic diagram showing an example of image areas 81-83 predicted to exist at time t for each of three floating objects with tracking IDs 71-73. Image area 81 of floating object with tracking ID 71 is predicted to move upward in frame image 6423 based on position and area information 71(t-2) at time t-2, two frames prior, and position and area information 71(t-1) at time t-1, one frame prior. Image area 82 of floating object with tracking ID 72 is predicted to move downward in frame image 6423 based on position and area information 72(t-2) at time t-2, two frames prior, and position and area information 72(t-1) at time t-1, one frame prior. Image area 83 of floating object with tracking ID 73 is predicted to move rightward in frame image 6423 based on position and area information 73(t-2) at time t-2, two frames prior, and position and area information 73(t-1) at time t-1, one frame prior.

[0051] The threshold setting unit 6532 is configured to set a threshold for detecting floatable objects based on the tracking information 6434 for each tracking ID of the floatable object being tracked. For example, the threshold setting unit 6532 may read the most recent threshold value 64342 used for each floatable object being tracked from the tracking information 6434, and initially set a threshold value equal to that value. Alternatively, the threshold setting unit 6532 may read the most recent multiple threshold values ​​64342 used for the floatable object being tracked from the tracking information 6434, calculate the average, minimum, maximum, or median of the threshold values, and initially set a threshold value equal to the calculated value. The threshold setting unit 6532 sends threshold information to the binarization unit 6533 for each tracking ID of the floatable object being tracked. The threshold information includes the tracking ID of the floatable object and a threshold value.

[0052] Furthermore, the threshold setting unit 6532 is configured to set a different threshold for the tracking ID when it receives tracking failure information including the tracking ID from the matching unit 6534. The threshold setting unit 6532 is configured to repeat the process of setting a different threshold each time it receives tracking failure information, until the total number of thresholds set for the same tracking ID reaches the upper limit. For a tracking ID for which the total number of set thresholds has reached the upper limit, the threshold setting unit 6532 responds to the matching unit 6534 that the total number of set thresholds has reached the upper limit.

[0053] The method by which the threshold value setting unit 6532 sequentially changes the threshold value as described above is arbitrary. For example, the threshold value setting unit 6532 may sequentially change the threshold value in accordance with a certain rule. As the certain rule, for example, the following rule can be considered. <Example of rule> The threshold value to be used first is the most recent threshold value 64342 used for the floating object of the tracking ID. The threshold values ​​Th to be used thereafter are curr is the previously used threshold value Th prev It is determined by the following formula: Th curr =Th prev +a (1) where a is the step size, and may be, for example, a=5. curr The upper limit of Th max The upper limit Th max When the threshold value reaches , the next threshold value to be used is set to a value smaller by the step width a than the first threshold value, and the threshold values ​​to be used thereafter are determined from the previously used threshold value using the following formula: curr =Th prev -a... (2) where Th curr The lower limit of Th min Let's say.

[0054] Threshold value range (Th min ~Th max ) and the interval a may be determined from the histogram of image features linked to the tracking information 6434 of the tracking ID, changes in background brightness information, etc. In this case, the threshold value range and interval a may be determined from the maximum value, minimum value, and variance of the histogram.

[0055] According to the above rule, the threshold value setting unit 6532 gradually increases the threshold value from the initial value, and when the upper limit is reached, gradually decreases the threshold value from the initial value. For example, if the threshold value initially used is 50 and the step size a is 5, the threshold value setting unit 6532 sequentially changes the threshold value to 50, 55, 60, 65, ..., until the upper limit Th is reached. max After that, the lower limit Th is changed to 45, 40, 35, ... min The threshold value will be changed until

[0056] However, the threshold setting unit 6532 may change the threshold so that it gradually decreases from the initial value, and when it reaches the lower limit, so that it gradually increases from the initial value. Alternatively, the threshold setting unit 6532 may change the threshold alternately in an increasing and decreasing direction. For example, if the initial threshold value is 50 and the step size a is 5, the threshold setting unit 6532 may change the threshold value in the following order: 50, 55, 45, 60, 40, 65, ...

[0057] The binarization unit 6533 is configured to input the frame image 6423 from the image information 642, the prediction information from the prediction unit 6531, and the threshold value information from the threshold value setting unit 6532, and to binarize, for each tracking ID, the image area of ​​the floating matter corresponding to that tracking ID in the frame image using the threshold value set for that tracking ID. The binarization unit 6533 outputs the binarization information for each tracking ID to the matching unit 6534. The binarization information includes, for example, the tracking ID, the capture time of the frame image, a binarized image of the image area, and the threshold value used for the binarization. For example, the binarization unit 6533 binarizes the image area 81 predicted for the floating matter corresponding to tracking ID 71 in the frame image 6423 shown in FIG. 8 using the threshold value set for that tracking ID 71 by the threshold value setting unit 6532, and outputs the binarization information including the binarized image to the matching unit 6534. Thereafter, when threshold information including the tracking ID is input again from the threshold setting unit 6532, the binarization unit 6533 will again binarize the image area 81 using the threshold included in this input threshold information, and will again output the binarized information including the binarized image to the matching unit 6534.

[0058] The matching unit 6534 is configured to input the binary information for each tracking ID from the binarization unit 6533, and detect floating objects that match the information on floating objects included in the tracking information 6434 related to the tracking ID from the binary image included in the binary information. In the present embodiment, the information on floating objects included in the tracking information 6434 is an area 64344 and an image feature 64345. As described above, the area 64344 is, for example, the height, width, length, etc. of the floating object, and the image feature 64345 is the brightness distribution, average brightness, median brightness, minimum value, maximum value, etc. of the floating object.

[0059] The matching unit 6534 extracts a region and image feature similar to the region 64344 and image feature 64345 for each floatable object detected from the binarized image, and compares the extracted region and image feature with the region 64344 and image feature 64345 in the tracking information 6434 for the floatable object to determine a floatable object that matches the floatable object being tracked. For example, the matching unit 6534 calculates a score for each floatable object detected from the binarized image, the higher the degree of match, and determines the floatable object with the highest score among the floatable objects whose calculated score is equal to or greater than a predetermined lower limit as the floatable object that matches the floatable object being tracked. When the matching unit 6534 determines a floatable object that matches the floatable object being tracked, it outputs tracking success information to the update unit 6535, the tracking ID of the determined floatable object being tracked, the shooting time of the frame image, the threshold value used for binarization, and the position, region, and image features of the successfully tracked floatable object.

[0060] On the other hand, if the binarized image does not contain any floating matter with a score equal to or greater than the lower limit for each tracking ID, the matching unit 6534 determines that the floating matter for the tracking ID could not be detected. In this case, the matching unit 6534 outputs tracking failure information including the tracking ID to the threshold setting unit 6532. In response to this tracking failure information, the threshold setting unit 6532 sets a different threshold if the total number of thresholds has not reached the upper limit, but if the upper limit has been reached, it responds to that effect to the matching unit 6534. For floating matter that is being tracked and for which the matching unit 6534 receives a response indicating that the total number of thresholds has reached the upper limit, the matching unit 6534 terminates tracking. However, it is possible to update only the prediction result as the tracking result a certain number of times. For example, for a floating object whose tracking has been completed, the matching unit 6534 may predict the position of the floating object at the time of shooting of the frame image to be binarized for the subsequent multiple frame images from the position 64343 in the most recent entry of the tracking information 6434, and send tracking success information including this predicted position, shooting time, etc. to the update unit 6535.

[0061] The update unit 6535 is configured to receive tracking success information for each tracking ID from the comparison unit 6534, and update the tracking information 6434 in accordance with this tracking success information. The tracking success information includes the tracking ID of the floating object, the capture time of the frame image, the threshold value used for binarization, and the position, area, and image features of the floating object that was successfully tracked. The update unit 6535 secures a new entry in the tracking information 6434 identified by the ID of the floating object that was successfully tracked, and sets the capture time of the frame image, the threshold value used for binarization, the position, area, and image features of the floating object that was successfully tracked in the tracking success information in the time 64341, threshold value 64342, position 64343, area 64344, and image feature 64345 items of the secured entry.

[0062] 9 is a flowchart showing an example of the processing of the tracking unit 653. An example of the operation of the tracking unit 653 will be described below with reference to FIG.

[0063] It is assumed that the tracking information group 643 includes tracking information 6434 for each floating object tracked up to the frame image at the tracking start point, which has been generated in advance by the tracking unit 653 using an arbitrary method. Figure 9 shows an example of the processing performed by the tracking unit 653 on frame images after the tracking start point. An example of the arbitrary method is, for example, a method in which multiple binary images are generated from each frame image at the tracking start point using multiple thresholds, and all floating objects comprehensively detected from these multiple binary images are assigned tracking IDs 6432, and tracking information 6434 is generated consisting of a single entry in which the shooting time of the frame image at the tracking start point, the threshold at which detection was successful, the position, area, and image features of the floating object are set in the fields of time 64341, threshold value 64342, position 64343, area 64344, and image feature 64345. However, this is not limiting.

[0064] First, the tracking unit 653 focuses on the shooting time of the frame image at the tracking start point (step S10). Next, the tracking unit 653 generates a set of tracking IDs that are alive at the current shooting time (step S11). A tracking ID that is alive at the current shooting time is a tracking ID 6432 that has tracking information 6434 in which the current shooting time is set in the time 64341 field. Next, the tracking unit 653 focuses on the next frame image (step S12). At the start, the frame image immediately after the frame image at the tracking start point becomes the next frame image. If there is no next frame image (YES in step S13), the tracking unit 653 ends the processing of FIG. 9.

[0065] If there is a next frame image (NO in step S13), the tracking unit 653 focuses on one of the active tracking IDs (step S14). Next, the tracking unit 653 predicts an image area for the currently focused tracking ID using the prediction unit 6531 (step S15). Next, the tracking unit 653 sets a threshold for the currently focused tracking ID using the threshold setting unit 6532 (step S16). Next, the tracking unit 653 binarizes the image area predicted for the currently focused tracking ID in the currently focused frame image using the threshold set for the currently focused tracking ID, generating a binarized image (step S17). Next, the tracking unit 653 compares the area and image feature of each floating object detected in the generated binarized image with the floating object area 64344 and image feature 64345 recorded in the tracking information 6434 for the currently focused tracking ID, and calculates a score representing the degree of match (step S18). Next, the tracking unit 653 determines whether or not there is a score equal to or greater than the threshold value using the matching unit 6534 (step S19). Next, if there is a score equal to or greater than the threshold value, the tracking unit 653 updates the tracking information 6434 of the tracking ID under focus with information on the floating object with the maximum score using the updating unit 6535 (step S20). Then, the tracking unit 653 proceeds to step S23.

[0066] On the other hand, if there is no score equal to or greater than the threshold value, the tracking unit 653 determines whether the total number of threshold values ​​set for the currently-watched tracking ID by the threshold value setting unit 6532 has reached an upper limit (step S21). Next, if the total number of threshold values ​​set for the currently-watched tracking ID by the threshold value setting unit 6532 has not reached the upper limit, the tracking unit 653 returns to step S16 to change the threshold value and continue tracking, and repeats the same processing as described above. Also, if the total number of threshold values ​​set for the currently-watched tracking ID by the threshold value setting unit 6532 has reached the upper limit, the tracking unit 653 ends tracking of the currently-watched tracking ID (step S22) and proceeds to step S23.

[0067] In step S23, the tracking unit 653 shifts its attention to the next active tracking ID. If there is a next active tracking ID (NO in step S24), the tracking unit 653 returns to step S15, focuses on the active tracking ID, and repeats the same processing as described above. If there is no next active tracking ID (YES in step S24), the tracking unit 653 shifts its attention to the shooting time of the currently active frame image (step S25), since it has completed processing for all active tracking IDs at the currently active shooting time. The tracking unit 653 then repeats the same processing as described above, starting with step S11, which generates a set of active tracking IDs at the currently active shooting time. When the tracking unit 653 has finished processing up to the frame image with the last shooting time included in the image information 642 (YES in step S13), it ends the processing of FIG. 9 .

[0068] As described above, the information processing device 6 according to this embodiment is configured to record, for each floating object being tracked, the threshold value that has functioned effectively in the binarization process up to the previous frame image in the tracking information 6434, and to use a threshold value determined based on the recorded threshold value in the binarization for detecting the floating object from the current frame image. Therefore, the information processing device 6 can stably extract the floating object to be tracked, compared to when the same threshold value is used for all floating objects.

[0069] Furthermore, the information processing device 6 according to the present embodiment is configured to use another threshold value determined based on the recorded threshold value when it fails to detect a floating object from a binarized image by binarization using a set threshold value for each floating object being tracked. Therefore, the information processing device 6 can stably extract the floating object to be tracked compared to when the same multiple threshold values ​​are used for all floating objects.

[0070] Next, a modification of this embodiment will be described.

[0071] In the above embodiment, the threshold setting unit 6532 of the tracking unit 653 sets one threshold value for each tracking ID at a time. However, the threshold setting unit 6532 may set any number M of threshold values ​​for each tracking ID at a time, which may be two or more. In this case, in step S16 shown in FIG. 9 , the threshold setting unit 6532 sets M threshold values ​​that are different from each other. In step S17, the binarization unit 6533 generates M binary images by binarizing the image area predicted in step S15 using the M threshold values. In step S18, the matching unit 6534 calculates a score indicating the degree of consistency between information (area and brightness information) about one or more floating objects present in each of the M binary images and the area and brightness information in the tracking information 6434. In step S19, the matching unit 6534 checks whether any of the scores calculated in step S18 is equal to or greater than the threshold value. Also, in step S20, the matching unit 6534 updates the tracking information group with the information on the floating object with the highest score among them.

[0072] The threshold setting unit 6532 may use any method to set M thresholds at once for each tracking ID. For example, a method may be used in which all thresholds are generated in order using the method described above in <Example of Rule>, and M thresholds are extracted at a time from the beginning of the sequence of thresholds. The maximum value of M is the total number of thresholds. If M is set to the same as the total number of thresholds, the threshold setting unit 6532 will initially set all thresholds at once for each tracking ID.

[0073] In the above embodiment, the tracking unit 653 includes the prediction unit 6531. However, the prediction unit 6531 may be omitted from the tracking unit 653. In this case, the binarization unit 6533 generates a binarized image by binarizing the entire region of the frame image or the liquid region in the container 2 using the threshold value set by the threshold setting unit 6532 for each floating matter being tracked.

[0074] Second Embodiment Next, a second embodiment of the present invention will be described with reference to Fig. 10. Fig. 10 is a block diagram of an information processing device in this embodiment. Note that this embodiment will describe an outline of the information processing device of the present invention.

[0075] Referring to FIG. 10 , an information processing device 10 in this embodiment is an information processing device that detects and tracks any number of objects in the foreground of an image as separate individuals through binarization processing, and is configured to include a memory unit 11, a threshold setting unit 12, a binarization unit 13, a matching unit 14, and an update unit 15.

[0076] The memory unit 11 is configured to store tracking information for each individual, including threshold values ​​used in the binarization process and information about the detected individual. The threshold setting unit 12 is configured to set a predetermined number of threshold values ​​based on the threshold values ​​included in the tracking information stored in the memory unit 11. The binarization unit 13 is configured to binarize an image using the predetermined number of threshold values ​​set by the threshold setting unit 12 to generate a predetermined number of binary images. The matching unit 14 is configured to detect an object that matches the information about the individual included in the tracking information from the binary images generated by the binarization unit 13. The update unit 15 is configured to update the tracking information with information about the object that matches the threshold values ​​used to generate the binary images in which the matching unit 14 successfully detected the object.

[0077] The information processing device 10 configured as described above operates as follows. The storage unit 11 stores tracking information for each individual, including the threshold value used in the binarization process and information about the detected individual. The threshold value setting unit 12 sets a predetermined number of threshold values ​​based on the threshold value included in the tracking information stored in the storage unit 11. Next, the binarization unit 13 binarizes the image using the predetermined number of threshold values ​​set by the threshold value setting unit 12 to generate a predetermined number of binary images. Next, the matching unit 14 detects an object that matches the information about the individual included in the tracking information from the binary images generated by the binarization unit 13. Next, the update unit 15 updates the tracking information with the threshold value used to generate the binary image in which the matching unit 14 successfully detected the object and the information about the object that matches.

[0078] According to the information processing device 10 configured and operating as described above, the threshold value at which an object was successfully detected is recorded in the memory unit 11 for each individual object, and a predetermined number of threshold values ​​are set for each object being tracked based on the recorded threshold values, and binarization is performed. Therefore, compared to when the same threshold value is used for all objects, it is possible to stably extract the object to be tracked.

[0079] Although the present invention has been described above with reference to the above-mentioned embodiments, the present invention is not limited to the above-mentioned embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0080] For example, in the above-described embodiment, the present invention is applied to the detection and tracking of floating objects in a liquid, but the objects to be detected and tracked are not limited to floating objects in a liquid, and any other objects may be detected and tracked. For example, the present invention may be applied to an information processing device that detects and tracks drones flying over buildings and the like from images captured by a surveillance camera that captures the scenery of buildings and the like.

[0081] Furthermore, for example, the information processing device may use a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an MPU (Micro Processing Unit), an FPU (Floating Number Processing Unit), a PPU (Physics Processing Unit), a TPU (Tensor Processing Unit), a quantum processor, a microcontroller, or a combination thereof, instead of the above-mentioned CPU.

[0082] The present invention can be applied to general image processing in which binarization is utilized to detect and track any object from an image.

[0083] Some or all of the above embodiments may be described as, but are not limited to, the following supplementary notes: [Supplementary Note 1] An information processing device that detects and tracks any number of objects that appear in the foreground of an image as separate individuals through a binarization process, comprising: a storage means that stores tracking information for each individual, the tracking information including thresholds used in the binarization process and information about the detected individuals; a threshold setting means that sets a predetermined number of thresholds based on the thresholds included in the tracking information; a binarization means that binarizes the image using the set predetermined number of thresholds to generate a predetermined number of binary images; a matching means that detects an object that matches information about the individual included in the tracking information from the generated binary images; and an update means that updates the tracking information with the thresholds used to generate the binary images that successfully detected the object and information about the matching object. [Supplementary Note 2] The information processing device according to Supplementary Note 1, further comprising: a prediction means for predicting an image region where the individual will next be present using information about the individual included in the tracking information; and the binarization means binarizes the predicted image region using the set predetermined number of thresholds to generate a predetermined number of binary images. [Supplementary Note 3] The information processing device according to Supplementary Note 2, wherein if the matching means fails to detect the object, the threshold setting means sets a predetermined number of other thresholds based on the thresholds included in the tracking information; the binarization means binarizes the image region using the set predetermined number of other thresholds to generate a predetermined number of other binary images; and the matching means detects an object that matches the information about the individual included in the tracking information from the generated predetermined number of other binary images. [Supplementary Note 4] The information processing device according to Supplementary Note 3, wherein the threshold setting means repeats the process of setting a predetermined number of other thresholds based on the thresholds included in the tracking information until the total number of set thresholds reaches an upper limit. [Supplementary Note 5] The information processing device according to any of Supplements 1 to 4, wherein the predetermined number is 1. [Supplementary Note 6] The information processing device according to any one of Supplementary Notes 1 to 4, wherein the predetermined number is two or more.[Supplementary Note 7] An information processing method for detecting and tracking any number of foreground objects in an image as separate individuals through binarization processing, comprising: storing, for each individual, tracking information including thresholds used in the binarization processing and information on the detected individuals; setting a predetermined number of thresholds based on the thresholds included in the tracking information; binarizing the image using the set predetermined number of thresholds to generate a predetermined number of binary images; detecting, from the generated binary images, an object that matches information on the individual included in the tracking information; and updating the tracking information with the thresholds used to generate the binary images in which the object was successfully detected and information on the matching object. [Supplementary Note 8] The information processing method according to Supplementary Note 7, further comprising: predicting an image region in which the individual will next be present using information on the individual included in the tracking information; and, in generating the binary images, binarizing the predicted image region using the set predetermined number of thresholds to generate a predetermined number of binary images. [Supplementary Note 9] The information processing method according to Supplementary Note 8, further comprising: if detection of the object fails, setting a predetermined number of other thresholds based on the threshold included in the tracking information; binarizing the image region using the set predetermined number of other thresholds to generate a predetermined number of other binary images; and detecting an object that matches information of the individual included in the tracking information from the generated predetermined number of other binary images. [Supplementary Note 10] The information processing method according to Supplementary Note 9, further comprising: repeating the process of setting a predetermined number of other thresholds based on the threshold included in the tracking information until the total number of set thresholds reaches an upper limit. [Supplementary Note 11] The information processing method according to any of Supplements 7 to 10, wherein the predetermined number is 1. [Supplementary Note 12] The information processing method according to any of Supplements 7 to 10, wherein the predetermined number is 2 or more.[Supplementary Note 13] A computer-readable recording medium having recorded thereon a program for causing a computer, which detects and tracks any number of foreground objects in an image as separate individuals through a binarization process, to perform the following steps: storing tracking information for each individual, the tracking information including thresholds used in the binarization process and information on the detected individuals; setting a predetermined number of thresholds based on the thresholds included in the tracking information; binarizing the image using the set predetermined number of thresholds to generate a predetermined number of binary images; detecting an object from the generated binary images that matches information on the individual included in the tracking information; and updating the tracking information with the thresholds used to generate the binary images in which the object was successfully detected and information on the matching object.

[0084] REFERENCE SIGNS LIST 1 Inspection system 2 Container 3 Flow induction device 4 Lighting device 5 Camera device 6 Information processing device 10 Information processing device 11 Storage unit 12 Threshold value setting unit 13 Binarization unit 14 Collation unit 15 Update unit 61 Communication I / F unit 62 Operation input unit 63 Screen display unit 64 Storage unit 65 Arithmetic processing unit

Claims

1. An information processing device that detects and tracks any number of objects in the foreground of an image as separate individuals through binarization processing, a storage means for storing tracking information for each individual, the tracking information including the threshold value used in the binarization process and information on the detected individual; a threshold value setting means for setting a predetermined number of threshold values ​​based on the threshold values ​​included in the tracking information; a binarization means for binarizing the image using the set predetermined number of threshold values ​​to generate a predetermined number of binary images; a matching means for detecting an object that matches information about an individual included in the tracking information from the generated binary image; an updating means for updating the tracking information with the threshold value used to generate the binary image in which the object was successfully detected and information on the matching object; An information processing device comprising:

2. a prediction unit for predicting an image region where the individual will next be present using information about the individual included in the tracking information; the binarization means binarizes the predicted image area using the set predetermined number of threshold values ​​to generate a predetermined number of binary images; The information processing device according to claim 1 .

3. the threshold setting means sets a predetermined number of other thresholds based on the thresholds included in the tracking information when the matching means fails to detect the object; the binarization means binarizes the image region using the set predetermined number of different threshold values ​​to generate a predetermined number of different binary images; the matching means detects an object that matches information about the individual included in the tracking information from the generated predetermined number of other binary images.

3. The information processing device according to claim 1.

4. the threshold setting means repeats the process of setting a predetermined number of other thresholds based on the thresholds included in the tracking information until the total number of set thresholds reaches an upper limit; The information processing device according to claim 3 .

5. the predetermined number is 1; 3. The information processing device according to claim 1.

6. the predetermined number is 2 or more; 3. The information processing device according to claim 1.

7. An information processing method for detecting and tracking any number of foreground objects in an image as separate individuals through binarization processing, comprising: storing tracking information for each individual, the tracking information including the threshold value used in the binarization process and information about the detected individual; setting a predetermined number of thresholds based on the thresholds included in the tracking information; binarizing the image using the set predetermined number of threshold values ​​to generate a predetermined number of binary images; Detecting an object that matches information about an individual included in the tracking information from the generated binary image; updating the tracking information with the threshold value used to generate the binary image in which the object was successfully detected and information on the matching object; Information processing methods.

8. Furthermore, predicting an image region in which the individual will next be present using information about the individual contained in the tracking information; In generating the binary images, the predicted image area is binarized using the set predetermined number of threshold values ​​to generate a predetermined number of binary images. The information processing method according to claim 7.

9. moreover, If the object detection fails, a predetermined number of other thresholds are set based on the thresholds included in the tracking information; binarizing the image region using the set predetermined number of other threshold values ​​to generate a predetermined number of other binary images; detecting an object that matches information about the individual included in the tracking information from the generated predetermined number of other binary images; 9. The information processing method according to claim 7 or 8.

10. A computer that detects and tracks any number of foreground objects in an image as separate individuals through binarization processing. a process of storing tracking information for each individual, the tracking information including the threshold value used in the binarization process and information on the detected individual; setting a predetermined number of thresholds based on the thresholds included in the tracking information; a process of binarizing the image using the set predetermined number of threshold values ​​to generate a predetermined number of binary images; A process of detecting an object that matches information about an individual included in the tracking information from the generated binary image; updating the tracking information with the threshold value used to generate the binary image in which the object was successfully detected and information on the matching object; A program to perform the following.