Image processing device, image processing method, and image processing program

Abstract

To continue tracking of an object even when the object is not detected temporarily.SOLUTION: An image processing device includes a control unit that executes: processing of acquiring a first frame image of a video of a road captured by a camera; processing of extracting an object on the road from the first frame image; processing of detecting a first position of the object extracted from the first frame image and setting a timer associated with the object; processing of advancing the timer when the object is not extracted at the first position in a second frame image acquired after the first frame image; and processing of continuing tracking of the object until the timer expires.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and an image processing program. [Background technology]

[0002] A technique has been proposed for detecting an object from frame images in a video captured by a camera and tracking the detected object (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-206934 Summary of the Invention [Problem to be solved by the invention]

[0004] In the past, tracking of objects was based on the overlap and positional relationship between frames of the tracking frames set for the object. Therefore, if an object is temporarily not detected in the frame images due to an obstruction between the object being tracked and the camera, or if a sudden change in brightness occurs due to lighting being turned on or sunlight halation, tracking of the object may fail.

[0005] An object of one aspect of the disclosed technology is to provide an image processing device, an image processing method, and an image processing program that can continue tracking an object even if the object is temporarily no longer detected. [Means for solving the problem]

[0006] One aspect of the disclosed technology is exemplified by the following image processing device: The image processing device includes a control unit that executes the following processes: acquiring a first frame image of a video in which a road is captured by a camera; extracting an object on the road from the first frame image; detecting a first position of the object extracted from the first frame image and setting a timer associated with the object; and, if the object is not extracted at the first position in a second frame image acquired after the first frame image, advancing the timer; and continuing to track the object until the timer expires.

[0007] With an image processing device having the above configuration, even if the object is no longer detected in the frame image, tracking of the object can be continued until the timer expires. Therefore, with this image processing device, tracking of the object can be continued even if the object is temporarily no longer detected. Here, the control unit may stop tracking the object using the frame when the timer expires.

[0008] The image processing device may further include the following feature: The process of advancing the timer includes a process of resetting the timer when the object is extracted at the first position in a third frame image acquired after the second frame image. By providing the image processing device with this feature, even in a case where an obstruction passes between the camera and the object multiple times, the timer is reset after the obstruction passes, making it possible to continue tracking of the object.

[0009] The image processing device may further include the following feature: the control unit is configured to When the timer expires, the image processing device starts tracking the object, and when the timer expires, ends tracking the object, and if tracking of the object continues for a predetermined time or more, executes a process of issuing a notification that the object is present on the road. If the object is continuously tracked for a predetermined time or more before the timer expires, the object is considered to be a static object on the road. By having such features, the image processing device can issue a notification of the presence of an object that is static on the road and therefore may obstruct vehicle travel.

[0010] The image processing device may further include the following feature. The control unit sets a frame for the object extracted from the first frame image to be used for tracking the position of the object on the video, and when a number of pixels forming another object detected in the second frame image that overlaps at least a portion of the frame is equal to or greater than a first threshold and overlaps with the frame, the control unit suppresses execution of the process of advancing the timer. When the number of pixels forming the other object that overlap with the frame is less than the first threshold, it is unlikely that the object will be hidden behind the other object, and it is therefore possible that the object has already moved. By including such a feature, the image processing device can advance the timer when it is considered that the object has already moved, thereby bringing tracking closer to completion. Furthermore, when it is considered that the object has not yet moved, the process of advancing the timer is suppressed, thereby allowing tracking of the object to continue.

[0011] The image processing device may further include the following feature: for the object extracted from the first frame image, the control unit sets a frame used for tracking the position of the object on the video, and when, among pixels forming another object detected in the second frame image at a position at least partially overlapping the frame, a number equal to or greater than a first threshold is present overlapping the position where the object was present in the first frame image, the control unit suppresses execution of the process of advancing the timer. By including such a feature, tracking of the object can be continued even if the object is temporarily no longer detected.

[0012] The image processing device may further include the following feature. The control unit sets a frame for the object extracted from the first frame image to be used for tracking the position of the object on the video, and when the size of another object detected in the second frame image at a position at which at least a portion of the object overlaps the frame is equal to or greater than a second threshold, the control unit suppresses execution of the process of advancing the timer. It is considered unlikely that the object will not be extracted in the frame image because the object is hidden behind another small object. In other words, if the object is not extracted from the frame image in such a case, it is considered that the object has already moved. By including this feature, the image processing device can advance the timer when it is considered that the object has already moved, thereby bringing tracking closer to completion. Furthermore, when it is considered that the object has not yet moved, the process of advancing the timer is suppressed, thereby allowing tracking of the object to continue.

[0013] The image processing device may further include the following feature: The control unit executes a process of setting a second timer, which indicates the duration for tracking the object using the frame, in association with the frame, and a process of advancing the second timer when the object is extracted at the position where the frame is set in the second frame image, and the suppression is executed when the second timer is equal to or greater than a third threshold. Objects that are dangerous to vehicles traveling on roads are considered to be objects that do not move even due to wind, etc. By including such a feature, the image processing device can perform the process of setting a second timer in association with the frame until the second timer is equal to or greater than the third threshold. By suppressing the advancement of the timer for the object being continuously tracked, it becomes easier to continue tracking the more dangerous object.

[0014] The disclosed technology can also be understood from the aspects of an image processing method and an image processing program. [Effects of the Invention]

[0015] According to the disclosed technology, tracking of an object can be continued even if the object is temporarily no longer detected. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 is a diagram illustrating an example of a falling object detection system according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of the detection device according to the embodiment. [Figure 3] FIG. 3 is a diagram illustrating an example of a processing block of the detection device according to the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of foreground extraction by the extraction unit. [Figure 5] FIG. 5 is a diagram illustrating an example of tracking of the foreground by the tracking unit in the embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of a management information table stored in the auxiliary storage unit. [Figure 7]FIG. 7 is a diagram illustrating an example of timer processing by the tracking unit. [Figure 8] FIG. 8 is a diagram illustrating an example of a processing flow of the detection device according to the embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of a processing block of a detection device according to a first modified example. [Figure 10] FIG. 10 is a diagram showing an example of a management information table according to the first modified example. [Figure 11] FIG. 11 is a diagram illustrating an example of the operation of a timer set by the tracking unit in the first modified example. DETAILED DESCRIPTION OF THE INVENTION

[0017] <Application example> An application example of the present invention will be described. A detection device 2 according to the application example of the present invention is used to monitor a road 500, for example, as illustrated in Fig. 1. The detection device 2 is a device that detects objects falling on the road 500 based on frame images acquired from a camera 1. Details of the hardware configuration of the detection device 2 will be described later, but the detection device 2 is an information processing device that includes a processor and a storage unit.

[0018] The detection device 2 extracts the foreground by removing the background from the first frame image input from the camera 1. The extraction of the foreground can be performed, for example, by background subtraction. The detection device 2 sets a tracking frame for the extracted foreground and starts tracking the foreground. Here, the detection device 2 sets a timer in association with the set tracking frame.

[0019] If another object is photographed at the position where the tracking frame is set in a second frame image acquired after the first frame image, the detection device 2 advances the timer. If the foreground is extracted at the position of the set tracking frame before the timer expires, the detection device 2 continues tracking the foreground using the tracking frame. In other words, according to this application example, if the state in which the foreground does not appear in the frame image is temporary, tracking of the foreground can be continued.

[0020] <Embodiment> Hereinafter, an embodiment will be described with reference to the drawings. FIG. 1 is a diagram showing an example of a falling object detection system 100 according to an embodiment. The falling object detection system 100 is a system that detects an object that has fallen onto a road 500. The falling object detection system 100 includes a camera 1, a detection device 2, and a display device 3. The camera 1 and the detection device 2 are connected to a computer network N1. The detection device 2 and the display device 3 are connected by a connection cable L1.

[0021] The road 500 is, for example, a motorway or a national expressway, but may also be a general national road, a prefectural road, or a municipal road.

[0022] Camera 1 is a video camera that captures images of road 500 to be monitored. Camera 1 may be, for example, a network camera that transmits captured images via computer network N1. Camera 1 is, for example, a digital video camera that uses a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) as an imaging element. Video data captured by camera 1 is transmitted to detection device 2 via computer network N1. The frame rate of the video data captured by camera 1 is, for example, 1 fps.

[0023] The detection device 2 is an information processing device that detects a fallen object on the road 500 based on video acquired from the camera 1 via the computer network N1. When the detection device 2 detects a fallen object on the road 500, it outputs the video of the detected fallen object to the display device 3 via the connection cable L1.

[0024] The display device 3 is a display monitored by the traffic wardens P1. The display device 3 is, for example, a Liquid Crystal Display (LCD), a Plasma Display Panel (PDP), an inorganic electroluminescence (EL) panel, or an organic EL panel. The display device 3 displays the information output by the detection device 2.

[0025] 2 is a diagram illustrating an example of the hardware configuration of a detection device 2 according to an embodiment. The detection device 2 includes a central processing unit (CPU) 201, a main memory unit 202, an auxiliary memory unit 203, a communication unit 204, a connection terminal 205, and a bus B1. The CPU 201, the main memory unit 202, the auxiliary memory unit 203, the communication unit 204, and the connection terminal 205 are interconnected by the bus B1.

[0026] The CPU 201 is also referred to as a microprocessor unit (MPU) or processor. The CPU 201 is not limited to a single processor and may have a multi-processor configuration. Furthermore, a single CPU 201 connected via a single socket may have a multi-core configuration. At least a portion of the processing performed by the CPU 201 may be performed by a processor other than the CPU 201, such as a dedicated processor, such as a digital signal processor (DSP), a graphics processing unit (GPU), a numerical calculation processor, a vector processor, or an image processing processor. Furthermore, at least a portion of the processing performed by the CPU 201 may be performed by an integrated circuit (IC) or other digital circuit. Furthermore, at least a portion of the CPU 201 may include an analog circuit. Examples of integrated circuits include large-scale integrated circuits (LSIs), application-specific integrated circuits (ASICs), and programmable logic devices (PLDs). Examples of PLDs include field-programmable gate arrays (FPGAs). The CPU 201 may be a combination of a processor and an integrated circuit. The combination is called, for example, a microcontroller unit (MCU), a system-on-a-chip (SoC), a system LSI, a chipset, etc. In the detection device 2, the CPU 201 expands a program stored in the auxiliary storage unit 203 into a work area of ​​the main storage unit 202, and executes the program to control the peripheral devices. This allows the detection device 2 to execute processing that meets a predetermined purpose. The main memory unit 202 and the auxiliary memory unit 203 are recording media that the detection device 2 can read.

[0027] The main storage unit 202 is exemplified as a storage unit that is directly accessed by the CPU 201. The main storage unit 202 includes a random access memory (RAM) and a read only memory (ROM).

[0028] The auxiliary storage unit 203 stores various programs and various data on a readable and writable recording medium. The auxiliary storage unit 203 is also called an external storage device. The auxiliary storage unit 203 stores an operating system (OS), various programs, various tables, etc. The OS includes a communication interface program that exchanges data with external devices connected via the communication unit 204. The external devices include, for example, other information processing devices and external storage devices connected via a computer network, etc. The auxiliary storage unit 203 may be, for example, part of a cloud system, which is a group of computers on a network.

[0029] The auxiliary storage unit 203 is, for example, an erasable programmable ROM (EPROM), a solid state drive (SSD), a hard disk drive (HDD), etc. Furthermore, the auxiliary storage unit 203 is, for example, a compact disc (CD) drive device, a digital versatile disc (DVD) drive device, a Blu-ray (registered trademark) disc (BD) drive device, etc. Furthermore, the auxiliary storage unit 203 may be provided by a network attached storage (NAS) or a storage area network (SAN).

[0030] The communication unit 204 is, for example, an interface with the computer network N1, and communicates with external devices such as the camera 1 via the computer network N1.

[0031] The connection terminal 205 is a connection terminal to which the connection cable L1 is connected. When the detection device 2 detects a falling object, it outputs an alarm to the display device 3 via the connection cable L1.

[0032] <Detection device 2 processing block> 3 is a diagram showing an example of a processing block of the detection device 2 according to the embodiment. The detection device 2 includes an extraction unit 21, a tracking unit 22, and a reporting unit 23. The detection device 2 executes the processes of each unit of the detection device 2, such as the extraction unit 21, the tracking unit 22, and the reporting unit 23, by having the CPU 201 execute a computer program deployed in an executable manner in the main memory unit 202.

[0033] The extraction unit 21 generates an extracted image by extracting the foreground from a frame image in a video received from the camera 1. The extraction of the foreground from a frame image is performed, for example, by background subtraction. FIG. 4 is a diagram showing an example of foreground extraction by the extraction unit 21. In the example of FIG. 4, a frame image I1 at time T and a frame image I2 at time T+1, which is the time immediately following time T, are input to the extraction unit 21. Frame image I1 shows only a road 500, while frame image I2 shows a box O1 and a passenger car O2 on the road 500. The extraction unit 21 extracts the box O1 as the foreground based on the difference between frame image I1 and frame image I2, and generates an extracted image D1 including the box O1 and the passenger car O2. The extracted image D1 is, for example, an image in which the box O1 and the passenger car O2 extracted as the foreground are shown in white and the road 500 in the background has been removed.

[0034] Returning to FIG. 3, the tracking unit 22 tracks the foreground (for example, a box O1 and an automobile O2) extracted by the extraction unit 21. If the tracked foreground is an object that has fallen onto the road 500, the tracking unit 22 causes the reporting unit 23 to report it. For example, the tracking unit 22 determines that a foreground that has been tracked continuously for a determination time or more set for falling object determination is a fallen object that is not moving on the road 500. Furthermore, the tracking unit 22 determines that a foreground whose tracking duration is less than the determination time is not a fallen object. In the example of FIG. 4, for example, the automobile O2 passes by in less than the determination time, and is therefore determined not to be a fallen object. The determination time may be determined based on, for example, the time required for the automobile O2 to pass through the shooting range of the camera 1. Details of tracking by the tracking unit 22 will be described later.

[0035] When the tracking unit 22 detects an object that has fallen onto the road 500, the reporting unit 23 notifies the traffic warden P1 of the detection of the fallen object by displaying a screen showing the detection on the display device 3. The reporting unit 23 may notify the traffic warden P1 of the detection of the object that has fallen onto the road 500 by, for example, outputting a sound from a speaker.

[0036] (Tracking of foreground by tracking unit 22) 5 is a diagram showing an example of foreground tracking by the tracking unit 22 in an embodiment. The tracking unit 22 sets tracking frames W1 and W2 for the box O1 and the automobile O2 extracted as the foreground in the extracted image D1 generated by the extraction unit 21, respectively. The tracking frame W1 is a rectangular frame set to surround the periphery of the box O1. The tracking frame W2 is a rectangular frame set to surround the periphery of the automobile O2. The tracking unit 22 tracks the box O1 and the automobile O2 using the tracking frames W1 and W2.

[0037] The tracking unit 22 also sets a timer for each of the tracking frame W1 and the tracking frame W2. The timer is decremented when tracking using the tracking frame W1 is not possible due to, for example, another object appearing between the box O1 and the camera 1 or a sudden change in brightness caused by the headlights of an automobile O2 traveling on the road 500 being turned on. The tracking unit 22 stores a management information table for managing the set tracking frames W1 and W2 in the auxiliary storage unit 203.

[0038] FIG. 6 is a diagram illustrating an example of the management information table 221 stored in the auxiliary storage unit 203. The management information table 221 includes the following fields: "ID," "Position," "Size," and "Timer." "ID" stores information uniquely assigned to each of the tracking frames W1 and W2. "Position" stores information indicating the positions of the tracking frames W1 and W2. Examples of information indicating the positions of the tracking frames W1 and W2 include the coordinates of the upper left corners of the tracking frames W1 and W2. "Size" stores information indicating the sizes of the tracking frames W1 and W2. Examples of information indicating the sizes of the tracking frames W1 and W2 include the width and height of the tracking frames W1 and W2. "Timer" stores the remaining time of a timer set for the tracking frames W1 and W2 by the tracking unit 22. FIG. 6 illustrates an example in which a two-second timer is set when the tracking frames W1 and W2 are set. The unit of the "timer" is, for example, seconds. The information stored in the management information table 221 is updated by the tracking unit 22.

[0039] (Timer processing by tracking unit 22) 7A to 7C are diagrams illustrating an example of timer processing by the tracking unit 22. In the example of Figures 7A to 7C, a tracking frame W1 is set, and a two-second timer is set in the tracking frame W1.

[0040] 7A illustrates an example in which tracking is successfully performed in tracking frame W1, where tracking is started, from time T, to time T+1, which is one second after time T, to time T+2, which is two seconds after time T. In such a case, the tracking unit 22 does not decrement the timer set for tracking frame W1.

[0041] FIG. 7B illustrates a case where the box O1 is not extracted in the area where the tracking frame W1 is set at time T+1 due to, for example, another object appearing between the box O1 and the camera 1 temporarily (for one second in the example of FIG. 7B). In FIG. 7B, the dashed rectangle illustrates that the box O1 was not extracted at time T+1. In this case, the tracking unit 22 decrements the timer set for the tracking frame W1 at time T+1 by one second. At time T+2, the box O1 is extracted at the position where the tracking frame W1 is set, and the tracking unit 22 resets the timer to two seconds. That is, in FIG. 7B, even if the box O1 is temporarily not extracted by the extraction unit 21, tracking of the box O1 continues.

[0042] 7C illustrates a case where a box O1 is not extracted in the area where the tracking frame W1 is set after time T+1. In this case, at time T+1, the tracking unit 22 decrements the timer set for the tracking frame W1 by one second. Furthermore, at time T+1, the tracking unit 22 decrements the timer set for the tracking frame W1 by another one second. As a result, the timer set for the tracking frame W1 becomes 0 seconds at time T+1, and the tracking unit 22 ends tracking using the tracking frame W1.

[0043] <Processing flow of detection device 2> 8 is a diagram showing an example of a processing flow of the detection device 2 according to the embodiment. Hereinafter, an example of the processing flow of the detection device 2 will be described with reference to FIG.

[0044] In step S1, the extraction unit 21 acquires an image from the camera 1. In step S2, the extraction unit 21 extracts the foreground from the image acquired in step S1.

[0045] In step S3, the tracking unit 22 sets a tracking frame for the foreground extracted in step S2. The tracking unit 22 stores the position and size of the set tracking frame in the management information table 221.

[0046] In step S4, the tracking unit 22 refers to the management information table 221 and determines whether or not a tracking frame exists that is the same in position and size as the tracking frame set in step S3. At this time, the tracking unit 22 may determine that the tracking frame is in the same position and size if the difference in position and size is within a preset tolerance. If a tracking frame in the same position and size exists (YES in step S4), the process proceeds to step S5. If a tracking frame in the same position and size does not exist (NO in step S4), the process proceeds to step S8.

[0047] In step S5, the tracking unit 22 updates the management information table 221. The tracking unit 22 associates the ID of the tracking frame set in step S2 with the ID of a tracking frame that has been determined to have the same position and size as the tracking frame. Furthermore, the tracking unit 22 deletes from the management information table 221 the information of the tracking frame that has been determined to have the same position and size as the tracking frame.

[0048] In step S6, the tracking section 22 processes the timer set in the tracking frame. The timer processing is as described with reference to Figures 7A to 7C.

[0049] In step S7, the tracking unit 22 detects whether the timer set in the tracking frame is below 0. If the timer is less than 0 (YES in step S7), the process proceeds to step S9. If the timer is equal to or greater than 0 (NO in step S7), the process proceeds to step S1.

[0050] In step S8, the tracking unit 22 assigns a new ID to the tracking frame set in step S2. Then, the process proceeds to step S6.

[0051] <Effects of the embodiment> In the embodiment described above, the tracking unit 22 sets a timer for the tracking frames W1 and W2. The timer is decremented by the tracking unit 22 if the extraction unit 21 is temporarily unable to extract the box O1 or the automobile O2. If the extraction unit 21 extracts the box O1 or the automobile O2 before the timer reaches "0," the tracking unit 22 continues tracking using the tracking frames W1 and W2. Therefore, according to this embodiment, it is permissible to lose sight of the box O1 or the automobile O2 within the time set by the timer. Consequently, tracking using the tracking frames W1 and W2 can be continued even if a vehicle or the like temporarily passes between the camera 1 and the box O1 or the automobile O2.

[0052] In this embodiment, the tracking unit 22 does not detect as a fallen object any foreground object that is not tracked continuously for the determination time or longer. This prevents an object such as a piece of paper that quickly moves across the road 500 and does not obstruct travel on the road 500, such as a passenger car O2, from being reported to the traffic warden P1 as a fallen object. Furthermore, in this embodiment, the tracking unit 22 detects as a fallen object any foreground object that is tracked continuously for the determination time or longer until the timer expires. Then, the reporting unit 23 can report the detection of a fallen object by the tracking unit 22 to the traffic warden P1.

[0053] <First Modification> In the embodiment described above, even if a vehicle or the like passes between the camera 1 and the box O1 or the automobile O2, the box O1 or the automobile O2 can be lost within the time set in the timer, and tracking using the tracking frames W1 and W2 can continue. In the first modified example, a configuration will be described in which the timer set in the tracking frames W1 and W2 is not decreased under certain conditions.

[0054] 9 is a diagram showing an example of a processing block of a detection device 2A according to Modification 1. The detection device 2A differs from the detection device 2 according to the embodiment in that it includes a tracking unit 22A instead of the tracking unit 22.

[0055] FIG. 10 is a diagram showing an example of a management information table 221A according to the first modified example. The management information table 221A differs from the management information table 221 according to the embodiment in that it further includes the fields of "duration" and "exception time." The "duration" stores the time during which tracking has been performed continuously by the tracking frames W1 and W2. Immediately after the tracking frames W1 and W2 are set, "0" is stored in "duration." The "exception time" stores the time during which a state in which a predetermined condition, which will be described later, is satisfied. The units of "duration" and "exception time" are, for example, "seconds."

[0056] The tracking unit 22A stores the duration for which tracking by the tracking frames W1 and W2 continues in the "duration" field of the management information table 221A. Even if a vehicle passes between the box O1 or the car O2 and the camera 1, the tracking unit 22A does not decrease the timers set for the tracking frames W1 and W2 if a predetermined condition is met. The predetermined condition can be the following conditions (1) to (3). For example, if the following condition (1) or (2) is met, the tracking unit 22A decreases the timer set for the tracking frames W1 and W2. , W2 does not decrement the timer set for the tracking frames W1 and W2. Furthermore, the tracking unit 22A may not decrement the timer set for the tracking frames W1 and W2, for example, when the following condition (3) is met in addition to at least one of the following conditions (1) or (2). By taking the following predetermined conditions into consideration, tracking is prevented from being interrupted when a vehicle crosses or when brightness suddenly changes due to vehicle headlights, etc. (1) The number of foreground pixels present within the tracking frames W1 and W2 is equal to or greater than a first threshold. (2) The size (width or height) of the foreground is equal to or greater than a second threshold. (3) The duration is equal to or longer than a third threshold value. Here, the third threshold value is set to, for example, a time shorter than the determination time for the tracking unit 22A to detect a falling object.

[0057] In addition, if the exceptional time that satisfies the above conditions (1) to (3) becomes equal to or greater than a fourth threshold, the tracking unit 22A may reset the timers set in the tracking frames W1 and W2 to "0."

[0058] 11A and 11B are diagrams illustrating an example of the operation of a timer set by the tracking unit 22A in Modification 1. In the example of Fig. 11A and Fig. 11B, the tracking unit 22 sets the third threshold value for the duration to 5 seconds.

[0059] FIG. 11A illustrates a case where the duration of the tracking frame W1 is 5 seconds or more at time T. At time T, the timer is set to 2 seconds and the duration is set to 5 seconds. At time T+1, a truck O3 appears between the box O1 and the camera 1, causing the box O1 to be no longer extracted by the tracking unit 22. The tracking unit 22A calculates the number of pixels within the tracking frame W1 of the truck O3, which appears in the foreground of the extracted image. Here, it is assumed that the calculated number of pixels is equal to or greater than a first threshold. The tracking unit 22A also calculates the width and height of the truck O3 in the extracted image. Here, it is assumed that at least one of the calculated width and height is equal to or greater than a second threshold. Because the above conditions (1) to (3) are satisfied, the tracking unit 22A does not decrease the timer set for the tracking frame W1 even if the box O1 has not been extracted. Therefore, in FIG. 11A, the timer at time T+1 is maintained at 2 seconds. The duration is calculated in total from time T and is 6 seconds.

[0060] At time T+2, the truck O3 is no longer present in the extracted image, and the box O1 is extracted by the extraction unit 21. Therefore, the tracking unit 22A sets a tracking frame W1 for the box O1. Furthermore, because the above conditions (1) to (3) are satisfied at time T+1, the total duration calculated from time T is 7 seconds.

[0061] FIG. 11B illustrates a case where the duration of tracking frame W1 is less than 5 seconds at time T. At time T, the timer is set to 2 seconds and the duration is set to 0 seconds. At time T+1, a truck O3 appears between box O1 and camera 1, causing box O1 to no longer be extracted by tracking unit 22. Furthermore, the duration at time T+1 is 0 seconds. Because the duration is 0 seconds, tracking unit 22A determines that the above conditions (1) to (3) are not met, and decrements the timer by 1 second. As a result, the timer at time T+1 becomes 1 second.

[0062] At time T+2, the truck O3 disappears from the extracted image, and the box O1 is extracted by the extraction unit 21. Therefore, the tracking unit 22A sets a tracking frame W1 for the box O1. Furthermore, since the box O1 was extracted and the tracking frame W1 was set before the timer set for the tracking frame W1 reached 0, the tracking unit 22A The timer is reset to 2 seconds. In addition, because box O1 was extracted and tracking frame W1 was set before the timer set in tracking frame W1 reached 0, tracking unit 22A calculates the total duration from time T to 2 seconds.

[0063] For example, if the box O1 is not extracted even though the above conditions (1) and (2) are not satisfied, it is possible that the object to be tracked, such as the box O1 or the automobile O2, has already moved. Also, if the above condition (3) is not satisfied, it is possible that the object to be tracked is an object that was temporarily present on the road 500. In the first modification, by applying the above conditions (1) to (3) as conditions for not decreasing the timer, it is possible to prevent an object that is temporarily present on the road 500 from being detected.

[0064] <Other variations> In the first modified example, the predetermined condition is "(1) the number of foreground pixels present in tracking frames W1 and W2 is equal to or greater than a first threshold value," but this predetermined condition (1) may be modified as follows: An example of a predetermined condition obtained by modifying (1) is condition (1A) "the number of pixels of an obstructing object present in tracking frames W1 and W2 at time T+1 at the pixel positions of the foreground present in tracking frames W1 and W2 at time T is equal to or greater than a first threshold value."

[0065] For example, if the above condition (1A) or (2) is met, the tracking unit 22A does not decrement the timers set for the tracking frames W1 and W2. Furthermore, the tracking unit 22A may not decrement the timers set for the tracking frames W1 and W2 if the above condition (3) is met in addition to at least one of the above conditions (1A) or (2). Even with this processing, tracking using the tracking frames W1 and W2 can continue even if a vehicle or the like passes temporarily between the camera 1 and the box O1 or the automobile O2.

[0066] In this embodiment, the tracking unit 22 tracks the foreground using a tracking frame set for the foreground. However, tracking of the foreground by the tracking unit 22 is not limited to using a tracking frame. The tracking unit 22 may track the foreground, for example, based on a first position of the foreground in frame image I1 at time T and a second position of the foreground in frame image I2 at time T+1. For example, the tracking unit 22 may advance the timer if the foreground is not extracted at the first position in frame image I2. Here, the position of the foreground in frame image I1 and the position of the foreground in frame image I2 do not need to exactly match, and some deviation is permissible.

[0067] The embodiments and modifications disclosed above can be combined with each other.

[0068] <Computer-readable recording medium> An information processing program that causes a computer or other machine or device (hereinafter referred to as a computer, etc.) to realize any of the above functions can be recorded on a computer-readable recording medium. Then, by having the computer, etc. read and execute the program from this recording medium, the function can be provided.

[0069] Here, a computer-readable recording medium refers to a recording medium that stores information such as data and programs electrically, magnetically, optically, mechanically, or chemically and can be read by a computer. Among such recording media, those that can be removed from a computer include, for example, flexible disks, magneto-optical disks, Compact Disc Read Only Memory (CD-ROM), Compact Disc-Recordable (CD-R), and Compact Disc. -Rewriterable (CD-RW), Digital Versatile Disc (DVD), Blu-ray Disc (BD), Digital Audio Tape (DAT), 8mm tape, flash memory, external hard disk drives, solid state drives (SSD), etc. In addition, there are built-in hard disk drives, SSDs, ROMs, etc. as recording media fixed to computers, etc.

[0070] <Appendix 1> A process of acquiring a first frame image of a video of a road (500) captured by a camera (1); A process of extracting objects (O1, O2) on the road (500) from the first frame image; a process of detecting a first position of the object (O1, O2) extracted from the first frame image and setting a timer associated with the object (O1, O2); a process of advancing the timer when the object (O1, O2) is not extracted at the first position in a second frame image acquired after the first frame image; and a control unit (201) that executes a process of continuing to track the objects (O1, O2) until the timer expires. Image processing device (2). <Appendix 2> the process of advancing the timer includes a process of resetting the timer when the object (O1, O2) is extracted at the first position in a third frame image acquired after the second frame image. The image processing device (2) according to appendix 1. <Appendix 3> The control unit (201) When the timer is set, start tracking the objects (O1, O2); When the timer expires, the tracking of the object (O1, O2) is terminated; If the tracking of the object (O1, O2) continues for a predetermined time (determination time) or more, a process of issuing a notification that the object (O1, O2) is present on the road is further executed. An image processing device (2) according to appendix 1 or 2. <Appendix 4> The control unit (201) for the object (O1, O2) extracted from the first frame image, setting a frame (W1, W2) to be used for tracking the position of the object (O1, O2) on the video; In the second frame image, when a number of pixels forming another object (O1, O2) detected at a position where at least a part of the object overlaps the frame (W1, W2) is equal to or greater than a first threshold and overlaps the frame, execution of the process of advancing the timer is suppressed. An image processing device (2) according to any one of appendices 1 to 3. <Appendix 5> The control unit (201) for the object (O1, O2) extracted from the first frame image, setting a frame (W1, W2) to be used for tracking the position of the object (O1, O2) on the video; In the second frame image, pixels forming other objects (O1, O2) detected at a position at which at least a part of the objects overlaps with the frame (W1, W2) are selected based on a first threshold value. If the number of pixels is equal to or greater than the number of pixels present in the first frame image and overlaps the position where the object was present, execution of the process of advancing the timer is suppressed. The image processing device according to claim 1. <Appendix 6> The control unit (201) for the object (O1, O2) extracted from the first frame image, setting a frame (W1, W2) to be used for tracking the position of the object (O1, O2) on the video; When the size of another object (O1, O2) detected in the second frame image at a position where at least a part of the object overlaps with the frame (W1, W2) is equal to or larger than a second threshold, execution of the process of advancing the timer is suppressed. An image processing device (2) according to any one of appendices 1 to 5. <Appendix 7> The control unit (201) a process of setting a second timer corresponding to the frame (W1, W2) that indicates a duration for tracking the object (O1, O2) using the frame (W1, W2); If the object (O1, O2) is extracted at the position where the frame is set in the second frame image, a process of advancing the second timer is executed; the inhibiting is performed when the second timer is greater than or equal to a third threshold. An image processing device (2) according to any one of appendices 4 to 6. <Appendix 8> A process of acquiring a first frame image of a video of a road (500) captured by a camera (1); A process of extracting objects (O1, O2) on the road (500) from the first frame image; a process of detecting a first position of the object (O1, O2) extracted from the first frame image and setting a timer associated with the object (O1, O2); a process of advancing the timer when the object (O1, O2) is not extracted at the first position in a second frame image acquired after the first frame image; and continuing to track the objects (O1, O2) until the timer expires. Image processing methods. <Appendix 9> A process of acquiring a first frame image of a video of a road (500) captured by a camera (1); A process of extracting an object on the road (500) photographed by the camera (1) from the first frame image; a process of detecting a first position of the object (O1, O2) extracted from the first frame image and setting a timer associated with the object; a process of advancing the timer when the object (O1, O2) is not extracted at the first position in a second frame image acquired after the first frame image; and causing the computer (2) to execute a process of continuing to track the objects (O1, O2) until the timer expires. Image processing program. [Explanation of symbols]

[0071] 1. Camera 2. Detection device 2A··Detection device 3...Display device 21...Extraction part 22 Tracking section 22A Tracking unit 23. Reporting Department 100··Falling Object Detection System 500...road 201 CPU 202...Main memory 203...Auxiliary storage section 204··Communications Department 205 Connection terminal 221 Management Information Table 221A Management Information Table B1 Bus D1 Extracted image I1··Frame image I2··Frame Image L1 Connection cable N1: Computer Network O1··Box O2··Passenger car O3...lorry P1...Traffic warden W1 Tracking Frame W2 Tracking Frame

Claims

1. A process of acquiring a first frame image of a video of a road captured by a camera; A process of extracting an object on the road from the first frame image; a process of detecting a first position of the object extracted from the first frame image and setting a timer associated with the object; a process of advancing the timer when the object is not extracted at the first position in a second frame image acquired after the first frame image; and a control unit that executes a process of continuing tracking of the object until the timer expires. Image processing device.

2. the processing of advancing the timer includes processing of resetting the timer when the object is extracted at the first position in a third frame image acquired after the second frame image. The image processing device according to claim 1 .

3. The control unit When the timer is set, start tracking the object; When the timer expires, the tracking of the object is terminated; If the tracking of the object continues for a predetermined time or more, a process of issuing a notification that the object is present on the road is further executed. The image processing device according to claim 1 .

4. The control unit setting a frame for tracking the position of the object on the video, the frame being used for tracking the position of the object extracted from the first frame image; suppressing execution of the process of advancing the timer when, in the second frame image, a number of pixels equal to or greater than a first threshold value among pixels forming another object detected at a position at least partially overlapping the frame are present overlapping the frame; The image processing device according to claim 1 .

5. The control unit setting a frame for tracking the position of the object on the video, the frame being used for tracking the position of the object extracted from the first frame image; when a number of pixels equal to or greater than a first threshold value among pixels forming another object detected in a position at which at least a part of the object overlaps the frame in the second frame image are present overlapping the position where the object existed in the first frame image, execution of the process of advancing the timer is suppressed; The image processing device according to claim 1 .

6. The control unit setting a frame for tracking the position of the object on the video, the frame being used for tracking the position of the object extracted from the first frame image; when a size of another object detected in the second frame image at a position where at least a part of the object overlaps with the frame is equal to or larger than a second threshold, execution of the process of advancing the timer is suppressed; The image processing device according to claim 1 .

7. The control unit setting a second timer associated with the frame, the second timer indicating a duration for tracking the object using the frame; If the object is extracted at the position where the frame is set in the second frame image, a process of advancing the second timer is executed; the suppressing is performed when the second timer is greater than or equal to a third threshold. The image processing device according to claim 4 .

8. A process of acquiring a first frame image of a video of a road captured by a camera; A process of extracting an object on the road from the first frame image; a process of detecting a first position of the object extracted from the first frame image and setting a timer associated with the object; a process of advancing the timer when the object is not extracted at the first position in a second frame image acquired after the first frame image; and continuing to track the object until the timer expires. Image processing methods.

9. A process of acquiring a first frame image of a video of a road captured by a camera; A process of extracting an object on the road from the first frame image; a process of detecting a first position of the object extracted from the first frame image and setting a timer associated with the object; a process of advancing the timer when the object is not extracted at the first position in a second frame image acquired after the first frame image; and continuing to track the object until the timer expires. Image processing program.

Images