Device for monitoring unattended objects, system equipped with the same for monitoring unattended objects, and method for monitoring unattended objects

The unattended object monitoring apparatus provides users with visual progress information and alerts about unattended objects by tracking and generating overlays on captured images, addressing the limitations of existing systems in displaying hidden objects.

DE112016006300B4Active Publication Date: 2024-11-28PANASONIC INTELLECTUAL PROPERTY MANAGEMENT CO LTD
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
DE112016006300
Authority / Receiving Office
DE · DE
Patent Type
Patents
Current Assignee / Owner
Priority Date
2016-01-25
Filing Date
2016-11-07
Publication Date
2024-11-28
Estimated Expiration
2036-11-07

AI Technical Summary

Technical Problem

Existing surveillance systems struggle to provide users with progress information of unattended objects after they appear in a monitoring area and fail to visually display such objects when they are hidden behind other objects.

Method used

An unattended object monitoring apparatus that includes an image retrieval unit, an object tracking unit, an unattended object detecting unit, a progress information generator, and a message image generator, which tracks appearing objects, detects unattended objects, generates progress information, and overlays this information onto captured images to provide visual alerts and details.

Benefits of technology

Enables users to receive intuitive and easy-to-understand progress information about unattended objects, including elapsed time and state changes, and ensures that hidden objects are visibly displayed, enhancing the detection and monitoring process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

Device for monitoring unattended objects, which detects an unattended object left in a monitoring area based on a captured image of the monitoring area, which is imaged by an imaging device, wherein the device comprises: a processor; and a memory that stores an instruction, wherein the device further comprises, as a configuration, the following when the instruction stored in memory is executed by the processor: an image retrieval unit that retrieves a captured image of a monitoring area mapped by the imaging device; an object tracking unit that detects an appearing object that appears in the monitored area from the captured image and tracked between the recorded images for each appearing object; a unit for detecting unattended objects that identifies the appearing object that is not moved beyond a predetermined time, based on a tracking result for each appearing object by the object tracking unit as the unattended object detected; a progress information generator that produces progress information of the unattended object after its appearance in the monitored area, based on the tracking result for each appearing object by the object tracking unit; and a message image generator that creates a message image by displaying the progress information in the recorded image, characterized by: that the device further includes: an image layer generator that creates an image layer comprising an image of the appearing object for each appearing object if a plurality of the appearing objects are detected, and a background image generator that creates a background image by removing the objects that appear in the captured image, wherein the message image generator processes each image layer in the order of detection of the objects appearing that are contained in each image layer, placed on the background image...
Need to check novelty before this filing date? Find Prior Art

Description

TECHNICAL AREA

[0001] The present disclosure relates to a device for monitoring unattended objects which detects an unattended object left in a monitoring area based on recorded images of the monitoring area which is imaged by an imaging device, a system equipped with the same for monitoring unattended objects and a method for monitoring unattended objects. STATE OF THE ART

[0002] The state of the art involves monitoring suspicious, unattended objects in spaces such as train stations, airports, and the like, used by a large number of unspecified individuals, for purposes of crime prevention and similar objectives. Monitoring an unattended object is generally carried out by an observer who monitors recorded images of a surveillance area captured by a surveillance camera (imaging device). However, it is necessary to automatically detect an unattended object based on recorded images of a surveillance area in order to reduce the observer's workload and improve the accuracy of detection. Various techniques for automatically detecting an unattended object based on recorded images have therefore been proposed (see PTL 1).In the prior art technique of PTL 1, a difference is obtained between an input image of a target monitoring area, which is captured and entered at a predetermined time interval, and an initial image of the target monitoring area in order to cut out the image of a person in the target monitoring area, wherein the person images, which are obtained successively at the predetermined time interval, are compared with each other in order to detect an object image that is to be separated from the person images, and, if the object image is detected, the time for which the object of the image has been left behind is measured, so that an alarm is issued when a predetermined time has elapsed.

[0003] If an unattended object is detected, it is also desirable to be able to specify the owner of the unattended object. Therefore, various techniques for specifying the owner of an unattended object based on captured images have been proposed (see PTL 2 and PTL 3). In the prior art technique of PTL 2, an image of an unattended object and an image of a bystander within a predetermined distance range of the unattended object are extracted from a captured image, and log data comprising the image of the unattended object and the image of the bystander are extracted to provide a user with information to specify the owner of the unattended object.Furthermore, in the state-of-the-art PTL 3 technology, within a predetermined period with an endpoint immediately before the detection time at which an unattended object (suspicious object) is first detected, all objects whose detection areas overlap with a detection area of ​​the suspicious object are detected as candidate objects of the owner of the unattended object, and information (incident information) including an object image of the detected candidate object is output to provide the user with information to specify the owner of the unattended object.

[0004] JP2011055270A is concerned with an information transmission apparatus. The information transmission apparatus comprises a detection unit for detecting an object from an image; a determination unit for determining a state of a detected object; a transmission unit for transmitting object information concerning the detected object to processing apparatuses communicable via a network as image attribute information; a reception unit for receiving a request concerning the attribute information to be transmitted by the transmission unit, from the processing apparatuses communicable via the network; and a control unit for determining the content of the attribute information to be transmitted to the processing apparatuses, based on the values included in the request received by the reception unit.

[0005] JP2012235300A is concerned with speedily providing reference information for specifying a person performing leaving or carrying-away in a leaving or carrying-away detection system.A leaving or carrying-away detection system includes a camera imaging a monitoring area; a data storage medium storing an imaged image; a leaving / carrying-away determining section; an occurrence time operating section operating occurrence time of leaving or carrying-away; an image acquirement processing section acquiring the imaged image of the monitored area imaged at occurrence time from the data storage medium ; a peripheral person extracting section extracting an image of a peripheral person within a prescribed distance range from a foreign matter being an object of leaving or carrying-away from the acquired imaged image; and a log data creating section creating leaving log data or carrying-away log data, which include the image of the foreign matter being the object of leaving or carrying-away and the image of the peripheral person.

[0006] US2010271478A1 discloses an apparatus includes a recording unit configured to record a video image, a generation unit configured to generate, when a first type of event in which a state of an object changes is detected from the video image, an index associating the first type and a second type of event related to the first type in the video image, and a playback unit configured to play back, when the first type is specified, a video image concerning the second type corresponding to the specified first type based on the index. Citation list of patent literature PTL 1: Unexamined Japanese patent specification no. 1-245395 PTL 2: Unexamined Japanese patent no. 2012-235300 PTL 3: Unexamined Japanese patent specification no. 2011-49646 BRIEF SUMMARY OF THE INVENTION

[0007] A device with the features of claim 1 and a method with the features of claim 7 are claimed.

[0008] The device for monitoring unattended objects of the present disclosure is a device for monitoring unattended objects that detects an unattended object left in a monitoring area based on a captured image of the monitoring area mapped by an imaging device, and comprises an image retrieval unit that retrieves a captured image of the monitoring area mapped by the imaging device, an object tracking unit that detects an appearing object that appears in the monitoring area from the captured image and tracks it between the captured images for each appearing object, and an unattended object detection unit that detects an appearing object that is not moved beyond a predetermined time.Based on a tracking result for each appearing object detected by the object tracking unit as an unattended object, a progress information generator that generates progress information of the unattended object after its appearance in the monitored area based on the tracking result for each appearing object by the object tracking unit, and a message image generator that generates a message image by overlaying the progress information onto the captured image.

[0009] According to the present disclosure, it is possible to provide a user with progress information about the unattended object after its appearance in the monitored area. In this way, it is also possible for the user to record changes in the state of the unattended object or details of its appearance until the alarm is triggered. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 is a schematic representation of a configuration of a system for monitoring unattended objects according to a first embodiment. Fig. 2A is a schematic representation showing an example in which an appearing object (a person with a bag) that appears in a surveillance area is detected from a recorded image. Fig. 2B is a schematic representation showing an example in which an object (bag) that reappears in the monitored area is detected from the recorded image. Fig. 3A is a schematic representation showing an example of a message image if the progress information is an elapsed time since the unattended object appeared in the monitoring area. Fig. 3B is a schematic representation showing another example of the message image if the progress information is an elapsed time since the unattended object appeared in the monitoring area. Fig. 4A is a schematic representation showing an example of the message image if the progress information consists of time-sequence images of the unattended object since the time the unattended object appears in the monitoring area. Fig. 4B is a schematic representation showing an example of time sequence images if the progress information consists of time sequence images of the unattended object since the time the unattended object appears in the monitored area. Fig. 5A is a schematic representation where the progress information is the elapsed time since the unattended object appears in the monitoring area, a schematic representation showing an example of the message image representing the elapsed time by a color change of a frame image, and a schematic representation showing an example of the message image where the color of the frame image is blue. Fig. 5B is a schematic representation where the progress information is the elapsed time since the unattended object appears in the monitoring area, a schematic representation showing an example of the message image representing the elapsed time by a color change of a frame image, and a schematic representation showing an example of the message image where the color of the frame image is yellow. Fig. 5C is a schematic representation where the progress information is the elapsed time since the unattended object appears in the monitoring area, a schematic representation showing an example of the message image representing the elapsed time by a color change of a frame image, and a schematic representation showing an example of the message image where the color of the frame image is red. Fig. Figure 6 is a flowchart showing the processing sequence in a device for monitoring unattended objects according to the first embodiment. Fig. Figure 7 is a schematic representation of a configuration of a system for monitoring unattended objects according to a second embodiment. Fig. Figure 8 is a graph that represents a feature set level of a local area on images taken in a time sequence. Fig. Figure 9A is a schematic representation showing an example of image layers of a background object and an emerging object A. Fig. 9B is a graphical representation showing an example of an image layer of an appearing object B. Fig. 10A is a graphic representation showing an example of a background image. Fig. Figure 10B is a schematic representation showing an image, with each image layer placed on top of the background image. Fig. 10C is a view that displays a message image, enhanced by showing the progress information and the frame image within the image. Fig. 10B is generated. Fig. Figure 11 is a flowchart showing the processing sequence in a device for monitoring unattended objects according to the second embodiment. DESCRIPTION OF EXECUTION FORMS

[0010] Before describing the embodiments, the problems of the prior art are briefly outlined. Incidentally, the prior art of PTL 1 had a problem: it was not possible to provide a user with progress information about an unattended object after its appearance in a monitored area. Providing the user with progress information about the unattended object after its appearance in the monitored area is beneficial, as it enables the user to track changes in the unattended object's state and its appearance until an alarm is triggered.

[0011] Furthermore, with the state-of-the-art techniques of PTL2 and PTL3, it is possible to specify the owner of an unattended object, but there was a problem that the user could not see the unattended object if the unattended object was hidden behind another appearing object, such as the user or another unattended object.

[0012] Even if the unattended object is hidden behind another appearing object, it is advantageous if it is possible to display the unattended object in the image, as the user can check the state of the unattended object.

[0013] The present invention was developed in view of these problems of the prior art, and a first object of the present disclosure is to provide a device for monitoring unattended objects which is capable of providing a user with progress information of an unattended object after its appearance in a monitoring area, a system equipped with the same for monitoring unattended objects and a method for monitoring unattended objects.Furthermore, a second objective of the present disclosure is to provide a device for monitoring unattended objects that is capable of displaying an unattended object in an image, a system equipped with the same for monitoring unattended objects, and a method for monitoring unattended objects even if the unattended object is hidden behind another appearing object.

[0014] A first disclosure made to solve the aforementioned problem is a device for monitoring unattended objects, which detects an unattended object left in a monitoring area based on a captured image of the monitoring area mapped by an imaging device, and comprises an image retrieval unit that retrieves a captured image of the monitoring area mapped by the imaging device, an object tracking unit that detects an appearing object that appears in the monitoring area from the captured image and tracks it between captured images for each appearing object, and an unattended object detection unit that detects an appearing object that is not moved beyond a predetermined time.based on a tracking result for each appearing object detected by the object tracking unit as the unattended object, a progress information generator that generates progress information of the unattended object after its appearance in the monitored area based on the tracking result for each appearing object by the object tracking unit, and a message image generator that generates a message image by overlaying the progress information onto the captured image.

[0015] According to the device for monitoring unattended objects of the first disclosure, it is possible to provide the user with the progress information of the unattended object after its appearance in the monitoring area.

[0016] Furthermore, the progress information in a second revelation includes the elapsed time since a predetermined time after the appearance in the monitoring area in the first revelation.

[0017] According to the device for monitoring unattended objects of the second disclosure, it is possible to visually represent the elapsed time since a predetermined time after the appearance in the monitoring area to a user in a visually easily understandable way.

[0018] Furthermore, the progress information in a third revelation includes a time-sequence picture of an unmonitored object since a predetermined time after its appearance in the monitoring area in the first or second revelation.

[0019] According to the device for monitoring unattended objects of the third disclosure, it is possible to indicate to the user in a visually easily understandable way the change in the state of the unattended object since the predetermined time after its appearance in the monitoring area.

[0020] Furthermore, in a fourth revelation, if an appearing object is detected, a frame image surrounding the appearing object is superimposed on the message image in one of the revelations one to three.

[0021] According to the device for monitoring unattended objects of the fourth disclosure, if an appearing object is detected, the appearing object can be displayed to the user in a visually easily understandable manner.

[0022] Furthermore, in a fifth revelation, the message image generator changes the color of the frame image according to the time that has elapsed since a predetermined time after the appearance of the unattended object in the surveillance area in the fourth revelation.

[0023] According to the device for monitoring unattended objects of the fifth disclosure, it is possible to visually represent to a user, in a visually easily understandable way, the elapsed time since a predetermined time after the appearance of the unattended object in the monitoring area.

[0024] Furthermore, a sixth revelation includes an image layer generator that creates an image layer containing an image of the appearing object for each appearing object, and a background image generator that creates a background image by removing appearing objects from a captured image, wherein a message image generator places each image layer onto the background image in order of detecting appearing images contained in each image layer, if a plurality of appearing objects is detected in any of the revelations one to five.

[0025] According to the device for monitoring unattended objects as described in the sixth disclosure, it is possible to display the unattended object in the image, even if the unattended object is hidden behind another appearing object.

[0026] Furthermore, a seventh disclosure is a system for monitoring unattended objects, comprising a device for monitoring unattended objects according to any one of disclosures one to six, an imaging device for mapping the monitoring area, and a display device that displays a message image generated by the device for monitoring unattended objects.

[0027] Furthermore, an eighth disclosure is a method for monitoring unattended objects for detecting an unattended object left in a monitoring area, based on a captured image of the monitoring area mapped by an imaging device, which involves retrieving a captured image of the monitoring area mapped by the imaging device, detecting an appearing object that appears in the monitoring area from the captured image and tracking between the captured images for each appearing object, and detecting the appearing object that is not displaced beyond a predetermined time, based on a tracking result for each appearing object.This includes generating progress information of the unmonitored object after its appearance in the monitored area, based on the tracking result for each appearing object, and generating a notification image by overlaying the progress information onto the captured image.

[0028] The following describes embodiments with reference to drawings. (First embodiment)

[0029] First, a system for monitoring unattended objects 1 according to a first embodiment is described with reference to Fig. This system for monitoring unattended objects 1 is described in sections 1 to 6. It is a system for detecting an unattended object left in the monitored area based on the captured image of the monitored area, which is mapped by the imaging device 2, and triggering an alarm if an unattended object is detected. It can be applied to monitoring suspicious unattended objects in a space, such as train stations and airports, used by a large number of unspecified persons. The following describes a case in which the system for monitoring unattended objects 1, as disclosed here, is applied to monitoring a suspicious unattended object in a train station building. Examples of unattended objects include items (for example, paper bags and briefcases), persons, and the like.An unattended object is not specifically limited and can be various other objects, such as a wheelchair, an electric car, and the like.

[0030] Fig. Figure 1 is a schematic representation of a configuration of a system for monitoring unattended objects 1 according to the first embodiment. As in Fig. As shown in Figure 1, the system for monitoring unattended objects 1 is configured to include an imaging device 2, an unattended object monitoring device 3, and a display device 4. The display device 4 serves as a signaling device for reporting an alarm to an observer. The unattended object monitoring device 3 is connected to the imaging device 2 and the display device 4 via wired or wireless communication, such as a local area network (LAN).The connection form between the device for monitoring unattended objects 3 and the imaging device 2 and the display device 4 is not specifically restricted and can, for example, be a connection via a network, such as the Internet, or the devices can be connected to each other by a communication cable using a universal interface (for example, a USB (Universal Serial Bus) interface).

[0031] The imaging device 2 is a general imaging device, such as a CCD camera or the like, installed on a wall, ceiling, or the like, and images a predetermined monitoring area sequentially as a still image or a moving image. The imaging device 2 is not specifically limited with respect to its form, function, arrangement, quantity, and the like, as long as a monitoring area can be imaged sequentially, and various modifications are possible. For example, the imaging device 2 can be a panoramic camera installed on a wall or the like and capable of imaging 180 degrees with one unit, or an omnidirectional camera installed on a ceiling or the like and capable of imaging 360 degrees with one unit.An image of the monitored area, captured sequentially by the imaging device 2, is entered into the device for monitoring unattended objects 3.

[0032] The display device 4 is a general display device, such as a monitor (a display), which is installed in a monitoring room or the like, in which the observer performs monitoring work, and displays a message image (see Fig. 4), which is output by the device for monitoring unattended objects 3. Details of the message image are described later. The display device 4 is not specifically limited with regard to its form, function, arrangement, quantity, and the like, as long as the message image can be displayed, and various modifications are possible. For example, the display device 4 can be a display screen of a mobile device carried by the observer when leaving the monitored area for a walkthrough or the like.

[0033] The device for monitoring unattended objects 3 is a general computer device and comprises a central processing unit (CPU) that fully performs various types of information processing, control of peripheral devices, or the like based on a predetermined control program; a random-access memory (RAM) that acts as a working area of ​​the CPU; a read-only memory (ROM) that stores control programs executed by the CPU or data; a network interface that performs communication processing over a network; and the like, which is not shown in detail.Various functions (for example, the generation of progress information) of the unattended object monitoring device 3, which will be described in detail later, can be implemented by the CPU, which executes a predetermined control program (for example, a progress information generation program). The unattended object monitoring device 3 is not limited to a computer device; it is also possible to use another information processing device (such as a server, etc.) capable of performing the same functions. Furthermore, at least some of the functions of the unattended object monitoring device 3 can be replaced by other known hardware processing.

[0034] The unattended object monitoring device 3 issues an alarm via the display device 4 when a predetermined time has elapsed since the detection of the unattended object. Specifically, an alarm is issued by displaying information indicating that an alarm has been triggered on the message screen output to the display device 4. For example, when an unattended object is detected, the unattended object is surrounded by a frame image, and when a predetermined time has elapsed since the detection of the unattended object, the color of the frame image can change to a prominent color, or an alarm can be issued by causing the frame image to flash. Furthermore, the unattended object monitoring device 3 can control an alarm triggering device (not shown) to issue an alarm.In addition to a general alarm triggering device, such as an alarm buzzer, an alarm lamp or the like, a sound generation device can be used as the alarm triggering device, which makes a predetermined alarm message sound by speech.

[0035] As in Fig. As shown in Figure 1, the device for monitoring unattended objects 3 comprises an image retrieval unit 11, an image processor 12, and a message image generator 13, and the image processor 12 comprises an object tracking unit 21, an unattended object detection unit 22, and a progress information generator 23. Each unit is controlled by a controller (not shown).

[0036] The image retrieval unit 11 is connected to the imaging device 2 and retrieves a captured image of the monitored area from the imaging device 2. The image retrieval unit 11 is connected to the image processor 12 and the message image generator 13, and the captured image retrieved from the imaging device 2 by the image retrieval unit 11 is input into the image processor 12 and the message image generator 13.

[0037] The image processor 12 retrieves a captured image from the image retrieval unit 11. If the captured image is a still image taken at a predetermined time interval, it is retrieved as still image data. If the captured image is a moving image, it is retrieved as still image data (image frame data) extracted from the moving image at a predetermined time interval. The captured image (still image data) retrieved by the image processor 12 from the image retrieval unit 11 is processed by the object tracking unit 21, the unattended object detection unit 22, and the progress information generator 23.

[0038] The object tracking unit 21 detects an emerging object appearing in the monitored area from the captured image and tracks it between captured images after detection for each emerging object. The emerging object can be a person, a moving object such as a wheelchair, an electric vehicle, or the like, driven by a person, or any other manned or unmanned moving object, or the like. The detection of the emerging object is performed by comparing the captured background image with the captured image in advance or by comparing two adjacent images captured in sequential order.Specifically, a dissimilarity value (difference or correlation value) of pixel values ​​(for example, brightness value) is obtained between the captured images to be compared. Pixels whose obtained dissimilarity exceeds a predetermined threshold are extracted, and an area in which the extracted pixels are clustered in close proximity is detected as an emerging object. When an emerging object is detected from the captured image, a unique identification code is assigned to each detected emerging image, and after the emerging object is detected, tracking is performed between the captured images. If a new emerging object appears near the already detected emerging image, it is determined that the new emerging object has been separated from the already detected emerging object.In this case, the identification code of the already detected appearing object is not changed, and a newly appearing object is assigned a new identification code and managed accordingly. When an appearing object is detected by the object tracking unit 21, the detection result is entered into the message image generator 13.

[0039] Fig. Figure 2A is a graphic representation showing an example where an object 31 appearing in the monitored area is detected from the captured image by the object tracking unit 21. In the example of Fig. 2A, the appearing object 31 is a person 33 with a paper bag 32. At this time, the paper bag 32 and the person 33 are not individually identified, but detected as one appearing object 31. Fig. Figure 2B is a graphic representation showing an example where a newly appearing object 34 in the monitored area is detected from the captured image by the object tracking unit 21. In the example of Fig. 2B, the paper bag 32, which is separated from person 33, is detected as the newly appearing object 34. Subsequently, the paper bag 32 is tracked as the newly appearing object 34. Furthermore, the appearing object 31 (i.e., person 33), which is separated from the paper bag 32, is tracked in its current state as an appearing object 31.

[0040] The Unattended Object Detection Unit 22 detects an unattended object based on the tracking result for each appearing object by the Object Tracking Unit 21. Specifically, for each appearing object detected by the Object Tracking Unit 21, a time shift is obtained in the captured image of the appearing object, and an appearing object that does not move beyond a predetermined time, that is, remains stationary beyond a predetermined time, is detected as an unattended object. When an unattended object is detected, a unique identification code is assigned to each detected unattended image, and tracking is performed between the captured images after the detection of the unattended object.When an unattended object is detected by the unit for detecting unattended objects 22, the detection result is entered into the message image generator 13.

[0041] The progress information generator 23 generates progress information of the unattended object after the time at which the unattended object appears in the monitoring area (hereinafter referred to as the "time of object appearance"), based on the tracking result for each appearing object by the object tracking unit 21. The progress information can be described as the elapsed time since a predetermined time after the time of object appearance, the elapsed time since a predetermined time since the detection of the unattended object, and a time sequence image of the unattended object after a predetermined time after the time of object appearance.The progress information is not limited to this and can include, for example, various other pieces of information or images, such as an image and time of the unattended object's appearance in the monitored area, an image and time of the unattended object's detection, and the like. The progress information generated by the progress information generator 23 is entered into the message image generator 13.

[0042] The message image generator 13 retrieves a captured image from the image retrieval unit 11. If the progress information of the unattended object is input by the progress information generator 23, the message image generator 13 creates a message image by overlaying the progress information of the unattended object onto the captured image. Specifically, the message image generator 13 creates an image that includes the progress information of the unattended object, based on the progress information input by the progress information generator 23, and overlays this image onto the captured image to create a message image. If the progress information input by the progress generator 23 is an image that includes progress information, a message image is created by overlaying this image onto the captured image.

[0043] If, in addition, an emerging object is detected by the object tracking unit 21, the message image generator 13 further overlays a frame image surrounding the emerging object onto the message image. If no unattended object is detected by the unit for detecting unattended objects 22, the message image generator 13 overlays a frame image surrounding the unattended object onto the captured image and generates a message image. The message image generated by the message image generator 13 is output to the display device 4. If no unattended object is detected by the object tracking unit 21, the message image generator 13 outputs the captured image, retrieved by the image retrieval unit 11, in its current state to the display device 4.

[0044] Fig. 3A and Fig. Figure 3B are schematic representations showing an example of the message image when the progress information is the elapsed time since the object appeared. In this case, the message image generator 13 creates an image (hereinafter referred to as the "time counter image") indicating the elapsed time since the object appeared and overlays the time counter image onto the captured image to create a message image. The message image generator 13 also overlays the frame image surrounding the unattended object onto the message image.

[0045] In the example of Fig. 3A and Fig. 3B, a paper bag 41 is detected as an unattended object, and a timer image 42 is superimposed in a position on the top of the paper bag 41 in the captured image. Additionally, the frame image 43 surrounding the paper bag 41 is superimposed on the message image. In the example of Fig. 3A is the time counter image 42 of "3:15", which indicates that the elapsed time since the moment the paper bag 41, which is an unattended object, appeared in the monitored area is 3 minutes 15 seconds, as shown in the recorded image. In the example of Fig. 3B is also the time counter image 42 of “8:59”, which indicates that the elapsed time since the time at which the paper bag 41, which is an unattended object, appeared in the surveillance area is 8 minutes 59 seconds, shown in the recorded image.

[0046] As already mentioned, it is possible to display the elapsed time since the appearance of paper bag 41, which is an unattended object, in the monitored area in a visually easily understandable way for the user if the time counter image 42 is superimposed on the captured image. It is also possible to display the unattended object (paper bag 41) to the user in a visually easily understandable way by surrounding it with a frame image 43. If another appearing object or another unattended object besides paper bag 41 is detected in the captured image, it is possible to display these objects to the user in a visually easily understandable way if they are surrounded by a frame image. In the example of Fig. 3A and Fig. 3B described the case where the progress information represents the elapsed time since the object's appearance, but the progress information can also represent the elapsed time since a predetermined point in time after the object's appearance. Furthermore, the progress information can represent the elapsed time since the unmonitored object is detected (or a predetermined point in time thereafter).

[0047] Fig. 4A and Fig. Figure 4B shows a schematic representation illustrating an example of the message image when the progress information is a time-sequence image of an unattended object after the time of its appearance. In this case, the message image generator 13 overlays the time-sequence image of the unattended object, input by the progress information generator 23, onto the captured image and generates a message image. The message image generator 13 also overlays the frame image surrounding the unattended object onto the message image.

[0048] In the example of Fig. 4A A person 51 who has collapsed on the ground is detected as an unattended object, and a time-counting image 52 is superimposed in a position on top of the person 51 in the captured image. Furthermore, a frame image 53 is superimposed, surrounding the person 51. In the example of Fig. 4A is the time counter image 52 of “10:00”, which indicates that the elapsed time since the appearance of person 51, who is an unattended object, is 10 minutes, superimposed in the recorded image.

[0049] This example shows how an alarm is triggered if 10 minutes have passed since the object first appeared and an unattended object is detected. That is, the time in Fig. 4A is the time at which an alarm is triggered. When an alarm is triggered, as in Fig. As shown in Figure 4B, the observer operating the device for monitoring unattended objects 3 can display a time-sequence image 54 of the person 51, who is an unattended object, in the frame image 53 via an input device (not shown), such as a keyboard or a mouse. The time-sequence image 54 is displayed as a still image (frame-wise image) or as a moving image at predetermined time intervals. The elapsed time of the time-sequence image 54 is also displayed in the time-sequence image 54. In the example of Fig. In image 4B, person 51 is standing upright at 4 minutes and 50 seconds, and in the images after 5 minutes 00 seconds, person 51 has collapsed.

[0050] As already mentioned, it is possible to display the change in the state of the unattended object (person 51) after the time of the object's appearance to the user in a visually easily understandable way if the time-sequence image 54 of person 51, who is an unattended object, is superimposed on the captured image. In the example of Fig. 4B states that person 51 was standing upright until 4 minutes and 50 seconds, but collapsed at 5 minutes 00 seconds. Consequently, the observer can record the state before person 51 collapsed and the time at which person 51 collapsed. In the examples of Fig. 4A and Fig. Section 4B described the case where the progress information is the time-sequence image of the unattended object since the time of its appearance, but the progress information can also be the elapsed time since a predetermined time after the time of the object's appearance. Furthermore, the progress information can be the elapsed time since the time the unattended object is detected (or a predetermined time thereafter).

[0051] The display position of the time sequence image 54 of person 51 is not limited to within the frame image 53, but can, for example, be a different position in the message image or on another screen. Furthermore, the display position or the display method of the elapsed time of the time sequence image 54 can vary. Additionally, the time at which the observer displays the time sequence image 54 can be the time before an alarm is issued. For example, the time sequence image 54 can be displayed at the time of Fig. 4B is displayed, at which point person 51 collapsed. Since it is possible in this way to detect the change in person 51's condition earlier than at the time an alarm is triggered, it is possible to react more quickly.

[0052] Fig. Figures 5A to 5C are schematic diagrams showing an example of the message image, where the progress information is the elapsed time since the object appeared, indicated by a change in the frame image color. In this case, the message image generator 13 overlays a frame image surrounding the unattended object onto the captured image and changes the frame image color according to the elapsed time. In the example of Fig. From 5A to 5C, the color of the frame image changes in the order of blue, yellow and red according to the elapsed time.

[0053] In the example of Fig. In steps 5A to 5C, a briefcase 61 is detected as an unattended object, and a framed image 62 surrounding the briefcase 61 is superimposed on the captured image. Fig. 5A to 5C will be the color of frame image 62 in the order from blue (see Fig. 5A), Yellow (see Fig. 5B) and red (see Fig. 5C) changed as time passes. Fig. 5A is a message image covering a period from the time the briefcase 61, which is an unattended object, appears, until a predetermined time has elapsed. Fig. 5B is a message image from a period of time from Fig. 5A until a predetermined time has elapsed, and Fig. 5C is a message image from a period of time from Fig. 5B until a predetermined time has elapsed. In this example, an alarm is set to sound at the time of Fig. 5C is output, and the alarm is triggered by changing the color of frame image 62 to red. The time interval for changing the color of frame image 62 can be set as desired. Furthermore, the color of frame image 62 that should change according to the elapsed time can also be set as desired. Additionally, the shape, brightness, and other characteristics of frame image 62 can be changed or set to flash not only according to the color of frame image 62, but also according to the elapsed time.

[0054] As mentioned previously, it is possible to display the elapsed time since the appearance of briefcase 61, which is an unattended object, in the monitoring area in a visually easily understandable way for the user by changing the color of the frame image according to the elapsed time. In the example of Fig. Section 5 described the case where the progress information is the elapsed time since the object's appearance, but the progress information can also be the elapsed time since a predetermined point in time after the object's appearance. Furthermore, the progress information can be the elapsed time since the unmonitored object is detected (or a predetermined point in time thereafter).

[0055] Next, the processing sequence in the device for monitoring unattended objects 3 of the system for monitoring unattended objects 1 according to the first embodiment, which is described in Fig. 1 is shown, with reference to the flowchart of Fig. 6 described.

[0056] First, the image retrieval unit 11 retrieves a captured image from the imaging device 2 and inputs the captured image into the image processor 12 and the message image generator 13 (step ST101). Subsequently, the object tracking unit 21 detects an appearing object (see Fig. 2A and Fig. 2B), which appears in the monitored area, from the captured image (step ST102). If an appearing object is detected (step ST102: Yes), the processing proceeds to step ST103; if no appearing object is detected (step ST102: No), the processing returns to step ST101.

[0057] In step ST103, the object tracking unit 21 tracks between the captured images after detecting an emerging object for each detected emerging object. In the next step, ST104, the unattended object detection unit 22 detects an emerging object that remains stationary for a predetermined time as an unattended object based on the tracking result for each emerging object by the object tracking unit 21. If an unattended object is detected (step ST104: Yes), processing proceeds to step ST105, and if no unattended object is detected (step ST104: No), processing returns to step ST101.

[0058] In step ST105, the progress information generator 23 produces progress information for the unattended object after the time of object appearance, based on the tracking result for each appearing object by the object tracking unit 21. As mentioned earlier, the progress information can include the elapsed time since a predetermined time after the time of object appearance, the elapsed time since a predetermined time since the detection of the unattended object, and a time sequence image of the unattended object after a predetermined time after the time of object appearance.

[0059] Next, in step ST106, the message image generator 13 overlays the progress information generated by the progress information generator 23 onto the captured image and creates a message image (see Fig. 3 to 5). Furthermore, the message image generator 13 overlays the frame image surrounding the appearing object onto the message image. The generated message image is output to the display device 4.

[0060] If a predetermined time has elapsed since the detection of the unattended object, then the device for monitoring unattended objects 3 in step ST107 issues an alarm via the display device 4 or the alarm triggering device, not shown.

[0061] As already mentioned, according to the first embodiment, it is possible to provide the user with progress information about the unattended object after its appearance in the monitored area, since a notification image can be generated by displaying the progress information of the unattended object after the time of its appearance in the captured image. Furthermore, it is possible to display the progress information of the unattended object to the user in an intuitive and easily understandable manner. (Second embodiment)

[0062] Next, a system for monitoring unattended objects 1 according to the second embodiment is described with reference to Fig. 7 to 11. The device for monitoring unattended objects 3 according to the second embodiment differs from the device for monitoring unattended objects 3 according to the first embodiment, which is described in Fig. Figure 1 shows that the device for monitoring unattended objects 3 further comprises an image layer generator 24 and a background image generator 25. Since the other points are the same as in the first embodiment described above, the same reference numerals are used and their description is omitted.

[0063] When multiple objects are detected appearing in the monitored area, the image layer generator 24 creates an image layer containing the image of each detected object. Specifically, multiple objects are identified based on the feature set level of the local area in the captured image, and an image layer is created for each identified object. Furthermore, if the object appearing is unmonitored and a background object is present behind it, an image layer containing the background object is also created. Creating the background object image layer is not strictly necessary.

[0064] Fig. Figure 8 is a graph representing a feature set level of a local area in images (frames) recorded sequentially over time. In the example of Fig. 8. A background object, an appearing object A, and an appearing object B are identified according to the feature set level. The image layer generator 24 creates an image layer for each object identified in this way. Each image layer is created by cutting out each object from the background image.

[0065] Fig. 9A is an image layer that is based on the feature set level at time t1 in the graph of Fig. 8 is generated. In the example of Fig. 9A is the appearing object A in Fig. 8 an unattended object, and a briefcase 71 is detected as an unattended object (appearing object A). Furthermore, the briefcase 71 is positioned on a bench 72, and the bench 72 is detected as a background image of the briefcase 71, which is an unattended object. As shown on the right side of Fig. As shown in Figure 9A, the image layer generator 24 creates an image layer L1, which includes an image of the bench 72, which is a background object, and an image layer L2, which includes an image of the briefcase 71, which is an unattended object (appearing object A).

[0066] Fig. 9B is an image layer that is based on the feature set level at time t2 in the graph of Fig. 8 is generated. In the example of Fig. 9B becomes a person 73 as an appearing object B in Fig. 8 detected. This person 73 moves from left to right in the image and at time t2 is in front of briefcase 71 and bench 72. That is, at time t2, briefcase 71 and bench 72 are hidden behind person 73. As on the right side of Fig. As shown in 9B, the image layer generator 24 creates an image layer L3 that includes the image of person 73, who is the appearing object B.

[0067] Background image generator 25 creates a background image by removing the objects that appear in the captured image. Fig. Figure 10A is a graphical representation showing the background image generated by the background image generator 25. Specifically, the image area of ​​the appearing object is used as a foreground image based on the difference between the captured image and the time at which the first appearing object is detected (in the example of Fig. 9. The foreground image is extracted from the captured image at time t1 (when the appearing object A is detected) and the captured image at an earlier time. Then, a background image is created by removing the foreground image from the captured image at the time the first appearing object is detected. Alternatively, an image of the monitored area captured in advance, in a situation where no appearing object is present, can be used as the background image.

[0068] As in Fig. As shown in Figure 10B, the message image generator 13 places the image layers L1 to L3 generated by the image layer generator 24 onto the background image (see Fig. 10A) in the detection sequence from the back side. The image layers L1 and L2 are positioned so that at least some of the objects contained in these image layers are visible. In the example of Fig. In 10B, image layers L1 to L3 are superimposed with their positions shifted from one another by a predetermined distance in the left-to-right direction. Consequently, the briefcase 71 and the bench 72, which are hidden behind person 73, can be displayed in the image. The direction and distance by which the positions of image layers L1 to L3 are shifted can be arbitrarily defined. If at least some of the objects contained in image layers L1 to L2 are visible even when the image layers are superimposed as they are, the positions of image layers L1 to L3 do not need to be shifted from one another. Furthermore, as long as the observer can recognize that image layers L1 to L3 are superimposed, they can be positioned as they are.In this case, for example, a means for changing the order of stacking the image layers, such as tabs (not shown) added to the image frames, any other control buttons (not shown), or the like, is provided, and the image layer can be displayed by pressing the means on the back.

[0069] As also in Fig. As shown in Figure 10C, the message image generator 13 creates a message image by further displaying the progress information and the frame image, which are generated by the progress information generator 23, in the image (see Figure 10C). Fig. 10B), where each image layer is placed on top of the background image. In the example of Fig. 10C, the progress information, consists of the time counter image 74 from "6:18" and the frame images 75 to 77, which surround the person 73, the briefcase 71, and the bench 72, respectively. The message image generated by the message image generator 13 is output to the display device 4.

[0070] Next, the processing sequence in the device for monitoring unattended objects 3 of the system for monitoring unattended objects 1 according to the second embodiment, which is described in Fig. 6 is shown, with reference to the flowchart of Fig. 11 described.

[0071] The processing in the device for monitoring unattended objects 3 according to the second embodiment differs from the first embodiment in that steps ST201 to ST204 are added between steps ST105 and ST106 of the processing in the device for monitoring unattended objects 3 according to the first embodiment, which are described in the flowchart of Fig. Section 6 was described. The following primarily describes the added steps ST201 to ST204.

[0072] If an emerging object is detected in the monitoring area in step ST105, step ST201, following step ST105, determines whether multiple emerging objects have been detected in the monitoring area. If multiple emerging objects are detected (step ST201: Yes), processing continues to step ST202; if no multiple emerging objects are detected (step ST201: No), processing proceeds to step ST106.

[0073] In step STST202, the image layer generator 24 creates an image layer that includes the image of the appearing object for each appearing object. If the appearing object is also an unattended object and a background object is present behind it, an image layer is also created that includes the image of the background object (see Fig. 9A and Fig. 9B). Subsequently, in step STST203, the image layer generator 24 creates a background image (see Fig. 10A).

[0074] In the next step ST204, the message image generator 13 places each image layer generated by the image layer generator 24 onto the background image generated by the background image generator 25, so that at least some of the objects contained in each image layer are visible (see Fig. 10B). Then, in step ST106 after step ST204, the message image generator 13 further incorporates the progress information and the frame image generated by the progress information generator 23 into the image, with each image layer superimposed on the background image to create a message image (see Fig. 10C) to generate, and outputs the image to the display device 4.

[0075] As already mentioned, according to the second embodiment, it is possible to display the unattended object in the image even if the unattended object is hidden behind another object, since every appearing object and every background object contained in the captured image can be displayed in the image if multiple appearing objects are detected in the monitored area. Furthermore, the background object located behind the unattended object can also be displayed in the image.

[0076] Although the present disclosure is described on the basis of specific embodiments, these embodiments are merely examples, and the present disclosure is not limited by these embodiments. The components of the device for monitoring unattended objects according to the present disclosure, the system equipped with it for monitoring unattended objects, and the method for monitoring unattended objects described in the preceding embodiment are not all necessarily required, but can be selected appropriately without deviating from the scope of protection of the present disclosure. COMMERCIAL USABILITY

[0077] A device for monitoring unattended objects, a system equipped with the same for monitoring unattended objects, and a method for monitoring unattended objects can be used as a device for monitoring unattended objects that is capable of providing a user with progress information of an unattended object after its appearance in a monitoring area, a system equipped with the same for monitoring unattended objects, and a method for monitoring unattended objects, and the like. REFERENCE MARKS OF THE DRAWINGS 1 SYSTEM FOR MONITORING UNATTENDED OBJECTS 2. IMAGE DEVICE 3 DEVICE FOR MONITORING UNATTENDED OBJECTS 4 INDICATOR DEVICE (REPORTING DEVICE) 11 IMAGE RETRIEVAL UNIT 12 IMAGE PROCESSOR 13 MESSAGE IMAGE GENERATOR 21 OBJECT TRACKING UNIT 22 UNIT FOR DETECTING UNATTENDED OBJECTS 23 PROGRESS INFORMATION GENERATOR 24-layer image generator 25 BACKGROUND IMAGE GENERATOR

Claims

[1] An unattended object monitoring device that detects an unattended object left in a monitoring area based on a captured image of the monitoring area imaged by an imaging device, the device comprising: a processor; and a memory that stores an instruction, wherein the apparatus, upon execution of the instruction stored in the memory by the processor, further comprises as a configuration: an image retrieval unit that retrieves a captured image of a surveillance area imaged by the imaging device; an object tracking unit that detects an appearing object appearing in the surveillance area from the captured image and tracked between the captured images for each appearing object; an unattended object detection unit which detects the appearing object which is not moved beyond a predetermined time, based on a tracking result for each appearing object detected by the object tracking unit as the unattended object; a progress information generator that generates progress information of the unattended object after the appearance in the monitoring area based on the tracking result for each appearing object by the object tracking unit; and a message image generator that creates a message image by overlaying the progress information in the captured image characterized by , that the device further comprises: an image layer generator that generates an image layer comprising an image of the appearing object for each appearing object if a plurality of the appearing objects are detected, and a background image generator that creates a background image by removing the appearing objects from the captured image, wherein the message image generator generates each image layer in the order of detection of the appearing objects contained in each image layer, placed on the background image.. [2] The unattended object monitoring device according to claim 1, wherein the progress information includes an elapsed time since a predetermined time after the appearance in the monitoring area. [3] An unattended object monitoring device according to claim 1 or 2, wherein the progress information includes a time-series image of the unattended object after a predetermined time after the appearance in the monitoring area. [4] An unattended object monitoring device according to any one of claims 1 to 3, wherein, if the appearing object is detected, the notification image generator further superimposes a frame image surrounding the appearing object on the notification image. [5] An unattended object monitoring apparatus according to claim 4, wherein the message image generator changes the color of the frame image according to an elapsed time since a predetermined time after the appearance of the unattended object in the monitoring area. [6] System for monitoring unattended objects, comprising: the device for monitoring unattended objects according to one of claims 1 to 5; an imaging device for imaging a surveillance area; and a display device that displays a message image generated by the unattended object monitoring device. [7] A method for monitoring unattended objects for detecting an unattended object left in a monitoring area based on a captured image of the monitoring area imaged by an imaging device, the method comprising: Retrieving a captured image of the surveillance area imaged by the imaging device; Detecting an appearing object appearing in the surveillance area from the captured image and tracking between the captured images for each appearing object; Detecting the appearing object that is not moved for a predetermined time based on a tracking result for each appearing object as the unattended object; Generating progress information of the unattended object after the appearance in the monitoring area based on the tracking result for each appearing object; and Generating a message image by displaying the progress information in the captured image, characterized by , that for each appearing object an image layer is generated which comprises an image of the appearing object if a plurality of the appearing objects are detected, and that a background image is created by removing the appearing objects from the recorded image, each image layer being superimposed on the background image in the order of detection of the appearing objects contained in the respective image layer.

Citation Information

Patent Citations

  • Leaving monitoring device

    JP2011049646A

  • Information transmission apparatus and information transmission method

    JP2011055270A

  • Leaving or carrying-away detection system and method for generating leaving or carrying-away detection record

    JP2012235300A

  • Apparatus for playing back recorded video images related to event,and method thereof

    US20100271478A1

  • JP002011055270A