Monitoring device, monitoring method, and monitoring program

Abstract

The present invention includes: a detection means for detecting objects and people in frame images included in a moving image; a stationary object detection means for detecting objects that are stationary among the detected objects as stationary objects; an owner identification means for extracting the people detected at the position closest to the stationary objects as owner candidates from each of a plurality of frame images including the objects detected as the stationary objects, and identifying the most frequently extracted people as the owners of the objects; and an unattended object detection means for detecting objects that are unattended, among the objects detected as the stationary objects, as unattended objects.

Description

Monitoring device, monitoring method, and monitoring program

[0001] The present disclosure relates to a monitoring device, a monitoring method, and a monitoring program.

[0002] When an abandoned object such as a lost item is found in a facility, work is carried out to identify its owner. As a related technique, for example, Patent Literature 1 describes a system that identifies the person who left the abandoned object and tracks the movement of that person.

[0003] International Publication No. 2018 / 198373

[0004] However, the technology described in Patent Document 1 leaves room for improvement in the accuracy of identifying the owner of an abandoned object.

[0005] An object of the present disclosure is to provide a monitoring device, a monitoring method, and a monitoring program that can improve the accuracy of identifying the owner of an abandoned object.

[0006] The surveillance device according to the present disclosure includes a detection means for detecting objects and people from frame images included in a moving image, a stationary object detection means for detecting stationary objects from among the detected objects as stationary objects, an owner identification means for extracting as owner candidates the person detected at a position closest to the object from each of a plurality of frame images including the object detected as a stationary object, and identifying the person who has been extracted the most times as the owner of the object, and an abandoned object detection means for detecting abandoned objects from among the objects detected as stationary objects as abandoned objects.

[0007] In the monitoring method according to the present disclosure, a computer detects objects and people from frame images included in a moving image, detects stationary objects among the detected objects as stationary objects, extracts the person detected closest to the object from each of a plurality of frame images including the object detected as a stationary object as a candidate owner, identifies the person who has been extracted the most times as the owner of the object, and detects abandoned objects among the objects detected as stationary objects as abandoned objects.

[0008] The monitoring program according to the present disclosure causes a computer to perform a detection process for detecting objects and people from frame images included in a moving image, a stationary object detection process for detecting stationary objects from among the detected objects as stationary objects, an owner identification process for extracting the person detected at the position closest to the object from each of a plurality of frame images including the object detected as a stationary object as an owner candidate and identifying the person who has been extracted the most times as the owner of the object, and an abandoned object detection process for detecting abandoned objects from among the objects detected as stationary objects as abandoned objects.

[0009] According to the present disclosure, it is possible to increase the accuracy of identifying the owner of an abandoned object.

[0010] FIG. 1 is a block diagram illustrating an example of a functional configuration of a monitoring device; FIG. 2 is an explanatory diagram illustrating an example of an outline of a detection process; FIG. 3 is an explanatory diagram illustrating an example of an outline of IoU; FIG. 4 is an explanatory diagram illustrating an example of an outline of an owner identification process; FIG. 5 is a flowchart illustrating an example of an operation of a monitoring device; FIG. 6 is a block diagram illustrating an example of a configuration of a computer; and FIG. 7 is a block diagram illustrating an example of a main part of a monitoring device.

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In each drawing, identical or corresponding elements are designated by the same reference numerals, and duplicate explanations will be omitted as necessary for clarity. Unless otherwise specified, predetermined values ​​such as predetermined values ​​and threshold values ​​are stored in advance in a storage device accessible from a device that uses the values. Furthermore, unless otherwise specified, the storage unit is composed of one or any number of storage devices.

[0012] 1 is a block diagram illustrating an example of the functional configuration of a monitoring device according to the present disclosure. The monitoring device 100 of this embodiment includes a detection unit 101, a feature extraction unit 102, a stationary object detection unit 103, an owner identification unit 104, an abandoned object detection unit 105, an owner information update unit 106, and a data output unit 107.

[0013] The monitoring device 100 receives, as input, video images of a monitored space captured by an imaging device such as a fixed camera. The detection unit 101 detects objects and people from frame images included in the video. The detection unit 101 detects objects such as suitcases and bags that may be left unattended. The detection unit 101 also detects human faces from the frame images. The detection unit 101 stores the rectangular coordinates of the objects, people, and faces detected from the frame images in a storage unit (not shown). Hereinafter, the area of ​​the frame image specified by the rectangular coordinates will also be simply referred to as a rectangle.

[0014] 2A and 2B are explanatory diagrams illustrating an example of an outline of the detection process performed by the detection unit 101. Fig. 2A shows a frame image included in a moving image, and Fig. 2B shows the result of the detection process performed by the detection unit 101 on the frame image of Fig. 2A.

[0015] Figure 2(a) shows an example of a frame image including a person pulling a suitcase (corresponding to person A pulling suitcase A), a person standing next to a bag (corresponding to person B standing next to bag B), and a person standing next to a bag (corresponding to person C standing next to bag C).

[0016] Figure 2(b) illustrates an example in which object A (corresponding to suitcase A), person A, face A (corresponding to person A's face), object B (corresponding to bag B), person B, face B (corresponding to person B's face), and object C (corresponding to bag C) and person C are detected through the detection process.

[0017] As described above, the detection unit 101 stores the rectangular coordinates of objects, people, and faces detected from frame images in a storage unit (not shown). Therefore, Fig. 2(b) shows the object A rectangle, person A rectangle, face A rectangle, object B rectangle, person B rectangle, face B rectangle, object C rectangle, and person C rectangle detected by the detection unit 101. Note that in the example shown in Fig. 2, the face of person C has not been detected. Therefore, Fig. 2(b) does not show a face rectangle corresponding to the face of person C.

[0018] The feature extraction unit 102 can extract facial features and human figure features from the frame images as feature information of the person detected by the detection unit 101. The facial features are features extracted from a face image of a person (i.e., corresponding to a face rectangle). The human figure features are features extracted from a whole-body image of a person (i.e., corresponding to a human rectangle).

[0019] The facial features and human-shaped features extracted by the feature extraction unit 102 can be used, for example, for same-person determination, which determines whether or not people detected from different frame images are the same person. However, same-person determination using human-shaped features may not function correctly if the person's clothing has changed.

[0020] When a face rectangle detected from a frame image overlaps with a rectangle of a person separately detected, the feature extraction unit 102 associates the face with the person. For example, in the example shown in FIG. 2B, the face A rectangle overlaps the person A rectangle. In this case, the feature extraction unit 102 extracts facial features from the face A rectangle and human-like features from the person A rectangle. The feature extraction unit 102 then associates the extracted facial features and human-like features with each other, assuming that they are extracted from the same person, and stores them in a storage unit (not shown).

[0021] 2B, the face B rectangle overlaps the person B rectangle. In this case, the feature extraction unit 102 extracts facial features from the face B rectangle and human-like features from the person B rectangle. The feature extraction unit 102 then stores the extracted facial features and human-like features in a storage unit (not shown) in association with each other, assuming that they are extracted from the same person.

[0022] 2B, no face rectangle corresponding to the face of person C is detected. In this case, the feature extraction unit 102 extracts human-shaped features from the rectangle of person C. Then, the feature extraction unit 102 determines that there is no corresponding face feature and stores only the extracted human-shaped features in a storage unit (not shown).

[0023] The stationary object detection unit 103 detects a stationary object from among the objects detected from the frame images as a stationary object. Specifically, the stationary object detection unit 103 detects a stationary object from a frame image based on a comparison result between the frame image and a frame image preceding the frame image.

[0024] The stationary object detection unit 103 can apply various methods to determine whether an object to be detected is stationary. An example of a method for determining whether an object is stationary is shown below. However, the operation of the stationary object detection unit 103 is not limited to the method shown below.

[0025] The stationary object detection unit 103 compares, for each object detected in one frame image, the IoU (Intersection over Union) with the detection result of that object in a frame image preceding that frame image.

[0026] FIG. 3 is an explanatory diagram illustrating an example of an overview of IoU. IoU is an index that represents the degree of overlap between two regions. IoU is expressed, for example, as a value between 0.0 and 1.0, with a larger value indicating a greater overlap between the regions. As shown in FIG. 3, IoU is the intersection of two regions divided by the union of the two regions.

[0027] For example, the stationary object detection unit 103 calculates IoU by comparing the area specified by the object A rectangle in the nth frame image with the area specified by the object A rectangle in the (n-1)th frame image. If IoU is equal to or greater than a predetermined threshold, the stationary object detection unit 103 determines that object A is not moving.

[0028] The stationary object detection unit 103 may also determine object movement by comparing a frame image with a predetermined number of frame images preceding the frame image (e.g., the immediately preceding five consecutive frame images). For example, the stationary object detection unit 103 compares the area specified by the object A rectangle in the nth frame image with each of the n-1st to n-5th frame images, and calculates the IoU for each. If all IoUs are equal to or greater than a predetermined threshold, the stationary object detection unit 103 determines that object A is not moving.

[0029] The stationary object detection unit 103 determines an object that has been determined to be stationary a predetermined number of times (e.g., 10 times) in succession as being stationary. That is, the stationary object detection unit 103 detects an object that has been determined to be stationary in each of a predetermined number of consecutive frame images as being stationary.

[0030] As described above, the stationary object detection unit 103 detects an object as a stationary object when the degree of overlap between the area of ​​the object in a frame image and the area of ​​the object in the frame image preceding the frame image in each of a plurality of consecutive frame images is equal to or greater than a predetermined threshold.

[0031] The stationary object detection unit 103 stores information (for example, rectangular coordinates) of an object detected as a stationary object and information indicating that it is a stationary object in association with each other in a storage unit (not shown).

[0032] The stationary object detection unit 103 determines that object A is moving if the IoU between the area specified by the object A rectangle in the nth frame image and the area specified by the object A rectangle in the (n-1)th frame image is less than a predetermined threshold. If the stationary object detection unit 103 determines that an object previously detected as a stationary object is moving, it determines that the object is not a stationary object and does not detect it as a stationary object. In addition, the stationary object detection unit 103 associates information about the object with information indicating that it is not a stationary object and stores the information in a storage unit (not shown).

[0033] The owner identification unit 104 identifies the owner of an object detected as a stationary object from the detected person. The owner identification unit 104 can apply various methods to identify the owner of a stationary object. An example of a method for identifying the owner of a stationary object is shown below. However, the operation of the owner identification unit 104 is not limited to the method shown below.

[0034] The owner identification unit 104 extracts, as owner candidates, the person detected closest to an object detected as a stationary object from each of a plurality of frame images including the object. For example, assume that object A is detected as a stationary object in the nth frame image. In this case, the owner identification unit 104 extracts, as owner candidates, the person closest to object A from each of a predetermined number of frame images before the nth frame image (for example, the n-9th to n-1th frame images). The owner identification unit 104 may also extract owner candidates from the nth frame image using the same criteria.

[0035] The owner identification unit 104 identifies the person who has been extracted most frequently as an owner candidate as the owner. For example, the owner identification unit 104 determines whether the people extracted as owner candidates are the same person using characteristic information (e.g., human feature amounts) of each person. Then, based on the result of the same person determination, the owner identification unit 104 identifies the person who has been extracted most frequently as an owner candidate as the owner.

[0036] The owner identification unit 104 associates information about an object detected as a stationary object (e.g., rectangular coordinates) with characteristic information about a person identified as the owner of the object (e.g., human-like features and facial features), and stores the information in a storage unit (not shown). Note that if facial features of the person identified as the owner have not been extracted, the owner identification unit 104 stores only the human-like features as the characteristic information about the person.

[0037] 4 is an explanatory diagram illustrating an example of an overview of the owner identification process executed by the owner identification unit 104. In FIG. 4, (a) corresponds to the n-4th frame image, (b) corresponds to the n-3rd frame image, (c) corresponds to the n-2th frame image, (d) corresponds to the n-1th frame image, and (e) corresponds to the nth frame image. In other words, the status of the monitored object changes in the order of (a), (b), (c), (d), and (e).

[0038] 4 shows an example in which object B is detected as a stationary object and the owner of object B is identified. For example, the stationary object detection unit 103 detects object B as a stationary object in the nth frame image of FIG. 4( e). The owner identification unit 104 extracts owner candidates from the frame image in which the stationary object is detected (i.e., the nth frame image) and the four consecutive frame images immediately preceding it (i.e., the n-4th to n-1th frame images).

[0039] In this case, the owner identification unit 104 extracts the person detected at the position closest to object B in the n-4th frame image (person B in the example of FIG. 4( a)) as an owner candidate. The owner identification unit 104 also extracts the person detected at the position closest to object B in the n-3th frame image (person B in the example of FIG. 4( b)) as an owner candidate. The owner identification unit 104 also extracts the person detected at the position closest to object B in the n-2th frame image (person B in the example of FIG. 4( c)) as an owner candidate. The owner identification unit 104 also extracts the person detected at the position closest to object B in the n-1th frame image (person A in the example of FIG. 4( d)) as an owner candidate. The owner identification unit 104 also extracts the person detected at the position closest to object B in the nth frame image (person A in the example of FIG. 4( e)) as an owner candidate.

[0040] The owner identification unit 104 determines whether the owner candidates extracted from each of the n-4th to nth frame images are the same person using feature information (e.g., human feature amounts) of each person. In the example shown in FIG. 4, the owner identification unit 104 determines that the owner candidates extracted from each of the n-4th to n-2th frame images are the same person, person B. The owner identification unit 104 also determines that the owner candidates extracted from each of the n-1th to nth frame images are the same person, person A. Based on the determination result, the owner identification unit 104 identifies person B, who has been extracted the most times as an owner candidate, as the owner of object B.

[0041] The abandoned object detection unit 105 detects an abandoned object among objects detected as stationary objects as an abandoned object. When an object detected as a stationary object is detected in each of a predetermined number of consecutive frame images but a person identified as the owner of the object is not detected, the abandoned object detection unit 105 detects the object as an abandoned object.

[0042] For example, the abandoned object detection unit 105 reads, for each object detected as a stationary object, characteristic information (e.g., humanoid characteristics) of a person identified as the owner of the object from a storage unit (not shown). Next, the abandoned object detection unit 105 uses the read characteristic information to determine whether the person identified as the owner is present in a newly input frame image. If a stationary object is present in a newly input frame image but its owner is not, the abandoned object detection unit 105 increments the value of an owner-absent counter by 1, which counts the number of times it has been determined that the owner is not present. An owner-absent counter is provided for each stationary object. If a stationary object is not present in a newly input frame image, or if both a stationary object and its owner are present, the abandoned object detection unit 105 resets the value of the corresponding owner-absent counter to 0. If the value of the owner-absent counter reaches a predetermined value, the abandoned object detection unit 105 detects the stationary object corresponding to the owner-absent counter as an abandoned object.

[0043] The abandoned object detection unit 105 associates information about an object detected as an abandoned object (e.g., rectangular coordinates) with characteristic information about a person identified as the owner of the object (e.g., human features and facial features), and stores the associated information in a storage unit (not shown). If facial features of the person identified as the owner have not been extracted, the abandoned object detection unit 105 stores only the human features as characteristic information about the person. The abandoned object detection unit 105 also associates information about an object detected as an abandoned object with information indicating that the object is an abandoned object, and stores the associated information in a storage unit (not shown).

[0044] If an object previously detected as an abandoned object does not exist in a newly input frame image, or if both the object and its owner exist, the abandoned object detection unit 105 determines that the object is not an abandoned object and does not detect it as an abandoned object. Furthermore, the stationary object detection unit 103 associates information about the object with information indicating that it is not an abandoned object and stores the information in a storage unit (not shown).

[0045] The owner information update unit 106 identifies the same person from the frame image based on the facial features of the person identified as the owner of the object detected as a still object, and stores the human features of the identified person in a memory unit (not shown) as the human features of the person.

[0046] For example, a case will be described in which object A is detected as a stationary object and person A is identified as the owner of object A. In this case, the owner information update unit 106 reads feature information of person A (for example, human feature amounts and facial feature amounts) from a storage unit (not shown). Next, the abandoned object detection unit 105 uses the read facial feature amounts to determine whether person A is identical to a person detected in a newly input frame image. If there is a person determined to be identical, the abandoned object detection unit 105 updates the human feature amounts of the person determined to be identical as the human feature amounts of person A and stores the updated human feature amounts in a storage unit (not shown). The owner information update unit 106 performs the same process for each object detected as a stationary object.

[0047] If a person identified as the owner of a still object changes clothes or moves to a location with a significantly different lighting environment, the human feature values ​​extracted from the frame images will change. In this case, there is a possibility that the same person determination based on the human feature values ​​will not function correctly. Therefore, the owner information update unit 106 updates the human feature values ​​of the person identified as the owner of the still object based on newly input frame images. This configuration prevents difficulty in identifying the owner. As a result, the accuracy of identifying the owner of an abandoned object can be improved.

[0048] It should be noted that there are cases where, like person C, facial features are not stored in the storage unit as the owner's feature information, and only human-like features are stored. In this case, the owner information update unit 106 uses the human-like features to determine whether the person detected in the newly input frame image is the same as person C. If there is a person determined to be the same as person C and the facial features of that person have been extracted, the abandoned object detection unit 105 stores the facial features of that person in the storage unit (not shown) as the facial features of person C.

[0049] The data output unit 107 outputs information indicating an object detected as an abandoned object and a person identified as the owner of the object. The data output unit 107 may also output information indicating an object detected as a stationary object and a person identified as the owner of the object. The information output by the data output unit 107 includes, for example, rectangular coordinates of the detected object, feature information of the identified person (i.e., facial feature amounts and human-shaped feature amounts), detection date and time, and identification date and time.

[0050] The data output unit 107 outputs information indicating an object detected as an abandoned object (or a stationary object) and a person identified as the owner of the object to a display device (not shown) such as a display device for display. The data output unit 107 may also output information indicating an object detected as an abandoned object (or a stationary object) and a person identified as the owner of the object to a storage unit (not shown) of an external device for storage.

[0051] Next, a description will be given of the operation of the monitoring device 100. Fig. 5 is a flowchart illustrating the operation of the monitoring device.

[0052] The detection unit 101 sequentially receives frame images included in a moving image of a space to be monitored captured by an imaging device such as a fixed camera. The following describes a case where the detection unit 101 receives the n-th frame image.

[0053] The detection unit 101 detects objects and people from the n-th frame image (step S101). The detection unit 101 also detects human faces from the n-th frame image.

[0054] Next, the feature extraction unit 102 extracts feature information of the person detected from the n-th frame image (step S102). For example, the feature extraction unit 102 extracts face feature information and human figure feature information as the feature information of the person.

[0055] Next, the stationary object detection unit 103 detects, as stationary objects, objects that are stationary among the objects detected from the n-th frame image (step S103).

[0056] For example, the stationary object detection unit 103 calculates the IoU of the area specified by the detected object rectangle in the nth frame image and each of the frame images before the nth frame image (e.g., the n-5th to n-1th frame images). Next, the stationary object detection unit 103 determines that an object whose IoU is equal to or greater than a predetermined threshold is stationary. The stationary object detection unit 103 then detects, as stationary objects, objects determined to be stationary in each of a predetermined number of consecutive frame images (e.g., the n-9th to nth frame images).

[0057] Next, if a stationary object is detected in the nth frame image, the monitoring device 100 proceeds to step S105 (Y in step S104). On the other hand, if a stationary object is not detected in the nth frame image, the monitoring device 100 ends the processing for the nth frame image (N in step S104).

[0058] Next, the owner identification unit 104 extracts, from each of a plurality of frame images including the object detected as a stationary object, a person detected at a position closest to the object as an owner candidate (step S105).

[0059] For example, the owner identification unit 104 extracts as an owner candidate a person who is closest to an object detected as a stationary object from a frame image prior to the nth frame image (for example, the n-9th to n-1th frame images). The owner identification unit 104 may also extract owner candidates from the nth frame image using the same criteria.

[0060] Next, the owner identification unit 104 determines whether the people extracted as owner candidates are the same person by using the characteristic information (e.g., human features) of each person. Then, based on the result of the same person determination, the owner identification unit 104 identifies the person who has been extracted most frequently as an owner candidate as the owner (step S106).

[0061] Next, the owner identification unit 104 associates information about the object detected as a stationary object (e.g., rectangular coordinates) with characteristic information about the person identified as the owner of the object (e.g., humanoid features and facial features), and stores the information in a memory unit (not shown) (step S107).

[0062] Next, the abandoned object detection unit 105 detects, as an abandoned object, an object that has been left out of the objects detected as stationary objects in the n-th frame image (step S108).

[0063] For example, the abandoned object detection unit 105 detects a stationary object as an abandoned object when a stationary object is detected in each of a predetermined number of consecutive frame images (for example, frame images n-9 to n), but its owner is not detected.

[0064] Next, the owner information update unit 106 identifies the same person from the n-th frame image based on the facial feature values ​​of the owner of the still object, and stores the human-like feature values ​​of the identified person in a storage unit (not shown) as the human-like feature values ​​of the owner of the still object (step S109).

[0065] The owner information update unit 106 executes the process of step S109 for each object detected as a stationary object in the nth frame image. Note that for an object detected as a stationary object for the first time in the nth frame image, there is no room to update the owner's human features to the latest information, so the process of step S109 may be omitted.

[0066] There may be cases where facial features are not stored in the storage unit as the owner's feature information, and only human-like features are stored. In this case, the owner information update unit 106 may execute the following process in the process of step S109. That is, the owner information update unit 106 uses the human-like features of the owner to determine whether the owner is identical to the person detected in the nth frame image. If a person determined to be identical exists and the facial features of the person have been extracted, the abandoned object detection unit 105 stores the facial features of the person in the storage unit (not shown) as the facial features of the owner.

[0067] Next, the data output unit 107 outputs information indicating the object detected as the abandoned object and the person identified as the owner of the object (step S110).

[0068] 5 does not limit the operation of the monitoring device 100 of the present disclosure. For example, the process of step S102 may be executed when it is determined in the process of step S104 that a stationary object has been detected. Furthermore, for example, the process of step S108 and the process of step S109 may be executed in the reverse order or in parallel.

[0069] Next, the effects of this embodiment will be described. In this embodiment, the owner identification unit 104 extracts, as an owner candidate, a person detected closest to an object from each of a plurality of frame images including the object detected as a stationary object. The owner identification unit 104 also performs a same-person determination for the people extracted as owner candidates using characteristic information about each person. Then, based on the result of the same-person determination, the owner identification unit 104 identifies the person who has been extracted most frequently as an owner candidate as the owner. That is, in this embodiment, the owner of a stationary object is identified using a plurality of frame images. This configuration can improve the accuracy of identifying the owners of stationary objects and abandoned objects.

[0070] Furthermore, in this embodiment, the owner information update unit 106 identifies a person identified as the owner of an object detected as a stationary object from the frame image based on the facial feature values ​​of the person, and stores the human feature values ​​of the identified person in a storage unit (not shown) as the human feature values ​​of the person. That is, the owner information update unit 106 updates the human feature values ​​of the person identified as the owner of the stationary object based on the newly input frame image. This configuration prevents difficulty in identifying the owner when the person identified as the owner of the stationary object changes clothes or moves to a location with a significantly different lighting environment. As a result, the accuracy of identifying the owners of stationary objects and abandoned objects can be improved.

[0071] 6 is a block diagram illustrating a configuration of a computer according to the present disclosure. A CPU 1000 executes processing in accordance with a monitoring program stored in a storage device 1001, thereby realizing the functions of the monitoring device 100 according to the above embodiment.

[0072] That is, the CPU 1000 executes processing in accordance with the monitoring program stored in the memory device 1001, thereby realizing the functions of the detection unit 101, feature extraction unit 102, stationary object detection unit 103, owner identification unit 104, abandoned object detection unit 105, owner information update unit 106 and data output unit 107 of the monitoring device 100 shown in FIG. 1.

[0073] The storage device 1001 is, for example, a non-transitory computer-readable medium. The non-transitory computer-readable medium includes various types of tangible storage media. Specific examples of the non-transitory computer-readable medium include semiconductor memory (e.g., mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), and flash ROM).

[0074] The memory 1002 is realized by, for example, a RAM (Random Access Memory), and is a storage means for temporarily storing data when the CPU 1000 executes processing.

[0075] Next, an overview of the present disclosure will be described. FIG. 7 is a block diagram illustrating the main components of a monitoring device. The monitoring device 10 (e.g., corresponding to monitoring device 100) shown in FIG. 7 includes: detection means 11 (implemented by a detection unit 101 in the embodiment) that detects objects and people from frame images included in a video; stationary object detection means 12 (implemented by a stationary object detection unit 103 in the embodiment) that detects stationary objects among the detected objects as stationary objects; owner identification means 13 (implemented by an owner identification unit 104 in the embodiment) that extracts the person detected closest to the object from each of multiple frame images including the detected stationary object as an owner candidate and identifies the person who has been extracted the most frequently as the owner of the object; and abandoned object detection means 14 (implemented by an abandoned object detection unit 105 in the embodiment) that detects abandoned objects among the detected stationary objects as abandoned objects. With this configuration, the monitoring device 10 identifies the owner of a stationary object using multiple frame images. As a result, the accuracy of identifying the owners of stationary objects and abandoned objects can be improved.

[0076] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Each embodiment can be combined with other embodiments as appropriate.

[0077] Each drawing is merely an example for describing one or more embodiments. Each drawing may not relate to only one particular embodiment, but may also relate to one or more other embodiments. As will be understood by those skilled in the art, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings to create, for example, an embodiment not explicitly shown or described. Not all features or steps shown in any one drawing are necessary to describe an exemplary embodiment, and some features or steps may be omitted. The order of steps described in any drawing may be changed as appropriate.

[0078] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0079] (Supplementary Note 1) A surveillance device comprising: a detection means for detecting objects and people from frame images included in a moving image; a stationary object detection means for detecting a stationary object from among the detected objects as a stationary object; an owner identification means for extracting a person detected at a position closest to the object from each of a plurality of frame images including the object detected as a stationary object as an owner candidate, and identifying the person who has been extracted the most times as the owner of the object; and an abandoned object detection means for detecting an abandoned object from among the objects detected as stationary objects as an abandoned object.

[0080] (Supplementary Note 2) The monitoring device according to Supplementary Note 1, further comprising: a feature extraction means capable of extracting human-like features and facial features from a frame image as feature information of a detected person; the owner identification means storing information of an object detected as a still object and feature information of a person identified as the owner of the object in a memory unit in association with each other; and an owner information update means identifying a person identified as the owner of an object detected as a still object from a frame image based on the facial features of the same person, and storing the human-like features of the identified person in the memory unit as the human-like features of the person.

[0081] (Supplementary Note 3) The monitoring device according to Supplementary Note 1 or Supplementary Note 2, wherein the owner identification means extracts owner candidates of the one object from a frame image preceding a frame image in which the one object is detected as a stationary object.

[0082] (Supplementary Note 4) The monitoring device according to Supplementary Note 3, wherein the owner identification means performs a person identification determination for the persons extracted as owner candidates using characteristic information of each person, and identifies the person who has been extracted as an owner candidate the most times based on the determination result as the owner.

[0083] (Supplementary Note 5) The monitoring device according to Supplementary Note 3, wherein the stationary object detection means detects a stationary object from a frame image based on a comparison result between the frame image and a frame image preceding the frame image.

[0084] (Supplementary Note 6) The monitoring device according to Supplementary Note 5, wherein the stationary object detection means detects an object as a stationary object when a degree of overlap between an area of ​​the object in a frame image and an area of ​​the object in a frame image preceding the frame image is equal to or greater than a predetermined threshold value in each of a plurality of consecutive frame images.

[0085] (Supplementary Note 7) The monitoring device described in any one of Supplementary Note 1 to Supplementary Note 6, wherein the abandoned object detection means detects an object as an abandoned object when an object detected as a stationary object is detected in each of a plurality of consecutive frame images but a person identified as the owner of the object is not detected.

[0086] (Supplementary Note 8) The monitoring device according to any one of Supplementary Note 1 to Supplementary Note 7, further comprising an output unit that outputs information indicating an object detected as an abandoned object and a person identified as the owner of the object.

[0087] (Supplementary Note 9) A surveillance method characterized in that a computer detects objects and people from frame images included in a moving image, detects stationary objects from the detected objects as stationary objects, extracts the person detected closest to the object from each of a plurality of frame images including the object detected as a stationary object as an owner candidate, identifies the person extracted the most times as the owner of the object, and detects abandoned objects from the objects detected as stationary objects as abandoned objects.

[0088] (Supplementary Note 10) A monitoring program for causing a computer to execute the following processes: a detection process for detecting objects and people from frame images included in a moving image; a stationary object detection process for detecting stationary objects from among the detected objects as stationary objects; an owner identification process for extracting, as owner candidates, the person detected at a position closest to the object from each of a plurality of frame images including the object detected as a stationary object, and identifying the person who has been extracted the most times as the owner of the object; and an abandoned object detection process for detecting, as abandoned objects, objects that have been detected as stationary objects.

[0089] Some or all of the elements (e.g., configurations and functions) described in Supplementary Notes 2 to 8 that are dependent on Supplementary Note 1 may also be dependent on Supplementary Notes 9 and 10 in the same dependency relationship as Supplementary Notes 2 to 8. Some or all of the elements described in any Supplementary Note may be applied to various hardware, software, recording means for recording software, systems, and methods.

[0090] This application claims priority based on Japanese Patent Application No. 2024-062652, filed on April 9, 2024, the disclosure of which is incorporated herein in its entirety by reference.

[0091] 10, 100 Monitoring device 11 Detection means 12 Stationary object detection means 13 Owner identification means 14 Abandoned object detection means 101 Detection unit 102 Feature extraction unit 103 Stationary object detection unit 104 Owner identification unit 105 Abandoned object detection unit 106 Owner information update unit 107 Data output unit 1000 CPU 1001 Storage device 1002 Memory

Claims

1. A surveillance device comprising: a detection means for detecting objects and people from frame images included in a moving image; a stationary object detection means for detecting stationary objects from among the detected objects as stationary objects; an owner identification means for extracting as owner candidates the person detected closest to the object from each of a plurality of frame images including the object detected as a stationary object, and identifying the person who has been extracted the most times as the owner of the object; and an abandoned object detection means for detecting abandoned objects from among the objects detected as stationary objects as abandoned objects.

2. A surveillance device as described in claim 1, further comprising: a feature extraction means capable of extracting human-like features and facial features from frame images as feature information of a detected person; said owner identification means storing information of an object detected as a stationary object and feature information of a person identified as the owner of said object in a memory unit in association with each other; and an owner information update means for identifying a person identified as the owner of an object detected as a stationary object from frame images based on the facial features of the same person, and storing the human-like features of the identified person in the memory unit as the human-like features of the person.

3. A monitoring device according to claim 1 or 2, wherein said owner identification means extracts candidate owners of a certain object from a frame image preceding a frame image in which the certain object is detected as a stationary object.

4. The monitoring device according to claim 3, wherein the owner identification means performs a person identification process for the people extracted as owner candidates using the characteristic information of each person, and identifies the person who has been extracted most frequently as an owner candidate based on the results of the identification process as the owner.

5. The monitoring device according to claim 3, wherein said stationary object detection means detects a stationary object from a frame image based on a comparison result between the frame image and a frame image preceding the frame image.

6. A monitoring device as described in claim 5, wherein the stationary object detection means detects an object as a stationary object when the degree of overlap between the area of ​​the object in a frame image and the area of ​​the object in a frame image preceding the frame image in each of a plurality of consecutive frame images is equal to or greater than a predetermined threshold.

7. A surveillance device as claimed in claim 1 or claim 2, wherein the abandoned object detection means detects an object as an abandoned object when an object detected as a stationary object is detected in each of a plurality of consecutive frame images but a person identified as the owner of the object is not detected.

8. A monitoring device according to claim 1 or claim 2, further comprising an output means for outputting information indicating an object detected as an abandoned object and a person identified as the owner of the object.

9. A surveillance method characterized by the following: a computer detects objects and people from frame images included in a moving image; detects stationary objects from among the detected objects as stationary objects; extracts the person detected closest to the object from each of a plurality of frame images including the object detected as a stationary object as a candidate owner; identifies the person extracted the most times as the owner of the object; and detects abandoned objects from among the objects detected as stationary objects as abandoned objects.

10. A monitoring program for causing a computer to execute the following processes: a detection process for detecting objects and people from frame images included in a video; a stationary object detection process for detecting stationary objects from among the detected objects as stationary objects; an owner identification process for extracting the person detected closest to the object from each of a plurality of frame images including the object detected as a stationary object as an owner candidate, and identifying the person who has been extracted the most times as the owner of the object; and an abandoned object detection process for detecting abandoned objects from among the objects detected as stationary objects as abandoned objects.

Images