Monitoring device, monitoring method, program, and monitoring system

Abstract

To provide a monitoring device, a monitoring method, a program, and a monitoring system that can determine whether processing is necessary on the basis of the relationship between a target object separated from its owner, its owner, and a finder who has found the target object, estimated by image processing.SOLUTION: A monitoring device includes: an acquisition unit that acquires images generated by at least one imaging means; an image processing unit that processes the acquired images to detect a target object that has become separated from the owner and its owner, and detect a finder who has picked up the target object; and a determination unit that uses the detection results of the owner and the finder by the image processing unit to estimate the relationship between the owner and the finder, and determines whether to execute a specified process on the basis of the estimated results.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a monitoring device, a monitoring method, a program, and a monitoring system. [Background technology]

[0002] Patent Document 1 describes an image monitoring device that determines whether a person has changed their possessions from an image of the person, and detects the transfer of possessions between people based on that determination. This image monitoring device detects possessions, even if the possessions are difficult to obtain image information about in advance or do not extend beyond the person's area in the image, and monitors suspicious individuals based on the possessions in a space where many people are coming and going. A same-person image extraction means extracts a first person image and a second person image in which the same person appears from images taken at different times, and a different area detection means detects a different area having a brightness feature not included in the first person image and the second person image. The possession determination means determines that a change in the possession of the person has occurred when the different area detection means detects a different area having a size equal to or greater than a predetermined reference value. The handover detection means also refers to the person positions stored in the person information storage means to calculate the distance between the person whose belongings have disappeared and the person whose belongings have appeared, and determines that the person whose belongings have disappeared has handed over the belongings to the person whose belongings have appeared if the distance between the people is within a predetermined distance range. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-16344 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology described in Patent Document 1 simply determines from images whether an item has been handed over between a person whose belongings have been detected as missing and a person whose belongings have been detected as appearing. On the other hand, if an object is separated from its owner and then acquired by a person who has acquired it, there may or may not be a need for action. [Means for solving the problem]

[0005] The monitoring device in the present disclosure comprises: an acquisition means for acquiring an image generated by at least one imaging means; an image processing means for processing the acquired image to detect an object separated from its owner and the owner, and to detect a finder who has found the object; a determination means for determining the relationship between the owner and the finder based on the detection result of the owner and the finder by the image processing means, and determining whether or not a predetermined process needs to be performed based on the result of the determination; Equipped with.

[0006] The monitoring system in the present disclosure comprises: At least one camera; a monitoring device connected to the camera, The monitoring device an acquisition means for acquiring an image generated by the camera; a detection means for detecting an object separated from its owner and the owner of the object by processing the acquired image, and for detecting a finder who has found the object; and a decision means for inferring the relationship between the owner and the finder using the processing result, and for deciding whether or not a predetermined process needs to be performed depending on the inferred result.

[0007] The monitoring method in the present disclosure includes: One or more computers acquiring an image generated by at least one imaging means; By processing the acquired image, a target object separated from its owner and the owner are detected, and a finder who found the target object is detected; The method includes using the detection results of the owner and the finder to estimate the relationship between the owner and the finder, and determining whether or not a predetermined process needs to be executed based on the estimation result.

[0008] The program in this disclosure is an acquisition process for acquiring images generated by at least one imaging means; image processing for detecting an object separated from its owner and the owner by processing the acquired image, and detecting a finder who found the object; a determination process of estimating a relationship between the owner and the finder using the detection results of the owner and the finder obtained by the image processing, and determining whether or not a predetermined process needs to be executed based on the estimation result; It is a program for causing a computer to execute the above. [Effects of the Invention]

[0009] According to the present disclosure, it is possible to obtain a monitoring device, a monitoring method, a program, and a monitoring system that can determine whether or not processing is necessary based on the relationship between a target object that has become separated from its owner, its owner, and the finder who found the target object, as estimated by image processing. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a functional block diagram illustrating a configuration example of a monitoring device according to the present disclosure. [Figure 2] 10 is a flowchart illustrating an example of a processing operation of a monitoring device according to the present disclosure. [Figure 3] 1 is a diagram conceptually illustrating an example of a system configuration of a monitoring system according to the present disclosure. [Figure 4] FIG. 1 is a diagram illustrating an example of a configuration of a computer that realizes a monitoring device according to the present disclosure. [Figure 5]1 is a diagram for explaining a method for estimating a relationship between monitoring devices according to the present disclosure; [Figure 6] 1 is a diagram for explaining a method for estimating a relationship between monitoring devices according to the present disclosure; [Figure 7] FIG. 10 is a diagram illustrating an example of a data structure of camera information. [Figure 8] FIG. 2 is a diagram illustrating an example of a data structure of image information. [Figure 9] FIG. 10 is a diagram illustrating an example of a data structure of lost item information. [Figure 10] FIG. 10 is a diagram illustrating an example of a data structure of detection information. [Figure 11] FIG. 10 is a diagram illustrating an example of a data structure of relationship information. [Figure 12] 3 is a flowchart showing a detailed example of processing operations of a relationship estimation process in the flow of FIG. 2. [Figure 13] 10 is a flowchart showing a second example of detailed processing operations of the relationship estimation process in the flow of FIG. 2. [Figure 14] 10 is a flowchart showing a third example of detailed processing operations of the relationship estimation process in the flow of FIG. 2. [Figure 15] 10 is a flowchart showing a fourth example of detailed processing operations of the relationship estimation processing in the flow of FIG. 2. [Figure 16] 10 is a flowchart showing a fifth example of detailed processing operations of the relationship estimation process in the flow of FIG. 2. [Figure 17] FIG. 10 is a functional block diagram showing a configuration example of a second monitoring device according to the present disclosure. [Figure 18] FIG. 10 is a diagram conceptually illustrating an example of the system configuration of a second monitoring system according to the present disclosure. [Figure 19] 10 is a flowchart illustrating an example of a processing operation of a main part of a monitoring device according to the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, in this disclosure, the drawings relate to one or more embodiments. In all drawings, similar components are given similar reference numerals and descriptions thereof are omitted as appropriate. In each of the following drawings, configurations of parts that are not related to the essence of this disclosure are omitted and are not shown.

[0012] In this disclosure, "acquisition" includes at least one of a device going to retrieve data or information stored in another device or storage medium (active acquisition) and a device inputting data or information output from another device (passive acquisition). Examples of active acquisition include making a request or inquiry to another device and receiving a reply, and accessing and reading information from another device or storage medium. Examples of passive acquisition include receiving information that is distributed (or transmitted, pushed, etc.). Furthermore, "acquisition" may also mean selecting and acquiring data or information from received data or information, or selecting and receiving distributed data or information.

[0013] <Example of functional configuration> As shown in FIG. 1, the monitoring device 100 includes an acquisition unit 102, an image processing unit 104, and a determination unit 106. The acquisition unit 102 acquires an image generated by at least one imaging means (a camera 5, which will be described later). The image processing unit 104 processes the acquired image to detect the target object that has moved away from its owner and its owner, as well as to detect the finder who has found the target object. The decision unit 106 uses the detection results of the owner and finder by the image processing unit 104 to estimate the relationship between the owner and finder, and determines whether or not a predetermined process needs to be performed based on the estimation result.

[0014] Here, the target object is an object that has been lost or left behind by the owner.

[0015] <Example of operation> 2, in the monitoring device 100, first, the acquisition unit 102 acquires an image generated by at least one imaging means (camera 5, described later) (step S101). Then, the image processing unit 104 processes the acquired image (step S103) to detect a target object separated from its owner and its owner (step S105), and also detects the finder who found the target object (step S107).

[0016] Then, the determination unit 106 estimates the relationship between the owner and the finder using the detection results of the owner and the finder by the image processing unit 104 (step S109), and determines whether or not a predetermined process needs to be performed based on the estimated result (step S111).

[0017] According to this monitoring device 100, the image processing unit 104 processes the image acquired by the acquisition unit 102 to detect the target object separated from its owner and its owner, as well as the finder who found the target object. The determination unit 106 then uses the detection results of the owner and the finder by the image processing unit 104 to estimate the relationship between the owner and the finder, and determines whether or not to execute a predetermined process based on the estimation result. This provides a monitoring device, monitoring method, program, and monitoring system that can determine whether or not to execute a process based on the relationship between the target object separated from its owner and its owner, which is estimated by image processing, and the finder who found the target object.

[0018] A detailed example of the monitoring device 100 will be described below.

[0019] (First embodiment) <System Overview> FIG. 3 is a diagram conceptually illustrating the system configuration of a monitoring system 1 according to the present disclosure. The monitoring system 1 detects an object 20, such as a lost or forgotten item, in a public facility such as a train station, in a public transportation vehicle, etc. However, the area to be monitored is not limited to these, and may be, for example, a store, a parking lot, an aisle, a road, etc. Furthermore, the monitoring system 1 detects the owner 10 who dropped the object 20 and the finder 30 who found the object 20 through image processing, estimates the relationship between the owner 10 and the finder 30, and determines whether or not to execute a predetermined process depending on the estimation result.

[0020] Here, the relationship includes whether the owner 10 and the finder 30 are the same person, whether they are different people, and if they are different people, whether they are acquaintances or strangers.

[0021] The monitoring system 1 includes a monitoring device 100 and at least one camera 5 (imaging means). The monitoring system 1 may further include an image processing device 200. The monitoring device 100, the camera 5, and the image processing device 200 are connected to each other via a communication network 3. The monitoring device 100 and the image processing device 200 are computers such as personal computers and server computers. Alternatively, the monitoring device 100 and the image processing device 200 may be computers that realize communication devices such as smartphones, tablet terminals, and mobile phones. The monitoring device 100 and the image processing device 200 may also be realized by combining multiple computers. The image processing device 200 may also be included in the monitoring device 100. In other words, the image processing device 200 may be hardware that is integrated with the monitoring device 100, or may be hardware that is separate from the monitoring device 100.

[0022] The functions of the monitoring device 100 and the image processing device 200 may be realized by installing an application program on various computers and starting the program. Alternatively, the functions of the monitoring device 100 and the image processing device 200 may be realized in a form provided to a user by accessing a server computer on a cloud via a network such as the Internet from a communication device such as an operation terminal (for example, SaaS (Software as a Service)).

[0023] The monitoring device 100 has a storage device 120. The storage device 120 may be provided inside or outside the monitoring device 100. In other words, the storage device 120 may be a piece of hardware that is integrated with the monitoring device 100, or may be a piece of hardware that is separate from the monitoring device 100. Furthermore, the storage device 120 may be realized by multiple storage devices.

[0024] The camera 5 includes an imaging element such as a lens and a charge-coupled device (CCD) image sensor or a complementary metal oxide semiconductor (CMOS) image sensor, and is, for example, a network camera such as an Internet Protocol (IP) camera. The network camera has, for example, a wireless local area network (LAN) communication function and is connected to the monitoring device 100 or the image processing device 200 via a communication network 3, i.e., a relay device (not shown) such as a router. These cameras 5 may be so-called monitoring cameras, at least one of which is installed in a station, a store, a facility, a vehicle, etc. The camera 5 may also include a mechanism for controlling the orientation of the camera body and lens, zooming, focusing, etc., in accordance with the movement of a person.

[0025] Images generated by the camera 5 are preferably captured in real time and transmitted to the monitoring device 100 or the image processing device 200. When an image is transmitted to the monitoring device 100, the image is transmitted from the monitoring device 100 to the image processing device 200, and the image processing result is returned to the monitoring device 100. However, the image transmitted to the monitoring device 100 or the image processing device 200 does not have to be transmitted directly from the camera 5, but may be an image delayed by a predetermined time. Images captured by the camera 5 may be temporarily stored in another storage device, and the monitoring device 100 or the image processing device 200 may read them from the storage device sequentially or at predetermined intervals. Furthermore, the images transmitted to the monitoring device 100 or the image processing device 200 are preferably moving images, but may also be frame images transmitted at predetermined intervals or still images.

[0026] The image processing performed by the image processing device 200 includes various analytical processes of image data, such as pattern recognition based on predetermined conditions, determining pass / fail by comparing with a reference value, recognizing an object by matching with the feature of a predetermined object, and measuring the size of the object. Representative algorithms for deriving features in object recognition include, but are not limited to, SIFT (Scale-Invariant Feature Transform) and HOG (Histograms of Oriented Gradients).

[0027] Furthermore, when the monitoring device 100 performs image processing to identify multiple people in an image and track each person, it may extract facial feature information from each person and perform a matching process using the extracted feature information to identify and track whether the people are the same person. However, the feature information used in the matching process may be other biometric information or feature information of areas other than the face, in addition to facial feature information. The feature information used in the matching process may be at least one of biometric information such as face, iris, veins, fingerprints, and ears. Alternatively, the feature information used in the matching process may be information indicating at least one of the following characteristics: a person's gait, height, shoulder width, proportions of body parts, clothing (shape, color, material, etc.), hairstyle (including hair color), accessories (e.g., hats, glasses, accessories), and belongings (e.g., bags, umbrellas, canes, etc.). Alternatively, the feature information may be a combination of at least two of these types of feature information.

[0028] <Hardware configuration example> The monitoring device 100 according to the present disclosure is realized by a computer 1000 shown in Fig. 4. The camera 5 and image processing device 200 shown in Fig. 3, and a communication device 300 shown in Fig. 18, which will be described later, are also realized by the computer 1000.

[0029] The computer 1000 includes a bus 1010 , a processor 1020 , a memory 1030 , a storage device 1040 , an input / output interface 1050 , and a network interface 1060 .

[0030] The bus 1010 is a data transmission path for transmitting and receiving data among the processor 1020, memory 1030, storage device 1040, input / output interface 1050, and network interface 1060. However, the method of connecting the processor 1020 and the like to each other is not limited to bus connection.

[0031] The processor 1020 is implemented by a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like.

[0032] The memory 1030 is a main storage device realized by a RAM (Random Access Memory) or the like.

[0033] The storage device 1040 is an auxiliary storage device realized by a hard disk drive (HDD), a solid state drive (SSD), a memory card, a read only memory (ROM), or the like. The storage device 1040 stores program modules that realize each function of the monitoring device 100 (e.g., the acquisition unit 102, the image processing unit 104, the determination unit 106, and the execution unit 108 described below). The processor 1020 loads each of these program modules into the memory 1030 and executes them, thereby realizing each function corresponding to the program module. The storage device 1040 may also function as the storage device 120 of the monitoring device 100.

[0034] The program module may be recorded on a recording medium. The recording medium on which the program module is recorded may include a non-transitory, tangible medium usable by the computer 1000, and the program code readable by the computer 1000 (processor 1020) may be embedded in the medium.

[0035] The input / output interface 1050 is an interface for connecting the computer 1000 with various input / output devices. The input / output interface 1050 also functions as a communication interface for performing short-range wireless communication such as Bluetooth (registered trademark) and NFC (Near Field Communication). Furthermore, the input / output interface 1050 also functions as a communication interface for performing mobile wireless communication via a mobile communication network or the like.

[0036] The network interface 1060 is an interface for connecting the computer 1000 to a communication network 3. This communication network 3 is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network). The method for connecting the network interface 1060 to the communication network 3 may be a wireless connection or a wired connection.

[0037] The computer 1000 is then connected to necessary equipment (e.g., displays, touch panels, operation buttons, touchpads, keyboards, mice, speakers, microphones, cameras, printers, etc. of the monitoring device 100, image processing device 200, and communication device 300, and camera 5) via the input / output interface 1050 or the network interface 1060.

[0038] Each component of the monitoring device 100 according to the present disclosure in each figure is realized by any combination of hardware and software of the computer 1000 in Figure 4. Those skilled in the art will understand that there are many variations in the realization method and device. The functional block diagrams showing the monitoring device 100 according to the present disclosure in each figure show blocks of logical functional units, rather than a configuration in hardware units.

[0039] Furthermore, the monitoring device 100 and image processing device 200 of the present disclosure may be realized as a single chip or device, or may be realized in a form that can be attached to another device. For example, the monitoring device 100 and the image processing device 200 may be built into the camera 5.

[0040] <Example of functional configuration> The following explanation will be given with reference to FIG. The monitoring device 100 includes an acquisition unit 102, an image processing unit 104, and a determination unit 106. The acquisition unit 102 acquires an image generated by at least one camera 5 (imaging means).

[0041] Camera information 130 may be stored in storage device 120 as information relating to at least one camera 5. As shown in FIG. 7, camera information 130 includes, for each camera 5, identification information (camera ID in the figure) that can identify the camera 5 and installation location information that indicates the installation location of the camera 5, in association with each other. The installation location information includes, for example, information that can identify the shooting location, such as the name of a railway line, the name of a station, and information indicating the installation location within the station. The installation location information may be the address of the location where camera 5 is installed, or may be the latitude and longitude. Camera information 130 may further include information that indicates shooting conditions such as the shooting range, shooting direction, and angle of view.

[0042] Image information 132 may be stored in storage device 120 as information relating to images acquired from each camera 5. As shown in Fig. 8, image information 132 includes, for each camera 5, identification information (camera ID in the figure) that can identify the camera 5, date and time information indicating the date and time of shooting (or a period), and image data information, all associated with each other. The image data information includes, for example, information indicating the storage device, directory, folder, etc. that stores the image data, and the file name of the image data or a path including the directory and file name.

[0043] However, the path for storing the image data acquired by the acquisition unit 102 may be set according to, for example, a predetermined rule. For example, the image data acquired by the acquisition unit 102 may be saved as image data with a file name indicating the date and time under a folder having a folder name indicating the camera ID. Alternatively, a folder having a folder name indicating the date and time may be created, and image data of the corresponding camera 5 at the corresponding date and time may be saved in the folder. In this case, the image information 132 may not be necessary. However, in a configuration in which images taken before the detection of the target object 20 are not used in the process of estimating the relationship between the owner 10 and the finder 30, the image data may not necessarily be stored. Also, at least a portion of the image data acquired from the camera 5 may be stored.

[0044] The image processing unit 104 processes the acquired image to detect the target object 20 that has moved away from the owner 10 and the owner 10, as well as to detect the finder 30 who has found the target object 20. More specifically, the image processing unit 104 uses image processing to detect objects that have fallen to the floor or that have been placed on the floor, chair, table, etc., away from the person in the image as target objects 20, and detects the person in question as the owner 10 of the target object 20.

[0045] Alternatively, the image processing unit 104 may use image processing to detect an object that has been left on the floor, a chair, a table, or the like for a predetermined time or longer as the target object 20, and detect a person who picked up the target object 20 as the finder 30. Then, the image processing unit 104 may detect the owner 10 of the target object 20 using an image taken before the target object 20 was detected.

[0046] Specifically, the image processing unit 104 first detects an object that has been stationary (not moving) in the image for a predetermined time or more (for example, the predetermined time is 30 minutes, but is not limited to this), and determines whether or not the object has been touched by a person, or whether or not a person is present around the object (within a predetermined range). The image processing unit 104 identifies an object that has not been touched by a person or has no person present around it for a predetermined time or more, and detects the object as a target object 20.

[0047] Next, the image processing unit 104 detects through image processing that an object (target object 20) that was left on the floor, chair, table, etc. has been taken away by a person, and designates the taken-away object as the target object 20, and designates the person who took away the target object 20 as the finder 30. Then, the image processing unit 104 detects the person who placed the target object 20 on the floor, chair, table, etc. by processing images taken before the time when it was detected that the target object 20 had been taken away, and designates the detected person as the owner 10.

[0048] The image processing unit 104 is realized by the image processing device 200 in Fig. 3. For example, the image processing unit 104 transmits the image acquired by the acquisition unit 102 to the image processing device 200, and receives the results of processing by the image processing device 200 from the image processing device 200. However, as described above, the image processing device 200 may be integrated with the monitoring device 100, in which case data transmission and reception processing with the image processing device 200 is not performed.

[0049] The target object 20 is an item that has been lost (lost property) by the owner 10 or an object that has been left behind by the owner 10. The target object 20 is not particularly limited to, for example, a suitcase, a bag, a package, a document, a wallet, a key chain, an umbrella, gloves, a hat, clothing, glasses, a watch, an ornament, precious metals, cash, cards, a camera, a mobile phone, a smartphone, a personal computer, an electronic device, a hazardous material, etc.

[0050] Next, the image processing unit 104 detects the person who picked up the target object 20 as the finder 30 through image processing.

[0051] The image processing unit 104 associates the detected target object 20 with the owner 10 and stores them in the storage device 120 as lost item information 140. As shown in Fig. 9, for example, the lost item information 140 includes, for each target object 20, identification information of the target object 20, target object information, date and time information, identification information of the owner 10, owner information, identification information of the finder 30, and finder information, all of which are associated with each other.

[0052] The identification information of the target object 20 is identification information uniquely assigned to the target object 20 (target object ID in the drawing). The target object information includes feature information of the target object 20, and is, for example, information indicating the feature amount of the target object 20 extracted from an image. Furthermore, the target object information may be information indicating the features of the target object 20 (for example, a black men's long wallet). The target object information may include, for example, the name of the target object (for example, a business bag, a smartphone, etc.) and information indicating the attributes of the target object (for example, information indicating the color, material, shape, whether it is for women, children, etc.). The date and time information is information indicating the date and time when the target object 20 was detected.

[0053] The identification information of the owner 10 is identification information (owner ID in the drawing) that is uniquely assigned to the owner 10 of the target object 20. The owner information includes characteristic information of the owner 10, and is, for example, information indicating characteristic amounts extracted from the face area and non-face area of ​​the owner 10 extracted from the image. Furthermore, the owner information may be information indicating the characteristics of the owner 10 (for example, a medium-sized, medium-built office worker in his 20s wearing a suit), and may include, for example, information indicating the attributes of the owner 10 (for example, information indicating age, sex, occupation, clothing, appearance, hairstyle (color), etc.).

[0054] The identification information of the finder 30 is identification information (finder ID in the drawing) that is uniquely assigned to the finder 30 who found the target object 20. Similar to the owner information, the finder information also includes characteristic information of the finder 30, and is, for example, information indicating the amount of characteristics extracted from the face area and non-face area of ​​the finder 30 extracted from the image. Furthermore, the finder information may be information indicating the characteristics of the finder 30 (for example, a woman in her 30s, with children, etc.), and may also include information indicating the attributes of the finder 30 (for example, information indicating age, sex, occupation, clothing, appearance, hairstyle (color), etc.).

[0055] It should be noted that the owner 10, the target object 20, and the finder 30 may each be associated with multiple owners 10, multiple target objects 20, and multiple finders 30. For example, if it is not possible to identify a single person who lost a target object 20, multiple people can be associated with the target object 20 as owners 10. For example, if a single person leaves multiple target objects 20 behind, multiple target objects 20 can be associated with the owner 10. For example, if it is not possible to identify a single person who found a target object 20, multiple people can be associated with the target object 20 as finders 30. Furthermore, if it is detected that the target object 20 has been handed over from the person who originally found it to another person, another person can be associated with the target object 20 as a second finder 30.

[0056] The image processing unit 104 can use the feature information stored in the lost item information 140 to identify (track) people and objects in other images by a matching process to determine whether they are the owner 10, the target object 20, or the finder 30. The other images are at least either images before or after the image in which the target object 20 was first detected.

[0057] For example, in the example shown in FIG. 5 , the image processing unit 104 detects in image 400a that the owner 10 has dropped the target object 20. Then, in a subsequent image 400b, the image processing unit 104 detects that the finder 30 has picked up the target object 20. The image processing unit 104 detects that the owner 10 and the finder 30 are different people. Then, the image processing unit 104 tracks the owner 10, the target object 20, and the finder 30 using the image after the target object 20 was detected. Then, in a subsequent image 400c, the image processing unit 104 can detect that the finder 30 has delivered and handed over the target object 20 to the owner 10.

[0058] In the example shown in FIG. 6 , the image processing unit 104 detects in image 410b that the owner 10 has dropped the target object 20. Then, in the subsequent image 410c, the image processing unit 104 detects that a person walking behind the owner 10 has picked up the target object 20. The image processing unit 104 traces back to image 410a, which precedes image 410b in which the target object 20 was detected, and detects that person Pa and person Pb included in image 410a are the owner 10 and the finder 30, respectively. Then, for example, the image processing unit 104 detects the time during which the distance between the owner 10 and the finder 30 is within a predetermined distance. Because the time during which the distance between the owner 10 and the finder 30 is within the predetermined distance is equal to or greater than a threshold, the determination unit 106 can determine that the relationship between the owner 10 and the finder 30 is acquaintances.

[0059] The determining unit 106 estimates the relationship between the owner 10 and the finder 30 using the detection results of the owner 10 and the finder 30 by the image processing unit 104. As described above, the relationship includes whether the owner 10 and the finder 30 are the same person (the owner himself / herself), whether they are different people, and if they are different people, whether they are acquaintances or strangers.

[0060] The following are examples of methods for estimating a relationship, but the methods are not limited to these. In addition, a plurality of the following methods may be combined. (Example of estimation method 1: Using relative position or distance transitions) The image processing unit 104 identifies changes in at least one of the relative positions and distance between the owner (lost owner) 10 and the finder 30 through image processing. The image processing unit 104 detects the owner 10 and the finder 30 using at least one of images taken before and after the target object 20 is detected, and identifies changes in at least one of the relative positions and distances.

[0061] The image processing unit 104 identifies the time during which the relative positions of the owner 10 and the finder 30 are in a predetermined positional relationship as a transition of the relative positions between the owner 10 and the finder 30. The determination unit 106 estimates the relationship between the owner 10 and the finder 30 using the time during which the relative positions between the owner 10 and the finder 30 identified by the image processing unit 104 are in the predetermined positional relationship. For example, one example of a predetermined positional relationship is a positional relationship in which the finder 30 follows behind the owner 10 at a certain distance. Another example of a predetermined positional relationship is a positional relationship in which the owner 10 and the finder 30 are side-by-side. For example, in the former positional relationship, it is assumed that the finder 30 is stalking the owner 10, and in the latter positional relationship, it is assumed that the owner 10 and the finder 30 are acquaintances who are traveling together. The image processing unit 104 identifies the time during which the finder 30 is moving in this positional relationship in which he is following behind the owner 10 at a certain distance. Alternatively, the image processing unit 104 identifies the time during which the owner 10 and the finder 30 are moving in a side-by-side positional relationship.

[0062] Furthermore, when multiple people (for example, a group of four or five people, such as students returning home or a group of relatives) are moving in a group lined up front, behind, left and right, the positional relationship between the owner 10 and the finder 30 may be other than side-by-side. The image processing unit 104 may identify the time when the owner 10 and the finder 30 are moving in a group positional relationship within a group of multiple people including other people in the vicinity. Here, the predetermined range is, for example, a circular area with a diameter of 1.5 meters. In the case of multiple people, it is assumed that the owner 10 and the finder 30 are present within this predetermined range while their positional relationship is subtly changing.

[0063] If the time period during which the finder 30 has been following the owner 10 at a certain distance, as determined by the image processing unit 104, is equal to or greater than a threshold (for example, 10 minutes or more), the determination unit 106 presumes that the finder 30 is a stranger who is following the owner 10. On the other hand, if the time period during which the owner 10 and the finder 30 have been moving side-by-side, as determined by the image processing unit 104, is equal to or greater than a threshold, the determination unit 106 presumes that the owner 10 and the finder 30 are acquaintances who are traveling together.

[0064] Furthermore, the image processing unit 104 may identify changes in the distance between the owner (lost person) 10 and the finder 30. For example, the image processing unit 104 identifies the time during which the distance between the owner 10 and the finder 30 is within a predetermined distance. If the time during which the owner 10 and the finder 30 are within a predetermined distance as determined by the image processing unit 104 is equal to or longer than a threshold (for example, 5 minutes or longer), the determination unit 106 presumes that the owner 10 and the finder 30 are acquaintances traveling together.

[0065] However, in situations where there is no movement, such as when waiting for a train to arrive on a station platform, the two people may be different people even if the distance between them is short for a time equal to or longer than the threshold. Therefore, the threshold may be changed depending on the situation (for example, the threshold may be set longer (e.g., 20 minutes) compared to when both people are moving). The situation may be identified based on the installation location of the camera 5 (e.g., a station platform), the time, etc., or may be identified by processing the image. In the case of image processing, the image processing unit 104 may identify the situation as such when, for example, a predetermined percentage or more of people other than the owner 10 and the finder 30 in the image are not moving.

[0066] The image processing unit 104 stores the detection results of the owner 10 and the finder 30 obtained by image processing in the storage device 120 as detection information 150. In addition, the determination unit 106 stores information indicating the relationship between the owner 10 and the finder 30 in the storage device 120 as relationship information 160.

[0067] As shown in FIG. 10, the detection information 150 includes, for each target object 20, identification information of the target object 20 (target object ID in the figure), date and time information indicating the date and time when the target object 20 was detected, identification information of the owner 10 of the target object 20 (owner ID in the figure), identification information of the finder 30 of the target object 20 (finder ID in the figure), and information indicating the detection result of the image processing unit 104, all associated with each other.

[0068] The information indicating the detection results of the image processing unit 104 includes at least one of information indicating the distance between the owner 10 and the finder 30, the time during which the owner 10 and the finder 30 are within a predetermined distance, and the time during which the relative positions of the owner 10 and the finder 30 are in a predetermined positional relationship. Furthermore, the information indicating the detection results of the image processing unit 104 may include at least one of information indicating at least one of the facial direction and line of sight of at least one of the owner 10 and the finder 30, information indicating the facial expression of at least one of the owner 10 and the finder 30, and information indicating the movement of at least one of the hand and arm of at least one of the owner 10 and the finder 30, which will be described later.

[0069] As shown in FIG. 11, the relationship information 160 includes, for each target object 20, the identification information of the target object 20 (target object ID in the figure), date and time information indicating the date and time the target object 20 was detected, identification information of the owner 10 of the target object 20 (owner ID in the figure), identification information of the finder 30 of the target object 20 (finder ID in the figure), and relationship information indicating the relationship, all associated with each other.

[0070] The date and time information can be used in a process of determining whether the target object 20 will be returned to the owner 10 after a predetermined time has elapsed. The relationship information indicating the relationship between the owner 10 and the finder 30 may include information indicating at least one of a stranger, an acquaintance, or the finder 30, or information indicating that the relationship is undetermined if the relationship cannot be identified. Furthermore, the information indicating the relationship may further include information indicating a specific relationship such as parent-child, husband and wife, siblings, friends, or a couple.

[0071] (Example of estimation method 2: Method using face direction or gaze) The image processing unit 104 may further detect at least one of the facial direction and the line of sight of at least one of the owner 10 and the finder 30. In other words, the detection result by the image processing unit 104 may include at least one of the facial direction and the line of sight of at least one of the owner 10 and the finder 30. The image processing unit 104 may also identify the time during which both the owner 10 and the finder 30 turn their faces or their lines of sight toward each other.

[0072] The determining unit 106 estimates the relationship between the owner 10 and the finder 30 using at least one of the face direction and the line of sight of at least one of the owner 10 and the finder 30 detected by the image processing unit 104. For example, if both the owner 10 and the finder 30 are turning their faces or looking at each other, it can be assumed that the owner 10 and the finder 30 are having a conversation. Therefore, the determining unit 106 will assume that the relationship between the owner 10 and the finder 30 is that of acquaintances. On the other hand, if only the finder 30 is turning their face or looking at the owner 10, the determining unit 106 may assume that the finder 30 is a stranger who is stalking the owner 10.

[0073] Furthermore, the determining unit 106 may use the time during which both the owner 10 and the finder 30 are turning their faces or looking at each other as the detection result of the image processing unit 104. For example, if the time specified by the image processing unit 104 is equal to or greater than a threshold, the determining unit 106 estimates that the relationship between the owner 10 and the finder 30 is that of acquaintances.

[0074] (Example of estimation method 3: Using facial expressions) The image processing unit 104 may further detect the facial expression of at least one of the owner 10 and the finder 30. In other words, the detection result by the image processing unit 104 may include the facial expression of at least one of the owner 10 and the finder 30. The facial expression may include, for example, a smile. The facial expression may also include gestures such as nodding to each other, or whether or not they are talking. The determining unit 106 estimates the relationship between the owner 10 and the finder 30 using the facial expression detected by the image processing unit 104. For example, if the determining unit 106 detects that one or both of the owner 10 and the finder 30 are smiling, it estimates that the relationship between the owner 10 and the finder 30 is that of acquaintances.

[0075] (Example of estimation method 4: Method using hand movements) The image processing unit 104 may further detect hand movements of at least one of the owner 10 and the finder 30. In other words, the detection result by the image processing unit 104 may include hand movements of at least one of the owner 10 and the finder 30. The hand movements include, for example, at least one of greetings or signals such as raising a hand towards the other person, waving to the other person, or beckoning to the other person, holding hands, or folding arms. The determining unit 106 estimates the relationship between the owner 10 and the finder 30 using the hand movement detected by the image processing unit 104. For example, if the determining unit 106 detects a movement in which at least one of the owner 10 and the finder 30 is waving their hand towards the other, it estimates that the relationship between the owner 10 and the finder 30 is that of acquaintances.

[0076] (Example of estimation method 5: Method using attributes or features) The image processing unit 104 may further detect at least one of the attributes and characteristics of the owner 10 and the finder 30. In other words, the detection result by the image processing unit 104 may include at least one of the attributes and characteristics of the owner 10 and the finder 30. The attributes and characteristics of the owner 10 and the finder 30 include, for example, attribute information indicating age, sex, occupation, clothing, appearance, hairstyle (color), etc., and at least one of feature information of the facial area, biometric information, and feature information of areas other than the face.

[0077] The determining unit 106 estimates the relationship between the owner 10 and the finder 30 using at least one of the attributes and characteristics of the owner 10 and the finder 30. Here, it estimates whether the owner 10 and the finder 30 are the same person (principal person). The determination unit 106 compares at least one of the attributes and characteristics of the owner 10 and the finder 30, and if the similarity is equal to or greater than a threshold, it presumes that the relationship is between the two people. On the other hand, if the similarity is less than the threshold, it presumes that the relationship is between two people other than the owner 10 and the finder 30.

[0078] However, two levels of similarity may be provided. For example, the second threshold value is lower than the first threshold value, and when the similarity is less than the first threshold value and equal to or greater than the second threshold value, the determining unit 106 may infer that the relationship between the owner 10 and the finder 30 is that of close relatives (i.e., acquaintances). In this case, when the similarity is less than the second threshold value, the determining unit 106 may infer that the relationship is that of strangers.

[0079] In another example, if the image processing unit 104 detects that the owner 10 is under 10 years old and the finder 30 is in their 40s, the determination unit 106 may infer that the owner 10 and the finder 30 are parent and child (or guardian) and that their relationship is acquaintances (parent and child).

[0080] Furthermore, a combination of the above-described methods for estimating the relationship may be used, and the attributes or characteristics of the owner 10 and the finder 30 may be used, as well as the facial direction or gaze of the owner 10 and the finder 30. For example, if the image processing unit 104 detects that both the owner 10 and the finder 30 are in their 60s, one is female and the other is male, the similarity in the facial features of the owner 10 and the finder 30 is equal to or greater than a threshold, and the time that the facial direction or gaze of the owner 10 and the finder 30 are directed toward each other is equal to or greater than a threshold, the determination unit 106 may estimate that the owner 10 and the finder 30 are siblings and that their relationship is acquaintances (siblings).

[0081] Next, the predetermined processing will be described. The decision unit 106 decides whether or not to execute the predetermined processing according to the result of the estimation. First, if it is estimated that the owner 10 and the finder 30 are the same person, the determining unit 106 determines that the predetermined process is not necessary. On the other hand, if it is estimated that the owner 10 and the finder 30 are different people, and the target object 20 is not returned after a certain time has passed, there is a possibility that it has been picked up or stolen. In this case, it is necessary to take measures such as reporting to nearby security guards, searching for the finder 30, recovering the target object 20, and returning it to the owner 10. Therefore, the determination unit 106 determines whether or not to execute a predetermined process depending on the estimation result.

[0082] The predetermined processing is exemplified below, but is not limited to these, and a combination of the following may also be used. (Processing example 1) A security guard or a person in charge of lost property notifies a recognizable communication device. (Processing example 2) Display on a display device near the camera 5. (Processing Example 3) An audio output or an alarm sound is output from an audio processing device around the camera 5. (Processing Example 4) The finder 30 is tracked using images acquired from a plurality of cameras 5. The execution of the predetermined processing will be explained in the following embodiment.

[0083] <Example of operation> The following description will be given with reference to FIG. 2 and FIGS. The flow in Fig. 2 is started at a predetermined timing. The predetermined timing may be, for example, a time when monitoring is required. The flow in Fig. 2 may be executed in real time or may be executed on recorded images.

[0084] 2, the acquisition unit 102 acquires an image generated by at least one camera 5 (step S101). Then, the image processing unit 104 processes the acquired image (step S103) to detect the target object 20 that has moved away from the owner 10 and the owner 10 of the target object 20 (step S105). Furthermore, the image processing unit 104 processes the acquired image (step S103) to detect the finder 30 who found the target object 20 (step S107).

[0085] If the target object 20 is detected in step S105, the monitoring device 100 executes the processes from step S105 onwards. If the target object 20 is not detected in step S105, the monitoring device 100 returns to step S101 and repeats the image processing.

[0086] Then, the determination unit 106 estimates the relationship between the owner and the finder using the detection results of the owner and the finder by the image processing unit 104 (step S109), and determines whether or not a predetermined process needs to be performed based on the estimated result (step S111).

[0087] The relationship estimation process will be described below for each of the above-mentioned example methods for estimating the relationship. The relationship estimation process is executed after the target object 20, the owner 10, and the finder 30 are detected in steps S105 and S107.

[0088] (Example of estimation method 1: Using relative position or distance transitions) 12, after the target object 20, the owner 10, and the finder 30 are detected in steps S105 and S107 of FIG. 2, the acquisition unit 102 acquires an image before the target object 20 was detected (step S121). Then, the image processing unit 104 identifies a change in at least one of the relative position and distance between the owner 10 and the finder 30 through image processing (step S123). Here, as a change in the distance between the owner 10 and the finder 30, the image processing unit 104 identifies a time during which the distance between the owner 10 and the finder 30 is within a predetermined distance. In another example, the image processing unit 104 may identify a time during which the relative positions of the owner 10 and the finder 30 are in a predetermined positional relationship.

[0089] Then, the determining unit 106 determines whether the time specified by the image processing unit 104 is equal to or greater than a threshold (step S125). If the time specified by the image processing unit 104 is not equal to or greater than the threshold (NO in step S125), the determining unit 106 determines that the relationship between the owner 10 and the finder 30 is strangers (step S127). On the other hand, if the time specified by the image processing unit 104 is equal to or greater than the threshold (YES in step S125), the determining unit 106 determines that the relationship between the owner 10 and the finder 30 is acquaintances (step S129). After steps S127 and S129, the monitoring apparatus 100 proceeds to step S111 in FIG.

[0090] (Example of estimation method 2: Method using face direction or gaze) 13, after the target object 20, the owner 10, and the finder 30 are detected in steps S105 and S107 of FIG. 2, the acquisition unit 102 acquires an image before the target object 20 is detected (step S121). Then, the image processing unit 104 identifies the facial orientation or line of sight of the owner 10 and the finder 30 through image processing (step S133). Here, the image processing unit 104 identifies whether the facial orientation or line of sight of the owner 10 and the finder 30 are facing each other (step S135).

[0091] If the image processing unit 104 determines that the owner 10 and the finder 30 are not facing each other in terms of their facial orientation or line of sight (NO in step S135), the determining unit 106 determines that the owner 10 and the finder 30 are strangers (step S127). On the other hand, if the image processing unit 104 determines that the owner 10 and the finder 30 are facing each other in terms of their facial orientation or line of sight (YES in step S135), the determining unit 106 determines that the owner 10 and the finder 30 are acquaintances (step S129). After steps S127 and S129, the monitoring apparatus 100 proceeds to step S111 in FIG.

[0092] (Example of estimation method 3: Using facial expressions) 14, after the target object 20, the owner 10, and the finder 30 are detected in steps S105 and S107 of FIG. 2, the acquisition unit 102 acquires an image before the target object 20 is detected (step S121). Then, the image processing unit 104 identifies the facial expression of at least one of the owner 10 and the finder 30 through image processing (step S143). If the image processing unit 104 does not detect a smile or conversation between the owner 10 and the finder 30 (NO in step S145), the determination unit 106 determines that the owner 10 and the finder 30 are strangers (step S127). On the other hand, if the image processing unit 104 detects a smile or conversation between the owner 10 and the finder 30 (YES in step S145), the determination unit 106 determines that the owner 10 and the finder 30 are acquaintances (step S129). After steps S127 and S129, the monitoring apparatus 100 proceeds to step S111 in FIG.

[0093] (Example of estimation method 4: Method using hand movements) 15, after the target object 20, the owner 10, and the finder 30 are detected in steps S105 and S107 of FIG. 2, the acquisition unit 102 acquires an image before the target object 20 is detected (step S121). Then, the image processing unit 104 identifies the hand movement of at least one of the owner 10 and the finder 30 through image processing (step S153). If the image processing unit 104 determines that the hand movement of the owner 10 or the finder 30 is not a greeting or a signal to the other person (NO in step S155), the determination unit 106 determines that the relationship between the owner 10 and the finder 30 is strangers (step S127). On the other hand, if the image processing unit 104 determines that the hand movement of the owner 10 or the finder 30 is a greeting or a signal to the other person (YES in step S155), the determination unit 106 determines that the relationship between the owner 10 and the finder 30 is acquaintances (step S129). After steps S127 and S129, the monitoring apparatus 100 proceeds to step S111 in FIG.

[0094] (Example of estimation method 5: Method using attributes or features) 16, after the target object 20, the owner 10, and the finder 30 are detected in steps S105 and S107 of FIG. 2, the acquisition unit 102 acquires an image before the target object 20 is detected (step S121). Then, the image processing unit 104 identifies at least one of the attributes and characteristics of the owner 10 and the finder 30 through image processing (step S163). Then, the image processing unit 104 performs a comparison process between at least one of the attributes and characteristics of the owner 10 and the finder 30, and calculates the similarity.

[0095] If the similarity calculated by the image processing unit 104 is not equal to or greater than the threshold (NO in step S165), the determining unit 106 determines that the relationship between the owner 10 and the finder 30 is strangers (step S127). On the other hand, if the similarity calculated by the image processing unit 104 is equal to or greater than the threshold (YES in step S165), the determining unit 106 determines that the relationship between the owner 10 and the finder 30 is the same person (step S167). After steps S127 and S167, the monitoring apparatus 100 proceeds to step S111 in FIG.

[0096] As described above, according to this embodiment, the monitoring device 100 includes an acquisition unit 102, an image processing unit 104, and a determination unit 106. The image processing unit 104 processes the image acquired by the acquisition unit 102 to detect the target object 20 that has moved away from the owner 10 and the owner 10, and to detect the finder 30 who found the target object 20. The determination unit 106 then uses the detection results of the owner 10 and the finder 30 by the image processing unit 104 to estimate the relationship between the owner 10 and the finder 30, and determines whether or not to execute a predetermined process depending on the estimation result. This makes it possible to obtain a monitoring device, monitoring method, program, and monitoring system that can determine whether or not processing is necessary based on the relationship between the target object 20 that has moved away from the owner 10, the owner 10, and the finder 30 who found the target object 20, as estimated by image processing.

[0097] (Second embodiment) The second monitoring device 100 of Fig. 17 according to the present disclosure is the same as the monitoring device 100 of Fig. 1 except that it further includes an execution unit 108. Note that each element of Fig. 17 may be combined with each element of one or more embodiments to the extent that no contradiction occurs.

[0098] <System Overview> 18, a second monitoring system 1 according to the present disclosure includes a communication device 300 used by a security guard 40 in addition to the monitoring system 1 of FIG. 3. The communication device 300 is connected to the monitoring device 100 via a communication network 3. Note that the communication network 3 connecting the communication device 300 and the monitoring device 100, the communication network 3 connecting the camera 5 and the monitoring device 100, and the communication network 3 connecting the image processing device 200 and the monitoring device 100 may be a combination of different networks.

[0099] The communication device 300 is a computer 1000 such as a personal computer, a smartphone, or a tablet terminal. The communication device 300 may be a portable device or a stationary device. An application having a function of receiving and outputting a notification from the monitoring device 100 is pre-installed and activated on the communication device 300. Alternatively, the notification from the monitoring device 100 may be received by accessing and logging in to a website provided by the monitoring system 1 using a browser on the communication device 300. Furthermore, there may be multiple communication devices 300, and each may be a different device (for example, a smartphone and a personal computer).

[0100] <Example of functional configuration> As shown in FIG. 17, the monitoring device 100 includes the same acquisition unit 102, image processing unit 104, and decision unit 106 as in FIG. 1, and further includes an execution unit 108. When the determination unit 106 determines that a predetermined process is necessary, the execution unit 108 executes the predetermined process. For example, the execution unit 108 notifies the communication device 300 that the security guard 40 can recognize.

[0101] Furthermore, if the determining unit 106 determines that a predetermined process is necessary, the image processing unit 104 processes the image to track the finder 30 who found the target object 20. The image processing unit 104 performs a comparison process using the characteristic information of the finder 30 stored in the lost item information 140 with characteristic information of a person detected from an image generated by at least one camera 5, thereby identifying a person whose similarity is equal to or exceeds a threshold as the finder 30 and tracking the finder 30.

[0102] Note that this similarity is different from the similarity used when estimating the relationship between the owner 10 and the finder 30 described above, and is used in a matching process to determine whether a person detected in a different image is the finder 30 to be tracked. Therefore, for example, the threshold for similarity used here may be different from the threshold for similarity used when estimating the relationship between the owner 10 and the finder 30 described above. For example, when estimating the relationship between the owner 10 and the finder 30, the threshold for similarity used to identify that the owner 10 and the finder 30 are different people may be lower than the threshold for similarity used in the matching process to track the finder 30. The threshold for similarity used to identify that the owner 10 and the finder 30 are the same person may be higher than the threshold for similarity used in the matching process to track the finder 30, or the same threshold may be used.

[0103] The execution unit 108 may cause an image including the finder 30 being tracked by the image processing unit 104 to be displayed on the display of the communication device 300 of the security guard 40 .

[0104] Furthermore, the determining unit 106 determines that a predetermined process is necessary if it is estimated that the owner 10 and the finder 30 are different people. The determining unit 106 may determine that a predetermined process is not necessary if it is estimated that the owner 10 and the finder 30 are the same person. In this case, however, the executing unit 108 may perform the predetermined process of displaying information indicating that the finder 30 is the owner 10 on the display of the communication device 300 of the security guard 40.

[0105] As described above, the predetermined process is at least one of the processes shown in (Processing Example 1) to (Processing Example 4). For example, in processing example 1, the execution unit 108 notifies the communication device 300 that can be recognized by the security guard 40 or the lost property management officer. The execution unit 108 performs at least one of the following actions on the communication device 300: displaying the notification on a display, outputting sound from a speaker, and outputting vibration to a vibrator. Upon receiving the notification and accepting a predetermined operation from the security guard 40, the execution unit 108 may cause detailed information about the notification to be displayed on a display. The detailed information about the notification may include, for example, an image containing at least one of the target object 20, the owner 10, and the finder 30. The detailed information about the notification may further include information indicating the installation location of the camera 5 that captured the image and information indicating the date and time the image was captured. The detailed information about the notification may further include information indicating the relationship between the owner 10 and the finder 30 identified by the determination unit 106.

[0106] By checking the detailed information of the notification displayed on the display, the security guard 40 can search for the owner 10 or the finder 30 in the vicinity and call out to them. The security guard 40 can tell the owner 10 that the object 20 has been dropped, and can tell the finder 30 to go through the procedure for reporting lost property so that the object 20 can be returned to the owner 10.

[0107] In processing example 2, the execution unit 108 may display, on a display device (for example, a signage) installed near the camera 5, information indicating that a lost item has been detected and the location where the target object 20, which is the lost item, was detected. Alternatively, the execution unit 108 may display detailed information about the above-mentioned notification content on a display device installed near the camera 5. However, information about the owner 10 and the finder 30 shall be handled lawfully.

[0108] In processing example 3, the execution unit 108 outputs audio or a warning sound from an audio processing device (e.g., a speaker) installed near the camera 5. For example, the execution unit 108 outputs messages such as "A lost item has been detected," or "If you have found the lost item, please report it." In other words, the execution unit 108 outputs a message to make the owner 10 aware of the loss, or to urge the finder 30 to report the found object 20, or to inform the finder that the found object 20 is being monitored, thereby discouraging theft.

[0109] In processing example 4, the execution unit 108 causes the image processing unit 104 to track the finder 30 using images acquired from the multiple cameras 5. Furthermore, the execution unit 108 may cause the display of the communication device 300 of the security guard 40 to display an image of the finder 30 being tracked and information indicating the location of the finder 30.

[0110] <Example of operation> The monitoring device 100 operates as shown in Fig. 19. The flow in Fig. 19 starts after the decision unit 106 decides whether or not it is necessary to execute a predetermined process in step S111 of Fig. 2. If the decision unit 106 decides that it is necessary to execute the predetermined process (YES in step S201), the execution unit 108 executes the predetermined process (step S203). If the decision unit 106 decides that it is not necessary to execute the predetermined process (NO in step S201), the execution unit 108 bypasses step S203, does not execute the predetermined process, and ends this process.

[0111] As described above, in this embodiment, the monitoring device 100 further includes an execution unit 108. When the determination unit 106 determines that a predetermined process is necessary, the execution unit 108 executes the predetermined process. For example, the execution unit 108 notifies the communication device 300 that can be recognized by the security guard 40. In this way, the monitoring device 100 of this embodiment not only achieves the same effects as the above-described embodiment, but also can execute appropriate processing when it is determined that a predetermined processing is necessary.

[0112] For example, the communication device 300 of the security guard 40 is notified that the target object 20 has been detected, and the security guard 40 can search for the owner 10, the target object 20, and the finder 30, and return the target object 20 to the owner 10 or encourage the finder 30 to report the target object 20. This provides an opportunity for the target object 20 to be returned to the owner 10. In addition, the execution unit 108 outputs information such as the fact that an item has been lost from a nearby display device or speaker, which can make the owner 10 realize that he or she has lost the item, or encourage the finder 30 to refrain from stealing the target object 20 and report it as lost property.

[0113] Although the embodiments of the present invention have been described above with reference to the drawings, these are merely examples of the present invention, and various other configurations can also be adopted.

[0114] (Other embodiments) Another example of the present disclosure may be a program that causes at least one computer to execute the above-disclosed method, or a computer-readable recording medium on which such a program is recorded. This recording medium includes a non-transitory tangible medium. The computer program comprises computer program code which, when executed by a computer, causes the computer to perform the monitoring method on a monitoring device.

[0115] Any combination of the above components, and conversion of the present disclosure into a method, device, system, recording medium, computer program, etc., are also valid aspects of the present disclosure.

[0116] Furthermore, the various components of the present disclosure do not necessarily have to be independent entities, but may be formed as a single member by multiple components, one component may be formed from multiple components, one component may be a part of another component, or part of one component may overlap with part of another component, etc.

[0117] Furthermore, although the methods and computer programs disclosed herein describe a number of steps in a sequential order, the order in which the steps are performed does not limit the order in which the steps are performed. Therefore, when implementing the methods and computer programs disclosed herein, the order of the steps can be changed as long as it does not cause any problems in terms of the content.

[0118] Furthermore, the steps of the method and computer program disclosed herein are not limited to being executed at different times, and may be executed while one step is being executed, or may be executed partially or entirely overlapping with another step, etc.

[0119] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate. In this disclosure, when information about the owner 10, the target object 20, and the finder 30 is acquired and used, it is assumed that this is done lawfully.

[0120] In addition, although the flowcharts used in the above description show multiple steps (processes) in a sequential order, the order of the steps executed in each embodiment is not limited to the order shown. In each embodiment, the order of the steps shown in the drawings can be changed as long as it does not cause any problems in terms of the content.

[0121] Some or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes. 1. an acquisition means for acquiring an image generated by at least one imaging means; an image processing means for processing the acquired image to detect an object separated from its owner and the owner, and to detect a finder who has found the object; a decision means for inferring the relationship between the owner and the finder using the detection results of the owner and the finder by the image processing means, and for deciding whether or not to execute a predetermined process based on the inferred result. 2. In the monitoring device described in 1., the image processing means identifies a change in at least one of the relative positions and the distance between the owner and the finder; The determining means estimates the relationship using the identified transition. 3. In the monitoring device according to 1. or 2., the detection result includes at least one of a facial orientation and a line of sight of at least one of the owner and the finder; The determination means estimates the relationship using at least one of the face direction and the line of sight. 4. In the monitoring device according to any one of 1. to 3., the detection result includes a facial expression of at least one of the owner and the finder; The determination means estimates the relationship using the facial expression. 5. In the monitoring device according to any one of 1. to 4., the detection result includes a hand movement of at least one of the owner and the finder; The determining means estimates the relationship using the hand movement. 6. In the monitoring device according to any one of 1. to 5., the detection result includes at least one of attributes and characteristics of the owner and the finder; The determining means estimates the relationship using at least one of the attribute and the feature. 7. In the monitoring device according to any one of 1. to 6., further comprising an execution unit that executes the predetermined processing when the determination unit determines that the predetermined processing is necessary; The execution means is a monitoring device that notifies a security guard via a communication device that can be recognized by the security guard. 8. In the monitoring device according to any one of 1. to 7., When the determining means determines that the predetermined processing is necessary, the image processing means processes the image to track the finder who found the target object. 9. In the monitoring device according to any one of 1. to 8., The determining means The monitoring device determines that the predetermined process is necessary when the relationship is estimated to be between the person and the person other than the person. 10. In the monitoring device described in 2., the image processing means identifies a time when the owner and the finder are within a predetermined distance; The determination means estimates the relationship using the identified time.

[0122] 11. At least one camera; a monitoring device connected to the camera, The monitoring device an acquisition means for acquiring an image generated by the camera; a detection means for detecting an object separated from its owner and the owner of the object by processing the acquired image, and for detecting a finder who has found the object; a decision means for using the processed result to estimate the relationship between the owner and the finder, and for deciding whether or not a predetermined process needs to be performed depending on the estimated result. 12. One or more computers acquiring an image generated by at least one imaging means; By processing the acquired image, a target object separated from its owner and the owner are detected, and a finder who found the target object is detected; A monitoring method that uses the detection results of the owner and the finder to infer a relationship between the owner and the finder, and determines whether or not a predetermined process needs to be executed depending on the inferred result. 13. Acquiring an image generated by at least one imaging means; image processing for detecting an object separated from its owner and the owner by processing the acquired image, and detecting a finder who found the object; a determination process of estimating a relationship between the owner and the finder using the detection results of the owner and the finder obtained by the image processing, and determining whether or not a predetermined process needs to be executed based on the estimation result; A program that causes a computer to execute the following. 14. Acquiring an image generated by at least one imaging means; image processing for detecting an object separated from its owner and the owner by processing the acquired image, and detecting a finder who found the object; a determination process of estimating a relationship between the owner and the finder using the detection results of the owner and the finder obtained by the image processing, and determining whether or not a predetermined process needs to be executed based on the estimation result; A computer-readable recording medium storing a program for causing a computer to execute the above.

[0123] Furthermore, some or all of the configurations described in Supplementary Notes 2 to 10, which are subordinate to Supplementary Note 1 (monitoring device), may also be subordinate to Supplementary Note 11 (monitoring system), Supplementary Note 12 (monitoring method), Supplementary Note 13 (program), and Supplementary Note 14 (recording medium) in the same subordinate relationship as Supplementary Note 2 to 10. Furthermore, not limited to Supplementary Note 11 to Supplementary Note 14, some or all of the configurations described as appendices may be subordinate to various hardware, software, various recording means for recording software, or systems, within the scope of each of the above-mentioned embodiments. [Explanation of symbols]

[0124] 1. Surveillance System 3. Communication Network 10 owners 20 Target Object 30 Founder 40 Security Guard 100 Monitoring equipment 102 Acquisition Department 104 Image processing section 106 Decision Section 108 Executive Department 120 Storage device 130 Camera Information 132 Image Information 140 Lost and Found Information 150 Detection Information 160 Relationship Information 200 Image processing device 300 Communication Equipment 1000 computers 1010 Bus 1020 processor 1030 memory 1040 Storage Device 1050 Input / Output Interface 1060 Network Interface

Claims

1. an acquisition means for acquiring an image generated by at least one imaging means; an image processing means for processing the acquired image to detect an object separated from its owner and the owner, and to detect a finder who has found the object; a decision means for inferring the relationship between the owner and the finder using the detection results of the owner and the finder by the image processing means, and for deciding whether or not to execute a predetermined process based on the inferred result.

2. 2. The monitoring device according to claim 1, the image processing means identifies a change in at least one of the relative positions and the distance between the owner and the finder; The determining means estimates the relationship using the identified transition.

3. 3. The monitoring device according to claim 1, the detection result includes at least one of a facial orientation and a line of sight of at least one of the owner and the finder; The determination means estimates the relationship using at least one of the face direction and the line of sight.

4. 3. The monitoring device according to claim 1, the detection result includes a facial expression of at least one of the owner and the finder; The determination means estimates the relationship using the facial expression.

5. 3. The monitoring device according to claim 1, the detection result includes at least one of attributes and characteristics of the owner and the finder; The determining means estimates the relationship using at least one of the attribute and the feature.

6. 3. The monitoring device according to claim 1, further comprising an execution unit that executes the predetermined processing when the determination unit determines that the predetermined processing is necessary; The execution means is a monitoring device that notifies a security guard via a communication device that can be recognized by the security guard.

7. 3. The monitoring device according to claim 2, the image processing means identifies a time when the owner and the finder are within a predetermined distance; The determination means estimates the relationship using the identified time.

8. At least one camera; a monitoring device connected to the camera, The monitoring device an acquisition means for acquiring an image generated by the camera; a detection means for detecting an object separated from its owner and the owner of the object by processing the acquired image, and for detecting a finder who has found the object; a decision means for using the processed result to estimate the relationship between the owner and the finder, and for deciding whether or not a predetermined process needs to be performed depending on the estimated result.

9. One or more computers acquiring an image generated by at least one imaging means; By processing the acquired image, a target object separated from its owner and the owner are detected, and a finder who found the target object is detected; A monitoring method that uses the detection results of the owner and the finder to infer a relationship between the owner and the finder, and determines whether or not a predetermined process needs to be executed depending on the inferred result.

10. an acquisition process for acquiring images generated by at least one imaging means; image processing for detecting an object separated from its owner and the owner by processing the acquired image, and detecting a finder who found the object; a determination process of estimating a relationship between the owner and the finder using the detection results of the owner and the finder obtained by the image processing, and determining whether or not a predetermined process needs to be executed based on the estimation result; A program that causes a computer to execute the following.

Images