Monitoring system, monitoring software, and monitoring method

Abstract

To provide a monitoring system capable of reducing an erroneous detection rate for persons, monitoring software, and a monitoring method.SOLUTION: A monitoring system includes a storage apparatus, an imaging apparatus, and an arithmetic apparatus which acquires a captured image generated by the imaging apparatus. The arithmetic apparatus performs: determination processing for determining whether a candidate of an immobile body is a mobile body or an immobile body when the candidate of the immobile body is detected in the captured image; storage processing for storing the candidate of the immobile body as one of immobile bodies in the storage apparatus when a result of the determination processing indicates the immobile body; collation processing for collating the candidate of the mobile body with one or more immobile bodies stored in the storage apparatus when the candidate of the mobile body is detected in the captured image; and monitoring processing for determining that the candidate of the mobile body is a mobile body when a result of the collation processing indicates absence of a matched immobile body, and monitoring the mobile body. The arithmetic apparatus does not monitor the candidate of the mobile body when the result of the collation processing indicates presence of the matched immobile body, .SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a monitoring system, a monitoring software, and a monitoring method. [Background technology]

[0002] Patent document 1 discloses a danger detection device that detects the posture and bedding height of the person being cared for by analyzing images captured by a camera, determines whether the person being cared for is in a dangerous position based on these data, and issues an alert if it is determined that the person is in a dangerous position. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-10383 Summary of the Invention [Problem to be solved by the invention]

[0004] Pose estimation technology, which detects people and their postures in captured images, can sometimes mistakenly identify non-human objects as people. Such false human detections can be reduced to some extent by removing two-dimensional people (or objects) based on 3D distance information from a TOF camera or other device, or by removing stationary objects using frame-to-frame subtraction. However, two-dimensional person removal cannot remove false positives of three-dimensional objects such as chairs, and frame-to-frame subtraction cannot remove cases where a difference occurs due to something passing in front of a three-dimensional object such as a chair. In these cases, non-human objects may be mistakenly monitored continuously.

[0005] Therefore, in a surveillance system that detects people using posture estimation technology in captured images, there is room for improvement in terms of reducing the rate of false person detection.

[0006] The present invention has been made in view of the above-mentioned current situation, and has as its object to provide a monitoring system, monitoring software, and monitoring method that can reduce the rate of false human detection. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems and achieve the object, (1) a monitoring system according to a first aspect of the present disclosure is a monitoring system comprising a storage device, an imaging device that generates captured images, and an arithmetic device that acquires the captured images generated by the imaging device, wherein the arithmetic device further performs the following steps when a candidate for an unmoving object is detected in the captured image: a determination process for determining whether the candidate for an unmoving object is a moving object or a unmoving object; and if the result of the determination process is that the candidate for an unmoving object is an unmoving object, a storage process for storing the candidate for an unmoving object in the storage device as one of the unmoving objects; a matching process when a candidate for a moving object is detected in the captured image; and a monitoring process for determining that the candidate for a moving object is a moving object and monitoring the moving object if the result of the matching process is that there is a matching unmoving object; and does not monitor the candidate for a moving object if the result of the matching process is that there is a matching unmoving object.

[0008] (2) In the monitoring system described in (1) above, the computing device may further delete the oldest of the stored immovable objects when the number of immovable objects stored in the storage device reaches a predetermined value.

[0009] (3) In the monitoring system described in (1) or (2) above, the calculation device may further perform a second storage process to store the candidate unmoving body that was determined to be a moving body as a result of the determination process and the candidate moving body that was determined to be a moving body as a result of the matching process in the storage device, respectively, as one of the moving bodies; in the determination process, a second matching process may be performed to match the candidate unmoving body with one or more of the moving bodies stored in the storage device; if the result of the second matching process shows that there is no matching moving body, the candidate unmoving body may be determined to be a unmoving body; and if the result of the second matching process shows that there is a matching moving body, the candidate unmoving body may be determined to be a moving body.

[0010] (4) In the monitoring system described in (3) above, the computing device may further delete the oldest of the stored moving objects when the number of the moving objects stored in the storage device reaches a predetermined value.

[0011] (5) In the monitoring system described in any of (1) to (4) above, if the result of the judgment process is an unmoving body, the calculation device may not monitor the candidate unmoving body, and if the result of the judgment process is a moving body, the calculation device may determine that the candidate unmoving body is a moving body and monitor the moving body.

[0012] (6) In the monitoring system described in any one of (1) to (5) above, the computing device may further perform a detection process to detect an object whose pose has been estimated in the captured image, and a primary determination process to determine whether the object detected by the detection process is a candidate for a moving object or a candidate for a stationary object.

[0013] (7) Furthermore, the monitoring software according to the second aspect of the present disclosure is monitoring software that operates on an arithmetic unit connected to a storage device and an image capturing device that generates captured images, and causes the arithmetic unit to perform the following: a determination process for determining whether a candidate for an unmoving object is a moving object or a unmoving object when the candidate for an unmoving object is detected in the captured image; a storage process for storing the candidate for an unmoving object in the storage device as one of the unmoving objects if the result of the determination process is that the candidate for an unmoving object is an unmoving object; a matching process for comparing the candidate for a moving object with one or more of the unmoving objects stored in the storage device when the candidate for a moving object is detected in the captured image; and a monitoring process for determining that the candidate for a moving object is a moving object and monitoring the moving object if the result of the matching process is that there is a matching unmoving object; and does not cause the arithmetic unit to perform the process for monitoring the candidate for a moving object.

[0014] (8) In the monitoring software described in (7) above, the arithmetic device may further be configured to execute a process of deleting the oldest of the stored immovable objects when the number of immovable objects stored in the storage device reaches a predetermined value.

[0015] (9) In the monitoring software described in (7) or (8) above, the arithmetic device may further be caused to execute a second storage process in which the candidate unmoving body determined to be a moving body as a result of the judgment process and the candidate moving body determined to be a moving body as a result of the matching process are each stored as one of the moving bodies in the storage device, and in the judgment process, a second matching process may be executed in which the candidate unmoving body is matched with one or more of the moving bodies stored in the storage device, and if the result of the second matching process shows that there is no matching moving body, the arithmetic device may be caused to execute a process in which the candidate unmoving body is a unmoving body, and if the result of the second matching process shows that there is a matching moving body, the arithmetic device may be caused to execute a process in which the candidate unmoving body is a moving body.

[0016] (10) In the monitoring software described in (9) above, the arithmetic device may further execute a process of deleting the oldest of the stored moving objects when the number of the moving objects stored in the storage device reaches a predetermined value.

[0017] (11) In the monitoring software described in any of (7) to (10) above, if the result of the judgment process is an unmoving body, the arithmetic device may not be made to execute a process of monitoring the candidate unmoving body, and if the result of the judgment process is a moving body, the arithmetic device may be made to judge the candidate unmoving body as a moving body and execute a process of monitoring the moving body.

[0018] (12) In the surveillance software described in any of (7) to (11) above, the arithmetic device may further execute a detection process for detecting an object whose posture has been estimated in the captured image, and a primary determination process for determining whether the object detected by the detection process is a candidate for a moving object or a candidate for a stationary object.

[0019] (13) Furthermore, a monitoring method according to a third aspect of the present disclosure includes an image acquisition step of acquiring a captured image generated by a photographing device; a determination step of determining whether a candidate for an unmoving body is a moving body or a unmoving body when the candidate for an unmoving body is detected in the captured image; a storage step of storing the candidate for an unmoving body in a storage device as one of the unmoving bodies when the result of the determination step is a unmoving body; a comparison step of comparing the candidate for a moving body with one or more of the unmoving bodies stored in the storage device when the candidate for a moving body is detected in the captured image; and a monitoring step of determining that the candidate for a moving body is a moving body and monitoring the moving body when the result of the comparison step is that there is no matching unmoving body.

[0020] (14) The monitoring method described in (13) above may further include a step of deleting the oldest of the stored immovable objects when the number of immovable objects stored in the storage device reaches a predetermined value.

[0021] (15) The monitoring method described in (13) or (14) above may further include a step of performing a second storage process to store the candidate unmoving body determined to be a moving body as a result of the determination process and the candidate moving body determined to be a moving body as a result of the matching process in the storage device, respectively, as one of the moving bodies; the determination process may perform a second matching process to match the candidate unmoving body with one or more of the moving bodies stored in the storage device; if the second matching process results in no matching moving body, the candidate unmoving body may be determined to be a unmoving body; and if the second matching process results in a matching moving body, the candidate unmoving body may be determined to be a moving body.

[0022] (16) The monitoring method described in (15) above may further include a step of deleting the oldest of the stored moving objects when the number of moving objects stored in the storage device reaches a predetermined value.

[0023] (17) The monitoring method described in any of (13) to (16) above may further include a step of determining that the candidate for an unmoving body is a moving body if the result of the determination step is a moving body, and monitoring the moving body.

[0024] (18) The monitoring method described in any one of (13) to (17) above may further include a detection step of detecting an object whose pose has been estimated in the captured image, and a primary determination step of determining whether the object detected by the detection step is a candidate for a moving object or a candidate for a stationary object. [Effects of the Invention]

[0025] According to the present disclosure, it is possible to provide a monitoring system, monitoring software, and monitoring method that can reduce the rate of false human detection. [Brief explanation of the drawings]

[0026] [Figure 1] 1 is a schematic diagram showing the configuration of a monitoring system according to a first embodiment. [Figure 2]4 is a flowchart illustrating a process of storing an unmoving object by the arithmetic device of the monitoring system according to the first embodiment. [Figure 3] 6 is a flowchart illustrating a specific example of a storage process performed by the arithmetic device of the monitoring system according to the first embodiment. [Figure 4] 10A and 10B are schematic diagrams showing examples of candidates for unmoving objects detected in a captured image; [Figure 5] 3 is a schematic diagram showing an example of an unmoving object already stored in a storage device of the surveillance system according to the first embodiment. FIG. [Figure 6] FIG. 10 is a schematic diagram for explaining a method for calculating an area match rate using IoU. [Figure 7] 5 is a flowchart illustrating a process of matching moving object candidates by the arithmetic device of the surveillance system according to the first embodiment. [Figure 8] 5 is a flowchart illustrating a specific example of a matching process performed by the arithmetic device of the monitoring system according to the first embodiment. [Figure 9] 10 is a schematic diagram illustrating a case where a non-human, immobile object is erroneously detected as a candidate moving object, showing a scene in which a jacket draped over a chair is erroneously detected. [Figure 10] 10 is a schematic diagram illustrating a case where a non-human, stationary object is erroneously detected as a candidate moving object, showing a scene in which a person moves while overlapping with a jacket that has been erroneously detected. [Figure 11] 10 is a schematic diagram illustrating a case where a non-human, immobile object is erroneously detected as a candidate moving object, showing a scene in which a person passes by a jacket that has been erroneously detected. [Figure 12] 10 is a flowchart illustrating processing by a computing device of the monitoring system according to the second embodiment. [Figure 13] 10 is a flowchart illustrating a specific example of a monitoring process performed by a computing device of a monitoring system according to a second embodiment.

[0027] Hereinafter, embodiments of a monitoring system, monitoring software, and monitoring method according to the present disclosure will be described in detail with reference to the drawings. The monitoring target of the monitoring system, monitoring software, and monitoring method according to the present disclosure is not particularly limited as long as it is a moving object, but the following description will focus on an example in which a person is the target.

[0028] In the following description, repeated descriptions of components having the same or similar functions will be omitted as appropriate.

[0029] (Embodiment 1) First, an overview of the monitoring system according to the first embodiment will be described. In the monitoring system according to this embodiment, image patterns of non-human, unmoving objects that have been mistakenly detected in the past are registered, and image patterns of moving objects detected thereafter are compared with the registered patterns. If a matching registered pattern is found, the moving object is not considered a monitoring target. This makes it possible to remove objects (e.g., non-human objects) that have been mistakenly detected as moving objects from the monitoring targets.

[0030] FIG. 1 is a schematic diagram showing the configuration of a monitoring system according to the first embodiment.

[0031] A monitoring system 100 according to this embodiment is used in facilities such as nursing homes and hospitals. The monitoring system 100 detects monitored individuals, such as residents and hospitalized patients, as moving objects from captured images and monitors them. As shown in FIG. 1, the monitoring system 100 according to this embodiment includes a storage device 10, an image capture device 20 that generates captured images, and a computing device 30 that acquires the captured images generated by the image capture device 20.

[0032] The image capturing device 20 is a camera installed in the room of the monitored person and captures images of the monitored person within the room area. The image capturing device 20 generates captured images, for example, infrared images, and transmits the generated captured images to the computing device 30 via a communication network. The captured images are moving images. The image capturing and generation of the captured images by the image capturing device 20 may be performed continuously at all times, or may be performed in response to the monitored person entering the capture range.

[0033] A room area is an area where a monitored person resides, is hospitalized, stays for a long period of time, etc. A room area includes both a room where a monitored person stays alone (a private room) and a room where multiple monitored people stay together. If a facility has multiple room areas, a camera device 20 may be installed in each room area, or a camera device 20 may be installed in some of the room areas.

[0034] Furthermore, the image capturing device 20 may be installed in a common area of ​​the facility and may capture images of the monitored person within the common area. A common area is an area where the monitored person is permitted to enter, pass through, or stay, excluding living rooms, and examples thereof include corridors, dining rooms, rehabilitation rooms, recreation rooms, and visiting areas. Multiple image capturing devices 20 may be installed in one common area.

[0035] The computing device 30 is a computer and is placed in an office or the like of the facility. The computing device 30 analyzes the captured image generated by the image capturing device 20, and performs monitoring if the monitored person in the captured image is a moving object.

[0036] The arithmetic device 30 may include a communication unit, a display unit (display device), an input unit, and a control unit.

[0037] The communication unit is a mechanism for transmitting and receiving data to and from the image capturing device 20 and the storage device 10 via a communication network or the like.

[0038] The display unit is a mechanism that displays various screens to present information to facility staff, and specifically includes, for example, a liquid crystal display device.

[0039] The input unit is a mechanism for accepting operational input, and specifically includes, for example, a keyboard and a mouse.

[0040] The control unit is a mechanism for controlling the operation of the arithmetic device 30, and executes various processes described below. In practice, each process is executed by storing a program corresponding to each process, such as a monitoring program, in a ROM or non-volatile memory, and loading and executing the program into the CPU.

[0041] The monitoring program may be pre-installed in the monitoring system 100, or may be provided to the user as an application program that can run on a general-purpose OS, recorded on a computer-readable recording medium, or via a network.

[0042] The storage device 10 is a device for storing data, and is configured with storage devices such as a HDD, nonvolatile RAM, and ROM. The storage device 10 is used to store programs executed by the arithmetic unit 30 (control unit) and to temporarily store data when the control unit operates.

[0043] Next, the processing by the arithmetic device 30 will be described with reference to Figures 2 to 11. The arithmetic device 30 is mainly characterized by performing a process of storing (registering) unmoving objects and a process of matching candidates for moving objects with the stored unmoving objects.

[0044] First, the process of storing an unmoving object will be described with reference to Fig. 2. Fig. 2 is a flowchart illustrating the process of storing an unmoving object performed by the arithmetic device of the surveillance system according to the first embodiment.

[0045] 2, the arithmetic device 30 first acquires a captured image generated by the imaging device 20 (S10), and when a candidate for an unmoving object is detected in the captured image, performs a determination process to determine whether the candidate for an unmoving object is a moving object or a stationary object (S11). This determination process may be performed by comparing the candidate for an unmoving object with a history of past moving objects stored in the storage device 10. This comparison process will be described in detail as a second comparison process in the second embodiment.

[0046] Here, "candidate for unmoving object" means an object that has been detected as an unmoving object at least once before the determination process, such as an unmoving object detected by inter-frame difference before the determination process.

[0047] If the result of the determination process is an unmoving object (S12: unmoving object), the arithmetic device 30 performs a storage process to store (register) the candidate unmoving object as one of the unmoving objects in the storage device 10 (S13). Examples of information to be registered in the storage device 10 include a photographed image of the unmoving object, the posture coordinates of the unmoving object, and the angle (direction) of the imaging device 20 when the unmoving object was photographed.

[0048] As a result of the above, the image pattern of the immovable object (for example, an object other than a person) to be removed can be registered in the storage device 10.

[0049] As will be described later, when the result of the determination process is an unmoving object, the arithmetic device 30 may calculate the matching rate (e.g., area matching rate) between the unmoving object candidate and each unmoving object already stored in the storage device 10, and determine whether or not to store the unmoving object candidate in the storage device 10 according to the calculated matching rate. For example, when there is an unmoving object whose matching rate is equal to or greater than a predetermined threshold, the unmoving object candidate may not be stored in the storage device 10, and when there is no unmoving object whose matching rate is equal to or greater than the threshold, the unmoving object candidate may be stored in the storage device 10 as one of the unmoving objects.

[0050] 2, if the result of the determination process is an unmoving object, the calculation device 30 may not monitor the candidate unmoving object (S14). On the other hand, if the result of the determination process is a moving object (S12: moving object), the calculation device 30 may determine that the candidate unmoving object is a moving object and monitor the moving object (S15).

[0051] The arithmetic device 30 may execute the storage process according to the flow shown in Fig. 3. Fig. 3 is a flowchart illustrating a specific example of the storage process performed by the arithmetic device of the surveillance system according to embodiment 1. Fig. 4 is a schematic diagram showing an example of a candidate for an unmoving object detected in a captured image.

[0052] In this case, as shown in Fig. 4, the calculation device 30 first calculates an area 41 of the candidate immobile object in the captured image (S20). Specifically, the calculation device 30 calculates the coordinates of the smallest rotated rectangle that contains all of the pose-estimated parts (skeleton points). Fig. 4 shows an example in which a jacket (a three-dimensional object) draped over a chair is mistakenly detected.

[0053] Fig. 5 is a schematic diagram showing an example of an unmoving object already stored in the storage device of the surveillance system according to embodiment 1. Fig. 6 is a schematic diagram for explaining a method for calculating an area match rate using IoU.

[0054] Next, the arithmetic device 30 calculates an area matching rate between the calculated area 41 of the candidate immovable object and the area 42 of each immovable object already stored in the storage device 10 as shown in Fig. 5 (S21). Fig. 5 shows an example in which a person (plane) on a poster hung on a wall is stored as an immovable object in the storage device 10. As the area matching rate, for example, IoU (Intersection over Union) can be used as shown in Fig. 6.

[0055] As a result, if there is an immovable object whose area matching rate is equal to or greater than a predetermined threshold (for example, 50%) (S22: Yes), the arithmetic device 30 ends the storage process without storing the candidate immovable object in the storage device 10 (S23), and if there is no immovable object (S22: No), the arithmetic device 30 stores the candidate immovable object in the storage device 10 as one of the immovable objects (S24). In the example shown in FIGS. 4 to 6, the area area between the area area 41 of the erroneously detected jacket and the area area 42 of the person on the registered poster is 0%, so if there are no other immovable objects in the storage device 10, the erroneously detected jacket will be stored in the storage device 10 as an immovable object.

[0056] The arithmetic device 30 may perform the storage process using a FIFO (First In, First Out) method, and if, as a result of storing the immovable objects in S24, the number of immovable objects (items) stored in the storage device 10 reaches a predetermined value (for example, 50 items) (S25: Yes), the oldest immovable object stored in the storage device 10 may be deleted (S26). On the other hand, if the number of immovable objects (items) stored in the storage device 10 is less than the predetermined value (for example, 50 items), the storage process may be terminated without deletion.

[0057] Next, a process for comparing candidates for moving objects with stored unmoving objects will be described. Fig. 7 is a flowchart illustrating the process for comparing candidates for moving objects performed by the arithmetic device of the surveillance system according to the first embodiment.

[0058] 7, when a candidate moving object is detected in a photographed image, the arithmetic device 30 performs a matching process (S30) to match the candidate moving object with one or more immovable objects stored in the storage device 10. As will be described later, this matching process may be performed by calculating the difference between the photographed image of the candidate moving object and the photographed image of the immovable object to be matched.

[0059] Here, a "candidate for a moving object" means an object that has been detected as a moving object at least once before the matching process, such as a moving object detected by inter-frame differences before the matching process.

[0060] If the result of the matching process is that there is no matching stationary object (S31: No), the arithmetic device 30 determines that the candidate moving object is a moving object and performs a monitoring process to monitor the moving object (S32). On the other hand, if the result of the matching process is that there is a matching stationary object (S31: Yes), the arithmetic device 30 does not monitor the candidate moving object (S33). As a result, even if a stationary object is mistakenly detected as a candidate moving object (for example, if a person or the like passes by and is mistakenly detected as a candidate moving object), if the stationary object is registered in the storage device 10, it can be removed from the monitoring targets as a non-moving object (person). This reduces the rate of false person detection by the monitoring system 100.

[0061] Here, in the matching process, a moving object candidate matches an unmoving object stored in the storage device 10, but it does not necessarily mean that the two (their captured images) match perfectly, and it may also mean that the difference between the two captured images is equal to or less than a predetermined threshold. That is, the calculation device 30 may determine that there is a matching unmoving object when the difference between the captured image of the moving object candidate and the captured image of the unmoving object to be matched is equal to or less than a predetermined threshold, and may determine that there is no matching unmoving object when the difference exceeds the predetermined threshold.

[0062] In addition, the immovable objects to be compared at this time may be all immovable objects stored in the storage device 10, or at least some of the immovable objects stored in the storage device 10, for example, immovable objects whose area matching rate with the candidate moving objects is above a predetermined threshold.

[0063] The arithmetic device 30 may execute the matching process according to the flow shown in FIG. 8. FIG. 8 is a flowchart illustrating a specific example of the matching process performed by the arithmetic device of the surveillance system according to the first embodiment. First, a specific example in which a non-human, immovable object is erroneously detected as a candidate for a moving object will be described. FIG. 9 is a schematic diagram illustrating a case in which a non-human, immovable object is erroneously detected as a candidate for a moving object, showing a scene in which a jacket draped over a chair is erroneously detected. FIG. 10 is a schematic diagram illustrating a case in which a non-human, immovable object is erroneously detected as a candidate for a moving object, showing a scene in which a person moves while overlapping with the erroneously detected jacket. FIG. 11 is a schematic diagram illustrating a case in which a non-human, immovable object is erroneously detected as a candidate for a moving object, showing a scene in which a person passes beside the erroneously detected jacket.

[0064] As shown in Figure 9, if a jacket (a three-dimensional object) draped over a chair is mistakenly detected as a person, the jacket is a stationary object and can be removed from the monitoring target if it is left as is. However, as shown in Figures 10 and 11, if a person moves while overlapping with the erroneously detected jacket, i.e., if at least a portion of the jacket is hidden, inter-frame differences occur, and as a result, the jacket is detected as a candidate moving object. In such a case, the person could not be removed from the monitoring target using planar person removal or inter-frame differences, but can be removed as a stationary object using the matching process according to this embodiment.

[0065] In the matching process shown in Fig. 8, the calculation device 30 first calculates an area 43 of a candidate moving object in the captured image (S40) as shown in Fig. 11. Specifically, the coordinates of the smallest rotated rectangle that contains all of the parts (skeleton points) whose postures have been estimated are calculated.

[0066] Next, the arithmetic device 30 calculates the area matching rate between the calculated area 43 of the candidate moving object and the area of ​​each immovable object stored in the storage device 10 (S41). The immovable objects stored in the storage device 10 correspond to, for example, the jacket hanging on the chair shown in FIG. 4 or the person on the poster shown in FIG. 5. The area matching rate can be, for example, IoU.

[0067] Next, the arithmetic device 30 extracts immovable objects having an area matching rate equal to or greater than a predetermined threshold (e.g., 25%) from among the immovable objects stored in the storage device 10 (S42). In the example shown in Fig. 11, the area matching rate with the person on the poster shown in Fig. 5 is 0%, so the immovable object is not extracted, but the area matching rate with the jacket shown in Fig. 4 is about 80%, so the immovable object is extracted.

[0068] At this time, the extracted non-moving objects may be limited to those whose angle of the camera device 20 at the time of shooting matches the angle of the camera device 20 at the time of shooting the candidate moving object (for example, when the difference between the two angles is within a predetermined threshold). In other words, non-moving objects whose angle of the camera device 20 at the time of shooting does not match the angle of the candidate moving object may be excluded from the processing targets in the next step S43 and thereafter.

[0069] Next, the arithmetic unit 30 calculates the difference between the captured image of the candidate moving object and the captured image of each extracted stationary object (S43). Specifically, the arithmetic unit 30 calculates the difference in brightness between the area of ​​the candidate moving object in the captured image and the area of ​​the stationary object in the captured image.

[0070] As a result, if all differences are below a predetermined threshold (S44: Yes), the calculation device 30 determines that there is a matching immovable object and terminates the matching process (S45), and if there is even one difference that exceeds the threshold (S44: No), the calculation device 30 determines that there is no matching immovable object and terminates the matching process (S46).

[0071] In the example shown in Figure 11, the difference between area 43 and area 41 of the jacket shown in Figure 4 is below the threshold, so there is a matching immobile object, i.e., the jacket shown in Figure 11 is determined to be an immobile object and will not be monitored.

[0072] In this embodiment, when performing monitoring processing of a moving object, the computing device 30 may detect the behavior of the moving object from the captured image and determine whether the monitored person is in danger based on the detection result. If it is determined that the monitored person is in danger, the computing device 30 may activate an alarm device to notify the facility manager of the danger. Note that, as a method for detecting the behavior of the moving object from the captured image, for example, a behavior detection model based on machine learning may be used.

[0073] The alarm device is a device that notifies facility staff and the like that a monitored person is in a dangerous state, and is placed in the facility's staff room, nurse station, etc. The alarm device may be an LCD display device, an information terminal carried by the staff (such as a smartphone or tablet), a speaker that emits sound, or a light that emits light. The alarm device issues an alert to staff about a monitored person that is determined to be in a dangerous state. The alert may be issued by displaying an alert on the display unit of the alarm device, or by sound, light emission, vibration, etc.

[0074] As described above, the monitoring system 100 according to this embodiment can reduce the rate of false human detection.

[0075] (Embodiment 2) The monitoring system of embodiment 2 differs from the monitoring system of embodiment 1 in that the calculation device 30 further performs a process of detecting candidates for moving or stationary objects from the captured image before the judgment process, and a process of storing (registering) objects judged to be moving objects, and also performs a process of comparing candidates for stationary objects with the stored moving objects in the judgment process.

[0076] In this embodiment, the image capturing device 20 includes a depth sensor such as an FOT camera in addition to an image sensor to acquire the three-dimensional coordinates of the joints. The depth sensor recognizes the position and depth (depth) of the subject. The depth sensor may be a pattern illumination type sensor.

[0077] The processing by the arithmetic device 30 in this embodiment will be described with reference to Fig. 12 and Fig. 13. Fig. 12 is a flowchart illustrating the processing by the arithmetic device of the monitoring system according to embodiment 2. In Fig. 12, the same reference numerals are used to denote processing that is common to embodiment 1.

[0078] As shown in FIG. 12, in this embodiment, after acquiring a captured image generated by the image capture device 20 (S10), the computing device 30 first performs a detection process to detect an object whose pose has been estimated in the captured image (S50). In particular, an object estimated to be a person is detected from the captured image using a person detection model based on machine learning, and multiple skeleton points of the object are extracted for each time-series frame, and the coordinate displacements of the skeleton points over time are estimated. The coordinates of each skeleton point may be three-dimensional coordinates including depth (distance) information based on a depth sensor. Note that the object detected by this detection process may be an actual person or a non-human object.

[0079] Next, the arithmetic device 30 performs a tracking process for the object detected by the detection process (S51). Specifically, if the detected object is an object that has already been detected, the location information of the detected object is added to the registered ID. If the detected object is an object that has not yet been detected, a new ID is assigned and the location information of the detected object is added to that ID.

[0080] Next, the arithmetic device 30 performs a primary determination process to determine whether the object detected by the detection process is a moving object candidate or a stationary object candidate (S52). In detail, the arithmetic device 30 determines whether the detected object is a moving object or a stationary object by taking the difference between frames of the captured image, and sets the detected object as a moving object candidate or a stationary object candidate, respectively.

[0081] If the result of the primary determination process is that the candidate is a moving object (S53: moving object), the calculation device 30 performs the matching process described in the first embodiment (S30). That is, the candidate is matched with one or more unmoving objects stored in the storage device 10.

[0082] If the result of the primary determination process is that the object is a candidate for an unmoving object (S53: moving object), the arithmetic device 30 performs the determination process described in the first embodiment (S11). That is, it determines whether the candidate for an unmoving object is a moving object or a unmoving object. In this way, in this embodiment, the arithmetic device 30 determines whether the detected object is a moving object or a unmoving object in the primary determination process and the subsequent determination process (secondary determination process).

[0083] In addition, the calculation device 30 performs a second storage process (S54) in which the candidate stationary object determined to be a moving object as a result of the judgment process (S12: moving object) and the candidate moving object determined to be a moving object as a result of the matching process (S31: moving object) are stored (registered) as one of the moving objects in the storage device 10.

[0084] Specifically, for example, the activity amount of the moving object is stored as a person activity amount in an ID determined in the object tracking process. Here, the person activity amount is a value based on the image difference amount calculated in the primary determination process or the matching process. The person activity amount is updated, for example, at 5 FPS. Person activity (after update) = Person activity (before update) + Image difference - Activity decay constant The activity amount attenuation constant is a constant obtained experimentally. In the primary determination process, the calculation device 30 calculates this human activity amount, and if the human activity amount is equal to or greater than a predetermined threshold, determines that the object is a candidate for a moving object, and if it is less than the threshold, determines that the object is a candidate for a stationary object.

[0085] The calculation device 30 may perform the second storage process using a FIFO method, and when the number of moving objects (items) stored in the storage device 10 reaches a predetermined value as a result of storing the moving objects in S54, the oldest moving object stored in the storage device 10 may be deleted. On the other hand, when the number of moving objects (items) stored in the storage device 10 is less than the predetermined value, the second storage process may be terminated without deletion.

[0086] Then, in the determination process (secondary determination process) of S11, the arithmetic device 30 performs a second matching process to match the candidate unmoving object with one or more moving objects stored in the storage device. If the result of the second matching process shows that there is no matching moving object, the arithmetic device 30 determines that the candidate unmoving object is an unmoving object (S12: unmoving object). If the result of the second matching process shows that there is a matching moving object, the arithmetic device 30 determines that the candidate unmoving object is a moving object (S12: moving object). In detail, the arithmetic device 30 checks whether an object having the same ID as the candidate unmoving object is stored in the storage device 10 as a moving object (an object whose human activity amount is equal to or greater than a predetermined threshold), and determines that there is a matching moving object if there is a history of such a moving object, and determines that there is no matching moving object if there is no history of such a moving object.

[0087] In this embodiment, the arithmetic device 30 performs a moving object monitoring process after the second storage process (S55).

[0088] The arithmetic device 30 may execute the monitoring process (S55) according to the flow shown in Fig. 13. Fig. 13 is a flowchart illustrating a specific example of the monitoring process by the arithmetic device of the monitoring system according to the second embodiment.

[0089] In this case, the computing device 30 first checks the reliability of the 3D coordinates of the moving object (S60). Specifically, based on the 3D landmark information of the moving object acquired by the depth sensor, the computing device 30 determines whether the object is appropriate as a person. For example, if the moving object exists on a flat surface or if the moving object is exhibiting an unpredictable posture, the computing device 30 determines that the moving object is unreliable. If the 3D coordinates of the moving object are reliable (S61: Yes), the computing device 30 may detect the behavior of the moving object from the captured image and determine whether the monitored person is in danger based on the detection result (S62). On the other hand, if the 3D coordinates of the moving object are unreliable (S61: Yes), the computing device 30 may continue to monitor the moving object without detecting its behavior (S63).

[0090] In this embodiment, as in the first embodiment, the rate of false human detection can be reduced.

[0091] Although the embodiments have been described above with reference to the drawings, the present disclosure is not limited to the above-described embodiments. Furthermore, the configurations of the respective embodiments may be appropriately combined or modified without departing from the spirit and scope of the present disclosure. [Industrial Applicability]

[0092] As described above, the present disclosure provides a technology useful for reducing the false human detection rate in a surveillance system that uses captured images. [Explanation of symbols]

[0093] 100: Surveillance system 10:Storage device 20:Photography equipment 30: Arithmetic device 41, 42, 43: Area area

Claims

1. A storage device; an imaging device that generates a captured image; a computing device that acquires the captured image generated by the imaging device, The computing device further includes a detection process for detecting an object whose pose has been estimated in the captured image; a tracking process of the object detected by the detection process; a primary determination process for determining whether the object detected by the detection process is a moving object candidate or a stationary object candidate; a determination process for determining whether a candidate for an unmoving object is a moving object or a unmoving object when the candidate for an unmoving object is detected in the captured image; a storage process for storing the candidate for the unmoving object in the storage device as one of the unmoving objects when the result of the determination process is an unmoving object; a comparison process for comparing, when a candidate moving object is detected in the captured image, the candidate moving object with one or more of the unmoving objects stored in the storage device; If there is no matching unmoving object as a result of the comparison process, a monitoring process is performed in which the candidate moving object is determined to be a moving object and the moving object is monitored; If there is a matching non-moving object as a result of the matching process, the candidate moving object is not monitored; In the primary determination process, the arithmetic device calculates a person activity amount based on an image difference amount, which is an inter-frame difference of the captured image, and an experimentally determined activity amount attenuation constant, and updates the person activity amount based on the calculated person activity amount, the image difference amount, and the activity amount attenuation constant, and determines that the object detected by the detection process is a candidate for a moving object if the updated person activity amount is equal to or greater than a predetermined threshold, and determines that the object detected by the detection process is a candidate for a non-moving object if the updated person activity amount is less than the predetermined threshold. A monitoring system characterized by:

2. The monitoring system described in claim 1, characterized in that the computing device further deletes the oldest of the stored unmoving objects when the number of unmoving objects stored in the storage device reaches a predetermined value.

3. the computing device further performs a second storage process to store the candidate for the non-moving object determined to be a moving object as a result of the determination process and the candidate for the moving object determined to be a moving object as a result of the matching process in the storage device as one of the moving objects; In the determination process, a second matching process is performed to match the candidate unmoving object with one or more moving objects stored in the storage device; If there is no matching moving object as a result of the second matching process, the candidate for the non-moving object is determined to be a non-moving object; 3. The monitoring system according to claim 1, wherein if a matching moving object is found as a result of the second comparison process, the candidate for the unmoving object is determined to be a moving object.

4. The monitoring system described in claim 3, characterized in that the computing device further deletes the oldest of the stored moving objects when the number of moving objects stored in the storage device reaches a predetermined value.

5. When the result of the determination process is an unmoving object, the calculation device does not monitor the unmoving object candidate, 5. The monitoring system according to claim 1, wherein, when the result of the determination process indicates a moving object, the candidate for the unmoving object is determined to be a moving object, and the moving object is monitored.

6. A monitoring system described in any one of claims 1 to 5, characterized in that in the primary judgment process, the arithmetic device calculates the updated person activity amount by adding the image difference amount to the person activity amount before the update and subtracting the activity amount attenuation constant.

7. A monitoring software that runs on a computing device connected to a storage device and an image capturing device that generates captured images, a detection process for detecting an object whose pose has been estimated in the captured image, in the computing device; a tracking process of the object detected by the detection process; a primary determination process for determining whether the object detected by the detection process is a moving object candidate or a stationary object candidate; a determination process for determining whether a candidate for an unmoving object is a moving object or a unmoving object when the candidate for an unmoving object is detected in the captured image; a storage process for storing the candidate for the unmoving object in the storage device as one of the unmoving objects when the result of the determination process is an unmoving object; a comparison process for comparing, when a candidate moving object is detected in the captured image, the candidate moving object with one or more of the unmoving objects stored in the storage device; If there is no matching unmoving object as a result of the comparison process, the candidate moving object is determined to be a moving object, and a monitoring process is executed to monitor the moving object; If there is a matching unmoving object as a result of the matching process, the process of monitoring the moving object candidate is not executed, In the primary determination process, a person activity amount is calculated based on an image difference amount, which is an inter-frame difference of the captured image, and an experimentally determined activity amount attenuation constant, and the person activity amount is updated based on the calculated person activity amount, the image difference amount, and the activity amount attenuation constant, and if the updated person activity amount is equal to or greater than a predetermined threshold, the object detected by the detection process is determined to be a candidate for a moving object, and if the updated person activity amount is less than the predetermined threshold, the object detected by the detection process is determined to be a candidate for a non-moving object. The monitoring software is characterized by:

8. an image acquisition step of acquiring a photographed image generated by the photographing device; a detection step of detecting an object whose pose has been estimated in the captured image; a tracking step of the object detected by the detecting step; a primary determination step of determining whether the object detected by the detection step is a moving object candidate or a non-moving object candidate; a determination step of determining whether a candidate for an unmoving object is a moving object or a unmoving object when the candidate for an unmoving object is detected in the captured image; a storage step of storing the candidate for the unmoving object in a storage device as one of the unmoving objects when the result of the determination step is an unmoving object; a matching step of matching a candidate moving object with one or more of the unmoving objects stored in the storage device when the candidate moving object is detected in the captured image; a monitoring step of determining that the candidate moving object is a moving object and monitoring the moving object when there is no matching unmoving object as a result of the comparing step, In the primary determination step, a person's activity amount is calculated based on an image difference amount, which is a difference between frames of the captured image, and an experimentally determined activity amount attenuation constant, and the person's activity amount is updated based on the calculated person's activity amount, the image difference amount, and the activity amount attenuation constant. If the updated person's activity amount is equal to or greater than a predetermined threshold, the object detected in the detection step is determined to be a candidate for a moving object, and if the updated person's activity amount is less than the predetermined threshold, the object detected in the detection step is determined to be a candidate for a non-moving object. A monitoring method comprising:

Images