Monitoring device, monitoring method and program

Abstract

To ensure safety in remote operation.SOLUTION: A monitoring device includes a detection unit for detecting a person candidate from a captured image of a prescribed area, a determination unit for determining whether detection of the person candidate is an erroneous detection or not, and a control unit for regulating remote operation on an operation target device that can be remotely operated if the detection of the person candidate is not an erroneous detection.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a monitoring device, a monitoring method, and a program. [Background technology]

[0002] With the evolution of networks, devices can now be remotely controlled via a network. Patent Document 1 discloses a system for remotely controlling multiple devices on a network. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2007-195111 Summary of the Invention [Problem to be solved by the invention]

[0004] An operator who remotely controls a device may not be able to see the remote environment. Therefore, for example, if a person is near the device being remotely controlled, the person may be put in danger when the device is operated by remote control.

[0005] The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to ensure safety in remote control. [Means for solving the problem]

[0006] A monitoring device according to one aspect of the present invention includes a detection unit that detects human candidates from an image captured of a specified area, a determination unit that determines whether the detection of the human candidate is a false detection, and a control unit that restricts remote operation of a remotely controllable target device if the detection of the human candidate is not a false detection.

[0007] For example, there is a case where a person is performing maintenance work on an operation target device. If the operation target device is remotely controlled while the person is performing the maintenance work on the operation target device, the person may be exposed to danger. If the detection of a human candidate is not a false detection, that is, if a person is detected within a predetermined area, the remote operation of the operation target device can be restricted, thereby preventing the operation target device from being operated by remote control, thereby ensuring the safety of the person.

[0008] The monitoring device includes a storage unit that stores images of objects other than people within the predetermined area, and the determination unit determines whether the detection of the human candidate is a false detection based on feature amounts of the objects and feature amounts of the human candidate. By determining whether the detection of the human candidate is a false detection based on the feature amounts of the objects and the feature amounts of the human candidate, the determination accuracy is improved.

[0009] The determination unit determines a degree of difference between the object and the human candidate based on the feature amount of the object and the feature amount of the human candidate, and determines whether or not the detection of the human candidate is a false detection based on the degree of difference. By determining whether or not the detection of the human candidate is a false detection based on the degree of difference between the object and the human candidate, determination accuracy is improved.

[0010] When the detection of the human candidate is a false detection, the determination unit determines that the image of the human candidate is the image of the object, and stores the image of the object in the storage unit. It is possible.

[0011] The determination unit determines whether the detection of the human candidate is a false positive based on the feature amount of the object for an overlapping area between a rectangular area including the object in the image of the object and a rectangular area including the human candidate in the image of the human candidate, and the feature amount of the human candidate for the overlapping area. By using the feature amount of the object and the feature amount of the human candidate for the overlapping area, it is possible to more accurately determine whether the detection of the human candidate is a false positive.

[0012] The feature amount of the object is a value calculated from pixel values ​​of the image of the object or a value calculated from pixel values ​​of a rectangular region including the object in the image of the object. The feature amount of the human candidate is a value calculated from pixel values ​​of the image of the human candidate or a value calculated from pixel values ​​of a rectangular region including the human candidate in the image of the human candidate. The pixel value may be a pixel value in a grayscale image. The pixel value may be at least one of RGB pixel values ​​in a color image.

[0013] The detection unit detects the human candidate from the captured image by excluding the human candidate that is equal to or smaller than a predetermined size. If the size of the human candidate in the captured image is equal to or smaller than a predetermined size, the human candidate is likely to be an object other than a human. Furthermore, if the size of the human candidate in the captured image is equal to or smaller than a predetermined size and the human candidate is a human, the distance between the human and the device to be operated is likely to be large. If the distance between the human and the device to be operated is large, the human will not be exposed to danger even if the device to be operated operates. By detecting the human candidate from the captured image by excluding the human candidate that is equal to or smaller than a predetermined size, false detections are suppressed and the load of the detection process is reduced.

[0014] The detection unit detects the human candidates from the captured image by excluding the human candidates within a predetermined range set in advance in the captured image. The predetermined range is, for example, a range in which a person cannot exist or a range in which a person is likely not to exist, but is not limited to this range. The human candidates within the predetermined range are likely to be objects other than a person. By detecting the human candidates from the captured image by excluding the human candidates within the predetermined range, false detections are suppressed and the load of the detection process is reduced.

[0015] The detection unit detects the human candidate from the captured image captured by at least one of a visible light camera, a TOF camera, and a thermal camera.

[0016] The determination unit determines whether the detection of the human candidate is a false positive for a plurality of frames, and the control unit restricts the remote control of the operation target device when the detection of the human candidate is not a false positive for a predetermined number of frames or more among the plurality of frames, thereby improving the accuracy of determining the presence or absence of a human within a predetermined area.

[0017] The present invention can also be understood as a monitoring method including at least a part of the above-described processing, a program for causing a computer to execute at least a part of the above-described processing, or a computer-readable recording medium on which such a program is non-temporarily recorded. It can also be understood as a monitoring system including at least a part of the above-described processing. The above configurations and processing can be combined to constitute the present invention as long as no technical contradictions arise. [Effects of the Invention]

[0018] According to the present invention, safety in remote control can be ensured. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 1 is a diagram illustrating the configuration of a monitoring system. [Figure 2]FIG. 2 is a diagram illustrating an example of a hardware configuration of the monitoring device. [Figure 3] FIG. 3 is a diagram illustrating an example of the functional configuration of the monitoring device. [Figure 4] FIG. 4 is a flowchart of the monitoring process. [Figure 5] FIG. 5 shows a comparison image and an image of a person candidate. [Figure 6] FIG. 6 is a diagram showing an example of a kernel in the overlap region for the comparison image. [Figure 7] FIG. 7 is a diagram showing an example of a kernel in an overlap region for an image of a human candidate. [Figure 8] FIG. 8 is a diagram showing an example of an image of a human candidate. DETAILED DESCRIPTION OF THE INVENTION

[0020] Hereinafter, application examples and embodiments will be described with reference to the drawings. The application examples and embodiments described below are aspects of the present application and do not limit the scope of the rights of the present application.

[0021] <Application example> FIG. 1 is a diagram illustrating the configuration of a monitoring system. The monitoring system will be described with reference to FIG. 1. The monitoring system includes a monitoring device (control device) 1 and an imaging device 2 that captures an image of a predetermined area (target area) in a factory or the like. The imaging device 2 may be installed in a fixed position or may be installed so as to be movable. The imaging device 2 is, for example, a visible light camera, a TOF camera, or a thermal camera. The imaging device 2 may also include at least one of a visible light camera, a TOF camera, and a thermal camera. The visible light camera captures an image of a target object within the wavelength range of visible light. The visible light camera may be a camera with a standard lens with a field angle of approximately 45 degrees, or may be a wide-angle camera with a field angle of approximately 60 degrees or more. The wide-angle camera may be an ultra-wide-angle camera such as a fisheye camera, an omnidirectional camera, or a spherical camera. The TOF camera captures an image of the target object and detects the distance to the target object. The thermal camera detects infrared rays emitted from the target object to visualize the temperature distribution of the target object. The number of imaging devices 2 may be one or more. The monitoring device 1 acquires an image of a predetermined area from the imaging device 2. The monitoring device 1 analyzes the image of the predetermined area to detect, recognize, or track a person (human body) within the predetermined area.

[0022] In the example shown in FIG. 1, a device 3 is installed within a predetermined area. The device 3 is a remotely operable device (operation target device). The device 3 may be, for example, a manufacturing device such as a robot. The number of devices 3 may be one or more. The device 3 may operate in response to instructions from a person, such as a worker, within the predetermined area. The device 3 is configured to be remotely operable upon receiving an operation command from outside. For example, the device 3 receives an operation command from an information processing device 4 located at a location away from the predetermined area, whereby the device 3 is remotely operated by the information processing device 4. The information processing device 4 is a device equipped with a communication function, such as a personal computer, a tablet terminal, or a smartphone. The number of information processing devices 4 may be one or more.

[0023] The monitoring device 1 analyzes an image of a predetermined area captured by the imaging device 2, and detects whether or not a person is present within the predetermined area. When the monitoring device 1 detects a person within the predetermined area, it restricts remote operation of the device 3. For example, a person may be performing maintenance work on the device 3. While the person is performing maintenance work on the device 3, if the device 3 is remotely operated and operates, the person may be exposed to danger. By restricting remote operation of the device 3, the device 3 cannot be operated by remote control, and the safety of the person is ensured.

[0024] <Description of the embodiment> Fig. 2 is a diagram illustrating an example of the hardware configuration of the monitoring device 1. The hardware configuration of the monitoring device 1 will be described with reference to Fig. 2. The monitoring device 1 includes a processor 110, a main memory device 120, an auxiliary memory device 130, a communication interface 140, and an output device 150.

[0025] The processor 110 reads out a program stored in the auxiliary storage device 130 into the main storage device 120 and executes it to realize the functions of the functional components shown in Fig. 3. The processor 110 is, for example, a CPU (Central Processing Unit). The memory 10 is, for example, a semiconductor memory such as a RAM (Random Access Memory) or a ROM (Read Only Memory). The auxiliary storage device 130 is, for example, a non-volatile memory such as a hard disk drive or a solid state drive. The communication interface 140 is an interface for performing wired or wireless communication. The output device 150 is, for example, a device for performing output such as a display.

[0026] The monitoring device 1 may be a general-purpose computer such as a personal computer, server computer, tablet terminal, or smartphone, or may be an embedded computer such as an on-board computer. Some of the functions of the monitoring device 1 may be realized by dedicated hardware devices such as ASICs or FPGAs. The monitoring device 1 is connected to the imaging device 2 via a wired or wireless connection and receives image data captured by the imaging device 2. The monitoring device 1 and the imaging device 2 may be integrated, or may be separate devices.

[0027] <Functional configuration> An example of the functional configuration of the monitoring device 1 will be described with reference to Fig. 3. Fig. 3 is a diagram illustrating an example of the functional configuration of the monitoring device 1. The monitoring device 1 has an image acquisition unit 10, a detection unit 11, a position acquisition unit 12, a size acquisition unit 13, a detection result storage unit 14, a template storage unit 15, a determination unit 16, and a control unit 17.

[0028] The image acquisition unit 10 acquires a captured image from the imaging device 2. The image acquisition unit 10 may store the acquired captured image in the auxiliary storage device 130. The detection unit 11 analyzes the captured image and detects human candidates (human body candidates) from the captured image. The predetermined area may be the entire imaging area (imaging range) of the imaging device 2, or may be a part of the imaging area of ​​the imaging device 2. The predetermined area may include the area where the equipment 3 is installed and the area around the equipment 3. The predetermined area may be the area around the equipment 3.

[0029] The detection unit 11 can detect human candidates from the captured image using a human detection algorithm. The human detection algorithm can use, for example, detection using HOG features and VM, detection using Haar-like features based on the brightness difference of the image, or detection using deep learning such as YOLO or SSD. The detection unit 11 stores an image of the detected human candidate in the detection result storage unit 14. The image of the detected human candidate may be, for example, an image of a rectangular area including the human candidate. The rectangular area including the human candidate may be a circumscribing rectangular area surrounding the human candidate.

[0030] The detection unit 11 may detect human candidates from the captured image by excluding human candidates that are equal to or smaller than a predetermined size that is set in advance. If the size of the human candidate in the captured image is equal to or smaller than a predetermined size, there is a high possibility that the human candidate is an object other than a human. Furthermore, if the size of the human candidate in the captured image is equal to or smaller than a predetermined size and the human candidate is a human, there is a high possibility that the distance between the person and the device 3 is large. If the distance between the person and the device 3 is large, the person will not be exposed to danger even if the device 3 operates. By the detection unit 11 detecting human candidates from the captured image by excluding human candidates that are equal to or smaller than a predetermined size, false detections are suppressed and the load of the detection process is reduced. The user can use an information processing device that can communicate with the monitoring device 1 via wired or wireless communication to exclude human candidates. The information processing device may be, for example, a personal computer, a tablet terminal, a smartphone, etc. The user may specify or set the size mask by inputting information about the predetermined size into monitoring device 1.

[0031] The detection unit 11 may detect human candidates from the captured image by excluding human candidates within a predetermined range set in advance in the captured image. The predetermined range is, for example, a range in which a person cannot exist or a range in which a person is likely not to exist, but is not limited to this range. Human candidates within the predetermined range are likely to be objects other than humans. By the detection unit 11 detecting human candidates from the captured image by excluding human candidates within a predetermined range set in advance in the captured image, false detections are suppressed and the load of the detection process is reduced. The user may use an information processing device to specify or set a predetermined range (area mask) for excluding human candidates. The user may specify or set the area mask by inputting information about the predetermined range into the monitoring device 1.

[0032] The position acquisition unit 12 acquires coordinate values ​​of the detection position of the human candidate detected by the detection unit 11. The coordinate values ​​of the detection position are XY coordinates in the captured image, and are, for example, coordinate values ​​of the four corners or the center of a rectangular area including the human candidate detected by the detection unit 11. The position acquisition unit 12 may acquire, as the coordinate values ​​of the detection position, coordinate values ​​in a coordinate system whose origin is the center point of the captured image, or coordinate values ​​in a coordinate system whose origin is one of the four corners of the captured image. The position acquisition unit 12 stores information on the acquired coordinate values ​​of the detection position in the detection result holding unit 14 in association with the image of the human candidate detected by the detection unit 11.

[0033] The size acquisition unit 13 acquires the size (width and height) of a rectangular area including a human candidate detected by the detection unit 11. The size acquisition unit 13 may calculate the size of the rectangular area including the human candidate from the coordinate values ​​of the four corners of the rectangular area. The size acquisition unit 13 stores information about the size of the rectangular area including the human candidate in the detection result holding unit 14 in association with an image of the human candidate.

[0034] The detection result storage unit 14 stores image information of the human candidates. The image information includes, for example, an image (frame) of the human candidate, coordinate values ​​of the detected position of the human candidate, and the size of a rectangular area including the human candidate. Furthermore, the image information of each human candidate may include information on the image capture time or information indicating the image capture order (for example, a frame number).

[0035] The template storage unit 15 stores images of objects as templates (false detection templates) for the detection unit 11 to compare with images of human candidates. The template storage unit 15 can store images of objects other than humans within a predetermined area as templates in advance. The template storage unit 15 can also store images of human candidates (objects) that the detection unit 11 has determined to be falsely detected as additional templates. The template storage unit 15 is an example of a storage unit.

[0036] The determination unit 16 compares the image of the object with the image of the human candidate. The determination unit 16 may acquire the image of the object from the template storage unit 15. The determination unit 16 determines whether or not the detection of the human candidate is a false detection based on the feature amount of the object and the feature amount of the human candidate. By determining whether or not the detection of the human candidate is a false detection based on the feature amount of the object and the feature amount of the human candidate, the determination accuracy is improved. The feature amount of the object may be a value calculated from the pixel values ​​of the image of the object, or may be a value calculated from the pixel values ​​of a rectangular area including the object in the image of the object. The feature amount of the human candidate may be a value calculated from the pixel values ​​of the image of the human candidate, or may be a value calculated from the pixel values ​​of a rectangular area including the human candidate in the image of the human candidate. The pixel value may be a pixel value in a grayscale image. The pixel value may be at least one of the pixel values ​​of RGB in a color image. The feature amount of the object and the feature amount of the human candidate The characteristic amount is not limited to a value calculated from pixel values, but may be other values.

[0037] The determination unit 16 may determine the degree of difference between the object and the human candidate based on the feature amount of the object and the feature amount of the human candidate, and determine whether the detection of the human candidate is an erroneous detection based on the degree of difference between the object and the human candidate. Determining whether the detection of the human candidate is an erroneous detection based on the degree of difference between the object and the human candidate improves the determination accuracy. When the determination unit 16 determines that the detection of the human candidate is an erroneous detection, the determination unit 16 may determine that the image of the human candidate is an image of the object, and store the image of the object in the template holding unit 15. In this way, the image of the human candidate when it is determined that the detection of the human candidate is an erroneous detection can be stored in the template holding unit 15 as an image of the object.

[0038] The determination unit 16 may determine whether the detection of the human candidate is a false positive based on the feature amount of the object for the overlapping area between the rectangular area including the object in the image of the object and the rectangular area including the human candidate in the image of the human candidate, and the feature amount of the human candidate for the overlapping area. By using the feature amount of the object and the feature amount of the human candidate for the overlapping area, it becomes possible to more accurately determine whether the detection of the human candidate is a false positive.

[0039] When the control unit 17 determines that the detection of a human candidate is not an erroneous detection, it restricts remote control of the device 3. That is, when a human is detected within a predetermined area, the control unit 17 restricts remote control of the device 3. Restricting remote control of the device 3 includes disabling all of the multiple types of remote control of the device 3 or disabling some of the multiple types of remote control of the device 3. Restricting remote control of the device 3 includes at least disabling remote control related to the operation of the device 3. For example, the control unit 17 may restrict remote control of the device 3 by disabling remote control related to the operation of the device 3 and enabling remote control of data transmission and reception in the device 3. The control unit 17 may restrict remote control of the device 3 by blocking all or some of the multiple operation commands from the information processing device 4 to the device 3. For example, the control unit 17 may disable remote control of the device 3. Disabling remote control may include disabling all of the multiple types of remote control or disabling some of the multiple types of remote control. When remote operation of the device 3 is disabled, the device 3 does not accept all or some of the multiple types of operation commands from the information processing device 4.

[0040] <Storage process> At least one of the detection unit 11 and the determination unit 16 extracts an image of an object other than a person from the captured image and stores the extracted image of the object as a template in the template storage unit 15. The image of the object may be, for example, an image of a rectangular area including the object. The user may specify the image of the object in the captured image. If the user recognizes that the image of a person candidate detected by the detection unit 11 is an image of an object, the user may specify the image of the person candidate as an image of the object. The user may input information regarding the image specification to the monitoring device 1. The user may send information regarding the image specification to the monitoring device 1 using an information processing device. At least one of the detection unit 11 and the determination unit 16 extracts an image of an object specified by the user from the captured image and stores the extracted image of the object in the template storage unit 15. Storing the image of the object in the template storage unit 15 is performed as a pre-processing when the monitoring device 1 is installed.

[0041] The position acquisition unit 12 acquires coordinate values ​​of the extracted position of the object. The coordinate values ​​of the extracted position of the object are XY coordinates in the captured image, and are, for example, coordinate values ​​of the four corners or the center of a rectangular area including the object. The position acquisition unit 12 may acquire, as the coordinate values ​​of the extracted position of the object, coordinate values ​​in a coordinate system whose origin is the center point of the captured image, or coordinate values ​​in a coordinate system whose origin is one of the four corners of the captured image. The position acquisition unit 12 acquires information on the coordinate values ​​of the extracted position of the object as an extraction coordinate. The template is stored in the template holding unit 15 in association with the image of the object.

[0042] The size acquisition unit 13 acquires the size (width and height) of a rectangular area including the extracted object. The size acquisition unit 13 may calculate the size of the rectangular area including the object from the coordinate values ​​of the four corners of the rectangular area. The size acquisition unit 13 stores information about the size of the rectangular area including the object in the template holding unit 15 in association with an image of the object.

[0043] <Monitoring process> FIG. 4 is a flowchart of the monitoring process (monitoring operation) in the monitoring device 1. The overall flow of the monitoring process will be described with reference to FIG. 4. The flowchart in FIG. 4 illustrates an example of processing that is repeated for each detected human candidate in each frame of an image captured by the imaging device 2. Here, a state in which a human in a predetermined area is not detected is called an undetected state, and a state in which a human in a predetermined area is detected is called a detected state. In the undetected state, undetected state monitoring (first monitoring) is set as the monitoring mode. In the detected state, detected state monitoring (second monitoring) is set as the monitoring mode. At the start of the flowchart in FIG. 4, the state is the undetected state, and undetected state monitoring is set as the monitoring mode.

[0044] In step S101, the image acquisition unit 10 acquires a captured image from the imaging device 2. In step S101, the detection unit 11 analyzes the captured image and detects human candidates. In step S101, the detection unit 11 stores the images of the detected human candidates in the detection result storage unit 14. Also in step S101, the position and size of the detected human candidates are acquired by the position acquisition unit 12 and the size acquisition unit 13, respectively. The information on the position and size of the human candidates is stored in the detection result storage unit 14 as image information together with the human candidate images.

[0045] In step S102, the determination unit 16 compares the comparison image with the image of the human candidate. The comparison image is an image of an object stored in the template storage unit 15. In step S103, the determination unit 16 determines whether the detection of the human candidate is an erroneous detection based on the feature amounts of the object and the feature amounts of the human candidate.

[0046] The process of determining whether or not the detection of a human candidate is an erroneous detection based on the feature amounts of an object and the feature amounts of a human candidate will be described. The determination unit 16 calculates the degree of difference between the comparison image and the image of the human candidate. FIG. 5 shows a comparison image 200 and an image 300 of a human candidate. The determination unit 16 determines whether or not the degree of difference in the overlapping area between the comparison image 200 and the image 300 of the human candidate satisfies a predetermined condition. The comparison image 200 is an image of a rectangular area including an object. The image 300 of the human candidate is an image of a rectangular area including the human candidate. Note that the dotted lines surrounding the comparison image 200 and the image 300 of the human candidate in FIG. 5 indicate the outlines of the captured images.

[0047] The determination unit 16 calculates the sum of squares of the differences in pixel values ​​within a predetermined range in the overlapping region. Fig. 6 shows an example in which the predetermined range in the overlapping region for the comparison image 200 is a 9x9 range (kernel). Fig. 7 shows an example in which the predetermined range in the overlapping region for the human candidate image 300 is a 9x9 range (kernel).

[0048] For example, as shown in Figures 6 and 7, when the predetermined range in the overlapping region is a 9x9 range (kernel), the determination unit 16 calculates the sum of squares D(x, y) of the differences in each pixel value in the predetermined range in the overlapping region based on the following (Equation 1). Calculating the sum of squares D(x, y) makes it possible to deal with fluctuations in the image, improving robustness. In the following (Equation 1), B is the pixel value of the comparison image 200, and C in the following (Equation 1) is the pixel value of the human candidate image 300.

number

[0049] The determination unit 16 compares the sum of squares D(x, y) of the differences between pixel values ​​within a predetermined range in the overlap region with a first threshold (Th1). The determination unit 16 calculates the number of pixels (hereinafter referred to as first pixels) whose sum of squares D(x, y) is equal to or greater than the first threshold (Th1). The first threshold (Th1) may be determined by simulation or machine learning. In FIG. 8, the first pixels in the human candidate image 300 are indicated by hatching.

[0050] The determination unit 16 calculates the ratio (A1) of the number of first pixels to the total number of pixels in a predetermined range, and determines whether the ratio (A1) is equal to or greater than a second threshold (Th2). The second threshold (Th2) may be calculated by simulation or machine learning. If the ratio (A1) is equal to or greater than the second threshold (Th2), the determination unit 16 determines that the degree of difference between the human candidate image and the comparison image satisfies a predetermined condition. If the ratio (A1) is less than the second threshold (Th2), the determination unit 16 determines that the degree of difference between the comparison image and the human candidate image does not satisfy the predetermined condition.

[0051] If the degree of difference between the comparison image and the image of the human candidate satisfies a predetermined condition, the determination unit 16 determines that the detection of the human candidate is not an erroneous detection (step S103: NO), and the process proceeds to step S104. In step S104, the determination unit 16 determines that the image of the human candidate is an image of a person. In this case, the determination unit 16 determines that a person is present within the predetermined area. In step S104, the determination unit 16 or the control unit 17 sets detection status monitoring as the monitoring mode.

[0052] If the degree of difference between the comparison image and the image of the human candidate does not satisfy the predetermined condition, the determination unit 16 determines that the detection of the human candidate is an erroneous detection (step S103: YES), and the process proceeds to step S105. In step S105, the determination unit 16 determines that the image of the human candidate is not an image of a person. In this case, the determination unit 16 determines that no person is present within the predetermined area. The determination unit 16 may determine that the image of the human candidate is an image of an object, and store the image of the object in the template holding unit 15. In step S105, the determination unit 16 or the control unit 17 sets undetected state monitoring as the monitoring mode.

[0053] When the device 3 is in the detected state, the control unit 17 disables remote control of the device 3. When remote control is disabled, the device 3 does not accept all or some of the multiple types of operation commands from the information processing device 4. When the device 3 is in the undetected state, the control unit 17 enables remote control of the device 3. When remote control is enabled, the device 3 accepts operation commands from the information processing device 4.

[0054] The determination unit 16 may determine whether or not the detection of a human candidate is an erroneous detection for multiple frames. The determination unit 16 may determine the presence or absence of a human in a predetermined area based on multiple frames. The determination unit 16 may also determine the presence or absence of a human in a predetermined area by integrating multiple frames. By determining the presence or absence of a human in a predetermined area based on multiple frames, the accuracy of determining the presence or absence of a human in the predetermined area is improved.

[0055] When the image of a human candidate is determined to be an image of a person for a first predetermined number of frames or more out of a plurality of consecutive frames in an undetected state, the determination unit 16 may determine that a person is present in the predetermined area. The plurality of consecutive frames may be frames that are consecutive in time series, or may be frames with consecutive frame numbers. The first predetermined number may be set to an arbitrary value. It is determined.

[0056] For example, if the images of the human candidates are determined to be images of a person for four or more of five consecutive frames in the undetected state, the determination unit 16 may determine that a person is present in the predetermined area. Specifically, the following process is performed.

[0057] The processing of steps S101 to S103 is performed for five consecutive frames, and if the degree of difference between the comparison image and the human candidate image satisfies a predetermined condition for four or more of the five consecutive frames, the determination unit 16 determines that the detection of the human candidate is not an erroneous detection, and the processing proceeds to step S104. In this case, the undetected state transitions to a detected state, and detection state monitoring is set as the monitoring mode. The control unit 17 sets the device 3 to remote operation disabled.

[0058] The processing of steps S101 to S103 is performed for five consecutive frames, and if the degree of difference between the comparison image and the human candidate image does not satisfy the predetermined condition for two or more of the five consecutive frames, the determination unit 16 determines that the detection of the human candidate is an erroneous detection, and the processing proceeds to step S105. In this case, the undetected state is maintained, and the undetected state monitoring is also maintained for the monitoring mode. The control unit 17 maintains the setting of enabling remote control of the device 3.

[0059] If the image of the human candidate is determined to be an image of a person for a second predetermined number or more of consecutive frames in the detection state, the determination unit 16 may determine that a person is present in the predetermined area. The second predetermined number is set to an arbitrary value. The second predetermined number is a value smaller than the first predetermined number. In the detection state, it is considered that a person will not suddenly disappear from the image, so the second predetermined number, which is a value smaller than the first predetermined number, is used as a threshold for determining the presence or absence of a person in the predetermined area.

[0060] For example, if the images of the human candidates are determined to be images of a person for two or more of five consecutive frames in the detection state, the determination unit 16 may determine that a person is present in the predetermined area. Specifically, the following process is performed.

[0061] The processing of steps S101 to S103 is performed for five consecutive frames, and if the degree of difference between the comparison image and the human candidate image satisfies a predetermined condition for two or more of the five consecutive frames, the determination unit 16 determines that the detection of the human candidate is not an erroneous detection, and the processing proceeds to step S104. In this case, the detection state is maintained, and detection state monitoring is also maintained in the monitoring mode. The control unit 17 maintains the setting of disabling remote control of the device 3.

[0062] The processing of steps S101 to S103 is performed for five consecutive frames, and if the degree of difference between the comparison image and the image of the human candidate does not satisfy the predetermined condition for one of the five consecutive frames, the determination unit 16 determines that the detection of the human candidate is an erroneous detection, and the process proceeds to step S105. Furthermore, the processing of steps S101 to S103 is performed for five consecutive frames, and if the degree of difference between the comparison image and the image of the human candidate does not satisfy the predetermined condition for all of the five consecutive frames, the determination unit 16 determines that the detection of the human candidate is an erroneous detection, and the process proceeds to step S105. In these cases, the detection state transitions to an undetected state, and undetected state monitoring is set as the monitoring mode. The control unit 17 sets the device 3 to remote control enabled.

[0063] Although the above describes an example of a process for calculating the degree of difference between the comparison image and the image of the human candidate, the process is not limited to this. The determination unit 16 may compare the comparison image with the image of the human candidate and calculate the degree of similarity between the comparison image and the image of the human candidate. The determination unit 16 may determine whether the degree of similarity between the comparison image and the image of the human candidate satisfies a predetermined condition.

[0064] The monitoring device 1 analyzes an image of a predetermined area captured by the imaging device 2, and if a person is detected within the predetermined area, it restricts remote operation of the device 3. If the device 3 is operated by remote control while a person is performing maintenance work on the device 3, the person may be exposed to danger. According to this embodiment, by restricting remote operation of the device 3, the device 3 cannot be operated by remote control, thereby ensuring the safety of the person.

[0065] A plurality of imaging devices 2 may be installed. The detection unit 11 may analyze captured images of a predetermined area captured by the plurality of imaging devices 2 and detect human candidates from the captured images. By analyzing captured images of a predetermined area captured by the plurality of imaging devices 2, it is possible to more accurately detect the presence or absence of a human in the predetermined area. At least one of the plurality of imaging devices 2 may be a visible light camera. At least one of the plurality of imaging devices 2 may be a TOF camera. At least one of the plurality of imaging devices 2 may be a thermal camera.

[0066] The above-described embodiments merely exemplify exemplary configurations of the present invention. The present invention is not limited to the specific embodiments described above, and various modifications are possible within the scope of the technical concept. Furthermore, the above configurations and processes can be combined with each other to form the present invention as long as no technical contradictions arise. For example, although rectangular regions are exemplified above, regions of shapes other than rectangular (polygonal, elliptical, free-form, etc.) may also be used.

[0067] The present invention can also be understood as a monitoring method including at least a part of the above-mentioned processing, a program for causing a computer to execute at least a part of the above-mentioned processing, or a computer-readable recording medium on which such a program is non-temporarily recorded.

[0068] <Appendix 1> a detection unit (11) that detects human candidates from a captured image of a predetermined area; a determination unit (16) that determines whether the detection of the human candidate is an erroneous detection; a control unit (17) that restricts remote operation of a remotely operable operation target device when the detection of the human candidate is not an erroneous detection; A monitoring device (1) comprising: <Appendix 2> a holding unit (15) for holding an image of an object other than a person within the predetermined area; The determination unit (16) determines whether or not the detection of the human candidate is an erroneous detection based on the feature amount of the object and the feature amount of the human candidate. 1. The monitoring device (1) according to claim 1. <Appendix 3> the determination unit (16) determines a degree of difference between the object and the human candidate based on the feature amount of the object and the feature amount of the human candidate, and determines whether or not the detection of the human candidate is an erroneous detection based on the degree of difference; 2. The monitoring device (1) according to claim 2. <Appendix 4> When the detection of the human candidate is a false detection, the determination unit (16) determines that the image of the human candidate is the image of the object, and stores the image of the object in the storage unit (15). 4. The monitoring device (1) according to claim 2 or 3. <Appendix 5> The determination unit (16) determines the human candidate based on the feature amount of the object in an overlapping area between a rectangular area including the object in the image of the object and a rectangular area including the human candidate in the image of the human candidate, and the feature amount of the human candidate in the overlapping area. Determine whether the detection is a false positive. 5. A monitoring device (1) according to any one of appendices 2 to 4. <Appendix 6> the feature amount of the object is a value calculated from pixel values ​​of the image of the object or a value calculated from pixel values ​​of a rectangular region including the object in the image of the object, The feature amount of the human candidate is a value calculated from pixel values ​​of an image of the human candidate or a value calculated from pixel values ​​of a rectangular region including the human candidate in the image of the human candidate. 6. A monitoring device (1) according to any one of appendices 2 to 5. <Appendix 7> The detection unit (11) detects the human candidates from the captured image by excluding the human candidates that are equal to or smaller than a predetermined size that is set in advance. 7. A monitoring device (1) according to any one of appendices 1 to 6. <Appendix 8> The detection unit (11) detects the human candidates from the captured image by excluding the human candidates within a predetermined range set in advance in the captured image. 8. A monitoring device (1) according to any one of appendices 1 to 7. <Appendix 9> The detection unit (11) detects the human candidate from the captured image captured by at least one of a visible light camera, a TOF camera, and a thermal camera. 9. A monitoring device (1) according to any one of appendices 1 to 8. <Appendix 10> The determination unit (16) determines whether the detection of the human candidate is an erroneous detection for a plurality of frames, the control unit (17) restricts the remote operation of the operation target device when the detection of the human candidate is not an erroneous detection for a predetermined number or more of the plurality of frames. A monitoring device (1) according to any one of appendices 1 to 9. <Appendix 11> a detection step of detecting a human candidate from a captured image of a predetermined area; a determination step of determining whether the detection of the human candidate is an erroneous detection; a control step of restricting remote operation of a remotely controllable operation target device when the detection of the human candidate is not an erroneous detection; A monitoring method comprising: <Appendix 12> On the computer, a detection step of detecting a human candidate from a captured image of a predetermined area; a determination step of determining whether the detection of the human candidate is an erroneous detection; a control step of restricting remote operation of a remotely controllable operation target device when the detection of the human candidate is not an erroneous detection; A program to execute. [Explanation of symbols]

[0069] 1: Monitoring device 2: Imaging device 3:Equipment 4: Information processing equipment 10: Image acquisition unit 11: Detection unit 12: Position acquisition part 13: Size acquisition section 14: Detection result storage unit 15: Template holder 16: Judgment section 17: Control unit

Claims

1. a detection unit that detects human candidates from a captured image of a predetermined area; a determination unit that determines whether the detection of the human candidate is an erroneous detection; a control unit that restricts remote operation of a remotely controllable operation target device when the detection of the human candidate is not an erroneous detection; A monitoring device comprising:

2. a storage unit for storing an image of an object other than a person within the predetermined area; the determination unit determines whether or not the detection of the human candidate is an erroneous detection based on the feature amount of the object and the feature amount of the human candidate. The monitoring device of claim 1 .

3. the determination unit determines a degree of difference between the object and the human candidate based on the feature amount of the object and the feature amount of the human candidate, and determines whether or not the detection of the human candidate is an erroneous detection based on the degree of difference; The monitoring device according to claim 2 .

4. When the detection of the human candidate is a false detection, the determination unit determines that the image of the human candidate is the image of the object, and stores the image of the object in the storage unit. The monitoring device according to claim 2 .

5. the determination unit determines whether or not the detection of the human candidate is an erroneous detection based on the feature amount of the object in an overlapping area between a rectangular area including the object in the image of the object and a rectangular area including the human candidate in the image of the human candidate, and the feature amount of the human candidate in the overlapping area; The monitoring device according to claim 2 .

6. the feature amount of the object is a value calculated from pixel values ​​of the image of the object or a value calculated from pixel values ​​of a rectangular region including the object in the image of the object, The feature amount of the human candidate is a value calculated from pixel values ​​of an image of the human candidate or a value calculated from pixel values ​​of a rectangular region including the human candidate in the image of the human candidate. The monitoring device according to claim 2 .

7. the detection unit detects the human candidates from the captured image by excluding the human candidates that are equal to or smaller than a predetermined size that is set in advance. The monitoring device of claim 1 .

8. the detection unit detects the human candidate from the captured image by excluding the human candidate within a predetermined range set in advance in the captured image. The monitoring device of claim 1 .

9. the detection unit detects the human candidate from the captured image captured by at least one of a visible light camera, a time-of-flight camera, and a thermal camera; The monitoring device of claim 1 .

10. the determination unit determines whether or not the detection of the human candidate is an erroneous detection for a plurality of frames; the control unit restricts the remote operation of the operation target device when the detection of the human candidate is not a false detection for a predetermined number or more of the plurality of frames. A monitoring device according to any one of claims 1 to 9.

11. a detection step of detecting a human candidate from a captured image of a predetermined area; a determination step of determining whether the detection of the human candidate is an erroneous detection; a control step of restricting remote operation of a remotely controllable operation target device when the detection of the human candidate is not an erroneous detection; A monitoring method comprising:

12. On the computer, a detection step of detecting a human candidate from a captured image of a predetermined area; a determination step of determining whether the detection of the human candidate is an erroneous detection; a control step of restricting remote operation of a remotely controllable operation target device when the detection of the human candidate is not an erroneous detection; A program to execute.

Images