Monitoring device, monitoring system, monitoring method, and monitoring program

Abstract

This monitoring device (300) comprises: a tracking unit that detects a moving body (10) on the basis of one or more images acquired by one or more imaging devices (70) provided at different positions from the moving body (10) being tracked; a first setting unit that sets a virtual viewpoint according to the movement of the moving body (10); a generation unit that generates a virtual image that is from the virtual viewpoint on the basis of the one or more images; and a detection unit that detects a detection target in the virtual image.

Description

Monitoring device, monitoring system, monitoring method, and monitoring program

[0001] The present disclosure relates to a monitoring device, a monitoring system, a monitoring method, and a monitoring program.

[0002] In order to detect an object approaching a vehicle, there is a technology that detects objects around the vehicle using a LiDAR (Light Detection and Ranging) or a camera mounted on the vehicle.

[0003] For example, the system disclosed in International Publication No. 2019 / 038914 (Patent Document 1) converts an image captured by a camera mounted on a vehicle into a virtual image captured from a virtual viewpoint, and displays the virtual image as a surrounding image showing the situation around the vehicle.

[0004] Japanese Patent Laid-Open Publication No. 2009-211335 (Patent Document 2) discloses a technique for using an image of a subject captured by a camera to generate a virtual viewpoint image of the subject as viewed from a viewpoint different from that of the camera.

[0005] International Publication No. 2019 / 038914 Japanese Patent Application Laid-Open No. 2009-211335

[0006] The method disclosed in Patent Document 1 has a problem in that it cannot detect a target, such as a person or an object, if the target is located in a blind spot of a moving object such as a vehicle. Furthermore, Patent Document 2 does not propose a solution to this problem.

[0007] An object of the present disclosure is to be able to detect a detection target even when the detection target is located in a blind spot of a moving object.

[0008] A monitoring device according to one aspect of the present disclosure includes a tracking unit that detects a moving object based on one or more images acquired by one or more imaging devices provided at a location separate from the moving object to be tracked, a first setting unit that sets a virtual viewpoint in accordance with the movement of the moving object, a generation unit that generates a virtual image viewed from the virtual viewpoint based on the one or more images, and a detection unit that detects the object to be tracked on the virtual image.

[0009] Preferably, the first setting unit sets the virtual viewpoint to a position directly above the moving object.

[0010] Preferably, the first setting unit changes the virtual viewpoint when the reliability of the detection target detected by the detection unit is less than a predetermined value.

[0011] Preferably, the monitoring device further includes a second setting unit that sets an alarm area on the virtual image, and a determination unit that determines whether the position on the virtual image of the detection target detected by the detection unit is included in the alarm area.

[0012] Preferably, the monitoring device further includes a second setting unit that sets an alarm area in the space of the world coordinate system, and a determination unit that determines whether the position in the world coordinate system of the detection object detected by the detection unit is included in the alarm area.

[0013] Preferably, the second setting unit sets the warning area in accordance with a position of the moving object.

[0014] Preferably, the second setting unit sets a predetermined area as the warning area.

[0015] A monitoring system according to another aspect of the present disclosure includes the above-described monitoring device and an alarm device that outputs a warning notice based on the determination result of the determination unit.

[0016] A surveillance method according to another aspect of the present disclosure, executed by one or more computers, includes detecting a moving object based on one or more images captured by one or more image capture devices provided at locations separate from the moving object to be tracked, setting a virtual viewpoint to track the movement of the moving object, generating a virtual image viewed from the virtual viewpoint based on the one or more images, and detecting a detection target in the virtual image.

[0017] A monitoring program according to another aspect of the present disclosure causes one or more computers to execute the above monitoring method.

[0018] According to the present disclosure, it is possible to detect a detection target even when the detection target is located in a blind spot of a moving object.

[0019] 6 is a diagram showing an example of a monitoring system in embodiment 1 and an example of an area to be monitored by the monitoring system. FIG. 7 is a diagram showing an example of an alarm area in embodiment 1. FIG. 8 is a diagram showing an example of a hardware configuration of a monitoring device in embodiment 1. FIG. 9 is a diagram showing an outline of the processing of the monitoring device in embodiment 1. FIG. 10 is a flowchart showing the procedure of monitoring processing by the monitoring device in embodiment 1. FIG. 11 is a diagram showing an example of a monitoring system and an example of an area to be monitored by the monitoring system in embodiment 2. FIG. 12 is a diagram showing the monitored area shown in FIG. 6 as viewed from directly above. FIG. 13 is a diagram showing an outline of the processing of the monitoring device in embodiment 2. FIG. 14 is a diagram showing an example of a virtual image in embodiment 2. FIG. 15 is a flowchart showing part of the procedure of monitoring processing by the monitoring device in embodiment 2. FIG. 16 is a flowchart showing the remaining part of the procedure of monitoring processing by the monitoring device in embodiment 2. FIG. 17 is a diagram showing a first modified example regarding the type of lens of each of a plurality of cameras and the arrangement of a plurality of cameras. FIG. 18 is a diagram showing a second modified example regarding the type of lens of each of a plurality of cameras and the arrangement of a plurality of cameras.

[0020] Hereinafter, embodiments and modifications according to the present disclosure will be described with reference to the drawings. In the following description, the same parts and components are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed descriptions thereof will not be repeated. Note that the embodiments and modifications described below may be selectively combined as appropriate.

[0021] First Embodiment FIG. 1 is a diagram showing an example of a monitoring system according to a first embodiment and an example of an area to be monitored by the monitoring system.

[0022] The monitoring system 1 in the first embodiment includes a plurality of cameras 70, a monitoring device 300, and an alarm device 500. The plurality of cameras 70 is an example of "one or more imaging devices" in the present disclosure. In the example shown in FIG. 1 , the plurality of cameras 70 includes two cameras (camera 71a and camera 71b). However, the number of the plurality of cameras 70 may be three or more. The plurality of cameras 70, the monitoring device 300, and the alarm device 500 are connected to each other via a network. The network may be a wired network or a wireless network.

[0023] The monitoring system 1 tracks the moving object 10 and sets an alert area in accordance with the movement of the moving object 10. The moving object 10 is a tracking target of the monitoring system 1. The moving object 10 is, for example, a work vehicle such as a forklift, an AMR (Autonomous Mobile Robot), or a drone. In the example shown in FIG. 1 , the moving object 10 is a forklift. The moving object 10 travels in a monitoring target area 51. In the example shown in FIG. 1 , the moving object 10 moves horizontally in the monitoring target area 51. The monitoring target area 51 is, for example, a predetermined area such as a factory, a warehouse, or an office.

[0024] The monitoring system 1 generates a virtual image viewed from a virtual viewpoint based on images obtained from the multiple cameras 70, and detects a detection target in the virtual image. The virtual image has a different viewpoint from the images obtained from the multiple cameras 70. Each of the multiple cameras 70 is a network camera (IP (Internet Protocol) camera). The type of lens of each of the multiple cameras 70 is, for example, a normal lens. The detection target includes at least one of a person and an object. In the example shown in FIG. 1 , a person 21 is detected as the detection target. The monitoring system 1 outputs a warning notification when the position of the detection target is included in the warning area.

[0025] More specifically, the multiple cameras 70 are provided at positions separate from the mobile object 10. In the example shown in Fig. 1 , the multiple cameras 70 are fixed to the ceiling 15 of the monitored area 51. The ceiling 15 is high enough to enable the multiple cameras 70 to obtain a bird's-eye view of the monitored area 51. Each of the multiple cameras 70 captures images of the monitored area 51 and acquires video of the monitored area 51. Each of the multiple cameras 70 compresses the acquired video data using an image compression technology such as H.264 and transmits the compressed video data (video stream) to the monitoring device 300.

[0026] The monitoring device 300 receives multiple images from multiple cameras 70. The monitoring device 300 detects the moving object 10 based on the multiple images. The monitoring device 300 selects two cameras (camera 71a and camera 71b in the example shown in FIG. 1 ) from the multiple cameras 70 that are suitable for capturing the moving object 10, and sets the two cameras as a stereo camera pair to be used for stereo matching. Based on the images from the two cameras, the monitoring device 300 identifies the position of the moving object 10 and generates a virtual image viewed from a virtual viewpoint. The monitoring device 300 detects a detection target in the virtual image and sets an alarm area in the virtual image.

[0027] The monitoring device 300 sets a virtual viewpoint by following the movement of the moving object 10. The monitoring device 300 sets the virtual viewpoint at a position that provides a bird's-eye view of the moving object 10 and its surroundings. In the example shown in FIG. 1 , the moving object 10 moves horizontally, and the area to be monitored 51 has good visibility from above. In such a case, the monitoring device 300 sets the virtual viewpoint to a position A1 directly above the moving object 10. That is, the monitoring device 300 sets the point at infinity directly above the moving object 10 as the virtual viewpoint.

[0028] By setting the virtual viewpoint in this way, it is possible to detect the detection target and determine the proximity of the detection target to the moving body 10 on the same virtual image. The monitoring device 300 detects the detection target on the virtual image and sets a warning area on the virtual image.

[0029] Now, with reference to FIG. 2, the warning area in embodiment 1 will be described. FIG. 2 is a diagram showing an example of a warning area in embodiment 1. A virtual image V1 is an example of a virtual image when the virtual viewpoint is set to position A1 shown in FIG. 1. The monitoring device 300 sets a warning area on the virtual image V1. The monitoring device 300 sets the warning area according to the position of the moving body 10. More specifically, the monitoring device 300 sets an area within a predetermined range with the position of the moving body 10 as the base point as the warning area. In the example shown in FIG. 2, the monitoring device 300 sets an area P1 within a radius of n meters centered on the position of the moving body 10 as the warning area. n is a number greater than zero.

[0030] Referring again to FIG. 1 , the monitoring device 300 determines whether the moving body 10 and the detection target are in proximity to each other on the virtual image. More specifically, the monitoring device 300 determines whether the position of the detection target on the virtual image is included in the warning area. If the position of the detection target on the virtual image is included in the warning area, the monitoring device 300 determines that the moving body 10 and the detection target are in proximity to each other and transmits an instruction to the alarm device 500 to output a warning notification. Referring again to FIG. 2 , in the example shown in FIG. 2 , the position of the person 21 on the virtual image V1 is included in area P1, which is the warning area. Therefore, the monitoring device 300 transmits an instruction to output a warning notification to the alarm device 500.

[0031] 1 , the alarm device 500 includes a display 501 and a speaker 502. The alarm device 500 outputs a warning notification when it receives an instruction to output a warning notification. As an example, when the alarm device 500 receives an instruction to output a warning notification, the display 501 displays a warning screen and the speaker 502 outputs a warning sound. The warning screen is a screen that notifies the user that a detection target is present within the warning area. As an example, the display 501 displays a virtual image as the warning screen, and highlights the detection target on the virtual image so that the user can see where the detection target is located within the warning area.

[0032] 1 , a person 21, who is the detection target, is hidden from the person 11 driving the mobile body 10 by a wall 31. In other words, the person 21, who is the detection target, is located in a blind spot of the mobile body 10. Therefore, if a camera is installed on the mobile body 10, the person 21, who is the detection target, cannot be detected from the video captured by the camera. However, in the surveillance system 1, multiple cameras 70 are installed in positions separate from the mobile body 10. Therefore, the detection target can be detected even if it is located in a blind spot of the mobile body 10.

[0033] 3 is a diagram illustrating an example of the hardware configuration of the monitoring device according to embodiment 1. The monitoring device 300 includes a processor 301, a memory 302, a storage 303, and a communication interface 304. The processor 301, the memory 302, the storage 303, and the communication interface 304 are connected via a bus 399.

[0034] The processor 301 is, for example, a central processing unit (CPU). The processor 301 reads a program 311 stored in a storage 303, loads the program 311 into a memory 302, and executes the program 311.

[0035] The memory 302 is configured as a volatile storage device such as a dynamic random access memory (DRAM) or a static random access memory (SRAM).

[0036] The storage 303 is configured by a non-volatile storage device such as a hard disk drive (HDD), a solid state drive (SSD), or a flash memory.

[0037] The storage 303 stores a program 311. The program 311 includes a plurality of computer-readable instructions for controlling the monitoring device 300. The program 311 includes a "monitoring program" in this disclosure. By executing the program 311, the processor 301 controls each unit of the monitoring device 300 and realizes various processes according to this embodiment.

[0038] The program 311 may be provided not as a standalone program but as part of an arbitrary program. In this case, the program 311 cooperates with the arbitrary program to realize the processing according to this embodiment. Even if the program does not include some of these modules, this does not deviate from the spirit of the monitoring device 300 according to this embodiment. Furthermore, some or all of the functions provided by the program 311 may be realized by dedicated hardware.

[0039] The storage 303 stores camera parameters 312 of the multiple cameras 70. The camera parameters 312 include internal parameters of the multiple cameras 70 and external parameters of the multiple cameras 70.

[0040] The storage 303 stores video data 313 received from the multiple cameras 70. The video data 313 is, for example, a video stream.

[0041] The communication interface 304 exchanges data with devices connected to the network (such as the multiple cameras 70 and the alarm device 500) via the network.

[0042] 4 is a diagram showing an overview of the processing of the monitoring device in embodiment 1. Monitoring device 300 includes video processing unit 351, tracking unit 352, first setting unit 353, generation unit 354, detection unit 355, second setting unit 356, and determination unit 357. Video processing unit 351, tracking unit 352, first setting unit 353, generation unit 354, detection unit 355, second setting unit 356, and determination unit 357 are realized by processor 301 executing program 311.

[0043] The video processing unit 351 receives video from the multiple cameras 70. That is, the video processing unit 351 receives multiple video streams from the multiple cameras 70. The video processing unit 351 extracts one or more frame images from each video stream. The video processing unit 351 associates the multiple frame images extracted from the multiple video streams by their capture times.

[0044] It should be noted that if images captured at the same time arrive at the image processing unit 351 at approximately the same time and the difference in the image capturing time can be ignored, then the image processing unit 351 does not need to perform the process of associating frame images.

[0045] The tracking unit 352 detects the moving object 10 and identifies the position of the moving object 10 based on multiple images captured by multiple cameras 70.

[0046] More specifically, first, the tracking unit 352 detects the moving body 10 from a plurality of frame images extracted by the image processing unit 351 and associated by the shooting time using object detection AI (Artificial Intelligence).

[0047] Next, the tracking unit 352 selects two cameras from the multiple cameras 70 that are suitable for capturing the moving object 10, and sets the two cameras as a pair of stereo cameras to be used for stereo matching. In the example shown in Fig. 1, the tracking unit 352 sets the camera 71a and the camera 71b as a pair of stereo cameras.

[0048] Next, the tracking unit 352 uses a known position estimation technique using stereo matching to identify the position of the moving object 10 in real space. More specifically, the tracking unit 352 identifies the position of the moving object 10 in real space based on a plurality of frame images extracted from two videos captured by two cameras set as a stereo camera pair and the camera parameters of the two cameras among the camera parameters 312.

[0049] The first setting unit 353 sets a virtual viewpoint in accordance with the movement of the moving body 10. The first setting unit 353 sets the virtual viewpoint at a position that provides a bird's-eye view of the moving body 10 and the area around the moving body 10. As in the example shown in Fig. 1 , when the moving body 10 moves horizontally and the monitoring target area 51 has good visibility from above, the first setting unit 353 sets the virtual viewpoint to a position directly above the moving body 10. In other words, the first setting unit 353 sets the point at infinity directly above the moving body 10 as the virtual viewpoint.

[0050] The generation unit 354 generates a virtual image seen from a virtual viewpoint based on a plurality of images acquired by a plurality of cameras 70 .

[0051] More specifically, the generation unit 354 generates a virtual video viewed from the virtual viewpoint set by the first setting unit 353 based on a plurality of frame images extracted from two videos acquired by two cameras set as a stereo camera pair. Note that the generation unit 354 may generate a virtual video viewed from the virtual viewpoint set by the first setting unit 353 using videos from all of the cameras. Any existing technology is used to generate the virtual video. An example of the existing technology is an image generation technology using homography transformation.

[0052] When an image generation technique using homography transformation is used to generate a virtual image, the generation unit 354 uses epipolar geometry to extract a feature point (hereinafter referred to as "feature point 1") of a frame image of an image captured by one of two cameras (e.g., camera 71a) set as a stereo camera pair, and a feature point (hereinafter referred to as "feature point 2") of a frame image of an image captured by the other of the two cameras (e.g., camera 71b). The generation unit 354 uses camera parameters of the two cameras to generate a homography transformation matrix that links feature point 1 and feature point 2. The generation unit 354 uses this transformation matrix, the frame image of the image captured by one of the two cameras, and the frame image of the image captured by the other of the two cameras to generate a virtual image viewed from the virtual viewpoint set by the first setting unit 353.

[0053] The detection unit 355 uses object detection AI to detect a detection target on the virtual image generated by the generation unit 354. The object detection AI includes skeleton detection AI. To improve the detection accuracy of the detection target, the detection unit 355 may zoom in or out on the virtual image to change the field of view, or rotate the virtual image to adjust the orientation of the virtual image.

[0054] The object detection AI takes a virtual image as input and outputs the attributes of the detection target shown in the virtual image, the position of the detection target, and a reliability score of the detection target. The attributes of the detection target include, for example, a person, a cart, a desk, or a chair. When the detection target is a person, the attributes of the detection target further include information for distinguishing between people, such as whether the detection target is wearing glasses or the color of the detection target's clothing. Planar coordinates indicating the position of the detection target in the virtual image are output as the position of the detection target. For example, when the attribute of the detection target is a person, planar coordinates indicating the position of each skeleton point of the person in the virtual image are output. The reliability score of the detection target is an index representing the reliability of the detection prediction result. For example, when the attribute of the detection target output by the object detection AI is a person, the reliability score of the detection target indicates the probability that the detection target is a person. When two or more people are detected overlapping, when two or more objects are detected overlapping, or when a person and an object are detected overlapping, the reliability score of the detection target is low.

[0055] If the reliability score of the detection target detected by the detection unit 355 is less than a predetermined value, the first setting unit 353 may change the virtual viewpoint in accordance with a predetermined rule. The predetermined value is, for example, 80%. The predetermined rule is set so that the detection accuracy of the detection target by the detection unit 355 increases by repeatedly changing the virtual viewpoint. When the virtual viewpoint is changed, the generation unit 354 generates a virtual image seen from the changed virtual viewpoint, and the detection unit 355 detects the detection target in the virtual image.

[0056] The second setting unit 356 sets a warning area on the virtual image generated by the generation unit 354. The second setting unit 356 sets the warning area according to the position of the moving body 10. More specifically, the second setting unit 356 sets an area within a predetermined range with the position of the moving body 10 as the base point as the warning area. As an example, the second setting unit 356 sets an area within a radius of n meters centered on the position of the moving body 10 as the warning area, where n is a number greater than zero.

[0057] The determination unit 357 determines whether or not a warning notification is necessary. More specifically, the determination unit 357 determines whether or not the position on the virtual image of the detection target detected by the detection unit 355 is included in the warning area. If the position on the virtual image of at least one detection target detected by the detection unit 355 is included in the warning area, the determination unit 357 determines that a warning notification is necessary and transmits an instruction to output a warning notification to the warning device 500. On the other hand, if the position on the virtual image of any detection target detected by the detection unit 355 is not included in the warning area, the determination unit 357 determines that a warning notification is not necessary.

[0058] The warning device 500 outputs a warning notification based on the determination result of the determining unit 357. More specifically, the warning device 500 outputs a warning notification when an instruction to output a warning notification is received.

[0059] FIG. 5 is a flowchart showing the procedure of the monitoring process performed by the monitoring device according to the first embodiment.

[0060] In step S1, the processor 301 receives multiple video streams from multiple cameras and extracts one or more frame images from each video stream.

[0061] Next, in step S2, the processor 301 detects the moving object 10 from the multiple frame images extracted in step S1.

[0062] Next, in step S3, the processor 301 selects two cameras from the multiple cameras 70 that are suitable for capturing the moving body 10, and sets the two cameras as a pair of stereo cameras to be used for stereo matching.

[0063] Next, in step S4, the processor 301 determines the position of the moving body 10 in real space based on multiple frame images extracted from two images acquired by two cameras set as a stereo camera pair and the camera parameters of the two cameras among the camera parameters 312.

[0064] Next, in step S5, the processor 301 sets a virtual viewpoint. As an example, the processor 301 sets the virtual viewpoint to a position directly above the moving body 10.

[0065] Next, in step S6, the processor 301 generates a virtual image viewed from the virtual viewpoint set in step S5 based on multiple frame images extracted from two images acquired by two cameras set as a stereo camera pair.

[0066] Next, in step S7, the processor 301 detects the detection target on the virtual image generated in step S6.

[0067] Next, in step S8, the processor 301 sets a warning area on the virtual image generated in step S6.

[0068] Next, in step S9, processor 301 determines whether a warning notification is necessary. If the position on the virtual image of at least one detection target detected in step S7 is included in the warning area, processor 301 determines that a warning notification is necessary. On the other hand, if the position on the virtual image of none of the detection targets detected in step S7 is included in the warning area, processor 301 determines that a warning notification is not necessary.

[0069] If a warning notification is necessary (YES in step S9), processor 301 proceeds to step S10. On the other hand, if a warning notification is not necessary (NO in step S9), processor 301 returns the process to step S1.

[0070] In step S10, the processor 301 transmits an instruction to output a warning notification to the alarm device 500. After step S10, the processor 301 returns the process to step S1. The monitoring process is always running.

[0071] In this way, in the monitoring system 1 according to the first embodiment, the multiple cameras 70 are provided at positions separate from the moving object 10. Therefore, the monitoring system 1 according to the first embodiment can detect the detection target even if the detection target is located in a blind spot of the moving object 10.

[0072] Furthermore, in the monitoring system 1 according to the first embodiment, the warning area is set according to the position of the mobile object 10. Therefore, when the mobile object 10 is moving, the warning area changes according to the movement of the mobile object 10. When a predetermined area is set as the warning area, even if the mobile object 10 is located far from that area, a warning notification is output when a detection target is present in that area. However, in such a case, a warning notification is not actually necessary. In contrast, in the monitoring system 1 according to the first embodiment, the warning area is set according to the position of the mobile object 10. Therefore, the monitoring system 1 according to the first embodiment can reduce unnecessary warning notifications compared to when a predetermined area is set as the warning area.

[0073] Furthermore, the monitoring system 1 in the first embodiment generates a virtual image from a different viewpoint than the camera image based on an image captured by a camera (hereinafter referred to as "camera image"), and detects the detection target in the virtual image. When multiple detection targets overlap in the camera image, it is difficult to detect the detection target in the camera image. Furthermore, even when there is only one detection target in the camera image, the detection accuracy of the detection target may be improved by changing the viewpoint and the angle at which the detection target is viewed. Generally, when the moving object 10 moves only horizontally, it is unlikely that multiple detection targets will overlap in a virtual image viewed from directly above the moving object 10. The monitoring system 1 in the first embodiment generates a virtual image viewed from directly above the moving object 10, and detects the detection target in the virtual image. Therefore, the monitoring system 1 in the first embodiment improves the detection accuracy of the detection target.

[0074] Furthermore, the monitoring system 1 in embodiment 1 generates a virtual image based on camera images, sets an alarm area on the virtual image, and performs a proximity determination between the moving body 10 and the detection target. Because the camera's subject also has three-dimensional depth, even if two objects appear close in a two-dimensional camera image, they are not necessarily close in three-dimensional space. For example, when viewing a person 21 and the moving body 10 from position B1 shown in FIG. 1 , the two may appear close to each other. However, it is unclear whether the two are close to each other in three-dimensional space. In other words, even if an alarm area is set on the two-dimensional camera image, proximity determination may not be possible. In such cases, it is better to set a virtual viewpoint at infinity directly above the moving body 10 and perform proximity determination on the virtual image viewed from the virtual viewpoint. The monitoring system 1 in embodiment 1 sets a virtual viewpoint at infinity directly above the moving body 10, sets an alarm area on the virtual image viewed from the virtual viewpoint, and performs proximity determination between the moving body 10 and the detection target. Therefore, the monitoring system 1 in embodiment 1 can more accurately perform proximity determination between the moving body 10 and the detection target.

[0075] In the first embodiment, the first setting unit 353 sets the virtual viewpoint at a position directly above the moving body 10, but the first setting unit 353 may also set the virtual viewpoint at a position shifted from a position directly above the moving body 10.

[0076] [Embodiment 2] In embodiment 1, the monitoring device 300 sets an alarm area on a virtual image. In contrast, in embodiment 2, the monitoring device sets an alarm area in the space of a world coordinate system. In embodiment 2, differences from embodiment 1 will be mainly described.

[0077] Fig. 6 is a diagram showing an example of a monitoring system and an example of an area to be monitored by the monitoring system in embodiment 2. Fig. 7 is a diagram showing the area to be monitored shown in Fig. 6 as seen from directly above.

[0078] 6 and 7 , a monitoring system 1A according to the second embodiment includes a plurality of cameras 70, a monitoring device 300A, and an alarm device 500. The plurality of cameras 70 are an example of "one or more imaging devices" in the present disclosure. In the example shown in FIGS. 6 and 7 , the plurality of cameras 70 include eight cameras (camera 71a, camera 71b, camera 72a, camera 72b, camera 73a, camera 73b, camera 74a, and camera 74b). However, the number of the plurality of cameras 70 is not limited to eight. The plurality of cameras 70, the monitoring device 300A, and the alarm device 500 are connected to one another via a network. The network may be a wired network or a wireless network.

[0079] The monitoring system 1A tracks the mobile object 10 and sets an alarm area in accordance with the movement of the mobile object 10. The mobile object 10 is a tracking target of the monitoring system 1A. In the example shown in FIG. 6, the mobile object 10 is an AMR. The mobile object 10 travels through a monitoring target area 51A. In the example shown in FIG. 6, the mobile object 10 moves horizontally through the monitoring target area 51A. The monitoring target area 51A is a predetermined area such as a factory, warehouse, or office.

[0080] The monitoring system 1A generates a virtual image viewed from a virtual viewpoint based on images obtained from multiple cameras 70, and detects a detection target in the virtual image. The virtual image has a viewpoint different from that of the images obtained from the multiple cameras 70. In the example shown in Fig. 6, a person 21 and a person 22 are detected as detection targets. The monitoring system 1A outputs a warning notification when the position of the detection target is included in the warning area.

[0081] More specifically, the multiple cameras 70 are installed at positions separate from the mobile object 10. In the example shown in FIG. 6, the multiple cameras 70 are fixed to a ceiling 15A of the monitored area 51A. The ceiling 15A is low. As shown in FIG. 7, the monitored area 51A has passages ST1 and ST2. The passages ST1 and ST2 are narrow passages surrounded by walls 31, 32, 33, and 34, and visibility from above the monitored area 51A is poor. The mobile object 10 travels along the passages ST1 and ST2. The eight cameras are positioned so as to be able to overlook each passage of the monitored area 51A and to utilize the distance measurement technology of the stereo camera. Each of the multiple cameras 70 photographs the monitored area 51A and acquires an image of the monitored area 51A. Each of the multiple cameras 70 transmits the acquired image data to the H. The video data is compressed using an image compression technology such as H.264, and the compressed video data (video stream) is transmitted to the monitoring device 300A.

[0082] The monitoring device 300A receives a plurality of images from a plurality of cameras 70. The monitoring device 300A detects the moving object 10 based on the plurality of images. The monitoring device 300A selects one or more pairs of stereo cameras to be used for stereo matching from two or more cameras located around the moving object 10 among the plurality of cameras 70. In the example shown in FIGS. 6 and 7 , the monitoring device 300A selects the pair of cameras 71a and 71b, the pair of cameras 72a and 72b, the pair of cameras 73a and 73b, and the pair of cameras 74a and 74b as the stereo camera pairs.

[0083] For each pair, the monitoring device 300A identifies the position of the moving object 10 based on the images from the two cameras that make up the pair and generates a virtual image seen from a virtual viewpoint. The monitoring device 300A detects a detection target in the virtual image and sets an alarm area in the space of the world coordinate system W.

[0084] 7 , when visibility from above the monitored area 51A is poor, the monitoring device 300A sets a virtual viewpoint at the center position of the two cameras that make up the stereo camera pair. Because the stereo camera pair is made up of cameras positioned around the mobile object 10, it can be said that the monitoring device 300A sets the virtual viewpoint to follow the movement of the mobile object 10. The monitoring device 300A generates a virtual image seen from the virtual viewpoint and detects a detection target in the virtual image.

[0085] The monitoring device 300A converts the position of the detection target detected in the virtual image into a position in a world coordinate system W and sets an alarm area in the world coordinate system W. The world coordinate system W is aligned with the world coordinate system used when calculating the camera parameters 312. The XY plane of the world coordinate system W is also aligned with the plane on which the mobile object 10 is traveling. The monitoring device 300A determines whether the mobile object 10 and the detection target are in proximity to each other in the world coordinate system W. More specifically, the monitoring device 300A determines whether the position of the detection target in the world coordinate system W is included in the alarm area. If the position of the detection target in the world coordinate system W is included in the alarm area, the monitoring device 300A determines that the mobile object 10 and the detection target are in proximity to each other and transmits an instruction to the alarm device 500 to output a warning notification. In the example shown in FIG. 6 , the positions of the person 21 and the person 22 in the world coordinate system W (positions Q2 and Q3) are included in area P2, which is the alarm area. Therefore, the monitoring device 300A transmits an instruction to the alarm device 500 to output a warning notification.

[0086] When the warning device 500 receives an instruction to output a warning notification, it outputs the warning notification in the same manner as in the first embodiment.

[0087] 6 and 7 , people 21 and 22, who are the detection targets, are hidden from the moving body 10 by a wall 31. In other words, people 21 and 22, who are the detection targets, are located in the blind spot of the moving body 10. Therefore, if a camera is installed in the moving body 10, people 21 and 22, who are the detection targets, cannot be detected from the video captured by the camera. However, in the surveillance system 1A, camera 70 is installed in a position separate from the moving body 10. Therefore, even if the detection target is located in the blind spot of the moving body 10, the detection target can be detected.

[0088] 8 is a diagram showing an outline of the processing of the monitoring device in embodiment 2. The monitoring device 300A in embodiment 2 has the same hardware configuration as the monitoring device 300.

[0089] Monitoring device 300A includes video processing unit 351, tracking unit 352A, first setting unit 353A, generation unit 354, detection unit 355, second setting unit 356A, and determination unit 357A. Video processing unit 351, generation unit 354, and detection unit 355 have been described in embodiment 1. Therefore, only a brief description of video processing unit 351, generation unit 354, and detection unit 355 will be given, and tracking unit 352A, first setting unit 353A, second setting unit 356A, and determination unit 357A will be described in detail.

[0090] The video processing unit 351 , tracking unit 352A, first setting unit 353A, generation unit 354 , detection unit 355 , second setting unit 356A, and determination unit 357A are realized by the processor 301 executing the program 311 .

[0091] The video processing unit 351 receives video from the multiple cameras 70 and processes the video in the same manner as in the first embodiment.

[0092] The tracking unit 352A detects the moving body 10 and identifies the position of the moving body 10 based on multiple images captured by multiple cameras 70.

[0093] More specifically, first, the tracking unit 352A detects the moving object 10 in the same manner as in the first embodiment.

[0094] Next, the tracking unit 352A selects one or more pairs of stereo cameras to be used for stereo matching from two or more cameras located around the moving object 10 among the multiple cameras 70. In the example shown in FIGS. 6 and 7 , the tracking unit 352A selects the following pairs of stereo cameras: a pair of camera 71a and camera 71b, a pair of camera 72a and camera 72b, a pair of camera 73a and camera 73b, and a pair of camera 74a and camera 74b. In the following description, the pair of camera 71a and camera 71b will be referred to as the "first pair."

[0095] The tracking unit 352A sets the cameras 71a and 71b that make up the first pair as a stereo camera pair. The tracking unit 352A identifies the position of the moving object 10 in real space based on the images from the cameras 71a and 71b in the same manner as in the first embodiment.

[0096] The first setting unit 353A sets a virtual viewpoint at a position A2 (see FIG. 6 ) that is the center between the cameras 71 a and 71 b. Because the stereo camera pair is made up of cameras positioned around the moving body 10, it can be said that the first setting unit 353A sets the virtual viewpoint by following the movement of the moving body 10.

[0097] The generation unit 354 generates a virtual video viewed from the virtual viewpoint set by the first setting unit 353A in the same manner as in the first embodiment.

[0098] The detection unit 355 detects the detection target on the virtual image generated by the generation unit 354 in the same manner as in the first embodiment. Here, the virtual image in the second embodiment will be described with reference to FIG. 9. FIG. 9 is a diagram showing an example of the virtual image in the second embodiment. The virtual image V2 is an example of the virtual image when the virtual viewpoint is set to the center position A2 (see FIG. 6) between the cameras 71a and 71b. The virtual image V2 shows the person 21 who is the detection target, the person 22 who is the detection target, the moving body 10, and the wall 31.

[0099] 8 , if the reliability score of the detection target detected by the detection unit 355 is equal to or greater than a predetermined value, the tracking unit 352A changes the pair of stereo cameras to the next pair. The predetermined value is, for example, 80%. Then, for the next pair, the tracking unit 352A identifies the position of the moving object 10, the first setting unit 353A sets a virtual viewpoint, the generation unit 354 generates a virtual image seen from the virtual viewpoint, and the detection unit 355 detects the detection target.

[0100] On the other hand, if the reliability score of the detection target detected by the detection unit 355 is less than a predetermined value, the first setting unit 353A changes the virtual viewpoint according to a predetermined rule. The predetermined rule is set so that repeated changes of the virtual viewpoint increase the detection accuracy of the detection target by the detection unit 355. As an example, the first setting unit 353A changes the position of the virtual viewpoint by a predetermined amount along a virtual line connecting two cameras set as a stereo camera pair in a direction away from the detection target whose reliability score is less than a predetermined value. When the camera 71a and the camera 71b are set as a stereo camera pair, the first setting unit 353A changes the position of the virtual viewpoint by a predetermined amount toward the camera 71b along a virtual line L1 (see FIG. 7 ) connecting the cameras 71a and 71b. Note that the position of the virtual viewpoint after the change is not limited to being on the virtual line connecting the two cameras set as a stereo camera pair.

[0101] After changing the virtual viewpoint, the generation unit 354 generates a virtual image seen from the changed virtual viewpoint, and the detection unit 355 detects the detection target in the virtual image generated by the generation unit 354. The change of the virtual viewpoint, the generation of a virtual image seen from the changed virtual viewpoint, and the detection of the detection target are performed until the reliability scores of all detection targets in the vicinity of the detection target whose reliability score has been determined to be less than the predetermined value become equal to or greater than the predetermined value.

[0102] Note that the monitoring system 1A also tolerates a case where the reliability scores of all detection targets in the vicinity of a detection target whose reliability score is determined to be less than a predetermined value do not become equal to or greater than a predetermined value even if the first setting unit 353A changes the virtual viewpoint a predetermined number of times or more. In other words, if the reliability scores of all detection targets in the vicinity of a detection target whose reliability score is determined to be less than a predetermined value do not become equal to or greater than a predetermined value even if the first setting unit 353A changes the virtual viewpoint a predetermined number of times or more, the tracking unit 352A changes the pair of stereo cameras to the next pair.

[0103] When the detection unit 355 has detected the detection targets for all pairs, the process proceeds to processing by the second setting unit 356A. The second setting unit 356A calculates coordinates in the world coordinate system W for the moving body 10 and the detection targets detected by the detection unit 355. Note that the detection targets for which coordinates in the world coordinate system W are calculated may include all detection targets detected by the detection unit 355, or may include only detection targets of all detection targets detected by the detection unit 355 whose reliability scores are equal to or greater than a predetermined value.

[0104] More specifically, the second setting unit 356A calculates coordinates indicating the center position of the ground contact surface of the moving body 10 in the world coordinate system W. Furthermore, for each detection target, the second setting unit 356A calculates coordinates indicating the position of the detection target in the world coordinate system W using transformation parameters used when generating the virtual image in which the detection target is detected. In the example shown in FIG. 6 , the second setting unit 356A calculates coordinates indicating the center position Q1 of the ground contact surface of the moving body 10, coordinates indicating the position Q2 of a person 21 as a detection target, and coordinates indicating the position Q3 of a person 22 as a detection target in the world coordinate system W. The center position Q1, position Q2, and position Q3 are all located on the XY plane of the world coordinate system W.

[0105] There may be cases where a detection target detected in a virtual image generated based on images acquired from one stereo camera pair is also detected in a virtual image generated based on images acquired from another stereo camera pair. The second setting unit 356A can determine whether or not the detection targets have been detected in duplicate based on the coordinates of the detection targets in the world coordinate system W. However, if it is difficult to determine whether or not the detection targets have been detected in duplicate based on the coordinates, the second setting unit 356A may estimate whether or not the detection targets have been detected in duplicate by taking into account the attributes of the detection targets detected by the detection unit 355, etc.

[0106] The second setting unit 356A sets the warning area in the space of the world coordinate system W. The second setting unit 356A sets the warning area in accordance with the position of the moving body 10. More specifically, the second setting unit 356A sets an area within a predetermined range with the position of the moving body 10 as the base point in the space of the world coordinate system W as the warning area.

[0107] As an example, the second setting unit 356A sets an area P2 (see FIG. 6 ) on the XY plane of the world coordinate system W and within a radius of n meters from the position of the moving object 10 (for example, the center position Q1), as the warning area, where n is a number greater than zero.

[0108] As another example, the second setting unit 356A may set, as the warning area, a three-dimensional area within a predetermined range with the position of the moving body 10 (e.g., the center position Q1) as the base point in the space of the world coordinate system W. For example, if the moving body 10 is an object that moves in a three-dimensional space, such as a drone, the second setting unit 356A may set a three-dimensional area as the warning area.

[0109] The determination unit 357A determines whether or not a warning notification is necessary. More specifically, the determination unit 357A determines whether or not the position in the world coordinate system W of the detection target detected by the detection unit 355 is included in the warning area. If the position in the world coordinate system W of at least one detection target detected by the detection unit 355 is included in the warning area, the determination unit 357A determines that a warning notification is necessary and transmits an instruction to output a warning notification to the warning device 500. On the other hand, if the position in the world coordinate system W of none of the detection targets detected by the detection unit 355 is included in the warning area, the determination unit 357A determines that a warning notification is not necessary.

[0110] When the warning device 500 receives an instruction to output a warning notification, it outputs the warning notification in the same manner as in the first embodiment.

[0111] Fig. 10 is a flowchart showing a part of the procedure of the monitoring process by the monitoring device in embodiment 2. Fig. 11 is a flowchart showing the remaining part of the procedure of the monitoring process by the monitoring device in embodiment 2. The same processes as those described in Fig. 5 are assigned the same step numbers, and detailed description will not be repeated.

[0112] First, the processor 301 performs the process of step S1. Next, the processor 301 performs the process of step S2. After step S2, the processor 301 proceeds to the process of step S3A.

[0113] In step S3A, the processor 301 selects one or more pairs of stereo cameras to be used for stereo matching from two or more cameras located around the moving body 10 among the multiple cameras 70.

[0114] Next, in step S3B, the processor 301 selects one pair from the one or more pairs selected in step S3A, and sets the two cameras that make up that pair as a stereo camera pair.

[0115] Next, the processor 301 performs the process of step S4. After step S4, the processor 301 proceeds to the process of step S5A.

[0116] In step S5A, processor 301 sets a virtual viewpoint. When a virtual viewpoint is set for the first time for the stereo camera pair set in step S3B, processor 301 sets the virtual viewpoint at the center position of the two cameras set as the stereo camera pair in step S3B. On the other hand, when detection is insufficient in step S7A (described later) (YES in step S7A), processor 301 changes the virtual viewpoint according to predetermined rules and sets the changed virtual viewpoint.

[0117] Next, the processor 301 performs the process of step S6. Next, the processor 301 performs the process of step S7. After step S7, the processor 301 proceeds to the process of step S7A.

[0118] In step S7A, the processor 301 determines whether or not the detection of the detection target is insufficient. If the reliability score of the detection target detected in step S7 is less than a predetermined value, the processor 301 determines that the detection of the detection target is insufficient. On the other hand, if the reliability score of the detection target detected in step S7 is equal to or greater than a predetermined value, the processor 301 determines that the detection of the detection target is not insufficient.

[0119] If the detection of the detection target is insufficient (YES in step S7A), processor 301 returns the process to step S5A and changes the virtual viewpoint, whereas if the detection of the detection target is not insufficient (NO in step S7A), processor 301 transitions the process to step S7B.

[0120] In step S7B, processor 301 determines whether any unused stereo camera pairs remain among the one or more pairs of stereo cameras selected in step S3A. If any unused stereo camera pairs remain among the one or more pairs of stereo cameras selected in step S3A (YES in step S7B), processor 301 returns the process to step S3B, selects a pair of stereo cameras that has not yet been used, and sets the two cameras that make up that pair as the stereo camera pair. On the other hand, if any unused stereo camera pairs remain among the one or more pairs of stereo cameras selected in step S3A (NO in step S7B), processor 301 transitions the process to step S7C.

[0121] In step S7C, the processor 301 calculates the coordinates in the world coordinate system W for the moving body 10 and the detection target detected in step S7.

[0122] Next, in step S8A, the processor 301 sets a warning area in the space of the world coordinate system W.

[0123] Next, in step S9A, the processor 301 determines whether or not a warning notification is necessary. If the position in the world coordinate system W of at least one detection target detected in step S7 is included in the warning area, the processor 301 determines that a warning notification is necessary. On the other hand, if the position in the world coordinate system W of none of the detection targets detected in step S7 is included in the warning area, the processor 301 determines that a warning notification is not necessary.

[0124] If a warning notification is necessary (YES in step S9A), processor 301 proceeds to step S10. After step S10, processor 301 returns the process to step S1. On the other hand, if a warning notification is not necessary (NO in step S9A), processor 301 returns the process to step S1.

[0125] In this way, in the monitoring system 1A according to the second embodiment, the multiple cameras 70 are provided at positions separate from the moving object 10. Therefore, the monitoring system 1A according to the second embodiment can detect the detection target even if the detection target is located in a blind spot of the moving object 10.

[0126] Furthermore, in the monitoring system 1A according to the second embodiment, an alarm area is set according to the position of the mobile object 10. Therefore, when the mobile object 10 is moving, the alarm area changes according to the movement of the mobile object 10. When a predetermined area is set as the alarm area, an alarm notification is output when a detected object is present in the area, even if the mobile object 10 is located far from the area. However, in such a case, an alarm notification is not actually necessary. In contrast, in the monitoring system 1A according to the second embodiment, an alarm area is set according to the position of the mobile object 10. Therefore, the monitoring system 1A according to the second embodiment can reduce unnecessary alarm notifications compared to when a predetermined area is set as the alarm area.

[0127] Furthermore, the monitoring system 1A in the second embodiment generates a virtual image from a different viewpoint from the camera image based on the camera image, and detects the detection target in the virtual image. When multiple detection targets overlap in the camera image, it is difficult to detect the detection target in the camera image. Furthermore, even when there is only one detection target in the camera image, the detection accuracy of the detection target may be improved by changing the viewpoint and the angle from which the detection target is viewed. The monitoring system 1A in the second embodiment generates a virtual image from a different viewpoint from the camera image, and detects the detection target in the virtual image. Therefore, the monitoring system 1A in the second embodiment improves the detection accuracy of the detection target.

[0128] Furthermore, when the reliability score of the detection target is less than a predetermined value, the monitoring system 1A according to the second embodiment changes the virtual viewpoint and detects the detection target again, thereby improving the detection accuracy of the detection target.

[0129] Furthermore, the monitoring system 1A in the second embodiment sets an alarm area in the space of the world coordinate system W, and performs proximity determination between the moving body 10 and the detection target on the world coordinate system W. Therefore, even if it is difficult to generate a virtual image that allows proximity determination, the monitoring system 1A in the second embodiment can perform proximity determination between the moving body 10 and the detection target.

[0130] [Modification 1 common to Embodiments 1 and 2] The monitoring systems 1 and 1A may include a ranging sensor such as a Time of Flight (TOF) or LiDAR at the position of each camera. In this case, the tracking units 352 and 352A use the ranging sensor to identify the position of the moving object 10 in real space. When the position of the moving object 10 is identified using a ranging sensor, it is not necessary to arrange two cameras in pairs. This allows for more flexibility in determining the orientation and arrangement of the cameras. Furthermore, the number of cameras can be reduced.

[0131] [Modification 2 common to Embodiments 1 and 2] The monitoring system 1, 1A may include one camera as one or more imaging devices. The one camera is provided at a location separate from the moving object 10. When one camera is provided as one or more imaging devices, the monitoring system 1, 1A further includes a ranging sensor. The ranging sensor is installed at the location of the one camera. The tracking unit 352, 352A detects the moving object 10 based on one image captured by the one camera. The tracking unit 352, 352A also identifies the position of the moving object 10 in real space using the ranging sensor. Furthermore, the generation unit 354 generates a virtual image viewed from a virtual viewpoint based on the one image captured by the one camera.

[0132] [Modification 3 common to Embodiments 1 and 2] The type of lens of each of the plurality of cameras 70 is not limited to a normal lens. As another example, the type of lens of each of the plurality of cameras 70 may be a wide-angle lens or a fisheye lens.

[0133] FIG. 12 illustrates a first modified example regarding the type of lens of each of the multiple cameras and the arrangement of the multiple cameras. As an example, multiple cameras 80 may be installed in the monitored area 51A instead of the multiple cameras 70, and a ranging sensor may be installed at each camera position. The multiple cameras 80 include cameras 81, 82, 83, and 84. Each of the multiple cameras 80 is a network camera. Each of the multiple cameras 80 is a fisheye camera with a 180-degree angle of view. For example, the virtual image generated from the images captured by cameras 71a and 71b in the second embodiment can be generated from flattened images of the fisheye images captured by cameras 81 and 82, or from flattened images of the fisheye images captured by cameras 81 and 83. Furthermore, ranging sensors installed at each of the multiple cameras 80 may be used to measure the distance to the moving body 10 and the distance to the detection target, thereby identifying the positions of the moving body 10 and the detection target.

[0134] 13 is a diagram showing a second modified example regarding the type of lens of each of the multiple cameras and the arrangement of the multiple cameras. The multiple cameras 90 include a camera 91, a camera 92, and a camera 93. Each of the multiple cameras 90 is a network camera. Each of the multiple cameras 90 is a fisheye camera. As an example, multiple cameras 90 may be provided in the monitored area 51A instead of the multiple cameras 70, and the multiple cameras 90 may be arranged with their fisheye lenses facing downward, as shown in FIG.

[0135] [Modification 4 common to Embodiments 1 and 2] The one or more cameras serving as the one or more imaging devices do not have to be fixed. The one or more cameras serving as the one or more imaging devices may be movable, such as a camera mounted on a drone.

[0136] [Variation 5 common to Embodiments 1 and 2] The warning area does not have to be circular, and may be an ellipse extending in the traveling direction of the moving object 10. Furthermore, the second setting units 356 and 356A may increase or decrease the size of the area of ​​the warning area in the traveling direction of the moving object 10 according to the speed of the moving object 10.

[0137] [Variation 6 common to Embodiments 1 and 2] The warning area does not have to be an area in the vicinity of the mobile object 10. In other words, the warning area may be an area that does not include the location of the mobile object 10. Furthermore, the second setting unit 356, 356A may set a plurality of warning areas.

[0138] [Variation 7 common to Embodiments 1 and 2] The second setting unit 356, 356A may accept the designation of an area that is not set as a warning area (hereinafter referred to as an "exclusion area"). When an exclusion area is designated and the exclusion area is included in an area within a predetermined range based on the position of the mobile object 10, the second setting unit 356, 356A may set, as the warning area, the area within the predetermined range based on the position of the mobile object 10, excluding the exclusion area.

[0139] An area designated as an excluded area is, for example, an area that is separated by a wall or the like and cannot be entered by the mobile body 10. Even if such an area is included within a predetermined range based on the position of the mobile body 10 and a detection target is present in such an area, the mobile body 10 will not come into contact with the detection target. Therefore, a warning notification is unnecessary. By designating an excluded area, unnecessary warning notifications can be reduced.

[0140] [Variation 8 Common to Embodiments 1 and 2] Even if the location of a detection target is included in the warning area, the determination unit 357, 357A may exclude the detection target from the scope of a warning notification based on at least one of the attributes of the detection target and the state of the detection target. The state of the detection target may include, for example, at least one of the direction of movement of the detection target and the speed of movement of the detection target. For example, if a detection target is detected behind the moving object 10 moving in the opposite direction from the moving object 10, the possibility of contact between the moving object 10 and the detection target is low. Therefore, in such a case, the determination unit 357, 357A may determine that a warning notification is not necessary and may not send an instruction to output a warning notification to the alarm device 500.

[0141] [Modification 9 common to Embodiments 1 and 2] The second setting unit 356, 356A may set a predetermined area as the warning area. In this case, by setting an area with a high volume of pedestrians as the predetermined area, it is possible to reduce contact between people and the moving object 10.

[0142] [Modification 10 common to Embodiments 1 and 2] The alarm device 500 may include either a display 501 or a speaker 502. The alarm device 500 may also include a warning light. If the alarm device 500 includes a warning light, the warning light is turned on in response to the alarm device 500 receiving an instruction to output a warning notification.

[0143] [Modification 11 common to Embodiments 1 and 2] When the location of the detection target is included in the warning area, the monitoring devices 300 and 300A may transmit an instruction to output a warning notification to the warning device 500 and also transmit a stop instruction to the mobile object 10. Furthermore, when the location of the detection target is included in the warning area, the monitoring devices 300 and 300A may transmit a stop instruction to the mobile object 10 instead of transmitting an instruction to output a warning notification to the warning device 500.

[0144] Furthermore, when the position of the detection target is included in the warning area, the monitoring device 300, 300A may transmit an instruction to output a warning notification to the warning device 500, causing the warning device 500 to output a warning sound. Thereafter, when the distance between the moving object 10 and the detection target becomes less than a predetermined distance, the monitoring device 300, 300A may transmit an instruction to stop the moving object 10, causing the moving object 10 to stop.

[0145] [Modification 12 common to Embodiments 1 and 2] When generating a virtual image, AI or the like may be used to complement parts not captured in a camera image to generate the virtual image. For example, even if there is no image to refer to as a real image, it may be possible to estimate the shape of an object in the uncaptured part from a portion of the image captured by the camera. In this way, the AI ​​may complement parts that cannot be seen based on previously learned data to generate a virtual image. Technologies that automatically generate images viewed from any viewpoint using deep learning (e.g., Neural Radiance Fields (NeRF) or Stylized Neural Implicit Representations for 3D Scenes (SNeRF)) are known.

[0146] [Additional Notes] The above-described embodiment and modifications include the following technical ideas.

[0147] [Configuration 1] A monitoring device comprising: a tracking unit that detects a moving object based on one or more images acquired by one or more imaging devices provided at a position separate from the moving object to be tracked; a first setting unit that sets a virtual viewpoint in accordance with the movement of the moving object; a generation unit that generates a virtual image seen from the virtual viewpoint based on the one or more images; and a detection unit that detects a detection target on the virtual image.

[0148] [Configuration 2] The monitoring device according to Configuration 1, wherein the first setting unit sets the virtual viewpoint to a position directly above the moving object.

[0149] [Configuration 3] The monitoring device according to Configuration 1, wherein the first setting unit changes the virtual viewpoint when the reliability of the detection target detected by the detection unit is less than a predetermined value.

[0150] [Configuration 4] The monitoring device according to any one of Configurations 1 to 3, further comprising: a second setting unit that sets a warning area on the virtual image; and a determination unit that determines whether the position on the virtual image of the detection target detected by the detection unit is included in the warning area.

[0151] [Configuration 5] The monitoring device according to any one of Configurations 1 to 3, further comprising: a second setting unit that sets an alarm area in a space of a world coordinate system; and a determination unit that determines whether the position in the world coordinate system of the detection target detected by the detection unit is included in the alarm area.

[0152] [Configuration 6] The monitoring device according to Configuration 4 or 5, wherein the second setting unit sets the warning area depending on the position of the moving object.

[0153] [Configuration 7] The monitoring device according to Configuration 4 or 5, wherein the second setting unit sets a predetermined area as the warning area.

[0154] [Configuration 8] A monitoring system comprising the monitoring device according to configuration 4 or 5, and an alarm device that outputs a warning notice based on the determination result of the determination unit.

[0155] [Configuration 9] A surveillance method executed by one or more computers, comprising: detecting a moving object to be tracked based on one or more images acquired by one or more imaging devices provided at a position separate from the moving object; setting a virtual viewpoint to follow the movement of the moving object; generating a virtual image viewed from the virtual viewpoint based on the one or more images; and detecting a detection target on the virtual image.

[0156] [Configuration 10] A monitoring program that causes one or more computers to execute the monitoring method according to Configuration 9.

[0157] The embodiments disclosed herein should be considered to be illustrative in all respects and not restrictive. The scope of the present invention is defined by the claims, not by the above description, and is intended to include all modifications within the meaning and scope of the claims.

[0158] 1, 1A Surveillance system, 10 Moving object, 11, 21, 22 Person, 15, 15A Ceiling, 31, 32, 33, 34 Wall, 51, 51A Surveillance target area, 70, 71a, 71b, 72a, 72b, 73a, 73b, 74a, 74b, 80, 81, 82, 83, 84, 90, 91, 92, 93 Camera, 300, 300A Surveillance device, 301 Processor, 302 Memory, 303 Storage, 304 Communication interface, 311 Program, 312 Camera parameters, 313 Image data, 351 Image processing unit, 352, 352A Tracking unit, 353, 353A First setting unit, 354 Generation unit, 355 Detection unit, 356, 356A Second setting unit, 357, 357A Determination unit, 500 alarm device, 501 display, 502 speaker, A1, A2, B1, Q2, Q3 position, L1 virtual line, P1, P2 area, Q1 center position, ST1, ST2 passage, V1, V2 virtual image, W world coordinate system.

Claims

1. A surveillance device comprising: a tracking unit that detects a moving object based on one or more images acquired by one or more imaging devices provided at a position separate from the moving object to be tracked; a first setting unit that sets a virtual viewpoint in accordance with the movement of the moving object; a generation unit that generates a virtual image viewed from the virtual viewpoint based on the one or more images; and a detection unit that detects a detection target on the virtual image.

2. The monitoring device according to claim 1, wherein the first setting unit sets the virtual viewpoint to a position directly above the moving object.

3. The monitoring device according to claim 1, wherein the first setting unit changes the virtual viewpoint when the reliability of the detection target detected by the detection unit is less than a predetermined value.

4. The monitoring device of claim 1, further comprising: a second setting unit that sets an alarm area on the virtual image; and a judgment unit that judges whether the position on the virtual image of the detection target detected by the detection unit is included in the alarm area.

5. The monitoring device according to claim 1, further comprising: a second setting unit that sets an alarm area in the space of a world coordinate system; and a judgment unit that judges whether the position in the world coordinate system of the detection target detected by the detection unit is included in the alarm area.

6. A monitoring device as described in claim 4 or 5, wherein the second setting unit sets the warning area according to the position of the moving object.

7. The monitoring device according to claim 4 or 5, wherein the second setting unit sets a predetermined area as the alarm area.

8. A monitoring system comprising the monitoring device according to claim 4 or 5, and an alarm device that outputs a warning notice based on the determination result of said determination section.

9. A surveillance method executed by one or more computers, comprising: detecting a moving object to be tracked based on one or more images acquired by one or more imaging devices provided at a position separate from the moving object; setting a virtual viewpoint to follow the movement of the moving object; generating a virtual image viewed from the virtual viewpoint based on the one or more images; and detecting a detection target on the virtual image.

10. A monitoring program for causing one or more computers to execute the monitoring method according to claim 9.

Images