Control device, control method, and program

Abstract

This control device comprises a processor that controls a monitoring camera which monitors a monitoring region. The processor makes it possible to switch the operation mode of the monitoring camera between a first monitoring mode in which an object present in the monitoring region is detected and tracked, and a second monitoring mode in which the object is imaged in response to a manual operation performed with respect to the monitoring camera. The processor divides the monitoring region into a first region and a second region, and switches the operating mode from the first monitoring mode to the second monitoring mode in response to an object, which was being tracked with the first monitoring mode in the first region, entering the second region.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a control device, a control method, and a program. [Background technology]

[0002] JP 2006-523043 A describes a method for detecting a moving object and controlling a surveillance system that includes a processing module adapted to receive image information from at least one image-forming sensor. The surveillance system performs motion detection analysis on the captured images and controls the camera in a specific manner when a moving object is detected.

[0003] Japanese Patent Application Laid-Open Publication No. 11-250369 describes a mobile fire monitoring device that has a surveillance camera that captures images of a specified area, a monitoring control device that is installed in relation to the surveillance camera and processes surveillance-related information, and an operating control device that communicates with the monitoring control device and processes the surveillance-related information.

[0004] Japanese Patent Application Laid-Open Publication No. 2004-056473 describes a monitoring control device that is provided with a neural network (NW) that outputs recognition information corresponding to images captured by a camera based on learning results, a control means that performs control based on this recognition information, a short-term memory means that temporarily saves image data, and a storage means that records this image data, and that the NW learns the relationship between the image and the degree of urgency of the event depicted in the image, recognizes the degree of urgency corresponding to the camera image, and the control means controls the frame rate of the image data recorded in the storage means based on the recognition information of the NW. Summary of the Invention

[0005] One embodiment of the technique of the present disclosure provides a control device, a control method, and a program that can reduce the burden on a user involved in monitoring an object. [Means for solving the problem]

[0006] The control device of the present disclosure is a control device that includes a processor that controls a surveillance camera that monitors a surveillance area, and the processor is capable of switching the operating mode of the surveillance camera between a first monitoring mode that detects and tracks objects present within the surveillance area, and a second monitoring mode that captures images of the objects in response to manual operations performed on the surveillance camera.The control device divides the surveillance area into a first area and a second area, and switches the operating mode from the first monitoring mode to the second monitoring mode in response to an object that was being tracked in the first area in the first monitoring mode entering the second area.

[0007] In the first monitoring mode, the processor preferably pans and tilts the monitoring camera to change the imaging range, thereby capturing an image of the monitoring area.

[0008] Preferably, when an object that has been tracked in the first monitoring mode in the first area enters the second area, the processor causes the surveillance camera to zoom in to change the area containing the object.

[0009] In the second monitoring mode, the processor preferably causes the monitoring camera to perform at least one of panning, tilting, and zooming in response to a manual operation.

[0010] In the second monitoring mode, the processor preferably outputs an image captured by the monitoring camera after manual operation as training data for machine learning.

[0011] In the second monitoring mode, the processor preferably outputs the history of the manual operation as training data for machine learning.

[0012] The second monitoring mode is preferably a mode in which, in addition to the first monitoring mode, an image of an object is captured in response to a manual operation performed on the monitoring camera.

[0013] In the second monitoring mode, it is preferable that the processor detects an object that appears in an image captured by the monitoring camera after manual operation, and causes the monitoring camera to track the detected object by performing at least one of panning, tilting, and zooming.

[0014] Preferably, the processor switches the operation mode from the second monitoring mode to the first monitoring mode in response to the object leaving the second region.

[0015] In the first monitoring mode, the processor preferably detects the object using a trained model based on machine learning.

[0016] The control method disclosed herein is a control method for controlling a surveillance camera that monitors a surveillance area, and includes making the operating mode of the surveillance camera switchable between a first surveillance mode that detects and tracks objects present in the surveillance area and a second surveillance mode that captures images of the objects in response to manual operations performed on the surveillance camera, dividing the surveillance area into a first area and a second area, and switching the operating mode from the first surveillance mode to the second surveillance mode in response to an object that was being tracked in the first surveillance mode in the first area entering the second area.

[0017] The program disclosed herein causes a computer to execute processes including: enabling the operating mode of a surveillance camera to be switched between a first monitoring mode in which an object present within a monitoring area is detected and tracked, and a second monitoring mode in which an image of the object is captured in response to manual operations performed on the surveillance camera; dividing the monitoring area into a first area and a second area; and switching the operating mode from the first monitoring mode to the second monitoring mode in response to an object that was being tracked in the first area in the first monitoring mode entering the second area. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a schematic diagram illustrating an example of the overall configuration of a monitoring system. [Figure 2]FIG. 10 is a conceptual diagram illustrating division of a monitoring area. [Figure 3] FIG. 2 is a block diagram showing an example of a hardware configuration of a monitoring camera and a management device. [Figure 4] FIG. 2 is a block diagram showing an example of the functions of a CPU included in the management device. [Figure 5] FIG. 10 is a conceptual diagram illustrating an example of an object detection process. [Figure 6] FIG. 10 is a conceptual diagram showing an example of automatic tracking of a dangerous object in an automatic monitoring mode. [Figure 7] FIG. 10 is a conceptual diagram illustrating an example of switching the operation mode from an automatic monitoring mode to a manual monitoring mode. [Figure 8] FIG. 10 is a conceptual diagram showing an example of manual PTZ in manual monitoring mode. [Figure 9] 10 is a flowchart illustrating an example of the flow of a monitoring process. [Figure 10] FIG. 10 is a conceptual diagram showing a teacher data output process according to a first modified example. [Figure 11] FIG. 10 is a conceptual diagram showing a teacher data output process according to a second modified example. [Figure 12] 13 is a flowchart showing an example of the flow of a monitoring process according to a third modified example. [Figure 13] 10 shows an example of automatic tracking in a manual monitoring mode according to a third modified example. [Figure 14] FIG. 10 is a block diagram showing an example of how an imaging processing program stored in a storage medium is installed in a computer. DETAILED DESCRIPTION OF THE INVENTION

[0019] Hereinafter, examples of a control device, a control method, and a program according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] CPU is an abbreviation for "Central Processing Unit". NVM is an abbreviation for "Non-volatile memory". RAM is an abbreviation for "Random Access Memory". IC is an abbreviation for "Integrated Circuit". ASIC is an abbreviation for "Application Specific Integrated Circuit". PLD is an abbreviation for "Programmable Logic Device". FPGA is an abbreviation for "Field-Programmable Gate Array". SoC is an abbreviation for "System-on-a-chip". SSD is an abbreviation for "Solid State Drive". USB is an abbreviation for "Universal Serial Bus". HDD is an abbreviation for "Hard Disk Drive". EEPROM is an abbreviation for "Electrically Erasable and Programmable Read Only Memory". EL is an abbreviation for "Electro-Luminescence". I / F is an abbreviation for "Interface". CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor." CCD is an abbreviation for "Charge Coupled Device." SWIR is an abbreviation for "Short Wave Infra-Red." LAN is an abbreviation for "Local Area Network."

[0022] 1, a monitoring system 10 includes a monitoring camera 12 and a management device 16. The monitoring system 10 is, for example, a system for monitoring a construction site. The monitoring camera 12 is installed in a high position, such as on the roof of a building near the construction site.

[0023] The management device 16 is used by a user such as a site supervisor who supervises workers at a construction site. The user uses the management device 16 to monitor, for example, whether any danger has occurred at the construction site during work. In particular, the user uses the management device 16 to monitor the movement of dangerous objects such as heavy machinery. The monitoring system 10 is a system for reducing the monitoring burden on the user. Note that dangerous objects are an example of an "object" related to the technology of the present disclosure.

[0024] The surveillance camera 12 includes an imaging device 18 and a rotation device 20. The imaging device 18 captures an image of a subject by receiving, for example, light in the visible wavelength band reflected from the subject. The imaging device 18 may also capture an image of a subject by receiving near-infrared light, which is light in the short-wave infrared wavelength band reflected from the subject. The short-wave infrared wavelength band refers to, for example, a wavelength band of approximately 900 nm to 2500 nm. Light in the short-wave infrared wavelength band is also generally referred to as SWIR light.

[0025] The imaging device 18 is attached to a swivel device 20. The swivel device 20 swivels the imaging device 18. For example, the swivel device 20 changes the imaging direction of the imaging device 18 in a pan direction and a tilt direction. The pan direction is, for example, the horizontal direction. The tilt direction is, for example, the vertical direction.

[0026] The rotation device 20 includes a base 22, a panning rotation member 24, and a tilting rotation member 26. The panning rotation member 24 is cylindrical and attached to the upper surface of the base 22. The tilting rotation member 26 is arm-shaped and attached to the outer circumferential surface of the panning rotation member 24. The imaging device 18 is attached to the tilting rotation member 26. The tilting rotation member 26 rotates around a tilt axis TA parallel to the horizontal direction, thereby changing the imaging direction of the imaging device 18 in the tilt direction.

[0027] The base 22 supports from below the panning rotation member 24. The panning rotation member 24 rotates about a pan axis PA that is parallel to the vertical direction, thereby changing the imaging direction of the imaging device 18 in the pan direction.

[0028] The base 22 has built-in drive sources (for example, a pan motor 24A and a tilt motor 26A shown in FIG. 3). The drive source of the base 22 is mechanically connected to the pan motor 24A and the tilt motor 26A. For example, the drive source of the base 22 is connected to the pan rotation member 24 and the tilt rotation member 26 via a power transmission mechanism (not shown). The pan rotation member 24 rotates about a pan axis PA by receiving power from the drive source of the base 22, and the tilt rotation member 26 rotates about a tilt axis TA by receiving power from the drive source of the base 22.

[0029] As shown in Fig. 1, the monitoring system 10 generates a captured image by capturing an image of an imaging range 31 set within a monitoring area 30 using an imaging device 18. The monitoring system 10 pans and tilts to change the imaging range 31, thereby capturing an image of the entire monitoring area 30. A construction site as the monitoring area 30 contains various subjects such as heavy machinery and workers. Heavy machinery includes cranes, power shovels, roller vehicles, bulldozers, dump trucks, etc.

[0030] The imaging device 18 is, for example, a digital camera having an image sensor (not shown). The image sensor receives subject light indicative of a subject, photoelectrically converts the received subject light, and outputs an electrical signal having a signal level according to the amount of received light as image data. The image data output by the image sensor corresponds to the captured image. The image sensor is a CMOS image sensor, a CCD image sensor, or the like. The imaging device 18 may capture a color image or a monochrome image. The captured image may be a still image or a moving image.

[0031] The imaging device 18 also has a zoom function. The zoom function is a function for reducing or enlarging (i.e., zooming in or out) the imaging range 31. The zoom function provided by the imaging device 18 is an optical zoom function that moves a zoom lens, or an electronic zoom function that performs image processing on image data. Note that the zoom function provided by the imaging device 18 may be a combination of the optical zoom function and the electronic zoom function.

[0032] The management device 16 includes a management device main body 13, a reception device 14, and a display 15. The management device main body 13 has a built-in computer 40 (see FIG. 3) and controls the entire monitoring system 10. The reception device 14 and the display 15 are connected to the management device main body 13.

[0033] The receiving device 14 receives various instructions from a user who uses the monitoring system 10. Examples of the receiving device 14 include a keyboard, a mouse, and / or a touch panel. The various instructions received by the receiving device 14 are grasped by the management apparatus main body 13. The display 15 displays various information (e.g., images and text) under the control of the management apparatus main body 13. Examples of the display 15 include a liquid crystal display and an EL display.

[0034] The surveillance camera 12 is communicably connected to the management device 16 via a communication network NT (such as the Internet or a LAN), and operates under the control of the management device main body 13. The connection between the surveillance camera 12 and the management device 16 may be either a wired connection or a wireless connection.

[0035] The management device 16 acquires captured images output from the imaging device 18 of the surveillance camera 12 and detects dangerous objects (heavy machinery, suspended loads, etc.) captured in the captured images using a trained model based on machine learning. When a specific object is detected, the management device 16 causes the surveillance camera 12 to pan, tilt, and zoom so as to track the detected object. Hereinafter, the operation of changing the imaging range 31 by panning, tilting, and zooming will be referred to as "PTZ." Furthermore, the operation of changing the imaging range 31 in accordance with the detection result of an object captured in the captured image will be referred to as "automatic PTZ."

[0036] Furthermore, the management device 16 enables an operation to change the imaging range 31 in response to a user's operation of the receiving device 14. Hereinafter, the operation to change the imaging range 31 in response to an instruction given to the receiving device 14 will be referred to as "manual PTZ." In manual PTZ, the user can set the imaging range 31 to any position and size within the monitoring area 30 by operating the receiving device 14.

[0037] In the following, an operating mode in which an automatic PTZ is used to detect and track an object present in the monitoring area 30 will be referred to as an "automatic monitoring mode." In addition to the automatic monitoring mode, an imaging mode in which imaging of an object in response to manual operation performed on the monitoring camera 12 is enabled will be referred to as a "manual monitoring mode." In other words, the "manual monitoring mode" is, for example, an imaging mode in which tracking of an object detected and tracked in the "automatic monitoring mode" is continued (the "automatic monitoring mode" is continued to be executed), and imaging of the object in response to manual operation.

[0038] A user can switch the monitoring mode of the monitoring system 10 between an automatic monitoring mode and a manual monitoring mode. The automatic monitoring mode is an example of a "first monitoring mode" according to the technology of the present disclosure. The manual monitoring mode is an example of a "second monitoring mode" according to the technology of the present disclosure. The "manual monitoring mode," which is an example of the "second monitoring mode," may enable a user to disable the function of the "automatic monitoring mode," i.e., the function of tracking an object detected in the automatic monitoring mode (i.e., not execute the automatic monitoring mode), in response to a user instruction via the reception device 14.

[0039] The management device 16 divides the monitoring area 30 into multiple areas and monitors the movement of dangerous objects, etc. As an example, as shown in FIG. 2, the management device 16 divides the monitoring area 30 into six areas, designating one of the six areas as an area of ​​interest R2 and the remaining five areas as normal areas R1. The area of ​​interest R2 is an area where construction work is being carried out and is highly dangerous. In the example shown in FIG. 2, the area of ​​interest R2 is an area where multiple dangerous objects (heavy machinery, etc.) are gathered and working. The normal area R1 is an example of a "first area" according to the technology of the present disclosure. The area of ​​interest R2 is an example of a "second area" according to the technology of the present disclosure.

[0040] Area information RI (see FIGS. 3 and 4) indicating a normal area R1 and an area of ​​interest R2 within the monitoring area 30 is set in advance. For example, the management device 16 sets an area within the monitoring area 30 where work using heavy machinery is scheduled to be carried out as the area of ​​interest R2. The management device 16 manages the position coordinates of the normal area R1 and the area of ​​interest R2 in association with the amount of panning and tilting of the monitoring camera 12. This allows the management device 16 to determine whether the imaging range 31 captured by the monitoring camera 12 corresponds to the normal area R1 or the area of ​​interest R2.

[0041] The area information RI may not be preset information, but may be dynamically changing information. For example, an area within the monitoring area 30 where there is a large amount of movement of heavy machinery or workers may be set as the area of ​​interest R2. For example, the monitoring area 30 is determined by calculating a motion vector for each position within the captured image output from the imaging device 18 of the monitoring camera 12, and an area with a large amount of movement is set as the area of ​​interest R2. In this case, the monitoring area 30 changes the area of ​​interest R2 when the area with a large amount of movement changes.

[0042] The shapes of the normal area R1 and the area of ​​interest R2 are not limited to rectangular. When the area of ​​interest R2 is set based on a motion vector, the shape of the area of ​​interest R2 may be dynamically changed to match the shape of an area with a large amount of motion.

[0043] The management device 16 monitors the entire monitoring area 30 by panning and tilting the monitoring camera 12 within the monitoring area 30 and changing the position of the imaging range 31, for example, along a path 32 shown in Fig. 2. In the example shown in Fig. 2, the size of the imaging range 31 when monitoring the entire monitoring area 30 is larger than the area of ​​interest R2, but the size of the imaging range 31 may also be smaller than the area of ​​interest R2.

[0044] 3, the rotation device 20 of the surveillance camera 12 includes a controller 34. The controller 34 controls the operations of the pan motor 24A, the tilt motor 26A, and the imaging device 18 under the control of the management device 16.

[0045] The management device main body 13 of the management device 16 includes a computer 40. The computer 40 has a CPU 42, an NVM 44, a RAM 46, and a communication I / F 48. The management device 16 is an example of a "control device" according to the technology of the present disclosure. The computer 40 is an example of a "computer" according to the technology of the present disclosure. The CPU 42 is an example of a "processor" according to the technology of the present disclosure.

[0046] The CPU 42, NVM 44, RAM 46, and communication I / F 48 are connected to a bus 49. In the example shown in Fig. 3, for convenience of illustration, one bus is shown as the bus 49, but multiple buses may be used. The bus 49 may be a serial bus or a parallel bus including a data bus, an address bus, a control bus, etc.

[0047] The NVM 44 stores various types of data. Here, examples of the NVM 44 include various types of nonvolatile storage devices such as an EEPROM, an SSD, and / or an HDD. The RAM 46 temporarily stores various types of information and is used as a work memory. Examples of the RAM 46 include a DRAM or an SRAM.

[0048] A program PG is stored in the NVM 44. The CPU 42 reads out a necessary program from the NVM 44 and executes the read program PG on the RAM 46. The CPU 42 controls the entire monitoring system 10, including the management device 16, by executing processing in accordance with the program PG.

[0049] The communication I / F 48 is an interface realized by hardware resources such as FPGA, etc. The communication I / F 48 is communicably connected to the controller 34 of the surveillance camera 12 via the communication network NT, and transmits and receives various information between the CPU 42 and the controller 34.

[0050] The bus 49 is also connected to the accepting device 14 and the display 15, and the CPU 42 operates in accordance with instructions received by the accepting device 14 and causes the display 15 to display various types of information.

[0051] The NVM 44 also stores a trained model LM for performing the above-mentioned object detection. The trained model LM is a trained model for object detection generated by performing machine learning using a plurality of training data. The NVM 44 also stores training data TD. The training data TD is training data for additional learning that is used to perform additional learning on the trained model LM. The training data TD is data based on user operations acquired in manual monitoring mode.

[0052] The NVM 44 also stores the above-mentioned area information RI.

[0053] 4, the CPU 42 executes operations based on the program PG to realize multiple functional units. The program PG causes the CPU 42 to function as a camera control unit 50, an image acquisition unit 52, a display control unit 53, an object detection unit 54, and a training data output unit 55.

[0054] The camera control unit 50 controls the controller 34 of the surveillance camera 12 to cause the imaging device 18 to perform imaging operations and zoom, and to cause the swivel device 20 to pan and tilt. In other words, the camera control unit 50 causes the surveillance camera 12 to perform imaging operations and change the imaging range 31.

[0055] The camera control unit 50 also performs switching control to switch the operation mode of the surveillance camera 12 between an automatic monitoring mode and a manual monitoring mode. In the automatic monitoring mode, the camera control unit 50 executes automatic PTZ, which changes the imaging range 31 in accordance with the result of object detection by the object detection unit 54. In the manual monitoring mode, the camera control unit 50 enables manual PTZ, which changes the imaging range 31 in accordance with an instruction given to the receiving device 14, in addition to automatic PTZ.

[0056] The image acquisition unit 52 acquires the captured image P output from the surveillance camera 12 when the camera control unit 50 causes the surveillance camera 12 to capture an image. The image acquisition unit 52 supplies the captured image P acquired from the surveillance camera 12 to the display control unit 53. The display control unit 53 displays the captured image P supplied from the image acquisition unit 52 on the display 15.

[0057] In addition, in the automatic monitoring mode, the image acquisition unit 52 supplies the captured image P acquired from the monitoring camera 12 to the object detection unit 54. In addition, in the manual monitoring mode, the image acquisition unit 52 supplies the captured image P acquired from the monitoring camera 12 to the object detection unit 54 and the teacher data output unit 55.

[0058] The object detection unit 54 performs a dangerous object detection process to detect a dangerous object captured in the captured image P using the learned model LM stored in the NVM 44. When the object detection unit 54 detects a dangerous object, it supplies the detection result of the dangerous object to the display control unit 53 and the camera control unit 50. Based on the detection result supplied from the object detection unit 54, the display control unit 53 displays the detected dangerous object on the display 15 in an identifiable manner. Based on the detection result supplied from the object detection unit 54, the camera control unit 50 changes the position of the imaging range 31 so that the detected dangerous object is located in the center of the imaging range 31. In other words, the camera control unit 50 changes the position of the imaging range 31 so as to track the dangerous object.

[0059] The camera control unit 50 also tracks a dangerous object detected in the normal area R1, and when the dangerous object enters the area of ​​interest R2, performs zooming to change the area including the dangerous object, and switches the operation mode of the surveillance camera 12 from automatic surveillance mode to manual surveillance mode. In the manual surveillance mode, the user can change the imaging range 31 using manual PTZ to check the dangerous object in detail.

[0060] In the manual monitoring mode, the teacher data output unit 55 generates and outputs teacher data TD in response to manual operations performed by the user on the monitoring camera 12 using the reception device 14. The teacher data TD output by the teacher data output unit 55 is stored by the NVM 44. The teacher data TD is, for example, a history of manual operations (i.e., the amount of operation related to manual PTZ). That is, in the manual monitoring mode, the history of manual operations performed when the user performs detailed checks on dangerous objects is output as teacher data TD.

[0061] The trained model LM is configured using a neural network. The trained model LM is configured using, for example, a deep neural network (DNN), which is a multilayer neural network that is the subject of deep learning. As the DNN, for example, a convolutional neural network (CNN) that targets images is used.

[0062] 5 shows an example of object detection processing by the object detection unit 54 using the trained model LM. In this embodiment, the trained model LM is configured by CNN. The object detection unit 54 inputs a captured image P as an input image to the trained model LM. The trained model LM generates a feature map FM representing the feature amounts of the captured image P using a convolutional layer.

[0063] The object detection unit 54 slides a window W of various sizes over the feature map FM and determines whether an object candidate exists within the window W. If the object detection unit 54 determines that an object candidate exists within the window W, it cuts out an image 60 within the window W, including the object candidate, from the feature map FM and inputs the cut-out image 60 to a classifier. The classifier outputs a label and a score for the object candidate included in the image 60. The label indicates the type of object. The score indicates the probability that the object candidate is an object of the type indicated by the label. Based on the label and score, the object detection unit 54 determines whether the object candidate is a dangerous object (such as heavy machinery), and if so, outputs detection information for the dangerous object. In the example shown in FIG. 5, a crane truck captured in the captured image P is detected as a dangerous object K.

[0064] In the example shown in FIG. 5, one dangerous object K is detected from the captured image P, but two or more dangerous objects K may be detected.

[0065] FIG. 6 shows an example of automatic tracking of a dangerous object K in the automatic monitoring mode. As described above, in the automatic monitoring mode, the position of the image capture range 31 is changed along the route 32 (see FIG. 2) set within the monitoring area 30, and the object detection unit 54 performs a dangerous object detection process each time the position of the image capture range 31 is changed. As shown in FIG. 6 as an example, when a dangerous object K is detected within the image capture range 31 in the normal area R1, the camera control unit 50 changes the position of the image capture range 31 so as to track the dangerous object K. In the example shown in FIG. 6, the position of the image capture range 31 is changed by panning and tilting the monitoring camera 12 in accordance with the movement of the dangerous object K so that the dangerous object K is positioned at the center.

[0066] When two or more dangerous objects are detected by the dangerous object detection process, the camera control unit 50 sets one dangerous object as the tracking target, for example, based on the label and the score.

[0067] FIG. 7 shows an example of switching the operation mode from the automatic monitoring mode to the manual monitoring mode. As shown in FIG. 7 as an example, when the camera control unit 50 detects that the dangerous object K that it has been tracking has entered the attention area R2, it causes the monitoring camera 12 to zoom in so as to enlarge the area including the dangerous object K. As a result, the dangerous object K is displayed in an enlarged form within the captured image P. Furthermore, when the camera control unit 50 detects that the dangerous object K that it has been tracking has entered the attention area R2, it switches the operation mode from the automatic monitoring mode to the manual monitoring mode. The user can confirm the dangerous object K by the enlarged display of the dangerous object K. If the user wants to check a part of the dangerous object K in more detail, the user can change the imaging range 31 using the manual PTZ.

[0068] FIG. 8 shows an example of manual PTZ in manual monitoring mode. As shown in FIG. 8 as an example, the camera control unit 50 changes the imaging range 31 in response to the user's operation of the receiving device 14. In the example shown in FIG. 8, the user focuses on the area below the hook H of a crane truck, which is a dangerous object K, and performs manual PTZ to enlarge the area below the hook H. This is because, in the case of a crane truck, the area below the hook H (or below the suspended load if a load is hung from the hook H) is highly dangerous, and it is preferable to check in detail whether workers or the like are present. In this way, the area of ​​interest that the user focuses on for detailed check of the dangerous object K varies depending on the type of dangerous object K. For example, if the dangerous object K is a roller cart, the user sets the area in front of the rollers as the area of ​​interest and checks in detail whether workers or the like are present.

[0069] After the user performs manual PTZ, the training data output unit 55 outputs the operation amounts (pan operation amount, tilt operation amount, and zoom operation amount) related to the manual PTZ as training data TD. It is not easy to detect a region of interest within a dangerous object K that the user focuses on for detailed confirmation by object detection, but by performing machine learning using training data TD that represents a history of manual operations for detailed confirmation, it becomes possible to detect the dangerous object K and also to estimate the region of interest.

[0070] Next, the operation of the monitoring system 10 will be described with reference to FIG.

[0071] Fig. 9 shows a flowchart illustrating an example of the flow of monitoring processing executed by the CPU 42. The flow of monitoring processing shown in Fig. 9 is an example of a "control method" according to the technique of the present disclosure. For convenience of explanation, the description will be given on the assumption that imaging by the imaging device 18 is performed at a default frame rate.

[0072] 9, first, in step S10, the camera control unit 50 starts the automatic monitoring mode. When the automatic monitoring mode starts, the monitoring camera 12 performs an imaging operation targeting the imaging range 31 (see FIG. 2) set within the monitoring area 30. After step S10, the monitoring process proceeds to step S11.

[0073] In step S11, the image acquisition unit 52 acquires the captured image P output from the surveillance camera 12 and supplies the acquired captured image P to the object detection unit 54. At this time, the captured image P is displayed on the display 15 via the display control unit 53. After step S11, the surveillance process proceeds to step S12.

[0074] In step S12, the object detection unit 54 uses the learned model LM to perform a dangerous object detection process to detect a dangerous object K from the captured image P (see FIG. 5). After step S12, the monitoring process proceeds to step S13.

[0075] In step S13, the camera control unit 50 determines whether or not a dangerous object K has been detected by the object detection unit 54. If an object has not been detected in step S13, the determination is negative, and the monitoring process proceeds to step S14. If an object has been detected in step S13, the determination is positive, and the monitoring process proceeds to step S15.

[0076] In step S14, the camera control unit 50 pans or tilts the surveillance camera 12 to change the imaging range 31 in the panning or tilting direction along the path 32 (see FIG. 2). After step S14, the monitoring process returns to step S11. In step S11, the image acquisition unit 52 again performs the captured image acquisition process.

[0077] In step S15, the camera control unit 50 performs automatic PTZ to change the imaging range 31 in accordance with the detection result of the dangerous object K detected by the object detection unit 54 (see FIG. 6). That is, the camera control unit 50 automatically tracks the dangerous object K. After step S15, the monitoring process proceeds to step S16.

[0078] In step S16, the camera control unit 50 determines whether the dangerous object K being tracked has entered the area of ​​interest R2 from the normal area R1. The determination of whether the dangerous object K has entered the area of ​​interest R2 is made, for example, based on the relationship between the coordinates related to the pan and tilt of the monitoring camera 12 and the coordinates of the area of ​​interest R2. In step S16, if the dangerous object K being tracked has not entered the area of ​​interest R2, the determination is negative, and the monitoring process returns to step S15. In step S16, if the dangerous object K being tracked has entered the area of ​​interest R2, the determination is positive, and the monitoring process proceeds to step S17.

[0079] In step S17, the camera control unit 50 causes the monitoring camera 12 to zoom in so as to enlarge the area including the dangerous object K (see FIG. 7). After step S17, the monitoring process proceeds to step S18.

[0080] In step S18, the camera control unit 50 switches the operation mode of the surveillance camera 12 from the automatic surveillance mode to the manual surveillance mode. After step S18, the surveillance process proceeds to step S19.

[0081] In step S19, the camera control unit 50 performs manual PTZ to change the imaging range 31 in accordance with an instruction given by the user to the receiving device 14 (see FIG. 8). If manual PTZ is not performed by the user, the camera control unit 50 automatically tracks the dangerous object K by automatic PTZ. After step S19, the monitoring process proceeds to step S20.

[0082] In step S20, the teacher data output unit 55 outputs the operation amounts (pan operation amount, tilt operation amount, and zoom operation amount) related to the manual PTZ as teacher data TD (see FIG. 8). After step S20, the monitoring process proceeds to step S21.

[0083] In step S21, the camera control unit 50 performs manual PTZ to change the imaging range 31 in response to an instruction given by the user to the receiving device 14. That is, manual monitoring using manual PTZ is continued by the user. Note that if manual PTZ is not performed by the user, the camera control unit 50 automatically tracks the dangerous object K using automatic PTZ. After step S21, the monitoring process proceeds to step S22.

[0084] In step S22, the camera control unit 50 determines whether the dangerous object K being tracked has left the area of ​​interest R2. Whether the dangerous object K being tracked has left the area of ​​interest R2 is determined, for example, based on the relationship between the coordinates related to the pan and tilt of the monitoring camera 12 and the coordinates of the area of ​​interest R2. If the dangerous object K being tracked has left the area of ​​interest R2 in step S22, the monitoring process returns to step S10. If the dangerous object K being tracked has not left the area of ​​interest R2 in step S22, the monitoring process proceeds to step S23.

[0085] In step S23, the camera control unit 50 determines whether or not a condition for terminating the monitoring process (hereinafter referred to as the "termination condition") has been satisfied. One example of the termination condition is that an instruction to terminate the monitoring process has been accepted by the accepting device 14. In step S23, if the termination condition has not been satisfied, the determination is negative, and the monitoring process returns to step S21. In step S23, if the termination condition has been satisfied, the determination is positive, and the monitoring process ends.

[0086] As described above, management device 16 as a control device can switch between a first monitoring mode (automatic monitoring mode) and a second monitoring mode (manual monitoring mode) that, in addition to the first monitoring mode, enables capturing images of an object in response to a manual operation performed on monitoring camera 12. Management device 16 divides monitoring area 30 into a first area (normal area R1) and a second area (attention area R2), and switches the operation mode from the first monitoring mode to the second monitoring mode in response to an object (dangerous object K) that has been tracked in the first area in the first monitoring mode entering the second area. This reduces the burden on the user associated with monitoring the object.

[0087] Furthermore, in the first monitoring mode, the management device 16 captures the monitored area 30 by panning and tilting the monitoring camera 12 to change the imaging range 31, so that the monitored area 30 can be made wider than the imaging range 31.

[0088] Furthermore, when an object that has been tracked in automatic monitoring mode in the first area enters the second area, the management device 16 causes the surveillance camera 12 to zoom to change the area that includes the object, thereby reducing the burden on the user associated with zooming operations.

[0089] Furthermore, in the second monitoring mode, the management device 16 causes the monitoring camera to perform at least one of panning, tilting, and zooming in response to manual operation, allowing the user to check the details of the object according to their own intention.

[0090] Furthermore, in the second monitoring mode, the management device 16 outputs the history of manual operations as training data obtained by machine learning, and therefore can efficiently collect training data TD for improving monitoring accuracy.

[0091] Furthermore, since management device 16 switches the operation mode from the second monitoring mode to the first monitoring mode in response to the object leaving the second area, it is possible to reduce the burden on the user involved in monitoring the object. Furthermore, since management device 16 detects the object using a trained model based on machine learning in the first monitoring mode, it is possible to reduce the burden on the user involved in monitoring the object.

[0092] Various modifications of the above embodiment will be described below.

[0093] [First Modification] In the above embodiment, the teacher data output unit 55 outputs the operation amounts (pan operation amount, tilt operation amount, and zoom operation amount) related to the manual operation (manual PTZ) as the teacher data TD. Instead, in the first modified example, as shown in an example in FIG. 10, the teacher data output unit 55 outputs the coordinates of the imaging range 31 before and after the manual operation as the teacher data TD. In the example shown in FIG. 10, (xa1, ya1) and (xb1, yb1) respectively represent the upper left coordinate and lower right coordinate of the imaging range 31 before the manual operation. (xa2, ya2) and (xb2, yb2) respectively represent the upper left coordinate and lower right coordinate of the imaging range 31 after the manual operation.

[0094] [Second Modification] 11 as an example, in the second modified example, the training data output unit 55 outputs the captured image P captured by the surveillance camera 12 after manual operation (manual PTZ) as training data TD. The captured image P after manual operation represents an area of ​​interest focused on by the user, and therefore, by machine learning the features of structures such as heavy machinery (for example, tires of the heavy machinery, traffic cones, etc.) that appear in the captured image P after manual operation, it becomes possible to automatically estimate the area focused on by the user in an image including a dangerous object K.

[0095] [Third Modification] In the above embodiment, after a dangerous object K enters the area of ​​interest R2 and a manual operation is performed in manual monitoring mode, causing the teacher data output unit 55 to output teacher data TD, manual monitoring continues until the dangerous object K leaves the area of ​​interest R2. In contrast, in the third modified example, after a manual operation is performed in manual monitoring mode, causing the teacher data output unit 55 to output teacher data TD, an object in the area of ​​interest that the user has focused on is set as a tracking target by manual operation, and the dangerous object K is automatically tracked until it leaves the area of ​​interest R2.

[0096] Fig. 12 shows an example of the flow of the monitoring process according to this modification. In this modification, steps S30 and S31 are executed between steps S20 and S22 instead of step S21 shown in Fig. 9. In this modification, after the teacher data output unit 55 outputs the teacher data TD in step S20, the monitoring process proceeds to step S30.

[0097] In step S30, the camera control unit 50 sets, as a tracking target, an object that appears in the captured image P captured by the monitoring camera 12 after manual operation. For example, the camera control unit 50 sets the object as a tracking target based on the result of object detection detected by the object detection unit 54. After step S30, the monitoring process proceeds to step S31.

[0098] In step S31, the camera control unit 50 , additional The automatic PTZ is performed to track the target, thereby changing the imaging range 31. The camera control unit 50 causes the surveillance camera 12 to perform at least one of panning, tilting, and zooming. After step S31, the surveillance process proceeds to step S22.

[0099] In this modification, if the determination in step S23 is negative, the monitoring process returns to step S31.

[0100] Fig. 13 shows an example of the automatic tracking performed by steps S30 and S31. As an example, as shown in Fig. 13, the camera control unit 50 sets the hook H of the crane truck as the object to be tracked, which is an object that appears in the captured image P after manual operation, and executes automatic PTZ to track the hook H.

[0101] In this way, according to this modified example, it is possible to automatically track the part that the user has checked in detail in manual monitoring mode, thereby reducing the burden on the user in monitoring the target object.

[0102] In the above embodiment and each of the above modifications, in the automatic monitoring mode, the monitoring area 30 is monitored by having the monitoring camera 12 pan and tilt, and moving the imaging range 31 along a path 32 (see FIG. 2) within the monitoring area 30. Alternatively, the monitoring camera 12 may be configured not to pan and tilt, but to have the angle of view set to a "wide angle" so that the imaging range 31 of the monitoring camera 12 includes the entire monitoring area 30, and to capture an image of a bird's-eye view of the entire monitoring area 30.

[0103] In this case, the camera control unit 50 tracks the dangerous object K based on the overhead image, and when the dangerous object K being tracked enters the area of ​​interest R2, it switches the operation mode to manual monitoring mode and switches the angle of view of the surveillance camera 12 to "standard." After this, the user manually switches the angle of view of the surveillance camera 12 to "telephoto" and checks in detail. Note that "wide angle," "standard," and "telephoto" express relative relationships and do not represent specific positions such as both ends and the center of the focal length of a zoom lens.

[0104] Furthermore, in each of the above embodiments, the program PG for the monitoring process is stored in the NVM 44 (see FIG. 3), but the technology of the present disclosure is not limited to this, and as an example, the program PG may be stored in any portable storage medium 100 that is a non-transitory storage medium such as an SSD or a USB memory, as shown in FIG. 14. In this case, the program PG stored in the storage medium 100 is installed in the computer 40, and the CPU 42 executes the above-described monitoring process in accordance with the program PG.

[0105] Alternatively, the program PG may be stored in a storage device such as another computer or server device connected to the computer 40 via a communication network (not shown), and the program PG may be downloaded and installed in the computer 40 in response to a request from the management device 16. In this case, the monitoring process is executed by the computer 40 in accordance with the installed program PG.

[0106] The hardware resources that execute the above-described monitoring process can be various processors, as listed below. Examples of processors include a CPU, which is a general-purpose processor that functions as a hardware resource that executes the monitoring process by executing software, i.e., a program PG, as described above. Examples of processors include dedicated electrical circuits, such as FPGAs, PLDs, or ASICs, which are processors with circuit configurations specifically designed to execute specific processes. Each processor has built-in or connected memory, and executes the monitoring process by using the memory.

[0107] The hardware resource that executes the monitoring process may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the monitoring process may be a single processor.

[0108] Examples of systems configured with a single processor include, first, a system in which one processor is configured with a combination of one or more CPUs and software, as typified by client and server computers, and this processor functions as a hardware resource that executes monitoring processing. Second, a system in which a processor is used to implement the functions of the entire system, including multiple hardware resources that execute monitoring processing, on a single IC chip, as typified by SoCs. In this way, monitoring processing is implemented using one or more of the various processors described above as hardware resources.

[0109] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements.

[0110] Furthermore, the above-described monitoring process is merely an example, and it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[0111] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0112] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0113] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. A control device including a processor for controlling a surveillance camera that monitors a surveillance area, The processor: The operation mode of the surveillance camera is a first surveillance mode in which an object present in the surveillance area is detected and tracked; a first monitoring mode for capturing an image of the object in response to a manual operation performed on the monitoring camera; Dividing the monitoring area into a first area and a second area; When the object being tracked in the first area in the first monitoring mode enters the second area, the operation mode is switched from the first monitoring mode to the second monitoring mode, and the monitoring camera is caused to zoom in so as to enlarge an area including the object; In the second monitoring mode, the monitoring camera is caused to perform at least one of panning, tilting, and zooming in response to the manual operation, and an image captured by the monitoring camera after the manual operation and a history of the manual operation are output as training data; switching the operation mode from the second monitoring mode to the first monitoring mode in response to the object moving from the second area to the first area; generating a trained model by performing machine learning using the training data; In the first monitoring mode, the object is detected and tracked using the generated trained model. Control device.

2. The processor: In the first monitoring mode, the monitoring camera is panned and tilted to change an imaging range, thereby capturing an image of the monitoring area. The control device according to claim 1 .

3. The second monitoring mode is a mode in which, in addition to the first monitoring mode, an image of the object is captured in response to a manual operation performed on the monitoring camera. The control device according to claim 1 or 2.

4. The processor: In the second monitoring mode, an object that appears in an image captured by the monitoring camera after the manual operation is detected, and the monitoring camera is made to track the detected object by performing at least one of panning, tilting, and zooming. The control device according to claim 3 .

5. A control method for controlling a surveillance camera that monitors a surveillance area, comprising: The operation mode of the surveillance camera is switchable between a first surveillance mode in which an object present in the surveillance area is detected and tracked, and a second surveillance mode in which an image of the object is captured in response to a manual operation performed on the surveillance camera; Dividing the monitoring area into a first area and a second area; switching the operation mode from the first monitoring mode to the second monitoring mode in response to the object being tracked in the first area in the first monitoring mode entering the second area, and causing the monitoring camera to zoom in so as to enlarge an area including the object; in the second monitoring mode, causing the monitoring camera to perform at least one of panning, tilting, and zooming in accordance with the manual operation, and outputting an image captured by the monitoring camera after the manual operation and a history of the manual operation as training data; switching the operation mode from the second monitoring mode to the first monitoring mode in response to the object moving from the second area to the first area; generating a trained model by performing machine learning using the training data; In the first monitoring mode, detecting and tracking the object using the generated trained model; A control method comprising:

6. The operation mode of the surveillance camera can be switched between a first surveillance mode in which an object present in a surveillance area is detected and tracked, and a second surveillance mode in which an image of the object is captured in response to a manual operation performed on the surveillance camera. Dividing the monitoring area into a first area and a second area; switching the operation mode from the first monitoring mode to the second monitoring mode in response to the object being tracked in the first area in the first monitoring mode entering the second area, and causing the monitoring camera to zoom in so as to enlarge an area including the object; in the second monitoring mode, causing the monitoring camera to perform at least one of panning, tilting, and zooming in accordance with the manual operation, and outputting an image captured by the monitoring camera after the manual operation and a history of the manual operation as training data; switching the operation mode from the second monitoring mode to the first monitoring mode in response to the object moving from the second area to the first area; generating a trained model by performing machine learning using the training data; In the first monitoring mode, detecting and tracking the object using the generated trained model; A program for causing a computer to execute a process including the above.

Images