Control device, control method, and program

Abstract

This control device is provided with a processor that controls a monitoring camera. The processor is capable of performing switching between a first monitoring mode in which the processor makes the monitoring camera perform image-capturing to acquire a first captured image and changes an image-capturing range in accordance with a received instruction and a second monitoring mode in which the processor makes the monitoring camera perform image-capturing to acquire a second captured image and uses a learned model obtained by machine learning to detect an object appearing in the second captured image and changes the image-capturing range in accordance with a result of detection. The processor outputs the first captured image acquired in the first monitoring mode as a training image with respect to the machine learning.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a control device, a control method, and a program. [Background technology]

[0002] Japanese Patent Application Laid-Open Publication No. 2004-056473 describes a monitoring control device that is provided with a neural network (NW) that outputs recognition information corresponding to images captured by a camera based on learning results, a control means that performs control based on this recognition information, a short-term memory means that temporarily saves image data, and a storage means that records this image data, and that the NW learns the relationship between the image and the degree of urgency of the event depicted in the image, recognizes the degree of urgency corresponding to the camera image, and the control means controls the frame rate of the image data recorded in the storage means based on the recognition information of the NW.

[0003] JP 2006-523043 A describes a method for detecting a moving object and controlling a surveillance system, which includes a processing module adapted to receive image information from at least one image-forming sensor. The surveillance system performs motion detection analysis on the captured images and controls the camera in a specific manner when a moving object is detected.

[0004] JP 2009-516480 A discloses a method and system for a video surveillance system including a plurality of video cameras each having a field of view, the cameras configured to perform at least one of zooming to change the camera field of view, tilting to rotate the camera about a horizontal tilt axis, and panning to rotate the camera about a vertical pan axis. The system further includes a processor configured to receive a signal representing an image within the field of view of at least one of the video cameras, recognize a target using the received signal, determine a direction to the target from the camera recognizing the target, and transmit the determined direction to the other cameras of the plurality of video cameras. Summary of the Invention

[0005] One embodiment of the technology disclosed herein provides a control device, a control method, and a program that can efficiently collect teacher images to be used in machine learning. [Means for solving the problem]

[0006] The control device of the present disclosure is a control device that includes a processor for controlling a surveillance camera, and the processor switches between a first monitoring mode in which the surveillance camera captures an image to obtain a first captured image and changes the capturing range in accordance with given instructions, and a second monitoring mode in which the surveillance camera captures an image to obtain a second captured image and uses a trained model through machine learning to detect objects that appear in the second captured image and changes the capturing range in accordance with the detection results, and outputs the first captured image captured in the first monitoring mode as a teacher image for machine learning.

[0007] Preferably, the processor outputs the first captured image as a teacher image in response to a manual operation performed on the surveillance camera.

[0008] The manual operation is a switching operation from the second monitoring mode to the first monitoring mode, and it is preferable that the processor outputs the first captured image acquired in the first monitoring mode after switching from the second monitoring mode to the first monitoring mode as a teacher image.

[0009] The surveillance camera is capable of changing the imaging range by changing at least one of pan, tilt, and zoom, and it is preferable that the switching operation is an operation of changing at least one of pan, tilt, and zoom in the second surveillance mode.

[0010] Preferably, after switching from the second monitoring mode to the first monitoring mode, the processor outputs the first captured image as a teacher image in response to a given output instruction.

[0011] It is preferable that the processor outputs the second captured image acquired in the second monitoring mode before the switch as a teacher image, with a judgment result indicating that the image is unsuitable for object detection, and outputs the first captured image acquired in the first monitoring mode after the switch as a teacher image, with a judgment result indicating that the image is suitable for object detection.

[0012] Preferably, after switching from the second monitoring mode to the first monitoring mode, the processor switches to the second monitoring mode if no operation is performed for a certain period of time in the first monitoring mode.

[0013] It is preferable that, after switching from the first monitoring mode to the second monitoring mode, if a manual operation is performed after a certain time has elapsed since the previous manual operation, the processor does not output the second captured image as a teacher image.

[0014] Preferably, the processor detects an object captured in the teacher image and adds position information of the detected object in the teacher image to the teacher image.

[0015] The processor preferably sets a lower detection standard for object detection when detecting an object captured in the teacher image than for object detection when detecting an object captured in the second captured image.

[0016] The processor preferably performs a location information change process that changes the location information in response to a given instruction.

[0017] Preferably, the processor determines the position of an object appearing in the teacher image in response to a given instruction, and adds position information of the determined object in the teacher image to the teacher image.

[0018] In addition to the teacher image, the processor preferably outputs an extended image generated by performing an extension process on the teacher image as the teacher image.

[0019] The extension process is preferably at least one of inversion, reduction, noise addition, and style change using deep learning.

[0020] The control method disclosed herein includes being able to switch between a first monitoring mode in which a surveillance camera captures an image to obtain a first captured image and changes the capturing range in response to given instructions, and a second monitoring mode in which a surveillance camera captures an image to obtain a second captured image and uses a trained model through machine learning to detect objects that appear in the second captured image and changes the capturing range in response to the detection results, and includes outputting the first captured image captured in the first monitoring mode as a training image for machine learning.

[0021] The program disclosed herein allows a computer to switch between a first monitoring mode in which a surveillance camera captures an image to obtain a first captured image and changes the capturing range in response to given instructions, and a second monitoring mode in which a surveillance camera captures an image to obtain a second captured image and uses a trained model through machine learning to detect objects that appear in the second captured image and changes the capturing range in response to the detection results, and causes a computer to execute processing that includes outputting the first captured image captured in the first monitoring mode as a training image for machine learning. [Brief explanation of the drawings]

[0022] [Figure 1] 1 is a schematic configuration diagram showing an example of the overall configuration of a monitoring system according to a first embodiment. [Figure 2] FIG. 2 is a block diagram showing an example of a hardware configuration of a monitoring camera and a management device. [Figure 3] FIG. 2 is a block diagram showing an example of the functions of a CPU included in the management device. [Figure 4] FIG. 10 is a conceptual diagram showing an example of manual PTZ in manual monitoring mode. [Figure 5] FIG. 10 is a conceptual diagram illustrating an example of an object detection process. [Figure 6]FIG. 10 is a conceptual diagram showing an example of automatic PTZ in automatic monitoring mode. [Figure 7] FIG. 10 is a conceptual diagram showing an example in which an object is erroneously detected in the automatic monitoring mode. [Figure 8] FIG. 10 is a conceptual diagram showing an example in which manual PTZ is performed in automatic monitoring mode. [Figure 9] FIG. 10 is a conceptual diagram illustrating an example of a learning process. [Figure 10] 10 is a flowchart showing an example of the flow of a monitoring process according to the first embodiment. [Figure 11] 10 is a flowchart showing an example of the flow of a monitoring process according to the second embodiment. [Figure 12] 11 is a flowchart showing an example of the flow of a monitoring process according to the third embodiment. [Figure 13] 13 is a flowchart showing an example of the flow of a monitoring process according to the fourth embodiment. [Figure 14] FIG. 10 is a conceptual diagram showing a first modified example of the teacher image output process. [Figure 15] FIG. 10 is a conceptual diagram showing a second modified example of the teacher image output process. [Figure 16] FIG. 10 is a conceptual diagram showing a third modified example of the teacher image output process. [Figure 17] FIG. 10 is a conceptual diagram showing a fourth modified example of the teacher image output process. [Figure 18] FIG. 10 is a conceptual diagram illustrating a modified example of object detection. [Figure 19] FIG. 10 is a block diagram showing an example of how an imaging processing program stored in a storage medium is installed in a computer. DETAILED DESCRIPTION OF THE INVENTION

[0023] Hereinafter, examples of a control device, a control method, and a program according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0024] First, the terms used in the following description will be explained.

[0025] CPU is an abbreviation for "Central Processing Unit". NVM is an abbreviation for "Non-volatile memory". RAM is an abbreviation for "Random Access Memory". IC is an abbreviation for "Integrated Circuit". ASIC is an abbreviation for "Application Specific Integrated Circuit". PLD is an abbreviation for "Programmable Logic Device". FPGA is an abbreviation for "Field-Programmable Gate Array". SoC is an abbreviation for "System-on-a-chip". SSD is an abbreviation for "Solid State Drive". USB is an abbreviation for "Universal Serial Bus". HDD is an abbreviation for "Hard Disk Drive". EEPROM is an abbreviation for "Electrically Erasable and Programmable Read Only Memory". EL is an abbreviation for "Electro-Luminescence". I / F is an abbreviation for "Interface". CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor." CCD is an abbreviation for "Charge Coupled Device." SWIR is an abbreviation for "Short Wave Infra-Red." LAN is an abbreviation for "Local Area Network."

[0026] [First embodiment] As an example, as shown in FIG. 1, a monitoring system 10 includes a monitoring camera 12 and a management device 16. The monitoring system 10 is, for example, a system for monitoring a construction site. The monitoring camera 12 is installed at a high location, such as the rooftop of a building near the construction site. The management device 16 is used by a user, for example, a site supervisor who supervises workers at the construction site. The user uses the management device 16 to monitor, for example, whether any danger has arisen at the construction site during work. The monitoring system 10 is a system for reducing the monitoring burden on the user.

[0027] The surveillance camera 12 includes an imaging device 18 and a rotation device 20. The imaging device 18 captures an image of a subject by receiving, for example, light in the visible wavelength band reflected from the subject. The imaging device 18 may also capture an image of a subject by receiving near-infrared light, which is light in the short-wave infrared wavelength band reflected from the subject. The short-wave infrared wavelength band refers to, for example, a wavelength band of approximately 900 nm to 2500 nm. Light in the short-wave infrared wavelength band is also generally referred to as SWIR light.

[0028] The imaging device 18 is attached to a swivel device 20. The swivel device 20 swivels the imaging device 18. For example, the swivel device 20 changes the imaging direction of the imaging device 18 in a pan direction and a tilt direction. The pan direction is, for example, the horizontal direction. The tilt direction is, for example, the vertical direction.

[0029] The rotation device 20 includes a base 22, a panning rotation member 24, and a tilting rotation member 26. The panning rotation member 24 is cylindrical and attached to the upper surface of the base 22. The tilting rotation member 26 is arm-shaped and attached to the outer circumferential surface of the panning rotation member 24. The imaging device 18 is attached to the tilting rotation member 26. The tilting rotation member 26 rotates around a tilt axis TA parallel to the horizontal direction, thereby changing the imaging direction of the imaging device 18 in the tilt direction.

[0030] The base 22 supports from below the panning rotation member 24. The panning rotation member 24 rotates about a pan axis PA that is parallel to the vertical direction, thereby changing the imaging direction of the imaging device 18 in the pan direction.

[0031] The base 22 has built-in drive sources (for example, the pan motor 24A and tilt motor 26A shown in FIG. 2). The drive source of the base 22 is mechanically connected to the pan motor 24A and the tilt motor 26A. For example, the drive source of the base 22 is connected to the pan rotation member 24 and the tilt rotation member 26 via a power transmission mechanism (not shown). The pan rotation member 24 rotates about the pan axis PA by receiving power from the drive source of the base 22, and the tilt rotation member 26 rotates about the tilt axis TA by receiving power from the drive source of the base 22.

[0032] As shown in Fig. 1, the monitoring system 10 generates a captured image by capturing an image of an imaging range 31 set within a monitoring area 30 using an imaging device 18. The monitoring system 10 captures an image of the entire monitoring area 30 by panning and tilting to change the imaging range 31. A construction site as the monitoring area 30 contains various subjects such as heavy machinery and workers. Heavy machinery includes power shovels, bulldozers, crane trucks, dump trucks, etc.

[0033] The imaging device 18 is, for example, a digital camera having an image sensor (not shown). The image sensor receives subject light indicative of a subject, photoelectrically converts the received subject light, and outputs an electrical signal having a signal level according to the amount of received light as image data. The image data output by the image sensor corresponds to the captured image. The image sensor is a CMOS image sensor, a CCD image sensor, or the like. The imaging device 18 may capture a color image or a monochrome image. The captured image may be a still image or a moving image.

[0034] The imaging device 18 also has a zoom function. The zoom function is a function for reducing or enlarging (i.e., zooming in or out) the imaging range 31. The zoom function provided by the imaging device 18 is an optical zoom function that moves a zoom lens, or an electronic zoom function that performs image processing on image data. Note that the zoom function provided by the imaging device 18 may be a combination of the optical zoom function and the electronic zoom function.

[0035] The management device 16 includes a management device main body 13, a reception device 14, and a display 15. The management device main body 13 has a built-in computer 40 (see FIG. 2) and controls the entire monitoring system 10. The reception device 14 and the display 15 are connected to the management device main body 13.

[0036] The receiving device 14 receives various instructions from a user who uses the monitoring system 10. Examples of the receiving device 14 include a keyboard, a mouse, and / or a touch panel. The various instructions received by the receiving device 14 are grasped by the management apparatus main body 13. The display 15 displays various information (e.g., images and text) under the control of the management apparatus main body 13. Examples of the display 15 include a liquid crystal display and an EL display.

[0037] The surveillance camera 12 is communicably connected to the management device 16 via a communication network NT (such as the Internet or a LAN), and operates under the control of the management device main body 13. The connection between the surveillance camera 12 and the management device 16 may be either a wired connection or a wireless connection.

[0038] The management device 16 acquires captured images output from the imaging device 18 of the surveillance camera 12 and detects specific objects (e.g., heavy machinery) that appear in the captured images using a trained model based on machine learning. When the management device 16 detects a specific object, it causes the surveillance camera 12 to pan, tilt, and zoom so as to track the detected object. Hereinafter, the operation of changing the imaging range 31 by panning, tilting, and zooming will be referred to as "PTZ." Furthermore, the operation of changing the imaging range 31 in accordance with the detection result of an object that appears in the captured image will be referred to as "automatic PTZ."

[0039] Furthermore, the management device 16 enables an operation to change the imaging range 31 in response to a user's operation of the receiving device 14. Hereinafter, the operation to change the imaging range 31 in response to an instruction given to the receiving device 14 will be referred to as "manual PTZ." In manual PTZ, the user can set the imaging range 31 to any position and size within the monitoring area 30 by operating the receiving device 14.

[0040] In the following, the monitoring mode in which the monitoring area 30 is monitored using manual PTZ is referred to as the "manual monitoring mode," and the monitoring mode in which the monitoring area 30 is monitored using automatic PTZ is referred to as the "automatic monitoring mode." The user can switch the monitoring mode of the monitoring system 10 between the manual monitoring mode and the automatic monitoring mode. The manual monitoring mode is an example of a "first monitoring mode" according to the technology of the present disclosure. The automatic monitoring mode is an example of a "second monitoring mode" according to the technology of the present disclosure.

[0041] 2, the rotation device 20 of the surveillance camera 12 includes a controller 34. The controller 34 controls the operation of the pan motor 24A, the tilt motor 26A, and the imaging device 18 under the control of the management device 16.

[0042] The management device main body 13 of the management device 16 includes a computer 40. The computer 40 has a CPU 42, an NVM 44, a RAM 46, and a communication I / F 48. The management device 16 is an example of a "control device" according to the technology of the present disclosure. The computer 40 is an example of a "computer" according to the technology of the present disclosure. The CPU 42 is an example of a "processor" according to the technology of the present disclosure.

[0043] The CPU 42, NVM 44, RAM 46, and communication I / F 48 are connected to a bus 49. In the example shown in Fig. 2, for convenience of illustration, one bus is shown as the bus 49, but multiple buses may be used. The bus 49 may be a serial bus or a parallel bus including a data bus, an address bus, a control bus, etc.

[0044] The NVM 44 stores various types of data. Here, examples of the NVM 44 include various types of nonvolatile storage devices such as an EEPROM, an SSD, and / or an HDD. The RAM 46 temporarily stores various types of information and is used as a work memory. Examples of the RAM 46 include a DRAM or an SRAM.

[0045] A program PG is stored in the NVM 44. The CPU 42 reads out a necessary program from the NVM 44 and executes the read program PG on the RAM 46. The CPU 42 controls the entire monitoring system 10, including the management device 16, by executing processing in accordance with the program PG.

[0046] The communication I / F 48 is an interface realized by hardware resources such as FPGA, etc. The communication I / F 48 is communicably connected to the controller 34 of the surveillance camera 12 via the communication network NT, and transmits and receives various information between the CPU 42 and the controller 34.

[0047] The bus 49 is also connected to the accepting device 14 and the display 15, and the CPU 42 operates in accordance with instructions received by the accepting device 14 and causes the display 15 to display various types of information.

[0048] The NVM 44 also stores a trained model LM for performing the above-mentioned object detection. The trained model LM is a trained model for object detection generated by machine learning using a plurality of training images in which specific objects are captured. The NVM 44 also stores training images TD. The training images TD are training images for additional learning that are used to cause the trained model LM to perform additional learning. The training images TD are images that satisfy predetermined conditions among the captured images acquired by the surveillance camera 12.

[0049] 3, the CPU 42 executes operations based on the program PG to realize multiple functional units. The program PG causes the CPU 42 to function as a camera control unit 50, a mode switching control unit 51, an image acquisition unit 52, a display control unit 53, an object detection unit 54, a teacher image output unit 55, and a machine learning unit 56.

[0050] The camera control unit 50 controls the controller 34 of the surveillance camera 12 to cause the imaging device 18 to perform imaging operations and zoom, and to cause the swivel device 20 to pan and tilt. In other words, the camera control unit 50 causes the surveillance camera 12 to perform imaging operations and change the imaging range 31.

[0051] The mode switching control unit 51 performs switching control to switch the monitoring mode of the monitoring system 10 between an automatic monitoring mode and a manual monitoring mode based on an instruction received by the receiving device 14. In the manual monitoring mode, the mode switching control unit 51 causes the camera control unit 50 to perform manual PTZ, which changes the imaging range 31 in accordance with an instruction given to the receiving device 14. In the automatic monitoring mode, the mode switching control unit 51 causes the camera control unit 50 to perform automatic PTZ, which changes the imaging range 31 in accordance with the result of object detection by the object detection unit 54.

[0052] The image acquisition unit 52 acquires the captured image P output from the surveillance camera 12 when the camera control unit 50 causes the surveillance camera 12 to capture an image. The image acquisition unit 52 supplies the captured image P acquired from the surveillance camera 12 to the display control unit 53. The display control unit 53 displays the captured image P supplied from the image acquisition unit 52 on the display 15.

[0053] In the manual monitoring mode, the image acquisition unit 52 supplies the captured image P acquired from the monitoring camera 12 as a first captured image P1 to the teacher image output unit 55. On the other hand, in the automatic monitoring mode, the image acquisition unit 52 supplies the captured image P acquired from the monitoring camera 12 to the object detection unit 54 as a second captured image P2.

[0054] The object detection unit 54 detects a specific object (e.g., heavy machinery) that appears in the second captured image P2 using the trained model LM stored in the NVM 44. The object detection unit 54 supplies the detection result of the object detection to the display control unit 53 and the camera control unit 50. Based on the detection result supplied from the object detection unit 54, the display control unit 53 displays the detected object on the display 15 in an identifiable manner. Based on the detection result supplied from the object detection unit 54, the camera control unit 50 changes the imaging range 31 so that the detected object is located in the center of the imaging range 31 and so that the detected object is enlarged.

[0055] The teacher image output unit 55 stores the first captured image P1 as a teacher image TD in the NVM 44 in response to a manual operation performed by the user on the surveillance camera 12 using the reception device 14. In this embodiment, in response to a switching operation performed by the user to switch from the automatic monitoring mode to the manual monitoring mode using the reception device 14, the teacher image output unit 55 stores the first captured image P1 acquired in the manual monitoring mode after the switch as a teacher image TD in the NVM 44. The switching operation is an operation to change at least one of pan, tilt, and zoom.

[0056] The machine learning unit 56 updates the trained model LM by performing additional learning on the trained model LM using the teacher images TD stored in the NVM 44. For example, when a certain number of teacher images TD have been accumulated in the NVM 44, the machine learning unit 56 causes the trained model LM to perform additional learning using the accumulated multiple teacher images TD. When the trained model LM is updated, the object detection unit 54 performs object detection using the updated trained model LM.

[0057] The trained model LM is configured using a neural network. The trained model LM is configured using, for example, a deep neural network (DNN), which is a multilayer neural network that is the subject of deep learning. As the DNN, for example, a convolutional neural network (CNN) that targets images is used.

[0058] Fig. 4 shows an example of manual PTZ in manual monitoring mode. In the example shown in Fig. 4, two heavy machinery H1 and H2 are shown as objects in the captured image P displayed on the display 15. The user can change the imaging range 31 to an area of ​​interest by operating the keyboard, mouse, or the like serving as the receiving device 14. Fig. 4 shows an operation in which heavy machinery H2, near which a person is present, is focused as a monitoring target, and the imaging range 31 is changed so that the area of ​​interest including the heavy machinery H2 coincides with the imaging range 31.

[0059] 5 shows an example of object detection processing by the object detection unit 54 using the trained model LM. In this embodiment, the trained model LM is configured by CNN. The object detection unit 54 inputs the second captured image P2 as an input image to the trained model LM. The trained model LM generates a feature map FM representing the feature amounts of the second captured image P2 using a convolutional layer.

[0060] The object detection unit 54 slides a window W of various sizes over the feature map FM and determines whether an object candidate exists within the window W. If the object detection unit 54 determines that an object candidate exists within the window W, it cuts out an image R within the window W containing the object candidate from the feature map FM and inputs the cut-out image R to a classifier. The classifier outputs a label and a score for the object candidate contained in the image R. The label represents the type of object. The score represents the probability that the object candidate is an object of the type represented by the label. In the example shown in FIG. 5, a heavy machine H1 is extracted as an object candidate, and the classifier determines that the label of the heavy machine H1 is a "power shovel." Furthermore, the score representing the probability that the heavy machine H1 is a "power shovel" is "0.90."

[0061] The object detection unit 54 outputs, as a detection result, the position information of the image R including an object whose score is equal to or greater than a certain value, the label, and the score. Note that, although one object is detected from the second captured image P2 in the example shown in Fig. 5, two or more objects may be detected.

[0062] As shown in FIG. 6 as an example, the display control unit 53 displays a rectangular frame F in the captured image P so as to surround the object detected by the object detection unit 54. The display control unit 53 also displays a label L indicating the type of object in the frame F near the frame F. The display control unit 53 may also display a score. If two or more objects are detected, the display control unit 53 displays multiple frames F in the captured image P.

[0063] Fig. 6 shows an example of automatic PTZ in automatic monitoring mode. In the example shown in Fig. 6, two heavy machinery H1 and H2 are captured as objects in the captured image P displayed on the display 15, with the heavy machinery H1 being detected by the object detection unit 54 and the heavy machinery H2 not being detected as an object. In this case, the camera control unit 50 performs control to change the imaging range 31 so that the area including the heavy machinery H1 coincides with the imaging range 31. As a result, automatic PTZ is performed to track the heavy machinery H1.

[0064] In addition, if there are two or more objects detected by the object detection unit 54 in the captured image P, for example, the camera control unit 50 controls to change the imaging range 31 so that the area containing the object with the highest score matches the imaging range 31.

[0065] FIG. 7 shows an example in which an object is erroneously detected by the object detection unit 54 in the automatic monitoring mode. In the example shown in FIG. 7, a vehicle that is not a heavy machine is erroneously detected as a power shovel, and automatic PTZ is performed to track the vehicle. However, when an object that is not a monitoring target is erroneously detected in this way, the monitoring target may be located in another area. For example, in the example shown in FIG. 7, the user focuses on heavy machine H2, which has a person nearby, as a monitoring target. In this case, the user operates the receiving device 14 to perform manual PTZ so that the area of ​​interest including heavy machine H2 matches the imaging range 31. In this way, after an object is detected in the automatic monitoring mode, if the object is not the user's intended monitoring target, the user may operate the receiving device 14 to perform manual PTZ.

[0066] Fig. 8 shows an example in which manual PTZ is performed in the automatic monitoring mode. In the example shown in Fig. 8, in response to automatic PTZ being performed on a car that has been erroneously detected by the object detection unit 54, the user performs manual PTZ so that the user's intended area of ​​interest (see Fig. 7) coincides with the imaging range 31. In the automatic monitoring mode, when the user operates the receiving device 14 to perform manual PTZ, the mode switching control unit 51 switches the monitoring mode from the automatic monitoring mode to the manual monitoring mode.

[0067] In response to the monitoring mode being switched from the automatic monitoring mode to the manual monitoring mode, the teacher image output unit 55 outputs the first captured image P1 after switching to the manual monitoring mode as the teacher image TD. For example, when the monitoring mode is switched to the manual monitoring mode by the user performing manual PTZ, the teacher image output unit 55 outputs the first captured image P1 at the time when the manual PTZ is stopped as the teacher image TD.

[0068] 9 shows an example of a learning process in which a trained model LM undergoes additional learning using a teacher image TD. In the learning process, a teacher image TD is input to the trained model LM. A correct answer label L1 indicating the type of object included in the first captured image P1 is assigned to the teacher image TD. Since the object captured in the first captured image P1 is an object that was not detected by the object detection unit 54, the correct answer label L1 is assigned by, for example, a user by determining the type of object.

[0069] The trained model LM outputs a detection result RT based on the input training image TD. The detection result RT is the above-mentioned label L and score. A loss calculation is performed using a loss function based on this detection result RT and the correct label L1. Then, various coefficients (weighting coefficients, biases, etc.) of the trained model LM are updated according to the result of the loss calculation, and the trained model LM is updated according to the update settings.

[0070] The label L may simply be a label indicating whether the detected object is correct or not (for example, whether it is heavy machinery or not). In this case, for example, the label L is expressed as a binary value of "1" or "0", with the correct label L1 being "1" and the incorrect label L0 being "0". The correct label L1 is an example of a "judgment result that is suitable for object detection" according to the technology of the present disclosure. The incorrect label L0 is an example of a "judgment result that is not suitable for object detection" according to the technology of the present disclosure.

[0071] In this way, since the object that the user has set as the monitoring target using the manual PTZ is likely to be the correct object, the first captured image P1 with the correct label L1 attached is output as the training image TD. In this way, by performing additional learning on the trained model LM using the training image TD, the accuracy of object detection is improved. It also becomes possible to detect new types of objects. For example, if a bulldozer, a type of heavy machinery, could not be detected in the automatic monitoring mode, additional learning of a training image TD including the bulldozer that the user has set as the monitoring target in the manual monitoring mode makes it possible to newly detect the bulldozer as heavy machinery.

[0072] Next, the operation of the monitoring system 10 will be described with reference to FIG.

[0073] Fig. 10 shows a flowchart illustrating an example of the flow of monitoring processing executed by the CPU 42. The flow of monitoring processing shown in Fig. 10 is an example of a "control method" according to the technology of the present disclosure. For convenience of explanation, the description will be given on the assumption that imaging by the imaging device 18 is performed at a default frame rate.

[0074] 10, first, in step S10, the mode switching control unit 51 causes the camera control unit 50 to start operation in the automatic monitoring mode. When the automatic monitoring mode starts, the monitoring camera 12 performs an imaging operation targeting the imaging range 31 (see FIG. 1) set within the monitoring area 30. After step S10, the monitoring process proceeds to step S11.

[0075] In step S11, the image acquisition unit 52 acquires the captured image P output from the surveillance camera 12 and supplies it as a second captured image P2 to the object detection unit 54. At this time, the captured image P is displayed on the display 15 via the display control unit 53. After step S11, the surveillance process proceeds to step S12.

[0076] In step S12, the object detection unit 54 performs an object detection process using the trained model LM to detect a specific object (e.g., heavy machinery) that appears in the second captured image P2 (see FIG. 5). After step S12, the monitoring process proceeds to step S13.

[0077] In step S13, the camera control unit 50 determines whether or not an object has been detected by the object detection unit 54. If an object has not been detected in step S13, the determination is negative, and the monitoring process proceeds to step S14. If an object has been detected in step S13, the determination is positive, and the monitoring process proceeds to step S15.

[0078] In step S14, the camera control unit 50 pans or tilts the surveillance camera 12, thereby changing the imaging range 31 in the pan direction or tilt direction (see FIG. 1). After step S14, the monitoring process returns to step S11. In step S11, the image acquisition unit 52 again performs the captured image acquisition process.

[0079] In step S15, the camera control unit 50 performs automatic PTZ (see FIGS. 6 and 7) to change the imaging range 31 in accordance with the detection result of the object detected by the object detection unit 54. After step S15, the monitoring process proceeds to step S16.

[0080] In step S16, the mode switching control unit 51 determines whether the monitoring mode has been switched from the automatic monitoring mode to the manual monitoring mode by the user operating the receiving device 14 to perform manual PTZ. If the monitoring mode has not been switched to the manual monitoring mode in step S16, the determination is negative, and the monitoring process returns to step S15. If the monitoring mode has been switched to the manual monitoring mode in step S16 (see FIG. 8), the determination is positive, and the monitoring process proceeds to step S17. For example, if the user operates the receiving device 14 to perform manual PTZ in the automatic monitoring mode, the determination is positive.

[0081] In step S17, the camera control unit 50 performs manual PTZ (see FIG. 4) to change the imaging range 31 in accordance with an instruction given by the user to the accepting device 14. After step S17, the monitoring process proceeds to step S18.

[0082] In step S18, the teacher image output unit 55 outputs the first captured image P1 acquired in the manual monitoring mode as a teacher image TD (see FIG. 8). After step S18, the monitoring process proceeds to step S19.

[0083] In step S19, the mode switching control unit 51 determines whether the monitoring mode has been switched from the manual monitoring mode to the automatic monitoring mode by the user operating the receiving device 14. If the monitoring mode has not been switched to the automatic monitoring mode in step S19, the determination is negative, and the monitoring process proceeds to step S20. If the monitoring mode has been switched to the automatic monitoring mode in step S19, the monitoring process returns to step S10.

[0084] In step S20, the mode switching control unit 51 determines whether or not a condition for terminating the monitoring process (hereinafter referred to as the "termination condition") has been satisfied. One example of the termination condition is that an instruction to terminate the monitoring process has been accepted by the accepting device 14. In step S20, if the termination condition has not been satisfied, the determination is negative, and the monitoring process returns to step S17. In step S20, if the termination condition has been satisfied, the determination is positive, and the monitoring process is terminated.

[0085] As described above, the management device 16 as a control device can switch between a manual monitoring mode in which the monitoring camera 12 captures an image to acquire a first captured image P1 and changes the image capture range 31 in response to given instructions, and an automatic monitoring mode in which the monitoring camera 12 captures an image to acquire a second captured image P2 and uses a machine learning trained model LM to detect objects captured in the second captured image P2 and changes the image capture range 31 in response to the detection result. The management device 16 then outputs the first captured image P1 captured in the manual monitoring mode as a training image TD for machine learning. In this way, the technology disclosed herein allows efficient collection of training images TD for use in machine learning without the user having to perform any special operations.

[0086] Furthermore, the management device 16 outputs the first captured image P1 as the teacher image TD in response to a manual operation performed on the surveillance camera 12. This manual operation is a switching operation from automatic monitoring mode to manual monitoring mode, and the management device 16 outputs the first captured image acquired in manual monitoring mode after switching from automatic monitoring mode to manual monitoring mode as the teacher image TD. Furthermore, the surveillance camera 12 can change the imaging range 31 by changing at least one of pan, tilt, and zoom, and the switching operation is an operation to change at least one of pan, tilt, and zoom in automatic monitoring mode. In this way, according to the technology disclosed herein, it is possible to efficiently collect teacher images TD in accordance with the user's intentions.

[0087] [Second embodiment] In the first embodiment, an example was shown in which the first captured image was output as a teacher image TD in response to an operation of switching the monitoring mode from automatic monitoring mode to manual monitoring mode, but in the second embodiment, the first captured image is output as a teacher image TD in response to an output instruction given by the user.

[0088] Fig. 11 shows the operation of the monitoring system 10 according to the second embodiment. As shown in Fig. 11, in this embodiment, step S30 is added between step S17 and step S18. The other steps are the same as those in the first embodiment.

[0089] In this embodiment, after manual PTZ is started in step S17, the monitoring process proceeds to step S30.

[0090] In step S30, it is determined whether or not the user has issued an output instruction by operating the receiving device 14. For example, the user issues an output instruction by operating a mouse serving as the receiving device 14 and clicking a dedicated button displayed on the display 15. If an output instruction has been issued in step S30, the determination is affirmative, and the monitoring process proceeds to step S18. If an output instruction has not been issued in step S30, the determination is negative, and the monitoring process proceeds to step S19.

[0091] In step S18, similarly to the first embodiment, the teacher image output unit 55 performs a teacher image output process to output the first captured image P1 acquired in the manual monitoring mode as a teacher image TD.

[0092] Thus, in this embodiment, after switching from automatic monitoring mode to manual monitoring mode, the management device 16 outputs the first captured image P1 as a teacher image TD in accordance with the given output instruction, so that the teacher image TD can be collected efficiently according to the user's intention.

[0093] [Third embodiment] In the first embodiment, an example was shown in which the first captured image was output as a teacher image TD in response to an operation of switching the monitoring mode from automatic monitoring mode to manual monitoring mode, but in the third embodiment, in addition to the first captured image, the second captured image P2 acquired in the automatic monitoring mode before switching is output as a teacher image TD.

[0094] Fig. 12 shows the operation of the monitoring system 10 according to the third embodiment. As shown in Fig. 12, in this embodiment, step S40 is added between step S16 and step S17. The other steps are the same as those in the first embodiment.

[0095] In this embodiment, if the mode is switched to the manual monitoring mode in step S16, the determination is affirmative and the monitoring process proceeds to step S40.

[0096] In step S40, the teacher image output unit 55 outputs the second captured image P2 (see FIG. 8) acquired in the automatic monitoring mode before switching as a teacher image TD. When the user switches from the automatic monitoring mode to the manual monitoring mode, it is considered that the object detection by the object detection unit 54 in the second captured image P2 acquired in the automatic monitoring mode before switching was erroneous, as shown in FIG. 8, and therefore the teacher image output unit 55 outputs the second captured image P2 as a teacher image TD to which an incorrect label L0 has been assigned. After step S40, the monitoring process proceeds to step S17.

[0097] In this embodiment, in step S18, the teacher image output unit 55 outputs the first captured image P1 acquired in the manual monitoring mode as a teacher image TD to which the correct label L1 has been assigned.

[0098] As described above, in this embodiment, the management device 16 outputs the second captured image P2 acquired in the automatic monitoring mode before switching as a teacher image TD to which an incorrect label L0 has been assigned, and outputs the first captured image P1 acquired in the manual monitoring mode after switching as a teacher image TD to which a correct label L1 has been assigned. As a result, in this embodiment, the correct label L1 or the incorrect label L0 can be automatically assigned to the teacher image TD, reducing the user's effort. Furthermore, by performing additional learning on the trained model LM using the second captured image P2 in addition to the first captured image P1, the detection accuracy of object detection is improved.

[0099] [Fourth embodiment] Next, a fourth embodiment will be described. The fourth embodiment is a modification of the third embodiment. In the third embodiment, when the monitoring mode is switched from the automatic monitoring mode to the manual monitoring mode, the second captured image P2 acquired in the automatic monitoring mode before the switch is output as the teacher image TD. In the fourth embodiment, when a certain condition is satisfied after the monitoring mode is switched from the automatic monitoring mode to the manual monitoring mode, the second captured image P2 acquired in the automatic monitoring mode before the switch is output as the teacher image TD.

[0100] 13 shows the operation of the monitoring system 10 according to the fourth embodiment. In this embodiment, in step S19, the mode switching control unit 51 determines whether or not an operation has been performed for a certain period of time in the manual monitoring mode after switching from the automatic monitoring mode to the manual monitoring mode (i.e., whether or not a state of no operation has continued for a certain period of time). If no operation has been performed for the certain period of time in step S19, the determination is affirmative, and the monitoring process returns to step S10. If an operation has been performed before the certain period of time has elapsed in step S19, the determination is negative, and the monitoring process proceeds to step S20. That is, in this embodiment, if no operation has been performed for a certain period of time in the manual monitoring mode after switching from the automatic monitoring mode to the manual monitoring mode, the mode transitions to the automatic monitoring mode.

[0101] In this embodiment, step S50 is added between step S16 and step S40. The other steps are the same as those in the third embodiment.

[0102] In this embodiment, if the mode is switched to the manual monitoring mode in step S16, the determination is affirmative and the monitoring process proceeds to step S50.

[0103] In step S50, the mode switching control unit 51 determines whether the current switching operation is performed after a certain time has elapsed since the previous switching operation. Specifically, the mode switching control unit 51 starts timing from the point in time when the monitoring mode is switched to the manual monitoring mode as a result of the affirmative determination in step S16, and determines whether the time that has elapsed since the monitoring mode is switched to the automatic monitoring mode as a result of the affirmative determination in step S19 until the determination is again affirmative in step S16 is within the certain time.

[0104] In step S50, if the current switching operation is not performed after a certain time has elapsed since the previous switching operation, the determination is negative and the monitoring process proceeds to step S40. In step S50, if the current switching operation is performed after a certain time has elapsed since the previous switching operation, the determination is positive and the monitoring process proceeds to step S17.

[0105] Thus, in this embodiment, if a certain time has elapsed since the previous switching operation, the teacher image output unit 55 does not output the second captured image P2 acquired in the automatic monitoring mode before switching as the teacher image TD. This corresponds to, for example, a situation in which the user switches the monitoring mode to the manual monitoring mode, then leaves the location of the management device 16 and continues to perform no operations, causing the mode to switch back to the automatic monitoring mode, and then returns to the location of the management device 16 and switches back to the manual monitoring mode. In such a situation, it is highly likely that the user did not observe the second captured image P2 acquired in the automatic monitoring mode immediately before switching to the manual monitoring mode, and it is unlikely that the user switched the monitoring mode to the manual monitoring mode because they determined that the object detection was a false detection. In other words, it is likely that the user performed the switching operation simply to return to the manual monitoring mode after switching from the manual monitoring mode to the automatic monitoring mode due to a continued state of no operations.

[0106] In this way, in this embodiment, it is possible to prevent the second captured image P2 from being output as the teacher image TD under circumstances not intended by the user.

[0107] Next, FIGS. 14 to 17 show various modified examples of the teacher image output process by the teacher image output unit 55. FIG.

[0108] [First Modification] Fig. 14 shows a first modified example of the teacher image output process. As shown in Fig. 14, in the first modified example, the teacher image output unit 55 detects an object that appears in the teacher image TD and adds position information of the detected object in the teacher image TD to the teacher image TD.

[0109] For example, the teacher image output unit 55 uses the trained model LM to detect an object from the first captured image P1 to be output as the teacher image TD, and adds position information of the detected object to the first captured image P1. Then, the teacher image output unit 55 outputs the first captured image P1 with the added position information as the teacher image TD.

[0110] When the teacher image output unit 55 outputs the second captured image P2, the teacher image output unit 55 may perform a similar position information addition process on the second captured image P2.

[0111] Furthermore, it is preferable that the detection standard for object detection when the teacher image output unit 55 detects an object using the trained model LM is lower than the detection standard for object detection when the object detection unit 54 detects an object using the trained model LM. For example, the detection standard is a lower limit value of the score for determining that an object candidate is a specific object. For example, when the object detection unit 54 detects an object using the trained model LM, it determines that the object is heavy machinery if the score is 0.9 or higher, and when the teacher image output unit 55 detects an object using the trained model LM, it determines that the object is heavy machinery if the score is 0.7 or higher.

[0112] In this way, by lowering the detection criteria for object detection when detecting an object reflected in the teacher image TD below the detection criteria for object detection when detecting an object reflected in the second captured image P2 in automatic monitoring mode, the detection accuracy of the trained model LM is improved, making it possible to detect objects that could not be detected before.

[0113] [Second Modification] Fig. 15 shows a second modified example of the teacher image output process. As shown in Fig. 15, in addition to the position information addition process shown in the first modified example, the second modified example allows the user to change the position information.

[0114] In this embodiment, the teacher image output unit 55 uses the trained model LM to detect an object from the first captured image P1 to be output as a teacher image TD, and displays the position information of the detected object together with the first captured image P1 on the display 15 via the display control unit 53. The user can change and determine the position information displayed on the display 15. For example, the user can change and determine the position, shape, and size of a rectangular frame representing the position information of the object using the receiving device 14. In the example shown in FIG. 15, when the trained model LM detects a person who is not a heavy machine as the object, the user changes the position information so that the position information represents the area of ​​the heavy machine H2.

[0115] The teacher image output unit 55 changes the position information in accordance with an instruction given to the reception device 14, and outputs the first captured image P1 to which the changed position information has been added as a teacher image TD.

[0116] When the teacher image output unit 55 outputs the second captured image P2, the teacher image output unit 55 may perform a similar position information change process on the second captured image P2.

[0117] In this way, according to this modified example, the user can change the location information to an appropriate location, thereby improving the accuracy of additional learning of the trained model LM.

[0118] [Third Modification] In the third variant, the teacher image output unit 55 does not perform object detection using the learned model LM, but determines the position of an object reflected in the teacher image TD in accordance with instructions given by the user, and adds position information of the determined object in the teacher image TD to the teacher image TD.

[0119] 16 as an example, the teacher image output unit 55 displays a first captured image P1 to be output as a teacher image TD on the display 15 via the display control unit 53. In response to instructions given to the receiving device 14, the teacher image output unit 55 determines the position of a heavy machine H2 that appears in the first captured image P1 to be output as the teacher image TD, and outputs the first captured image P1 to which the position information of the heavy machine H2 has been added as the teacher image TD. For example, the user can determine the position of an object by changing the position, shape, and size of a rectangular frame that represents the position information of the object using the receiving device 14.

[0120] Similarly, when the second captured image P2 is to be output, the teacher image output section 55 can add position information in accordance with an instruction given by the user.

[0121] According to this modification, the user can determine the position of an object that appears in the teacher image TD, thereby improving the accuracy of machine learning of the trained model LM.

[0122] [Fourth Modification] In the fourth modified example, the teacher image output unit 55 performs an extension process on the teacher image TD to pad it out, in order to further improve the accuracy of the machine learning of the trained model LM. As an example, as shown in Fig. 17, the teacher image output unit 55 outputs, as the teacher image TD, an extended image P1E obtained by inverting the first captured image P1, in addition to the first captured image P1 to be output as the teacher image TD. This increases the number of teacher images TD, thereby improving the accuracy of the machine learning of the trained model LM.

[0123] The extension process for generating the extended image P1E is not limited to inversion, but may be at least one of inversion, reduction, noise addition, and style change using deep learning.

[0124] Similarly, when the second captured image P2 is to be output, the teacher image output unit 55 can increase the number of teacher images TD by performing the extension process.

[0125] The various processes described in the first to fourth modified examples may be performed after the teacher image TD output from the teacher image output unit 55 is stored in a storage device such as the NVM 44.

[0126] In the above-described embodiments and modifications, heavy machinery is detected as an object from a captured image, but the detected object is not limited to heavy machinery. For example, as shown in FIG. 18, a barricade B set up around a heavy machinery H1 to ensure safety may be detected. Alternatively, after detecting the heavy machinery H1, it may be detected in stages to determine whether or not a barricade B has been set up around the heavy machinery H1. The above-described various techniques can also be applied to barricade detection, just as in the case of heavy machinery.

[0127] The technology of the present disclosure is particularly useful in cases where training images cannot be easily obtained, such as heavy machinery at a construction site, barricades, and the like.

[0128] Furthermore, in each of the above embodiments, the program PG for monitoring processing is stored in the NVM 44 (see FIG. 2), but the technology of the present disclosure is not limited to this, and as an example, the program PG may be stored in any portable storage medium 100 that is a non-transitory storage medium such as an SSD or a USB memory, as shown in FIG. 19. In this case, the program PG stored in the storage medium 100 is installed in the computer 40, and the CPU 42 executes the above-described monitoring processing in accordance with the program PG.

[0129] Alternatively, the program PG may be stored in a storage device such as another computer or server device connected to the computer 40 via a communication network (not shown), and the program PG may be downloaded and installed in the computer 40 in response to a request from the management device 16. In this case, the monitoring process is executed by the computer 40 in accordance with the installed program PG.

[0130] The hardware resources that execute the above-described monitoring process can be various processors, as listed below. Examples of processors include a CPU, which is a general-purpose processor that functions as a hardware resource that executes the monitoring process by executing software, i.e., a program PG, as described above. Examples of processors include dedicated electrical circuits, such as FPGAs, PLDs, or ASICs, which are processors with circuit configurations specifically designed to execute specific processes. Each processor has built-in or connected memory, and executes the monitoring process by using the memory.

[0131] The hardware resource that executes the monitoring process may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the monitoring process may be a single processor.

[0132] Examples of systems configured with a single processor include, first, a system in which one processor is configured with a combination of one or more CPUs and software, as typified by client and server computers, and this processor functions as a hardware resource that executes monitoring processing. Second, a system in which a processor is used to implement the functions of the entire system, including multiple hardware resources that execute monitoring processing, on a single IC chip, as typified by SoCs. In this way, monitoring processing is implemented using one or more of the various processors described above as hardware resources.

[0133] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements.

[0134] Furthermore, the above-described monitoring process is merely an example, and it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[0135] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0136] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0137] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. A control device including a processor for controlling a surveillance camera, The processor: Switching between a first monitoring mode in which a monitoring camera captures an image to obtain a first captured image and changes the image capture range in response to a given instruction, and a second monitoring mode in which the monitoring camera captures an image to obtain a second captured image and detects an object appearing in the second captured image using a trained model through machine learning and changes the image capture range in response to the detection result, outputting the first captured image acquired in the first monitoring mode as a training image for the machine learning; after switching from the first monitoring mode to the second monitoring mode, if the manual operation is performed after a certain time has elapsed since the previous execution of the manual operation of switching from the second monitoring mode to the first monitoring mode, the second captured image is not output as the teacher image. Control device.

2. the processor outputs the first captured image as the teacher image in response to a manual operation performed on the surveillance camera. The control device according to claim 1 .

3. The processor outputs the first captured image acquired in the first monitoring mode after switching from the second monitoring mode to the first monitoring mode as the teacher image. The control device according to claim 2 .

4. the surveillance camera is capable of changing the imaging range by changing at least one of pan, tilt, and zoom; the switching operation is an operation of changing at least one of pan, tilt, and zoom in the second monitoring mode. The control device according to claim 3 .

5. the processor outputs the first captured image as the teacher image in response to a given output instruction after switching from the second monitoring mode to the first monitoring mode. The control device according to claim 3 or 4.

6. The processor: The second captured image acquired in the second monitoring mode before the switching is assigned a determination result that the second captured image is unsuitable for detecting the object, and is output as the teaching image; and the first captured image acquired in the first monitoring mode after the switching is assigned a determination result that matches the detection of the object, and is output as the teaching image; The control device according to any one of claims 3 to 5.

7. the processor switches to the second monitoring mode when no operation is performed for a certain period of time in the first monitoring mode after switching from the second monitoring mode to the first monitoring mode; The control device according to claim 6.

8. The processor: detecting an object captured in the teacher image, and adding position information of the detected object in the teacher image to the teacher image; The control device according to any one of claims 1 to 7.

9. The processor: a detection standard for object detection when detecting an object captured in the teacher image is lowered to be lower than a detection standard for object detection when detecting an object captured in the second captured image; The control device according to claim 8.

10. The processor: performing a location information change process to change the location information in response to a given instruction; The control device according to claim 8 or 9.

11. The processor: determining a position of an object appearing in the teacher image in accordance with a given instruction, and adding position information of the determined object in the teacher image to the teacher image; The control device according to any one of claims 1 to 7.

12. The processor: In addition to the teacher image, an extended image generated by performing an extension process on the teacher image is output as the teacher image. The control device according to any one of claims 1 to 11.

13. The extension process is at least one of inversion, reduction, noise addition, and style change using deep learning. The control device according to claim 12.

14. a first monitoring mode in which a monitoring camera captures an image to obtain a first captured image and changes the image capture range in response to a given instruction, and a second monitoring mode in which the monitoring camera captures an image to obtain a second captured image and detects an object appearing in the second captured image using a trained model through machine learning and changes the image capture range in response to the detection result; outputting the first captured image acquired in the first monitoring mode as a training image for the machine learning; after switching from the first monitoring mode to the second monitoring mode, if the manual operation for switching from the second monitoring mode to the first monitoring mode is performed after a certain time has elapsed since the previous execution of the manual operation, the second captured image is not output as the teacher image; A control method comprising:

15. a first monitoring mode in which a monitoring camera captures an image to obtain a first captured image and changes the image capture range in response to a given instruction, and a second monitoring mode in which the monitoring camera captures an image to obtain a second captured image and detects an object appearing in the second captured image using a trained model through machine learning and changes the image capture range in response to the detection result; outputting the first captured image acquired in the first monitoring mode as a training image for the machine learning; after switching from the first monitoring mode to the second monitoring mode, if the manual operation for switching from the second monitoring mode to the first monitoring mode is performed after a certain time has elapsed since the previous execution of the manual operation, the second captured image is not output as the teacher image; A program for causing a computer to execute a process including the above.

16. A control device having a processor for controlling a surveillance camera, The processor: Switching between a first monitoring mode in which a monitoring camera captures an image to obtain a first captured image and changes the image capture range in response to a given instruction, and a second monitoring mode in which the monitoring camera captures an image to obtain a second captured image and detects an object appearing in the second captured image using a trained model through machine learning and changes the image capture range in response to the detection result, In response to a manual operation of switching from the second monitoring mode to the first monitoring mode, the first captured image acquired after the switching is output as a training image for the machine learning; When switching, a judgment result indicating that the second captured image acquired before the switching is unsuitable for detecting the object is assigned to the second captured image acquired before the switching, and a judgment result indicating that the first captured image is suitable for detecting the object is assigned to the first captured image acquired after the switching, and the two images are output as the teacher images, after switching from the second monitoring mode to the first monitoring mode, if there is no operation for a certain period of time in the first monitoring mode, switching to the second monitoring mode; after switching from the first monitoring mode to the second monitoring mode, if the manual operation is performed after a certain time has elapsed since the previous execution of the manual operation, the second captured image is not output as the teacher image. Control device.

Images