Control device, control system, control method, and program

Abstract

To provide a control device that can reduce user discomfort caused by occurrence of communication delay.SOLUTION: A control device (300) for remotely operating an imaging device (100) includes a communication unit (304) that communicates with the imaging device (100), a notification unit (305) that notifies a user, and a control unit (301) that controls notification by the notification unit (305) on the basis of whether an amount of communication delay in the communication and a zoom state of the imaging device meet predetermined conditions.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a control device, a control system, a control method, and a program. [Background technology]

[0002] Conventionally, there is known a control device that allows a user to remotely control a camera to perform shooting and image production even if the user is not actually at the shooting site. When controlling a camera remotely, a communication delay may occur in communication between the camera installed at the shooting site and a control device such as a PC or controller installed at the user's base that controls the camera. When a large communication delay occurs, the difference between the position of a subject in an image displayed to the user and the actual position of the subject becomes large, and the user's operation may become an unintended operation, which makes the user feel uncomfortable.

[0003] Patent document 1 relates to a communication device that can be remotely controlled from an external device, and discloses a communication device that sets the communication interval to be short in remote control mode and displays a UI (User Interface) warning until the communication interval is shortened. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-216639 A Summary of the Invention [Problem to be solved by the invention]

[0005] In the communication device disclosed in Patent Document 1, a UI warning is displayed based only on the communication interval, so that the discomfort felt by the user due to the occurrence of communication delays may not be resolved.

[0006] Therefore, an object of the present invention is to provide a control device that can reduce the discomfort felt by a user due to the occurrence of communication delays. [Means for solving the problem]

[0007] A control device as one aspect of the present invention is a control device that remotely operates an imaging device, and has a communication unit that communicates with the imaging device, a notification unit that notifies a user, and a control unit that determines whether or not a communication delay amount in the communication and a zoom state of the imaging device satisfy predetermined conditions, and controls the notification by the notification unit based on the result of the determination.

[0008] Other objects and features of the present invention will be described in the following embodiments. Effect of the Invention

[0009] According to the present invention, it is possible to provide a control device capable of reducing the discomfort felt by a user due to the occurrence of communication delays. [Brief description of the drawings]

[0010] [Figure 1] FIG. 2 is a schematic diagram of a system in each embodiment. [Diagram 2] FIG. 2 is a block diagram of a system in each embodiment. [Diagram 3] 4 is a flowchart showing a basic operation of a controller in each embodiment. [Figure 4] 4 is a flowchart showing a basic operation of the camera in each embodiment. [Diagram 5] FIG. 1 is an explanatory diagram of problems in each embodiment. [Figure 6] 5 is a flowchart showing the operation of a controller in the first embodiment. [Figure 7] 4 is a flowchart showing the operation of the information processing device in the first embodiment. [Figure 8] FIG. 4 is an explanatory diagram of a table for determining tracking possibility in the first embodiment. [Figure 9] 4A to 4C are explanatory diagrams of the photographing angle of view and the subject size for each zoom state in the first embodiment. [Figure 10] 13 is a composite image of a captured image and a message according to a tracking possibility in the first embodiment. [Figure 11] 5 is a flowchart showing calculation of a zoom state in the first embodiment. [Figure 12] FIG. 4 is an explanatory diagram of a correction process of tracking possibility in the first embodiment. [Figure 13] 13 is a flowchart showing generation of an assist menu in the second embodiment. [Figure 14] FIG. 11 is an explanatory diagram of an assist menu in the second embodiment. [Figure 15] 10 is a flowchart showing the operation of an information processing device in the second embodiment. [Figure 16] 13 is a flowchart showing the operation of a controller in the third embodiment. [Figure 17] FIG. 13 is an explanatory diagram of a table for determining tracking possibility in the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0012] (First embodiment) First, an overview of a control system (camera control system) 10 in a first embodiment of the present invention will be described with reference to Fig. 1. Fig. 1 is a schematic diagram of the control system 10. The control system 10 includes a camera (imaging device) 100, an information processing device 200, and a controller (control device) 300.

[0013] The camera 100 and the information processing device 200 are connected to a network formed on a LAN (Local Area Network) 400, and the controller 300 is connected to another LAN 500. The LAN 400 and the LAN 500 are connected via the Internet 600, forming a network in which the devices can communicate with each other using a communication protocol. The communication medium may be wired or wireless. The camera 100 and the information processing device 200 are installed in locations physically close to each other, and the controller 300 is installed in a remote location away from them. For this reason, a large communication delay may occur when communication is performed via the Internet 600, compared to communication within the LAN 400 or the LAN 500.

[0014] The camera 100 captures an image of a predetermined range of an area and outputs the captured image to the information processing device 200 or the controller 300 via a network. Images may be transferred from the camera 100 to the information processing device 200 via an image transmission cable such as SDI or High-Definition Multimedia Interface (HDMI (registered trademark)). The camera 100 includes a driving unit 109, which will be described later, and is capable of pan / tilt operations for changing the imaging direction.

[0015] The information processing device 200 has a function of receiving an image captured by the camera 100 and inferring the position of a subject in the received image using a learning model. The information processing device 200 can then transmit an instruction to control the camera 100 based on the inference result.

[0016] Controller 300 can remotely acquire images output by camera 100 and control and set the camera based on user operations by accessing camera 100 via Internet 600. Note that an image in this embodiment refers to either a still image or each frame of a video, and this embodiment is applicable to either.

[0017] Next, the hardware configuration of the control system 10 will be described with reference to Fig. 2. Fig. 2 is a block diagram of the control system 10. The camera 100 has a CPU (control unit) 101, a ROM 102, a RAM 103, an image input I / F (Interface) 104, a network I / F 105, an image processing unit 106, an image sensor (imaging element) 107, a drive I / F 108, and a drive unit 109. The camera 100 also has an internal bus 110 that connects the above-mentioned components to each other so that they can communicate with each other.

[0018] The CPU 101 controls each component of the camera 100, thereby controlling the entire device. The ROM 102 is a non-volatile storage device such as a flash memory, HDD, SSD, or SD card, and is used as a permanent storage area for the OS, various programs, and various data, as well as a short-term storage area for various data. The RAM 103 is a high-speed storage device such as a DRAM, into which the OS, various programs, and various data are loaded, and is also used as a working area for the OS and various programs. The image output I / F 104 is an interface for outputting images captured by an image sensor 107 (described later) to the outside, and is configured by a Serial Digital Interface (SDI) or HDMI (registered trademark).

[0019] The network I / F 105 is an I / F for connecting to the above-mentioned LAN 400, and is responsible for communication with external devices such as the information processing device 200 and the controller 300 via a communication medium such as Ethernet (registered trademark). Although it has been described that remote camera control of the camera 100 is performed via the network I / F 105, it may be performed via another I / F such as a serial communication I / F (not shown). An image sensor 107 such as a CCD sensor or a CMOS sensor is connected to the image processing unit 106, which converts image data acquired from the image sensor 107 into a predetermined format, compresses it as necessary, and transfers it to the RAM 103.

[0020] Drive unit 109 is a mechanical mechanism and optical system for changing the imaging direction of camera 100, and is composed of a mechanical drive system, a drive motor, lenses, etc. Based on instructions received from CPU 101 via drive I / F 108, drive unit 109 performs rotation such as pan / tilt operations for directing the imaging angle of view in the horizontal or vertical direction, and zoom operations for optically changing the imaging angle of view.

[0021] The information processing device 200 includes a CPU 201, a ROM 202, a RAM 203, a network I / F 204, an image output I / F 205, a user input I / F 206, an inference unit 207, an image input I / F 208, and an internal bus 209 that interconnects the various components. The CPU 201 controls the various components of the information processing device 200, thereby controlling the entire device. The ROM 202 is a non-volatile storage device such as a flash memory, HDD, SSD, SD card, etc., and is used as a permanent storage area for the OS, various programs, and various data, as well as a short-term storage area for various data. The RAM 203 is a high-speed storage device such as a DRAM, into which the OS, various programs, and various data are loaded, and is also used as a work area for the OS and various programs.

[0022] The network I / F 204 is an I / F for connecting to the above-mentioned LAN 400, and is responsible for communication with the camera 100 via a communication medium such as Ethernet. Examples of communication include transmission of a control command to the camera 100 and reception of a camera image from the camera 100. The image output I / F 205 is an interface for outputting an image to the outside for operation of the information processing device 200, and is configured with SDI or HDMI (registered trademark). For example, a display (not shown) having a liquid crystal panel or an organic EL panel can be connected. The user input I / F 206 receives an instruction from a user and transmits an instruction signal to the CPU 201. As a specific example, it is an interface for connecting to an input device such as a mouse, a keyboard, or a touch panel, and is configured with a USB (Universal Serial Bus) or the like. The image output I / F 205 and the user input I / F 206 configure a user interface between the user and the information processing device 200.

[0023] The inference unit 207 estimates the position and presence or absence of a predetermined object from an image received from an image input I / F 208 (described later) and is configured with a calculation device specialized for image processing and inference processing, such as a so-called GPU (Graphics Processing Unit). Although a GPU is generally effective for use in learning processing, a reconfigurable logic circuit such as an FPGA (Field-Programmable Gate Array) may realize the same function. The processing of the inference unit 207 may be performed by the CPU 201. In this embodiment, an area representing a person's face in the received image is detected, and the coordinates of the upper left vertex and the lower right vertex of the circumscribing rectangle are output. The image input I / F 208 is an interface for receiving an image from the camera 100 and is configured with SDI or HDMI (registered trademark).

[0024] The controller 300 includes a CPU (control unit) 301, a ROM 302, a RAM 303, a network I / F (communication unit) 304, a display unit (notification unit) 305, a user input I / F 306, and an internal bus 307 that interconnects the various components. The CPU 301 controls the various components of the controller 300, thereby controlling the entire device. The ROM 302 is a non-volatile storage device such as a flash memory, HDD, SSD, SD card, etc., and is used as a permanent storage area for the OS, various programs, and various data, as well as a short-term storage area for various data. The RAM 303 is a high-speed storage device such as a DRAM, into which the OS, various programs, and various data are loaded, and is also used as a work area for the OS and various programs.

[0025] The network I / F 304 is an I / F for connecting to the LAN 500, and is responsible for communication with the camera 100 and external devices via a communication medium such as Ethernet. Examples of communication include sending a control command to the camera 100 and receiving a camera image from the camera 100. The display unit 305 is, for example, a liquid crystal panel or an organic EL panel, and displays images acquired from the camera 100 and a setting screen of the controller 300. The user input I / F 306 is an interface for receiving operations from the user on the controller 300. The user input I / F 306 is, for example, a button, a dial, a joystick, or a touch panel.

[0026] Next, an operation of controlling the camera 100 by the controller 300, which is a basic operation in the control system 10, will be described. First, with reference to Fig. 3, a basic operation in which the controller 300 transmits a control command to the camera 100 based on a user operation will be described. Fig. 3 is a flowchart showing the basic operation of the controller 300. Note that the camera 100 is controlled based on the control command transmitted from the controller 300, and the operation of the camera 100 will be described later. This control flow starts when the CPU 301 detects that a user operation has been performed on the user input I / F 306.

[0027] First, in step S101, the CPU 301 of the controller 300 detects a joystick operation by a user via the user input I / F 306. Next, in step S102, the CPU 301 acquires the operation direction and operation amount of the joystick from the user input I / F 306. A specific example of the joystick is an analog output specification that uses a voltage output from a variable resistor provided in each of the pan direction and tilt direction. The CPU 301 can grasp the angular velocity in each of the pan direction and tilt direction by reading a digital value obtained by passing the voltage input from the joystick through an A / D conversion unit (not shown). In this embodiment, the A / D conversion unit can read a value corresponding to the angular velocity as each component in the pan direction and tilt direction, with a value in a predetermined range, for example, 0 to 1023, depending on the operation amount.

[0028] Next, in step S103, CPU 301 converts the joystick operation direction and operation amount into a control command (control value) in accordance with a protocol predefined as a control method for camera 100, using the joystick operation direction and operation amount as an angular velocity in the pan direction and an angular velocity in the tilt direction. In this way, CPU 301 acquires the pan / tilt drive speed and drive direction based on user operation. Alternatively, the drive amount may be acquired based on user operation. CPU 301 then writes out the control command to RAM 303. Next, in step S104, CPU 301 reads out the control command written out to RAM 303 in step S103, and transmits it to camera 100 via network I / F 304.

[0029] Next, a basic operation of camera 100 when a control command is received from controller 300 will be described with reference to Fig. 4. Fig. 4 is a flowchart showing the basic operation of camera 100. This flow starts when CPU 101 detects that a control command has arrived at network I / F 105.

[0030] First, in step S201, CPU 101 reads out a control command received via network I / F 105, and writes it to RAM 103. Next, in step S202, CPU 101 reads out values ​​of the operation direction and operation amount for each of the pan direction and tilt direction from the control command written out to RAM 103 in step S201.

[0031] Next, in step S203, CPU 101 derives drive parameters for panning and tilting in a desired direction at a desired speed based on the values ​​read out in step S202. Specifically, these are parameters for controlling motors (not shown) for the pan direction and tilt direction included in drive unit 109, and may be converted into drive parameters by referring to a conversion table previously stored in RAM 103 based on the operation amount included in the received control command.

[0032] Next, in step S204, CPU 101 controls drive unit 109 via drive I / F 108 based on the derived drive parameters. Drive unit 109 rotates based on the drive parameters derived by CPU 101. This allows camera 100 to change the imaging direction, that is, perform pan / tilt operations. In this way, by controlling controller 300 and camera 100, it is possible to control the rotation of camera 100 in response to user operation.

[0033] The respective operations of the controller 300 and the camera 100 when the camera 100 is controlled by the controller 300, which is the basic operation of the control system 10, have been described above.

[0034] Next, an example of the operation related to the problem of this embodiment will be described with reference to FIG. 5. FIG. 5 is an explanatory diagram of the problem of this embodiment. In FIG. 5, images 501 to 505 are images captured by the camera 100, and images 506 to 510 are images displayed on the controller 300. Also, each image is assumed to be displayed at times T1 to T5. Here, the camera 100 and the controller 300 are connected via the Internet 600, and are in a situation where a large communication delay may occur. Therefore, the image 501 captured by the camera 100 at time T1 reaches the controller 300 at time T2 and is displayed as an image 507. Similarly, the image 502 corresponds to the image 508, the image 503 corresponds to the image 509, and the image 504 corresponds to the image 510.

[0035] Communications 511-514 each indicate the transmission of an image from camera 100 to controller 300. Communications 515-517 each indicate the transmission of a control from controller 300 to camera 100. For example, communication 515 indicates that control is applied to image 502 transmitted from controller 300 at time T1 and received by camera 100 at time T2. The user operates controller 300 to control camera 100 to place the subject in the center of the angle of view.

[0036] The explanation will continue with a focus on the subject. At time T1, as shown in image 501, the subject is moving to the left, and at time T2, as shown in image 502, it can be confirmed by camera 100 that the subject is stationary. However, at time T1, the user is looking at image 506 in which the subject is moving to the left, and so performs an operation to turn the camera to the left, which operation becomes an operation on image 502 at time T2 as communication 515. As a result, at time T3, the subject has shifted to a position to the right as shown in image 503.

[0037] Furthermore, the image the user sees at time T2 is image 507, which arrives after image 501 has been communicated via communication 511, and the user who sees image 507 thinks that the subject is still moving to the left and operates controller 300 to aim camera 100 to the left. Communication 516 results in control of image 503, which moves further to the right at time T4 as shown in image 504. The user can confirm that the subject has stopped at time T3, but depending on the user's reaction speed, some control remains in communication 517. For this reason, at time T5 when all control is completed, the subject moves to a position significantly displaced from the center of the angle of view as shown in image 510.

[0038] Next, a process of presenting a warning message regarding pan / tilt control when controller 300 determines that it is difficult to track a subject by user operation of camera 100 based on the amount of communication delay will be described with reference to Fig. 6. A situation in which it is difficult to track a subject is, for example, a situation in which a communication delay causes a delay in the user's control of camera 100, and the subject is likely to be removed from the image of camera 100, as described with reference to Fig. 5.

[0039] FIG. 6 is a flowchart showing the operation of the controller 300, illustrating a series of steps in which the controller 300 acquires the amount of communication delay and the zoom state of the camera 100 and presents a warning UI to the user.

[0040] First, in step S301, CPU 301 determines whether or not a command indicating the end of this flow has been received (whether or not to continue the process) via network I / F 304 or user input I / F 306. If it is determined that the process is to be continued, the process proceeds to step S302. On the other hand, if it is determined that the process is not to be continued, the process ends.

[0041] In step S302, the CPU 301 measures the amount of communication delay in communication with the camera 100 via the network I / F 304. Methods for measuring the amount of communication delay include, but are not limited to, a method of measuring and averaging the time from transmission of a control request to a response for each of a plurality of communications, or a method of measuring a response to a control request including a specific command. The measured amount of communication delay is stored in the RAM 303 of the controller 300 by the CPU 301 as connection information for the camera 100 in the controller 300. Hereinafter, the explanation will be given assuming that the CPU 301 has acquired the amount of communication delay Lctrl.

[0042] Next, in step S303, the CPU 301 acquires the zoom status (Zstatus) of the camera 100 via the network I / F 304. In this embodiment, the CPU 301 transmits a zoom status acquisition command to the information processing device 200 via the network I / F 304. Then, the CPU 201 of the information processing device 200 calculates Zstatus from the focal length and focus position of the lens of the camera 100, and returns it to the controller 300.

[0043] Next, in step S303, the controller 300 acquires the zoom state of the camera 100. Here, a method for acquiring the zoom state will be described in detail with reference to Fig. 7. Fig. 7 is a flowchart showing the operation of the information processing device 200 that has received a control command via the network I / F 204. This flow starts when the CPU 201 detects that a zoom state acquisition command has arrived at the network I / F 204.

[0044] First, in step S401, the CPU 201 reads out a control command received via the network I / F 204 and writes it to the RAM 203. Next, in step S402, the CPU 201 calculates the shooting angle of view θ. Specifically, the CPU 201 acquires the focal length f [mm] of the lens of the camera 100 and the horizontal size x [mm] of the image sensor 107 via the network I / F 204. Then, the CPU 201 calculates the shooting angle of view θ using the formula 2×atan(x / (2×f)) and writes it to the RAM 203. Next, in step S403, the CPU 201 acquires the distance d at which the lens of the camera 100 is in focus via the network I / F 204 and writes it to the RAM 203.

[0045] Next, in step S404, the CPU 201 calculates the zoom status Zstatus. Specifically, the CPU 201 reads out the photographing angle of view θ and the distance d at which the lens is in focus from the RAM 203. The CPU 201 then predetermines the face size of the person to be tracked to be 0.15 m, and calculates Zstatus by calculating the ratio of the size of the subject to the photographing angle of view, 0.15 / (2×d×tan(θ / 2)). At this time, the inference unit 207 may obtain an image photographed by the camera 100 via the network I / F 204, and detect the subject (such as a face) by inference. If the subject is not detected, Zstatus may be stored as 0. This allows processing to be performed when the subject is not present in the image, which will be described later. As a method for determining whether or not the subject is present, other methods such as a method using other image processing such as pattern matching, or a method of obtaining a focusing result from the camera, may be used.

[0046] Next, in step S405, the CPU 201 transmits the Zstatus written to the RAM 203 to the controller 300 via the network I / F 204. After the controller 300 receives the Zstatus, this flow ends.

[0047] In the present embodiment, an example has been described in which the controller 300 acquires Zstatus from the information processing device 200 in step S303, but other methods may be used as long as they are capable of calculating Zstatus from the zoom information of the camera 100. For example, the camera 100 may execute the flow shown in Fig. 7, and acquire Zstatus from the controller 300 to the camera 100. Alternatively, the controller 300 may acquire the focal length f or the in-focus distance d from the camera 100 via the network I / F 304, and the CPU 301 may calculate Zstatus. In these embodiments, the information processing device 200 is not essential.

[0048] After CPU 301 stores Zstatus in RAM 303, the process proceeds to step S304. In step S304, CPU 301 determines whether or not the subject is present within the angle of view. CPU 301 references the value of Zstatus stored in RAM 303, and if the value is 0, the subject is not present within the angle of view, and the process thereafter ends. On the other hand, if the value of Zstatus is greater than 0, CPU 301 determines that the subject is present within the angle of view, and proceeds to step S305.

[0049] In step S305, CPU 301 performs a process of determining the tracking possibility based on the communication delay amount measured in step S302 and the zoom state of camera 100 acquired in step S303. Here, tracking possibility is the tracking performance in remotely controlling camera 100 by controller 300, and is a parameter indicating whether a subject can be tracked by manually panning / tilting camera 100 with the measured communication delay amount and zoom state.

[0050] Here, the tracking possibility calculated by the CPU 301 will be described in detail with reference to Fig. 8 and Fig. 9(A) to (C). Fig. 8 is an explanatory diagram of a table for determining the tracking possibility. In Fig. 8, the vertical axis indicates the magnitude of the communication delay amount Lctrl, and the horizontal axis indicates the zoom state Zstatus.

[0051] The CPU 301 acquires the communication delay amount Lctrl and the zoom state Zstatus from the RAM 303, and plots them on a table to acquire the traceability Traceability as a parameter corresponding to the plot point. In this embodiment, the values ​​of the traceability Traceability (information on the traceability performance) are classified into three types: "2" is a warning, "1" is a caution, and "0" is no problem, and the maximum value is "2." Note that the types of the values ​​of the traceability Traceability are not limited to three types, and may be two or more types.

[0052] As shown in FIG. 8, the communication delay amount Lctrl in this embodiment is classified into three stages. When a communication delay amount of 200 ms or less occurs, the communication delay amount is set to "small". When a communication delay amount of more than 200 ms and less than or equal to 500 ms occurs, the communication delay amount is set to "medium". When a communication delay amount of more than 500 ms occurs, the communication delay amount is set to "large". However, in this embodiment, the communication delay amount threshold is not limited to these, and other values ​​may be used. Also, in this embodiment, the threshold is classified into three stages, but is not limited to these, and may be two stages or more.

[0053] As shown in FIG. 8, the zoom state Zstatus in this embodiment is classified into three stages. When the zoom state Zstatus is 0.15 or less, the zoom state is set to "small". When the zoom state Zstatus is greater than 0.15 and less than 0.4, the zoom state is set to "medium". When the zoom state Zstatus is greater than 0.4, the zoom state is set to "large". In this embodiment, the specific values ​​of the zoom state threshold and the number of classifications are not limited to these, and other experimentally obtained values ​​and the number of classifications may be more finely classified as long as they are two or more. As an example, when the CPU 301 acquires a communication delay amount Lctrl of 200 ms and a zoom state Zstatus of 0.5, the value "1" of the block 807 on the table shown in FIG. 8 is acquired as the traceability Traceability.

[0054] In this manner, in the present embodiment, when the communication delay amount Lctrl and the zoom state Zstatus satisfy a predetermined condition, the CPU 301 acquires "0" as the traceability Traceability. On the other hand, when another predetermined condition is satisfied, the CPU 301 acquires "1". When neither condition is satisfied, the CPU 301 acquires "2".

[0055] 9(A) to 9(C) are explanatory diagrams of the photographing angle of view and the subject size for each zoom state. FIG. 9(A) shows a captured image 911 in which the camera 100 captures a subject 901 when the zoom state is "large" where the zoom state Zstatus is 0.4 or more. 921 represents the position of the subject's face. I1 represents the distance in the real world that appears in the captured image 911, and is measured in meters. As shown in FIG. 9(A), when the zoom state is "large", the angle of view is a bust-up shot in which the subject's chest and above are the focus of attention.

[0056] Fig. 9(B) shows a captured image 912 of a subject 902 captured by the camera 100 when the zoom state is "medium" where the zoom state Zstatus is greater than 0.15 and equal to or less than 0.4. 922 represents the position of the subject's face. I2 represents the distance in the real world shown in the captured image 912, measured in meters. As shown in Fig. 9(B), when the zoom state is "medium", the angle of view is a close-up of the upper body of the subject, with attention focused on the upper body.

[0057] Fig. 9(C) shows a captured image 913 of a subject 903 captured by the camera 100 when the zoom state is "small" where the zoom state Zstatus is 0.15 or less. 923 indicates the position of the subject's face. I3 indicates the distance in the real world captured in the captured image 913, and is measured in meters. As shown in Fig. 9(C), when the zoom state is "small", the angle of view is an overhead shot that captures the entire body of the subject and its surrounding environment.

[0058] Next, referring to Fig. 9(A) to (C), the mechanism by which a subject goes out of frame due to the communication delay amount Lctrl and the zoom state Zstatus will be described. As an example, a case will be described in which the communication delay amount Lctrl is 400 ms, the distance l2 is 3 m (corresponding to the zoom state "medium"), and a person walking briskly at 1.8 m / s is tracked by the camera 100 in the center of the image in a Hinomaru composition. A person walking briskly moves Lctrl x 1.8 [m / s] = 0.72 m during the time of the communication delay amount Lctrl. In other words, at the time when the camera 100 captures an image in which the subject is at a position 0.72 m from the center of the image, the user can finally confirm that the subject has started to move in the image.

[0059] Since the control using the controller 300 is performed after the user confirms that the subject has started to move, the camera 100 actually starts to move when the control command arrives at the camera 100 with a delay of the communication delay amount Lctrl. That is, the camera 100 actually starts to move when the person is at a position of 1.44 m from the center of the captured image. This is smaller than l2 / 2=1.5 m, so it means that the subject has not gone out of the frame. That is, if the user performs a pan / tilt operation to track the subject the moment he or she confirms that the subject has started to move, it means that in this case the pan / tilt operation can be started before the person goes out of the frame of the captured image.

[0060] Considering the above explanation of the frame-out mechanism, when the zoom state Zstatus is large, there is less time before the subject goes out of the frame. Therefore, as shown in FIG. 8, when the communication delay amount Lctrl is large and the zoom state Zstatus is large, the value of the traceability Traceability is determined so as to make the degree of warning stronger. On the other hand, when the communication delay amount Lctrl is small and the zoom state Zstatus is small, the value of the traceability Traceability is determined so as not to issue a warning. Note that each value is defined to explain the traceability Traceability, but this is merely for the purpose of concrete explanation and is not limited to this. After the CPU 301 determines the traceability Traceability and writes it to the RAM 303, the process proceeds to step S306 in FIG. 6.

[0061] In step S306, CPU 301 determines whether manual tracking is possible or not according to the value of Traceability determined in step S305. If the value of Traceability is 0, CPU 301 determines that manual tracking is possible and proceeds to step S301. On the other hand, if the value of Traceability is not 0, CPU 301 determines that manual tracking is not possible and proceeds to step S307. In step S307, CPU 301 generates a message (information on tracking performance) notifying the user that manual tracking of the subject is difficult and stores it in RAM 303. CPU 301 then reads out an image (warning UI) related to the message from RAM 303 and displays it on display unit 305, thereby presenting the message to the user.

[0062] 10(A) and (B) are composite images in which a captured image is superimposed on a message generated according to the value of Traceability acquired by the CPU 301. Fig. 10(A) shows a composite image in which a warning message 1001 in the case where Traceability is "1" is superimposed on a captured image 611. Fig. 10(B) shows a composite image in which a warning message 1002 in the case where Traceability is "2" is superimposed on a captured image 612.

[0063] The user can check the attention message 1001 and the warning message 1002 based on the communication delay during imaging and the size of the person within the angle of view. Therefore, the user can adjust the zoom value of the camera 100 or the speed of the pan / tilt operation of the camera 100 in advance based on the content of the message. After the display unit 305 displays the message (warning UI), the process proceeds to step S301. Alternatively, the CPU 301 may control the output of an image obtained from the camera 100 via the image output I / F 205, in which message data is superimposed, and the image is displayed on an external monitor (not shown). Also, a speaker or a vibration control unit may be provided in the controller 300, and a method of warning the user using sound or vibration may be adopted.

[0064] As described above, by referring to the amount of communication delay and the zoom state to determine whether or not manual tracking of the subject is possible, and by displaying a message if tracking is difficult, the user can set the camera in advance to enable manual tracking.

[0065] The method of calculating the zoom state described in step S303 may be another method. For example, the CPU 301 may use the ratio of the size of the person's face in the image captured by the camera 100 as the zoom state Zstatus. Specifically, the controller 300 transmits a control command to the information processing device 200 to obtain the size of the person's face in the image as the zoom state.

[0066] 11 is a flowchart of the operation (calculation process of the zoom state) of the information processing device 200 that receives a control command via the network I / F 204. This flow starts when the CPU 201 of the information processing device 200 detects that a zoom state acquisition command has arrived at the network I / F 204.

[0067] First, in step S1101, the CPU 201 reads out a control command received via the network I / F 204 and writes it to the RAM 203. Next, in step S1102, the CPU 201 acquires an image captured by the camera 100 via the network I / F 204 and acquires the size of the subject in the image. Specifically, the CPU 201 writes the captured image of the camera 100 acquired via the network I / F to the RAM 203, and the inference unit 207 inputs the image stored in the RAM 203 and infers the coordinates of the face of a person appearing in the image. Here, the coordinates of the face of a person in this embodiment are the two-dimensional coordinate values ​​of the upper left vertex and the lower right vertex of a rectangular area circumscribing the face of the person, which are used to express the position of the face of the person in the image. The inference unit 207 writes the coordinates of the face of the person to the RAM 203 and proceeds to step S1103.

[0068] In step S1103, CPU 201 obtains the ratio of the detected person within the angle of view as the zoom state by dividing the number of horizontal pixels of the rectangular area of ​​the person's face obtained in step S1102 by the number of horizontal pixels of the image. If no person is detected in the image, CPU 201 determines that there is no rectangular area of ​​the person's face, and sets the zoom state Zstatus to 0. After CPU 201 writes the zoom state Zstatus to RAM 203, the process proceeds to step S1104. In step S1104, CPU 201 transmits the zoom state Zstatus to controller 300 via network I / F 204. After CPU 301 of controller 300 stores the zoom state Zstatus in RAM 303, this flow ends.

[0069] According to this embodiment, the size of a person in a captured image can be obtained, and the zoom state Zstatus can be calculated regardless of individual differences in the size of the person's face, making it possible to determine the tracking feasibility parameters more accurately.

[0070] In the present embodiment, the controller 300 acquires the ratio of the size of the person's face in the image captured by the camera 100 from the information processing device 200. However, other methods may be applied as long as they can similarly obtain the ratio of the size of the face. For example, the camera 100 may have an inference unit, and the flow of FIG. 11 may be performed by the camera 100. In that case, the controller 300 may transmit a zoom state acquisition command to the camera 100. Alternatively, the controller 300 may have an inference unit, and the CPU 301 may acquire the image captured by the camera 100 via the network I / F 304, and the CPU 301 may perform the processes from step S1102 to step S1104. In these embodiments, the information processing device 200 is not essential.

[0071] Alternatively, the focal length of the camera 100 at the time of shooting may be used as the zoom state Zstatus acquired by the controller 300 from the information processing device 200 in step S303 of this embodiment. For example, when the focal length of the camera 100 is variable from fmin to fmax, the traceability Traceability may be set to "0" when the zoom state Zstatus is in the range of fmin to fmin+(fmax-fmin) / 3. Similarly, the traceability Traceability may be set to "1" when the focal length is in the range of fmin+(fmax-fmin) / 3 to fmin+2*(fmax-fmin) / 3. Also, the traceability Traceability may be set to "2" when the focal length is in the range of fmin+2*(fmax-fmin) / 3 to fmax. Also, the normalized value of the focal length of the camera 100 at the time of shooting or a value converted into a predetermined range may be used as the zoom state Zstatus. Furthermore, any other method may be used as long as it increases the traceability when the focal length is long (when the angle of view is narrow).

[0072] Even in this way, it is possible to display warnings and cautions in the same way. That is, when the focal length is large (when the zoom state is "large"), the angle of view becomes narrower, the subject becomes relatively large, and it becomes easier for it to go out of frame. In such a case, a warning can be displayed.

[0073] In this embodiment, an example in which a single person appears in the captured image has been described, but when multiple people are present, the inference unit 207 can detect only the specified person. As a method for detecting only the specified person, the inference unit 207 acquires the feature amount of the person selected and detected in advance and stores it in the RAM 203. Thereafter, the CPU 201 extracts a person whose feature amount detected by the inference unit 207 is highly similar to the feature amount of the selected person stored in the RAM 203. This makes it possible to continuously detect the person selected in advance. In this embodiment, the traceability Traceability is acquired based on two values, the communication delay amount Lctrl and the zoom status Zstatus, but the traceability Traceability may be corrected according to the position of the subject in the image.

[0074] Fig. 12 is an explanatory diagram of a correction process of the Traceability according to a detection result 1208 of the face position of a subject 1207 in a captured image 1201. In Fig. 12, the camera 100 acquires a captured image 1201 with a communication delay Lctrl of 300 ms and a zoom state Zstatus of 0.3. Therefore, the CPU 301 refers to the table shown in Fig. 5 and acquires the value "1" of block 805 as the Traceability.

[0075] A horizontal center line 1202 indicates the horizontal center line of the captured image 1201. Line segments 1203 and 1204 are drawn at positions shifted 1 / 6 to the left and right from the horizontal center line 1202, and if the subject 1207 is contained between these two line segments, the subject 1207 is unlikely to go out of frame. Line segments 1205 and 1206 are drawn at positions shifted 2 / 6 to the left and right from the horizontal center line 1202, and if the subject 1207 is contained outside these two line segments, even a small movement of the subject 1207 will cause it to go out of frame.

[0076] As described above, CPU 301 corrects Traceability according to the position of detection result 1208 indicating subject 1207 as follows: If detection result 1208 is between line segment 1203 and line segment 1205, or between line segment 1204 and line segment 1206, then Traceability is increased by "1." If detection result 1208 is outside line segment 1205 or outside line segment 1206, then Traceability is increased by "2."

[0077] In the example of FIG. 12, detection result 1208 indicating the position of subject 1207 is observed between line segments 1203 and 1205, and therefore there is a higher possibility of subject going out of frame than if subject 1207 were captured near horizontal center line 1202. For this reason, CPU 301 corrects Traceability by adding "+1" as described above, and generates a message according to the value of Traceability "2" in step S107. Note that if the result of the above-mentioned addition causes Traceability to exceed "2", it may be clipped to "2". Alternatively, if Traceability exceeds "2", a stronger message than the warning shown in FIG. 10(B) may be displayed in step S107 according to the value of Traceability.

[0078] This makes it possible to generate a message that can more accurately present to the user the likelihood of the subject 1207 going out of frame. The process of correcting the traceability according to the position of the subject in the image described above is not limited to this. Instead, the traceability may be corrected based on the distance between the subject and the camera 100. Or, both of these may be performed. As a means for acquiring the distance between the subject and the camera 100, there are a method of calculating from the focal length and focus position of the camera 100, a method of calculating from the focal length of the camera 100 and the size of the subject in the image, and a method of measuring using a distance measuring device (not shown). Note that any method that can acquire the distance between the subject and the camera may be used, and is not limited to these methods.

[0079] Next, a method of calculation from the focal length of the camera 100 acquired by the CPU 301 via the network I / F 304 and Zstatus acquired from the RAM 303 will be specifically described. First, the CPU 301 calculates the horizontal angle of view θh captured by the camera 100 from the focal length. Hereinafter, the horizontal angle of view θh will be described as 90°. Next, the CPU 301 calculates the horizontal real-world distance Lh at the position where the subject is captured from the zoom state Zstatus. In this embodiment, the zoom state Zstatus is 0.1, the size of the face of the subject whose position is acquired by the inference unit 207 is 0.15 m, and the distance Lh can be calculated as 0.15 / 0.1=1.5 m.

[0080] Thereafter, CPU 301 can obtain the distance x between the subject and camera 100 as an approximate value of Lh / (2×tan(θh / 2))=0.75 m. As a correction of the traceability performed by CPU 301, for example, when the distance x is smaller than a predetermined distance, the traceability is corrected by adding "+1". This is because the closer the subject captured by camera 100 is to camera 100, the faster the pan control and tilt control need to be performed.

[0081] After the CPU 301 generates a warning UI in step S307 and displays it on the display unit 305, if the zoom state Zstatus acquired by the CPU 301 in step S303 is 0 again, the CPU 301 may cause the display unit 305 to stop displaying the warning UI.

[0082] In this embodiment, the characteristic processing of the control system 10 including the controller 300 has been described, but the present invention is not limited to this. An information processing device (not shown) equipped with an application that realizes a similar function may be substituted. In this case, the warning UI needs to be displayed on a display (not shown) or other device having a display function via an image output I / F of the information processing device (not shown).

[0083] In the present embodiment, the CPU 301 measures the amount of communication delay with the camera 100 via the network I / F 304, but the present invention is not limited to this. The time required for communication between a device near the camera 100 and a device near the controller 300 may be measured as long as they are both via the Internet 600, which is the main cause of communication delay. For example, the time required for communication between a device connected to the LAN 400 and a device connected to the LAN 500 may be measured.

[0084] In the present embodiment, the process of acquiring the traceability Traceability by referring to the table shown in Fig. 8 based on two values ​​of the communication delay amount Lctrl and the zoom state Zstatus has been described, but the present invention is not limited to this. For example, a function for calculating the traceability Traceability using two values ​​of the communication delay amount Lctrl and the zoom state Zstatus as input may be prepared and determined. A specific function and the determined traceability Traceability in this case will be described below.

[0085] First, the CPU 301 reads out from the ROM 302 a first velocity v1 of the subject and a second velocity v2 that is faster than the first velocity v1, which are determined in advance. Next, the CPU 301 determines the size of the subject's face to be 0.15 m, and calculates the horizontal distance l in the real world of the image captured by the camera 100 as 0.15 / Zstatus. The CPU 301 calculates the distance lvi that the subject moves due to communication delay until the control of the camera 100 starts by vi×Lctrl×2 (i=1, 2). The CPU 301 compares lvi with the distance l / 2 [m] from the center of the captured image of the camera 100 to the edge of the image. If lv1≧l / 2, the traceability Traceability is determined to be "2". If lv2≧l / 2>lv1, the traceability Traceability is determined to be "1". If l / 2>lv2, the traceability Traceability is determined to be 0. This makes it possible to carry out this flow without preparing the table shown in FIG.

[0086] In this embodiment, the horizontal distance captured by the image of the camera 100 (for example, l1, l2, or l3 in FIG. 9) may be used as the zoom status Zstatus. This is because when the size of a person's face is defined as 0.15 [m], it can be expressed as an inverse relationship such as Zstatus=0.15 / l2, and since they are essentially synonymous, similar processing can be performed by appropriately modifying the judgment conditions of Zstatus. In other words, any method may be used as long as it can determine the difficulty of tracking a subject using the camera 100 from the communication delay amount Lctrl and the zoom status Zstatus.

[0087] Second embodiment Next, a second embodiment of the present invention will be described. In the first embodiment, a process was described in which it was determined whether or not the subject could be tracked by manual control based on the communication delay amount and the zoom state, and a message was displayed if tracking was difficult. In this embodiment, a process was described in which not only a message regarding a warning or caution is presented, but also an assist menu corresponding to the message is presented to the user to support camera control. This embodiment differs from the first embodiment in that an assist menu is displayed in conjunction with the controller 300 displaying a warning or caution message.

[0088] The system configuration in this embodiment is similar to the configuration of the control system 10 described in the first embodiment with reference to Fig. 1, and therefore the description thereof will be omitted. Also, the hardware configurations of the camera 100, the information processing device 200, and the controller 300 in this embodiment are similar to the configurations described in the first embodiment with reference to Fig. 2, and therefore the description thereof will be omitted.

[0089] In this embodiment, the controller 300 acquires the amount of communication delay when exchanging information via the Internet 600, and when it is determined that the tracking performance does not satisfy a predetermined standard, a process of presenting an assist menu related to pan / tilt control will be described. Specifically, the assist menu display of the controller 300 after generating the warning UI in step S307 in Fig. 6 and the control content of the camera 100 will be described. Note that the basic process performed by the controller 300 is similar to the process described in the first embodiment with reference to Fig. 3, and therefore the description thereof will be omitted.

[0090] 13 is a flowchart showing the generation of an assist menu performed in step S307 by the CPU 301. In this embodiment, it is assumed that the communication delay amount Lctrl is 300 ms, the zoom state Zstatus is 0.3, and the camera 100 is acquiring a captured image.

[0091] First, in step S1301, if the CPU 301 determines that the tracking possibility parameter is a value that makes it impossible to manually track the subject, it generates a GUI of an assist menu to inform the user that manual tracking of the subject is difficult. Then, the CPU 301 stores the GUI of the assist menu in the RAM 303.

[0092] FIG. 14 is an explanatory diagram of the GUI of the assist menu in this embodiment. The CPU 301 superimposes the assist menu 1402 on the captured image of the camera 100 and stores it in the RAM 303. Items 1403, 1404, and 1405 in the assist menu 1402 indicate the contents of the assist in this embodiment. That is, the CPU 301 can instruct at least one of pan control, tilt control, and zoom control (speed adjustment control or automatic subject tracking control) based on the user's selection. The CPU 301 acquires the selected menu ID via the user input I / F 306. The menu ID is a unique value determined according to each item. For example, when the item 1403 is selected, the CPU 301 may store the menu ID (1) in the RAM 303.

[0093] After display unit 305 reads the GUI of the assist menu from RAM 303 and starts displaying it, the process proceeds to step S1302 in Fig. 13. In step S1302, CPU 301 determines whether or not a predetermined time has passed since the menu ID was obtained. If CPU 301 determines that the predetermined time has passed since the menu ID was obtained, it determines that the user does not think that the assistance content presented in the assist menu generated by CPU 301 is unnecessary, and the process proceeds to step S1305. On the other hand, if the predetermined time has not passed since the menu ID was obtained, the process proceeds to step S1303.

[0094] In step S1303, CPU 301 determines whether or not a user selection has occurred for the assist menu generated in step S1301. CPU 301 determines whether or not a user selection has occurred by referring to RAM 303 and judging whether or not a menu ID has been acquired. If CPU 301 has acquired a menu ID, it is determined that a user selection has occurred, and the process proceeds to step S1304. On the other hand, if CPU 301 has not acquired a menu ID, it is determined that a user selection has not occurred, and the process proceeds to step S1302.

[0095] In step S1304, CPU 301 transmits a control command corresponding to the acquired menu ID via network I / F 304, and causes camera 100 to execute the command. In this embodiment, the control menus corresponding to menu IDs (1) to (3) will be described.

[0096] When CPU 301 acquires (1) as the menu ID, it transmits a control command to camera 100 via network I / F 304 to decrease the focal length of camera 100 in order to decrease the zoom state (decrease the zoom factor, reduce the subject size). The target value of the focal length is set automatically. In this embodiment, since the value of the zoom state Zstatus is 0.3, CPU 301 controls camera 100 to automatically zoom out until the zoom state Zstatus becomes 0.1. As a result, in the table shown in FIG. 8, while the Traceability before assist was "1" stored in block 805, by executing this assist, the traceability Traceability decreases to 0 stored in block 806.

[0097] When the CPU 301 acquires (2) as the menu ID, the CPU 301 issues a control command to increase the pan speed / tilt speed to the camera 100. In this embodiment, the value of the angular acceleration of the pan / tilt control is changed as the pan / tilt speed. The value of each acceleration after the change is automatically set. By increasing the angular acceleration of the pan / tilt by the camera 100, the maximum control speed of the pan / tilt set in the camera 100 can be reached more quickly. In particular, even if the distance between the camera 100 and the subject to be manually tracked is short, tracking is easy to perform. Note that in this embodiment, the angular acceleration of the pan / tilt control is described, but this is not limited to this. The angular velocity of the pan / tilt control may be increased, or both the angular velocity and the angular acceleration may be increased. Alternatively, a correction may be made to increase the pan / tilt speed inside the controller 300. In this way, any method that increases the amount of change in the pan / tilt is available.

[0098] When the CPU 301 acquires (3) as the menu ID, the CPU 301 transmits a control signal for starting automatic tracking to the information processing device 200 via the Internet 600.

[0099] 15 is a flowchart showing the operation of the information processing device 200 when the information processing device 200 receives a control signal for causing the CPU 301 to start automatic tracking via the Internet 600. This flow is initiated when the CPU 201 of the information processing device 200 receives an instruction for execution via the network I / F 204 or the user input I / F 206.

[0100] First, in step S1501, CPU 201 determines whether or not a command indicating the end of this flow has been received (whether or not to continue processing) via network I / F 204 or user input I / F 206. If it is determined that the processing is to be continued, the process proceeds to step S1502. On the other hand, if it is determined that the processing is not to be continued, the process ends.

[0101] In step S1502, the CPU 201 receives images captured by the camera 100 from the image input I / F 208 of the information processing device 200. The captured images are sequentially transmitted from the image output I / F 104 of the camera 100 in accordance with a predetermined frame rate, and the image input I / F 208 of the information processing device 200 sequentially writes the received captured images to the internal RAM 203. The captured images may also be received via the network I / F 204 and expanded in the internal RAM 203.

[0102] Next, in step S1503, the CPU 201 reads out the image from the RAM 203 and inputs it to the inference unit 207, and stores the type of subject inferred by the inference unit 207 and position information of the subject in the captured image in the RAM 203. The inference unit 207 has a trained model created using a machine learning method such as deep learning, receives an image as input data, and performs subject detection by outputting the type of subject such as a person, position information, and a score indicating the likelihood as output data. In this embodiment, the position information is coordinates indicating the center of gravity of a target in the image.

[0103] Next, in step S1504, CPU 201 transmits a command to inquire about current control information to camera 100 via network I / F 204, and stores the response in RAM 203. The control information is information related to driving such as the maximum angle, minimum angle, and current angle in panning and tilting, and the maximum angle of view, minimum angle of view, and current angle of view in zooming, and information related to images such as image resolution and format. In this embodiment, the above-mentioned information is mainly used, but the obtainable control information is not limited to these.

[0104] Next, in step S1505, CPU 201 calculates the pan and tilt directions and angular velocities so as to track the subject, based on the position information stored in RAM 203 in step S1503 and the control information of camera 100 acquired in step S1504. CPU 201 then converts this into a control command (control value) in accordance with a protocol previously determined as a method for controlling camera 100, and writes the control command to RAM 203.

[0105] Next, in step S1506, CPU 201 transmits the control command written to RAM 203 in step S1505 to camera 100 via network I / F 204. Note that the operation of camera 100 receiving the control command is the same as the operation of camera 100 receiving a control command from controller 300, as described in the first embodiment with reference to Fig. 4, and therefore a description thereof will be omitted. After CPU 201 executes control according to the menu ID, the process proceeds to step S1305 in Fig. 13.

[0106] In step S1305, the CPU 301 executes processing to stop displaying the warning UI. In this embodiment, the CPU 301 stops generating the GUI of the assist menu, thereby ending the display of the warning UI displayed by the controller 300. After the controller 300 ends displaying the warning UI, the processing to display the assist menu and perform control according to the selection is ended.

[0107] In this embodiment, the process of performing control according to the menu ID selected for the assist menu generated in step S1301 has been described, but the present invention is not limited to this. A predetermined process may be executed without waiting for a user selection. For example, the CPU 301 may generate a UI for the assist menu in step S1301, and the display unit 305 may start displaying it, and then the process may proceed to step S1304 after a certain time has elapsed, and the predetermined control may be started.

[0108] According to this embodiment, not only is it determined whether or not the subject can be tracked manually based on the communication delay amount and the zoom state, but if it is determined that tracking is difficult, an assist menu is displayed and control is performed according to the selection, making it easier for the user to capture an image by tracking the subject.

[0109] In this embodiment, as in the first embodiment, only a warning UI is notified, and the assist menu may be started automatically. For example, if only the "zoom down" assist process is possible, the assist process may be started without notification to the user or a selection from the user.

[0110] Third embodiment Next, a third embodiment of the present invention will be described. In the second embodiment, the process of determining whether the subject can be manually tracked based on the communication delay amount Lctrl and the zoom state Zstatus, and assisting the control of the camera 100 according to the result was described. In this embodiment, the traceability Traceability is calculated by referring to only the communication delay amount Lctrl, and a warning UI is generated and displayed. This embodiment differs from the second embodiment in that the controller 300 determines whether the subject can be manually tracked based only on the communication delay amount Lctrl, and displays an assist menu.

[0111] The system configuration in this embodiment is similar to the configuration of the control system 10 described in the first embodiment with reference to Fig. 1, and therefore the description thereof will be omitted. Also, the hardware configurations of the camera 100, the information processing device 200, and the controller 300 in this embodiment are similar to the configurations described in the first embodiment with reference to Fig. 2, and therefore the description thereof will be omitted.

[0112] With reference to Fig. 16, a process will be described in which controller 300 acquires the amount of communication delay when exchanging information via Internet 600 and presents an assist menu related to pan / tilt control when it is determined that the tracking performance does not satisfy a predetermined standard. Fig. 16 is a flowchart showing the operation of controller 300. Note that in Fig. 16, steps S1601 to S1603, S1605, and S1606 are similar to steps S301, S302, S304, S306, and S307 in Fig. 6, respectively, and therefore description thereof will be omitted.

[0113] In step S1604, the CPU 301 of the controller 300 obtains the communication delay amount Lctrl from the RAM 303, and determines the traceability Traceability.

[0114] Fig. 17 is an explanatory diagram of a table for determining traceability, showing the relationship between the magnitude of the communication delay amount Lctrl and the traceability Traceability. As shown in Fig. 17, this embodiment refers to only the communication delay amount Lctrl when determining the traceability Traceability, which differs from the first embodiment in that it refers to the communication delay amount Lctrl and the zoom status Zstatus as shown in Fig. 8.

[0115] The CPU 301 acquires the communication delay amount Lctrl from the RAM 303 and plots it on a table to acquire the traceability, which is a parameter corresponding to the plot point. In this embodiment, similar to the first embodiment, the traceability is classified into three types: a warning when the communication delay amount Lctrl is "2", a caution when the communication delay amount is "1", and no problem when the communication delay amount is "0". When the communication delay amount Lctrl acquired by the CPU 301 is "small", the CPU 301 acquires "0" corresponding to the block 1703 as the traceability. When the communication delay amount Lctrl acquired by the CPU 301 is "medium", the CPU 301 acquires "1" corresponding to the block 1702 as the traceability. When the communication delay amount Lctrl acquired by the CPU 301 is "large", the CPU 301 acquires "2" corresponding to the block 1701 as the traceability. After the CPU 301 determines the traceability in step S1604 in FIG. 16, the process proceeds to step S1605.

[0116] According to this embodiment, the controller 300 can calculate the traceability Traceability by referring only to the communication delay amount Lctrl, and can assist in controlling the camera 100.

[0117] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0118] According to each embodiment, it is possible to provide a control device, a control system, a control method, and a program capable of reducing the discomfort felt by a user due to the occurrence of communication delays.

[0119] The disclosure of each embodiment includes the following configurations and methods. (Configuration 1) A control device for remotely controlling an imaging device, A communication unit that communicates with the imaging device; A notification unit that notifies a user; A control device comprising: a control unit that determines whether a communication delay amount in the communication and a zoom state of the imaging device satisfy predetermined conditions, and controls a notification by the notification unit based on a result of the determination. (Configuration 2) The control device according to configuration 1, characterized in that, when the control unit determines that the communication delay amount and the zoom state do not satisfy the specified condition, the control unit controls the notification unit to notify a user of information indicating the result of the determination. (Configuration 3) The control unit is the communication delay amount is a first value and the zoom state is a third value; the communication delay amount is a first value, and the zoom state is a fourth value that is greater than the third value; determining that a predetermined condition is satisfied when the communication delay amount is a second value greater than the first value and the zoom state is a third value; 3. The control device according to configuration 1 or 2, wherein when the communication delay amount is the second value and the zoom state is the fourth value, it is determined that the predetermined condition is satisfied. (Configuration 4) 4. The control device according to any one of configurations 1 to 3, wherein the zoom state is an angle of view of the imaging device, and the smaller the angle of view, the larger the zoom state. (Configuration 5) 4. The control device according to any one of configurations 1 to 3, wherein the zoom state is a size of a subject in an image captured by the imaging device, and the larger the subject, the larger the zoom state. (Configuration 6) The control device according to any one of configurations 1 to 5, characterized in that, when the control unit determines that the predetermined condition is not satisfied, the control unit instructs the imaging device to perform at least one of pan control, tilt control, and zoom control via the communication unit. (Configuration 7) 7. The control device according to configuration 6, wherein at least one of the pan control, the tilt control, and the zoom control is a speed adjustment control or an automatic subject tracking control. (Configuration 8) The control device according to any one of configurations 1 to 7, characterized in that the control unit determines tracking performance based on the amount of communication delay and the zoom state, and determines that the specified condition is satisfied when the tracking performance satisfies a specified criterion. (Configuration 9) 9. The control device according to configuration 8, wherein the tracking performance is a parameter determined based on the amount of communication delay and the zoom state. (Configuration 10) The control device according to configuration 8 or 9, wherein the control unit determines the tracking performance based on at least one of a position of a subject in an image captured by the imaging device or a distance between the subject and the imaging device. (Configuration 11) The control device according to any one of configurations 1 to 10, characterized in that, when the control unit determines that the communication delay amount and the zoom state satisfy the specified condition while providing a notification based on the result of the determination, the control unit controls the notification unit to stop the notification. (Configuration 12) The control device according to any one of configurations 1 to 11, characterized in that, when the control unit determines that no subject is present in an image captured by the imaging device for a predetermined period while providing a notification based on the result of the determination, the control unit controls the notification unit to stop providing a notification based on the result of the determination. (Configuration 13) A receiving means for receiving an operation from a user is provided, The control device according to any one of configurations 1 to 12, characterized in that the control unit outputs at least one of pan control, tilt control, and zoom control to the imaging device via the communication unit based on a user's operation accepted by the acceptance means. (Configuration 14) 14. The control device according to any one of configurations 1 to 13, wherein the notification unit notifies the user by display, sound, or vibration. (Configuration 15) A control device for remotely controlling an imaging device, A communication unit that communicates with the imaging device; a control unit that determines whether a communication delay amount in the communication satisfies a predetermined condition; A receiving means for receiving an operation from a user, The control unit is determining at least one of a drive speed and a drive amount for at least one of pan control, tilt control, and zoom control based on the user's operation received by the receiving means, and performing a first control to instruct the imaging device via the communication unit; A control device characterized by performing a second control in which, when a communication delay amount in the communication does not satisfy a predetermined condition, at least one of a drive speed and a drive amount of at least one of pan control, tilt control, and zoom control is automatically determined, and an instruction is given to the imaging device via the communication unit. (Configuration 16) The control device according to configuration 15, characterized in that the second control is control for causing the imaging device to execute at least one of control for increasing a pan speed, control for increasing a tilt speed, control for zooming out, and automatic subject tracking control for controlling at least one of panning, tilting, and zooming based on a result of subject detection. (Configuration 17) The control device according to configuration 15 or 16, characterized in that when the control unit determines that a subject is present in an image captured by the imaging device and that the specified condition is not satisfied, the control unit instructs at least one of the pan control, the tilt control, or the zoom control via the communication unit. (Configuration 18) An imaging device; A control system comprising: a control device according to any one of configurations 1 to 17. (Method 1) A control method for remotely controlling an imaging device, comprising: communicating with the imaging device; a step of determining whether a communication delay amount in the communication and a zoom state of the imaging device satisfy predetermined conditions, and controlling a notification to a user based on the result of the determination. (Method 2) A control method for remotely controlling an imaging device, comprising: communicating with the imaging device; a step of determining whether or not a communication delay amount in the communication satisfies a predetermined condition, and if it is determined that the predetermined condition is not satisfied, instructing the imaging device to perform at least one of pan control, tilt control, and zoom control via the communication. (Configuration 19) A program for causing a computer to execute the control method according to Method 1 or 2.

[0120] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0121] 100 Camera (imaging device) 300 Controller (control device) 301 CPU (control unit) 304 Network I / F (Communication section) 305 Notification Department

Claims

1. A control device that remotely controls an imaging device, a communication unit that communicates with the imaging device; a notification unit that notifies the user; a control unit that determines whether a communication delay amount in the communication and a zoom state of the imaging device satisfy predetermined conditions, and controls notification by the notification unit based on the result of the determination.

2. The control device according to claim 1, characterized in that, when it is determined that the communication delay amount and the zoom state do not satisfy the specified condition, the control unit controls the notification unit to notify the user of information indicating the result of the determination.

3. The control unit the communication delay amount is a first value and the zoom state is a third value; a case where the communication delay amount is a first value and the zoom state is a fourth value that is greater than the third value; 2. The control device according to claim 1, wherein when the communication delay amount is a second value greater than the first value and the zoom state is a third value, it is determined that a predetermined condition is satisfied.

4. The control unit the communication delay amount is equal to or greater than a fifth value that is greater than the second value; the zoom state is equal to or greater than a sixth value that is greater than the fourth value; and The control device according to claim 3, wherein the control device determines that the predetermined condition is not satisfied when the communication delay amount is equal to or greater than the second value and less than the fifth value, and the zoom state is equal to or greater than the fourth value and less than the sixth value.

5. 2. The control device according to claim 1, wherein the zoom state is a field angle of the imaging device, and the smaller the field angle, the larger the zoom state.

6. 2. The control device according to claim 1, wherein the zoom state is a size of a subject in an image captured by the imaging device, and the larger the subject, the larger the zoom state.

7. The control device according to claim 1, wherein, when the control unit determines that the predetermined condition is not satisfied, the control unit instructs the imaging device to perform at least one of pan control, tilt control, and zoom control via the communication unit.

8. 8. The control device according to claim 7, wherein at least one of the pan control, the tilt control, and the zoom control is a speed adjustment control or an automatic subject tracking control.

9. The control device according to claim 1 , wherein the control unit determines tracking performance based on the amount of communication delay and the zoom state, and determines that the predetermined condition is met when the tracking performance meets a predetermined standard.

10. 10. The control device according to claim 9, wherein the tracking performance is a parameter determined based on the amount of communication delay and the zoom state.

11. The control device according to claim 9 , wherein the control unit determines the tracking performance based on at least one of a position of a subject in an image captured by the imaging device or a distance between the subject and the imaging device.

12. The control device according to claim 1, characterized in that the control unit controls the notification unit to stop the notification if it determines that the communication delay amount and the zoom state satisfy the specified condition while the notification based on the result of the determination is being made.

13. The control device according to claim 1, characterized in that the control unit controls the notification unit to stop the notification based on the result of the judgment if it determines that no subject is present in the image captured by the imaging device for a predetermined period while the notification based on the result of the judgment is being made.

14. comprising a receiving means for receiving an operation from a user, 2. The control device according to claim 1, wherein the control unit outputs at least one of pan control, tilt control, and zoom control to the imaging device via the communication unit based on the user's operation accepted by the accepting means.

15. The control device according to claim 1 , wherein the notification unit notifies the user by display, sound, or vibration.

16. A control device that remotely controls an imaging device, a communication unit that communicates with the imaging device; a control unit that determines whether a communication delay amount in the communication satisfies a predetermined condition; a receiving means for receiving an operation from a user, The control unit determining at least one of a drive speed and a drive amount for at least one of pan control, tilt control, and zoom control based on the user's operation received by the receiving means, and performing a first control to instruct the imaging device via the communication unit; A control device characterized by performing second control in which, when the communication delay amount in the communication does not satisfy a predetermined condition, the control device automatically determines at least one of the drive speed and drive amount of at least one of pan control, tilt control, or zoom control, and instructs the imaging device via the communication unit.

17. The control device according to claim 16, characterized in that the second control is a control that causes the imaging device to execute at least one of control to increase a pan speed, control to increase a tilt speed, control to zoom out, and automatic subject tracking control that controls at least one of panning, tilting, and zooming based on a result of subject detection.

18. The control device according to claim 16, characterized in that the control unit, when it is determined that a subject is present in an image captured by the imaging device and the predetermined condition is not satisfied, instructs at least one of the pan control, the tilt control, and the zoom control via the communication unit.

19. An imaging device; A control system comprising: a control device according to any one of claims 1 to 18.

20. A control method for remotely controlling an imaging device, comprising: communicating with the imaging device; a step of determining whether the amount of communication delay in the communication and the zoom state of the imaging device satisfy predetermined conditions, and controlling notification to the user based on the result of the determination.

21. A control method for remotely controlling an imaging device, comprising: communicating with the imaging device; a step of determining whether or not a communication delay amount in the communication satisfies a predetermined condition, and if it is determined that the predetermined condition is not satisfied, instructing the imaging device to perform at least one of pan control, tilt control, and zoom control via the communication.

22. 22. A program causing a computer to execute the control method according to claim 20 or 21.

Images