Controller and control method

Abstract

To provide a technique for reducing sudden change in photographing direction if detection on a tracking object from a photographic image ends in failure.SOLUTION: A controller calculates a direction to a tracking object as a tracking subject direction based upon a detection result when the tracking object can be detected from a photographic image, or controls a control speed in a photographing direction based upon a tracking subject direction which is detected in the past when the tracking object cannot be detected from the photographic image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an imaging control technique. [Background technology]

[0002] In recent years, advances in AI (Artificial Intelligence) have led to the development of automatic tracking technology that detects a subject from video footage captured by a camera and controls the pan and tilt of the camera to track the subject.

[0003] If another subject passes in front of the subject being tracked (tracked subject), or if the tracked subject moves and becomes hidden behind an obstacle such as a wall, the tracked subject may temporarily disappear from the camera's view. If the tracked subject disappears from the captured image, it will no longer be possible to detect it from the captured image.

[0004] In automatic tracking, if pan / tilt control is immediately stopped when the subject becomes temporarily undetectable due to occlusion, the camera's pan and tilt movements will come to an abrupt halt, resulting in poor quality footage.

[0005] Patent Document 1 discloses a technique in which, when a tracking target is blocked by an obstruction, the obstruction is set as a new tracking target. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-80221 Summary of the Invention [Problem to be solved by the invention]

[0007] However, when a tracking subject is blocked by another person moving in the opposite direction to the tracking subject's direction of travel, such as when two people pass each other, if the person in front is designated as the new tracking subject, the tracking subject's direction of movement will suddenly reverse. This can result in a sudden change in pan-tilt control, which can degrade the tracking quality of the captured video. This invention provides technology for reducing the occurrence of sudden changes in shooting direction when the tracking subject can no longer be detected from the captured image. [Means for solving the problem]

[0008] One aspect of the present invention is characterized in that, if a tracking target can be detected from a captured image, the direction toward the tracking target is calculated as the tracking target direction based on the detection result, and if a tracking target cannot be detected from a captured image, the control means controls the control speed of the shooting direction based on the tracking target direction calculated in the past. [Effects of the Invention]

[0009] According to the present invention, when the tracking target cannot be detected from the captured image, it is possible to reduce the occurrence of a sudden change in the shooting direction. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing an example of a system configuration. [Figure 2] FIG. 2 is a block diagram showing an example of the hardware configuration of each of the camera 100 and the workstation 200. [Figure 3] 10 is a flowchart of a process performed by the camera 100 to track and photograph a tracking subject. [Figure 4] 10 is a flowchart of a process performed by the workstation 200 to cause the camera 100 to track and photograph a subject to be tracked. [Figure 5] FIG. 10 is a diagram showing a display example in step S404. [Figure 6] FIG. 4 is a diagram showing an example of a display screen of the display unit 208 during subject tracking. [Figure 7] (a) is a graph showing the relationship between Wdiff and Vp, and (b) is a graph showing the relationship between the "angle difference between the angle of the pan direction of the tracked subject as seen from camera 100 and the current pan angle of camera 100" and the "pan control speed." [Figure 8] (a) is a diagram showing an image 800 captured by camera 100, (b) is a diagram showing a sphere 801 whose radius is the distance from camera 100 to the subject shown in the captured image, and (c) is a diagram showing the three-dimensional coordinates (X, Y, Z) of the tracked subject, as well as the pan angle PANtarget and tilt angle TILTtarget. [Figure 9] 4 is a diagram for explaining an example of subject detection characteristics of the inference unit 206. FIG. [Figure 10] FIG. 1 is a block diagram showing an example of the hardware configuration of a camera 1000. [Figure 11] 10 is a flowchart showing the tracking and photographing of a tracking subject by the camera 1000. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0012] [First embodiment] First, an example of the configuration of a system according to this embodiment will be described with reference to Fig. 1. As shown in Fig. 1, the system according to this embodiment includes a camera 100 as an imaging device that tracks and captures an image of a person 10 who is a subject to be tracked (tracked subject), and a workstation 200 as a control device that controls the operation of the camera 100.

[0013] In the system according to this embodiment, a camera 100 and a workstation 200 are connected to a network 300 such as a LAN or the Internet, and are configured to be able to communicate data with each other.

[0014] For example, workstation 200 can control the operation of camera 100 by transmitting, via network 300, to camera 100, a distribution request command requesting distribution of captured images and setting commands for setting various parameters. Camera 100 distributes captured images to workstation 200 in response to the distribution request command received from workstation 200, and stores various parameters in response to the setting command received from workstation 200. Furthermore, workstation 200 can control pan, tilt, and zoom (hereinafter also referred to as PTZ) of camera 100 by transmitting commands to camera 100 for controlling the PTZ.

[0015] Note that any connection format or communication protocol may be used between the camera 100 and the workstation 200 as long as data communication between the camera 100 and the workstation 200 is possible. For example, the camera 100 and the workstation 200 may be directly connected using a serial communication cable without going through a network.

[0016] In addition, in the system of this embodiment, the camera 100 and the workstation 200 are connected via a video cable 400, and the images captured by the camera 100 are transmitted to the workstation 200 via the video cable 400.

[0017] The method for transmitting captured images from camera 100 to workstation 200 is not limited to a specific method. For example, camera 100 may transmit captured images to workstation 200 via network 300. In other words, the configuration for transmitting and receiving various information such as commands and captured images between camera 100 and workstation 200 is not limited to a specific configuration.

[0018] Next, an example of the hardware configuration of each of the camera 100 and the workstation 200 will be described using the block diagram of Fig. 2. Note that the hardware configuration shown in Fig. 2 is merely an example of a hardware configuration that can be applied to each of the camera 100 and the workstation 200, and can be modified / altered as appropriate.

[0019] First, the camera 100 will be described. The camera 100 is a camera that tracks and photographs a tracking subject. If the camera 100 is a camera that captures moving images, the camera 100 transmits images of each frame in the moving images to the workstation 200 as captured images. On the other hand, if the camera 100 is a camera that captures still images periodically or irregularly, the camera 100 transmits the still images to the workstation 200 as captured images.

[0020] The CPU 101 executes various processes using computer programs and data stored in the RAM 102. As a result, the CPU 101 controls the overall operation of the camera 100 and executes or controls various processes that will be described as processes performed by the camera 100.

[0021] The RAM 102 has an area for storing images captured by the imaging unit 104, and an area for storing computer programs and data loaded from the storage device 103. The RAM 102 also has an area for storing various information received from the workstation 200 via the communication unit 106, and a work area used by the CPU 101 when executing various processes. In this way, the RAM 102 can provide various areas as needed.

[0022] The storage device 103 is a non-volatile storage device such as a flash memory, HDD, SSD, or SD card. The storage device 103 stores setting data for the camera 100, computer programs and data related to the startup of the camera 100, and computer programs and data related to the basic operation of the camera 100. The storage device 103 also stores computer programs and data for causing the CPU 101 to execute or control various processes described as processes performed by the camera 100.

[0023] The imaging unit 104 includes an optical system, an imaging element that converts light focused through the optical system into an electric charge (image signal), and an image processing circuit that generates a captured image based on the image signal. The imaging element may be, for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor. Alternatively, the imaging element may be a CCD (Charge Coupled Device) image sensor.

[0024] The video output unit 105 is an interface for transmitting images captured by the imaging unit 104 to the workstation 200 via a video cable 400. The video output unit 105 includes, for example, an SDI (Serial Digital Interface) or an HDMI (High-Definition Multimedia Interface) (registered trademark).

[0025] The communication unit 106 is an interface for performing data communication with the workstation 200 via the network 300. The PTZ driving unit 107 controls the imaging direction (pan angle, tilt angle) of the camera 100 (imaging unit 104) and the angle of view (zoom) of the camera 100 (imaging unit 104) based on a control command received from the workstation 200.

[0026] CPU 101, RAM 102, storage device 103, imaging unit 104, video output unit 105, communication unit 106, and PTZ drive unit 107 are all connected to a system bus 108. Each unit of camera 100 shown in Fig. 2 is driven by, for example, power obtained by rectifying externally supplied AC power to a predetermined voltage, or power supplied from a built-in battery (not shown).

[0027] Next, a description will be given of the workstation 200. The workstation 200 is used as an example of a computer device such as a PC (personal computer), a smartphone, or a tablet terminal device, and any computer device may be used as long as it can achieve similar functions.

[0028] The CPU 201 executes various processes using computer programs and data stored in the RAM 202. As a result, the CPU 201 controls the overall operation of the workstation 200 and executes or controls various processes that will be described as processes performed by the workstation 200.

[0029] The RAM 202 has an area for storing computer programs and data loaded from the storage device 203, and an area for storing various information received from the camera 100 by the communication unit 204. The RAM 202 also has an area for storing captured images received from the camera 100 by the video input unit 205. The RAM 202 also has a work area used by the CPU 201 and the inference unit 206 when executing various processes. In this way, the RAM 202 can provide various areas as needed.

[0030] The storage device 203 is a non-volatile storage device such as a flash memory, HDD, SSD, SD card, etc. The storage device 203 stores setting data for the workstation 200, computer programs and data related to the startup of the workstation 200, computer programs and data related to the basic operation of the workstation 200, etc. The storage device 203 also stores computer programs and data for causing the CPU 201 and the inference unit 206 to execute or control various processes described as processes performed by the workstation 200.

[0031] The communication unit 204 is an interface for performing data communication with the camera 100 via the network 300. The video input unit 205 is an interface for receiving captured images transmitted from the camera 100 via a video cable 400, and includes, for example, SDI and HDMI.

[0032] The inference unit 206 detects an object from an input image and outputs the position and size of the object in the image. The inference unit 206 is, for example, a computing device specialized for image processing and inference processing, such as a GPU (Graphics Processing Unit). While a GPU is generally effective for inference processing, equivalent functions may be realized by a reconfigurable logic circuit such as an FPGA (Field Programmable Gate Array). Furthermore, the processing of the inference unit 206 may be performed by the CPU 201.

[0033] The user input I / F 207 is a user interface such as a keyboard, mouse, or touch panel, and can be operated by the user to input various instructions and information to the workstation 200.

[0034] The display unit 208 has a liquid crystal screen or a touch panel screen, and can display the processing results of the CPU 201 and the inference unit 206 as images, text, etc. The display unit 208 may be a projection device such as a projector that projects images and text.

[0035] In this embodiment, an example is shown in which the workstation 200 has a display unit 208, but this configuration is not limited to this. For example, the display unit 208 may be omitted from the workstation 200, and a display monitor connected to the workstation 208 may be provided.

[0036] Next, the process performed by camera 100 to track and photograph a tracking subject will be described with reference to the flowchart in Fig. 3. The flowchart in Fig. 3 is a flowchart of the process executed by camera 100 (CPU 101) in response to detecting receipt of a control command transmitted from workstation 200.

[0037] In step S301, the CPU 101 receives a control command transmitted from the workstation 200 via the communication unit 106, and stores the received control command in the RAM 102.

[0038] In step S302, CPU 101 acquires the PTZ control speed of camera 100 from the control command stored in RAM 102 in step S301. The PTZ control speed includes the speed at which the pan angle (angle in the pan direction) of camera 100 is changed, the speed at which the tilt angle (angle in the tilt direction) of camera 100 is changed, and the speed at which the zoom of camera 100 is changed.

[0039] Then, based on the acquired PTZ control speed, CPU 101 acquires drive parameters, which are parameters for changing the pan angle of camera 100 at a specified speed, changing the tilt angle of camera 100 at a specified speed, and changing the zoom of camera 100 at a specified speed. Specifically, CPU 101 acquires drive parameters for driving and controlling the motors for the pan direction and tilt direction included in PTZ drive unit 107, and drive parameters for driving and controlling the motor of a zoom drive unit included in PTZ drive unit 107. For example, CPU 101 may acquire drive parameters corresponding to the PTZ control speed acquired from the control command from a "table in which drive parameters corresponding to various PTZ control speeds are registered in advance." In this way, the method for acquiring drive parameters based on a control command is not limited to a specific method.

[0040] In step S303, the CPU 101 controls the PTZ driving unit 107 based on the driving parameters acquired in step S302. This allows the camera 100 to perform pan-tilt-zoom operations (changing the pan angle, tilt angle, and zoom) in response to control commands from the workstation 200.

[0041] Next, the processing performed by workstation 200 to cause camera 100 to perform tracking and photographing of a tracking subject will be described with reference to the flowchart of FIG. 4. The flowchart of FIG. 4 is a flowchart of processing executed in response to CPU 201 detecting an instruction to perform tracking and photographing input by a user through user input I / F 207. The instruction to perform tracking and photographing may be input from an external device. In this case, the flowchart of FIG. 4 is a flowchart of processing executed in response to CPU 201 detecting that an instruction to perform tracking and photographing transmitted from the external device has been received via communication unit 204. In this way, the method for inputting the instruction to perform tracking and photographing is not limited to a specific input method. Furthermore, the trigger for starting the processing according to the flowchart of FIG. 4 is also not limited to a specific trigger.

[0042] In step S401, CPU 201 determines whether a command (end command) instructing the end of the processing according to the flowchart of Fig. 4 has been acquired via communication unit 204 or user input I / F 207. If the result of this determination is that an end command has been acquired, the processing according to the flowchart of Fig. 4 ends, and if an end command has not been acquired, the processing proceeds to step S402.

[0043] In step S402, the CPU 201 receives the captured image transmitted from the camera 100 via the video input unit 205, and stores the received captured image in the RAM 202. In step S403, the CPU 201 inputs the captured image stored in the RAM 202 in step S402 to the inference unit 206. The inference unit 206 performs subject detection processing on the input captured image.

[0044] Here, the subject detection processing by the inference unit 206 will be described. The inference unit 206 inputs an input photographed image into a trained model created using a machine learning technique such as deep learning, and performs arithmetic processing of the trained model, thereby outputting information (rectangular frame information) that defines a rectangular frame that includes the entire body (human body) of the subject detected from the photographed image. For example, the inference unit 206 outputs information indicating the center coordinates of the rectangular frame that includes the entire body of the subject in the photographed image and the size (height and width) of the rectangular frame as rectangular frame information.

[0045] Note that the rectangular frame information is not limited to this, and may be, for example, the coordinates of the upper left vertex and the lower right vertex of the rectangular frame. Furthermore, the rectangular frame is not limited to one that includes the entire subject, but may also be a rectangular frame that includes part of the subject, such as a person's head or face. In this case, the trained model used can be changed to one that outputs rectangular frame information that defines a rectangular frame that includes a desired area. Furthermore, the subject detection process by the inference unit 206 is not limited to a method that uses a trained model. For example, the inference unit 206 may use a template matching method in which a template image of the subject is registered in advance in the storage device 203 or the like, and an area in the captured image that has a high similarity to the template image is detected as the subject area.

[0046] In step S404, the CPU 201 superimposes a rectangular frame corresponding to the rectangular frame information on the captured image stored in the RAM 202 in step S402 and displays the superimposed rectangular frame on the display unit 208. An example of the display in step S404 is shown in FIG.

[0047] 5, a rectangular frame 511 corresponding to the rectangular frame information of subject 501 and a rectangular frame 512 corresponding to the rectangular frame information of subject 502 are superimposed on a captured image 500 including subjects 501 and 502. The size of rectangular frame 511 is the size indicated by the rectangular frame information of subject 501, and the center coordinates of rectangular frame 511 are the center coordinates indicated by the rectangular frame information of subject 501. The size of rectangular frame 512 is the size indicated by the rectangular frame information of subject 502, and the center coordinates of rectangular frame 512 are the center coordinates indicated by the rectangular frame information of subject 502.

[0048] In step S405, CPU 201 determines whether or not a tracking subject has been designated. If the result of this determination is that a tracking subject has been designated, the process proceeds to step S406, and if the tracking subject has not been designated, the process proceeds to step S401.

[0049] The method for designating the subject to be tracked is not limited to a specific method. For example, in the example of Fig. 5, when the user operates the mouse serving as input I / F 207 to place the mouse cursor over rectangular frame 511 and perform a click operation, CPU 201 may designate subject 501 corresponding to rectangular frame 511 as the subject to be tracked.

[0050] Furthermore, for example, the CPU 201 may designate as the tracking subject a subject corresponding to rectangular frame information indicating the largest size of the subjects detected from the captured image, or a subject corresponding to rectangular frame information indicating the center coordinates closest to the center position of the captured image.

[0051] In step S406, CPU 201 performs identification processing to identify the tracking subject from the subjects detected in step S403. If CPU 201 can identify the tracking subject from the subjects detected in step S403, CPU 201 sets the value of the "presence flag indicating whether the tracking subject is present in the captured image" stored in RAM 202 to true, indicating that the tracking subject is present, and stores in RAM 202 the rectangular frame information of the tracking subject from the rectangular frame information output in step S403.

[0052] On the other hand, if the CPU 201 is unable to identify the tracking subject from the subjects detected in step S403, it sets the value of the existence flag stored in the RAM 202 to false, which indicates that the subject does not exist.

[0053] Since the first step S406 is the first step S406 after the tracking subject is designated, the CPU 201 sets the value of the existence flag to true and stores in the RAM 202 rectangular frame information of the designated tracking subject.

[0054] In step S406 from the second time onwards, CPU 201 performs the process described below. CPU 201 calculates the similarity between the rectangular frame information of each subject detected in step S403 and the "rectangular frame information of the tracked subject" stored in RAM 202 (the "rectangular frame information of the tracked subject" most recently stored in RAM 202 in step S406). For example, CPU 201 calculates the IoU (Intersection Over Union) as the similarity between the rectangular frame corresponding to the rectangular frame information of each subject detected in step S403 and the rectangular frame corresponding to the "rectangular frame information of the tracked subject" stored in RAM 202 (the "rectangular frame information of the tracked subject" most recently stored in RAM 202 in step S406). The greater the overlap between the rectangular frames, the higher the IoU score.

[0055] Then, CPU 201 specifies the rectangular frame information for which the highest similarity is calculated among the rectangular frame information for each subject detected in step S403 as the rectangular frame information for the subject to be tracked. In other words, CPU 201 specifies the subject for which the highest similarity is calculated among the subjects detected in step S403 as the subject to be tracked.

[0056] If any of the similarities calculated for the rectangular frame information of each subject detected in step S403 is less than the threshold value, the CPU 201 determines that the tracking subject has not been identified.

[0057] Note that the above-described process for identifying a tracking subject from a captured image is an example, and is not limited to the above-described process.Furthermore, the process for determining whether a tracking subject exists in a captured image is not limited to the above-described process.

[0058] For example, the CPU 201 may incorporate a motion prediction process for the tracking subject in the calculation of the similarity. In this case, the CPU 201 may predict the rectangular frame of the tracking subject this time from the rectangular frame information of the tracking subject two times before and the previous time, and calculate the IoU between the predicted rectangular frame and the rectangular frame of each subject as the similarity. By incorporating motion prediction, it is possible to improve the accuracy of identifying the tracking subject when the subject is moving.

[0059] As another example, the CPU 201 may calculate the similarity using not only the rectangular frame information of the subject but also features of the captured image of the subject. For example, the CPU 201 may acquire image features such as a brightness histogram within the rectangular frame and incorporate the similarity between the features into the similarity calculation.

[0060] Alternatively, the CPU 201 may use a machine learning technique such as deep learning to acquire features using a trained model that outputs a feature vector of the human body, and incorporate the features into the similarity calculation.

[0061] In step S407, CPU 201 determines whether the value of the existence flag is true or false. If the result of this determination is that the value of the existence flag is true, it is determined that a tracking subject is present in the captured image, and the process proceeds to step S408. On the other hand, if the value of the existence flag is false, it is determined that a tracking subject is not present in the captured image, and the process proceeds to step S414.

[0062] In step S408, CPU 201 reads out rectangular frame information (center coordinates and size of the rectangular frame of the tracking subject) of the tracking subject stored in RAM 202. In step S409, CPU 201 calculates the speed (PT control speed) at which camera 100 changes the shooting direction (pan angle, tilt angle) to track and shoot the tracking subject. Here, the processing in step S409 will be described using a specific example shown in FIG. 6.

[0063] Fig. 6 shows an example of the display screen of display unit 208 during object tracking. Fig. 6(a) shows a display example of a captured image 600 including object 601, which is the object to be tracked, and object 602, which is not the object to be tracked. The number of pixels in the horizontal direction of captured image 600 is W [pixels], and the number of pixels in the vertical direction is H [pixels].

[0064] Rectangular frame 611 is a rectangular frame corresponding to the rectangular frame information of subject 601, and rectangular frame 612 is a rectangular frame corresponding to the rectangular frame information of subject 602. Position 631 indicates the "center coordinates of subject 601" indicated by the rectangular frame information of subject 601, and position 632 indicates the "center coordinates of subject 602" indicated by the rectangular frame information of subject 602. In addition, in FIG. 6, the size of the rectangular frame indicated by the rectangular frame information indicates the length of the diagonal of the rectangular frame. Size S [Pixel] indicates the size of the rectangular frame indicated by the rectangular frame information of subject 601, and size 642 indicates the size of the rectangular frame indicated by the rectangular frame information of subject 602. In addition, position 650 indicates the target position for tracking imaging. In this embodiment, workstation 200 controls the pan angle and tilt angle of camera 100 so that the center coordinates of the tracked subject in the captured image approach target position 650. In the following, for ease of explanation, the target position 650 is assumed to be the center position of the captured image, but this is not limiting, and the target position 650 may be any position on the captured image.

[0065] In such a case, CPU 201 calculates the horizontal distance Wdiff [Pixel] between position 631 and target position 650 and the vertical distance Hdiff [Pixel] between position 631 and target position 650. For example, CPU 201 obtains (the horizontal coordinate of target position 650 - the horizontal coordinate of position 631) as Wdiff [Pixel]. Also, for example, CPU 201 obtains (the vertical coordinate of target position 650 - the vertical coordinate of position 631) as Hdiff [Pixel].

[0066] Then, CPU 201 calculates the pan control speed Vp, which is the speed at which the pan angle of camera 100 is changed, by calculating the following equation 1, and calculates the tilt control speed Vt, which is the speed at which the tilt angle of camera 100 is changed, by calculating the following equation 2.

[0067] Vp=(Wdiff / W)×Kp1 … Formula 1 Vt=(Hdiff / H)×Kt1 … Formula 2 Here, Kp1 in Equation 1 and Kt1 in Equation 2 are both proportionality coefficients. In other words, when Equation 1 and Equation 2 are used, the greater the distance between the target position and the central coordinates of the tracked subject, the faster the control speed becomes, and the closer the distance, the slower the control speed becomes. Also, when the central coordinates of the tracked subject reach the target position and the distance becomes zero, the control speed becomes zero. If you want to increase the control speed according to the distance between the target position and the central coordinates of the tracked subject, you can simply increase the proportionality coefficients (Kp1 or Kt1).

[0068] Graph 701 shown in FIG. 7(a) is a graph showing the relationship between Wdiff and Vp shown in Equation 1. The horizontal axis is Wdiff, and the vertical axis is Vp. Note that the control speed may be set to zero when the distance between the target position and the central coordinates of the tracking subject is less than a threshold (i.e., when the distance is very small). For example, CPU 201 may calculate Vp using a formula such that Vp becomes zero in the section where Wdiff is less than or equal to a threshold, as shown in graph 702 shown in FIG. 7(a).

[0069] In the above example, the control speed is calculated using an equation in which the relationship between the distance and the control speed is linear. However, the equation for calculating the control speed is not limited to such a linear equation, and the control speed may be calculated using another equation as long as the equation has a relationship in which the control speed increases as the distance increases and decreases as the distance decreases. This is also true when the control speed is set to zero when the distance between the target position and the center coordinates of the tracking subject is less than a threshold (i.e., when the distance is very small).

[0070] In step S410, CPU 201 calculates the speed (Z control speed) at which the zoom is changed so that camera 100 can track and photograph the subject to be tracked. In this embodiment, CPU 201 controls the zoom of camera 100 so that the size of the subject to be tracked in the photographed image becomes a specified size.

[0071] In this embodiment, the "size of the rectangular frame of the tracking subject" indicated by the rectangular frame information of the tracking subject read in step S408 for the first time is used as the "prescribed size." However, this is not limiting, and the prescribed size may be, for example, a size specified by the user by operating the user input I / F 207. Furthermore, the CPU 201 may dynamically change the prescribed size.

[0072] The processing in step S410 will now be described with reference to a specific example shown in Fig. 6. The CPU 201 calculates the zoom control speed Vz, which is the speed at which the zoom of the camera 100 is changed, by calculating the following equation 3.

[0073] Vz=(Sb-S)×Kz…Equation 3 Here, Sb is the reference subject size [Pixel] (the specified size described above), and Kz is a proportionality coefficient. In other words, the zoom control speed increases as the difference between the reference subject size and the size of the tracked subject increases, and decreases as the difference decreases. Furthermore, when the size of the tracked subject matches the reference subject size, the zoom control speed becomes zero. This enables zoom control that keeps the size of the tracked subject constant.

[0074] As with pan-tilt control, if you want to increase the zoom control speed, you can increase the proportionality coefficient Kz. Also, to prevent zoom control from being performed when the difference between the size of the tracked subject and the size of the reference subject is less than a threshold (i.e., when the difference is minute), CPU 201 may set the zoom control speed to zero when the difference is less than a threshold.

[0075] In step S411, the CPU 201 transmits a command to the camera 100 via the communication unit 204 inquiring about a set (PTZ values) of the current pan angle, the current tilt angle, the current horizontal imaging angle of view, and the current vertical imaging angle of view. The CPU 201 then receives the set transmitted from the camera 100 in response to the command via the communication unit 204, and stores the received set in the RAM 202.

[0076] The pan angle and tilt angle are angles based on a predetermined orientation of the camera 100 (hereinafter referred to as the forward orientation). In the following description, the current pan angle of the camera 100 will be expressed as PANcur [degrees], and the current tilt angle of the camera 100 will be expressed as TILTcur [degrees]. The current horizontal imaging angle of view of the camera 100 will be expressed as FOVh [degrees], and the current vertical imaging angle of view of the camera 100 will be expressed as FOVv [degrees].

[0077] In step S412, CPU 201 calculates the direction of the tracking subject as seen from camera 100 (tracking subject direction) based on the front direction of camera 100. An example of calculating the tracking subject direction will be described with reference to FIG. 6(a). CPU 201 calculates the horizontal shooting angle of view FOVhp per shooting pixel approximately by calculating the following equation 4.

[0078] FOVhp=FOVh / W … Equation 4 Here, it is assumed that position 631 of the tracking subject is horizontally separated by Wdiff [Pixel] from the center position (target position 650) of the captured image 600. In this case, CPU 201 calculates the direction of the tracking subject in the pan direction, that is, the angle PANtarget [degrees] of the pan direction of the tracking subject as seen from camera 100, using the front direction of camera 100 as a reference, by calculating Equation 5 below.

[0079] PANtarget=FOVhpxWdiff+PANcur … Formula 5 The CPU 201 then stores the calculated angle PANtarget in RAM 202. Similarly, the CPU 201 calculates FOVvt = FOVv / H to approximately calculate the vertical imaging angle of view FOVvt per imaging pixel. Here, if position 631 of the tracking subject is vertically separated by Hdiff [Pixels] from the center position of the captured image 600, the CPU 201 calculates TILTtarget = FOVvt x Hdiff + TILEcur to calculate the direction of the tracking subject in the tilt direction, that is, the angle TILTtarget [degrees] of the tilt direction of the tracking subject as seen from the camera 100, with the front orientation of the camera 100 as the reference. The CPU 201 then stores the calculated angle TILTtarget in RAM 202.

[0080] The method for calculating the direction of a tracking subject is not limited to the above method. For example, CPU 201 may calculate the direction of a tracking subject taking into consideration that the shooting direction of camera 100 changes spherically around camera 100 as a result of rotation of PTZ driving unit 107. An example of calculation in this case will be described with reference to FIG. 8.

[0081] 8(a) is a diagram showing an image 800 captured by camera 100, with the position of the tracked subject being point P. Point P is separated from the center of captured image 800 by Wdiff [Pixel] in the horizontal direction and Hdiff [Pixel] in the vertical direction. The angular difference between the center of captured image 800 and point P calculated using Equation 4 is assumed to be PANdiff [degree] and Tiltdiff [degree].

[0082] 8(b) shows a sphere 801 in a three-dimensional space with the position of camera 100 as the origin O, and with the radius being the distance from camera 100 to the subject shown in the captured image. The front of camera 100 is in the X-axis direction in the three-dimensional space, and when camera 100 is driven by panning, the captured image rotates around the Z-axis, and when camera 100 is driven by tilting, the captured image rotates around the Y-axis. Therefore, if captured image 800 is an image captured with camera 100 facing forward, captured image 800 is positioned so as to be perpendicular to the X-axis, as shown in FIG. 8(b).

[0083] First, CPU 201 calculates the three-dimensional coordinates (x, y, z) of position P of the subject to be tracked in the state shown in Fig. 8(b) using the following equations 6, 7, and 8. Note that in this process, only the direction of the subject to be tracked needs to be known, so the radius of spherical surface 801 can be any value, but for simplicity of calculation, the radius is set to 1 in the following calculations.

[0084] x=1 … Equation 6 y=tan(PANdiff) … Equation 7 z=tan(TILTdiff) … Equation 8 Since these are coordinates when camera 100 is facing in the X-axis direction, CPU 201 then converts the above three-dimensional coordinates (x, y, z) based on the direction in which camera 100 is facing. This calculation can be performed by rotating the coordinate axes in three-dimensional space using, for example, a well-known three-dimensional rotational coordinate conversion calculation. Specifically, if the current pan angle and tilt angle of camera 100 are PANcur and TILTcur, respectively, CPU 201 can calculate the three-dimensional coordinates (X, Y, Z) of the tracking subject after rotating the coordinate axes by calculating the following equation 9.

[0085]

number

[0086] 8(c) shows the three-dimensional coordinates (X, Y, Z) of the tracking subject, as well as the pan angle PANtarget and tilt angle TILTtarget calculated using equation 9. Finally, CPU 201 calculates the pan angle PANtarget and tilt angle TILTtarget of the tracking subject relative to the reference direction of camera 100 by calculating equations 10 and 11 below using the three-dimensional position (X, Y, Z) of the tracking subject.

[0087] PANtarget =arctan(Y / X) … Equation 10 TILTtarget=arctan(Z / √(X 2 +Y 2 )) … Equation 11 Using the above equations 6 to 11, the CPU 201 can calculate the direction of the subject to be tracked, taking into account the rotation of the PTZ driver 107. In step S413, the CPU 201 generates control commands for causing the PTZ driver 107 to change the pan angle of the camera 100 at a pan control speed Vp, change the tilt angle of the camera 100 at a tilt control speed Vt, and change the zoom of the camera 100 at a zoom control speed Vz. The CPU 201 then transmits the generated control commands to the camera 100 via the communication unit 204.

[0088] Here, it is assumed that the photographed image 600 in Fig. 6(b) is input after the photographed image 600 in Fig. 6(a) is input from the camera 100 to the workstation 200. The photographed image 600 in Fig. 6(b) is an image photographed in a state where the subject 601 in the photographed image 600 in Fig. 6(a) has moved to the right and the subject 602, which is closer to the camera 100 than the subject 601, has moved to the left, so that the subject 601 is hidden behind the subject 602.

[0089] When the captured image 600 in Fig. 6(a) is input to the workstation 200, the subject 601, which is the subject to be tracked, can be detected from the captured image 600, and therefore a rectangular frame 611 is displayed around the subject 601. However, when the captured image 600 in Fig. 6(b) is input to the workstation 200, the subject 601 is hidden behind the subject 602 in the captured image 600, and therefore the subject 601 cannot be detected from the captured image 600. Therefore, no rectangular frame is displayed around the subject 601.

[0090] Therefore, when the captured image 600 of Fig. 6(a) is input to the workstation 200, the process of step S408 is executed after the process of step S407. On the other hand, when the captured image 600 of Fig. 6(b) is input to the workstation 200, the process of step S414 is executed after the process of step S407.

[0091] In step S414, CPU 201 determines whether the length of the period during which the value of the presence flag is false (that is, the length of the period during which the tracking subject does not exist in the captured image) is equal to or longer than a predetermined time.

[0092] If, as a result of this determination, the length of the period during which the value of the presence flag is false is equal to or longer than the predetermined time, CPU 201 determines that it is difficult to redetect the tracking subject, and ends the processing according to the flowchart in Fig. 4. Note that, when ending the processing, camera 100 may perform control so that the shooting direction and shooting angle of view are set to the user's preset. In this case, the shooting direction and shooting angle of view information set by the user in advance may be recorded, and CPU 201 may communicate a control command to camera 100 before ending the processing. On the other hand, if the length of the period during which the value of the presence flag is false is less than the predetermined time, the processing proceeds to step S415.

[0093] Note that this "predetermined time" may be any time, may be a preset time, may be a time set by the user through operation of user input I / F 207, or may be a time dynamically changed by CPU 201. Furthermore, it is not necessary to provide a timeout after the predetermined time has elapsed. For example, CPU 201 may perform processing from step S415 onward upon detecting the occurrence of an event. The event is not limited to a specific event, and may be, for example, "the user inputs a specific instruction through operation of user input I / F 207."

[0094] In step S415, CPU 201 reads the direction of the subject to be tracked calculated in the most recently executed step S412 from RAM 202. For example, assume that captured image 600 of Fig. 6(a) was acquired in the previous step S402, and captured image 600 of Fig. 6(b) is acquired in the current step S402. In this case, CPU 201 acquires the direction of the subject to be tracked in "captured image 600 of Fig. 6(a)" acquired in the previous step S402.

[0095] In step S416, similarly to step S411, the CPU 201 transmits a command to the camera 100 to inquire about the set via the communication unit 204. The CPU 201 then receives the set transmitted from the camera 100 in response to the command via the communication unit 204, and stores the received set in the RAM 202.

[0096] In step S417, CPU 201 calculates the pan control speed Vp and the tilt control speed Vt by calculating the following equations 12 and 13, respectively, based on the tracking subject direction (PAN target and TILT target) read out in step S415 and the current pan angle PANcur and the current tilt angle TILTcur included in the set acquired in step S416.

[0097] Vp=(PANtarget -PANcur )xKp2 … Equation 12 Vt=(TILTtarget-TILTcur)xKt2 … Equation 13 Kp2 in Equation 12 and Kt2 in Equation 13 are both proportionality coefficients. By using Equations 12 and 13, the control speed increases as the angular difference between the direction of the tracking subject immediately before the tracking subject disappears from the captured image and the current shooting direction of camera 100 increases, and the control speed decreases as the angular difference decreases.

[0098] In this way, when the workstation 200 can detect the tracking target from the captured image, it calculates the direction toward the tracking target as the tracking target direction based on the detection result. On the other hand, when the workstation 200 cannot detect the tracking target from the captured image, it controls the control speed of the shooting direction based on the tracking target direction calculated in the past.

[0099] Graph 711 shown in FIG. 7(b) is a graph showing the relationship between the "angle difference between the angle of the pan direction of the tracked subject as seen from camera 100 and the current pan angle of camera 100" and the "pan control speed" as shown in Equation 12. The horizontal axis represents the "angle difference between the angle of the pan direction of the tracked subject as seen from camera 100 and the current pan angle of camera 100," and the vertical axis represents the "pan control speed." A higher pan control speed can be achieved by increasing the proportionality coefficient Kp2. Note that when the "angle difference between the angle of the pan direction of the tracked subject as seen from camera 100 and the current pan angle of camera 100" is less than a threshold (i.e., when the angle difference is small), the pan control speed may be set to zero so that pan control is not performed. In this case, the CPU 201 may calculate the pan control speed using a formula such that the pan control speed becomes zero in the section where the "angle difference between the angle of the pan direction of the tracked subject as seen from the camera 100 and the current pan angle of the camera 100" is equal to or less than a threshold value, as shown in graph 712 in Figure 7(b).

[0100] As described above, step S417 uses the direction of the tracked subject calculated in the most recent execution of step S412. Therefore, step S412 may calculate the direction of the tracked subject taking into account the difference between the control cycles of steps S412 and S417. For example, CPU 201 may perform a motion prediction process for the tracked subject in step S412 to predict the position of the tracked subject in the next cycle, and then perform a similar calculation based on the difference between the predicted position of the tracked subject and the current shooting direction to calculate the direction of the tracked subject. Performing such prediction processing makes it possible to calculate the pan / tilt speed based on the position of the tracked subject at the time the pan / tilt speed is actually calculated.

[0101] As another example of calculation of the direction of a tracked subject, the CPU 201 may correct the direction of a tracked subject in consideration of the subject detection characteristics of the inference unit 206. Fig. 9 shows a diagram for explaining an example of the subject detection characteristics of the inference unit 206. Fig. 9 shows a state in which a subject 901 in a captured image 900 moves to the right and is blocked by an obstruction 960.

[0102] Since the subject 901 can be detected from the photographed image 900 in Fig. 9(a), a rectangular frame 911 of the subject 901, which is the detection result of the subject 901 by the inference unit 206, is superimposed on the photographed image 900. The position 931 is the center coordinate of the rectangular frame 911 of the subject 901 in the photographed image 900 in Fig. 9(a), and the width 941 is the horizontal length (number of pixels) of the rectangular frame 911.

[0103] If the inference unit 206 has the characteristic of being unable to detect a subject when the subject is obstructed even slightly, then in the captured image 900 of Figures 9(b) and (c), part of the subject 901 is obstructed by an obstructing object 960, and therefore the subject 901 is not detected from the captured image 900, and therefore the rectangular frame of the subject 901 is not superimposed on the captured image 900.

[0104] In the above description, if the captured image 900 of FIG. 9(a) was acquired in the previous step S402 and the captured image 900 of FIG. 9(b) was acquired in the current step S402, then in the current step S417, the CPU 201 calculates the control speed of the capturing direction so as to approach the direction toward position 931 (the direction of the subject to be tracked calculated based on the captured image 900 of FIG. 9(a)).

[0105] Therefore, in step S412, the CPU 201 may correct the calculated direction of the tracking subject to a direction shifted by width 941 in the direction of pan control speed Vp. The position on the captured image of the direction resulting from such correction is shown as position 932 in FIG. 9(c). Similar to FIG. 9(b), the captured image 900 in FIG. 9(c) is a captured image acquired in step S402 after the captured image 900 in FIG. 9(a).

[0106] As a result, if the captured image 900 of Figure 9(c) is acquired in this step S402, in this step S917, the capturing direction of the camera 100 can be determined based on the difference between the direction of the tracked subject corresponding to the position 932 and the current pan angle of the camera 100.

[0107] The amount of correction for the direction of the tracked subject does not necessarily have to be determined according to the size of the rectangular frame of the tracked subject, but may be a predetermined amount determined experimentally in advance. By correcting the direction of the tracked subject in this way, it becomes possible to suppress the influence of the subject detection characteristics of the inference unit 206 and to direct the pan / tilt in a direction closer to the direction in which the tracked subject was actually occluded.

[0108] Furthermore, the time used for comparison with the "predetermined time" in step S414 is not limited to the "length of time during which the value of the presence flag is false (i.e., the length of time during which the tracking subject is not present in the captured image)." For example, timing may be started at the timing when the PTZ of camera 100 stops, pointing in the direction of the tracking subject immediately before the tracking subject became unable to be identified by the processing of step S417. In this case, in step S414, CPU 201 may compare the duration of the "state in which camera 100 faces the direction of the tracking subject immediately before the tracking subject became unable to be identified, and the PTZ of camera 100 is stopped" with the predetermined time. By performing such control, the timeout is determined based on the elapsed time since pan / tilt stopped, making it easier for the user to understand that a timeout has occurred.

[0109] In this way, according to this embodiment, even if the tracking subject can no longer be identified from the captured image, pan / tilt control can be continued so that the shooting direction of camera 100 is directed in the direction in which the tracking subject was previously located.

[0110] If the camera operation is stopped immediately after the subject can no longer be detected, the pan / tilt will come to an abrupt halt, resulting in poor quality of the captured image. Therefore, by continuing pan / tilt control as in this embodiment, the quality of the captured image during tracking photography can be improved.

[0111] Furthermore, in tracking photography, the subject is detected and the camera is driven based on the detection results, so in principle, the camera drive lags behind the movement of the subject. Therefore, if the camera drive is stopped immediately after the subject can no longer be detected, the pan / tilt operation will stop even though the tracked subject has not yet reached the target position. However, by performing processing such as in this embodiment, it is possible to stop the pan / tilt movement when the pan / tilt movement catches up to the position where the tracked subject is hidden.

[0112] Furthermore, in cases where the tracked subject is hidden behind an object, it is highly likely that the subject will reappear from the point where it was hidden. In other words, by stopping pan / tilt at the position where the tracked subject has hidden, as in this embodiment, it is possible to stop pan / tilt at a position where it is easy to detect the tracked subject reappearing.

[0113] The process for calculating the control speed described in the first embodiment may be performed by the camera 100, the workstation 200, or a device separate from these devices. In other words, the control device that executes the process for calculating the control speed may be incorporated into the camera 100, the workstation 200, or may exist as a device separate from these devices. Furthermore, such a control device may be configured with hardware, software, or a combination of hardware and software.

[0114] [Second embodiment] In this embodiment, differences from the first embodiment will be described, and unless otherwise specified below, it will be assumed that the present embodiment is the same as the first embodiment. In this embodiment, a camera 1000 will be described that has the functions of the camera 100 and can execute the processing of the workstation 200 (processing according to the flowchart in FIG. 4). An example of the hardware configuration of the camera 1000 according to this embodiment will be described using the block diagram in FIG. 10.

[0115] The CPU 1001, RAM 1002, storage device 1003, imaging unit 1004, and PTZ driving unit 1007 are similar to the CPU 101, RAM 102, storage device 103, imaging unit 104, and PTZ driving unit 107 in FIG. 1, respectively, and therefore description thereof will be omitted.

[0116] The communication unit 1006 performs data communication with the outside via a network such as a LAN or the Internet. Note that the method of data communication with the outside by the communication unit 1006 is not limited to a specific data communication method.

[0117] The inference unit 1009 is similar to the inference unit 206, except that it acquires a captured image from the imaging unit 1004. The user input I / F 1010 is a user interface such as a button, switch, lever, or touch panel screen, and can be operated by the user to input various information and instructions to the camera 1000.

[0118] The CPU 1001, RAM 1002, storage device 1003, imaging unit 1004, communication unit 1006, PTZ driving unit 1007, inference unit 1009, and user input I / F 1010 are connected to a system bus 1008. Each unit of the camera 1000 shown in Fig. 10 is driven by, for example, power obtained by rectifying externally supplied AC power to a predetermined voltage, or power supplied from a built-in battery (not shown).

[0119] Next, tracking and photographing of a tracking subject by camera 1000 will be described with reference to the flowchart of Fig. 11. The flowchart of Fig. 11 is a flowchart of processing executed in response to CPU 1001 detecting an instruction to perform tracking and photographing input by a user operating user input I / F 1010. As with the first embodiment, the method of inputting the instruction to perform tracking and photographing is not limited to a specific input method.

[0120] In step S1101, the CPU 1001 determines whether or not a command (end command) instructing the end of the processing according to the flowchart of Fig. 11 has been acquired via the communication unit 1006 or the user input I / F 1010. If the result of this determination is that an end command has been acquired, the processing according to the flowchart of Fig. 11 ends, and if an end command has not been acquired, the processing proceeds to step S1102. In step S1102, the CPU 1001 acquires a captured image output from the imaging unit 1004 and stores the acquired captured image in the RAM 1002.

[0121] In step S1103, the CPU 1001 inputs the captured image stored in the RAM 1002 in step S1102 to the inference unit 1009. The inference unit 1009 performs subject detection processing by performing the same processing as that performed by the inference unit 206 on the input captured image, and outputs rectangular frame information that defines a rectangular frame that includes the entire body (human body) of the subject detected from the captured image.

[0122] In step S1105, CPU 1001 determines whether a tracking subject has been designated. If the result of this determination is that a tracking subject has been designated, the process proceeds to step S1106, and if a tracking subject has not been designated, the process proceeds to step S1101. As in the first embodiment, the method for designating a tracking subject is not limited to a specific method.

[0123] In step S1106, CPU 1001 performs the same processing as in step S406 above, thereby performing identification processing to identify the tracking subject from the subjects detected in step S1103. If CPU 1001 is able to identify the tracking subject from the subjects detected in step S1103, it sets the value of the presence flag stored in RAM 1002 to true, and stores in RAM 1002 the rectangular frame information of the tracking subject from the rectangular frame information output in step S1103.

[0124] On the other hand, if the CPU 1001 is unable to identify the tracking subject from the subjects detected in step S1103, it sets the value of the existence flag stored in the RAM 1002 to false.

[0125] Since the first step S1106 is the first step S1106 after the tracking subject is designated, the CPU 1001 sets the value of the presence flag to true and stores rectangular frame information of the designated tracking subject in the RAM 1002. In step S1106 from the second time onwards, the CPU 201 performs the same processing as in step S406 from the second time onwards.

[0126] In step S1107, CPU 1001 determines whether the value of the existence flag is true or false. If the result of this determination is that the value of the existence flag is true, the process proceeds to step S1108, and if the value of the existence flag is false, the process proceeds to step S1114.

[0127] In step S1108, CPU 1001 reads rectangular frame information of the tracking subject stored in RAM 1002. In step S1109, CPU 1001 performs the same process as in step S409 above to calculate the speed at which camera 100 changes the shooting direction (pan angle, tilt angle) to track and shoot the tracking subject.

[0128] In step S1110, CPU 1001 performs the same process as in step S410 above to calculate the speed at which the zoom is changed so that camera 100 can track and photograph the subject being tracked.

[0129] In step S1111, the CPU 1001 acquires a set of the current pan angle of the camera 100, the current tilt angle of the camera 100, the current horizontal imaging angle of view of the camera 100, and the current vertical imaging angle of view of the camera 100, for example, from the PTZ driving unit 1007. Then, the CPU 101 stores the acquired set in the RAM 1002.

[0130] In step S1112, CPU 1001 calculates the direction of the subject being tracked by performing the same process as in step S412 above. In step S1114, CPU 1001 determines whether the length of the period during which the value of the presence flag is false (i.e., the length of the period during which the subject being tracked does not exist in the captured image) is equal to or longer than a predetermined time. If the result of this determination is that the length of the period during which the value of the presence flag is false (i.e., the length of the period during which the subject being tracked does not exist in the captured image) is equal to or longer than the predetermined time, the processing according to the flowchart in Fig. 11 ends. On the other hand, if the length of the period during which the value of the presence flag is false (i.e., the length of the period during which the subject being tracked does not exist in the captured image) is shorter than the predetermined time, the processing proceeds to step S1115.

[0131] In step S1115, CPU 1001 reads the direction of the subject to be tracked calculated in the most recent step S1112 from RAM 1002. In step S1116, CPU 1001 performs the same process as in step S1111 above to acquire the set, and stores the acquired set in RAM 1002. In step S1117, CPU 1001 performs the same process as in step S417 above to calculate pan control speed Vp and tilt control speed Vt.

[0132] In step S1118, the CPU 1001 generates control commands for causing the PTZ driving unit 1007 to change the pan angle of the camera 1000 at a pan control speed Vp, change the tilt angle of the camera 1000 at a tilt control speed Vt, and change the zoom of the camera 1000 at a zoom control speed Vz. The CPU 1001 then performs the same processing as in step S302 above to obtain the PTZ control speed of the camera 1000 from the generated control command, and obtains driving parameters based on the PTZ control speed.

[0133] In step S1119, the CPU 1001 performs the same process as in step S303 above, thereby controlling the PTZ driving unit 1007 based on the driving parameters acquired in step S1118.

[0134] In the first and second embodiments, the case where the subject to be detected is a person has been described. However, the attribute of the subject to be detected is not limited to a person, and the subject may be an object with any attribute.

[0135] The numerical values, processing timing, processing order, processing subject, data (information) configuration / acquisition method / sending destination / sending source / storage location, etc. used in each of the above embodiments are given as examples to provide a concrete explanation, and are not intended to be limited to these examples.

[0136] In addition, some or all of the above-described embodiments may be used in appropriate combination, and some or all of the above-described embodiments may be selectively used.

[0137] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0138] The invention of this specification includes the following control device, control method, and computer program. (Item 1) A control device characterized by comprising a control means for, when a tracking target can be detected from a captured image, calculating the direction toward the tracking target as the tracking target direction based on the detection result, and, when a tracking target cannot be detected from the captured image, controlling the control speed of the shooting direction based on the tracking target direction calculated in the past. (Item 2) The control means When a tracking target is detected from a captured image, the control device calculates the direction of the tracking target based on the difference between the position of the tracking target detected from the captured image and the target position, and the current shooting direction and zoom. (Item 3) The control means 3. The control device according to item 1 or 2, characterized in that when a tracking target can be detected from a captured image, the control speed of the shooting direction is calculated based on the difference between the position of the tracking target detected from the captured image and the target position. (Item 4) The control means When a tracking target can be detected from a captured image, the control device according to any one of items 1 to 3 calculates a zoom control speed based on the difference between the size of the tracking target detected from the captured image and a specified size. (Item 5) The control means Item 1. A control device characterized in that, if the tracking target cannot be detected from the captured image, the shooting direction is changed if the tracking target cannot be detected for a predetermined period of time after the control speed of the shooting direction becomes zero. (Item 6) The control means 5. A control device according to any one of items 1 to 4, characterized in that, when a tracking target cannot be detected from a captured image, the control speed of the shooting direction is calculated based on the difference between the current shooting direction and the tracking subject direction calculated in the past. (Item 7) The control means 5. The control device according to any one of items 1 to 4, characterized in that, when a tracking target cannot be detected from a captured image, the control speed of the shooting direction is calculated based on the difference between the current shooting direction and the tracking target direction predicted based on the tracking target direction calculated in the past. (Item 8) The control means 5. The control device according to any one of items 1 to 4, characterized in that, when a tracking target cannot be detected from a captured image, the control speed of the shooting direction is calculated based on the difference between the current shooting direction and the tracking target direction calculated in the past and corrected based on the size of the tracking target. (Item 9) moreover, means for acquiring an image captured by an imaging device; a transmitting means for generating a command for changing the photographing direction at a control speed controlled by the control means and transmitting the generated command to the photographing device; 9. The control device according to any one of items 1 to 8, comprising: (Item 10) moreover, An imaging means for capturing an image; a driving means for changing the photographing direction at a control speed controlled by the control means; 9. The control device according to any one of items 1 to 8, comprising: (Item 11) A control method characterized by the fact that, when a tracking target can be detected from a captured image, the direction toward the tracking target is calculated as the tracking target direction based on the detection result, and, when a tracking target cannot be detected from a captured image, the control speed of the shooting direction is controlled based on the tracking target direction calculated in the past. (Item 12) A computer program for causing a computer to function as each means of the control device according to any one of items 1 to 9.

[0139] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0140] 10: Person 100: Camera 200: Workstation 300: Network 400: Video cable

Claims

1. A control device characterized by comprising a control means for, when a tracking target can be detected from a captured image, calculating the direction toward the tracking target as the tracking target direction based on the detection result, and, when a tracking target cannot be detected from the captured image, controlling the control speed of the shooting direction based on the tracking target direction calculated in the past.

2. The control means 2. The control device according to claim 1, wherein, when a tracking target is detected from a captured image, the direction of the tracking target is calculated based on the difference between the position of the tracking target detected from the captured image and a target position, and the current shooting direction and zoom.

3. The control means 2. The control device according to claim 1, wherein, when a tracking target is detected from a captured image, a control speed of the shooting direction is calculated based on a difference between the position of the tracking target detected from the captured image and a target position.

4. The control means 2. The control device according to claim 1, wherein, when a tracking target is detected from a captured image, the control speed of the zoom is calculated based on the difference between the size of the tracking target detected from the captured image and a specified size.

5. The control means The control device described in claim 1, characterized in that if the tracking target cannot be detected from the captured image, the shooting direction is changed if the time during which the tracking target cannot be detected continues for a predetermined period of time after the control speed of the shooting direction becomes zero.

6. The control means The control device according to claim 1, characterized in that, when a tracking target cannot be detected from the captured image, the control speed of the shooting direction is calculated based on the difference between the current shooting direction and the tracking target direction calculated in the past.

7. The control means 2. The control device according to claim 1, wherein, when a tracking target cannot be detected from the captured image, a control speed of the shooting direction is calculated based on the difference between the current shooting direction and the direction of the tracking target predicted based on the direction of the tracking target calculated in the past.

8. The control means 2. The control device according to claim 1, wherein, when a tracking target cannot be detected from a captured image, a control speed of the shooting direction is calculated based on a difference between the current shooting direction and a tracking target direction calculated in the past and corrected based on the size of the tracking target.

9. moreover, means for acquiring an image captured by an imaging device; a transmitting means for generating a command for changing the photographing direction at a control speed controlled by the control means and transmitting the generated command to the photographing device; The control device according to claim 1 , further comprising:

10. moreover, An imaging means for capturing an image; a driving means for changing the photographing direction at a control speed controlled by the control means; The control device according to claim 1 , further comprising:

11. A control method characterized by the fact that, when a tracking target can be detected from a captured image, the direction toward the tracking target is calculated as the tracking target direction based on the detection result, and, when a tracking target cannot be detected from a captured image, the control speed of the shooting direction is controlled based on the tracking target direction calculated in the past.

12. A computer program for causing a computer to function as each of the means of the control device according to any one of claims 1 to 9.

Images