Control apparatus, method, computer readable storage medium, and computer program product

Abstract

The invention provides a control apparatus, a method, a computer readable storage medium, and a computer program product. If the tracking object can be detected from the captured image, the control device calculates a direction toward the tracking object as a tracking subject direction based on a detection result, and if the tracking object cannot be detected from the captured image, controls a control speed in the camera direction based on the previously calculated tracking subject direction.

Description

Technical Field

[0001] The present disclosure generally relates to control devices, methods, computer-readable storage media, and computer program products, and more particularly to camera control technology. Background Art

[0002] In recent years, with the development of artificial intelligence (AI), automatic tracking technology has been proposed that detects a subject in a video captured by a camera and controls the pan and tilt of the camera to track the subject.

[0003] There are cases where the tracking subject may temporarily disappear from the camera's field of view when another subject passes in front of the subject being tracked (the tracking subject), or when the tracking subject moves and hides behind an obstacle such as a wall. If the tracking subject is not visible in the captured video, the tracking subject cannot be detected from the captured video.

[0004] In automatic tracking, if pan / tilt control is stopped immediately when a subject cannot be detected temporarily, the movement of the camera in the pan and tilt directions will stop abruptly, which leads to degradation of the captured video.

[0005] Japanese Patent Laid-Open No. 2012-80221 discloses a technology in which, for example, when a tracking target is blocked by the blocker, the blocker is set as a new tracking target.

[0006] When the tracking target is obscured by another person moving in the opposite direction of the tracking target, such as when a person passes by, setting the person in the foreground as the new tracking target will cause the tracking target's movement direction to suddenly reverse. As a result, sudden changes in pan / tilt control may occur, and the tracking quality of the captured video may deteriorate. Summary of the Invention

[0007] The present disclosure provides techniques for reducing the occurrence of sudden changes in camera direction when a tracking object can no longer be detected in a captured image.

[0008] According to a first aspect of the present disclosure, a control device includes: a memory storing a program; and a processor configured to, when executing the program, cause the control device to: calculate a direction toward a tracking object as a tracking subject direction based on a detection result when a tracking object can be detected from a captured image; and control a control speed in a camera direction based on a previously calculated tracking subject direction when the tracking object cannot be detected from the captured image.

[0009] According to a second aspect of the present disclosure, a control method includes: when a tracking object can be detected from a captured image, calculating a direction toward the tracking object as a tracking subject direction based on a detection result; and when the tracking object cannot be detected from the captured image, controlling a control speed in a camera direction based on the previously calculated tracking subject direction.

[0010] According to a third aspect of the present disclosure, a computer-readable storage medium stores a computer program for causing a computer to execute a method, the method comprising: when a tracking object can be detected from a captured image, calculating a direction toward the tracking object as a tracking subject direction based on a detection result; and when the tracking object cannot be detected from the captured image, controlling a control speed in a camera direction based on a previously calculated tracking subject direction.

[0011] According to a fourth aspect of the present disclosure, a computer program product includes a computer program for causing a computer to execute a method, the method comprising: when a tracking object can be detected from a captured image, calculating a direction toward the tracking object as a tracking subject direction based on a detection result; and when the tracking object cannot be detected from the captured image, controlling a control speed in a camera direction based on a previously calculated tracking subject direction.

[0012] Further features of the present disclosure will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a diagram showing a system configuration example.

[0014] Figure 2 A block diagram showing an example of the hardware configuration of a camera and a workstation.

[0015] Figure 3 This is a flowchart of processing performed when the camera tracks a tracking subject and captures an image of the tracking subject.

[0016] Figure 4 This is a flowchart of processing performed when a workstation causes a camera to track a tracking subject and capture an image of the tracking subject.

[0017] Figure 5 It shows Figure 4 FIG. 4 is a diagram showing an example of a display in step S404.

[0018] Figure 6A is a diagram showing an example of a display screen of a display unit during subject tracking.

[0019] Figure 6Bis a diagram showing an example of a display screen of a display unit during subject tracking.

[0020] Figure 7A This is a graph showing the relationship between the horizontal distance between positions and the control speed.

[0021] Figure 7B _ is a graph showing the relationship between the angle difference between the angle of the tracking object in the pan direction as viewed from the camera and the current pan angle of the camera, and the pan control speed.

[0022] Figure 8A A diagram showing a captured image obtained by a camera.

[0023] Figure 8B 3 is a diagram showing a sphere whose radius is the distance from the camera to the subject in the captured image.

[0024] Figure 8C 3D coordinates (X, Y, Z) of a tracking object, as well as a pan direction tracking angle and a pitch direction angle are shown.

[0025] Figure 9 A is a diagram showing an example of subject detection characteristics of the inference unit.

[0026] Figure 9 B is a diagram showing an example of subject detection characteristics of the inference unit.

[0027] Figure 9 C is a diagram showing an example of subject detection characteristics of the inference unit.

[0028] Figure 10 is a block diagram showing an example of the hardware configuration of a camera.

[0029] Figure 11 This is a flowchart performed when the camera tracks a tracking subject and captures an image of the tracking subject. DETAILED DESCRIPTION

[0030] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the present disclosure. In the embodiments, a plurality of features are described, not all of which are required, and a plurality of such features can be appropriately combined. In the accompanying drawings, the same reference numerals are given to the same or similar configurations, and redundant descriptions thereof are omitted.

[0031] First embodiment

[0032] First, refer to Figure 1 An example of a configuration of a system according to this embodiment is described. Figure 1As shown, the system according to the present embodiment includes a camera 100 serving as an imaging device and a workstation 200 serving as a control device for controlling the operation of the camera 100, wherein the camera 100 tracks a person 10 as a subject to be tracked (tracking subject) and captures an image of the person 10.

[0033] In the system according to the present embodiment, the camera 100 and the workstation 200 are connected to a network 300 such as a LAN or the Internet, and are configured to perform data communication with each other.

[0034] The workstation 200 can control the operation of the camera 100 by sending a distribution request command for requesting distribution of captured images and a setting command for setting various parameters to the camera 100 via the network 300. The camera 100 distributes the captured images to the workstation 200 in response to the distribution request command received from the workstation 200, and stores various parameters in response to the setting command received from the workstation 200. The workstation 200 can also control the pan, tilt, and zoom (hereinafter referred to as PTZ) of the camera 100 by sending a command for controlling the PTZ to the camera 100.

[0035] Any connection format or communication protocol between the camera 100 and the workstation 200 may be used as long as the connection format or communication protocol enables data communication between the camera 100 and the workstation 200. For example, the camera 100 and the workstation 200 may be directly connected to each other using a serial communication cable instead of a network.

[0036] In the system according to the present embodiment, the camera 100 and the workstation 200 are connected to each other via the video cable 400 , and a captured image captured by the camera 100 is transmitted to the workstation 200 via the video cable 400 .

[0037] The method for transmitting the captured image from the camera 100 to the workstation 200 is not limited to a specific method. For example, the camera 100 may transmit the captured image to the workstation 200 via the aforementioned network 300. In other words, the configuration for transmitting and receiving various information such as commands and captured images between the camera 100 and the workstation 200 is not limited to a specific configuration.

[0038] Next, we will refer to Figure 2 The block diagram in describes an example of the respective hardware configurations of the camera 100 and the workstation 200. Figure 2 The illustrated hardware configurations are merely examples of hardware configurations respectively applicable to the camera 100 and the workstation 200 and may be appropriately deformed / changed.

[0039] First, the configuration of camera 100 will be described. Camera 100 tracks a tracking subject and captures images of the tracking subject. If camera 100 captures moving images, camera 100 transmits images of frames of the moving images as captured images to workstation 200. If camera 100 captures still images, camera 100 transmits still images as captured images to workstation 200.

[0040] The CPU 101 executes various types of processing using computer programs and data stored in the RAM 102. The CPU 101 controls the overall operation of the camera 100, and executes or controls various types of processing performed by the camera 100.

[0041] The RAM 102 has an area for storing images captured by the imaging unit 104 and an area for storing computer programs and data loaded from the storage device 103. The RAM 102 also has an area for storing various types of information received from the workstation 200 by the communication unit 106, and a work area used when the CPU 101 executes various types of processing. This enables the RAM 102 to provide various types of areas as appropriate.

[0042] The storage device 103 is a nonvolatile storage device such as a flash memory, an HDD, an SSD, or an SD card. The storage device 103 stores configuration data of the camera 100, computer programs and data used when starting the camera 100, and computer programs and data related to the basic operation of the camera 100. The storage device 103 also stores computer programs and data for causing the CPU 101 to execute or control various types of processing performed by the camera 100.

[0043] The imaging unit 104 includes an optical system, an image sensor that converts light generated by the optical system into electric charge (image signal), and an image processing circuit that generates a captured image based on the image signal. For example, a complementary metal oxide semiconductor (CMOS) image sensor can be used as the image sensor. A charge coupled device (CCD) image sensor can also be used as the image sensor.

[0044] The video output unit 105 is an interface for transmitting the captured image captured by the camera unit 104 to the workstation 200 via the video cable 400. The video output unit 105 includes, for example, a serial digital interface (SDI) and a high-definition multimedia interface.

[0045] The communication unit 106 is an interface for performing data communication with the workstation 200 via the network 300. The PTZ drive unit 107 controls the imaging direction (pan angle and pitch angle) of the camera 100 (imaging unit 104) and the field of view (zoom) of the camera 100 (imaging unit 104) based on a control command received from the workstation 200.

[0046] The CPU 101 , the RAM 102 , the storage device 103 , the imaging unit 104 , the video output unit 105 , the communication unit 106 , and the PTZ drive unit 107 are all connected to a system bus 108 . Figure 2 The illustrated constituent components of the camera 100 are driven by electric power obtained by rectifying AC power supplied from an external power source into a predetermined voltage, or by electric power supplied from a built-in battery (not illustrated).

[0047] The configuration of the workstation 200 will now be described. The workstation 200 serves as an example of a computer device such as a personal computer (PC), a smartphone, or a tablet terminal, etc. Any computer device may be used as long as it can realize the functions associated with the workstation 200.

[0048] The CPU 201 executes various types of processing using computer programs and data stored in the RAM 202. The CPU 201 controls the overall operation of the workstation 200, and executes or controls various types of processing performed by the workstation 200.

[0049] The RAM 202 has an area for storing computer programs and data loaded from the storage device 203 and an area for storing various types of information received from the camera 100 via the communication unit 204. The RAM 202 also has an area for storing captured images received from the camera 100 by the video input unit 205. The RAM 202 has a work area used when the CPU 201 and the inference unit 206 execute various types of processing. This enables the RAM 202 to provide various types of areas as appropriate.

[0050] The storage device 203 is a nonvolatile storage device such as a flash memory, an HDD, an SSD, or an SD card. The storage device 203 stores configuration data of the workstation 200, computer programs and data used when starting the workstation 200, and computer programs and data related to the basic operation of the workstation 200. The storage device 203 also stores computer programs and data for causing the CPU 201 and the inference unit 206 to execute or control various types of processing described as processing performed by the workstation 200.

[0051] The communication unit 204 is an interface for performing data communication with the camera 100 via the network 300. The video input unit 205 is an interface for receiving a captured image transmitted from the camera 100 via the video cable 400, and includes, for example, SDI or HDMI.

[0052] The inference unit 206 detects objects from the input image and outputs the position and size of the objects in the image. The inference unit 206 is, for example, an arithmetic device dedicated to image processing and inference processing, such as a graphics processing unit (GPU). While GPUs are generally advantageous for inference processing, reconfigurable logic circuits such as field programmable gate arrays (FPGAs) can also be used to implement the same functionality. The processing of the inference unit 206 can also be performed by the CPU 201.

[0053] The user input I / F 207 is a user interface such as a keyboard, a mouse, or a touch panel, and the user can operate the user input I / F 207 to input various types of instructions and information to the workstation 200 .

[0054] The display unit 208 includes a liquid crystal screen or a touch panel screen, and can display processing results obtained by the CPU 201 and the inference unit 206 as images, text, etc. The display unit 208 may be a projection device such as a projector that projects images and text.

[0055] Although the present embodiment describes an example in which the workstation 200 includes the display unit 208, the present disclosure is not limited to this configuration. For example, a configuration in which the workstation 200 does not include the display unit 208 and a display monitor connected to the workstation 200 is also possible.

[0056] Next, we will refer to Figure 3 The flowchart in describes processing performed by the camera 100 to track a tracking subject and capture an image of the tracking subject. Figure 3 The flowchart is a flowchart of processing executed in response to the camera 100 (CPU 101) detecting the receipt of a control command transmitted from the workstation 200.

[0057] In step S301 , the CPU 101 receives a control command transmitted from the workstation 200 via the communication unit 106 , and stores the received control command in the RAM 102 .

[0058] In step S302, the CPU 101 acquires the PTZ control speed of the camera 100 from the control command stored in the RAM 102 in step S301. The PTZ control speed includes the speed of changing the pan angle (angle in the pan direction) of the camera 100, the speed of changing the pitch angle (angle in the pitch direction) of the camera 100, and the speed of changing the zoom of the camera 100.

[0059] Based on the acquired PTZ control speed, the CPU 101 acquires drive parameters for changing the pan angle of the camera 100 at a specified speed, changing the pitch angle of the camera 100 at a specified speed, and changing the zoom of the camera 100 at a specified speed. Specifically, the CPU 101 acquires drive parameters for driving and controlling the various pan / tilt motors included in the PTZ drive unit 107, as well as drive parameters for driving and controlling the motors of the zoom drive unit included in the PTZ drive unit 107. For example, the CPU 101 can acquire drive parameters corresponding to the PTZ control speed acquired from the control command from a "table in which drive parameters corresponding to various PTZ control speeds are pre-registered." Therefore, the method of acquiring drive parameters based on the control command is not limited to a specific method.

[0060] In step S303, the CPU 101 controls the PTZ drive unit 107 based on the drive parameters acquired in step S302. Thus, the camera 100 can perform pan, tilt, and zoom operations (operations for changing the pan angle, tilt angle, and zoom) based on control commands from the workstation 200.

[0061] Next, we will refer to Figure 4 The flowchart in describes processing performed by the workstation 200 to cause the camera 100 to track a tracking subject and capture an image of the tracking subject. Figure 4 The flowchart of FIG201 is a flowchart of processing executed in response to the CPU 201 detecting a tracking / photographing execution instruction input by the user via an operation of the user input I / F 207. The tracking / photographing execution instruction may be input by an external device. In this case, Figure 4 The flowchart of FIG201 will be a flowchart of processing executed in response to the CPU 201 detecting that a tracking / photographing execution instruction transmitted from an external device has been received via the communication unit 204. Therefore, the method for inputting the tracking / photographing execution instruction is not limited to a specific input method. Figure 4 The trigger of the process of the flowchart is not limited to a specific trigger.

[0062] In step S401, the CPU 201 determines whether a command for ending the operation according to the present invention has been obtained by the communication unit 204 or the user input I / F 207. Figure 4If it is determined that the end command has been obtained, then Figure 4 The processing of the flowchart in ends. If it is determined that the end command has not been obtained, the process moves to step S402.

[0063] In step S402, the CPU 201 receives the captured image transmitted from the camera 100 via the video input unit 205, and stores the received captured image in the RAM 202. In step S403, the CPU 201 inputs the captured image stored in the RAM 202 in step S402 to the inference unit 206. The inference unit 206 performs subject detection processing on the input captured image.

[0064] The subject detection processing performed by the inference unit 206 will be described. The inference unit 206 inputs the input captured image to a learning model created using a machine learning method such as deep learning, performs arithmetic processing on the learning model, and outputs information (rectangular frame information) defining a rectangular frame that includes the entire body (human body) of each subject detected in the captured image. For example, the inference unit 206 outputs information indicating the center coordinates of the rectangular frame and the size (height and width) of the rectangular frame as the rectangular frame information.

[0065] The rectangular frame information is not limited to this, and may be, for example, the coordinates of the upper left and lower right vertices of the rectangular frame. The rectangular frame is not limited to a frame including the entire body of the subject, but may be a rectangular frame including a part of the subject (for example, a person's head or face). In this case, the learning model to be used only needs to be replaced by a learning model that outputs rectangular frame information for defining a rectangular frame including the desired area. The subject detection processing performed by the inference unit 206 is not limited to a method using a learning model. For example, the inference unit 206 may use a template matching method in which a template image of the subject is pre-registered in the storage device 203 or the like and an area in the captured image having a high similarity to the template image is detected as an area of the subject.

[0066] In step S404 , the CPU 201 superimposes a rectangular frame corresponding to the rectangular frame information on the captured image stored in the RAM 202 in step S402 , and displays the resultant image on the display unit 208 . Figure 5 An example of the display in step S404 is shown in FIG.

[0067] exist Figure 5In the display example of FIG, a rectangular frame 511 corresponding to the rectangular frame information of the subject 501 and a rectangular frame 512 corresponding to the rectangular frame information of the subject 502 are superimposed and displayed on a captured image 500 including a subject 501 and a subject 502. The size of the rectangular frame 511 is the size indicated by the rectangular frame information of the subject 501, and the center coordinates of the rectangular frame 511 are the center coordinates indicated by the rectangular frame information of the subject 501. The size of the rectangular frame 512 is the size indicated by the rectangular frame information of the subject 502, and the center coordinates of the rectangular frame 512 are the center coordinates indicated by the rectangular frame information of the subject 502.

[0068] In step S405, the CPU 201 determines whether any tracking subject has been specified. If it is determined that any tracking subject has been specified, the process moves to step S406. If it is determined that no tracking subject has been specified, the process moves to step S401.

[0069] The method for specifying the tracking subject is not limited to a specific method. Figure 5 In the example of , when the user operates the mouse serving as the input I / F 207 to perform a click operation with the mouse cursor superimposed on the rectangular frame 511, the CPU 201 can specify the subject 501 corresponding to the rectangular frame 511 as a tracking subject.

[0070] For example, the CPU 201 may also designate the subject corresponding to the rectangular frame information indicating the largest size from among the subjects detected in the captured image as the tracking subject. The CPU 201 may also designate the subject corresponding to the rectangular frame information indicating the center coordinates closest to the center position of the captured image as the tracking subject.

[0071] In step S406, the CPU 201 performs recognition processing for identifying the tracking subject from the subjects detected in step S403. If the CPU 201 can recognize the tracking subject from the subjects detected in step S403, the CPU 201 sets the value of the "existence flag indicating whether the tracking subject exists in the captured image" stored in the RAM 202 to "true," indicating that the tracking subject exists, and stores the rectangular frame information of the tracking subject in the rectangular frame information output in step S403 in the RAM 202.

[0072] If the CPU 201 cannot identify the tracking subject from the subjects detected in step S403 , the CPU 201 sets the value of the existence flag stored in the RAM 202 to “false” indicating that “the tracking subject does not exist”.

[0073] Since the initial step S406 is the first instance of step S406 after the tracking subject is designated, the CPU 201 sets the value of the existence flag to “true” and stores the rectangular frame information designating the tracking subject in the RAM 202 .

[0074] In the second and subsequent instances of step S406, the CPU 201 performs the following processing. The CPU 201 calculates the similarity between the rectangular frame information of each subject detected in step S403 and the "rectangular frame information of the tracked subject" stored in RAM 202 (the "rectangular frame information of the tracked subject" stored in RAM 202 in the most recent step S406). For example, the CPU 201 calculates the intersection-over-union (IoU) score between the rectangular frame corresponding to the rectangular frame information of each subject detected in step S403 and the rectangular frame corresponding to the "rectangular frame information of the tracked subject" stored in RAM 202 (the "rectangular frame information of the tracked subject" stored in RAM 202 in the most recent step S406) as the similarity. The greater the overlap between the rectangular frames, the higher the IoU score.

[0075] Then, the CPU 201 identifies the rectangular frame information of the objects detected in step S403 for which the highest similarity has been calculated as the rectangular frame of the tracking object. In other words, the CPU 201 identifies the object for which the highest similarity has been calculated as the tracking object among the objects detected in step S403.

[0076] If the similarity calculated for the rectangular frame information of the subject detected in step S403 is smaller than the threshold value, the CPU 201 determines that it cannot recognize the tracking subject.

[0077] The above-described process for identifying a tracking subject from a captured image is an example, and the present disclosure is not limited to the above-described process. The process for determining whether a tracking subject exists in a captured image is also not limited to the above-described process.

[0078] For example, the CPU 201 can use a process for predicting the motion of the tracked subject in similarity calculations. In this case, the CPU 201 only needs to predict the current tracked subject's rectangular frame based on the information about the previously tracked subject's rectangular frames and calculate the IoU between the predicted rectangular frame and each of the subject's rectangular frames as similarity. By using motion prediction, the accuracy of identifying the tracked subject can be improved when the subject is moving.

[0079] As another example, the CPU 201 may calculate the similarity using not only the rectangular frame information of the subject but also the features of the subject associated with the captured image. For example, the CPU 201 may obtain image-related features such as a brightness histogram within the rectangular frame and embed the similarity between the features for similarity calculation.

[0080] Alternatively, the CPU 201 may obtain features using a learning model that outputs a human feature vector while using a machine learning method such as deep learning, and embed the features for similarity calculation.

[0081] In step S407, the CPU 201 determines whether the value of the presence flag is "true" or "false." If the presence flag is "true," the CPU 201 determines that a tracking subject exists in the captured image, and the process proceeds to step S408. If the presence flag is "false," the CPU 201 determines that a tracking subject does not exist in the captured image, and the process proceeds to step S414.

[0082] In step S408, the CPU 201 reads the rectangular frame information of the tracking object (center coordinates and size of the rectangular frame of the tracking object) stored in the RAM 202. In step S409, the CPU 201 calculates the speed (PT control speed) at which the camera 100 changes the camera direction (pan angle and pitch angle) to track the tracking object and capture an image of the tracking object. Figure 6A and Figure 6B The specific example shown is used to explain the processing in step S409.

[0083] Figure 6A and Figure 6B is a diagram showing an example of a display screen performed by the display unit 208 during subject tracking. Figure 6A A display example of a captured image 600 including a subject 601 as a tracking subject and a subject 602 not being tracked is shown. The number of pixels in the horizontal direction of the captured image 600 is W (pixels), and the number of pixels in the vertical direction is H (pixels).

[0084] The rectangular frame 611 is a rectangular frame corresponding to the rectangular frame information of the subject 601, and the rectangular frame 612 is a rectangular frame corresponding to the rectangular frame information of the subject 602. The position 631 indicates the "center coordinates of the subject 601" indicated by the rectangular frame information of the subject 601, and the position 632 indicates the "center coordinates of the subject 602" indicated by the rectangular frame information of the subject 602. Figure 6A and Figure 6B, the size of the rectangular frame indicated by each rectangular frame information is assumed to indicate the length of the diagonal line of the rectangular frame. Size S (pixels) indicates the size of the rectangular frame indicated by the rectangular frame information of subject 601, and size 642 indicates the size of the rectangular frame indicated by the rectangular frame information of subject 602. Position 650 indicates the position of the object being tracked. In this embodiment, the workstation 200 controls the pan and tilt angles of the camera 100 so that the center coordinates of the tracked object in the captured image approach the object position 650. Hereinafter, for the purpose of discussion, the object position 650 is the center position of the captured image, but the present disclosure is not limited thereto, and the object position 650 may be any position in the captured image.

[0085] In this case, the CPU 201 calculates the horizontal distance Wdiff (pixels) between the position 631 and the object position 650, and the vertical distance Hdiff (pixels) between the position 631 and the object position 650. For example, the CPU 201 calculates (the horizontal coordinate of the object position 650 - the horizontal coordinate of the position 631) as Wdiff (pixels). For example, the CPU 201 also calculates (the vertical coordinate of the object position 650 - the vertical coordinate of the position 631) as Hdiff (pixels).

[0086] Then, the CPU 201 calculates the pan control speed Vp as the speed of change of the pan angle of the camera 100 by calculating the following formula 1, and calculates the tilt control speed Vt as the speed of change of the tilt angle of the camera 100 by calculating the following formula 2:

[0087] Vp = (Wdiff / W) × Kp1…Formula 1

[0088] Vt = (Hdiff / H) × Kt1…Equation 2.

[0089] Kp1 in Formula 1 and Kt1 in Formula 2 are both scaling factors. That is, using Formulas 1 and 2, the greater the distance between the target position and the center coordinates of the tracked object, the faster the control speed, while the smaller the distance, the slower the control speed. When the center coordinates of the tracked object reach the target position and the distance becomes zero, the control speed becomes zero. If a faster control speed is desired based on the distance between the target position and the center coordinates of the tracked object, increasing the scaling factor (Kp1 or Kt1) is sufficient.

[0090] Figure 7AThe graph 701 shown is a graph showing the relationship between Wdiff and Vp represented by Formula 1. The horizontal axis indicates Wdiff, and the vertical axis indicates Vp. When the distance between the object position and the center coordinates of the tracking subject is less than a threshold value (i.e., when the distance is very small), the control speed may also be set to zero. For example, the CPU 201 may use a method such that Figure 7A Vp is calculated using the calculation formula in which Vp is zero in the interval where Wdiff is a threshold value or less as shown in graph 702 .

[0091] The above example describes the case where the control speed is calculated using a formula in which the relationship between distance and control speed is linear. However, the formula for calculating the control speed is not limited to this linear formula, and other formulas can be used to calculate the control speed as long as they have a relationship such that the control speed increases with increasing distance and decreases with decreasing distance. This is also true even if the control speed is set to zero when the distance between the target position and the center coordinates of the tracked object is less than the threshold value (i.e., when the distance is very small).

[0092] In step S410, the CPU 201 calculates the speed (Z control speed) at which the camera 100 changes its zoom to track the tracking subject and capture an image of the tracking subject. In this embodiment, the CPU 201 controls the zoom of the camera 100 so that the size of the tracking subject in the captured image is the defined size.

[0093] In this embodiment, the "size of the tracking subject's rectangular frame" indicated by the tracking subject's rectangular frame information initially read in step S408 is used as the "defined size." However, the present disclosure is not limited thereto. For example, the defined size may be a size specified by the user through an operation of the user input interface 207. Alternatively, the CPU 201 may dynamically change the defined size.

[0094] Will refer to Figure 6A and Figure 6B The specific example shown explains the processing in step S410. The CPU 201 calculates the zoom control speed Vz, which is the speed of change of the zoom of the camera 100, by calculating the following formula 3:

[0095] Vz=(Sb-S)×Kz…Formula 3.

[0096] Sb is the base subject size (pixels) (defined above), and Kz is the scaling factor. In other words, the larger the difference between the base subject size and the tracking subject size, the higher the zoom control speed, and the smaller the difference, the lower the zoom control speed. When the tracking subject size matches the base subject size, the zoom control speed is zero. This allows zoom control to maintain a constant size for the tracking subject.

[0097] As with pan / tilt control, if a faster zoom control speed is desired, it is sufficient to increase the scale factor Kz. If the difference between the size of the tracked subject and the size of the reference subject is smaller than a threshold value, the CPU 201 may also set the zoom control speed to zero, thereby preventing zoom control from being performed when the difference is smaller than the threshold value (i.e., when the difference is very small).

[0098] In step S411, the CPU 201 transmits a command for inquiring about a set of the current pan angle, the current pitch angle, the current horizontal angle of view, and the current vertical angle of view (PTZ values) to the camera 100 via the communication unit 204. Then, in response to the command, the CPU 201 receives the set transmitted from the camera 100 via the communication unit 204 and stores the received set in the RAM 202.

[0099] The pan angle and pitch angle are angles relative to a given orientation (hereinafter referred to as the front orientation) of the camera 100. In the following description, the current pan angle of the camera 100 is represented as PANcur (degrees), and the current pitch angle of the camera 100 is represented as TILTcur (degrees). In addition, the current horizontal field of view of the camera 100 is represented as FOVh (degrees), and the current vertical field of view of the camera 100 is represented as FOVv (degrees).

[0100] In step S412, the CPU 201 calculates the direction of the tracking subject (tracking subject direction) viewed from the camera 100 relative to the front orientation of the camera 100. Figure 6A An example of calculating the direction of the tracking subject is described below. The CPU 201 approximately calculates the horizontal field angle FOVhp of each pixel of the captured image by calculating the following formula 4:

[0101] FOVhp=FOVh / W...Formula 4.

[0102] Here, it is assumed that the position 631 of the tracking subject is horizontally spaced Wdiff (pixels) from the center position (target position 650) of the captured image 600. In this case, the CPU 201 calculates the tracking subject direction in the pan direction, that is, the angle PANtarget (degrees) of the tracking subject in the pan direction as viewed from the camera 100 relative to the frontal orientation of the camera 100, by calculating the following formula 5:

[0103] PANtarget=FOVhp×Wdiff+PANcur…Formula 5.

[0104] The CPU 201 then stores the calculated angle PANtarget in the RAM 202. Similarly, the CPU 201 calculates FOVvt = FOVv / H to approximately obtain the vertical field of view angle FOVvt per pixel of the captured image. Assuming that the position 631 of the tracking object is vertically spaced Hdiff (pixels) from the center position of the captured image 600, the CPU 201 calculates TILTtarget = FOVvt × Hdiff + TILEcur to obtain the tracking object direction in the pitch direction, that is, the angle TILTtarget (degrees) of the tracking object in the pitch direction as viewed from the camera 100 relative to the frontal orientation of the camera 100. The CPU 201 then stores the calculated angle TILTtarget in the RAM 202.

[0105] The method for calculating the tracking subject direction is not limited to the above method. For example, the CPU 201 may calculate the tracking subject direction taking into account that the camera direction of the camera 100 changes spherically around the camera 100 due to the rotation of the PTZ drive unit 107. Figures 8A to 8C Describe an example calculation for this situation.

[0106] Figure 8A is a diagram showing a captured image 800 captured by camera 100, in which the tracking subject is located at position P. Position P is horizontally spaced by Wdiff (pixels) and vertically spaced by Hdiff (pixels) from the center of captured image 800. It is assumed that the angular difference between the center of captured image 800 and position P, calculated by Formula 4, is defined as PANdiff (degrees) and Tiltdiff (degrees).

[0107] Figure 8BA sphere 801 is shown with a radius equal to the distance from the camera 100 to the subject in the captured image in a three-dimensional space with the position of the camera 100 as the origin O. Assume that the front of the camera 100 points in the X-axis direction in the three-dimensional space, and when the camera 100 is driven in the pan direction, the captured image rotates around the Z-axis, and when the camera 100 is driven in the tilt direction, the captured image rotates around the Y-axis. Therefore, if the captured image 800 is an image captured by the camera 100 facing the front, then Figure 8B The captured image 800 is shown at a position perpendicular to the X-axis.

[0108] Here, the CPU 201 first calculates using the following formulas 6, 7, and 8: Figure 8B The three-dimensional coordinates (x, y, z) of the position P of the tracking object in the state shown. Since it is sufficient to obtain only the direction of the tracking object in this process, the radius of the sphere 801 can be any value, but for ease of calculation, the radius is assumed to be 1 in the following calculation:

[0109] x=1 ...Formula 6

[0110] y=tan(PANdiff)…Formula 7

[0111] z=tan(TILTdiff)…Equation 8.

[0112] Since these are the coordinates when the camera 100 is facing in the X-axis direction, the CPU 201 then converts the three-dimensional coordinates (x, y, z) based on the direction the camera 100 is facing. For example, this calculation can be performed by rotating the coordinate axes in three-dimensional space using a known three-dimensional rotation coordinate conversion calculation. Specifically, if the current pan and tilt angles of the camera 100 are PANcur and TILTcur, respectively, the CPU 201 can calculate the three-dimensional coordinates (X, Y, Z) of the tracked object after rotating the coordinate axes by calculating the following formula 9:

[0113]

[0114] Figure 8C The three-dimensional coordinates (X, Y, Z) of the tracking object and the pan angle PANtarget and pitch angle TILTtarget calculated using Formula 9 are shown. Finally, the CPU 201 calculates the pan angle PANtarget and pitch angle TILTtarget of the tracking object relative to the reference direction of the camera 100 by calculating the following Formulas 10 and 11 based on the three-dimensional position (X, Y, Z) of the tracking object:

[0115] PANtarget = arctan(Y / X) ... Formula 10

[0116] TILTtarget=arctan(Z / √(X 2 +Y 2 )) …Formula 11.

[0117] Using Formulas 6 to 11 above, the CPU 201 can calculate the tracking subject direction while taking into account the rotation of the PTZ drive unit 107. In step S413, the CPU 201 generates a control command to cause the PTZ drive unit 107 to change the pan angle of the camera 100 at a pan control speed Vp, change the pitch angle of the camera 100 at a tilt control speed Vt, and change the zoom of the camera 100 at a zoom control speed Vz. The CPU 201 then transmits the generated control command to the camera 100 via the communication unit 204.

[0118] Here, assuming Figure 6A The captured image 600 shown is input from the camera 100 to the workstation 200 and then input to Figure 6B A captured image 600 is shown. Figure 6B The captured image 600 shown is an image in which Figure 6A The subject 601 in the captured image 600 shown moves to the right, and the subject 602 in front of the camera 100 relative to the subject 601 moves to the left, so that the subject 601 is hidden behind the subject 602 .

[0119] when Figure 6A When the captured image 600 is input to the workstation 200, the subject 601 as the tracking subject can be detected in the captured image 600, and thus the subject 601 is displayed with a rectangular frame 611. However, if Figure 6B When the captured image 600 is input to the workstation 200, the subject 601 is hidden behind the subject 602 in the captured image 600, and thus the subject 601 cannot be detected in the captured image 600. Therefore, no rectangular frame is displayed on the subject 601.

[0120] Therefore, if Figure 6A If the captured image 600 is input to the workstation 200, the process of step S407 is followed by the process of step S408. Figure 6B If the captured image 600 is input to the workstation 200 , the process of step S407 is followed by the process of step S414 .

[0121] In step S414 , the CPU 201 determines whether a period of time during which the value of the presence flag is “false” (ie, a period of time during which the tracking subject does not exist in the captured image) is not shorter than a predetermined period of time.

[0122] As a result of this determination, if the period of time during which the value of the existence flag is "false" is longer than or equal to the predetermined period of time, the CPU 201 determines that the situation is such that it is difficult to re-detect the tracking subject, and Figure 4 The processing of the flowchart in FIG. ends. At the end of the processing, the camera 100 may be controlled so that the camera direction and shooting angle of view are set in advance by the user. In this case, information related to the camera direction and shooting angle set in advance by the user may be recorded, and the CPU 201 may communicate a control command to the camera 100 before the end of the processing. If the period during which the value of the presence flag is "false" is shorter than the predetermined period, the process moves to step S415.

[0123] The "predetermined time period" may be any time period and may be a preset time period, a time period set by the user via an operation of the user input I / F 207, or a time period dynamically changed by the CPU 201. It is not necessary to set a timeout after the predetermined time period has elapsed. For example, the CPU 201 may perform the processing in step S415 and subsequent steps upon detecting the occurrence of an event. The event is not limited to a specific event and may be, for example, an event in which "the user has input a specific instruction by operating the user input I / F 207."

[0124] In step S415, the CPU 201 reads the tracking subject direction calculated in the most recently executed step S412 from the RAM 202. For example, assuming that the tracking subject direction is obtained in the previous step S402 Figure 6A The captured image 600 is obtained in step S402. Figure 6B In this case, the CPU 201 acquires the “ Figure 6A The tracking subject direction of the tracking subject in the captured image 600″.

[0125] In step S416, the CPU 201 transmits a command for inquiring about the above-mentioned set to the camera 100 via the communication unit 204 in the same manner as in the above-mentioned step S411. The CPU 201 then receives the set transmitted from the camera 100 in response to the command via the communication unit 204, and stores the received set in the RAM 202.

[0126] In step S417, the CPU 201 calculates the pan control speed Vp and the tilt control speed Vt by calculating the following Formula 12 and Formula 13, respectively, based on the tracking subject direction (PANtarget and TILTtarget) read in step S415 and the current pan angle PANcur and the current pitch angle TILTcur included in the set obtained in step S416.

[0127] Vp = (PANtarget -PANcur) × Kp2…Equation 12

[0128] Vt = (TILTtarget - TILTcur) × Kt2…Equation 13

[0129] Kp2 in Formula 12 and Kt2 in Formula 13 are both proportional coefficients. Using Formulas 12 and 13, the greater the angular difference between the direction of the tracking subject immediately before the tracking subject ceases to exist in the captured image and the current camera direction of the camera 100, the faster the control speed, and the smaller the angular difference, the slower the control speed.

[0130] Therefore, if the workstation 200 can detect a tracking target in the captured image, the workstation 200 calculates the direction toward the tracking target as the tracking subject direction based on the detection result. If the workstation 200 cannot detect any tracking target from the captured image, the workstation 200 controls the camera direction control speed based on the previously calculated tracking subject direction.

[0131] Figure 7B The graph 711 shown in is a graph showing the relationship between "the angle difference between the angle in the pan direction of the tracking object as viewed from the camera 100 and the current pan angle of the camera 100" and the "pan control speed" expressed by Formula 12. The horizontal axis indicates the "angle difference between the angle in the pan direction of the tracking object as viewed from the camera 100 and the current pan angle of the camera 100", and the vertical axis indicates the "pan control speed". If a faster pan control speed is desired, it can be achieved by increasing the proportional coefficient Kp2. If the "angle difference between the angle in the pan direction of the tracking object as viewed from the camera 100 and the current pan angle of the camera 100" is less than the threshold value (that is, if the angle difference is very small), the pan control speed can also be set to zero to prevent pan control. In this case, the CPU 201 can use a method such as Figure 7B The pan control speed is calculated using a formula in which the pan control speed becomes zero in an interval where the “angle difference between the angle in the pan direction of the tracking object viewed from the camera 100 and the current pan angle of the camera 100 ” is less than a threshold value as shown in graph 712 .

[0132] As described above, in step S417, the tracking subject direction calculated in the most recently executed step S412 is used. Therefore, in step S412, the tracking subject direction can be calculated taking into account the difference in control cycles between step S412 and step S417. For example, the CPU 201 can perform processing to predict the tracking subject's movement in step S412 to predict the tracking subject's position in the next cycle, and then perform the same arithmetic operation based on the difference between the predicted tracking subject position and the current camera direction to calculate the tracking subject direction. By performing this prediction processing, the pan / tilt speed can be calculated based on the tracking subject's position at the time the pan / tilt speed is actually calculated.

[0133] As another example of calculating the tracking subject direction, the CPU 201 may correct the tracking subject direction in consideration of subject detection characteristics of the inference unit 206 . Figure 9 A to C are diagrams illustrating examples of subject detection characteristics of the inference unit 206 . Figure 9 A to C show a state in which the subject 901 in the captured image 900 moves to the right and is blocked by an obstruction 960 .

[0134] Since it is possible to Figure 9 The subject 901 is detected in the captured image 900 of A, and thus a rectangular frame 911 of the subject 901 as a result of the detection of the subject 901 by the inference unit 206 is superimposed on the captured image 900. The position 931 is located at Figure 9 The center coordinates of the rectangular frame 911 of the subject 901 in the captured image 900 of A are shown, and the width 941 is the horizontal length (number of pixels) of the rectangular frame 911 .

[0135] If the inference unit 206 has a characteristic that the subject cannot be detected even if the subject is slightly obscured, the subject 901 will not be detected from the image. Figure 9 This is because a portion of the subject 901 is blocked by the occluder 960. Therefore, the rectangular frame of the subject 901 is not superimposed on these captured images 900.

[0136] In the above description, if Figure 9 The captured image 900 of A is obtained in the previous step S402 and Figure 9 The captured image 900 of B is acquired in step S402 of this time, and in step S417 of this time, the CPU 201 approaches the position 931 in the direction (based on Figure 9 The control speed of the camera direction is calculated by using the tracking subject direction calculated from the captured image 900 of A.

[0137] Therefore, in step S412, the CPU 201 may also correct the calculated tracking subject direction to a direction obtained by moving the calculated tracking subject direction by a width 941 in the direction of the pan control speed Vp. Figure 9 In C, the position on the captured image in the correction direction is shown as position 932. Figure 9 B, Figure 9 The captured image 900 of C is Figure 9 The captured image 900 of A is obtained after the captured image in step S402.

[0138] Therefore, if the Figure 9 If the captured image 900 of C is obtained, then in step S417 this time, the camera direction of the camera 100 may be determined based on the difference between the tracking object direction corresponding to the position 932 and the current pan angle of the camera 100 .

[0139] The present disclosure is not limited to determining the correction amount for the tracking subject direction based on the size of the tracking subject's rectangular frame; a predetermined amount determined in advance through experiments may also be used. By correcting the tracking subject direction in this manner, the pan / tilt direction can be adjusted to a direction closer to the direction in which the tracking subject is actually obscured, while suppressing the influence of the subject detection characteristics of the inference unit 206.

[0140] The time period used for comparison with the "predetermined time period" in step S414 is not limited to the "time period during which the value of the presence flag is 'false' (i.e., the time period during which the tracking subject does not exist in the captured image)." For example, the timer may be started at a timing at which the camera 100 faces the direction of the tracking subject immediately before the camera 100 is no longer able to recognize the tracking subject due to the processing in step S417, and the PTZ of the camera 100 is stopped. In this case, in step S414, the CPU 201 may compare the time period during which the state of "the camera 100 faces the direction of the tracking subject immediately before the camera 100 is no longer able to recognize the tracking subject, and the PTZ of the camera 100 is stopped" continues with the predetermined time period. By performing such control, the timeout is set based on the time elapsed from the stop of pan / tilt, so it is easier for the user to confirm that the timeout has occurred.

[0141] Therefore, according to the present embodiment, even if the tracking subject can no longer be identified from the captured image, the pan / tilt control can be continued so that the camera direction of the camera 100 faces the direction in which the tracking subject exists immediately ahead.

[0142] If the camera stops driving immediately after the subject can no longer be detected, panning / tilting will stop abruptly and the quality of the captured image will deteriorate. Therefore, by continuing panning / tilting control as in this embodiment, the quality of the captured image during tracking shooting can be improved.

[0143] Since tracking photography involves subject detection and camera drive based on the detection results, camera drive is, in principle, delayed relative to subject movement. Therefore, if camera drive is stopped immediately after the subject can no longer be detected, the pan / tilt operation will cease, even if the tracking subject has not yet reached the intended position. However, by performing the processing described in this embodiment, the pan / tilt operation can be stopped at the timing when the pan / tilt movement reaches a position where the tracking subject is hidden.

[0144] If the tracking subject is hidden in the shadow of an object, there is a possibility that the subject will reappear from the hidden position. In other words, by stopping the pan / tilt operation at the position where the tracking subject is hidden, as in this embodiment, the pan / tilt operation can be stopped at a position where the tracking subject is easily detected to be reappearing.

[0145] The process of calculating the control speed described in the first embodiment may be performed by the camera 100, the workstation 200, or a device separate from these devices. In other words, the control device that performs the process of calculating the control speed may be included in the camera 100, the workstation 200, or may exist as a device separate from these devices. Furthermore, such a control device may be configured as hardware, software, or a combination of hardware and software.

[0146] Second embodiment

[0147] This embodiment describes the differences from the first embodiment, and the other configurations are the same as those in the first embodiment unless otherwise described below. In this embodiment, a configuration having the functions of the camera 100 and capable of executing the processing of the workstation 200 (according to Figure 4 The camera 1000 of the process of the flowchart of FIG. Figure 10 An example of the hardware configuration of the camera 1000 according to the present embodiment is described.

[0148] CPU 1001, RAM 1002, storage device 1003, camera unit 1004 and PTZ drive unit 1007 are connected to the Figure 1 The illustrated CPU 101 , RAM 102 , storage device 103 , imaging unit 104 , and PTZ driving unit 107 are the same, and description thereof will be omitted.

[0149] The communication unit 1006 performs data communication with an external device via a network such as a LAN or the Internet, etc. The method by which the communication unit 1006 performs data communication with an external device is not limited to a specific data communication method.

[0150] The inference unit 1009 is similar to the inference unit 206 except that it acquires a captured image from the imaging unit 1004. The user input I / F 1010 is a user interface such as a button, switch, lever, or touch panel screen, and can be operated by the user to input various information and instructions to the camera 1000.

[0151] The CPU 1001 , the RAM 1002 , the storage device 1003 , the imaging unit 1004 , the communication unit 1006 , the PTZ driving unit 1007 , the inference unit 1009 , and the user input I / F 1010 are connected to a system bus 1008 . Figure 10 The constituent components of the camera 1000 shown in FIG. 1 are driven by power obtained by rectifying AC power supplied from an external power source into a predetermined voltage or by power supplied from a built-in battery (not shown).

[0152] Will refer to Figure 11 The flowchart in describes tracking photography of a tracking subject performed by the camera 1000. Figure 11 The flowchart is a flowchart of processing executed in response to the CPU 1001 detecting a tracking / photographing execution instruction input by the user via an operation of the user input I / F 1010. The method for inputting the tracking / photographing execution instruction is not limited to a specific input method.

[0153] In step S1101, the CPU 1001 determines whether the communication unit 1006 or the user input I / F 1010 has obtained the information for terminating the operation according to the embodiment of the present invention. Figure 11 If it is determined that the end command has been obtained, then Figure 11 The processing of the flowchart in ends. If it is determined that the end command has not been obtained, the process moves to step S1102. In step S1102, the CPU 1001 acquires the captured image output from the imaging unit 1004 and stores the acquired captured image in the RAM 1002.

[0154] In step S1103, the CPU 1001 inputs the captured image stored in the RAM 1002 in step S1102 to the inference unit 1009. The inference unit 1009 performs subject detection processing on the input captured image by performing the same processing as the inference unit 206, and outputs rectangular frame information defining a rectangular frame including the whole body (human body) of each subject detected from the captured image.

[0155] In step S1105, the CPU 1001 determines whether any tracking subject has been specified. If it is determined that a tracking subject has been specified, the process moves to step S1106. If it is determined that a tracking subject has not been specified, the process moves to step S1101. The method of specifying a tracking subject is not limited to a specific method.

[0156] In step S1106, the CPU 1001 performs the same processing as in step S406 above to perform recognition processing for recognizing the tracking subject from among the subjects detected in step S1103. If the CPU 1001 is able to recognize the tracking subject from among the subjects detected in step S1103, the CPU 1001 sets the value of the existence flag stored in the RAM 1002 to "true" and stores the rectangular frame information of the tracking subject in the rectangular frame information output in step S1103 in the RAM 1002.

[0157] If the CPU 1001 cannot identify the tracking subject from among the subjects detected in step S1103 , the CPU 1001 sets the value of the existence flag stored in the RAM 1002 to “false”.

[0158] Since the initial step S1106 is the first instance of step S1106 after the tracking subject is designated, the CPU 1001 sets the value of the existence flag to "true" and stores the rectangular frame information designating the tracking subject in the RAM 1002. In the second and subsequent instances of step S1106, the CPU 1001 performs the same processing as in the case of the second and subsequent instances of step S406.

[0159] In step S1107, the CPU 1001 determines whether the value of the existence flag is "true" or "false." As a result of the determination, if it is determined that the value of the existence flag is "true," the process moves to step S1108, and if the value of the existence flag is "false," the process moves to step S1114.

[0160] In step S1108, the CPU 1001 reads the rectangular frame information of the tracking subject stored in the RAM 1002. In step S1109, the CPU 1001 performs the same processing as in step S409 to calculate the speed at which the camera 100 changes the camera direction (pan angle and pitch angle) in order to track the tracking subject and capture an image of the tracking subject.

[0161] In step S1110 , the CPU 1001 performs the same processing as in step S410 described above to calculate the speed at which the camera 100 changes the zoom in order to track the tracking subject and capture an image of the tracking subject.

[0162] In step S1111, the CPU 1001 acquires a set of, for example, the current pan angle of the camera 100, the current pitch angle of the camera 100, the current horizontal field angle of the camera 100, and the current vertical field angle of the camera 100 from the PTZ drive unit 1007. The CPU 1001 then stores the acquired set in the RAM 1002.

[0163] In step S1112, the CPU 1001 performs the same processing as in step S412 above to calculate the tracking subject direction. In step S1114, the CPU 1001 determines whether the time period during which the value of the existence flag is "false" (i.e., the time period during which the tracking subject does not exist in the captured image) is not shorter than a predetermined time period. As a result of the determination, if the time period during which the value of the existence flag is "false" (i.e., the time period during which the tracking subject does not exist in the captured image) is longer than or equal to the predetermined time period, the CPU 1001 determines whether the tracking subject direction is shorter than a predetermined time period. Figure 11 If the period of time during which the value of the presence flag is false (ie, the period during which the tracking subject does not exist in the captured image) is shorter than the predetermined period of time, the process proceeds to step S1115.

[0164] In step S1115, the CPU 1001 reads the tracking subject direction calculated in the most recently executed step S1112 from the RAM 1002. In step S1116, the CPU 1001 performs the same processing as in step S1111 to acquire the aforementioned set and stores the acquired set in the RAM 1002. In step S1117, the CPU 1001 performs the same processing as in step S417 to calculate the pan control speed Vp and the tilt control speed Vt.

[0165] In step S1118, the CPU 1001 generates a control command to cause the PTZ drive unit 1007 to change the pan angle of the camera 1000 at the pan control speed Vp, change the pitch angle of the camera 1000 at the tilt control speed Vt, and change the zoom of the camera 1000 at the zoom control speed Vz. The CPU 1001 then performs the same processing as in step S302 above to obtain the PTZ control speed of the camera 100 based on the generated control command, and obtains a drive parameter based on the PTZ control speed.

[0166] In step S1119 , the CPU 1001 performs the same processing as in step S303 described above to control the PTZ drive unit 1007 based on the drive parameters obtained in step S1118 .

[0167] In the first and second embodiments, the case where the subject to be detected is a person is described. However, the attribute of the subject to be detected is not limited to a person, and the subject to be detected may be an object of any attribute.

[0168] Numerical values, processing timings, processing orders, processing operators, and configuration / acquisition methods / destinations / storage locations of data (information) and the like used in the above-described embodiments are merely examples provided for specific description, and the present disclosure is not limited to such examples.

[0169] Some or all of the above embodiments may be used in combination as appropriate. Some or all of the above embodiments may be used selectively.

[0170] Other embodiments

[0171] The embodiments of the present invention can also be implemented by the following method, that is, software (including computer program products of computer programs / instructions) that perform the functions of the above-mentioned embodiments is provided to a system or device through a network or various storage media, and a computer (central processing unit (CPU), microprocessing unit (MPU)) of the system or device reads and executes the computer program / instructions.

[0172] While the present disclosure has been described with reference to exemplary embodiments, it is to be understood that the present disclosure is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. A control device comprising: a memory storing a program; as well as a processor configured to, when executing the program, cause the control device to: When a tracking target can be detected from the captured image, a direction toward the tracking target is calculated as a tracking subject direction based on the detection result, and In a case where the tracking target cannot be detected from the captured image, the control speed in the camera direction is controlled based on the previously calculated tracking subject direction.

2. The control device according to claim 1, wherein: The processor is further configured to cause the control device to calculate the tracking object direction based on a difference between a position of the tracking object detected from the captured image and the object position, a current camera direction, and a zoom when the tracking object can be detected from the captured image.

3. The control device according to claim 1, wherein: The processor is further configured to cause the control device to calculate a control speed in the camera direction based on a difference between a position of the tracking object detected from the captured image and a position of the object when the tracking object can be detected from the captured image.

4. The control device according to claim 1, wherein: The processor is further configured to cause the control device to calculate a zoom control speed based on a difference between a size of the tracking object detected from the captured image and a defined size when the tracking object can be detected from the captured image.

5. The control device according to claim 1, wherein: The processor is further configured to cause the control device to: In a case where the tracking target cannot be detected from the captured image, the camera direction is changed in response to a period in which the tracking target cannot be detected continuing for a predetermined period of time after the control speed in the camera direction has become zero.

6. The control device according to claim 1, wherein: The processor is further configured to cause the control device to calculate a control speed in the camera direction based on a difference between a current camera direction and a previously calculated tracking subject direction when the tracking target cannot be detected from the captured image.

7. The control device according to claim 1, wherein: The processor is further configured to cause the control device to: When the tracking target cannot be detected from the captured image, the control speed in the camera direction is calculated based on the difference between the current camera direction and the tracking target direction predicted based on the previously calculated tracking target direction.

8. The control device according to claim 1, wherein: The processor is further configured to cause the control device to, when the tracking object cannot be detected from the captured image, calculate a control speed in the camera direction based on a difference between a current camera direction and a tracking object direction obtained by correcting a previously calculated tracking object direction according to a size of the tracking object.

9. The control device according to claim 1, wherein: The processor is further configured to cause the control device to: Obtaining images captured by a camera device, and generating a command for changing the direction of the camera at a controlled control speed, and The generated command is sent to the imaging device.

10. The control device according to claim 1, further comprising: a camera unit configured to capture images; as well as A driving unit is configured to change the camera direction at a controlled speed.

11. A control method comprising: When the tracking target can be detected from the captured image, a direction toward the tracking target is calculated as a tracking subject direction based on a detection result; as well as In a case where the tracking target cannot be detected from the captured image, the control speed in the camera direction is controlled based on the previously calculated tracking subject direction.

12. A computer-readable storage medium storing a computer program for causing a computer to execute a method, the method comprising: When the tracking target can be detected from the captured image, a direction toward the tracking target is calculated as a tracking subject direction based on a detection result; as well as In a case where the tracking target cannot be detected from the captured image, the control speed in the camera direction is controlled based on the previously calculated tracking subject direction.

13. A computer program product comprising a computer program for causing a computer to perform a method comprising: When the tracking target can be detected from the captured image, a direction toward the tracking target is calculated as a tracking subject direction based on a detection result; as well as In a case where the tracking target cannot be detected from the captured image, the control speed in the camera direction is controlled based on the previously calculated tracking subject direction.

Images