Control apparatus, control method, and program

Abstract

To enable an automatic tracking photographing in which a tracking delay is suppressed.SOLUTION: A control apparatus having control means of executing processing for controlling an imaging direction so as to fix a subject of a tracking object to a target position in an imaging image imaged by an imaging device, contains: acquisition means of acquiring a plurality of imaging images; detection means of detecting a position of the subject in the imaging image; prediction means of predicting the position of the subject after the predetermined time on the basis of the position of the subject; and control means of executing processing for controlling the imaging direction of the imaging device. In the case where the prediction position reaches the target position regulated by a predetermined pixel area or a predetermined line in the imaging image, the control means starts processing for controlling the imaging direction so as to fix the subject to the target position in the imaging image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to tracking technology. [Background technology]

[0002] In recent years, there has been a technology that detects an object from an image captured by an imaging device and controls the imaging direction of the imaging device so that the detected object remains within the imaging angle of view. When using such a technology in video production, it is important to perform smooth tracking shooting.

[0003] Patent Document 1 describes a method for tracking a tracking target without causing a large change in a captured image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 5809538 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in Patent Document 1, for example, after the subject reaches the capture position (for example, the center of the screen), the camera platform starts rotating from a stopped state. Therefore, if the subject is moving fast, it takes time for the driving speed of the camera platform to reach that speed. Therefore, tracking starts from a position that is quite far from the capture position. In other words, the subject may be far from the capture position, such as the center of the screen, which may result in an undesirable composition.

[0006] SUMMARY OF THE PRESENT EMBODIMENT An object of the present invention is to provide a control device that enables automatic tracking shooting with reduced tracking delay. [Means for solving the problem]

[0007] In order to solve the above problems, the control device in this embodiment is a control device having a control means that executes processing to control the imaging direction so that the subject to be tracked remains at a target position in an image captured by an imaging device, and has an acquisition means that acquires a plurality of captured images from the imaging device, a detection means that detects the position of the subject in the captured images acquired by the acquisition means, a prediction means that predicts the position of the subject a predetermined time ahead as a predicted position based on the positions of the subject in the plurality of captured images, and a control means that executes processing to control the imaging direction of the imaging device, and is characterized in that when the predicted position reaches the target position defined by a predetermined pixel area or a predetermined line in the captured image, the control means starts processing to control the imaging direction so as to keep the subject at the target position in the captured image. Effect of the Invention

[0008] According to the present invention, automatic tracking photography with suppressed tracking delay is possible. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a configuration diagram of an automatic tracking system according to a first embodiment; [Diagram 2] FIG. 1 is a block diagram showing an example of the functional configuration of a controller according to a first embodiment; [Diagram 3] Hardware configuration diagram of a controller according to the first embodiment [Figure 4] 1 is a flowchart of tracking control of an automatic tracking system according to a first embodiment. [Diagram 5] Image of camera pan head operation in the first embodiment [Figure 6] 11 is a flowchart of tracking control of an automatic tracking system according to a second embodiment. [Figure 7] Operational image of the camera platform in the second embodiment DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] (First embodiment) FIG. 1 is a configuration diagram of an automatic tracking system according to a first embodiment. The imaging system according to this embodiment includes a controller 100, a camera 200, a camera platform 300, and a LAN 400. The method of connection between the devices is not limited to a specific method. For example, the devices may be connected via a network such as a wireless LAN, or may be connected by a wired cable. In this imaging system, the imaging direction of the camera 200 can be changed by connecting the camera 200 to a camera platform 300 having a drive unit that drives in the horizontal and vertical directions. Furthermore, by issuing a control instruction to the drive unit of the camera platform 300 from the controller 100, it becomes possible to change the imaging direction of the camera 200 remotely. In addition, the controller 100 detects a subject from an image captured by the camera 200, and transmits a control instruction to the camera platform 300 so as to keep the subject at a preset position in the captured image, thereby making it possible to automatically track the subject.

[0011] Next, an example of the functional configuration of the controller 100 will be described with reference to Fig. 2. The controller 100 is composed of a position detection unit 101, a speed calculation unit 102, a position prediction unit 103, a storage unit 104, a determination unit 105, an acquisition unit 106, a relative angular velocity calculation unit 107, and a control unit 108, and constitutes an automatic tracking system that controls the camera platform camera 200. The main units that realize this embodiment are the position detection unit 101, the speed calculation unit 102, the position prediction unit 103, and the determination unit 105, and are executed by a CPU 1001 in Fig. 4 described later.

[0012] The position detection unit 101 acquires a captured image from the camera 200. That is, a plurality of consecutive captured images (frames) are acquired. Furthermore, the position detection unit 101 detects the subject position (center point) of the tracking target in the captured image by a process of detecting an object from the captured image using a learned model (hereinafter referred to as object detection AI). This detected subject position (center point) is output to the speed calculation unit 102 and the position prediction unit 103 as position information of the subject. Note that the subject position information is information in pixels [pix] according to the resolution of the captured image. For example, when the captured image has a FHD resolution of 1920×1080 and the lower left is set as the reference position X:0, Y:0, when the subject position is located at the center of the image, the position information is X:960, Y:540. In this embodiment, the position detection unit 101 detects the position of the subject, but this is not limited to this. For example, the subject position information may be acquired from the camera 200. At this time, the camera 200 detects the position of the subject from the captured image and transmits information about the frame and the subject position information to the controller 100. For simplicity, the current frame is referred to as frame t, and the frame immediately preceding the current frame whose position has been detected is referred to as frame t-1. In this embodiment, the detection speed of the object detection AI of the position detection unit 101 is described as 5 fps. In other words, when the time of the current frame t is 0s, the time of frame t-1 is -200 ms.

[0013] The speed calculation unit 102 acquires position information from the position detection unit 101. Furthermore, the one-way moving speed or speed vector of the subject is calculated as the speed of the subject from the subject position of the frame one or more frames before the acquired frame and the subject position of the acquired frame. The calculated speed of the subject is output to the position prediction unit 103. In this embodiment, the multiple frames used to calculate the subject speed are determined based on the detected speed of the position detection unit 101, and the unit of the calculated speed of the subject is described as [pix / s]. That is, the speed calculation unit 102 calculates the speed of the subject using the position information of the subject in the current frame t and the position information of the subject in frame t-1. In this embodiment, the speed of the subject is calculated from the position information of the subject in the acquired frame and the position information of the subject acquired before the frame, but is not limited to this. As long as the subject position can be predicted, the method of calculating the speed of the subject may be any method.

[0014] The position prediction unit 103 obtains the subject position in frame t from the position detection unit 101 and the subject's speed from the speed calculation unit 102. From information about the subject position and subject speed in frame t, the position where the subject will arrive after a predetermined time has elapsed is predicted. In this embodiment, a predicted position at a time according to the detection speed of the object detection AI of the position detection unit 101 is calculated, and the predicted position (predicted position) is explained as information expressed in pixels [pix]. In other words, when the detection speed of the object detection AI is 5 fps, if frame t: 0s, the position of the subject at frame t+1: 200 ms is calculated as the predicted position.

[0015] The storage unit 104 stores position information of the target position at which the camera head rotation starts. In this embodiment, the target position is a position that is predefined by a specific pixel area or line through which the subject passes. In this embodiment, the target position is described as information expressed in pixels [pix].

[0016] The determination unit 105 acquires information on the target position from the storage unit 104. Furthermore, the determination unit 105 acquires the predicted position predicted by the position prediction unit 103. Furthermore, it determines whether the predicted position matches the target position. In this embodiment, the determination unit 105 determines whether the predicted position of the subject matches the target position or passes through the target position, but this is not limited to this. For example, when the target position is predefined as a predetermined pixel area, it is determined that the target position and the predicted position match when the predicted position of the subject enters the pixel area. Also, when the target position is predefined by a predetermined line, it is determined that the predicted position of the subject has passed the target position when the X coordinate or Y coordinate of the predicted position of the subject matches the X coordinate or Y coordinate defined as the target position. In this embodiment, the method of determining that the predicted position matches or passes the target position is not limited to this. For example, a difference value may be calculated from the coordinates of the target position and the coordinates of the predicted position, and it may be determined that the target position and the predicted position match when the difference value becomes a predetermined value or less.

[0017] Acquisition unit 106 acquires angle of view information of the captured image captured by camera 200 through serial communication with camera 200. Here, angle of view information refers to information on the angle of view corresponding to the vertical and horizontal widths of the captured image obtained from camera 200, and is information on the imaging range as shown in FIG. 3. In this embodiment, the angle of view information may be obtained by acquiring the lens zoom value (lens data) of the camera and converting it into angle of view information by acquisition unit 106. This makes it possible to grasp the angle of view of the vertical and horizontal widths of the captured image.

[0018] The angular velocity calculation unit 107 acquires the angle of view information from the acquisition unit 106 and the subject's velocity [pix / s] from the velocity calculation unit 102. Furthermore, based on the angle of view information and the subject's velocity, it calculates a relative angular velocity indicating the relative velocity between the subject's velocity and the pan / tilt velocity of the camera 200 and the camera head 300. Then, it outputs the calculated relative angular velocity to the control unit 108. For example, the relative angular velocity of the horizontal axis in the pan direction specifies the angle of view of the width of the captured image based on the angle of view information. When the angle of view of the width is 90 [deg], the resolution of the width is 1920 [pix], and the subject's velocity is 192 [pix / s], the relative angular velocity is 9 [deg / s]. Here, the information on the image resolution (length and width) is acquired from the captured image acquired from the camera 200.

[0019] The control unit 108 performs rotation control on the pan head 300 based on the information on the relative angular velocity acquired from the angular velocity calculation unit 107 .

[0020] Next, the hardware configuration of the controller 100 in this embodiment will be described with reference to Fig. 2. The camera 100 in this embodiment has a CPU 1001, a ROM 1002, a RAM 1003, a network I / F 1004, a display unit 1005, an operation unit 1006, and an internal bus 1007 that enables mutual communication.

[0021] The CPU 1001 controls the entire device by controlling each component of the controller 100. The CPU 1001 corresponds to the position detection unit 101, the velocity calculation unit 102, the position prediction unit 103, the determination unit 105, the acquisition unit 106, and the angular velocity calculation unit 107 in FIG.

[0022] The ROM 1002 is used as a permanent storage area for the OS, various programs, and various data, and is also used as a short-term storage area for various data. The ROM 1002 functions as a part of the control unit 108 in FIG.

[0023] The RAM 1003 is a volatile, high-speed storage device such as a DRAM, into which the OS, various programs, and various data are loaded, and which is also used as a working area for the OS and various programs. The RAM 1003 corresponds to the storage unit 104 in FIG.

[0024] The network I / F 1004 is an interface for connecting to the above-mentioned LAN 400, and is responsible for communication with external devices such as the camera 200 and the camera platform 300 via a communication medium such as Ethernet (registered trademark).

[0025] The display unit 1005 is a display such as an LCD for displaying captured images acquired by the camera 200, various settings, and the like.

[0026] The operation unit 1006 is a mouse, a keyboard, or the like for accepting operations from a user and outputting the accepted results to the CPU 1001 .

[0027] Next, using the flowchart in Fig. 4, the control from when the pan head rotation is stopped to when automatic tracking starts in the first embodiment will be described using an example of application to automatic tracking photography of an aircraft. This flowchart is realized by loading an OS, various programs, and various data into a RAM (storage device) that temporarily stores computer programs executed by the CPU 1001, and executing them by the CPU 1001. This flowchart starts with the acquisition of a captured image from the camera 200, and ends when the relative angular velocity between the pan head 300 and the subject becomes 0 and the process transitions to automatic tracking. Also, an image of the operation of the pan head 300 and the camera 200 in the first embodiment is shown using Fig. 4.

[0028] In step S201, a captured image is acquired from the camera 200.

[0029] In step S202, the position detection unit 101 detects the subject position of the tracking target from the captured image acquired from the camera 200. The detected position information of the subject is stored in the RAM 1003.

[0030] In step S203, if the position information of the subject has been detected in a frame prior to the frame in which the position information of the subject was detected in step S202, the process proceeds to step S204, and if not, the process proceeds to step S201. That is, if the position information has been detected in only one frame in step S202, the speed cannot be calculated in step S204, and the process proceeds to step S201.

[0031] In step S204, the speed of the subject is calculated by the speed calculation unit 102 based on the subject position detected in step S202. Specifically, a method is used to calculate the subject speed by linear interpolation using the subject position in the current frame t and the subject position in frame t-1. At this time, the frame used for the speed calculation is determined based on the detection speed of the object detection AI. If the detection speed of the object detection AI is 5 fps, and the time of the current frame t is frame t: 0s, the speed of the subject is calculated based on the subject position in frame t-1: -200 ms. In this embodiment, the speed of the subject is calculated from frame t and frame t-1, but this is not limited to this. For example, when predicting the speed of the subject based on two or more subject positions, the subject speed is set to an approximate value using the least squares method. In addition, in the automatic tracking photography of aircraft takeoff and landing as in this embodiment, the speed of the subject is set to a speed value with the landing direction or takeoff direction being positive. The speed image is illustrated as the subject speed vector in FIG. 5.

[0032] In step S205, the position prediction unit 103 calculates the position coordinates of the subject at a predetermined time ahead based on the speed of the subject calculated in step S204. The calculated position coordinates are stored in the RAM 1003 as a predicted position. In this embodiment, the position of the subject at a predetermined time ahead based on the detection speed of the object detection AI is predicted. For example, in the case of a detection speed of the object detection AI of 5 fps, if the current time is t:0s, the position where the subject will arrive at t+1:200 ms is predicted, and the predicted coordinates are set as the predicted position. At this time, the predicted position is located ahead of the subject position in the traveling direction as shown in FIG. 5. In this embodiment, the position of the subject at a predetermined time ahead according to the detection speed of the object detection AI is set as the predicted position, but this is not limited thereto. For example, the position of the subject at a time ahead of the present time set in advance may be set as the predicted position, or the predetermined time may be determined according to the speed of the subject. When the predetermined time is determined according to the speed of the subject, the predicted position of the subject at a future time is used as the predicted position as the speed of the subject increases.

[0033] In step S206, the determination unit 105 acquires information on the target position from the RAM 1003, and determines whether the predicted position calculated in step S205 matches (passes) the target position. As shown in FIG. 5, it is determined whether the predicted position has reached or passed the target position. If it has reached or passed the target position, the process proceeds to step S207, and if it has not reached the target position, the process proceeds to step S202. The step processing up to this point corresponds to the period indicated by A in FIG. 5, during which the subject position is detected by the detection unit 103, but the head rotation of the camera head 300 remains stopped, and the subject position and the target position are not matched by the head rotation. For this reason, the head does not rotate in the direction opposite to the subject's traveling direction.

[0034] In step S207, when the predicted position reaches or passes the target position, the process transitions to automatic tracking with pan head rotation. Specifically, based on the information on the relative angular velocity calculated by the relative angular velocity calculation unit 107, the control unit 108 transmits a control command for rotation control to the pan head 300 so that the speed of the subject matches the pan head rotation speed and the relative angular velocity becomes zero. When the pan head 300 receives the control command, it accelerates from a stopped state and starts rotating. When the relative angular velocity becomes zero and the speed of the subject matches the rotation speed of the pan head 300, this control flowchart ends, and the process transitions to automatic tracking control in which the subject is tracked at the target position.

[0035] This technology can be applied to both landing and takeoff aircraft automatic tracking photography. In addition, the movement of the subject is not limited to the pan direction. For the tilt direction, which includes up and down movement, automatic tracking control is applied to keep the subject's position in the captured image at the target position.

[0036] This embodiment is particularly effective for automatic tracking and shooting of a subject that moves in one direction, and is also useful for automatic tracking and shooting of an object such as racing or track and field events in which the subject moves in one direction, in addition to automatic tracking and shooting of an aircraft.

[0037] This allows the camera to predict the subject's position after a certain time has passed based on the subject's moving speed, and when the predicted position reaches the target position, it transitions to automatic tracking with camera head rotation, enabling automatic tracking photography with reduced tracking delay.

[0038] In the present embodiment, the controller 100 is described as controlling the camera platform 300, but this is not limiting. For example, an imaging system may be configured with the camera 200 and the camera platform 300. In this case, the functional configuration of the controller 100 is executed by a CPU in the camera 200. Also, in the present embodiment, the camera 200 and the camera platform 300 are described as separate entities, but this is not limiting. For example, a camera having a pan / tilt drive unit may be used. In this case, a control command is output to the camera 200, and drive control is executed by the CPU in the camera 200.

[0039] Second Embodiment In the second embodiment, a method of accelerating more smoothly at the time of transition from A to B shown in Fig. 5 in the first embodiment will be described. This embodiment is particularly effective for a large camera platform that does not allow for rapid rotation or quick control response, or for shooting where sudden accelerated rotation is to be avoided.

[0040] The configuration diagram of the automatic tracking system of the second embodiment is the same as that of the first embodiment. The difference is that there are two position information of the target position 104, a first target position and a second target position. The first target position corresponds to the determination position at which the head rotation starts, similar to the target position of the first embodiment. The second target position in this embodiment corresponds to the determination position at which the acceleration rotation control is switched to the automatic tracking control. In this embodiment, when the subject travels in the forward direction, the second target position is located forward of the first target position.

[0041] Using the flowchart in Fig. 6, the control from the state where the pan head rotation is stopped to the start of automatic tracking in the second embodiment will be described using an example of the case where it is applied to aircraft automatic tracking photography. Also, Fig. 7 shows an image of the operation of the pan head camera 200. This flowchart is realized by the CPU 1001 executing the OS, various programs, and various data loaded into a RAM (storage device) that temporarily stores computer programs executed by the CPU 1001.

[0042] The control in steps S401 to S404 is the same as that in steps S201 to S204 in the first embodiment shown in FIG. 4, and therefore a description thereof will be omitted.

[0043] In step S405, determination unit 105 compares the predicted position with the first target position at which head rotation begins, and in step S406, it is determined whether the predicted position coincides with (reaches) the first target position. In this embodiment, as shown in FIG. 7, it is described as determining whether the predicted position has reached or passed the first target position. If it has reached or passed, the process proceeds to step S407, and if it has not reached, the process returns to step S402. The step processing up to this point corresponds to the period indicated by A in FIG. 7, during which the head rotation of head 300 remains stopped and the head does not rotate in the opposite direction to the traveling direction of the subject.

[0044] In step S407, in response to the fact that the predicted position has reached or passed the first target position, the process shifts to accelerated rotation control accompanied by pan head rotation.

[0045] In step S408, control unit 108 performs accelerated rotation control on camera head 300 based on the information on the relative angular velocity calculated by angular velocity calculation unit 107 so that the subject velocity and the camera head rotation velocity match and the relative angular velocity becomes zero. A large relative angular velocity results in rapid accelerated rotation, so smooth accelerated rotation can be achieved by performing accelerated rotation control at a speed obtained by multiplying the relative angular velocity by an acceleration coefficient of 1 or less. It is sufficient that the acceleration coefficient has a relationship such that the relative angular velocity becomes zero while the subject arrival position moves from the first target position to the second target position. Alternatively, accelerated rotation may be performed at a predetermined constant acceleration until the relative angular velocity becomes zero.

[0046] The control in steps S409 to S411 is the same as that in steps S401 to S404, and therefore a description thereof will be omitted. Since the accelerated rotation of the camera platform has begun, the speed of the subject calculated in S410 gradually decreases.

[0047] In step S412, the determination unit 105 compares the predicted position with a second target position at which automatic tracking is to be started.

[0048] In step S413, it is determined whether the second target position has been reached. As shown in FIG. 5, it is determined whether the predicted position has reached or passed the second target position. If it has reached or passed, proceed to step S414, and if it has not been reached, return to step S408. The step processing up to this point corresponds to the period indicated by A' in FIG. 5, during which the pan-and-tilt camera 200 accelerates and rotates until the relative angular velocity becomes 0. If the relative angular velocity becomes 0 before the predicted position reaches the second target position, a control instruction is issued to drive the pan-and-tilt camera 300 at a constant speed until the predicted position reaches the second target position.

[0049] In step S414, the process shifts to automatic tracking control for tracking the subject at the target position.

[0050] As a result, the position of the subject after a predetermined time has elapsed is predicted based on the moving speed of the subject, and when the subject arrival position reaches the first target position, the control shifts to accelerated rotation control accompanied by pan head rotation. Furthermore, the control gradually accelerates so that the subject speed and the pan head rotation speed match before the predicted position reaches the second target position. As a result, tracking delay is suppressed from the state where the pan head 300 is stopped until the subject reaches the second target position, and smooth tracking video can be captured. [Explanation of symbols]

[0051] 100 Controller 101 Position detection unit 102 Speed ​​calculation section 103 Position Prediction Unit 105 Judgment section 108 Control Unit 200 Cameras 300 Head

Claims

1. A control device having a control means for executing a process for controlling an imaging direction so that a subject to be tracked is kept at a target position in an image captured by an imaging device, a detection means for detecting a position of the subject in each of a plurality of captured images obtained from the imaging device; a prediction means for predicting a position of the subject a predetermined time ahead as a predicted position based on positions of the subject in the plurality of captured images; A control means for executing a process for controlling an imaging direction of the imaging device; having When the predicted position reaches the target position defined by a predetermined pixel area or a predetermined line in the captured image, the control means starts a process for controlling an imaging direction so as to keep the subject at the target position in the captured image. A control device comprising:

2. 2. The control device according to claim 1, wherein said prediction means predicts the position of the subject a predetermined time ahead based on a detection speed at which said detection means detects the position of the subject.

3. 2. The control device according to claim 1, further comprising a calculation unit that calculates a velocity of the subject from the positions of the subject in the plurality of captured images.

4. 4. The control device according to claim 3, wherein said prediction means predicts a position of the subject a predetermined time ahead as a predicted position based on the speed of the subject calculated by said calculation means.

5. A control device having a control means for executing a process for controlling an imaging direction so that a subject to be tracked is kept at a first target position in an image captured by an imaging device, a detection means for detecting a position of the subject in each of a plurality of captured images obtained from the imaging device; a calculation means for calculating a velocity of the subject based on a position of the subject in the plurality of captured images; a prediction means for predicting a position of the subject a predetermined time ahead as a predicted position from the speed of the subject calculated by the calculation means; A control means for executing a process for controlling an imaging direction of the imaging device; a setting means for setting a first target position and a second target position, the first target position and the second target position being defined in advance by a predetermined pixel area or a predetermined line in the captured image; having When the predicted position reaches the second target position, the control means starts a process for controlling an imaging direction so that a speed of the subject and a speed of changing the imaging direction match each other; When the predicted position reaches the first target position, the control means starts a process for controlling an imaging direction so as to keep the subject at the target position in the captured image.

2. The control device according to claim 1, wherein, when a moving direction of a subject is a forward direction, the first target position is located in front of the second target position.

6. An imaging device that captures an image; A camera platform having a driving means for changing an imaging direction of the imaging device; An imaging system including a control device having a control means for causing the driving means to execute a process for controlling an imaging direction, the imaging device has an output unit for outputting a plurality of captured images to the control device, The control means a detection means for detecting a position of the subject in each of a plurality of captured images obtained from the imaging device; a prediction means for predicting a position of the subject a predetermined time ahead as a predicted position based on positions of the subject in the plurality of captured images; A control unit that controls the driving unit of the camera platform to control an imaging direction; A transmission means for transmitting a control instruction of the control means to the camera head; having When the predicted position reaches the target position defined by a predetermined pixel area or a predetermined line in the captured image, the control means transmits a control instruction to the drive means to keep the subject at the target position in the captured image; The camera platform drives the driving means based on the control instruction from the control device.

1. An imaging system comprising:

7. A control method for a control device that executes a process for controlling an imaging direction so that a subject to be tracked is kept at a target position in an image captured by an imaging device, comprising: a detection step of detecting a position of the subject in each of a plurality of captured images acquired from the imaging device; a prediction step of predicting a position of the subject a predetermined time ahead as a predicted position based on the positions of the subject in the plurality of captured images detected in the detection step; a control step of executing a process for controlling an imaging direction of the imaging device; having When the predicted position reaches the target position defined by a predetermined pixel area or a predetermined line in the captured image, the control step starts a process for controlling an imaging direction so that the position of the subject in the captured image remains at the target position. A control method for a control device comprising:

8. A program for causing a computer to execute the control method according to claim 7.

Images