Control device, control method, and program

Abstract

To perform automatic tracking imaging suppressing occurrence of hunting.SOLUTION: A control device executes processing for controlling an imaging direction in such a manner that a subject of a tracking target is caught at a target position in a picked-up image which is picked up by an imaging apparatus. The control device comprises: detection means for detecting a position of the subject in the picked-up image; acquisition means for acquiring information relating to an angle of view from the imaging apparatus; calculation means for calculating a velocity of the subject; relative angle calculation means for calculating a relative angle between the position of the subject and the target position in at least one of panning and tilting directions; angular velocity calculation means for calculating a relative angular velocity between the velocity of the subject and a drive velocity in at least one of the panning and tilting direction in the picked-up image; and control means which executes processing for controlling the imaging direction of the imaging apparatus on the basis of the relative angle and the relative angular velocity.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to tracking technology. [Background technology]

[0002] In recent years, there has been a technology that detects an object from an image captured by an imaging device and controls the imaging direction of the imaging device based on the detection result so that the subject remains within the imaging angle of view. When using such technology in video production, it is important to automatically track and capture the subject.

[0003] Patent Document 1 describes a method of estimating a moving position of a subject from its speed and performing pan-tilt control (hereinafter referred to as PT control) to keep the subject at a target position designated by a user. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 3440916 Summary of the Invention [Problem to be solved by the invention]

[0005] However, when PT control of the camera head is performed so that the position of the subject in the captured image matches the target position, a transient response of the PT control occurs, resulting in hunting, due to the effect of the latency from when the camera head control is performed until the actual operation is completed. [Means for solving the problem]

[0006] In order to solve the above problems, the control device in this embodiment is a control device having a control means that executes processing to control an imaging direction so that a subject to be tracked remains at a target position in an image captured by an imaging device, and has a detection means that detects a position of the subject in each of a plurality of captured images acquired from the imaging device, an acquisition means that acquires information regarding an angle of view from the imaging device, a calculation means that calculates the velocity of the subject from the position of the subject in the plurality of captured images, a relative angle calculation means that calculates a relative angle between the position of the subject and at least one of the pan or tilt directions of the target position based on the position of the subject in the captured images and the target position, an angular velocity calculation means that calculates a relative angular velocity between the velocity of the subject and at least one drive velocity in the pan or tilt direction in the captured images from the information regarding the angle of view and the velocity of the subject, and a control means that executes processing to control the imaging direction of the imaging device based on the relative angle and the relative angular velocity. Effect of the Invention

[0007] According to the present invention, automatic tracking photography that suppresses the occurrence of hunting becomes possible. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a configuration diagram of an automatic tracking system according to a first embodiment; [Diagram 2] FIG. 1 is a block diagram showing an example of the functional configuration of a controller according to a first embodiment; [Diagram 3] FIG. 1 is a diagram for explaining a relative angle according to a first embodiment; [Figure 4] Hardware configuration diagram of a controller according to the first embodiment [Diagram 5] 1 is a flowchart of tracking control of an automatic tracking system according to a first embodiment. [Figure 6] 11 is a flowchart of tracking control of an automatic tracking system according to a second embodiment. [Figure 7] Operational image of the camera platform and camera in the second embodiment DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] (First embodiment) FIG. 1 is a configuration diagram of an automatic tracking system according to a first embodiment. The automatic tracking system according to this embodiment includes a controller 100, an imaging device 200, a camera platform 300, and a LAN 400. The method of connection between the devices is not limited to a specific method. For example, the devices may be connected by a wired cable. In this imaging system, the imaging direction of the camera 200 can be changed by connecting the camera 200 to a camera platform 300 having a drive unit that drives in the horizontal and vertical directions. Furthermore, by issuing a control instruction to the drive unit of the camera platform 300 from the controller 100, it becomes possible to change the imaging direction of the camera 200 remotely. In addition, the controller 100 detects a subject from an image captured by the camera 200, and transmits a control instruction to the camera platform 300 so as to keep the subject at a preset position in the captured image, thereby making it possible to automatically track the subject.

[0010] Next, an example of the functional configuration of the controller 100 will be described with reference to Fig. 2. The controller 100 is composed of a position detection unit 101, a storage unit 102, a determination unit 103, an acquisition unit 104, a relative angle calculation unit 105, a velocity calculation unit 106, a relative angular velocity calculation unit 107, and a control unit 108, and controls the camera platform 300. The main units for realizing this embodiment are the position detection unit 101, the determination unit 103, the acquisition unit 104, the relative angle calculation unit 105, the relative angular velocity calculation unit 107, and the control unit 108, and are executed by a CPU 1001 in Fig. 4 described later.

[0011] The position detection unit 101 acquires a captured image captured by the camera 200. The position detection unit 101 also detects the subject position (center point) of the tracking target in the captured image by a process of detecting an object from the captured image using a learned model (hereinafter referred to as object detection AI). Furthermore, this is stored as position information of the subject in the storage unit 102 described later. The subject position is information expressed in pixels [pix] with the resolution of the captured image as a unit. For example, if the captured image has a FHD resolution of 1920 x 1080 and the lower left is set as the reference position X: 0, Y: 0, when the subject position is located at the center of the image, the position information is X: 960, Y: 540. For example, if the captured image has a FHD resolution of 1920 x 1080 and the lower left is set as the reference position X: 0, Y: 0, when the subject position is located at the center of the image, the position information is X: 960, Y: 540. In this embodiment, the position detection unit 101 detects the position of the subject, but this is not limited to this. For example, the camera 200 itself may perform video analysis to obtain the subject's position information. In this case, the camera 200 detects the subject's position from the captured image, and transmits information about the frame and the subject's position information to the controller 100.

[0012] The storage unit 102 stores position information of a target position that displays a subject to be tracked on the screen. In this embodiment, the target position is a position that is predefined by a specific pixel area or line through which the subject passes. In this embodiment, the target position is described as information expressed in pixels [pix].

[0013] The determination unit 103 acquires information about the target position from the storage unit 102. Furthermore, the determination unit 103 calculates the distance of the subject position relative to the target position in the captured image as the subject distance [pix].

[0014] Acquisition unit 104 acquires, from camera 200, information on the angle of view of the captured image captured by camera 200. Here, the angle of view information refers to information on the angle of view corresponding to the vertical and horizontal widths of the captured image obtained from camera 200, and is information on the imaging range as shown in Fig. 3. The angle of view information may be obtained by acquiring the lens zoom value (lens data) of the camera and converting it into angle of view information by acquisition unit 104. As a result, the acquired information on the angle of view of the vertical and horizontal widths of the captured image is stored in storage unit 102, which will be described later.

[0015] The relative angle calculation unit 105 acquires the angle of view information from the acquisition unit 104 and the subject position from the storage unit 102. Based on the acquired angle of view information and the subject position, the relative angle θ [deg] between the subject position and the pan / tilt direction of the camera platform 300 (camera 200) is converted. The relative angle θ [deg] in this embodiment is described as the angle θ [deg] of the subject position from the left end of the angle of view as shown in FIG. 3, but is not limited thereto. For example, the angle θ [deg] of the subject position from the right end of the angle of view may be used. Specifically, in the case where the horizontal direction is the horizontal axis (pan direction), the angle of view of the width of the captured image is specified from the angle of view information, and the image resolution is specified from the captured image. If the angle of view of the width is 90 [deg], the resolution of the width is 1920 [pix], and the subject distance is 192 [pix], the relative angle is 9 [deg] from the ratio.

[0016] The velocity calculation unit 106 acquires position information from the position detection unit 101. Furthermore, the one-way moving velocity or velocity vector of the subject is calculated as the velocity of the subject from the subject position of one or more frames before the acquired frame and the subject position of the acquired frame. The calculated velocity of the subject is output to the relative angular velocity calculation unit 107. Here, for simplicity, the current frame is frame t, and the frame in which the position is detected one frame before the current frame is frame t-1. In this embodiment, the time required for detection by the object detection AI of the position detection unit 101 is 200 ms per frame. In other words, when the time of the current frame t is 0s, the time of frame t-1 is -200 ms. In this embodiment, the multiple frames used to calculate the subject velocity are determined based on the detection speed of the position detection unit 101, and the unit of the calculated velocity of the subject is [pix / s]. That is, the velocity calculation unit 106 calculates the velocity of the subject using the position information of the subject in the current frame t and the position information of the subject in frame t-1. In the present embodiment, the velocity of the subject is calculated from the position information of the subject in an acquired frame and the position information of the subject acquired prior to that frame, but this is not limited thereto. Any method for calculating the velocity of the subject may be used.

[0017] Relative angular velocity calculation section 107 acquires the angle of view information acquired by acquisition section 104, the subject velocity calculated by velocity calculation section 106, and control information output by control section 108 (described later) to camera head 300. Furthermore, it calculates a relative angular velocity ω indicating the relative velocity between the subject velocity and the pan-tilt velocity calculated based on the control information output to camera head 300.

[0018] Based on the information on the relative angle θ and the relative angular velocity ω, the control unit 108 performs rotation control on the pan head 300. The detailed control method will be described later.

[0019] Next, the hardware configuration of the controller 100 in this embodiment will be described with reference to Fig. 4. The camera 100 in this embodiment has a CPU 1001, a ROM 1002, a RAM 1003, a network I / F 1004, a display unit 1005, an operation unit 1006, and an internal bus 1007 that enables mutual communication.

[0020] The CPU 1001 controls the entire device by controlling each component of the controller 100. The CPU 1001 corresponds to the detection unit 101, the determination unit 103, the acquisition unit 104, the relative angle calculation unit 105, the velocity calculation unit 106, and the relative angular velocity calculation unit 107 in FIG.

[0021] The ROM 1002 is used as a permanent storage area for the OS, various programs, and various data, and is also used as a short-term storage area for various data. The ROM 1002 functions as a part of the control unit 108 in FIG.

[0022] The RAM 1003 is a volatile, high-speed storage device such as a DRAM, into which the OS, various programs, and various data are loaded, and which is also used as a working area for the OS and various programs. The RAM 1003 corresponds to the storage unit 102 in FIG.

[0023] The network I / F 1004 is an interface for connecting to the above-mentioned LAN 400, and is responsible for communication with external devices such as the camera 200 and the camera platform 300 via a communication medium such as Ethernet (registered trademark).

[0024] The display unit 1005 is a display such as an LCD for displaying captured images acquired from the camera 200, various settings, and the like.

[0025] The operation unit 1006 is a mouse, a keyboard, or the like for accepting operations from a user and outputting the accepted results to the CPU 1001 .

[0026] Next, automatic tracking control when this embodiment is applied to automatic tracking photography with an aircraft as a subject will be described with reference to the flowchart of Fig. 5. This flowchart is realized by loading an OS, various programs, and various data into a RAM (storage device) that temporarily stores a computer program executed by the CPU 1001, and executing the program by the CPU 1001. This flowchart also starts by acquiring a captured image from the camera 200 and detecting the subject, and is repeatedly executed until a command to end the user automatic tracking is received.

[0027] In step S201, the position detection unit 101 detects the position of the subject to be tracked from the captured image acquired from the camera 200. This process is repeated until the subject is detected from the captured image. Information regarding the detected subject position is output to the RAM 1003, and the process proceeds to step S202 and step S203.

[0028] In step S202, the distance between the target position and the subject position in the captured image is calculated as the subject distance [pix] in the determination unit 103. The calculated subject distance [pix] is stored in the RAM 1003, and the process proceeds to step S205.

[0029] In step S203, the velocity calculation unit 106 calculates the velocity [pix / s] of the subject based on the amount of change in the subject position in multiple captured images, and the process proceeds to step S207. Specifically, a method is used to calculate the velocity of the subject by linear interpolation based on one or more past subject positions and the current subject position. For simplicity, the current frame is frame t, and the frame in which the position is detected one frame before the current frame is frame t-1. In this embodiment, the time required for detection by the object detection AI of the position detection unit 101 is 200 ms per frame. In other words, when the time of the current frame t is 0s, the time of frame t-1 is -200 ms. In this case, if the velocity of the subject is calculated at two subject positions, the velocity of the subject is calculated based on the subject positions of t-1: -200 ms and t: 0 ms. When the velocity of the subject is predicted based on more subject positions, for example, the least squares method is used for approximation. In automatic tracking photography of aircraft taking off and landing, the velocity of the subject is a velocity value with the landing direction or takeoff direction being positive.

[0030] In step S204, the acquisition unit 104 acquires the field of view information from the camera 200. The acquired field of view information is stored in the RAM 1003.

[0031] In step S205, the relative angle calculation unit 105 converts the subject distance [pix] into a relative angle θ [deg] based on the angle of view information. The relative angle θ indicates the relative angle difference between the subject direction and the imaging direction of the camera 200 (the pan / tilt direction of the camera platform 300). Taking the horizontal direction as the horizontal axis (pan direction) as an example, the angle of view of the width of the captured image is specified from the angle of view information. Also, the image resolution is specified from the captured image. If the angle of view of the width is 90 [deg], the resolution of the width is 1920 [pix], and the subject distance is 192 [pix], the relative angle is 9 [deg] from the ratio. The relative angle θ calculated in step S205 is output to the control unit 108. In this embodiment, the relative angle is calculated as the relative angle of each of the pan and tilt directions, but is not limited to this. At least one of the pan direction or the tilt direction may be calculated as the relative angle.

[0032] In step S206, the control unit 108 multiplies the relative angle θ by a control coefficient Kp to calculate a control angle θ' [deg]. Adjusting the control coefficient Kp to increase the value of the control angle θ' strengthens the motion tracking response that captures the subject at the target position, but makes hunting more likely to occur. Reducing the value of the control angle θ' suppresses the tracking response and makes hunting less likely to occur, but there is a risk of losing tracking of the subject. In other words, increasing the value of the control angle θ' improves subject tracking ability, and controls the position of the subject in the captured image to match the target position.

[0033] In step S207, the relative angular velocity calculation unit 107 converts the subject's velocity [pix / s] into a relative angular velocity ω based on the angle of view information. The relative angular velocity ω indicates the difference between the subject's velocity and the camera head rotation speed. Taking an example where the pan direction is the horizontal axis, the horizontal angle of view of the captured image is specified based on the angle of view information. If the horizontal angle of view is 90 [deg], the horizontal resolution is 1920 [pix], and the subject's velocity is 192 [pix / s], the relative angular velocity is 9 [deg / s]. The relative angular velocity obtained by the conversion is output to the control unit 108.

[0034] In step S208, the control unit 108 multiplies the relative angular velocity by a control coefficient Kd to obtain a control angular velocity ω' [deg / s]. Increasing the value of the control coefficient Kd strengthens the response that matches the subject's speed and the camera head rotation speed. Reducing the value of the control coefficient Kd suppresses the response. This is a particularly effective method for matching the camera head rotation speed to the subject's speed when automatically tracking a subject with a constant moving speed. In other words, increasing the value of the control angular velocity ω' matches the driving speed of the subject and the PT, making it possible to capture video with little change between captured images.

[0035] In step S209, the control unit 108 calculates a pan head rotation acceleration value V by adding together the control angle θ' and the control angular velocity ω' obtained in steps S206 and S208, and outputs the calculated value as a control command to the pan head 300 to control the rotation of the pan head 300. Here, the control period will be described as the time interval during which control is performed. Specifically, the controller 100 outputs the pan head rotation acceleration value V calculated at the time interval of the control period T to the pan head 300 via the camera 200, and the pan head is controlled. V=(Kd×ω)T+(Kp×θ)=ω′T+θ′(T is the control period of the pan head 300) (Formula 1)

[0036] Although the change in the relative angle θ is affected by the latency between the control of the rotation of the pan head 300 and the completion of the actual operation, the relative angular velocity ω is an angular velocity predicted from the amount of change between one or more past subject positions and the current subject position, and therefore the effects of latency can be suppressed.

[0037] In other words, by combining control for matching the subject position in the captured image with the target position, and control for matching the subject speed with the pan head rotation speed, the effect of latency between the pan head 300 rotation control and the completion of the actual operation is suppressed. This enables automatic tracking shooting with suppressed hunting.

[0038] In this embodiment, the control coefficients Kp and Kd are described as being constant values, but are not limited to this. For example, they may be determined by referring to a table corresponding to the subject speed.

[0039] Second embodiment In the first embodiment, the control coefficients Kp and Kd are predetermined values, whereas in the second embodiment, the control coefficients Kp and Kd are changed according to the distance between the subject position and the target position. In this embodiment, the weighting of the control coefficients Kp and Kd is changed depending on whether the distance between the target position and the subject position is within a predetermined range, thereby enabling smoother automatic tracking.

[0040] In the second embodiment, the determination unit 103 further determines whether the subject is within a predetermined range based on the target position and the position of the subject. Here, the predetermined range in this embodiment is a specific pixel area indicated by the target position 503 shown by the dotted line in Fig. 7, and will be described as a range in which the subject is kept in a captured image when automatic tracking control is being performed or a range in which the subject is determined to be lost.

[0041] Next, the control of the camera platform 300 in the second embodiment will be described with reference to the flowchart in Fig. 6. Also, an image of the operation of the camera 200 and the camera platform 300 in the second embodiment will be shown with reference to Fig. 7. This flowchart is realized by loading the OS, various programs, and various data into a RAM (storage device) that temporarily stores a computer program executed by the CPU 1001, and executing the program by the CPU 1001. Also, this flowchart starts by acquiring a captured image from the camera 200 and detecting a subject, and is executed repeatedly until a command to end automatic tracking is received.

[0042] In step S301, determination unit 103 determines whether the subject position is within a predetermined range. If the subject position is not within the predetermined range, the process proceeds to step S302. If the subject position is within the range, the process proceeds to step S304.

[0043] In step S302, the control unit 108 calculates a rotation control command for the pan head 300 with a larger value of the control coefficient Kp, and outputs a control instruction to the pan head 300. At this time, the control image shown in A of Fig. 7 is obtained, and control is performed so that the subject position and the target position approach each other. After transmitting the control instruction to the pan head 300, the process proceeds to step S306.

[0044] In step S303, it is determined whether the relative angular velocity ω is greater than a predetermined value. Here, the predetermined value is described as a value arbitrarily set by the user, but is not limited to this. For example, it may be a value previously set on the controller 100 side. If it is greater than the predetermined value, the process proceeds to step S304, and if it is equal to or less than the predetermined value, the process proceeds to step S305.

[0045] In step S304, the control unit 108 calculates a rotation control command for the pan head 300 with a larger value of the control coefficient Kd, and outputs a control instruction to the pan head 300. At this time, the control image shown in B in FIG. 7 is obtained, the relative angular velocity ω converges to 0, and the speed of the subject and the rotation speed of the pan head are controlled to match. During this period, the effect of latency is small, and the occurrence of hunting can be suppressed. After sending the control command, the process proceeds to step S306.

[0046] In step S305, the control unit 108 calculates a rotation control command for the pan head 300 with an increased ratio of the control coefficient Kp, and outputs a control instruction to the pan head 300. At this time, the control image shown in C in Fig. 7 is obtained. As a result, the relative angle θ converges to 0, and the subject is captured and tracked at the center of the target position. After transmitting the control command, the process proceeds to step S306.

[0047] In step S306, if the controller 100 receives an end command for the automatic tracking, the process ends this control flow chart, otherwise the process proceeds to step S301.

[0048] In this embodiment, the control according to the above control flow chart has been described, but the present invention is not limited to this. For example, in step S305, a control command that balances the control coefficients Kp and Kd may be calculated. In practice, the balance between the control coefficients Kp and Kd is tuned according to the characteristics of the movement of the target object to be automatically tracked.

[0049] In this way, when combining control that reduces the amount of deviation between the subject and target position to zero and control that reduces the amount of deviation between the subject's speed and the pan head's rotation speed to zero, the values ​​of the control coefficients Kp and Kd are switched depending on whether the subject's position is within a specified range. Then, in order to track the subject at the target position, first the ratio of the control coefficients is adjusted so that the relative angular velocity ω converges to zero, and then the ratio of the control coefficients is adjusted so that the relative velocity θ converges to zero, thereby enabling automatic tracking shooting with suppressed hunting.

[0050] In this embodiment, the control coefficient Kp and the control coefficient Kd are described as being changed from a predetermined value to a larger value, but this is not limited thereto. For example, they may be changed from a predetermined value to a smaller value. In this case, the value of Kd is changed to a smaller value in step S302, the value of Kp in step S304, and the value of Kd in step S305, and the value of the control angle or the control angular velocity in Equation 1 is changed. Also, the control coefficient Kd or the control coefficient Kp may be determined by referring to a table according to the subject speed.

[0051] In the first and second embodiments, the camera 200 and the camera platform 300 are described as separate entities, but this is not limiting. For example, the camera may have a pan / tilt drive unit. In this case, a control command is output to the camera 200, and the drive control is executed by a CPU in the camera 200. [Explanation of symbols]

[0052] 100 Controller 101 Detection unit 102 Storage section 103 Judgment section 104 Acquisition Department 105 Relative angle calculation unit 106 Speed ​​calculation section 107 Relative angular velocity calculation unit 108 Control Unit 200 Cameras 300 Head

Claims

1. A control device having a control means for executing a process for controlling an imaging direction so that a subject to be tracked is kept at a target position in an image captured by an imaging device, a detection means for detecting a position of the subject in each of a plurality of captured images obtained from the imaging device; an acquisition means for acquiring information regarding an angle of view from the imaging device; a calculation means for calculating a velocity of the subject from a position of the subject in the plurality of captured images; a relative angle calculation means for calculating a relative angle between the position of the subject and at least one of a pan direction and a tilt direction of the target position based on the position of the subject in the captured image and the target position; an angular velocity calculation means for calculating a relative angular velocity between the velocity of the subject and at least one driving velocity in a pan or tilt direction in the captured image based on information about the angle of view and the velocity of the subject; a control means for executing a process for controlling an imaging direction of the imaging device based on the relative angle and the relative angular velocity; A control device comprising:

2. 2. The control device according to claim 1, wherein the control means executes a process for controlling an imaging direction of the imaging device in accordance with a control angle calculated based on the relative angle and a first control coefficient, and a control angular velocity calculated based on the relative angular velocity and a second control coefficient.

3. 3. The control device according to claim 2, further comprising a determination unit for determining whether the position of the subject is within a range of the target position that is set in advance within a predetermined range.

4. 4. The control device according to claim 3, wherein when said determining means determines that the position of said subject is not within said range of said target position, a value of said first control coefficient is increased.

5. When the determining means determines that the position of the object is within the range of the target position and when the relative angular velocity is greater than a predetermined value, the value of the second control coefficient is increased; 5. The control device according to claim 4, wherein when the determination means determines that the position of the subject is within the range of the target position and when the relative angular velocity is smaller than a predetermined value, the value of the first control coefficient is increased.

6. A control device having a control means for executing a process for controlling an imaging direction so that a subject to be tracked is kept at a target position in an image captured by an imaging device, a detection step of detecting a position of the subject in each of a plurality of captured images acquired from the imaging device; acquiring information regarding an angle of view from the imaging device; a calculation step of calculating a velocity of the subject from positions of the subject in the plurality of captured images; a relative angle calculation step of calculating a relative angle between the position of the subject and at least one of a pan direction and a tilt direction of the target position based on the position of the subject in the captured image and the target position; an angular velocity calculation step of calculating a relative angular velocity between the velocity of the subject and at least one driving velocity in a pan or tilt direction in the captured image based on information about the angle of view and the velocity of the subject; a control step of executing a process for controlling an imaging direction of the imaging device based on the relative angle and the relative angular velocity; A control method for a control device comprising the steps of:

7. A program for causing a computer to execute the control method according to claim 6.

Images