Imaging control apparatus, imaging control method, and program

Abstract

To provide a technique for suppressing a subject from being lost by controlling an imaging range on the basis of an angle of the subject.SOLUTION: A control apparatus comprising: a calculation unit configured to calculate an angle of an object from a part of the object detected from a captured image captured by an image capturing unit and a predetermined reference line; and a control unit configured to perform PTZ control on the image capturing unit based on the angle of the object calculated by the calculation unit and a parameter related to a preset angle.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an imaging control device, an imaging control method, and a program that automatically tracks a subject. [Background technology]

[0002] There is a technique for detecting the position of a subject from an image captured by an imaging device, and controlling the imaging direction of the imaging device based on the detection result so as to keep the subject within the imaging screen.

[0003] For example, Patent Document 1 discloses a method of performing pan / tilt control (tracking) so as to keep a subject within a captured image by calculating the moving speed from the subject position and estimating the subject moving position. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 4709101 Summary of the Invention [Problem to be solved by the invention]

[0005] However, if the subject's behavior deviates from prediction, such as if the subject suddenly changes trajectory, the method described in Patent Document 1 may delay pan / tilt drive in response to the subject's sudden change in behavior. For example, an aircraft may enter a steep descent if its fuselage falls below a certain angle relative to the horizon, making it difficult to predict its behavior and continue capturing images of the subject using the method described in Patent Document 1. Furthermore, if the subject being tracked cannot be detected from the captured video, it becomes difficult to continue tracking (the subject may be lost). Therefore, the present invention provides a technology for controlling the imaging range based on the subject's angle and preventing the subject from being lost. [Means for solving the problem]

[0006] One aspect of the present invention is characterized by comprising: a calculation means for calculating a part of a subject detected from an image captured by an imaging means and an angle of the subject from a predetermined reference line; and a control means for performing PTZ control on the imaging means based on the angle of the subject calculated by the calculation means and a parameter related to a preset angle. [Effects of the Invention]

[0007] According to the present invention, the imaging range is controlled based on the angle of the subject, and the subject is prevented from being lost. [Brief explanation of the drawings]

[0008] [Figure 1] System configuration diagram. [Figure 2] 2 is a block diagram showing an example of the hardware configuration of each of the controller 100 and the camera 200. [Figure 3] FIG. 2 is a block diagram showing an example of the functional configuration of a controller 100. [Figure 4] UI diagram of the tracking function, which tracks according to the target position setting. [Figure 5] 4 is a flowchart of a tracking process according to the first embodiment. [Figure 6] (a) is an illustration showing the detection frame of the detected aircraft part superimposed. (b) is an illustration showing the detection frame used to calculate the angle of the subject. (c) is an illustration showing how the angle of the subject is calculated when the reference line is set along the horizontal axis. (d) is an illustration showing how the angle of the subject is calculated when the reference line is set along the vertical axis. [Figure 7] 10 is a flowchart of a tracking process according to a second embodiment. [Figure 8] 10 is a flowchart of a tracking process according to a third embodiment. [Figure 9] An illustration showing the detection frames for the subject's body parts superimposed when the subject is a motorcycle. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] [Embodiment 1] FIG. 1 is a configuration diagram of an auto-tracking camera system according to a first embodiment of the present invention. The auto-tracking camera system includes a controller 100, a camera 200, and a network 300. The connection between these devices is not limited to a specific method. For example, the devices may be connected via a network such as a wireless LAN, or may be connected via a wired cable. In this embodiment, the camera 200, which has a drive unit that drives in the horizontal and vertical directions, can change its imaging direction by receiving control instructions from the controller 100 or by following instructions preset by an auto-tracking function or the like. Similarly, the imaging angle of view can be changed by driving the lens of the camera 200. Furthermore, the controller 100 detects a subject from an image captured by the camera 200 and transmits control instructions to the camera 200 to continuously capture the subject at a preset position (target position) in the captured image, thereby enabling automatic tracking of the subject.

[0011] (Controller hardware configuration) 2 is a block diagram showing the hardware configuration of the controller 100 according to this embodiment. The controller 100 includes a CPU 101, a RAM 102, a ROM 103, an image input I / F (interface) 104, a display unit 105, a network I / F 106, an operation I / F 107, and an internal bus 108 that enables mutual communication.

[0012] The CPU 101 is a central processing unit that controls the controller 100. The CPU 101 corresponds to an acquisition unit 111, a control unit 112, a part detection unit 114, and an angle calculation unit 115 in FIG.

[0013] RAM 102 is a storage device such as a DRAM (Dynamic Random Access Memory) that temporarily stores computer programs executed by CPU 101. RAM 102 also provides a work area into which the OS, various programs, and various data are loaded and which CPU 101 uses when executing processing. RAM 102 is also used as a work area for the OS and various programs. RAM 102 functions as part of control unit 118 in FIG. 3, which will be described later.

[0014] The ROM 103 is a non-volatile storage device such as a flash memory, HDD, SSD (Solid State Drive), SD card, etc. The ROM 103 is used as a permanent storage area for storing the OS, various programs, various data, and other programs used by the CPU 101 to control the controller 100, and is also used as a short-term storage area for various data. The ROM 103 corresponds to the storage unit 113 in FIG. 3, which will be described later.

[0015] The image input I / F 104 is an interface for receiving captured images from a camera 200 (described later), and is configured using SDI or HDMI (registered trademark).

[0016] The network interface (I / F) 106 is a network interface that communicates with external devices via a LAN (Local Area Network), and is responsible for communication with terminals (devices) via a communication medium such as Ethernet. When communication with an external device is performed based on an instruction from the CPU, it is performed via the network I / F 106.

[0017] The display unit 106 is configured, for example, with a liquid crystal panel and a backlight, and displays captured images and setting screens acquired from the terminal. In this embodiment, the display unit 106 displays captured images captured by the camera 200 and their settings. Note that, although an example is shown in this embodiment in which the controller 100 has the display unit 106, this configuration is not limiting. For example, a configuration in which a display monitor that displays captured images and a controller are separately present may also be used.

[0018] The operation I / F 107 is an interface for receiving operations from the user on the controller, and is an interface for inputting operation information from a plurality of buttons, joystick, and the like provided on the housing of the controller 100. The user operates operation members such as a joystick, which is an operation interface, or a GUI (Graphical User Interface) displayed on the display unit, to input operation information to the control unit. This allows the settings of the controller itself and control of the terminal connected to the controller.

[0019] (Camera hardware configuration) Next, the hardware configuration of the camera 200 according to this embodiment will be described with reference to FIG.

[0020] The camera 200 has a CPU 201, a RAM 202, a ROM 203, an imaging unit 204, a drive unit 205, an imaging optical system 206, an image processing unit 207, an image output interface (I / F) 208, a network I / F 209, and an internal bus 210 that enables mutual communication.

[0021] Camera 200 can output images captured by imaging unit 204 to an external device via a network or an image cable (not shown). Camera 200 also has a tracking function that automatically tracks a subject.

[0022] The CPU 201 is a central processing unit that controls the camera.

[0023] The RAM 202 is a storage device such as a DRAM (Dynamic Random Access Memory) that temporarily stores computer programs executed by the CPU 201. The RAM 202 also provides a work area into which the OS, various programs, and various data are loaded and which the CPU 201 uses when executing processing. It is also used as a work area for the OS and various programs.

[0024] The ROM 203 is a non-volatile storage device such as a flash memory, HDD, SSD (Solid State Drive), SD card, etc. The ROM 203 is used as a permanent storage area for storing the OS, various programs, various data, and other programs used by the CPU 201 to control the camera 200, and is also used as a short-term storage area for various data.

[0025] The imaging unit 204 includes a drive unit 205, an imaging optical system 206, and an image processing unit 207, which will be described later. The drive unit 205 drives a pan-tilt (PT) mechanism to change the imaging direction. The drive unit 205 also drives the imaging optical system 206 in the optical axis direction to enable zoom (Z) operation for changing the imaging angle of view.

[0026] The drive unit 205 is configured, for example, by a mechanical drive system that performs pan and tilt operations and a drive source motor, and is configured by an actuator such as a stepping motor and an encoder that detects pan and tilt positions. When the drive unit 205 receives a control instruction from the CPU 201 requesting information about the pan or tilt drive position of the drive unit 205, the drive unit 205 outputs information about the drive position corresponding to the control instruction. The drive unit 205 also drives the imaging unit 204 so that the pan and tilt values ​​correspond to the motor and the PTZ drive command from the CPU 201.

[0027] The imaging optical system 206 drives the imaging unit 204 to achieve a zoom value according to a PTZ drive command from the CPU 201. The imaging optical system 206 is a lens that focuses light from a subject on the imaging surface of an imaging element, and is composed of, for example, a zoom lens, a focus lens, and a shake correction lens. Furthermore, the zoom value (angle of view) can be changed by driving the imaging optical system 206 in the optical axis direction. The imaging element (not shown) of the imaging unit 204 captures an image of the subject and generates a captured image. The imaging element (not shown) of the imaging unit 204 converts the light from the subject focused by the imaging optical system 206 into an electrical signal for each pixel. Furthermore, an amplifier (not shown) amplifies the electrical signal converted by the imaging element and outputs it to an image processing unit 207, which will be described later. The imaging element is, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor.

[0028] The image processing unit 207 converts the electrical signal amplified by an amplifier (not shown) into a predetermined format, compresses it as necessary, and transfers it to the RAM 203. The image processing unit also performs image quality adjustment when acquiring a captured image, and cropping to cut out only a predetermined area of ​​the image data.

[0029] In this embodiment, the imaging unit 204 and the driving unit 205 are described as being integrated, but the imaging unit 204 and the driving unit 205 may be separate and detachable, like a camera that is mounted on a pan head and changes the imaging direction. Furthermore, while the imaging unit 204 and the imaging optical system 206 are described as being integrated, the imaging unit 204 and the imaging optical system 206 may be separate and detachable, like a lens-interchangeable camera.

[0030] The image output I / F 208 is an interface for outputting captured images to the outside. For example, it is configured by an SDI (Serial Digital Interface) or an HDMI (High-Definition Multimedia Interface) (registered trademark). Here, the image output I / F is connected to the image input I / F of the controller.

[0031] The network I / F 209 is an I / F for connecting to the above-mentioned network 300, and is responsible for communication with external devices such as the controller 100 via a communication medium such as Ethernet (registered trademark). Here, it has been described that the camera 200 is remotely controlled via the network I / F 209, but it may also be controlled via another I / F such as a serial communication I / F (not shown).

[0032] (Controller function configuration) Next, an example of the functional configuration of the controller 100 will be described with reference to Fig. 3. The controller 100 is made up of an acquisition unit 111, a control unit 112, a storage unit 113, a part detection unit 114, and an angle calculation unit 115.

[0033] The acquisition unit 111 acquires the captured image captured by the camera 200. The acquisition unit 111 also estimates and detects the type of tracking target in the captured image and the subject position (center point) by processing the captured image to detect objects using a trained model, and sets this as subject information. In this embodiment, the types of tracking targets include vehicles, humans, aircraft, ships, and animals. Among the subject information, information related to position is expressed in pixels (pix) according to the video resolution. For example, if the captured image has an FHD resolution of 1920 x 1080 and the reference position X:0, Y:0 is at the bottom left, the position information when the subject position is located in the center of the image is X:960, Y:540.

[0034] In this embodiment, the acquisition unit 111 detects the position of the subject, but this is not limiting. For example, the position information of the subject may be acquired from the camera 200. In this case, the camera 200 detects the position of the subject from the captured image, and transmits information about the frame and the position information of the subject to the controller 100.

[0035] The control unit 112 generates a control instruction for controlling any or all of the pan, tilt, and zoom driving of the camera 200 based on the subject angle parameters that can be taken by the subject, which are preset for each type of subject, and outputs the generated control instruction to the camera 200.

[0036] Storage unit 113 stores angle parameters relating to the angle of the subject that have been set in advance. In this case, the angle parameters are stored by controller 100, but this is not limitative. The angle parameters may be acquired when connected to camera 200, or camera 200 may output the angle parameters to controller 100 together with the captured image.

[0037] The part detection unit 114 detects parts that make up the subject detected by the acquisition unit 111. The "parts" referred to here are elements that characterize the subject, and the parts are detected based on a model previously trained by machine learning. For example, if the subject is an airplane, the main wings and tail fins are characteristic elements, and if the subject is a motorcycle, the tires, body, and pilot are characteristic elements. In this embodiment, parts are detected using machine learning, but this is not limited to this. For example, it is also possible to use a method that divides the subject into multiple segments using image processing technology, or a method that extracts feature points from an image and identifies parts. In this case, the parts to be detected may be determined based on the type of subject identified when the acquisition unit 102 detects the position information of the subject.

[0038] The angle calculation unit 115 calculates the angle of the subject from the position information of two or more constituent parts of the subject detected by the part detection unit 114. This angle may be a two-dimensional angle or a three-dimensional angle within the angle of view, as long as it allows the orientation of the subject to be estimated. Specifically, the angle of the subject is calculated as an acute angle between a pre-set (predetermined) reference line and a line connecting at least two or more parts of the subject. In this case, the reference line is set in advance as the horizontal or vertical axis of the captured image, but is not limited to this. For example, the horizon in real space may be used as the reference line, and this may be set in advance by calculation from the pan / tilt angle of the camera 200, acquiring the direction of travel of the aircraft to be tracked by taking a picture of the runway in advance, or calculation in virtual space. Specific methods for calculating the angle will be described later.

[0039] In this embodiment, automatic tracking is achieved by the controller 100 outputting control instructions, including drive control, to the camera 200, but the controller 100 and the camera 200 may be integrated. Also, some of the functions of the controller 100 described above may be executed by the camera 200.

[0040] (Tracking processing) The basic automatic tracking process (automatic tracking mode) in this embodiment will be described with reference to FIG. 4 . A rectangular frame 402 in a captured image 400 is a rectangular frame corresponding to the rectangular frame information of the subject 401. A position 411 indicates the "center coordinates of the subject 401" detected by the acquisition unit 111 in the captured image of the subject 401. A position 420 indicates a target position for tracking photography. In this embodiment, while the automatic tracking process is being performed, the controller 100 controls the drive unit 205 and the imaging optical system 206 of the camera 100 so that the center coordinates of the tracked subject in the captured image approach the target position 420. While the target position 420 is the center position of the captured image in FIG. 4 , the target position 420 may be any position or size on the captured image. The user can set the target position 420 to any position or size. In this case, the user sets the target position 420 beforehand when the automatic tracking process is performed, and then the automatic tracking process is performed.

[0041] (How to set the detection angle) A method for setting angle parameters in this embodiment will be described. In this embodiment, the user can set and change parameters related to the angle of the subject, and if the user does not set them, angle parameters that are predefined as initial settings will be automatically applied.

[0042] The user can select a method for setting the angle parameters. For example, when "1. Automatic Setting (Auto)," "2. Manual Setting (Manual)," "3. Initial Setting," etc. are displayed on the display unit 105, the user can select a method for setting the angle parameters via the operation I / F 107. The angle parameters set at this time are angle settings that allow tracking to continue using normal tracking processing. In addition, at this time, parameters are determined for each type of subject (vehicle, human, aircraft, ship, animal, etc.). For example, as a result of the user selecting the target history information, (type, angle setting A) is set to (vehicle, 0° to 5°), (human, 80° to 110°), (aircraft, -5° to 20°).

[0043] The storage unit 113 stores, for each subject, information on the angle of the subject specified by the setting method selected according to each setting method selection screen.

[0044] If the user selects the setting method "1. Automatic Setting (Auto)," the user can set the angle parameters based on, for example, history information related to past tracking functions. In other words, if a subject is lost during a past automatic tracking operation, the angle parameters are determined from the angle information of the subject before the loss. For example, if the user selects one or more history records in which a subject was lost from the history information of past automatic tracking operations, the angle parameters are determined from the history information so that the angle before the loss does not include the angle parameters.

[0045] If the user selects "2. Manual Setting", the user can set the angle parameters of the subject by inputting numerical values.

[0046] When the user selects the setting method "3. Initial setting", the angle parameters of the subject stored in advance in the storage unit 113 are applied.

[0047] The parameter relating to the angle of the subject is selected or set by the user as an angle at which the subject can be tracked, but is not limited to this. For example, an angle at which it is estimated that tracking the subject will be difficult may be set as a parameter.

[0048] Furthermore, although the parameters relating to the angle of the subject are set for each type of subject, this is not limiting and may be set according to the state of the subject, for example. In this case, the angle parameters applied when the aircraft is preparing to take off or is scheduled to take off may be different from the angle parameters applied when the aircraft is preparing to land.

[0049] (Tracking processing based on subject angle) Next, automatic tracking control when this embodiment is applied to automatic tracking photography with an aircraft as the subject will be described using the flowchart in Figure 5. This flowchart is realized by loading an OS, various programs, and various data into RAM 102 (a storage device) that temporarily stores computer programs executed by CPU 101, and executing them by CPU 101. This flowchart also starts by acquiring a captured image from camera 200 executing automatic tracking mode and detecting the subject, and is executed repeatedly until a command to end automatic tracking is received from the user.

[0050] In step S501, the acquisition unit 111 detects the type (aircraft), status (takeoff / landing), and position of the object to be tracked from the captured image acquired from the camera 200. At this time, the status is detected based on the type of object. For example, if the type of object is an aircraft, the status of the aircraft is determined from the captured image. In this embodiment, when the object is an aircraft, the status of the object is determined from the orientation of the high-lift device (flap) captured in the captured image, but this is not limited thereto. For example, the status of the object may be determined based on the captured image and information about the aircraft received from an air traffic control tower or the like. This process is repeated until the object is detected from the captured image. Information about the detected type, status, and position of the object (object information) is output to the control unit 112, and the process proceeds to step S502. At this time, the type of object is detected, but this is not limited thereto. For example, if the camera is installed at an airport and captures images of only specific objects such as aircraft, the type of object does not need to be detected. In this case, information about the object's position is detected and output to the control unit 112, and the process proceeds to step S502.

[0051] In step S502, part detection unit 114 detects parts that make up the subject detected in step S501. FIG. 6(a) shows image data in which aircraft 602, the subject in captured image 601, is detected. Also in FIG. 6(a), part detection frames 603-607 corresponding to information on the parts of the subject detected by part detection unit 114 illustrate the detection results of parts of aircraft 302. Part detection frame 303 illustrates the nose, part detection frame 304 the fuselage, part detection frame 305 and part detection frame 306 the main wings, and part detection frame 307 the tail, with two-point coordinate data at the top left and bottom right of each frame. In this embodiment, the part detection frame is a rectangle consisting of two-point coordinate data, but the shape and number of coordinate points of the part detection frame are not limited. After part detection unit 114 detects the parts, the process proceeds to step S503.

[0052] In step S503, angle calculation unit 115 calculates the angle of aircraft 602 from part detection frames 603 to 607 detected in step S502. Here, a method for calculating the angle of the subject from part detection frame 603 and part detection frame 604 will be described. The angle of the subject is calculated from two parts and a reference line determined for each type of subject. The parts used to calculate the angle will be described as being set in advance. For example, if the subject is an aircraft, the angle is calculated using a detection frame that detects the nose and fuselage, and if the subject is a vehicle, the angle of the subject is calculated using a detection frame that detects the front and rear wheels. In this way, it is assumed that the combination of parts used to calculate the angle of the subject is set in advance for each type of subject.

[0053] FIG. 6(b) shows an excerpt of the aircraft 602 and part detection frames 603 and 604 from FIG. 6(a). Center coordinates 612 and 613 of each part are calculated from two-point coordinate data 608 and 609 of part detection frame 603 and two-point coordinate frames 610 and 611 of part detection frame 604. The center coordinates 612 are calculated as ((X1+X2) / 2, (Y1+Y2) / 2), and the center coordinates 613 are calculated as ((X3+X4) / 2, (Y3+Y4) / 2). At this time, coordinates other than the center may be calculated for each part being imaged. Next, an angle Φ616 is calculated, which is the angle formed by a line 614 connecting the calculated center coordinates 612 and 613 and a dotted line 615 indicating the horizontal line that serves as the reference for the horizontal plane in the captured image. Here, angle Φ616 is calculated as the angle between a line calculated from the subject's location and the reference line, as in Figure 6(c) or 6(d). The dotted line in Figures 6(c) and 6(d) is the reference line, and the line is calculated from the nose of the aircraft and the frame of the moving object. When the reference line is in the horizontal axis direction, as in Figure 6(c), the downward side is taken as a negative value, and the calculation is made as a value between -180° and 180°. Furthermore, the angle of the subject is calculated so that it becomes an acute angle. When the reference line is in the vertical axis direction, as in Figure 6(d), the left side is taken as a negative value, and the calculation is made as a value between -180° and 180°.

[0054] Angle Φ616 can be calculated as follows from coordinates 617 ((X1+X2) / 2, (Y3+Y4) / 2)) on dotted line 615 that forms a triangle with coordinates 612 and 613.

[0055]

number

[0056] This angle Φ616 is the aircraft angle (angle of the subject).

[0057] Here, the reference horizontal line (dotted line 615) may be the horizon line, and is set in advance by calculation from the pan / tilt angle of camera 200, acquiring the direction of travel of the aircraft to be tracked by photographing the runway in advance, calculation in virtual space, etc. After calculating the angle of the subject in step S503, the calculation result is output to control unit 112, and the process proceeds to step S504.

[0058] In step S504, the control unit 112 determines whether the aircraft angle Φ616 calculated in step S503 is within the range of angle setting A, which is the angle setting set in the memory unit 113. Here, the parameters related to the angle of the aircraft 602 set in the memory unit 113 in this flow are shown in Table 1.

[0059] [Table 1]

[0060] In this embodiment, angle setting A is set to a range of angles determined based on the state of the subject, such as takeoff, landing, or flight. During takeoff, if the angle is within the range of angle setting A, the process proceeds to step S505, and if it is outside the range of angle setting A, the process proceeds to step S506. At this time, the range of angle setting A may be specified by the user, or may be determined from the average angle taken by the subject in each state. Alternatively, angle A may be determined according to the type of subject. If it is determined in step S504 that the angle is within the range of angle setting A, the process proceeds to step S505, and if it is determined that the angle is outside the range of angle setting A, the process proceeds to step S506.

[0061] In step S505, the control unit 112 generates a PTZ control instruction to continue normal tracking processing, and then outputs the generated PTZ control instruction to the camera 200, and the process proceeds to step S507.

[0062] In step S506, the control unit 112 generates a PTZ control instruction to zoom out from the current angle of view. If the aircraft angle 616 is not within the range of angle setting A set in the storage unit 113, there is a possibility that normal tracking processing will not be able to keep up with the subject's behavior and continue tracking the subject. In this case, the control unit 112 generates a PTZ control instruction to zoom out from the current angle of view. This zoom out may be performed by zooming out by a predetermined factor from the current angle of view, or may be performed within a range of sizes that allow tracking of the subject. Furthermore, if it is not possible to zoom out any further from the current angle of view, the user may be notified by displaying on the display unit 105 that the maximum angle of view has been reached. At this time, the user may be notified by sound. Furthermore, the control unit 112 outputs the generated PTZ control instruction to the camera 200, and the process proceeds to step S507.

[0063] In step S507, this flow ends if the controller 100 receives an automatic tracking end command input by the user via the operation I / F 107. If an end command has not been received, the flow proceeds to step S501, and this flow is repeated.

[0064] Although the present embodiment has been described as an example in which the present invention is applied to a video in which a subject is continuously shot in a predetermined composition by automatic tracking, the present invention is not limited to video. For example, the present invention can also be applied to automatic tracking photography of an aircraft during landing or takeoff, or photography during flight. Furthermore, the angle detection direction of the subject is not limited to the pitch angle.

[0065] This allows the aircraft angle of the subject to be detected and the image to be captured at an angle of view that corresponds to the angle of the subject, preventing the subject from being lost from the captured image even if a sudden change in flight occurs, and enabling continuous automatic tracking and shooting.

[0066] This embodiment is particularly effective for automatic tracking and photography of a subject that moves in one direction, and is also useful for racing, track and field events, and other events where the subject moves in one direction, in addition to automatic tracking and photography of aircraft.

[0067] [Embodiment 2] In the first embodiment, camera 200 is controlled based on angle setting A, which is a preset angle parameter for the subject, and the angle of the subject. In the second embodiment, in addition to angle setting A, which is estimated to enable tracking of the subject, angle parameters (angle settings) are set for angle settings which are estimated to possibly not be able to track the subject using normal tracking processing, and camera 200 is controlled according to each angle setting. The setting method for angle settings which are estimated to possibly not be able to track the subject using normal tracking processing is set in advance using the setting method described above.

[0068] 7 is a flowchart according to the second embodiment. This flowchart is implemented by loading an OS, various programs, and various data into RAM 102 (a storage device) that temporarily stores a computer program executed by CPU 101, and executing the program by CPU 101. This flowchart starts by acquiring a captured image from camera 200 and detecting a subject, and is executed repeatedly until a command to end automatic tracking is received from the user.

[0069] In step S701, the control unit 112 determines whether the aircraft angle calculated in S504 corresponds to the predefined angle setting B-1. The angle parameters applied in this embodiment are shown in Table 2.

[0070] If the angle setting B-1 is applicable, the process proceeds to step S702. If the angle setting B-1 is not applicable, the process proceeds to step S703.

[0071] [Table 2]

[0072] If it is within the range of angle setting B-1, the process proceeds to step S702. If it does not correspond to angle setting B-1, the process proceeds to step S703. In this embodiment, the angle setting for which normal tracking processing cannot be performed is set to angle setting B, and angle setting B is determined as two parameters, angle setting B-1 and angle setting B-2, but angle setting B may be two or more. If there are two or more angle settings B, the same angle setting determination process as in step S701 is repeatedly executed to determine which of the angle settings the aircraft angle corresponds to.

[0073] Angle setting B-1 is determined to correspond to the case where the angle of the aircraft is greater than 20°. In other words, if it is determined to be greater than the maximum value of angle setting A, the process proceeds to step S702. At this time, the aircraft, which was in a takeoff state, is now tilted further upward.

[0074] Angle setting B-2 is determined when the aircraft angle is less than -5°. In other words, the process transitions to step S703 when it is determined that the angle is less than the minimum value of angle setting A. At this time, the aircraft is in a state where the aircraft, which was in a takeoff state, is tilted downward.

[0075] In step S702, the control unit 112 calculates a PTZ control command based on angle setting B-1. Angle setting B-1 is an angle at which a subject taking off may rise suddenly. Therefore, if it is determined in step S701 that the angle setting corresponds to B-1, the control unit 112 generates a PTZ control command in step S702 to zoom in from the current angle of view. Since the aircraft 602 making a sudden climb is moving away from the camera 200 installed on the ground, its size within the current angle of view may become smaller than that of the current angle of view, and it may become impossible to recognize it as a subject. Therefore, the aircraft 602 performs PTZ control to zoom in so that the subject appears larger relative to the current angle of view. The control unit 112 outputs the generated PTZ control command to the camera 200, and the process proceeds to step S507.

[0076] In step S703, the control unit 112 calculates a PTZ control command based on angle setting B-2. Angle setting B-2 is a state in which a sudden descent of the subject, such as a crash, may occur. With the current angle of view, a sudden descent of the airplane may cause the subject to move out of the angle of view. Therefore, a PTZ control command is generated to zoom out the aircraft 602 from the current angle of view. The control unit 112 outputs the generated PTZ control command to the camera 200, and the process proceeds to step S507.

[0077] As described above, by further distinguishing between angle parameters that are estimated to make it difficult to continue tracking using normal tracking processing, PTZ control that is more suited to the aircraft's situation becomes possible.

[0078] [Embodiment 3] In the first embodiment, the camera 200 was controlled based on an angle setting estimated to enable tracking of a subject, but in the third embodiment, the camera 200 is further controlled based on an amount of angle change per predetermined time. In this embodiment, the description will be given assuming that the parameters (angle settings) related to the angle change used for control are set in advance. As in the first embodiment, the setting method allows the user to set and change the parameters related to the angle change, and if the user does not set them, the parameters that are predefined as initial settings will be automatically applied.

[0079] 8 is a flowchart relating to automatic tracking in embodiment 3. This flowchart is realized by loading an OS, various programs, and various data into RAM 102 (storage device) that temporarily stores computer programs executed by CPU 101, and executing them by CPU 101. This flowchart also starts by acquiring a captured image from camera 200 and detecting a subject, and is repeatedly executed until a command to end automatic tracking is received from the user.

[0080] In step S801, the angle calculation unit 115 calculates the amount of angle change based on the aircraft angle calculated using a captured image taken before the captured image used for the calculation in step S503 and the expected angle calculated in step S502. Specifically, captured images are sequentially transmitted from the camera 200 in accordance with a predetermined frame rate. Furthermore, since the angle calculation unit 115 calculates the aircraft angle for the sequentially transmitted captured images, the amount of angle change per time can be calculated from the angle A1 calculated at time t and the angle A0 calculated at time t-1. At this time, if the angle A0 has not been calculated at time t-1, such as when the subject is detected for the first time, step S801 is skipped and the process proceeds to step S507. After calculating the amount of angle change, the angle calculation unit 115 proceeds to step S802.

[0081] In step S802, if the angle change amount calculated in step S801 corresponds to a predefined setting related to angle change, the process proceeds to step S505. If the aircraft angle change amount does not correspond to a predefined setting related to angle change, the process proceeds to step S506.

[0082] By performing PTZ control based on the amount of change in angle as described above, it becomes possible to quickly perform PTZ control in response to changes in the angle of the subject.

[0083] [Other variations] Using Figure 9, tracking processing based on the angle of the subject will be explained using an example in which the subject is a motorcycle. Figure 9(a) is image data in which a motorcycle 902, the subject, traveling horizontally in the X direction is detected in a captured image 901 captured by camera 200. Part detection frames 903 to 906 illustrate the detection results of the parts of motorcycle 902 detected by part detection unit 114, with part detection frame 903 being the front wheel, part detection frame 904 being the body, part detection frame 905 being the rear wheel, and part detection frame 906 being the driver. In this embodiment, the part detection frame is a rectangle made up of the coordinates of two points, but the shape of the detection object frame and the number of coordinate points are not limited.

[0084] The angle with respect to a subject such as a motorcycle or vehicle is calculated based on the front and rear wheel locations. For example, if the subject is a motorcycle, angle calculation unit 115 calculates angle 909 formed by horizontal line 908 and line 907 connecting the center points of object detection frame 903 indicating the front wheel and object detection frame 905 indicating the rear wheel.

[0085] As shown in Figure 9(b), if the angle 909 formed by the line 907 and the horizontal line 908 is greater than the set angle, PTZ control is performed to zoom in or out from the current angle of view rather than normal tracking processing. At this time, if the center point of the detection frame for the rear wheel is higher in the Y-axis direction than the front wheel, that is, if the rear wheel of the motorcycle is higher than the front wheel, a fall is predicted, so PTZ control is performed to zoom in. On the other hand, if the center point of the detection frame for the front wheel is higher in the Y-axis direction than the rear wheel, that is, if the front wheel of the motorcycle is higher than the rear wheel, there is a possibility of acceleration in the X-axis direction, so PTZ control is performed to zoom out.

[0086] As described above, even if the subject is not an aircraft, by specifying the part to be used in calculating the angle of the subject for each type of subject, it is possible to perform PTZ control appropriately based on the condition of the subject.

[0087] The numerical values, processing timing, processing order, processing subject, data (information) configuration / acquisition method / sending destination / sending source / storage location, etc. used in each of the above embodiments are given as examples to provide a concrete explanation, and are not intended to be limited to these examples.

[0088] In addition, some or all of the above-described embodiments may be used in appropriate combination, and some or all of the above-described embodiments may be selectively used.

[0089] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0090] The invention of this specification includes the following control device, control method, and computer program. [Explanation of symbols]

[0091] 100 Controllers 111 Acquisition Department 112 Control Unit 113 Storage section 114 Part detection unit 115 Angle calculation unit 204 Imaging unit

Claims

1. a calculation means for calculating an angle of the object from a part of the object detected from the image captured by the imaging means and a predetermined reference line; and a control means for performing PTZ control on the imaging means based on the angle of the subject calculated by the calculation means and a parameter related to a preset angle. A control device characterized by:

2. The control device according to claim 1 , wherein the reference line is defined based on the horizontal axis or the vertical axis of the captured image.

3. 2. The control device according to claim 1, further comprising a detection unit for detecting at least two or more parts of the subject in the captured image.

4. 4. The control device according to claim 3, wherein the calculation means calculates the angle of the subject from a combination of predetermined parts from among the two or more parts.

5. an automatic tracking mode in which PTZ control is performed on the imaging means so that the subject is continuously imaged at a predetermined size; During the automatic tracking mode, If the angle of the subject is within the range of the parameter, the control means performs PTZ control on the imaging means so as to continue imaging the subject at the predetermined size; If the angle of the subject is not within the range of the parameter, the control means performs PTZ control on the imaging means so as to continue imaging the subject at a size larger or smaller than the predetermined size.

2. The control device according to claim 1.

6. During the automatic tracking mode, When the angle of the subject is smaller than the minimum value of the range of the parameter, the control means performs PTZ control on the imaging means so as to continue imaging the subject at a size smaller than the predetermined size.

6. The control device according to claim 5.

7. During the automatic tracking mode, When the angle of the subject is larger than the maximum value of the range of the parameter, the control means performs PTZ control on the imaging means so as to continue to image the subject at a size larger than the predetermined size.

6. The control device according to claim 5.

8. the parameter is a parameter relating to the amount of change in angle per unit time, the calculation means calculates a change in the angle of the object based on a first angle of the object and a second angle calculated before the first angle; the control means performs PTZ control on the imaging means based on the angle change amount and the parameter.

2. The control device according to claim 1.

9. the subject is an aircraft, and the calculation means calculates the angle of the subject from the nose and the moving object, which are the parts detected by the detection means; 4. The control device according to claim 3.

10. the subject is a motorcycle or a vehicle, and the calculation means calculates the angle of the subject from the front wheel and rear wheel, which are the parts detected by the detection means; 4. The control device according to claim 3.

11. a calculation step of calculating an angle of the subject from a part of the subject detected from an image captured by the imaging means and a predetermined reference line; a control step of performing PTZ control on the imaging means based on the angle of the subject calculated in the calculation step and a parameter related to a preset angle. A control method comprising:

12. A computer program for causing a computer to function as each of the means of the control device according to any one of claims 1 to 10.

Images