Camera attitude control device and camera attitude control system equipped therewith, camera attitude control method, camera attitude control program

Abstract

This invention provides a camera attitude control device that enables intuitive operation of a camera capable of pan / tilt operation, as well as a camera attitude control system, camera attitude control method, and camera attitude control program equipped therewith. [Solution] The mobile terminal 20 is a camera attitude control device that controls the attitude of a camera driven in the pan and tilt directions, and comprises an attitude detection unit 21 and a PTZ command generation unit 22. The attitude detection unit 21 acquires rotation information measured by an inertial measuring device (IMU). The PTZ command generation unit 22 controls the attitude of the PTZ camera 10 based on the rotation information acquired by the attitude detection unit 21.

Description

Technical Field

[0001] The present disclosure relates to, for example, a camera attitude control device that performs shooting such as video, a camera attitude control system including the same, a camera attitude control method, and a camera attitude control program.

Background Art

[0002] In recent years, for example, in a shooting studio, a conference hall, an event venue, a sports facility, etc., a remote camera that performs shooting while changing the shooting direction and shooting position by remote control is used. For example, Patent Document 1 discloses a camera position estimation unit that estimates a camera position with respect to a sphere using a camera image of the sphere and radius information of the sphere, and a camera position with respect to the sphere estimated by the camera position estimation unit, and a camera direction estimation unit that estimates a camera direction with respect to the sphere using reference information composed of a feature point of the sphere and its three-dimensional coordinates.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, the above conventional camera attitude estimation device has the following problems. That is, it was difficult to intuitively operate a camera capable of pan / tilt operation with the camera attitude estimation device disclosed in the above publication. An object of the present disclosure is to provide a camera attitude control device that can intuitively operate a camera capable of pan / tilt operation, a camera attitude control system including the same, a camera attitude control method, and a camera attitude control program. [Means for solving the problem]

[0005] The camera attitude control device according to this disclosure controls the attitude of a camera driven in the pan and tilt directions, and comprises a measurement information acquisition unit and an attitude control unit. The measurement information acquisition unit acquires rotation information measured by an inertial measuring device. The attitude control unit controls the attitude of the camera based on the rotation information acquired by the measurement information acquisition unit. [Effects of the Invention]

[0006] The camera attitude control device described herein allows for intuitive operation of a camera capable of pan / tilt control. [Brief explanation of the drawing]

[0007] [Figure 1] An external view showing the configuration of a PTZ camera operated by a portable terminal (camera attitude control device) according to one embodiment of the present disclosure. [Figure 2] A control block diagram showing the basic configuration of the PTZ camera in Figure 1. [Figure 3] Figure 1 is a control block diagram showing the configuration of a camera attitude control system comprising a PTZ camera and a portable terminal (camera attitude control device) for operating the PTZ camera. [Figure 4] Figure 3 is a flowchart showing the processing flow of camera attitude control by the mobile terminal (camera attitude control device). [Figure 5A] A flowchart showing the flow of the PTZ command generation process (absolute position control) generated in the camera attitude control process shown in Figure 4. [Figure 5B] A flowchart showing the flow of the PTZ command generation process (speed control) in the camera attitude control process shown in Figure 4. [Figure 6] A control block diagram showing the configuration of a camera attitude control device located within a PTZ camera according to another embodiment of the present disclosure. [Figure 7A]Figure 1 is a front view showing the PTZ camera positioned along the horizontal plane. [Figure 7B] Figure 1 is a front view showing the PTZ camera positioned on an inclined surface. [Figure 8] Figure 7 is a flowchart showing the processing flow of self-position estimation and camera attitude control by the camera attitude control device. [Figure 9] A control block diagram showing the configuration of a camera attitude control system comprising a portable terminal (camera attitude control device) and a PTZ camera according to yet another embodiment of the present disclosure. [Figure 10] Figure 9 is a flowchart showing the flow of attitude control processing and self-position estimation processing by the camera attitude control device. [Figure 11] A control block diagram showing the configuration of a camera attitude control device located within a PTZ camera according to yet another embodiment of the present disclosure. [Figure 12] A control block diagram showing the configuration of a mobile terminal (camera attitude control device) according to yet another embodiment of this disclosure. [Figure 13] A control block diagram showing the configuration of a camera attitude control device located within a PTZ camera according to yet another embodiment of the present disclosure. [Modes for carrying out the invention]

[0008] The embodiments will be described in detail below, with reference to the drawings as appropriate. However, unnecessary details may be omitted. For example, detailed explanations of already well-known matters and redundant explanations of substantially identical configurations may be omitted. This is to avoid the following explanation becoming unnecessarily verbose and to facilitate understanding for those skilled in the art. The applicant provides the accompanying drawings and the following description so that a person skilled in the art can fully understand the disclosure, and not intends to limit the subject matter described in the claims.

[0009] (Embodiment 1) A camera attitude control system including a mobile terminal (camera attitude control device) 20 (see FIG. 3) according to an embodiment of the present disclosure and a PTZ camera 10 operated by the mobile terminal 20 will be described as follows with reference to FIGS. 1 to 5B. (1) Configuration of the PTZ camera 10 The PTZ (Pan / Tilt / Zoom) camera 10, which is an operation target of the mobile terminal (camera attitude control device) 20 according to the present embodiment, is an imaging device driven in the pan direction / tilt direction / roll direction / zoom direction. For example, it is installed in a photography studio, a conference hall, an event venue, a sports facility, etc., and is controlled by a controller such as a PC (Personal Computer) to perform various shootings. As shown in FIG. 1, the PTZ camera 10 performs shooting while switching the shooting direction in the pan direction, tilt direction, and roll direction, respectively. Further, as shown in FIG. 1, the PTZ camera 10 incorporates a lens unit LU including a plurality of optical lenses in the camera head unit 11, and controls the relative positions of the optical lenses to change the shooting range (zoom range) and the focus position to perform shooting.

[0010] As shown in FIGS. 1 and 2, the PTZ camera 10 includes a base unit 10a, a turning unit 10b, a camera head unit 11, a rotation mechanism 11a, an imaging element 12, a lens driving unit (zoom mechanism) 13, a lens control unit 14, a pan direction driving mechanism 15a, a tilt direction driving mechanism 15b, and a pan / tilt direction control unit 16. As shown in FIG. 1, the base unit 10a is a member that constitutes the lower part of the PTZ camera 10, and is configured such that the turning unit 10b is turned relative to the base unit 10a by the pan direction driving mechanism 15a.

[0011] As shown in FIG. 1, the turning unit 10b is an arm-shaped member attached to the base unit 10a so as to be turnable, and is connected to the base unit 10a at its lower part and supports the camera head unit 11 in a rotatable state at its upper part. The camera head unit 11 is a substantially cylindrical member that encloses a lens unit LU including a plurality of optical lenses and an imaging device 12. As shown in FIG. 1, the camera head unit 11 is driven to pivot in the pan direction by a pan direction drive mechanism 15a and to change its orientation vertically in the tilt direction by a tilt direction drive mechanism 15b. Further, as shown in FIG. 1, the camera head unit 11 is driven to rotate in the roll direction by a rotation mechanism (not shown).

[0012] Thereby, it is possible to perform imaging while switching the imaging direction (pan / tilt / roll direction) by the PTZ camera 10. The rotation mechanism 11a rotationally drives the camera head unit 11 about the roll axis X shown in FIG. 1. Thereby, it is possible to perform imaging while rotating the video when imaging a subject. As shown in FIG. 2, the imaging device 12 is provided in the camera head unit 11, converts the light incident through a lens unit LU including a plurality of optical lenses such as a zoom lens into a signal, and generates digital image data of the subject.

[0013] The lens drive unit 13 is enclosed in the camera head unit 11 and drives to change the relative positions of a lens unit LU including a plurality of optical lenses such as a zoom lens. Thereby, it is possible to adjust the focus position with respect to the subject and to change the imaging range (zoom range) including the subject. The lens control unit 14 controls the lens drive unit 13 so as to adjust the imaging range and the focus position by changing the relative positions in the optical axis direction of a plurality of lenses such as a zoom lens and a focus lens included in the lens unit LU.

[0014] As shown in FIG. 1, the pan direction drive mechanism 15a pivotally drives the camera head unit 11 and the swivel unit 10b with respect to the base unit 10a. Thereby, it is possible to perform imaging while changing the imaging direction in the left-right direction with respect to the subject. As shown in Figure 1, the tilt drive mechanism 15b rotates the camera head 11 vertically relative to the swivel section 10b. This allows for shooting while changing the shooting direction vertically relative to the subject.

[0015] The pan / tilt direction control unit 16 controls the pan direction drive mechanism 15a and the tilt direction drive mechanism 15b so that when the zoom position (shooting range) changes due to the lens drive unit 13 controlled by the lens control unit 14, smooth footage is shot while the subject remains within the field of view. (2) Overall configuration of the mobile terminal 20 The mobile terminal (camera attitude control device) 20 according to this embodiment is, for example, a smartphone or a tablet terminal, which is connected to the PTZ camera 10 described above via a network and is used as a controller to operate the PTZ camera 10. As shown in Figure 3, the mobile terminal 20 includes an attitude detection unit (measurement information acquisition unit) 21, a PTZ command generation unit (attitude control unit) 22, a PTZ command transmission unit (communication unit) 23, a display control unit 24, and a display unit 25.

[0016] The orientation of the PTZ camera 10 controlled by the mobile terminal 20 includes at least one of the position (XYZ coordinates) and rotation angle of the PTZ camera 10. The attitude detection unit (measurement information acquisition unit) 21 is an inertial measurement unit (IMU) that detects three-dimensional inertial motion, and detects the movement (rotation and acceleration) of the mobile terminal 20 by measuring the tilt and acceleration of three axes.

[0017] The PTZ command generation unit (attitude control unit) 22 controls the attitude of the PTZ camera 10 based on the rotation and acceleration of the mobile terminal 20 detected by the attitude detection unit 21. Specifically, the PTZ command generation unit 22 generates control commands to change the attitude of the PTZ camera 10 in accordance with the attitude of the mobile terminal 20 (at least one of rotation and acceleration).

[0018] This allows a user possessing the mobile device 20 to intuitively operate the PTZ camera 10 by using the mobile device 20 as a controller and changing the orientation of the mobile device 20. The PTZ command transmission unit (communication unit) 23 transmits the PTZ commands generated by the PTZ command generation unit 22 to the PTZ camera 10. The PTZ commands transmitted from the PTZ command transmission unit 23 are control commands that control the attitude of the PTZ camera 10.

[0019] The display control unit 24 controls the display unit 25 to display, for example, various images and information received by the mobile terminal 20 via the network. The display control unit 24 also controls the display unit 25 to display, for example, images captured by the camera function provided in the mobile terminal 20. Furthermore, the display control unit 24 controls the display unit 25 to display images captured by the PTZ camera 10.

[0020] The display unit 25 is, for example, a touch-panel type liquid crystal display panel or an organic EL (Electro-Luminescence) panel provided on the outer surface of the mobile terminal 20, and is controlled by the display control unit 24. Here, the display control unit 24 controls the display screen of the display unit 25 to blink when the PTZ camera 10 reaches the end of at least one of the drive ranges in the pan direction and tilt direction. The display control unit 24 also controls the display screen of the display unit 25 to change to a warning color when the PTZ camera 10 reaches the end of the drive range in at least one of the pan direction and tilt direction.

[0021] This allows the user of the mobile terminal 20 to be notified via the display screen of the display unit 25 when the PTZ camera 10 reaches the end of its drive range in the pan or tilt direction, providing intuitive feedback that it can no longer be driven. Note that the display control by the display control unit 24 may be performed, for example, when the pan / tilt direction control unit 16 of the PTZ camera 10 recognizes that it is "driven within a range where it will not rotate further". Alternatively, it may be performed when a response is received that the current position of the PTZ camera 10 in the pan / tilt direction is located at an end in a preset driving range, after inquiring about the current position of the PTZ camera 10 in the pan / tilt direction.

[0022] Also, by displaying the live video of the PTZ camera 10 on the display screen of the display unit 25 of the mobile terminal 20, it is possible to view the live video of the remotely located PTZ camera 10 in real time with an operation feeling as if shooting with the camera function built into the mobile terminal 20 at hand. <Control Method of PTZ Camera 10> The control method of the PTZ camera 10 of the present embodiment will be described as follows using the flowchart shown in FIG. 4.

[0023] That is, in step S11, the attitude detection unit 21 of the mobile terminal 20 acquires measurement information (rotation angle, acceleration) from the IMU. Next, in step S12, the PTZ command generation unit 22 of the mobile terminal 20 generates a control command (PTZ command) for controlling the PTZ camera 10 based on the rotation information (rotation angle) included in the measurement information acquired in step S11.

[0024] Next, in step S13, the PTZ command transmission unit 23 of the mobile terminal 20 transmits the control command (PTZ command) generated in step S12 to the PTZ camera 10. Next, in step S14, it is determined whether to end the process as it is. Here, if the process is to be ended, it ends as it is, and if not, it proceeds to step S15.

[0025] Next, in step S15, since it was determined in step S14 that the process would not terminate, it is determined whether the pre-set processing interval time has elapsed. If it is determined that the processing interval time has elapsed, the process returns to step S11 and the transition process is repeated. On the other hand, if it is determined that the processing interval time has not elapsed, the process returns to step S14 and it is determined whether or not to terminate the process.

[0026] The control commands transmitted to the PTZ camera 10 can vary depending on the product, but in the PTZ camera 10 of this embodiment, for example, control using the HTTP protocol may be adopted as the control command for controlling the attitude of the PTZ camera 10. Here, there are two main types of control in the pan / tilt direction: (1) absolute position control and (2) velocity control.

[0027] In controlling the PTZ camera 10 of this embodiment, for example, the PTZ control command conversion process is performed according to the flowcharts shown in Figures 5A and 5B. In other words, as shown in Figure 5A, in the case of absolute position control command conversion, in step S21, the PTZ command generation unit 22 of the mobile terminal 20 performs absolute position control command conversion from the measurement information (rotation angle) detected by the attitude detection unit 21.

[0028] Next, in step S22, the PTZ command generation unit 22 clips the PTZ control range. Next, in step S23, the PTZ command generation unit 22 generates a URL that includes the rotation angle. As a result, the PTZ command transmission unit 23 sends a URL containing the rotation angle generated in the PTZ command generation unit 22 to the PTZ camera 10, allowing the mobile terminal 20 to intuitively control what is being sent to the PTZ camera 10.

[0029] On the other hand, as shown in Figure 5B, in the case of speed control command conversion, in step S31, the PTZ command generation unit 22 of the mobile terminal 20 performs speed control command conversion from the measurement information (rotation angle) detected by the attitude detection unit 21. Next, in step S32, the PTZ command generation unit 22 calculates the amount of change from the formula change amount = (rotation angle - previous rotation angle).

[0030] Next, in step S33, the PTZ command generation unit 22 calculates the control speed from the formula control speed = change amount × coefficient. Next, in step S34, the control speed is clipped within the PTZ control range. Next, in step S35, the PTZ command generation unit 22 generates a URL that includes the control speed.

[0031] Next, in step S36, the PTZ command generation unit 22 calculates the previous rotation angle from the formula: previous rotation angle = rotation angle. As a result, the PTZ command transmission unit 23 sends a URL including the control speed generated in the PTZ command generation unit 22 to the PTZ camera 10, allowing the mobile terminal 20 to intuitively control what is sent to the PTZ camera 10.

[0032] Furthermore, by performing speed control command conversion, the PTZ camera 10 can be controlled more smoothly. As a variation of the control mechanism, a dead zone may be provided in which the amount of operation (change) of the mobile terminal 20 is considered to be zero when the change is small. In this case, an effect similar to image stabilization can be obtained.

[0033] <Key Features> The portable terminal (camera attitude control device) 20 of this embodiment is a camera attitude control device that controls the attitude of a camera driven in the pan and tilt directions, as described above, and comprises an attitude detection unit 21 and a PTZ command generation unit 22, as shown in Figure 3. The attitude detection unit 21 acquires rotation information of the portable terminal 20 measured by an inertial measurement unit (IMU). The PTZ command generation unit 22 controls the attitude of the PTZ camera 10 based on the rotation information acquired by the attitude detection unit 21.

[0034] This allows for intuitive operation of the PTZ camera 10, which is capable of pan / tilt control, using mobile terminals 20 connected to each other via a network. (Embodiment 2) A camera attitude control device 120 for operating a PTZ camera 110 according to another embodiment of this disclosure will be described below with reference to Figures 6 to 8.

[0035] In addition, components having the same function as those described in Embodiment 1 above are denoted by the same reference numerals, and their detailed descriptions are omitted. The camera attitude control device 120 in this embodiment differs from Embodiment 1, in that it is located inside the portable terminal 20 used to remotely operate the PTZ camera 10, as shown in Figure 6.

[0036] Furthermore, the camera attitude control device 120 is not limited to being located inside the PTZ camera 110; for example, it may be integrated with the PTZ camera 110 using a connecting jig or the like. In other words, as shown in Figure 6, the camera attitude control device 120 of this embodiment is installed inside the PTZ camera 110 (for example, in the base portion 10a) and comprises an attitude detection unit 121, a PTZ command generation unit 122, and a self-position estimation unit 124.

[0037] The attitude detection unit 121, similar to the attitude detection unit 21 described in Embodiment 1 above, is an inertial measurement unit (IMU) that detects three-dimensional inertial motion, and detects the movement (rotation and acceleration) of the PTZ camera 110 by measuring the tilt and acceleration of three axes. Here, the PTZ camera 110 installed on the inclined surface shown in Figure 7B can be operated intuitively in the same way as above, by correcting the pan / tilt / roll control amount according to the inclination angle of the PTZ camera 110 at the time of installation, compared to controlling the PTZ camera 110 installed on the horizontal surface shown in Figure 7A.

[0038] For example, as shown in Figure 7B, if the angle of the inclined surface is θ, then when moving the PTZ camera 110 by a distance A in the horizontal direction, it is controlled to move by a distance of Asinθ in the tilt direction and Acosθ in the pan direction. This allows the PTZ camera 110 to be controlled to correct its orientation during installation, even when it is installed on an inclined surface, so that it can capture images along the horizontal direction.

[0039] Furthermore, in the case of a PTZ camera 110 in which the camera head 11 is driven in the roll direction by a rotation mechanism 11a, as in this embodiment, by rotating it by an angle θ in the roll direction, it is possible to capture a well-aligned image along the horizontal direction even when the PTZ camera 110 is installed on an inclined surface. In this case, when the PTZ camera 110, which is installed at an angle to the horizontal plane, is operated dynamically, for example, by mounting it on a crane, the angle θ of the inclined surface changes moment by moment. Therefore, by continuously detecting the angle θ of the inclined surface using an inertial measurement unit (IMU) or SLAM (Simultaneous Localization Mapping), etc., and appropriately controlling the roll, the PTZ camera 110 can be operated intuitively.

[0040] The PTZ command generation unit (posture control unit) 122 controls the posture of the PTZ camera 110 based on the rotation and acceleration of the PTZ camera 110 detected by the posture detection unit 121. Specifically, for example, the PTZ command generation unit 122 generates a control command so as to correct the posture of the PTZ camera 110 in accordance with the posture at the time of installation of the PTZ camera 110. Then, the PTZ command generation unit 122 transmits the generated control command to the pan / tilt direction control unit 16.

[0041] As a result, the PTZ camera 110 controls its posture in the pan / tilt direction based on the control command generated in the camera posture control device 120. The self-position estimation unit 124 acquires the acceleration included in the measurement information measured by the posture detection unit 121 using an inertial measurement unit (IMU), and estimates the position of the PTZ camera 110 based on the acquired acceleration. Specifically, the self-position estimation unit 124 double-integrates the measurement result (acceleration) in the inertial measurement unit (IMU) acquired by the posture detection unit 121 to calculate the position coordinates of the PTZ camera 110.

[0042] Note that the self-position estimated by the self-position estimation unit 124 includes at least one of the position (XYZ coordinates) and angle of the PTZ camera 110. <Control method of PTZ camera 110> The control method of the PTZ camera 110 according to this embodiment will be described as follows using the flowchart shown in FIG. 8.

[0043] That is, in step S41, the posture detection unit 121 of the camera posture control device 120 acquires measurement information (rotation angle, acceleration) from the IMU. Next, in step S42, the PTZ command generation unit 22 of the camera posture control device 120 generates a control command (PTZ command) for controlling the PTZ camera 110 based on the rotation information (rotation angle) included in the measurement information acquired in step S41.

[0044] Next, in step S43, the self-position estimation unit 124 of the camera attitude control device 120 calculates the position coordinates of the PTZ camera 110 by integrating the acceleration included in the measurement information acquired in step S41 twice. Next, in step S44, the PTZ command transmission unit 23 of the camera attitude control device 120 transmits the control command (PTZ command) generated in step S42 and the position coordinates of the PTZ camera 110 calculated in step S43 to the PTZ camera 110.

[0045] Next, in step S45, it is determined whether or not to terminate the process. If the process is to be terminated, it will end; otherwise, the process proceeds to step S46. Next, in step S46, since it was determined in step S45 that the process would not terminate, it is determined whether or not the pre-set processing interval time has elapsed. If it is determined that the processing interval time has elapsed, the process returns to step S41 and repeats the transition process. On the other hand, if it is determined that the processing interval time has not elapsed, the process returns to step S45 and determines whether or not to terminate the process.

[0046] The control commands transmitted to the PTZ camera 110 can vary depending on the product. In the PTZ camera 110 of this embodiment, as in Embodiment 1 above, control using the HTTP protocol may be adopted as the control command for controlling the attitude of the PTZ camera 10. (Embodiment 3) A camera attitude control system comprising a mobile terminal (camera attitude control device) 220 and a PTZ camera 10 operated by the mobile terminal 220, according to yet another embodiment of this disclosure, will be described below with reference to Figures 9 and 10.

[0047] In addition, components having the same function as those described in Embodiments 1 and 2 above are denoted by the same reference numerals, and their detailed descriptions are omitted. Furthermore, the portable terminal (camera attitude control device) 220 according to this embodiment is equipped with a display control unit 24 and a display unit 25, similar to the portable terminal 20 shown in Figure 2, but for the sake of convenience of explanation, they are not shown and their descriptions are omitted.

[0048] The portable terminal (camera attitude control device) 220 of this embodiment is similar to Embodiment 1 in that it is located outside the PTZ camera 10 (on the portable terminal 220), as shown in Figure 9. However, it differs from Embodiment 1 in that it includes an imaging unit 224, an attitude estimation unit (SLAM (Simultaneous Localization Mapping)) 225, and an integrated attitude estimation unit 226 for estimating the attitude of the portable terminal 220.

[0049] As shown in Figure 9, the portable terminal (camera attitude control device) 220 of this embodiment includes an attitude detection unit (IMU) 21, a PTZ command generation unit 22, a PTZ command transmission unit 23, an imaging unit 224, an attitude estimation unit (SLAM) 225, an integrated attitude estimation unit 226, and a self-position estimation unit 227. The imaging unit 224 is provided, for example, to photograph a subject on the back side of the mobile terminal 220, opposite to the operating surface, and captures video and images of the subject by operating the mobile terminal 220.

[0050] The imaging unit 224 may also be an imaging device provided for photographing subjects on the operating surface side of the mobile terminal 220. The attitude estimation unit (SLAM) 225 estimates the position of the mobile terminal 220 using multiple images captured by the imaging unit 224. More specifically, the attitude estimation unit 225 acquires feature points contained in the multiple images captured by the imaging unit 224 and estimates the self-position of the mobile terminal 220.

[0051] The integrated attitude estimation unit 226 integrates the attitude information of the mobile terminal 220 detected by the attitude detection unit (IMU) 21 and the attitude information of the mobile terminal 220 estimated by the attitude estimation unit (SLAM) 225 to estimate the attitude of the mobile terminal 220. Note that the integrated posture estimation unit 226 performs the most probable posture estimation from the results estimated by a plurality of different methods, for example, using a Kalman filter or the like.

[0052] By doing so, compared with the result estimated using only the detection result in the posture detection unit (IMU) 21, the estimation results in the posture estimation unit (SLAM) 225 are combined and integrated, so that the influence of drift can be eliminated and accurate posture estimation can be performed. That is, when performing posture estimation using only the detection result in the posture detection unit (IMU) 21, although it can be implemented with a simple configuration, there is a possibility that drift due to error accumulation may occur during long-term continuous operation.

[0053] Therefore, in the mobile terminal 220 of the present embodiment, by integrating, by the integrated posture estimation unit 226, the posture information estimated from the information of the feature points included in the plurality of images captured by the imaging unit 224 and the posture information detected by the inertial measurement unit (IMU) in the posture detection unit 21, it is possible to perform highly accurate posture estimation while taking advantage of the merits of each posture detection and posture estimation.

[0054] The self-position estimation unit 227 acquires the acceleration included in the measurement information measured by the inertial measurement unit (IMU) by the posture detection unit 21, and estimates the position of the mobile terminal 220 based on the acquired acceleration. Specifically, the self-position estimation unit 227 double-integrates the measurement result (acceleration) in the inertial measurement unit (IMU) acquired by the posture detection unit 21 to calculate the position coordinates of the mobile terminal 220.

[0055] Note that the self-position estimated by the self-position estimation unit 227 includes at least one of the position (XYZ coordinates) and angle of the PTZ camera 10. <Control method of PTZ camera 10> The control method of the PTZ camera 10 of the present embodiment will be described as follows using the flowchart shown in FIG. 10.

[0056] In other words, in step S51, the attitude detection unit 21 of the mobile terminal 220 acquires measurement information (rotation angle, acceleration) from the IMU. Next, in step S52, the imaging unit 224 of the mobile terminal 220 acquires multiple camera images. Next, in step S53, the self-position estimation unit 227 of the mobile terminal 220 calculates the position coordinates of the mobile terminal 220 by integrating the acceleration included in the measurement information acquired in step S51 twice.

[0057] Next, in step S54, the attitude estimation unit 225 of the mobile terminal 220 calculates the rotation angle and position coordinates of the mobile terminal 220 as SLAM results from the feature points included in the multiple camera images acquired in step S52. Next, in step S55, the integrated attitude estimation unit 226 of the mobile terminal 220 integrates the attitude detected based on the measurement information acquired in step S51 with the attitude estimated in step S54.

[0058] Next, in step S56, the PTZ command generation unit 22 of the mobile terminal 220 generates a control command (PTZ command) for controlling the PTZ camera 10 based on the attitude information integrated in step S55. Next, in step S57, the PTZ command transmission unit 23 of the mobile terminal 220 transmits the control command (PTZ command) generated in step S56 to the PTZ camera 10.

[0059] Next, in step S58, it is determined whether or not to terminate the process. If the process is to be terminated, it will end; otherwise, the process proceeds to step S59. Next, in step S59, since it was determined in step S58 that the process would not terminate, it is determined whether or not the pre-set processing interval time has elapsed. If it is determined that the processing interval time has elapsed, the process returns to step S51 and repeats the transition process. On the other hand, if it is determined that the processing interval time has not elapsed, the process returns to step S58 and determines whether or not to terminate the process.

[0060] The processes in steps S51 and S53 (IMU) and steps S52 and S54 (SLAM) may be performed in series or in parallel, as in this embodiment. (Embodiment 4) A camera attitude control device 320 for operating a PTZ camera 310 according to yet another embodiment of this disclosure will be described below with reference to Figure 11.

[0061] In addition, components having the same function as those described in Embodiments 1 to 3 above are denoted by the same reference numerals, and their detailed descriptions are omitted. The camera attitude control device 320 in this embodiment differs from embodiments 1 and 3 described above, in that it is located inside the PTZ camera 310, as shown in Figure 11, while the camera attitude control device 320 in this embodiment is located inside the portable terminals 20 and 220 used to remotely operate the PTZ camera 10.

[0062] Furthermore, the camera attitude control device 320 is not limited to being located inside the PTZ camera 310; it may also be configured to be integrated with the PTZ camera 310, for example, by using a connecting jig or the like. In other words, as shown in Figure 11, the camera attitude control device 320 of this embodiment is provided inside the PTZ camera 310 (for example, in the base portion 10a) and comprises an attitude detection unit 21, a PTZ command generation unit 22, self-position estimation units 324a, 324b, imaging unit 224, attitude estimation unit (SLAM) 225, and integrated attitude estimation unit 326.

[0063] The self-position estimation unit 324a acquires the acceleration included in the measurement information measured by the inertial measurement unit (IMU) of the attitude detection unit 21, and estimates the position of the PTZ camera 310 based on the acquired acceleration. Specifically, the self-position estimation unit 324a calculates the position coordinates of the PTZ camera 310 by integrating the measurement result (acceleration) from the inertial measurement unit (IMU) acquired by the attitude detection unit 21 twice.

[0064] The self-position estimation unit 324b calculates the position coordinates of the PTZ camera 310 from the attitude information of the PTZ camera 310 estimated by the attitude estimation unit 225. The integrated attitude estimation unit 326 integrates the detection results from the attitude detection unit (IMU) 21 and the estimation results from the attitude estimation unit (SLAM) 225 to estimate the attitude of the PTZ camera 310.

[0065] The PTZ command generation unit 22 generates control commands (PTZ commands) for controlling the PTZ camera 310 based on the attitude information integrated by the integrated attitude estimation unit 326. The PTZ command generation unit 22 then transmits the generated control commands to the punch / tilt direction control unit 16. As a result, the PTZ camera 310 has its attitude in the punch / tilt direction controlled based on control commands generated in the camera attitude control device 320.

[0066] (Embodiment 5) A mobile terminal (camera attitude control device) 420 according to yet another embodiment of this disclosure will be described below with reference to Figure 12. In addition, components having the same function as those described in Embodiments 1 to 4 above are denoted by the same reference numerals, and their detailed descriptions are omitted.

[0067] Furthermore, the portable terminal (camera attitude control device) 420 according to this embodiment is equipped with a display control unit 24 and a display unit 25, similar to the portable terminal 20 shown in Figure 2, but for the sake of convenience of explanation, they are not shown and their descriptions are omitted. The portable terminal (camera attitude control device) 420 of this embodiment differs from the above embodiment in that, as shown in Figure 12, it not only transmits PTZ commands to an external source for controlling the attitude of the PTZ camera 10, but also transmits attitude information to a CG synthesis device, etc., which will be described later, via a network.

[0068] In other words, as shown in Figure 12, the portable terminal (camera attitude control device) 420 of this embodiment includes an attitude detection unit 21, a PTZ command generation unit 22, a PTZ command transmission unit 23, an imaging unit 224, an attitude estimation unit (SLAM) 225, an integrated attitude estimation unit 226, an attitude information generation unit 427, and an attitude information transmission unit (communication unit) 428. The PTZ command transmission unit 23 transmits control commands generated by the PTZ command generation unit 22 to the PTZ camera 10 via the network, based on the attitude estimation results of the mobile terminal 420 integrated in the integrated attitude estimation unit 226.

[0069] The PTZ commands transmitted from the PTZ command transmission unit 23 are control commands that control the attitude of the PTZ camera 10. Furthermore, the posture information transmission unit 428 transmits the posture information generated by the posture information generation unit 427 to an external party via the network, based on the posture estimation results of the mobile terminal 420 integrated by the integrated posture estimation unit 226.

[0070] The transmission of attitude information is performed using a transmission protocol such as FreeD. This allows the transmitted pose information to be used in combination with a CG synthesis device synchronized via a network to generate, for example, a CG image that combines a background CG and a subject. (Embodiment 6) A camera attitude control device 520 for operating a PTZ camera 510 according to yet another embodiment of this disclosure will be described below with reference to Figure 13.

[0071] In addition, components having the same function as those described in Embodiments 1 to 5 above are denoted by the same reference numerals, and their detailed descriptions are omitted. The camera attitude control device 520 in this embodiment differs from embodiment 5 in that, as shown in Figure 13, it is located inside the PTZ camera 510 rather than inside the mobile terminal.

[0072] In other words, as shown in Figure 13, the camera attitude control device 520 of this embodiment is located inside the PTZ camera 510 and includes an attitude detection unit 21, a PTZ command generation unit 22, an imaging unit 224, an attitude estimation unit (SLAM) 225, self-position estimation units 324a, 324b, an integrated attitude estimation unit 326, an attitude information generation unit 427, and an attitude information transmission unit 428.

[0073] The PTZ command generation unit 22 generates control commands to control the attitude of the PTZ camera 510 based on the attitude estimation results of the PTZ camera 510 integrated in the integrated attitude estimation unit 326, and transmits them to the punch / tilt direction control unit 16. The PTZ commands generated by the PTZ command generation unit 22 are control commands that control the attitude of the PTZ camera 10.

[0074] Furthermore, the attitude information transmission unit 428 transmits the attitude information generated by the attitude information generation unit 427 to an external source via the network, based on the attitude estimation results of the PTZ camera 510 integrated in the integrated attitude estimation unit 326. The transmission of attitude information is performed using a transmission protocol such as FreeD. This allows, similar to Embodiment 5 described above, to generate a CG image by combining the transmitted pose information with a CG synthesis device synchronized via a network, for example, a CG image that combines a background CG and a subject.

[0075] [Other embodiments] Although one embodiment of the present disclosure has been described above, the present disclosure is not limited to the above embodiment, and various modifications are possible without departing from the gist of the disclosure. (A) In the above embodiments, examples of the camera attitude control device and camera attitude control method that implement the present disclosure have been described. However, the present disclosure is not limited thereto.

[0076] For example, this disclosure may be implemented as a camera attitude control program that causes a computer to execute the camera attitude control method described above. This camera attitude control program is stored in the memory (storage unit) of the camera attitude control device. The CPU reads the camera attitude control program stored in memory and instructs the hardware to execute each step. More specifically, the same effect as described above can be obtained by the CPU reading the camera attitude control program and executing each step described above.

[0077] Furthermore, this disclosure may be implemented as a recording medium storing a camera attitude control program. (B) In Embodiment 2 described above, an example was given of performing attitude control and self-position estimation of a PTZ camera based on the detection results of the attitude detection unit (IMU) 121. In Embodiment 3 described above, an example was given of performing attitude control and self-position estimation of a PTZ camera by combining the detection results of the attitude detection unit (IMU) 121 and the estimation results of the attitude estimation unit (SLAM) 225. However, the present invention is not limited thereto.

[0078] For example, the configuration may use the estimation results from the attitude estimation unit (SLAM) to perform at least one of attitude control and self-localization of the PTZ camera. (C) In the above embodiment, an example was described in which the display control unit 24 controls the display screen of the display unit 25 to blink or change to a warning color when the PTZ camera 10 reaches the end of the drive range in at least one of the pan and tilt directions. However, this disclosure is not limited thereto.

[0079] For example, when the PTZ camera reaches the end of the drive range in at least one of the pan and tilt directions, the system may be configured to include a vibration generating unit that generates vibrations instead of the display control described above. (D) In the above embodiment 1, an example was given in which the attitude detection unit (measurement information acquisition unit) 21, 121, etc., detects the attitude (position (XYZ coordinates), rotation angle) of a mobile terminal 20 or PTZ camera 110, etc., using an inertial measurement unit (IMU) that detects three-dimensional inertial motion. However, this disclosure is not limited to this.

[0080] For example, the system may be configured to acquire measurement information measured by an inertial measurement unit (IMU) located outside the device. (E) In the above embodiment, an example was given in which the configuration of the present disclosure is applied to attitude control of a remotely operated PTZ camera 10 or the like. However, the present disclosure is not limited thereto.

[0081] For example, the configuration of this disclosure may be applied to the attitude control of a camera recorder such as a single-lens reflex camera, instead of a PTZ camera. <Note> Based on the above description of embodiments, the following technologies are disclosed. (Technology 1) The camera attitude control device related to Technology 1 is A camera attitude control device that controls the attitude of a camera driven in the pan and tilt directions, A measurement information acquisition unit that acquires rotational information measured in an inertial measuring device, Based on the rotation information acquired by the measurement information acquisition unit, the attitude control unit controls the attitude of the camera, It is equipped with.

[0082] (Technology 2) The camera attitude control device relating to Technology 2 is a camera attitude control device relating to Technology 1, The measurement information acquisition unit acquires acceleration information measured by the inertial measurement device, The attitude control unit controls the attitude of the camera based on the rotation information and the acceleration information.

[0083] (Technology 3) The camera attitude control device relating to Technology 3 is a camera attitude control device relating to Technology 2, The measurement information acquisition unit acquires acceleration information measured by the inertial measurement device, The system further includes a self-position estimation unit that estimates the position of the camera based on the acceleration information acquired by the measurement information acquisition unit.

[0084] (Technology 4) The camera attitude control device relating to Technology 4 is a camera attitude control device relating to any one of Technologies 1 to 3, The measurement information acquisition unit acquires feature points included in multiple captured camera images, The attitude control unit controls the attitude of the camera based on the rotation information and the feature points.

[0085] (Technology 5) The camera attitude control device relating to Technology 5 is the camera attitude control device relating to Technology 4, The system further includes a self-position estimation unit that estimates the position of the camera based on the rotation information and feature points acquired by the measurement information acquisition unit. (Technology 6) The camera attitude control device according to Technology 6 is a camera attitude control device according to any one of Technologies 1 to 5, The camera comprises a base and a first imaging unit that is mounted on the base in a manner that allows it to rotate and performs imaging. The attitude control unit detects the attitude of the base unit from the information acquired by the measurement information acquisition unit and controls the attitude of the first imaging unit.

[0086] (Technology 7) The camera attitude control device relating to Technology 7 is the camera attitude control device relating to Technology 5, The system further includes a communication unit that transmits the camera's orientation, estimated by the self-position estimation unit, to an external source via a network. (Technology 8) The camera attitude control device relating to Technology 8 is a camera attitude control device relating to any one of Technologies 1 to 7, The inertial measurement device is attached to a portable terminal that operates the camera.

[0087] (Technology 9) The camera attitude control device according to Technology 9 is the camera attitude control device according to Technology 8, The mobile terminal transmits commands to control the camera's orientation via the network. (Technology 10) The camera attitude control device according to Technology 10 is a camera attitude control device according to Technology 8, The aforementioned mobile terminal has a display unit that displays the video being captured by the camera.

[0088] (Technology 11) The camera attitude control device relating to Technology 11 is a camera attitude control device relating to Technology 4, Multiple camera images containing the aforementioned feature points are captured by a second imaging unit mounted on a mobile terminal that operates the camera. (Technology 12) The camera attitude control device according to Technology 12 is a camera attitude control device according to Technology 11, The aforementioned mobile device is connected to the camera via a network.

[0089] (Technology 13) The camera attitude control device relating to Technology 13 is a camera attitude control device relating to Technology 11, The second imaging unit photographs the subject on the opposite side from the operating surface of the mobile device. (Technology 14) The camera attitude control device relating to Technology 14 is a camera attitude control device relating to Technology 11, The second imaging unit captures the subject on the operating surface side of the mobile device.

[0090] (Technology 15) The camera attitude control device relating to Technology 15 is a camera attitude control device relating to Technology 8, The mobile terminal further includes a vibration-generating unit that generates vibration when the camera reaches the end of at least one of the pan and tilt drive ranges. (Technology 16) The camera attitude control device according to Technology 16 is a camera attitude control device according to Technology 10, The mobile terminal further includes a display control unit that controls the display screen of the display unit to blink when the camera reaches the end of the drive range in at least one of the pan and tilt directions.

[0091] (Technology 17) The camera attitude control device according to Technology 17 is a camera attitude control device according to Technology 10, The mobile terminal further includes a display control unit that controls the display screen of the display unit to change to a warning color when the camera reaches the end of the drive range in at least one of the pan and tilt directions.

[0092] (Technology 18) The camera attitude control device according to Technology 18 is a camera attitude control device according to Technology 8, The aforementioned mobile device is fixed to the camera via a support. (Technology 19) The camera attitude control device related to Technology 19 is A camera attitude control device that controls the attitude of a camera driven in the pan and tilt directions, A measurement information acquisition unit that acquires feature points contained in multiple captured camera images, Based on the characteristic points acquired by the measurement information acquisition unit, an attitude control unit controls the camera's attitude, It is equipped with.

[0093] (Technology 20) The camera attitude control system related to Technology 20 is A camera attitude control device relating to any one of technologies 1 to 19, A mobile device for operating the aforementioned camera, It is equipped with. [Industrial applicability]

[0094] The camera attitude control device of this disclosure has the effect of allowing intuitive operation of a camera capable of pan / tilt movement, and is therefore widely applicable to the control of various imaging devices that can be driven in the pan / tilt direction. [Explanation of symbols]

[0095] 10 PTZ cameras 10a Base section 10b Swivel section 11 Camera head 11a Rotation mechanism 12 Image sensor 13. Lens drive unit (zoom mechanism) 14 Lens control unit 15a Pan direction drive mechanism 15b Tilt direction drive mechanism 16 Pan / Tilt Direction Control Unit 20. Mobile device (camera attitude control device) 21. Posture detection unit (measurement information acquisition unit) 22 PTZ command generation unit (attitude control unit) 23 PTZ Command Transmission Unit (Communication Unit) 24 Display Control Unit 25 Display section 110 PTZ camera 120 Camera attitude control device 121 Attitude detection unit 122 PTZ command generation unit (attitude control unit) 124 Self-position estimation part 220 Mobile device (camera attitude control device) 224 Imaging Department 225 Posture estimation section 226 Integrated posture estimation unit 227 Self-position estimation part 310 PTZ Camera 320 Camera attitude control device 324a,324b Self-position estimation part 326 Integrated posture estimation unit 420 Mobile device (camera attitude control device) 427 Posture information generation unit 428 Posture Information Transmission Unit (Communication Unit) 510 PTZ Camera 520 Camera attitude control device LU Lens Unit

Claims

1. A camera attitude control device that controls the attitude of a camera driven in the pan and tilt directions, A measurement information acquisition unit that acquires rotational information measured in an inertial measuring device, Based on the rotation information acquired by the measurement information acquisition unit, the attitude control unit controls the attitude of the camera, A camera attitude control device equipped with the following features.

2. The measurement information acquisition unit acquires acceleration information measured by the inertial measurement device, The attitude control unit controls the attitude of the camera based on the rotation information and the acceleration information. The camera attitude control device according to claim 1.

3. The measurement information acquisition unit acquires acceleration information measured by the inertial measurement device, The system further includes a self-position estimation unit that estimates the position of the camera based on the acceleration information acquired by the measurement information acquisition unit. The camera attitude control device according to claim 2.

4. The measurement information acquisition unit acquires feature points included in multiple captured camera images, The attitude control unit controls the attitude of the camera based on the rotation information and the feature points. The camera attitude control device according to claim 1 or 2.

5. The system further includes a self-position estimation unit that estimates the position of the camera based on the rotation information and feature points acquired by the measurement information acquisition unit. The camera attitude control device according to claim 4.

6. The camera comprises a base and a first imaging unit that is mounted on the base in a manner that allows it to rotate and performs imaging. The attitude control unit detects the attitude of the base unit from the information acquired by the measurement information acquisition unit and controls the attitude of the first imaging unit. The camera attitude control device according to claim 1 or 2.

7. The system further includes a communication unit that transmits the camera's orientation, estimated by the self-position estimation unit, to an external source via a network. The camera attitude control device according to claim 5.

8. The inertial measuring device is attached to a portable terminal that operates the camera. The camera attitude control device according to claim 1 or 2.

9. The mobile terminal transmits commands to control the camera's orientation via the network. The camera attitude control device according to claim 8.

10. The aforementioned mobile terminal has a display unit that displays the video being captured by the camera. The camera attitude control device according to claim 8.

11. Multiple camera images including the aforementioned feature points are captured by a second imaging unit mounted on a mobile terminal that operates the camera. The camera attitude control device according to claim 4.

12. The aforementioned mobile terminal is connected to the camera via the network. The camera attitude control device according to claim 11.

13. The second imaging unit photographs the subject on the opposite side from the operating surface of the mobile terminal. The camera attitude control device according to claim 11.

14. The second imaging unit photographs the subject on the operating surface side of the mobile terminal. The camera attitude control device according to claim 11.

15. The mobile terminal further includes a vibration-generating unit that generates vibration when the camera reaches the end of at least one of the pan and tilt drive ranges. The camera attitude control device according to claim 8.

16. The mobile terminal further includes a display control unit that controls the display screen of the display unit to blink when the camera reaches the end of at least one of the drive ranges in the pan direction and the tilt direction. The camera attitude control device according to claim 10.

17. The mobile terminal further includes a display control unit that controls the display screen of the display unit to change to a warning color when the camera reaches the end of at least one of the drive ranges in the pan direction and the tilt direction. The camera attitude control device according to claim 10.

18. The aforementioned mobile terminal is fixed to the camera via a support portion. The camera attitude control device according to claim 8.

19. A camera attitude control device that controls the attitude of a camera driven in the pan and tilt directions, A measurement information acquisition unit that acquires feature points contained in multiple captured camera images, Based on the characteristic points acquired by the measurement information acquisition unit, an attitude control unit controls the camera's attitude, A camera attitude control device equipped with the following features.

20. A camera attitude control device according to claim 1 or 19, A mobile device for operating the aforementioned camera, A camera attitude control system equipped with this system.

21. A camera attitude control method for controlling the attitude of a camera driven in the pan and tilt directions, A measurement information acquisition step in which rotational information measured in an inertial measuring device is acquired, A posture control step which controls the camera's posture based on the rotation information acquired in the measurement information acquisition step, A camera attitude control method that includes the following features.

22. A camera attitude control program that controls the attitude of a camera driven in the pan and tilt directions, A measurement information acquisition step in which rotational information measured in an inertial measuring device is acquired, A posture control step which controls the camera's posture based on the rotation information acquired in the measurement information acquisition step, A camera attitude control program that causes a computer to execute a camera attitude control method that includes the following features.

Images