Imaging device, control method, and program

Abstract

To realize a technique of performing framing as intended by a user even when there is a communication delay in imaging by remotely operating an imaging device.SOLUTION: An imaging device which can be remotely controlled by an operation device includes: imaging means for taking an image; communication means for communicating with the operation device; image processing means for detecting an object in the image; and control means for controlling the framing of the image including the object on the basis of the framing operation received by the operation device after the communication means sends the image to the operation device. The image processing means calculates the framing control amount on the basis of a delay time in communication processing with the operation device, a predicted moving amount of the object in the delay time, an actual moving amount of the object before and after the delay time, and the framing operation amount received from the operation device. The control means controls the framing on the basis of the framing control amount.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a technique for remotely controlling an imaging device. [Background technology]

[0002] There is known an imaging device that can perform framing control in the pan, tilt, and zoom directions by remotely operating it from an operating device. By receiving and displaying an image captured by the imaging device on the operating device, a user can perform framing operations while viewing the image and capture a desired subject.

[0003] Patent Document 1 describes a method for remotely controlling an imaging device to track a subject, in which the position of the subject is predicted taking communication delays into account, and pan and tilt operations are controlled based on the predicted position of the subject. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6557768 Summary of the Invention [Problem to be solved by the invention]

[0005] In Patent Document 1, when a user specifies a subject to be tracked, the imaging device automatically performs framing control for the subject to be tracked, but does not predict the subject position taking into account the user's framing operation. Also, in Patent Document 1, the user specifies the subject to be tracked by looking at an image received with a delay due to communication delay, so if the direction or speed of the subject's movement changes, there is a possibility that the user will not be able to specify the subject to be tracked as intended.

[0006] The present invention has been made in consideration of the above-mentioned problems, and its purpose is to realize a technology that enables framing to be performed as intended by the user even when there is a communication delay when remotely operating an imaging device to take a photograph. [Means for solving the problem]

[0007] In order to solve the above problems and achieve the object, the present invention provides an imaging device that can be remotely controlled by an operation device, the imaging device comprising: an imaging means for capturing an image; a communication means capable of communicating with the operation device; an image processing means for detecting a subject in the image; and after transmitting the image to the operation device by the communication means, In response to a framing operation performed by the operation device a framing operation received from the operation device; amount and a control unit for controlling framing of an image including the subject based on the above-mentioned. from a first time when the image to be transmitted to the operation device is captured to a second time when the framing operation amount is received from the operation device and the framing control is started. a delay acquisition unit that acquires a delay time; calculating a framing control amount based on a predicted movement amount of the subject within a delay time from the first time to the second time, which is predicted at the first time, an actual movement amount of the subject within the delay time, which is calculated based on an image captured at the first time and an image captured at the second time, and the framing operation amount received from the operation device; The control means performs the framing control based on the framing control amount. [Effects of the Invention]

[0008] According to the present invention, even if there is a communication delay when photographing by remotely operating an imaging device, it is possible to perform framing as intended by the user. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing the configuration of an imaging device and an operation device according to an embodiment of the present invention. [Figure 2] FIG. 1 is an external view of an imaging apparatus according to an embodiment of the present invention. [Figure 3] FIG. 2 is an external view of the operating device according to the embodiment. [Figure 4] FIG. 2 is an explanatory diagram of an input unit of the operation device according to the embodiment. [Figure 5] FIG. 2 is a block diagram showing the arrangement of an image processing unit of the imaging apparatus according to the first embodiment. [Figure 6] 4 is a flowchart showing a control example of the first embodiment. [Figure 7]FIG. 10 is a block diagram showing the arrangement of an information processing unit of the imaging apparatus according to the second embodiment. [Figure 8] 10 is a flowchart showing a control example of the second embodiment. [Figure 9] 4 is a timing chart of each processing step in a control example of the first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] Hereinafter, an embodiment in which the imaging device of the present invention is applied to a digital camera or the like capable of remote framing control, and the operation device of the present invention is applied to a remote controller capable of remotely operating the imaging device will be described in detail with reference to the accompanying drawings. Note that the operation device is not limited to a remote controller, and may be a communication terminal such as a smartphone, or an information processing terminal such as a tablet PC (personal computer).

[0012] <Device Configuration> First, the configuration and functions of the imaging device and operation device of this embodiment will be described with reference to FIGS.

[0013] FIG. 1 illustrates an example of the hardware configuration of an imaging device and an operation device according to this embodiment.

[0014] In FIG. 1, an imaging device 1 and an operation device 2 are connected so as to be able to communicate with each other. The operation device 2 can remotely control the imaging device 1. The imaging device 1 is remotely controlled by the operation device 2 to capture images and transmit the captured image data to the operation device 2. A user can remotely control the imaging device 1 using the operation device 2. The user can use the operation device 2 to perform framing operations and framing control to change the shooting range (angle of view) of the imaging device 1, thereby remotely monitoring images captured by the imaging device 1 in real time. The imaging device 1 also has a subject recognition function and a tracking function, and can automatically track a specified subject.

[0015] The imaging device 1 is placed in a location where it can capture an image of a subject. The operation device 2 is placed in a remote location where the user cannot directly operate the imaging device 1, or is carried by a user in a remote location where the user cannot directly operate the imaging device 1. The imaging device 1 and operation device 2 are connected by, for example, a wireless communication system, and are capable of transmitting and receiving information.

[0016] When the imaging device 1 captures an image of a moving subject by remotely operating the operation device 2, it is assumed that framing cannot be performed as intended by the user due to a communication delay between the imaging device 1 and the operation device 2. In this embodiment, a configuration and control for enabling framing to be performed as intended by the user in such a case will be described.

[0017] FIG. 2 illustrates an example of the external configuration of the imaging device 1 of this embodiment.

[0018] The imaging device 1 includes a control unit 101 , an image processing unit 102 , a communication unit 103 , a work memory 104 , a non-volatile memory 105 , an imaging unit 106 , and an optical unit 107 .

[0019] The control unit 101 performs arithmetic processing for controlling the imaging device 1. The control unit 101 includes a processor such as a CPU that controls the components of the imaging device 1.

[0020] The image processing unit 102 performs arithmetic processing of image data acquired by the imaging unit 106 , arithmetic processing of data for evaluation photometry acquired by the imaging unit 106 , and arithmetic processing of control data for controlling the optical unit 107 .

[0021] The communication unit 103 includes an interface for communicatively connecting with the controller device 2. The interface is, for example, a wireless communication interface that complies with a wireless communication method such as Wi-Fi (registered trademark), Bluetooth (registered trademark), infrared communication, or Wireless USB, or a public wireless communication method such as 4G or 5G. The communication unit 103 can transmit and receive information such as image data and operation signals to and from the controller device 2.

[0022] The work memory 104 is, for example, a RAM. The work memory 104 is used as a working area for expanding constants and variables for the operation of the control unit 101, programs read from a nonvolatile memory 105 (described later), and the like. The work memory 104 is also used as a buffer memory for temporarily storing image data captured by the imaging unit 106.

[0023] The nonvolatile memory 105 is, for example, a ROM. The nonvolatile memory 105 stores constants, programs, and the like for the operation of the control unit 101. The programs referred to here are programs for executing communication processing and control processing, which will be described later in this embodiment.

[0024] The imaging unit 106 has an image sensor composed of photoelectric conversion elements such as CCD or CMOS that converts a subject image into an electric signal, and an A / D converter that converts an analog signal output from the image sensor into a digital signal. Under the control of the control unit 101, the imaging unit 106 converts the subject image light formed by a lens included in the optical unit 107 (described later) into an electric signal using the image sensor, performs noise reduction processing, etc., and outputs image data consisting of a digital signal.

[0025] The optical unit 107 includes a group of lenses including a zoom lens and a focus lens, a shutter with an aperture function, and a mechanism for driving these optical members. The optical unit 107 drives the optical members to perform at least one of rotating the shooting range (angle of view) of the imaging device 1 around the pan (P) axis (horizontal direction) or tilt (T) axis (vertical direction), and moving it along the zoom (Z) axis (enlargement / reduction direction).

[0026] Note that, although the imaging device 1 of this embodiment is an example of a camera equipped with a PTZ (port-to-zero) function, the imaging device 1 is not limited to this. For example, the imaging device 1 may be capable of framing by moving or rotating the device itself, like a drone, or may be capable of framing by moving or rotating an external device, like a movable platform such as a gimbal.

[0027] The PTZ function is a function that allows remote control of the pan, tilt, and zoom of the imaging device 1. PTZ is an abbreviation of the initials of pan, tilt, and zoom. Pan is horizontal oscillation. Tilt is vertical oscillation. Zoom is zooming up (telephoto) and zooming out (wide angle).

[0028] FIG. 3 illustrates an example of the external configuration of the operating device 2 of this embodiment.

[0029] The operation device 2 includes a control unit 201 , an input unit 202 , a communication unit 203 , a work memory 204 , a non-volatile memory 205 , and a display unit 206 .

[0030] The control unit 201 performs arithmetic processing for controlling the operation device 2. The control unit 201 includes a processor such as a CPU that controls the components of the operation device 2.

[0031] The input unit 202 includes operation members that accept user operations and outputs operation signals corresponding to the user operations to the control unit 201. As shown in FIG. 3, the input unit 202 includes a joystick 202A and a slide bar 202B. As shown in FIG. 4, the input unit 202 can input the direction and magnitude of the operation as operation signals according to the direction and distance (stroke) of movement of the operation members. FIG. 4 illustrates operation positions of the joystick 202A and the slide bar 202B. Operation positions 202a, 202b, and 202c indicate the states in which the operation amount of the operation members is 0, an intermediate amount, and a maximum amount, respectively. The joystick 202A can input any operation direction of 360 degrees by moving the operation members, with the center position (neutral position) of the movable range of the operation members being the operation amount 0. The slide bar 202B can input any operation direction of 180 degrees, which is opposite to the operation amount, depending on the direction of movement of the operation members, with the center position (neutral position) of the movable range of the operation members being the operation amount 0.

[0032] The input unit 202 can operate at least one of pan, tilt, zoom, focus and aperture of the imaging device 1, and pan can be performed by operating the joystick 202A left and right, tilt by operating it forward and backward, and zoom by operating the slide bar 202B forward and backward.

[0033] The input unit 202 may have any form as long as it can input the direction and magnitude of an operation, and may be, for example, a touch panel that is integrated with the display unit 206 (described later) and that can input with the user's finger or a stylus. The operation members of the input unit 202 are not limited to the joystick 202A and the slider 202B, and may include, for example, a numeric keypad or other buttons, which may be configured as a touch panel.

[0034] The communication unit 203 includes an interface for communicatively connecting with the imaging device 1. The interface is, for example, a wireless communication interface that complies with a wireless communication method such as Wi-Fi (registered trademark), Bluetooth (registered trademark), infrared communication, or Wireless USB, or a public wireless communication method such as 4G or 5G. The communication unit 203 is capable of transmitting and receiving information such as image data and operation signals to and from the imaging device 1.

[0035] The work memory 204 is, for example, a RAM. The work memory 204 is used as a working area for expanding constants and variables for the operation of the control unit 201, programs read from the nonvolatile memory 205 (described later), and the like. The work memory 204 is also used as a buffer memory for temporarily storing image data received from the imaging device 1.

[0036] The nonvolatile memory 205 is, for example, a ROM. The nonvolatile memory 205 stores constants, programs, and the like for the operation of the control unit 201. The programs referred to here are programs for executing communication processing and control processing, which will be described later in this embodiment.

[0037] Furthermore, when the controller device 2 is a smartphone or a tablet PC, the nonvolatile memory 205 stores an OS (operating system) which is basic software executed by the control unit 201, and an application for remotely controlling the imaging device 1 in cooperation with the OS. The processing of the controller device 2 is realized by reading the software provided by the application.

[0038] The display unit 206 is a display device such as a liquid crystal display, an organic EL display, etc. The display unit 206 displays images received from the imaging device 1, a GUI (Graphical User Interface), and the like.

[0039] The operation device 2 cooperates with the imaging device 1 to realize remotely controlled shooting. The imaging device 1 generates a live view image by capturing an image of a subject and transmits the image to the operation device 2. The operation device 2 displays the live view image received from the imaging device 1. The user operates the input unit 202 while checking the image captured by the imaging device 1 on the display unit 206 of the operation device 2. The operation device 2 transmits an operation signal received by the input unit 202 to the imaging device 1. The imaging device 1 controls the optical unit 107 in accordance with the operation signal received from the operation device 2. By repeating the above-described operations, the user can frame the subject as intended and capture videos or still images, even if the subject is moving.

[0040] [First embodiment] The first embodiment will be described below.

[0041] The first embodiment is an example of a control method for framing the movement of a subject in the horizontal direction (pan direction) and vertical direction (tilt direction).

[0042] In the first embodiment, the subject moves on a plane perpendicular to the optical axis of the imaging device 1, and the apparent size of the subject in the frame is not taken into consideration in framing. The user remotely controls the operation device 2 to move the optical section 107 of the imaging device 1 in the pan direction and tilt direction to frame the image and capture the subject.

[0043] FIG. 5 is a block diagram showing the configuration and functions of the image processing unit 102 according to the first embodiment.

[0044] The image processing unit 102 has an object detection unit 401 , an object speed calculation unit 402 , a delay acquisition unit 403 , an object movement amount calculation unit 404 , and a framing control amount calculation unit 405 .

[0045] The subject detection unit 401 detects a predetermined subject (hereinafter, referred to as a main subject) in a live view image generated by the image capture unit 106, and outputs subject information, which is the detection result, to the subject speed calculation unit 402.

[0046] The subject speed calculation unit 402 calculates the speed of the main subject detected by the subject detection unit 401 from multiple frames of live view images generated by the imaging unit 106, and outputs the calculated subject speed information to the subject movement amount calculation unit 404.

[0047] The delay acquisition unit 403 acquires the delay time from when the live view image is captured by the imaging unit 106 until the optical unit 107 is driven by a framing operation, and outputs the acquired delay time information to the subject movement amount calculation unit 404.

[0048] Subject movement amount calculation unit 404 calculates a predicted movement amount of the main subject within the delay time based on the speed of the main subject calculated by subject speed calculation unit 402 and the delay time acquired by delay acquisition unit 403. Subject movement amount calculation unit 404 then outputs subject movement amount information, which is the calculation result, to framing control amount calculation unit 405. Subject movement amount calculation unit 404 also calculates the actual movement amount of the main subject based on live view images before and after the delay time, and outputs this to framing control amount calculation unit 405 together with the subject movement amount information.

[0049] Framing control amount calculation unit 405 calculates a framing control amount for optical unit 107 based on the predicted movement amount of the main subject calculated by subject movement amount calculation unit 404, the framing operation amount input from operation device 2, and the actual movement amount of the main subject before and after the delay time. Framing control amount calculation unit 405 then outputs the framing control amount that is the calculation result to control unit 101. Control unit 101 drives the optical members of optical unit 107 based on the framing control amount calculated by framing control amount calculation unit 405.

[0050] Next, the control of the first embodiment will be described with reference to FIGS.

[0051] Fig. 6 is a flowchart showing a control example of the first embodiment. Fig. 9 is a timing chart of each processing step in the control example of Fig. 6. In Fig. 9, the horizontal axis is a time axis including times T1 to T12, and the vertical axis shows each processing step in the control example of Fig. 6, indicating the temporal relationship between each processing step. The relationship of the length of time between each processing step in Fig. 9 is not necessarily as shown in Fig. 9.

[0052] 6 is realized by the control unit 101 of the imaging device 1 executing a program stored in the nonvolatile memory 105 and controlling each component of the imaging device 1. The processing in FIG. 6 is started when the imaging device 1 transitions to a shooting mode.

[0053] In S601 (time T1 in FIG. 9 ), the delay acquisition unit 403 of the image processing unit 102 acquires a delay time in the communication processing between the image capture device 1 and the operation device 2. The delay time is the time it takes for the image capture unit 106 of the image capture device 1 to capture a live view image, the image processing unit 102 to process the live view image, the communication unit 103 to transmit the live view image to the communication unit 203 of the operation device 2, the display unit 206 to display the live view image, and then for the communication unit 203 to transmit an operation signal input to the input unit 202 to the communication unit 103 of the image capture device 1, and for the optical members of the optical unit 107 to be driven based on the operation signal. In this case, it is assumed that the delay time between the communication units 103 and 203 will vary depending on the capacity and conditions of the transmission path and the congestion status of the transmission path. In this case, it is assumed that the delay time is sufficiently small compared to the delay time between the communication units 103 and 203, and that variation due to transmission conditions, etc. is also small. Therefore, the delay time can be obtained by, for example, measuring the time from when dummy image data is transmitted from the imaging device 1 to the operation device 2 until when a dummy operation signal is transmitted from the operation device 2 to the imaging device 1. To obtain a more accurate delay time, time related to other factors in the transmission path may be added. Note that the delay obtaining unit 403 may obtain the delay time sequentially during shooting, or may obtain the delay time in advance before shooting.

[0054] In S602, the imaging unit 106 captures a live view image, and the image processing unit 102 processes the live view image.

[0055] In S603, the communication unit 103 transmits a live view image to the communication unit 203 of the controller device 2. The controller device 2 displays the received live view image on the display unit 206, the input unit 202 accepts a framing-related operation from the user, and transmits an operation signal to the imaging device 1 via the communication unit 203. The operation signal includes the direction and amount of the framing-related operation.

[0056] In S604, the subject detection unit 401 detects the main subject in the live view image using a known method.

[0057] In S605 (time T3 in FIG. 9), control unit 101 determines whether multiple frames of live view images necessary for calculating the velocity of the main subject have been captured. If control unit 101 determines that multiple frames of live view images necessary for calculating the velocity of the main subject have been captured, control proceeds to S606; otherwise, control proceeds to S612.

[0058] In S606 (time T4 in FIG. 9), the subject velocity calculation unit 402 calculates the velocity of the main subject. The velocity of the main subject is calculated from the difference between the subject positions in multiple live view images most recently captured by the imaging unit 106. Note that three or more live view images may be used to calculate the subject velocity as well as the subject acceleration, which may be used to calculate the subject movement amount, which will be described later.

[0059] In S607 (time T8 in Figure 9), the subject movement amount calculation unit 404 calculates the predicted movement amount of the main subject within the delay time from the delay time acquired by the delay acquisition unit 403 and the subject speed calculated by the subject speed calculation unit 402.

[0060] In S608 (time T9 in FIG. 9), subject movement amount calculation section 404 calculates the actual movement amount of the main subject before and after the delay time.

[0061] In S609, control unit 101 determines whether or not it has been able to acquire the operation signal received from operation device 2, the predicted movement amount of the main subject, and the actual movement amount of the main subject before and after the delay time. If control unit 101 determines that it has been able to acquire the operation signal received from operation device 2, the predicted movement amount of the main subject, and the actual movement amount of the main subject before and after the delay time, it proceeds to S610; otherwise, it proceeds to S611.

[0062] In S610 (time T10 in FIG. 9), framing control amount calculation section 405 calculates the framing control amount. Framing control amount calculation section 405 calculates the framing control amount based on the predicted movement amount of the main subject calculated by subject movement amount calculation section 404, the framing operation amount in the operation signal received from operation device 2, and the actual movement amount of the main subject before and after the delay time. Note that the framing operation amount input by user operation includes an operation amount related to at least either the pan direction or the tilt direction.

[0063] Here, the predicted movement amounts of the main subject in the pan and tilt directions calculated by subject movement amount calculation unit 404 are defined as mb(p) and mb(t), respectively, and the framing operation amounts in the pan and tilt directions are defined as mc(p) and mc(t), respectively. Furthermore, the actual movement amounts of the main subject in the pan and tilt directions before and after the delay time are defined as ma(p) and ma(t), respectively, and the framing control amounts in the pan and tilt directions are defined as mx(p) and mx(t). mb(p), mb(t), ma(p), and ma(t) are defined by the angular displacement of the line connecting the optical axis of image capture unit 106 and the main subject. mc(p), mc(t), mx(p), and mx(t) are defined by the angular displacement of the optical axis of image capture unit 106. Framing control amount calculation unit 405 calculates mx(p) and mx(t) using the following Equations 1 and 2. (Formula 1) mx(p)=ma(p)+(mc(p)-mb(p)) (Formula 2) mx(t)=ma(t)+(mc(t)-mb(t)) In the above equations 1 and 2, (mc(p)-mb(p)) and (mc(t)-mb(t)) are the differences between the framing operation amount and the predicted movement amount of the main subject, and represent the subject placement adjustment amount, i.e., where the user wants to place the subject within the frame. In other words, the framing control amount is found by adjusting the actual movement amount of the main subject before and after the delay time with the subject placement adjustment amount.

[0064] In S611 (time T11 in FIG. 9), the control unit 101 drives the optical members of the optical unit 107 based on the framing control amount calculated by the framing control amount calculation unit 405.

[0065] In S612, the control unit 101 determines whether or not an instruction to end image capture has been received. If the control unit 101 determines that an instruction to end image capture has been received, it ends the process. If not, the control unit 101 returns the process to S601 and repeats the above-described process.

[0066] As described above, according to the first embodiment, even if there is a communication delay when photographing by remotely controlling the imaging device 1, accurate framing according to the user's intention is possible.

[0067] [Second embodiment] The second embodiment will now be described.

[0068] The second embodiment is an example of a control method for performing framing for the movement of the subject in the forward and backward directions (zoom directions) in addition to the movement of the subject in the horizontal direction (pan direction) and vertical direction (tilt direction) of the first embodiment.

[0069] In the second embodiment, the subject moves not only in the horizontal and vertical directions as in the first embodiment, but also in the optical axis direction (zoom direction) of the imaging device 1, and the apparent size of the subject in the frame is also taken into consideration in framing. The user performs framing operations in the zoom direction as well as the pan direction and tilt direction using the operation device 2 to capture an image of the subject.

[0070] FIG. 7 is a block diagram showing the configuration and functions of the image processing unit 102 according to the second embodiment.

[0071] The image processing unit 102 has an object detection unit 701, an object feature point detection unit 702, an object feature point speed calculation unit 703, a delay acquisition unit 704, an object feature point movement amount calculation unit 705, and a framing control amount calculation unit 706. The difference from the first embodiment is that the image processing unit 102 detects a plurality of feature points of the object, calculates the speed and movement amount of each feature point, and calculates the framing control amount from the calculated movement amount of each feature point.

[0072] Other configurations of the imaging device 1 and the operation device 2 are the same as those in the first embodiment.

[0073] The subject detection unit 701 detects a predetermined subject (hereinafter, referred to as a main subject) in a live view image generated by the image capture unit 106, and outputs subject information, which is the detection result, to the subject speed calculation unit 402.

[0074] The subject feature point detection unit 702 detects feature points of the main subject detected by the subject detection unit 701 .

[0075] The subject feature point velocity calculation unit 703 calculates the velocity of the feature points of the main subject detected by the subject detection unit 701 from multiple frames of live view images generated by the imaging unit 106, and outputs the calculated subject feature point velocity information to the subject feature point movement amount calculation unit 705.

[0076] The delay acquisition unit 704 acquires the delay time from when the live view image is captured by the imaging unit 106 until the optical unit 107 is driven by a framing operation, and outputs the acquired delay time information to the object feature point movement amount calculation unit 705.

[0077] Subject feature point movement amount calculation unit 705 calculates a predicted movement amount of the feature point of the main subject within the delay time, based on the speed of the feature point of the main subject calculated by subject feature point speed calculation unit 703 and the delay time acquired by delay acquisition unit 704. Subject feature point movement amount calculation unit 705 then outputs subject movement amount information, which is the calculation result, to framing control amount calculation unit 706. Subject feature point movement amount calculation unit 705 also calculates the actual movement amount of the feature point of the main subject based on live view images before and after the delay time, and outputs this to framing control amount calculation unit 706 together with the subject movement amount information.

[0078] Framing control amount calculation unit 706 calculates a framing control amount for optical unit 107 based on the predicted movement amount of the feature point of the main subject calculated by subject feature point movement amount calculation unit 705, the framing operation amount input from operation device 2, and the actual movement amount of the feature point of the main subject within the delay time. Framing control amount calculation unit 706 then outputs the framing control amount that is the calculation result to control unit 101. Control unit 101 drives the optical members of optical unit 107 based on the framing control amount calculated by framing control amount calculation unit 405.

[0079] Next, a control example of the second embodiment will be described with reference to FIGS.

[0080] Fig. 8 is a flowchart showing a control example of embodiment 2. The processing in Fig. 8 is realized by the control unit 101 of the imaging device 1 executing a program stored in the nonvolatile memory 105 and controlling each component of the imaging device 1. The processing in Fig. 8 is started when the imaging device 1 transitions to a shooting mode.

[0081] In S801 (time T1 in FIG. 9), the delay acquisition unit 704 of the image processing unit 102 acquires the delay time in the communication processing between the imaging device 1 and the operation device 2. The method and timing for acquiring the delay time are as described in S601 in FIG.

[0082] In S802, the imaging unit 106 captures a live view image, and the image processing unit 102 processes the live view image.

[0083] In S803, the communication unit 103 transmits a live view image to the communication unit 203 of the controller device 2. The controller device 2 displays the received live view image on the display unit 206, the input unit 202 accepts a framing-related operation from the user, and transmits an operation signal to the imaging device 1 via the communication unit 203. The operation signal includes the direction and amount of the framing-related operation.

[0084] In S804, the subject detection unit 701 detects the main subject in the live view image using a known method.

[0085] In S805, the subject feature point detection unit 702 detects feature points of the main subject. A plurality of feature points of the main subject are detected. A known method is used to detect feature points of the main subject.

[0086] In S806 (corresponding to time T3 in FIG. 9), control unit 101 determines whether multiple frames of live view images necessary for calculating the velocity of the feature points of the main subject have been captured. Here, it determines whether live view images for calculating the velocity of at least one of the multiple detected feature points have been captured. If control unit 101 determines that multiple frames of live view images necessary for calculating the velocity of the feature points of the main subject have been captured, control proceeds to S807; otherwise, control proceeds to S813.

[0087] In S807 (corresponding to time T4 in FIG. 9), the subject feature point velocity calculation unit 703 calculates the velocity of the feature point of the main subject. The velocity of the feature point of the main subject is calculated from the difference in the position of the feature point of the subject in multiple live view images most recently captured by the imaging unit 106. Note that three or more live view images may be used to calculate the velocity of the feature point of the main subject as well as the acceleration of the feature point of the subject, and this may be used to calculate the amount of movement of the feature point of the main subject, which will be described later.

[0088] In S808 (corresponding to time T8 in Figure 9), the object feature point movement amount calculation unit 705 calculates the predicted movement amount of the feature point of the main subject within the delay time, based on the delay time acquired by the delay acquisition unit 704 and the velocity of the feature point of the main subject calculated by the object feature point velocity calculation unit 703.

[0089] In S809 (corresponding to time T9 in FIG. 9), subject feature point movement amount calculation section 705 calculates the actual movement amount of the feature point of the main subject within the delay time.

[0090] In S810, the control unit 101 determines whether or not it has been able to acquire the operation signal received from the operation device 2, the predicted movement amount of the feature point of the main subject, and the actual movement amount of the feature point of the main subject within the delay time. If it has determined that it has been able to acquire the operation signal received from the operation device 2, the predicted movement amount of the feature point of the main subject, and the actual movement amount of the feature point of the main subject within the delay time, the control unit 101 proceeds to the process of S811; otherwise, the control unit 101 proceeds to the process of S813.

[0091] In S811 (corresponding to time T10 in FIG. 9), framing control amount calculation section 706 calculates a framing control amount. Framing control amount calculation section 706 calculates the framing control amount based on the predicted movement amount of the feature point of the main subject calculated by subject feature point movement amount calculation section 705, the framing operation amount in the operation signal received from operation device 2, and the actual movement amount of the feature point of the main subject within the delay time. Note that the framing operation amount input by user operation includes an operation amount related to at least either the pan direction or the tilt direction.

[0092] Here, the amount calculated by the object feature point movement amount calculation unit 705 is defined as follows. -Movement amount of feature points of the main subject Pan direction: mb(p)[1], mb(p)[2], ..., mb(p)[n] Tilt direction: mb(t)[1], mb(t)[2], ..., mb(t)[n] Framing control amount Pan direction: mc(p) Tilt direction: mc(t) Zoom direction: mc(z) Actual movement of the main subject's feature points during the delay time Pan direction: ma(p)[1], ma(p)[2], ..., ma(p)[n] Tilt direction: ma(t)[1], ma(t)[2], ..., ma(t)[n] Framing control amount Pan direction: mx(p) Tilt direction: mx(t) Zoom direction: mx(z) Note that [*] is a quantity related to the feature point detected at the *th time, and n is the number of feature points of the subject detected by the subject feature point detection unit 702.

[0093] mb(p), mb(t), ma(p), and ma(t) are defined as changes in the angles in the pan and tilt directions of the line connecting the optical axis of the imaging unit 106 and the main subject. mc(p), mc(t), mx(p), and mx(t) are defined as changes in the angles in the pan and tilt directions of the optical axis of the imaging unit 106. mc(z) and mx(z) are defined as changes in the distance (focal length) from the center of the lens of the optical unit 107 to the imaging unit 106. The framing control amount calculation unit 706 calculates mx(p) and mx(t) as averages of the framing control amounts for each feature point using the following equations 3, 4, 5, and 6. (Formula 3) mx(p)=(mx(p)[1]+mx(p)[2]+...+mx(p)[n]) / n (Formula 4) mx(p)[1]=ma(p)[1]+(mc(p)-mb(p)[1]) mx(p)[2]=ma(p)[2]+(mc(p)-mb(p)[2]) ... mx(p)[n]=ma(p)[n]+(mc(p)-mb(p)[n]) (Formula 5) mx(t)=(mx(t)[1]+mx(t)[2]+...+mx(t)[n]) / n (Formula 6) mx(t)[1]=ma(t)[1]+(mc(t)-mb(t)[1]) mx(t)[2]=ma(t)[2]+(mc(t)-mb(t)[2]) ... mx(t)[n]=ma(t)[n]+(mc(t)-mb(t)[n]) Furthermore, mx(z) is calculated from the movement amounts of multiple feature points. In this embodiment, an example using the movement amounts of two feature points will be described as the simplest example. mx(z) is calculated using the following Equations 7 and 8. (Formula 7) mx(z)=ma(z)+(mc(z)-mb(z)) (Formula 8) ma(z) = |ma(p)[i] - ma(p)[j]| mb(z) = |mb(p)[i] - mb(p)[j]| In Equation 8, i and j are two different arbitrary subject feature points. In this embodiment, an example has been described in which the directions of framing operation and framing control are the pan direction, tilt direction, and zoom direction, but by performing similar calculations, the directions of framing operation and framing control can be further expanded to include translation in the roll direction (direction of rotation around the optical axis) and in each of the up / down, left / right, and front / back directions.

[0094] In S812 (corresponding to time T11 in FIG. 9), the control unit 101 drives the optical members of the optical unit 107 based on the framing control amount calculated by the framing control amount calculation unit 706.

[0095] In S813, the control unit 101 determines whether or not an instruction to end image capture has been received. If the control unit 101 determines that an instruction to end image capture has been received, the control unit 101 ends the process; if not, the control unit 101 returns the process to S801 and repeats the above-described process.

[0096] Operations other than those described above are the same as those in the first embodiment.

[0097] As described above, according to the second embodiment, even if there is a communication delay when the imaging device 1 is remotely controlled to capture an image, accurate framing according to the user's intention is possible.

[0098] Furthermore, in contrast to the first embodiment, even when the subject moves in the direction of the optical axis of the imaging device 1, accurate framing according to the user's intention is possible.

[0099] In the above-described embodiment, an example of controlling the optical unit 107 in at least one of the pan direction, tilt direction, and zoom direction as framing control has been described, but this is not limited to this, and may also be realized, for example, by image cropping processing based on a framing operation.

[0100] The above-mentioned operations described as being performed by the control unit 101 of the imaging device 1 may be performed by a single piece of hardware, or the entire device may be controlled by multiple pieces of hardware (e.g., multiple processors or circuits) sharing the processing.

[0101] [Other embodiments] The present invention can also be realized by supplying a program that realizes one or more functions of each embodiment to a system or device via a network or storage medium, and having one or more processors in the computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0102] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0103] 1...imaging device, 101...control section, 102...image processing section, 103...communication section, 2...operation device, 201...control section, 202...input section, 203...communication section, 206...display section

Claims

1. An imaging device that can be remotely controlled by an operation device, an imaging means for capturing an image; a communication means capable of communicating with the operation device; image processing means for detecting a subject in the image; a control means for controlling framing of an image including the subject based on a framing operation amount received from the operation device in response to a framing operation performed on the operation device after transmitting the image to the operation device by the communication means, The image processing means a delay acquisition unit that acquires a delay time from a first time when an image to be transmitted to the operation device is captured to a second time when the framing control is started upon receiving the framing operation amount from the operation device in a communication process with the operation device, calculating a framing control amount based on a predicted movement amount of the subject within a delay time from the first time to the second time, which is predicted at the first time, an actual movement amount of the subject within the delay time, which is calculated based on an image captured at the first time and an image captured at the second time, and the framing operation amount received from the operation device; The imaging device, wherein the control means performs the framing control based on the framing control amount.

2. The image processing means a subject detection unit that detects a main subject in the image; a subject velocity calculation unit that calculates the velocity of a main subject from a plurality of frame images generated by the imaging means; a subject movement amount calculation unit that calculates a predicted movement amount of the main subject within the delay time based on the speed of the main subject and the delay time, and calculates an actual movement amount of the main subject within the delay time based on the image captured at the first time and the image captured at the second time; 2. The imaging device according to claim 1, further comprising: a framing control amount calculation unit that calculates a framing control amount based on a predicted movement amount of the main subject within the delay time, a framing operation amount received from the operation device, and an actual movement amount of the main subject within the delay time.

3. The image processing means a subject detection unit that detects a main subject in the image; a subject feature point detection unit that detects feature points of the main subject; a subject feature point velocity calculation unit that calculates the velocity of a feature point of a main subject from a plurality of frame images generated by the imaging means; a subject feature point movement amount calculation unit that calculates a predicted movement amount of the feature point of the main subject within the delay time based on the speed of the feature point of the main subject and the delay time, and calculates an actual movement amount of the feature point of the main subject within the delay time based on the image captured at the first time point and the image captured at the second time point; 2. The imaging device according to claim 1, further comprising: a framing control amount calculation unit that calculates a framing control amount based on a predicted movement amount of a feature point of the main subject within the delay time, a framing operation amount received from the operation device, and an actual movement amount of the feature point of the main subject within the delay time.

4. The imaging device according to claim 1 , wherein the delay acquisition unit acquires the delay time during imaging.

5. The imaging device according to claim 1 , wherein the delay acquisition unit acquires the delay time in advance before shooting.

6. The imaging device according to claim 4 or 5, characterized in that the delay acquisition unit acquires the delay by measuring the time from when the imaging device transmits a dummy image to the operation device until when the imaging device receives a dummy operation signal from the operation device.

7. 7. The imaging device according to claim 1, wherein the delay time is a time period from when the imaging means captures an image and transmits the image to the operation device until when the optical unit is driven based on the framing operation.

8. 8. The imaging device according to claim 1, wherein the direction of the framing operation and the framing control includes at least one of a pan direction, a tilt direction, and a zoom direction relative to an optical axis of the imaging means.

9. 9. The imaging device according to claim 8, wherein the framing control includes control for driving an optical unit in at least one of a pan direction, a tilt direction, and a zoom direction.

10. 9. The imaging device according to claim 8, wherein the framing control includes image cropping based on at least one of a pan direction, a tilt direction, and a zoom direction.

11. A control method for an imaging device that has an imaging means for capturing an image and can be remotely controlled by an operation device, comprising: communicating with the control device; detecting an object in the image; and after transmitting the image to the operation device, controlling the framing of the image including the subject based on a framing operation amount received from the operation device in response to a framing operation performed on the operation device, In the framing control, acquiring a delay time from a first time when an image to be transmitted to the operation device is captured to a second time when the framing operation amount is received from the operation device and the framing control is started in a communication process with the operation device; calculating a framing control amount based on a predicted movement amount of the subject within a delay time from the first time to the second time, which is predicted at the first time, an actual movement amount of the subject within the delay time, which is calculated based on an image captured at the first time and an image captured at the second time, and the framing operation amount received from the operation device; A control method comprising: performing the framing control based on the framing control amount.

12. A program for causing a computer to function as each of the means of the imaging device according to any one of claims 1 to 10.

Images