Imaging device, shake correction apparatus, control method, and program

Abstract

To provide an imaging device that can detect a subject to be tracked accurately at high speed and perform tracking photography, a shake correction apparatus, a control method, and a program.SOLUTION: An imaging device causes a beam direction changing device to change the beam direction of an imaging beam at high speed to continuously capture a main subject at a fixed position on an image pick-up device. The imaging device corrects the position of a focus lens by using the control direction of the focus lens in focusing and the speed at which the beam direction changing device changes the beam direction of the imaging beam, and registers in advance, of a plurality of feature quantities characterizing the main subject, a feature quantity to be used in detection of the main subject. The imaging device has first main subject detection means that detects the main subject by using the feature quantity registered in advance, and second main subject detection means that detects the main subject by using the plurality of feature quantities, and makes a switch to one of the main subject detection means to detect the position of the main subject according to the movement of detected subjects.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an imaging device, a motion compensation device, a control method, and a program, and more particularly to an imaging device, a motion compensation device, a control method, and a program for tracking and photographing a subject moving at high speed. [Background technology]

[0002] In order to track and photograph a subject moving at high speed, it is necessary to perform focus control according to the subject to be tracked and high-speed detection of the subject to be tracked, and various technologies have been proposed to address these needs.

[0003] Patent document 1 describes a technology that improves the tracking accuracy of fast-moving subjects by performing tracking calculations using movement trajectory information based on a pre-registered, predetermined route when it is easy to estimate the movement trajectory of the subject to be tracked, such as on a motorsports circuit.

[0004] Patent Documents 2 and 3 describe changing the value of the best focus correction amount according to the image height based on the output of the attitude state detection means, and eliminating the effects of misalignment of the optical system and light receiving element used for autofocus due to deterioration of the sub-mirror over time, etc. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-27436 [Patent Document 2] Japanese Patent Application Laid-Open No. 2000-19384 [Patent Document 3] Japanese Patent Application Laid-Open No. 2010-249922 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the technology of Patent Document 1 cannot improve the tracking performance of a fast-moving subject when the movement of the subject to be tracked is complex and it is difficult to estimate the movement trajectory.

[0007] On the other hand, if an imaging device is attached to a high-speed light direction change device, it is possible to track and photograph a fast-moving subject whose movement trajectory is difficult to estimate by registering a minimum of characteristics in advance. However, in this case, the imaging device itself needs to be rotated at high speed, and the mirror needs to be driven at high speed to change the light beam, so changes in the control position due to the vibrations and gravity that occur during this process must be corrected as needed.

[0008] Patent Documents 2 and 3 disclose methods for eliminating the effects of misalignment of the optical system or light receiving element, but do not disclose methods for correcting changes in the control position due to the effects of such vibrations and gravity as needed.

[0009] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an imaging device, a vibration reduction device, a control method, and a program that can accurately and quickly detect a subject to be tracked and perform tracking photography. [Means for solving the problem]

[0010] In order to solve the above problem, the imaging device of claim 1 of the present invention is an imaging device that keeps capturing a main subject at a fixed position on an imaging element by changing the ray direction of an imaging ray at high speed using a ray direction changing device, and is equipped with a focus adjustment means that drives and controls the position of a focus lens to perform focus adjustment, a control direction detection means that detects the control direction of the focus lens by the focus adjustment means, and an incident ray change detection means that detects the rate of change when the ray direction of the imaging ray is changed by the ray direction changing device, and is characterized in that the position of the focus lens is corrected by the focus adjustment means using the control direction detected by the control direction detection means and the rate of change detected by the incident ray change detection means.

[0011] In order to solve the above problem, a blur correction device according to claim 17 of the present invention is a blur correction device that uses light direction changing means to quickly change the ray direction of imaging light rays in order to keep capturing a main subject at a fixed position on an imaging element of an imaging device, and that corrects subject blur caused by the change in the ray direction of the imaging light rays, and is characterized in that the blur correction device comprises: main subject position acquisition means for acquiring the position of the main subject from the imaging device; and control means for controlling the light direction changing means in accordance with the acquired position of the main subject, and the imaging device comprises main subject position detection means for detecting the position of the main subject; subject feature pre-registration means for pre-registering feature amounts to be used when detecting the main subject from among a plurality of feature amounts that characterize the main subject; first main subject detection means for detecting the main subject from the detected subjects using only the feature amounts registered by the subject feature pre-registration means; and second main subject detection means for detecting the main subject from the detected subjects using the plurality of feature amounts, and the position of the main subject is detected by switching between the first main subject detection means and the second main subject detection means in accordance with movement of the detected subject. [Effects of the Invention]

[0012] According to the present invention, it is possible to accurately and quickly detect a subject to be tracked and perform tracking photography. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a diagram showing a schematic configuration of a tracking photography system including an imaging device and a high-speed light direction changing device according to an embodiment of the present invention; [Figure 2] FIG. 2 is a block diagram showing a hardware configuration of the tracking photography system. [Figure 3] 10 is a flowchart of a tracking shooting process executed in the imaging device. [Figure 4] 4 is a subroutine showing details of the main subject detection process in step S304 of FIG. 3. [Figure 5] 5 is a subroutine showing detailed processing of the second main subject detection operation in step S420 of FIG. 4. [Figure 6] 4 is a subroutine showing details of the pre-registration process of subject feature amounts in step S321 of FIG. 3. [Figure 7] 4 is a subroutine showing detailed processing of steps S307 to S311 in FIG. 3. [Figure 8] FIG. 1 is a diagram for explaining a general relationship between an image height and an image plane. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, a tracking and photographing system including an imaging device and a high-speed light direction changing device according to the present invention will be described using examples.

[0015] FIG. 1 is a diagram showing a schematic configuration of a tracking and photography system 100 according to an embodiment of the present invention.

[0016] As shown in FIG. 1, the tracking photography system 100 comprises an imaging device 1 equipped with an interchangeable lens 31, and a high-speed light direction changing device 50 for continuously capturing a main subject at a fixed position on the imaging element of the imaging device 1 by changing the light direction of the imaging light at high speed.

[0017] The high-speed light direction changing device 50 also includes a camera rotation drive device 54 , a mirror 57 , and a mirror drive motor 56 .

[0018] The camera rotation drive device 54 (imaging device rotation means) mounts the imaging device 1 so that it can rotate around the optical axis of the imaging light. By rotating the imaging device 1 around the optical axis, the imaging direction of the camera can be changed in the horizontal direction.

[0019] A mirror drive motor 56 (light beam bending means) drives a mirror 57 that bends the imaging light beam. For example, in the case of the arrangement of the imaging device 1 shown in Fig. 1, the imaging light beam is bent in the tilting direction by driving the mirror 57. On the other hand, when the imaging device 1 is rotated 90 degrees around the optical axis from the arrangement shown in Fig. 1, the imaging light beam is bent in the panning direction by driving the mirror 57.

[0020] The tracking photography system 100 changes the ray direction of the imaging light beam incident on the imaging optical system such as the interchangeable lens 31 in the panning and tilting directions by rotating the camera rotation drive device 54 and the mirror 57 at high speed. The high-speed light beam direction change device 50 also functions as a blur correction device that corrects subject blur of the main subject that occurs when the ray direction of the imaging light beam is changed.

[0021] FIG. 2 is a block diagram showing the hardware configuration of the tracking and photography system 100. As shown in FIG.

[0022] As shown in FIG. 2, the high-speed light direction changing device 50 includes a communication driver 51, a CPU 52, a camera rotation control circuit 53, and a mirror drive circuit 55 in addition to the camera rotation drive device 54, mirror 57, and mirror drive motor 56 described above.

[0023] The communication driver 51 is a communication unit for communicating information with the imaging device 1, and the CPU 52 is a processing unit that controls the entire high-speed light direction changing device 50, which has a built-in memory for calculations.

[0024] A camera rotation control circuit 53 controls the rotation drive of a camera rotation drive device 54 , and a mirror drive circuit 55 controls the drive of a mirror drive motor 56 .

[0025] The CPU 52 is a means for determining a control target position in the high-speed light beam direction change device 50, and calculates a rotation target value for the camera rotation drive device 54 and a target value for the bending angle of the imaging light beam caused by the mirror 57 using information obtained from the imaging device 1. Then, in accordance with these calculated values, the CPU 52 drives and controls the camera rotation drive device 54 and the mirror 57. This makes it possible to change the light beam direction of the imaging light beam incident on the imaging optical system in the panning direction and tilting direction at high speed.

[0026] The imaging device 1 includes a lens mount 2, a solid-state imaging element (hereinafter referred to as a sensor) 3, an imaging circuit 4, an A / D conversion circuit 5, a memory (hereinafter referred to as a VRAM) 6, an image display device (hereinafter referred to as an LCD) 7, and a storage memory 8.

[0027] The imaging device 1 further includes an EEPROM 9, an AE processing circuit 11, an AF processing circuit 12, a shake detection circuit 13, a shake detection sensor 14, a CPU 15, a timing generator (hereinafter referred to as TG) 16, and a sensor driver 17.

[0028] The imaging device 1 also includes a communication driver 21 , a sensor movement motor 22 , a sensor movement control circuit 20 , a motion vector detection circuit 18 , and a main subject detection circuit 19 .

[0029] The lens mount 2 is a mount portion for mounting an interchangeable lens 31, and the sensor 3 is an imaging element that forms an image of imaging light from the interchangeable lens 31 and photoelectrically converts the image.

[0030] The imaging circuit 4 receives the electrical signal photoelectrically converted by the sensor 3 and performs various image processing to generate a predetermined image signal, and the A / D conversion circuit 5 converts the analog image signal generated by the imaging circuit 4 into a digital image signal (image data).

[0031] The VRAM 6 is a memory such as a buffer memory that temporarily stores image data output from the A / D conversion circuit 5. The LCD 7 is a liquid crystal display device that displays menu screens and image data. The storage memory 8 is a storage memory made up of semiconductor memory or the like that stores image data. The LCD 7 is a vari-angle type liquid crystal display device, and although not shown in FIG. 1, the imaging device 1 is placed on the camera rotation drive device 54 in a state where it is opened 180 degrees. However, this is not limited to this as long as the configuration allows the user to check the display of the menu screens and image data. For example, it is also possible to check these displays on an external monitor using a terminal (not shown) on the side of the imaging device 1.

[0032] The EEPROM 9 is an electrically rewritable read-only memory that stores programs for various controls and data used to perform various operations. The AE processing circuit 11 performs automatic exposure (AE) processing on the image signal output from the A / D conversion circuit 5.

[0033] AF processing circuit 12 calculates an AF evaluation value and detects the amount of defocus in order to perform autofocus (AF) processing on the image data upon receiving the output from A / D conversion circuit 5. Shake detection sensor 14 is a sensor that detects movement of the imaging device, such as camera shake, and is composed of inertial sensors such as a gyro sensor and an accelerometer, and by using multiple of these sensors, shake on multiple axes can be detected.

[0034] A shake detection circuit 13 processes signals from a shake detection sensor 14, and a CPU 15 performs overall control of the imaging device 1, which includes a built-in memory for calculations.

[0035] The TG 16 generates a predetermined timing signal, and the sensor driver 17 controls the photoelectric conversion process by the sensor 3 .

[0036] The TG 16 outputs a predetermined timing signal to the CPU 15, the imaging circuit 4, and the sensor driver 17, and the CPU 15 performs various controls in synchronization with this timing signal. The imaging circuit 4 also receives the timing signal from the TG 16 and performs various image processing such as color signal separation in synchronization with this. Furthermore, the sensor driver 17 receives the timing signal from the TG 16 and drives the sensor 3 in synchronization with this.

[0037] A communication driver 21 (communication means) communicates information with the interchangeable lens 31 and the high-speed light direction change device 50, and a sensor movement motor 22 moves the sensor 3 in horizontal and vertical rotation directions. A sensor movement control circuit 20 controls the movement of the sensor movement motor 22. A motion vector detection circuit 18 detects the motion vector of the subject from the image signal output from the A / D conversion circuit 5.

[0038] Main subject detection circuit 19 (main subject position detection means) receives outputs from motion vector detection circuit 18, A / D conversion circuit 5 and CPU 15, and detects the position on sensor 3 of the main subject to be photographed.

[0039] Interchangeable lens 31 includes image stabilization lens 32, focus lens 33, aperture 34, which is a light amount adjustment means for controlling the amount of subject light beam transmitted through these lenses, and communication driver 35 for communicating with imaging device 1. Note that aperture 34, focus lens 33, and image stabilization lens 32 are controlled by a drive control circuit (not shown) that controls a drive motor (not shown) that drives them. Data used to perform various operations in interchangeable lens 31 is stored in advance in an EEPROM (not shown).

[0040] The storage memory 8, which is a storage medium for image data, etc., may take various forms. For example, it may be a fixed semiconductor memory such as a flash memory, or a semiconductor memory such as a card-type flash memory that is card-shaped or stick-shaped and is formed so as to be detachable from the imaging device 1. It may also be a magnetic storage medium such as a hard disk or floppy disk.

[0041] The operation of this embodiment configured as above will be described below.

[0042] The subject light beam that has been transmitted through the interchangeable lens 31 with its light intensity adjusted is formed as a subject image on the light receiving surface of the sensor 3. This subject image is converted into an electrical signal through photoelectric conversion processing by the sensor 3 and output to the imaging circuit 4. The imaging circuit 4 performs various types of signal processing on the input signal to generate a predetermined image signal.

[0043] This image signal is output to an A / D conversion circuit 5 and converted into a digital signal (image data), which is then temporarily stored in a VRAM 6 and then displayed as an image on an LCD 7 .

[0044] On the other hand, the image data stored in VRAM 6 is converted into an image data format suitable for storage and stored in storage memory 8. When playback operation is started, the image data stored in storage memory 8 is subjected to decoding and expansion processes, etc., and then temporarily stored in VRAM 6 and displayed as an image on LCD 7.

[0045] Meanwhile, the TG 16 outputs a predetermined timing signal to the CPU 15, the imaging circuit 4, and the sensor driver 17, and the CPU 15 performs various controls in synchronization with this timing signal. The imaging circuit 4 also receives the timing signal from the TG 16 and performs various image processing such as color signal separation in synchronization with this. Furthermore, the sensor driver 17 drives the sensor 3 in synchronization with this timing signal.

[0046] On the other hand, the image data digitized by the A / D conversion circuit 5 is output not only to the VRAM 6 mentioned above, but also to the AE processing circuit 11, the AF processing circuit 12, the motion vector detection circuit 18, and the main subject detection circuit 19.

[0047] In the AE processing circuit 11, an AE evaluation value according to the brightness of the subject is calculated by, for example, finding a weighted average value using the image data input from the A / D conversion circuit 5 and the main subject detection result by the main subject detection circuit 19, and this value is output to the CPU 15.

[0048] The CPU 15 calculates the exposure time and amplification of the sensor 3 and the aperture value of the diaphragm 34 based on the AE evaluation value from the AE processing circuit 11. Then, based on these calculation results, the CPU 15 controls the exposure time and amplification of the sensor 3 via the TG 16 and sensor driver 17, and transmits information about the aperture value of the diaphragm 34 to the interchangeable lens 31 via the communication driver 21. On the interchangeable lens 31 side, aperture drive processing and the like are performed based on the aperture value information transmitted from the CPU 15, and the aperture value of the diaphragm 34 is adjusted to be appropriate.

[0049] In this way, the CPU 15, TG 16, communication driver 21, diaphragm 34 of the interchangeable lens 31 and its drive control circuit play the role of means for changing exposure control.

[0050] The AF processing circuit 12 detects the amount of defocus by performing image correction for focus adjustment and a correlation calculation of the image data after the image correction on the image data input from the A / D conversion circuit 5. The CPU 15 calculates the amount and direction of drive of the focus lens 33 based on the detected amount of defocus, and transmits the calculation results as lens drive control information to the interchangeable lens 31 via the communication driver 21. The interchangeable lens 31 performs focus adjustment control to drive the focus lens 33 based on the lens drive control information transmitted from the CPU 15.

[0051] In this way, the focus lens drive control circuit, which is made up of the AF processing circuit 12, CPU 15, communication driver 21, and interchangeable lens 31, plays the role of focus adjustment means that controls the drive of the focus lens 33 to the in-focus position.

[0052] Furthermore, CPU 15 detects, based on the lens drive control information, whether the direction (control direction) in which the focus adjustment means drives and controls focus lens 33 is a direction in which the positioning accuracy of focus lens 33 to the control position will be reduced due to the influence of gravity. That is, CPU 15 plays the role of control direction detection means that detects the control direction by the focus adjustment means in order to identify the positioning accuracy of focus lens 33, which changes depending on the drive direction.

[0053] Furthermore, the motion vector detection circuit 18 receives the input digital image signal (standard image) and performs a correlation calculation with the digital image signal (reference image) of the previous frame for each divided area in response to an instruction from the CPU 15. The motion vector detection circuit 18 further obtains the motion vector of the subject based on the result of this correlation calculation.

[0054] Specifically, in the correlation calculation, the pixel shift amount that gives the highest correlation (smallest difference amount) is found for each divided area by calculating the difference between the standard image and the reference image while shifting the reference image by a predetermined number of pixels in the horizontal and vertical directions of the sensor 3. Next, the motion vector of the object is found by taking the pixel shift amount found as the amount of movement of the object in that area and the direction of pixel shift in the horizontal and vertical directions at that time as the direction of movement.

[0055] Incidentally, since the detection of a motion vector is a known technique described in Japanese Patent No. 3143173 and the like, a detailed description thereof will be omitted.

[0056] Main subject detection circuit 19 (main subject detection means) detects the position of the subject to be photographed. This detection of the subject position is performed using information processed by CPU 15, such as image data input from A / D conversion circuit 5, subject movement information obtained from outputs of motion vector detection circuit 18, blur detection circuit 13, and AF processing circuit 12, and user instruction information.

[0057] Then, main subject detection circuit 19 comprehensively detects a plurality of main subject regions from the detected subject positions, and transmits the ranking results to CPU 15.

[0058] The CPU 15 receives, via the communication drivers 21 and 51, from the high-speed light beam direction changing device 50, information regarding the maximum speed at which the camera rotation drive device 54 will drive the camera, the speed at the time of communication, information regarding change in direction after communication, and the maximum drive speed limit. The CPU 15 also receives information regarding the structure of the high-speed light beam direction changing device 50 that will affect the vibration of the tip of the mirror 57, as well as the maximum speed at which the direction of the imaging light beam incident on the mirror 57 will change, the speed at the time of communication, information regarding change in direction of the imaging light beam after communication, and the maximum drive speed limit. Examples of the information regarding the structure of the high-speed light beam direction changing device 50 include the overall size of the mirror 57, the size of the drive shaft of the mirror 57, and the size of the portion of the mirror 57 that bends the imaging light beam.

[0059] This enables the CPU 15 to determine whether there is a possibility that the image pickup device 1 or the mirror 57 will be driven at high speed by the high-speed light direction change device 50 thereafter.

[0060] The CPU 15 constantly acquires drive information such as the camera rotation drive speed of the camera rotation drive device 54 and the drive speed and drive information of the mirror 57 that bends the imaging light beam at regular intervals, but other information may also be acquired at specific timings such as when the imaging device 1 is started up or when the interchangeable lens 31 is attached to the imaging device 1, for example.

[0061] Then, the CPU 15 (incident light ray change detection means) uses the information obtained from the high-speed light ray direction change device 50 to determine the speed of change (speed, direction, and manner of change) of the direction of the imaging light ray.

[0062] Furthermore, the position of the subject to be photographed is transmitted from the imaging device 1 to the high-speed light direction changing device 50 every time the main subject detection circuit 19 detects the position.

[0063] In this way, CPU 15 plays the role of subject position notifying means for notifying high speed light direction changing device 50 (light direction changing means) of the main subject position.

[0064] In addition, the imaging device 1 transmits to the high-speed light beam direction change device 50 information such as the position of the focus lens 33, the focal length of the interchangeable lens 31 at that time, exposure control information, shooting timing, and AF-related information. The exposure control information includes information on the aperture value, shutter speed, and ISO sensitivity, while the AF-related information includes information on the focal length, focus sensitivity, and focus lens drive direction and speed. Generally, when focus sensitivity is high, the focus lens 33 is more susceptible to the effects of external vibrations and the like. Therefore, in this embodiment as well, information is exchanged between the imaging device 1 and the high-speed light beam direction change device 50, because the degree of influence of vibrations and the like caused by the drive of the high-speed light beam direction change device 50 varies depending on the focus sensitivity.

[0065] Using this information in the high-speed light direction changer 50, the CPU 52 may change the drive control of the camera rotation drive device 54 and the mirror 57.

[0066] Next, a tracking shooting process executed in the imaging device 1 will be described with reference to the flowchart shown in Fig. 3. This process is executed by the CPU 15 reading a program from the EEPROM 9 and expanding it in a RAM (not shown).

[0067] In this process, a main subject detection operation and an operation for correcting the control position of the focus adjustment means according to the detected main subject are executed. In addition, the main subject detection operation is switched appropriately between a first main subject detection operation (first main subject means) that detects the position of the subject by pre-registering subject features, and a second main subject detection (second main subject detection means) that detects the position of the subject using multiple subject features.

[0068] In FIG. 3, first, in step S301, when an instruction is given to turn on the power switch (not shown) of the imaging device 1, an initial operation is first executed.

[0069] Specifically, it performs processes such as initializing variables used in each process executed inside the imaging device 1, moving the camera rotation drive device 54 and mirror 57 to their initial positions, and acquiring information from the currently attached interchangeable lens 31. Note that the information acquired from the currently attached interchangeable lens 31 here includes, for example, information about the image stabilization lens 32, focus lens 33, aperture 34, and focal length.

[0070] In step S302, AE processing is executed to calculate the exposure time and amplification of the sensor 3 and the aperture value of the diaphragm 34. Next, based on the calculation results, the exposure time (accumulation time) of the sensor 3, the sensor amplification, and the drive of the diaphragm 34 are controlled, and the image formed on the sensor 3 is displayed on the LCD 7 as an image.

[0071] In step S303, it is determined whether or not subject feature pre-registration has been performed to register features to be used during photography. If subject feature pre-registration has not been performed (NO in step S303), the process proceeds to step S321. If subject feature pre-registration has been performed (YES in step S303), the process proceeds to step S304.

[0072] In step S304, a main subject detection process is performed.

[0073] First, depending on the movement of the subject at the time of detecting the position of the subject to be photographed, it is determined in step S321 whether to detect the subject position using only the registered subject feature amounts or to detect the subject position using multiple subject feature amounts.The main subject is then detected using the method determined.The position of the main subject detected here (main subject detection information) is acquired by CPU 52 (main subject position acquisition means / control means) via communication drivers 21, 51, and the drive of camera rotation drive device 54 and mirror drive motor 56 is controlled according to this acquired position.The main subject detection process in step S304 will be described in detail later with reference to FIG. 4.

[0074] In step S305, it is determined whether or not AE processing or AF processing (half-press operation) that is performed prior to release has been instructed. If these processes have not been instructed (NO in step S305), the process returns to step S302, and if they have been instructed (YES in step S305), the process proceeds to step S306. Note that if the user has difficulty half-pressing the release button on the imaging device 1 arranged as shown in FIG. 1, the user may perform the half-press operation using a cable release or remote control.

[0075] In step S306, AE processing is performed, and in step S307, AF processing is performed.

[0076] Specifically, CPU 15 first detects the defocus amount for the main subject determined by main subject detection circuit 19 using AF processing circuit 12, and calculates the drive amount and drive direction (lens drive control information) of focus lens 33 based on the detected defocus amount. Next, CPU 15 transmits the lens drive control information resulting from the calculation to interchangeable lens 31. As a result, drive processing of focus lens 33 is performed on the interchangeable lens 31 side, and a focused state is achieved.

[0077] In step S308, the speed of change in the direction of the imaging light beam is calculated from the driving information of the camera rotation driving device 54 and the mirror 57 that bends the imaging light beam, obtained from the high-speed light beam direction changing device 50.

[0078] In step S309, it is determined whether or not correction of the control position of the focus lens 33 is necessary based on the rate of change in the ray direction of the imaging light ray calculated in step S308 and the drive amount and drive direction of the focus lens 33 calculated in step S307. If the determination result shows that correction is necessary, the control position is corrected, and then the process proceeds to step S310. As described above, whether or not this correction is necessary is determined based on the positioning accuracy of the control position, which changes depending on the drive direction of the focus lens 33 during AF processing. This makes it possible to accurately focus on the main subject. Note that if correction of the control position is not necessary, the process proceeds to step S310 without correcting the control position.

[0079] In this way, CPU 15 serves as a control direction detection means for detecting the control direction of the focus adjustment means, and evaluates the positioning accuracy of the control position of focus lens 33 according to the detection result and the speed of change in the direction of the incident light beam.

[0080] The details of steps S307 to S309 and S311 will be described later with reference to FIG.

[0081] In step S310, it is determined whether a release command (full press operation) has been given, and if no command has been given (NO in step S310), the process returns to step S305, and if a command has been given (YES in step S310), the process proceeds to step S311. Note that if it is difficult for the user to fully press the release button on the imaging device 1 arranged as shown in Fig. 1, the user may perform the full press operation using a cable release or remote control.

[0082] In step S311, similar to step S309, it is determined whether or not the control position of focus lens 33 needs to be corrected in order to accurately focus on the main subject, and if so, the control position is corrected. At the same time, it is determined whether or not a change in exposure control and an accompanying correction by image processing are necessary, and if so, instructions are given to change the exposure control (here, adjusting the depth of field and ISO sensitivity) and correct by image processing, after which the process proceeds to step S310. Details of this change in exposure control and the determination of whether or not the accompanying correction by image processing is necessary will be described later using FIG. 7.

[0083] In step S312, exposure processing is performed in accordance with the instructions in step S310.

[0084] In step S313, if an instruction to perform correction by image processing has been given in step S311, background blurring is performed by image processing, and then the process returns to step S303.

[0085] Specifically, background blurring is performed by first equally dividing the area of ​​the captured image and then calculating distance information for each small area from the AF results for each small area. For example, this can be calculated based on the Gaussian imaging equation using a=b×f / (bf) (where a is the distance from the focus lens 33 to the subject, b is the distance from the focus lens 33 to the sensor 3, and f is the focal length of the focus lens 33).

[0086] A small area equidistant from the main subject is then determined as the main subject area, and an area of ​​any shape including the main subject area is identified by contour extraction.

[0087] Next, the image of this main subject region is separated and extracted, thereby separating the photographed image into a main subject image and a background image.

[0088] A spatial filter is applied to the background image to generate a blurred background image, which is then overlaid with the main subject image and synthesized to generate a blurred background image, which is then displayed on the LCD 7.

[0089] Note that background blurring is a known technique described in, for example, Japanese Patent Application Laid-Open No. 2009-218708, and therefore a detailed description thereof will be omitted.

[0090] On the other hand, if the process proceeds to step S321, a subject feature pre-registration process (subject feature pre-registration means) is performed to pre-register features to be used for detecting the subject position at the time of shooting, among a plurality of subject feature values ​​that characterize the subject. Details of this process will be described later with reference to FIG. 5.

[0091] Next, the main subject detection process performed in step S304 will be described with reference to FIG.

[0092] First, in steps S401 and S402, a determination is made as to whether to switch between the first and second main subject determination operations, and depending on the result of that determination, the process proceeds to either step S403, S404, or S420. This determination is made based on the movement of the subject during actual shooting.

[0093] First, when the subject being photographed is moving rapidly, it is necessary to detect the subject position at a higher speed, but when the subject is moving slowly, it is necessary to detect the subject position more accurately.

[0094] In step S401, it is determined whether the subject is moving rapidly, i.e., whether the absolute value of the amount of motion of the subject is greater than or equal to a threshold value. The amount of motion of the subject is calculated from the outputs of the blur detection circuit 13 and the motion vector detection circuit 18.

[0095] If it is determined that the subject is moving rapidly (YES in step S401), the process proceeds to step S403, where main subject detection circuit 19 executes a first main subject detection operation using the registered feature amount, and then the process proceeds to step S407. On the other hand, if it is not (NO in step S401), the process proceeds to step S402.

[0096] 4, if the subject feature amounts have not been pre-registered in step S321, the first main subject detection operation cannot be executed. Therefore, in this case, the process starts from step S420, and the second main subject detection operation using multiple feature amounts is executed by main subject detection circuit 19. Note that if the subject feature amounts have been pre-registered, the first main subject detection is selected for the initial main subject detection.

[0097] In step S402, the judgment indices 1 and 2 are calculated, and it is determined whether or not they are equal to or less than the respective judgment thresholds.

[0098] Determination index 1 is the standard deviation of the difference between the moving average of the movement of the detected subject and the original detection signal, and is an index for evaluating the variation in detection accuracy when the first main subject detection operation is performed. If the value of determination index 1 is large, the variation in detection accuracy in the first main subject detection operation is large, and it is necessary to change the content of the subject features pre-registered for the first main subject detection operation.

[0099] The judgment index 2 is the sum of the absolute values ​​of the difference between the moving average of the detected subject's movement and the original detection signal, and is an index for evaluating the degree of irregular movement of the subject being photographed. If the value of judgment index 2 is large, the subject is moving irregularly, and it is necessary to change the content of the subject feature amounts pre-registered for the first main subject detection operation.

[0100] Therefore, if the comparison result in step S402 shows that both determination indices 1 and 2 are equal to or less than the determination threshold, the process proceeds to step S403, where a first main subject detection operation is executed by main subject detection circuit 19. In this case, the contents of the subject feature amounts pre-registered for the first main subject detection operation are not changed. Then, the process proceeds to step S407.

[0101] If the comparison result in step S402 shows that one of the judgment indicators 1 and 2 exceeds the judgment threshold and the other is equal to or less than the judgment threshold, the process proceeds to step S404.

[0102] In step S404, a second main subject detection operation that prioritizes pre-registration is executed by main subject detection circuit 19. Specifically, after first executing the first main subject detection operation, the second main subject detection operation is executed using, of the multiple subject feature amounts used in the second main subject detection operation, subject feature amounts excluding the subject feature amounts used in the first main subject detection operation. Then, the process proceeds to step S405.

[0103] In step S405, it is determined whether or not to change the contents of the pre-registered subject features based on the execution result of the second main subject detection operation with priority given to pre-registration in step S404. Specifically, in step S404, if the main subject could not be detected in the first main subject detection operation, and if the difference between the subject positions detected in the first and second main subject detection operations is greater than a predetermined value, it is determined that the registration contents need to be changed. In this case (YES in step S405), the process proceeds to step S406, where the pre-registration contents are changed, and then the process proceeds to step S407. On the other hand, if the difference between the subject positions detected in the first and second main subject detection operations is equal to or less than the predetermined value in step S404 (NO in step S405), the process proceeds directly to step S407.

[0104] On the other hand, if the comparison result in step S402 shows that both determination indices 1 and 2 are greater than the determination threshold, the process proceeds to step S420, where a second main subject detection operation is executed by main subject detection circuit 19, and the process ends.

[0105] By doing this, if there is a large difference between the detection results of the second main subject detection and the first main subject detection, detailed main subject detection using the second main subject detection can be continued until the movement of the subject becomes large.

[0106] The first and second main subject detection operations will be described in detail later.

[0107] The respective determination thresholds of the determination indicators 1 and 2 used in the determination in step S402 are changed according to various conditions.

[0108] For both judgment indexes 1 and 2, the judgment threshold is changed according to the continuous shooting speed, the sensor frame rate during video recording, and the number of readout pixels or readout range associated therewith.

[0109] During high-speed continuous shooting at a high frame rate, the first main subject detection operation (detection using only pre-registered subject features) is used more frequently, so the determination thresholds for determination indices 1 and 2 are increased.

[0110] In addition, if the number of read pixels or read range used for detection is smaller than when all pixels that can be captured are used normally, the judgment thresholds for judgment indicators 1 and 2 are reduced to increase the frequency of using the second main subject detection operation (detection using multiple subject features).

[0111] Specifically, the following calculation formula is used: Threshold of judgment index 1 = α1 × continuous shooting frame rate ÷ reference frame rate + α2 × frame rate ÷ reference frame rate +α3 × total readout pixels ÷ total screen pixel count Threshold of judgment index 2 = β1 × continuous shooting frame rate ÷ reference frame rate + β2 × frame rate ÷ reference frame rate +β3 × total readout pixels ÷ total screen pixel count Here, α1, α2, α3, β1, β2, and β3 are predetermined values.

[0112] This is because, during high-speed continuous shooting and at a high frame rate, there is a high probability that a main subject moving at high speed will be photographed, and therefore it is necessary to detect the main subject at a higher speed.

[0113] Furthermore, if the number of pixels to be read or the read range to be used for detection is set small, the load required for detecting the main subject is reduced, and the probability of detection being performed quickly increases, thereby achieving higher accuracy.

[0114] Returning to FIG. 4, in steps S407 to S410, the position of the detected main subject is corrected.

[0115] In step S407, the position of the main subject on the screen is converted into a position in space.

[0116] Specifically, distance information about the main subject is first obtained based on the Gaussian imaging equation from the panning and tilting amounts of the image capture device 1 obtained from the output of the shake detection circuit 13 and the lens extension amount obtained from the output of the AF processing circuit 12. Next, the horizontal and vertical positions of the main subject in space are determined from the obtained distance information about the main subject. The position along the optical axis is the shooting distance of the main subject.

[0117] However, since these are theoretical values ​​obtained by calculation, they may differ from the actual spatial position of the main subject. If it is determined that there is a large error from the theoretical values ​​based on the design information of the interchangeable lens 31 actually used, the obtained theoretical values ​​are corrected using the shooting magnification of the interchangeable lens 31 (which varies depending on the distance and image height when in focus).

[0118] Next, in step S408, the history of the position information of the main subject that has been converted into space up to that point is read in. Assuming that the main subject is being detected at 30 fps, it is desirable to read a history of about one second in the past.

[0119] Then, in step S409, since it can be assumed that the main subject moves continuously in real space, this is utilized to correct the detected position of the main subject on the screen.

[0120] Due to influences such as control delays of the camera rotation drive device 54 and mirror 57, and vibrations at the tip of the mirror 57, the position of the main subject on the screen is observed with an error from the actual spatial movement of the subject.

[0121] Therefore, since it can be assumed that the main subject moves continuously in real space, this is utilized to correct the position of the main subject converted into the space obtained in step S407.

[0122] Specifically, the main subject position information obtained in step S407 is added to the history of position information of the main subject read in step S408, and the movement is approximated to an N-order function using the least squares method for correction. Assuming that the main subject is being detected at 30 fps, history from the past about one second is used, so it is desirable to approximate to an N-order function of about sixth degree.

[0123] The value at the current time obtained from the approximated Nth-order function is taken as the position of the main subject in space, and this is converted to a position on the screen to be the corrected position of the subject. This value is used as the position of the main subject in the processing of step S306.

[0124] In step S410, the position of the main subject in space at the current time calculated from the approximated Nth-order function is recorded as a history of position information, after which the process ends.

[0125] Next, detailed processing of the second main subject detection operation, which is executed in step S420 to detect the position of the subject by comprehensively determining multiple subject feature amounts, will be described using the flowchart in Fig. 5. This processing is executed by main subject detection circuit 19, and the earlier a main subject is detected, the higher the priority of that main subject.

[0126] The process in FIG. 5 starts after all main subject detection flags are turned off.

[0127] First, in step S201, it is determined whether or not a main subject area (AF measuring point) has been designated by the photographer. In this determination, first, main subject detection circuit 19 acquires AF point information from CPU 15. Then, based on the acquired AF point information, if the photographer has designated an arbitrary AF measuring point by operating a menu screen displayed on LCD 7 or by touching image data displayed on LCD 7, it is determined that an AF measuring point has been designated.

[0128] If an AF ranging point has been designated (YES in step S201), the process proceeds to step S202. If an AF ranging point has been designated by operating the menu screen, the designated AF ranging point is set as the main subject area. On the other hand, if an AF ranging point has been designated by touching the image data displayed on the LCD 7, a predetermined area centered on the AF ranging point and a nearby area containing a subject similar to the subject detected in that area are set as the main subject area.

[0129] For example, it is checked whether a face has been detected within a predetermined area centered on an AF range-finding point specified by a touch operation. The size of the predetermined area set by a touch operation may be determined in advance. The size may also be variable depending on the focal length of the focus lens 33.

[0130] If a face area is detected within the predetermined area, the main subject area is determined based on the center coordinates and size of the detected face area. Note that multiple face areas may be detected, some of which may be partially within the predetermined area. In this case, the detected face area whose center coordinates are closest to the center of the predetermined area, the face area with the largest size if they are equally close, or the face area detected first if they are the same size, is selected as the main subject area. Furthermore, if the detected face is large and its pupils are detectable, the face area containing two pupils or the face area containing the largest pupil is selected as the main subject area.

[0131] On the other hand, if an AF measuring point is not specified (NO in step S201), the process proceeds to step S203, where it is determined whether a face has been detected. If a face has not been detected (NO in step S203), the process proceeds to step S205. If a face has been detected (YES in step S203), the process turns on a face detection flag for the area where the face has been detected, and then proceeds to step S221.

[0132] In step S221, it is determined whether the detected face can be considered the same as the main subject up to the previous frame. If it can be considered the same (YES in step S221), proceed to step S204, select the detected face area as the main subject area, and then end this process. On the other hand, if it cannot be considered the same (NO in step S221), proceed to step S205.

[0133] If a face has been detected (YES in step S203) and the face size is equal to or larger than a certain size, pupil detection is performed and if either the left or right pupil is detected, the pupil detection flag is turned on for the area where the face has been detected, and then the process proceeds to step S221. In step S221, it is determined whether the face including the detected pupil can be considered to be the same as the face that was the main subject up to the previous frame.

[0134] If face authentication is enabled, after face detection, the system detects the facial features such as the pupils and nose, compares their shapes and positions with the facial authentication information, and authenticates each individual's face. Then, the system turns on the face detection flag and proceeds to step S221. Here, the facial authentication information refers to information indicating the shapes and positions of the facial features of each individual. In this case, in step S221, it is determined whether the subject identified by the authentication information can be considered the same as the main subject up to the previous frame. This determination of whether the subject can be considered the same is made using a function of the main subject detection circuit 19, which will be described in detail later, that tracks subjects presumed to be the same.

[0135] In step S205, it is determined whether a person or the like has been detected.

[0136] This determination is made by inputting the acquired image into a detection circuit with coefficients set using deep learning functions to determine its similarity to people or specific animals (e.g., dogs, cats, horses, deer, etc.), or to the shape of specific vehicles (cars, trains, airplanes), and obtaining the determination result as output.

[0137] If a person or the like is not detected (NO in step S205), proceed to step S207; if a person or the like is detected (YES in step S205), the person or the like detection flag is turned on for the area where a person or the like is detected, and then proceed to step S222.

[0138] In step S222, it is determined whether the detected person or the like can be considered the same as the main subject up to the previous frame. If it can be considered the same (YES in step S222), proceed to step S206, select the area of ​​the detected person or the like as the main subject area, and then end this processing. On the other hand, if it cannot be considered the same (NO in step S222), proceed to step S207.

[0139] In step S207, it is determined whether or not a moving object (a subject that moves in at least one of the horizontal, vertical and optical axis directions in the real space that the photographer is trying to capture within the screen) is present.

[0140] If no moving object is present (NO in step S207), the process proceeds to step S209; if a moving object is present (YES in step S207), the moving object detection flag is turned on for the area where the moving object is detected, and then the process proceeds to step S223.

[0141] In step S223, it is determined whether the detected moving object can be considered the same as the main subject up to the previous frame. If it can be considered the same (YES in step S223), proceed to step S208, select the moving object as the main subject area, and then end this process. On the other hand, if it cannot be considered the same (NO in step S223), proceed to step S209.

[0142] Here, the determination in step S207 as to whether or not a moving object is present is made using the outputs of the blur detection circuit 13, the motion vector detection circuit 18, and the AF processing circuit 12.

[0143] There may be cases where the output of the shake detection circuit 13 is small (when both the detected values ​​for the horizontal and vertical axes are less than a predetermined value), that is, when the photographer does not intentionally move the imaging device 1. In this case, if there is an area where the amount of motion of the motion vector detected by the motion vector detection circuit 18 is equal to or greater than a predetermined value Vc, that area is determined to be the main subject area.

[0144] In this case, the main subject region is calculated specifically as follows. First, main subject detection circuit 19 divides the light receiving surface of sensor 3 into 256 regions, 16 horizontally and 16 vertically. Next, motion vectors in the horizontal and vertical directions detected by motion vector detection circuit 18 in region nm, which is the nth region horizontally (n=1 to 16) and the mth region vertically (m=1 to 16), are obtained as Vhor(nm) and Vver(nm), respectively. Then, the amount of motion Vu(nm) for region nm is calculated from the obtained horizontal and vertical motion vectors as follows: Vu(nm)= √[Vhor(nm)×Vhor(nm)+Vver(nm)×Vver(nm)] If Vu(nm) is equal to or greater than a predetermined value Vc, the main subject detection circuit 19 determines that the region is moving, and merges adjacent moving regions if any exist. If there are multiple moving regions, the circuit 19 selects the one closest to the center of the screen as the main subject region.

[0145] On the other hand, when the output of shake detection circuit 13 is large (when either the horizontal or vertical axis detection value is equal to or greater than a predetermined value), that is, when the photographer is intentionally moving imaging device 1, the area exhibiting the same movement as this intentional movement is determined to be the main subject area. That is, if there is an area where the amount of movement of the motion vector detected by motion vector detection circuit 18 is equal to or less than a predetermined value Vd (≧Vc), that area is determined to be the main subject area.

[0146] Specifically, the main subject region in this case is calculated as follows. First, main subject detection circuit 19 calculates the amount of motion Vu(nm) of region nm detected by motion vector detection circuit 18 in region nm, as in the method described above. If the amount of motion Vu(nm) is equal to or less than a predetermined value Vd, that region is determined to be a region containing a moving object being pursued by the photographer, and if there are adjacent regions containing moving objects, they are merged. Furthermore, if there are multiple regions containing moving objects, the region closest to the center of the screen is selected as the main subject region.

[0147] If an area containing a moving object cannot be detected from the amount of movement of the motion vector detected by motion vector detection circuit 18, main subject detection circuit 19 determines that the moving object moving in the distance direction (optical axis direction) is the main subject area. This main subject area is detected by checking whether there is an AF point whose amount changes in the same direction over time from the distance or defocus amount obtained from AF processing circuit 12 (for example, an AF point whose distance is decreasing for five consecutive frames).

[0148] Specifically, the main subject area in this case is calculated as follows: First, the change in distance or defocus amount for each frame is checked for all AF points obtained from the AF processing circuit 12. If an AF point is extracted that changes in the same direction by a predetermined amount over a predetermined number of frames or more, the extracted AF point is determined to be an area containing a moving object, and if there are adjacent areas containing moving objects, they are merged. Furthermore, if there are multiple areas containing moving objects, the area closest to the center of the screen is selected as the main subject area.

[0149] In step S209, it is checked from the position and size on the screen whether an area having a specific pattern or an area with a high likelihood of being the main subject has been detected among a group of areas with similar colors or brightness.

[0150] If an area having a specific pattern or an area with a high likelihood of being the main subject is detected (YES in step S209), the main subject detection flag is turned on and the process proceeds to step S224. In step S224, it is determined whether the subject in the detected area can be considered to be the same as the main subject up to the previous frame. If it can be considered the same (YES in step S224), the process proceeds to step S210, where the detected area is selected as the main subject area and the process ends. On the other hand, if it cannot be considered the same (NO in step S224), the process proceeds to step S225.

[0151] Here, the determination of whether an area has a specific pattern is made based on information about the shape and pattern of the clothing worn by the person or parts of the clothing, and information about the shape and pattern of accessories. Specifically, the photographer evaluates the similarity of the image included in the area to be determined with pre-registered shapes and patterns, and the area where the similarity is greater than or equal to a certain value and is the highest is determined to be the main subject area. In step S209, detection of an area with this specific pattern takes priority over detection of an area with a collection of similar colors or brightness.

[0152] If there is no region with a similarity equal to or greater than a certain value, a region that does not border two of the four sides of the screen and includes a group of similar colors and brightness and is equal to or greater than a certain size is determined to be a region with a high likelihood of being the main subject. If there are multiple such regions, a region with a center of gravity of the group closest to the center of the screen is selected as a region with a high likelihood of being the main subject.

[0153] In step S225, a determination is made as to whether or not a main subject candidate has been detected in the processing up to that point by checking the detection flags. If the result of this determination is that any of the detection flags is on (YES in step S225), it is determined that a main subject candidate has been detected, and the process proceeds to step S226. On the other hand, if none of the detection flags is on (NO in step S225), no main subject candidate has been detected in the processing up to that point, and the process proceeds to step S211.

[0154] In step S226, the area with the highest priority is selected from among the areas with the detection flag turned on, and this area is set as the main subject area, and this process ends.

[0155] Specifically, if there is an area where the face detection flag is on, that area is designated as the main subject area, if there is an area where the face detection flag is off but the person etc. detection flag is on, that area is designated as the main subject area, if there is an area where both the face detection flag and the person etc. detection flag are off but the moving object detection flag is on, that area is designated as the main subject area, and if there is an area where both these flags are off but the main subject detection flag is on, that area is designated as the main subject area.

[0156] In step S211, it is determined from the processing results of the AF processing circuit 12 whether or not there are a plurality of subjects with a high proportion of AF points (distance measurement points) within the image plane where AF (distance measurement) is possible and at different distances.

[0157] If there are multiple subjects (YES in step S211), proceed to step S212, and the AF point that shows the closest AF result among the multiple AF points is set as the main subject area, and this process ends. In this case, since the entire screen is not the subject, such as in landscape photography, but rather is likely to be a commemorative photo with the landscape as the background, the closest subject is set as the main subject area.

[0158] If there are no such multiple subjects (NO in step S211), the process proceeds to step S213. In step S213, the area for performing AF is determined based on the AE processing result, shooting mode, shutter speed, aperture value, and strobe ON / OFF information set by the photographer obtained from the CPU 15.

[0159] Specifically, in step S213, first, the main subject area is determined based on the photographing mode and its accompanying conditions shown in Table 1 below.

[0160] [Table 1]

[0161] If none of the conditions in Table 1 apply, in step S213, the main subject area is determined based on the brightness and its accompanying conditions shown in Table 2 below.

[0162] [Table 2]

[0163] If multiple moving objects such as faces or people are detected, the main subject area is determined based on their detected position and size (faces) or their detected position (moving objects such as people), as described above. Main subjects with different priorities may also be detected, in which case the main subject area is determined according to the priority.

[0164] In this way, the main subject area is initially determined, but from the next frame onwards, priority is given to the same subject.

[0165] The main subject detection circuit 19 has the function of tracking a subject that is presumed to be the same as the subject detected as the main subject in the previous frame. With this function, if the subject detected as the main subject in the above procedure is presumed to be the same as the main subject up to the previous frame, that subject is given the highest priority as the main subject. Even if there is a main subject that was detected earlier in the above procedure, the main subject that is presumed to be the same is given priority.

[0166] Next, a function for tracking a subject that is estimated to be the same as the subject detected as the main subject in the previous frame will be described.

[0167] The method for estimating the same subject differs depending on which of the above conditions the main subject region is detected under.

[0168] If the photographer has specified an AF point of their choice, the area containing that AF point will always be considered the main subject area. However, because there is a possibility that the main subject may temporarily shift to an adjacent AF point, AF information from adjacent AF points above, below, left, and right is also acquired. Furthermore, if the AF information from the specified AF point changes significantly and nearly identical AF information is obtained from adjacent AF points, the adjacent AF point will be considered the main subject area. In other words, the subject specified by the photographer at the time of the photographer's intention will be considered the main subject, without any estimation of whether they are the same subject.

[0169] When the main subject area is determined based on the face detection result, the pupil detection result performed after face detection, or the detection result of a person, etc., first, a face, pupil, or person, etc. is detected. After that, if a face, pupil, or person of approximately the same size as the detected one is found in the same position as the previous frame (the difference in the center coordinates of the detection result is within a predetermined value), it is assumed to be the same subject, and the newly detected area is set as the main subject area. Note that when left and right pupils are detected, they are assumed to be the same subject if they are approximately the same size and located in the same position.

[0170] Although it requires a heavy computational load, it is also possible to identify the same subject using face recognition. This involves comparing the shape and position of facial organs such as the pupils and nose with face recognition information to authenticate each individual's face and determine whether the faces are presumed to be the same, thereby determining whether they can be presumed to be the same subject.

[0171] If the main subject area is determined according to the movement of a moving object, the moving object area is detected using the procedure described above, and the main subject area is selected from the detected position (center coordinates of the area) and the motion vector of that area. If the position and motion vector of the detected moving object area are the same as those detected in the previous frame, it is assumed to be the same object, and the newly detected area is set as the main subject area.

[0172] In addition, quadratic function approximation can be performed using motion vectors (amount and direction of movement) over several frames to estimate the movement of the subject, and if the difference in the amount of movement calculated from the motion vector relative to the estimated amount of movement is within a predetermined value, the subject is assumed to be the same, and that area can be used as the main subject area.

[0173] Similarly, for a moving object moving in the optical axis direction, if the position and motion vector of a newly detected moving object area are the same as those detected in the previous frame, it is assumed to be the same subject, and the newly detected area is set as the main subject area.

[0174] Similarly, when a region that is highly likely to be the main subject among regions containing clusters of similar colors and brightness is designated as the main subject region, if the size and position of a newly detected cluster of similar colors and brightness are similar to those detected in the previous frame, it is estimated to be the same subject, and the newly detected region is designated as the main subject region.

[0175] When the closest subject is determined to be the main subject region (step S212 in FIG. 5), a new main subject region is searched for without determining whether the subject is estimated to be the same as the subject.

[0176] In addition, if panning or tilting that exceeds the angle of view, an extreme change in brightness, a change to playback mode, recording stopping, or power off occurs, it is determined that shooting of the main subject up to that point has ended, and the main subject area information detected up to that point is erased.

[0177] Next, the subject feature pre-registration process in step S321 will be described with reference to FIG.

[0178] This process is a process for registering in advance, among a plurality of subject feature amounts that characterize a subject, feature amounts to be used for subject position detection during photography.

[0179] In most normal main subject detection processes, only the image taken immediately before shooting is used, but in this process, main subject detection is performed using multiple images, and the feature amounts of the main subject to be used during shooting are identified from the detection results.

[0180] First, in step S601, previously captured images are loaded into a list of images used when detecting pre-registered subject features. The images loaded here are selected by the photographer from among previously captured images taken with the imaging device 1 and stored in the storage memory 8 installed when step S601 is executed. For example, images of the photographer's own child at an athletic meet, images of the athlete being photographed, and images of their uniforms are examples of images loaded into the list here.

[0181] Furthermore, by replacing (attaching) the storage memory 8 when step S601 is executed, it is also possible to load a retouched image onto the wrist. For example, an image obtained by combining a previously taken image of the photographer's own child with an image stored in the replaced storage memory 8 of the clothes, socks, accessories, etc. that the child was wearing that day.

[0182] In step S602, if practice photography is possible before the actual photography, images selected by the photographer from among the practice images are also included in the list.

[0183] In step S603, accompanying information and an image relating to the subject are acquired.

[0184] In this acquisition, first, image features of the target player, etc. are acquired from publicly available video, for example, video acquired from the Internet. Next, the acquired image features are compared with the image features of the images added to the list in steps S601 and S602. As a result of the comparison, the acquired image features may include past incidental information about the individual subject, such as changes to the player's game uniform or jersey number, changes to the uniform design, or the use of a different uniform than usual. In this case, an image having those image features is acquired from the Internet, etc., and the acquired image and incidental information are added to the list. Incidental information may also include public information about the event by the organizer.

[0185] In steps S601 and S602, the accompanying information and images to be acquired in step S603 may be acquired in advance and added to the list.

[0186] Furthermore, if there are no retouched images in step S601, images of the clothes, socks, accessories, etc. worn by the photographer's own children on that day also fall under the category of accompanying information and images related to the subject of photography acquired in step S603.

[0187] Also, in step S603, a single image of clothing, socks, etc. may be added to the list. In this case, by linking the image selected in the next step S604 with this single image, it becomes possible to efficiently and accurately detect the main subject.

[0188] In step S604, an image to be used in the second main subject detection operation in step S605 is selected from the images added to the list in step S603. Image selection here can be performed automatically or manually after other processing, including correction.

[0189] When the type of uniform or outfit to be worn on that day is determined from the accompanying method obtained in step S603, the corresponding image is automatically selected.

[0190] Furthermore, if the clothes, socks, accessories, etc. worn that day are listed, the image is linked to the person wearing them. Here, if the face recognition result shows that only one person is listed in steps S601 and S602, the image is automatically linked to that person. On the other hand, if the face recognition result shows that multiple people are listed, the photographer is prompted to link each person.

[0191] This allows images of people such as the photographer's own child wearing blue and white horizontal striped socks, a player wearing the YY uniform with the number XX, a player wearing a red and purple game suit, or a player wearing a pink flower accessory on their chest to be linked to images of outfits and accessories on the wrist, making it possible to detect the main subject efficiently and accurately.

[0192] Furthermore, if a race car, airplane, or the like is listed, an information image regarding the similarity to the shape of the specific vehicle that corresponds to that day, and an information image regarding the color and brightness of the subject area are automatically selected from the accompanying information obtained in step S603.

[0193] Furthermore, when shooting a ball game, the main subject to be detected must be manually set to be the ball, a person, or the person closest to the ball.

[0194] Specifically, the user is first prompted to select something other than a person, such as a ball, by touching the screen of LCD 7. If the process ends without anything being touched, a "person" is determined to be the main subject to be detected. If something other than a person is touched, it is then determined whether the process ends without a person being touched or whether a person is touched. In the former case, something other than a person, such as a ball, is determined to be the main subject to be detected, and in the latter case, the person closest to the other object is determined to be the main subject to be detected. If a touch operation is accepted here, it is preferable to notify the user which part of the screen was touched by adding a frame to the area or displaying it in inverse video.

[0195] In step S605, the second main subject detection operation is performed sequentially on the list-mounted images selected in step S604. When the list is mounted, these images may be read from storage memory 8 and recorded in EEPROM 9 via CPU 15, or a separate memory may be provided to ensure sufficient capacity. Also, if the memory capacity of the image capture device 1 itself is insufficient, the images may be read from storage memory 8 each time.

[0196] If the main subject detection process has been completed for all selected list-mounted images (YES in step S606), proceed to step S607. On the other hand, if the process has not been completed (NO in step S606), return to step S605 and perform a second main subject detection operation on the next selected list-mounted image.

[0197] In step S607, the result of the second main subject detection operation in step S605 is used to rank the feature amounts of the subject to be used during shooting.

[0198] This ranking is performed according to the number of subjects to be tracked detected in the images, the load of the detection process, the reliability of the detection, and the number of images in which the subjects to be tracked are detected.

[0199] For images in which multiple subjects are detected as tracking targets, some further processing is required to narrow it down to a single tracking target, and it is not appropriate to rank the features obtained from such images highly. Therefore, images in which multiple subjects are detected are ranked lowest (10th place).

[0200] The ranking is as shown in Table 3 below, taking into account the load of the detection process and the reliability of the detection.

[0201] [Table 3]

[0202] The reliability is calculated by calculating an index based on contrast and noise (converted from ISO sensitivity) when detecting each subject in an image, and if the index >= the specified value 1, it is considered high, if the specified value 1 > the index >= the specified value 2, it is considered medium, and if the index < the specified value 2, it is considered low.

[0203] Load level 1 is the lightest load, load level 2 is the next lightest load, and load level 3 is the heaviest load. Load level 1 corresponds to an image in which a face has been detected (step S203 in FIG. 5) or an image in which a specific pattern has been detected and has a high probability of being the main subject (step S209 in FIG. 5). Load level 2 corresponds to an image in which an eye has been detected after face detection (step S203 in FIG. 5), an image in which a person has been detected after face detection (step S205 in FIG. 5), and an image in which a color luminance similarity has been detected and has a high probability of being the subject (step S209 in FIG. 5). Load level 3 corresponds to an image in which face authentication has been performed after face detection (step S203 in FIG. 5).

[0204] The above load levels are for standard cases that assume that the same processing circuitry, such as CPU 15 and main subject detection circuit 19, is used. Therefore, if a dedicated processing circuit is provided that can exclusively perform detection, an image that would normally be classified as load level 2 or 3 may be classified as load level 1.

[0205] If the main subject is detected using features that have a heavy processing load, there is a possibility that the tracking will be adversely affected due to detection delays, etc. Also, if the main subject is detected using features with low reliability, there is a possibility that the main subject will be misdetected or its position will be inaccurate. The above ranking was determined taking these factors into consideration.

[0206] Furthermore, the order of the feature amounts is changed according to the number of images in the area where the main subject to be tracked is detected.

[0207] Specifically, if the number of images in the area where the main subject to be tracked is detected is greater than the number of images in the next higher ranking, the rankings are swapped. This process starts with processing the image with the largest number of images, ranked 1, and continues until processing of the lowest ranking is complete. For example, if the number of images in the area where the main subject is detected is three in the fourth ranking and one in the third ranking, the rankings of the third- and fourth-ranked feature amounts are swapped.

[0208] This is because it can be determined that a feature that can be detected as a main subject in various situations has a high probability of being detected and a low probability of being overlooked as a main subject.

[0209] In step S608, each image is displayed according to the ranking in step S607, for example by superimposing a frame on the position where the feature amount is detected.

[0210] In response to this display, the photographer can delete or add images to the list and change their order.

[0211] The method involves checking the images displayed in order of rank together with the detection position displayed in a frame. If the photographer specifies an image that he or she wants to use preferentially for main subject detection as a result of this checking, that image is ranked first.

[0212] Furthermore, if there is an image in which the main subject was mistakenly detected, the image can be deleted or the main subject can be corrected by moving the displayed frame on the screen of LCD 7, for example. This correction can be made by changing the main subject's face to the clothing or accessories worn that day, etc. In this case, the second main subject detection operation in step S605 must be performed again, which corresponds to the processing in step S201 in Figure 5, and processing is performed at the main subject detection position specified by the photographer, and the feature amounts of that subject are pre-registered.

[0213] Additionally, it is possible to add a list-mounted image using the method of step S601.

[0214] This may result in a low ranking, and may be the case when the image was not ranked at all or when the image was not originally included in the list of images. In these cases, the load level in Table 3 is set to load level 1 in order to improve the ranking in the next detection.

[0215] When the display of the image in which the ranked subject features have been detected has finished, the process proceeds to step S609, where the photographer's approval for pre-registration as a feature is confirmed. If approval is received (YES in step S609), the process proceeds to step S610, where the highest ranked subject feature is pre-registered, and then this process ends. On the other hand, if approval is not received (NO in step S609), the process proceeds to step S611.

[0216] In step S611, it is determined whether a list-mounted image has been added or the main subject detection position has been corrected. If not (NO in step S611), the process returns to step S607, where the order of the subject features is changed according to the photographer's input, and then the processes from step S608 onwards are carried out. On the other hand, if added (YES in step S611), the process returns to step S605, where a second main subject detection operation is carried out based on the specified main subject detection position, etc., and then the processes from step S606 onwards are carried out.

[0217] In this way, in this embodiment, feature amounts simplified to a level that allows tracking are pre-registered, and only features specific to the main subject are detected, thereby enabling high-speed tracking of the subject.

[0218] Here, the operation of the first main subject detection means, which detects the subject position using only the pre-registered subject feature amounts, performed in step S403 will be described.

[0219] Unlike the second main subject detection operation, the first main subject detection operation detects the main subject only using the highest-order subject feature pre-registered in step S610. Therefore, the first main subject detection operation performs only the processing related to the relevant subject feature among the detailed processing of the second main subject detection operation shown in FIG. 5, and does not determine whether or not it is the same subject. Therefore, even if main subject detection fails in the first main subject detection operation, main subject detection is not performed using other subject feature values, and the operation proceeds to main subject detection of the next frame image. In this case, the detected main subject position is used as the previous detection position, and a corrected main subject position is determined using the method described in steps S408 to S410 of FIG. 4.

[0220] For example, if the presence of a specific pattern has been pre-registered as the highest-ranking main subject feature, step S209 is executed to evaluate the detection of an area having the specific pattern. If the result of this evaluation is that the specific pattern has been detected (YES in step S209), step S210 is executed, followed by steps S408 to S410. On the other hand, if the main subject has not been detected (NO in step S209), the previous detection position is used as the main subject position, and steps S408 to S410 are executed.

[0221] If the frequency of main subject detection failure is high when using only the top-ranked subject feature, even when including determination of whether or not the subject is the same, the order may be changed and the first main subject detection operation may be performed using subject feature values ​​ranked second or lower.

[0222] Furthermore, if the order is reversed and main subject detection fails frequently using only the highest-ranking subject feature, even when including the determination of whether or not the subjects are the same, steps S404 to S406 may be performed regardless of the determination result of step S402 in FIG. 4.

[0223] Here, the second main subject detection operation with priority given to pre-registration, which is performed in step S404, will be described.

[0224] 5, processing related to the relevant subject feature amount is performed first, and if main subject detection is successful, a determination is made as to whether the detected main subject is the same subject as in the previous frame. If a main subject that is presumed to be the same subject is detected, the processing of steps S408 to S410 is performed using this detection position.

[0225] On the other hand, if main subject detection fails, main subject detection is performed using subject feature amounts ranked second or lower. As a result, of the subject feature amounts used for main subject detection, the main subject position based on the highest-ranked subject feature amount that successfully detected the main subject and estimated that the detected main subject is the same subject as in the previous frame is used to perform the processing of steps S408 to S410.

[0226] When detection for all ranked feature amounts has been completed, detection is also performed for subject feature amounts that have not been ranked according to the operating procedure in Fig. 5. Then, the detected main subject position is used to perform the processing in steps S408 to S410.

[0227] For example, if the presence of a specific pattern is pre-registered as the highest-ranking subject feature, the detection of an area having the specific pattern in step S209 is evaluated, including a determination of whether or not it is the same subject (step S224).

[0228] If the result of this evaluation is that the subject is detected (YES in step S209, YES in step S224), the process of step S210 is performed to determine the main subject detection area. Otherwise, the second ranked subject feature (here, the main subject detected by face detection (step S203)) is evaluated, including determining whether it is the same subject (step S221).

[0229] If the result of this evaluation is that the subject is detected (YES in step S203, YES in step S221), the process of step S204 is performed to determine the main subject detection area. Otherwise, the subject feature value ranked third (here, person detection (step S205)) is evaluated, including determining whether it is the same subject (step S222).

[0230] If the result of this evaluation is that the main subject is detected (YES in step S205, YES in step S222), the process of step S206 is performed to determine the main subject detection region. When the evaluation of detection for all ranked subject feature amounts has been completed, the evaluation of detection for subject feature amounts that have not been ranked is then performed.

[0231] Here, the registration of subject feature amounts in step S406 will be described.

[0232] If the main subject cannot be detected in the first main subject detection operation, the main subject detection results for the subject features ranked second and below that were successful, including determination of whether they are the same subject, are replaced in order of rank. For example, if the subject features ranked first to fourth have been pre-registered, and only the second-ranked subject is successfully detected, the subject feature ranked second up to the previous time becomes the first-ranked subject feature, and the first-ranked subject feature becomes the second-ranked subject feature. Furthermore, if detection was successful for the second and fourth-ranked subjects, the subject feature ranked second up to the previous time becomes the first-ranked subject feature, and the fourth-ranked subject feature becomes the second-ranked subject feature. Furthermore, the first-ranked subject feature becomes the third-ranked subject feature, and the third-ranked subject feature becomes the fourth-ranked subject feature.

[0233] If the difference between the subject positions detected in the first and second main subject detection operations is greater than a predetermined value, first, main subject detection is performed using feature amounts ranked second or lower. Next, of the obtained detection results, the feature amount with the highest ranking that is the same subject as the subject in the previous frame and whose difference in subject position is less than a predetermined value is set as the first-ranked feature amount, and the feature amount that was ranked first the previous time is set as the second-ranked feature amount.

[0234] Here, the AF processing performed in step S307 and the calculation of the speed of change in the direction of light rays incident on the photographing optical system performed in step S308 will be described using steps S701 and S702 in Fig. 7. Furthermore, the correction of the control position of the focus lens performed in step S309 will be described using steps S703 to S706 in Fig. 7 and Fig. 8. Furthermore, the change in exposure control performed in step S311 and the determination of whether or not an instruction for correction by image processing is required will be described using steps S710 and S711 in Fig. 7.

[0235] In step S307, AF processing is performed on the area on the sensor 3 where the main subject was detected in step S304.

[0236] The CPU 15 detects the defocus amount of the area where the main subject is present using the AF processing circuit 12, calculates the drive amount and drive direction of the focus lens 33 from the detected defocus amount, and transmits the calculation result information to the interchangeable lens 31. Then, the interchangeable lens 31 performs drive processing of the focus lens 33 in accordance with the calculation result information transmitted by the CPU 15, and obtains a focused state.

[0237] Next, in step S701, drive information for the camera rotation drive device 54 and the mirror 57 that bends the imaging light beam is obtained from the high-speed light beam direction change device 50.

[0238] In step S702, the rotation speed of the imaging device 1 is calculated from the drive information of the camera rotation drive device 54, and the error in the control position of the focus lens 33 caused by the rotation of the imaging device 1 is estimated based on the calculated speed. Here, there is play in the drive mechanisms inside the interchangeable lens 31, not just the focus lens 33. However, even with this play, gravity and friction typically control the lens 31 to within a negligible error range from the ideal control position. However, if the interchangeable lens 31 is significantly shaken by the high-speed rotation of the imaging device 1, gravity-induced drop becomes dominant, causing the lens to shift to the lower end of the play range, resulting in a significant deviation from the ideal control position. Therefore, in this embodiment, when vibrations that cause gravity-induced drop to be dominant occur, the error in the control position of the focus lens 33 caused by the vibrations is estimated (first calculation means).

[0239] Furthermore, the angle of change per unit time of the mirror 57 that bends the light beam is calculated from the driving information of the mirror 57.

[0240] When the angle of change per unit time of mirror 57 is large and the image height is high such that a ray of light corresponding to the position of the main subject passes through the tip of mirror 57, an error in the control position of mirror 57 is likely to occur due to a delay in driving mirror 57 and shaking (vibration) of the tip, etc. Therefore, in this embodiment, the error in the control position of mirror 57 is estimated based on the angle of change per unit time of mirror 57 and the image height (second calculation means).

[0241] The general relationship between image height (unit: 1 / 2) and image surface (unit: μm) is shown in Figure 8. In Figure 8, the horizontal axis represents image height and the vertical axis represents image surface.

[0242] It is desirable for the image plane to be constant regardless of image height, as shown by the dotted line in FIG. 8 . However, in reality, errors occur in the image plane as the image height increases due to surface tilt caused by the optical axis of the interchangeable lens 31 not being perpendicular to the sensor 3 and field curvature caused by aberrations in the focus lens 33. For example, as shown in the actual image plane in FIG. 8 , the image plane varies by several tens of micrometers between when the image height is 60% and when the image height is 70%, which is only 10% higher. Here, while the magnitude of field curvature can be estimated from the design data and measurement data of the interchangeable lens 31 during manufacturing, estimating the magnitude of surface tilt is difficult because the surface tilt varies depending on how the interchangeable lens 31 is attached to the imaging device 1. Therefore, in this embodiment, the maximum value of the manufacturing tolerance of the interchangeable lens 31 is used as the magnitude of surface tilt.

[0243] Specifically, the amount of change in the light beam per unit time is calculated from the angle of change per unit time of mirror 57, and the error in the mirror control position is estimated. At the same time, the error in the control position of the driving mechanisms inside interchangeable lens 31, such as focus lens 33, is estimated (presumed) from the rotation per unit time of imaging device 1 by camera rotation drive device 54. Note that it is also possible to estimate only one of these errors based on the angle of change per unit time of mirror 57 or the magnitude of rotation per unit time of imaging device 1.

[0244] In step S703, evaluation is performed at the image height where AF processing is performed.

[0245] Specifically, first, the RMS value of the error in the control position of mirror 57 estimated in step S702 and the error in the control position of the focus lens due to the rotation of camera rotation drive device 54 is calculated. Next, it is determined whether this value is within the range of the difference from the position in real space corrected in step S409. If the result of this determination is that it is within the range of the difference, the difference from the position in real space is used for correction. On the other hand, if it is outside the range, the difference from the position in real space is used for correction, but since there is a possibility that further error has occurred, that error is absorbed by exposure control.

[0246] Correction using the difference from the position in real space is performed as follows.

[0247] The image plane at the image height (hereinafter referred to as AF image height) of the area on the image sensor (sensor 3) where the main subject was detected by the main subject detection process in step S304 and the image height (hereinafter referred to as corrected image height) of the area corrected from the position in real space is obtained based on the graph of estimated real image plane shown in Fig. 8. The image planes at the AF image height and the corrected image height are obtained, and the difference between them is calculated as the focus lens control position correction amount (hereinafter referred to as focus lens control position correction amount 1) due to image plane change.

[0248] When a lens attached to the imaging device 1 is used instead of the interchangeable lens 31, the magnitude of the final surface tilt obtained during surface tilt adjustment in the manufacturing process is recorded, and this information can be used to estimate the data of the real image plane. However, when an interchangeable lens 31 is attached to the imaging device 1, as in this embodiment, the surface tilt varies depending on how the interchangeable lens 31 is attached to the imaging device 1. Therefore, the average value of the difference in the image plane between the AF image height and the corrected image height in the first quadrant of FIG. 8 (the side with the larger amount of image plane change) and the difference in the image plane between the AF image height and the corrected image height in the second quadrant of FIG. 8 (the side with the smaller amount of image plane change) is set to the focus lens control position correction amount 1. In this case, there is a possibility of insufficient correction, and this amount is absorbed by exposure control.

[0249] Next, in step S704, communication is performed with the interchangeable lens 31 to obtain information for correcting the control position of the focus lens 33. This information includes the position of the focus lens 33 before and after drive processing on the interchangeable lens 31 side, and this value is used to determine whether the direction of drive control is upward (extension direction) or downward (retraction direction). This information also includes a focus lens control position correction amount 2.

[0250] The focus lens control position correction amount 2 is a value that is also used in normal AF processing, and is a value that is set in advance for the focus lens 33. Therefore, it can be obtained by communication with the interchangeable lens 31.

[0251] In step S705, it is determined whether focus lens control position correction is necessary. If it is determined that it is necessary (YES in step S705), the process proceeds to step S706, and if it is determined that it is not necessary (NO in step S705), the process proceeds to step S310.

[0252] Here, it is determined that it is unnecessary in the following two cases.

[0253] The first case is when the focus lens control position correction amount 1 is equal to or less than a predetermined value A1 (for example, a control drive amount equivalent to one-tenth of the depth of field) and the drive control direction is upward.

[0254] The second is when the focus lens control position correction amount 1 is less than a predetermined value A1 (for example, a control drive amount equivalent to 1 / 10 of the depth of field) and the drive speed of the camera rotation drive device 54 after the AF processing in step S307 is always less than a predetermined value (for example, 20° / sec).

[0255] Everything else is deemed necessary.

[0256] In step S706, focus lens position correction control is performed to accurately focus on the main subject.

[0257] Specifically, if the correction direction of the focus lens position is upward, that is, if the focus lens control position correction amount 1 is set in step S703 so as to move the focus lens 33 in the extension direction (upward), drive control is performed according to that setting. In other words, if the correction direction is upward, drive amount information consisting of the drive amount and drive direction is transmitted to the interchangeable lens 31 so as to drive and control the lens by the correction amount.

[0258] On the other hand, if the correction direction of the focus lens position is downward, that is, if focus lens control position correction amount 1 is set in step S703 so as to move the focus lens 33 in the retraction direction (downward), drive control is not performed with that setting. Specifically, drive amount information is transmitted to the interchangeable lens 31 so as to drive and control the focus lens 33 downward by focus lens control position correction amount 1+focus lens control position correction amount 2, and then drive and control the focus lens 33 upward by focus lens control position correction amount 2.

[0259] There may be cases where the direction of drive processing of the focus lens 33 performed on the interchangeable lens 31 side is downward, but the focus lens control position correction amount 1 = 0. In this case as well, in order to perform correction control drive in the upward direction with the focus lens control position correction amount 2, the drive amount (focus lens control position correction amount 2) and drive direction (upward) are transmitted to the interchangeable lens 31.

[0260] In this way, whether the focus lens position correction direction is upward or downward, the focus lens 33 is driven upward (opposite to the direction of gravity (specific direction)) and then the control position correction is terminated. This allows stable position control of the focus lens 33, which is pressed downward (in the direction of gravity).

[0261] Here, it is determined whether or not control position correction is necessary (step S705), and if necessary, focus lens control position correction is performed (step S706) in order to reduce release time lag.

[0262] Then, in step S310, it is determined whether or not a release command has been issued. If a command has been issued (YES in step S310), the process proceeds to step S707; if no command has been issued (NO in step S310), the process returns to step S305.

[0263] In steps S707 and S708, even if the focus lens control position has been corrected before the release command in step S706, the control position may need to be corrected again after the release command, so the same processing as in steps S702 and S703 is performed.

[0264] Then, in step S709, it is determined whether focus lens control position correction is necessary, as in step S705. If it is determined that it is necessary (YES in step S709), the process proceeds to step S710, where focus lens position correction control is performed to accurately focus on the main subject, as in step S706, and then the process proceeds to step S711. On the other hand, if it is determined that it is not necessary (NO in step S709), the process proceeds directly to step S711.

[0265] In step S711, it is determined whether or not to change the exposure control (aperture ISO sensitivity).

[0266] In this determination, first, the determination image plane value Zd is set according to the focus lens control position correction amount 1 calculated in step S703. Here, if a lens attached to the image pickup device 1 is used instead of the interchangeable lens 31, the focus lens control position correction amount 1 can be used as the determination image plane value Zd as is. However, when the interchangeable lens 31 is attached to the image pickup device 1 as in this embodiment, the influence of surface tilt must be taken into consideration, so the sum of half the difference between the image plane difference in the first quadrant and the image plane difference in the second quadrant in FIG. 8 is used as the determination image plane value Zd. In other words, the value of the determination image plane value Zd is set as follows:

[0267] 8 is Z1, and the image plane difference in the second quadrant is Z2. When a lens attached to the imaging device 1 is used instead of the interchangeable lens 31, the determined image plane value Zd is the focus lens control position correction amount 1, that is, Z1 or Z2. Here, whether Z1 or Z2 is used is selected based on the results of the adjustment of the surface tilt.

[0268] On the other hand, when the interchangeable lens 31 is attached to the imaging device 1, the judged image plane value Zd is calculated as follows, assuming Z1>=Z2. Judgment image plane value Zd = Focus lens control position correction amount 1 + (Z1 - Z2) / 2 =(Z1+Z2) / 2+(Z1-Z2) / 2 =Z1

[0269] Next, it is determined whether the determination image plane value Zd exceeds Fnδ / 2 (where δ is the allowable circle of confusion diameter, and a value such as 20 μm is generally used), which is half the depth of field calculated from the aperture value Fn set at that time. If the determination image plane value Zd exceeds the value of Fnδ / 2, it is determined that the exposure control (aperture ISO sensitivity) should be changed (YES in step S711), and the process proceeds to step S712. On the other hand, if the determination image plane value Zd is equal to or less than the value of Fnδ / 2, it is determined that the exposure control (aperture ISO sensitivity) should be maintained (NO in step S711), and the process proceeds to step S713.

[0270] In step S712, an instruction is given to the interchangeable lens 31 to change the aperture value so that the determined image plane value Zd+Fnδ / 2 falls within the depth. Specifically, an instruction is given to the interchangeable lens 31 so that the aperture value Fa after the change becomes (Zd+Fnδ / 2)÷δ.

[0271] If the aperture value is controlled in fixed increments (for example, in 1 / 3 step increments), the smallest aperture value Fb exceeding Fa may be instructed to the interchangeable lens 31 as the aperture value to be changed.

[0272] This reduces the effects of surface tilt and field curvature, making it possible to accurately focus on the main subject, even if the image height during AF processing changes.

[0273] In step S713, it is determined whether correction by image processing is required.

[0274] Specifically, the difference between the pre-change aperture value Fn used in the determination in step S711 and the post-change aperture value Fa used in the instruction in step S712 (hereinafter referred to as the aperture value difference between the front and rear) is calculated. Next, if the calculated aperture value difference between the front and rear satisfies any of the conditions shown in Table 4 below, it is determined that correction by image processing (background blurring processing) is necessary.

[0275] [Table 4]

[0276] If the difference between the front and rear aperture values ​​satisfies any of the conditions shown in Table 4, it is determined that correction by image processing (background blurring processing) is necessary (YES in step S713), and the process proceeds to step S714. On the other hand, if the difference between the front and rear aperture values ​​does not satisfy any of the conditions shown in Table 4, it is determined that correction by image processing is not necessary (NO in step S713), and the process ends.

[0277] The values ​​shown in Table 4 are provided for illustrative purposes only and are not limiting. The difference in aperture value before and after the change that requires correction by image processing can be determined according to the focal length and the aperture value before the change.

[0278] Here, even if it is determined in step S711 that exposure control should be maintained, the aperture value has not been changed and the difference between the aperture values ​​before and after is 0, so it is determined that no correction by image processing is necessary (NO in step S713), and this processing is terminated.

[0279] This allows the photographer to take a picture with the depth of field they desire.

[0280] As explained above, in this embodiment, high-speed light direction change device 50 (light direction change means) functions as a subject blur correction means that corrects subject blur of the main subject during tracking photography. Furthermore, CPU 15, TG 16, communication driver 21, and aperture 34 of interchangeable lens 31 and its drive control circuit function as an exposure control change means that changes exposure control. Furthermore, main subject detection circuit 19 functions as a main subject detection means that receives outputs from motion vector detection circuit 18, A / D conversion circuit 5, and CPU 15 and detects the position of the main subject to be photographed within the frame.

[0281] The CPU 15 also serves as a subject position notification means that notifies the high-speed light direction change device 50 of the subject position, a control direction detection means that detects the control direction of the focus lens 33 by the interchangeable lens 31 (focus adjustment means), and an incident light change detection means. Furthermore, the focus lens drive control circuit consisting of the AF processing circuit 12, CPU 15, communication driver 21, and interchangeable lens 31 serves as a focus adjustment means that drives and controls the focus lens to the in-focus position.

[0282] As explained above, in the present invention, when detecting a main subject, accurate and high-speed main subject detection is made possible by selectively using a first main subject detection means that uses pre-registered subject feature amounts and a second main subject detection means that uses multiple subject feature amounts.

[0283] At the same time, the system corrects for differences in the drive direction of the focus lens 33 to its control position, changes in the angle of the mirror 57 due to vibration, and changes in the control position due to the image height of the subject, and changes in exposure control taking into account correction errors, making it possible to accurately focus on the main subject.

[0284] In the present invention, the lens-interchangeable imaging device 1 has been described as an example, but the present invention may also be applied to a lens-attached imaging device, or to a digital video camera or the like.

[0285] Furthermore, with regard to the roles assigned to the CPU 15 in the present invention, a dedicated circuit for performing these roles may be provided.

[0286] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the present embodiment to a system or device via a network or a recording medium, and having one or more processors in the computer of the system or device read and run the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0287] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention.

[0288] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0289] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) An imaging device that keeps capturing a main subject at a fixed position on an imaging element by changing the ray direction of an imaging ray at high speed using a ray direction change device, the imaging device comprising: a focus adjustment means that drives and controls the position of a focus lens to perform focus adjustment; a control direction detection means that detects the control direction of the focus lens by the focus adjustment means; and an incident ray change detection means that detects the rate of change when the ray direction of the imaging ray is changed by the ray direction change device, characterized in that the focus adjustment means corrects the position of the focus lens using the control direction detected by the control direction detection means and the rate of change detected by the incident ray change detection means. (Configuration 2) The imaging device according to Configuration 1, wherein the light direction changing device has a light bending means for bending the imaging light beam with a mirror and an imaging device rotating means for rotatably mounting the imaging device. (Configuration 3) The imaging device according to Configuration 2 further comprises a first calculation means for calculating an error in the position of the focus lens, which is driven and controlled by the focus adjustment means, caused by rotation of the imaging device by the imaging device rotation means, a second calculation means for calculating an error in the control position of the mirror according to the angle of change of the mirror, and a main subject detection means for detecting the position on the imaging element of a main subject to be photographed, and characterized in that exposure control is changed according to the image height of the position of the main subject detected by the main subject detection means and the errors calculated by the first and second calculation means. (Configuration 4) An imaging device according to configuration 3, characterized in that if the focal length, the aperture value before the change in the exposure control, and the aperture step before and after the change in the exposure control satisfy predetermined conditions, correction is further performed by image processing. (Configuration 5) An imaging device described in any one of configurations 2 to 4, further comprising a communication means for communicating information with the light beam direction change device, wherein the communication means acquires information from the light beam direction change device for correcting the position of the focus lens, which is driven and controlled by the focus adjustment means. (Configuration 6) The imaging device described in Configuration 5, characterized in that the communication means receives from the light ray direction changing device at least one of information regarding the structure of the light ray direction changing device, the maximum speed at which the light ray direction of the imaging light ray is changed by the light ray direction changing device, the speed at the time of communication, and the change in the light ray direction after communication. (Configuration 7) The imaging device described in Configuration 5 or 6, characterized in that the communication means transmits to the light direction change device at least one of information regarding the drive control of the focus adjustment means, main subject detection information in the imaging device, exposure control information in the imaging device, focal length of the focus lens, and shooting timing. (Configuration 8) The imaging device according to configuration 7, wherein the information relating to the drive control of the focus adjustment means includes at least one of a focus distance, a focus sensitivity, and a drive direction and speed of the focus lens. (Configuration 9) The imaging device described in Configuration 6, characterized in that the information regarding the structure of the light ray direction changing device includes at least one of the overall size of the mirror, the size of the drive shaft of the mirror, and the size of the part that bends the imaging ray incident on the mirror. (Configuration 10) The imaging device according to any one of configurations 5 to 9, wherein the information communication with the light direction change device by the communication means is performed immediately before photographing. (Configuration 11) The imaging device according to any one of configurations 2 to 10, wherein the focus adjustment means drives the focus lens in the direction opposite to the specific direction, and then ends the position correction. (Configuration 12) The imaging device according to configuration 11, wherein the specific direction is the direction of gravity, and the imaging device rotation means places the imaging device so that the imaging direction faces upward. (Configuration 13) The imaging device according to configuration 11 or 12, characterized in that the necessity of correcting the position of the focus lens is confirmed at least immediately before shooting. (Configuration 14) The imaging device described in any one of configurations 11 to 13, characterized in that the focus adjustment means drives the focus lens in a direction opposite to the specific direction in accordance with the drive speed of the imaging device rotation means immediately before shooting. (Configuration 15) The imaging device according to configuration 14, wherein the specific direction is the direction of gravity, and the imaging device rotation means places the imaging device so that the imaging direction faces upward. (Configuration 16) The imaging device according to any one of configurations 1 to 15, characterized in that the imaging device comprises an interchangeable lens including the focus lens and a main body including the imaging element. (Configuration 17) A blur correction device that uses a light direction changing means to quickly change the ray direction of an imaging light ray in order to keep capturing a main subject at a fixed position on an imaging element of an imaging device, and that corrects subject blur caused by the change in the ray direction of the imaging light ray, comprising: a main subject position acquisition means for acquiring the position of the main subject from the imaging device; and a control means for controlling the light direction changing means in accordance with the acquired position of the main subject, wherein the imaging device comprises: a main subject position detection means for detecting the position of the main subject; a subject feature pre-registration means for pre-registering feature amounts to be used when detecting the main subject from among a plurality of feature amounts that characterize the main subject; a first main subject detection means for detecting the main subject from the detected subjects using only the feature amounts registered by the subject feature pre-registration means; and a second main subject detection means for detecting the main subject from the detected subjects using the plurality of feature amounts, wherein the position of the main subject is detected by switching between the first main subject detection means and the second main subject detection means in accordance with the movement of the detected subject. (Configuration 18) The image stabilization device according to Configuration 17, characterized in that the multiple features are any of facial recognition information indicating the shape and position of each individual's facial organs, information about the shape and pattern of clothing worn by a person and parts thereof, information about the shape and pattern of accessories, information about similarity to a person or a specific animal, information about similarity to the shape of a specific vehicle, information about the color and brightness of the subject area, and movement of the subject in the horizontal, vertical and optical axis directions. (Configuration 19) A shake correction device according to Configuration 17 or 18, characterized in that the registration in the subject feature pre-registration means is performed by detecting the subject to be photographed in advance using the second main subject detection means, and from the result, the subject features to be used in detecting the subject to be photographed in photography are registered in order of priority. (Configuration 20) The image stabilization device described in Configuration 19, characterized in that the detection by the second main subject detection means of the subject to be photographed in the pre-photographing is based on any one of a past photographed image taken by the imaging device, an image recorded on a recording medium attached to the imaging device and retouched from the past photographed image, a released video, public information about an event by the organizer, and accompanying information about each past subject. (Configuration 21) A shake correction device according to any one of configurations 17 to 20, characterized in that a determination threshold used to determine whether to use the first main subject detection means or the second main subject detection means is changed depending on any one of a continuous shooting speed, a sensor frame rate, a readout pixel count, and a readout range. (Configuration 22) The image stabilization device of any one of configurations 17 to 22, characterized in that the light ray direction changing means has a light ray bending means for bending the imaging light ray with a mirror and an imaging device rotation means for rotatably mounting the imaging device, and converts the detected position of the main subject on the screen into real space after estimating the position of the main subject detected by the first main subject detection means or the second main subject detection means based on any one of the image height of the main subject, vibration of the mirror, and error in the control position of the mirror due to a drive delay of the mirror, and error in the control position of the focus lens caused by rotation of the imaging device by the imaging device rotation means, thereby correcting the position of the subject. (Method 1) A control method for an imaging device that keeps capturing a main subject at a fixed position on an imaging element by rapidly changing the ray direction of an imaging ray using a ray direction change device, the control method comprising: a focus adjustment step that drives and controls the position of a focus lens to perform focus adjustment; a control direction detection step that detects the control direction of the focus lens in the focus adjustment step; and an incident ray change detection step that detects the rate of change when the ray direction of the imaging ray is changed by the ray direction change device, characterized in that the position of the focus lens is corrected in the focus adjustment step using the control direction detected in the control direction detection step and the rate of change detected in the incident ray change detection step. (Method 2) A control method for a motion compensation device that uses a light direction changing means to quickly change the ray direction of an imaging light beam in order to keep capturing a main subject at a fixed position on an imaging element of an imaging device, and that compensates for subject motion caused by the change in the ray direction of the imaging light beam, the control method comprising: a main subject position acquisition step of acquiring the position of the main subject from the imaging device; and a control step of controlling the light direction changing means in accordance with the acquired position of the main subject, wherein the control method further comprises: a main subject position detection step of detecting the position of the main subject, a subject feature pre-registration step of pre-registering feature amounts to be used when detecting the main subject, out of a plurality of feature amounts that characterize the main subject, a first main subject detection step of detecting the main subject from the detected subjects using only the feature amounts registered in the subject feature pre-registration step, and a second main subject detection step of detecting the main subject from the detected subjects using the plurality of feature amounts, wherein the position of the main subject is detected by switching between the first main subject detection step and the second main subject detection step in accordance with the movement of the detected subject. (Program 1) A program for causing a computer to function as each of the means of the imaging device described in any one of configurations 1 to 16. (Program 2) A program for causing a computer to function as each of the means of the image stabilization device and the imaging device described in any one of configurations 17 to 22. [Explanation of symbols]

[0290] 1. Imaging device 2 Lens mount 3 sensors 4. Imaging circuit 7 LCD 11 AE processing circuit 12 AF processing circuit 13 Shake detection circuit 14 Shake detection sensor 15,52 CPU 18 Motion vector detection circuit 19 Main subject detection circuit 21,35,51 Communication Driver 31 Interchangeable Lenses 50 High-speed light direction change device 54 Camera rotation drive device 56 Mirror drive motor 57 Mirror 100 Tracking and Shooting System

Claims

1. An imaging device that continues to capture a main subject at a fixed position on an imaging element by changing the light direction of an imaging light beam at high speed using a light direction changing device, a focus adjustment unit that drives and controls the position of the focus lens to perform focus adjustment; a control direction detection means for detecting a control direction of the focus lens by the focus adjustment means; an incident light beam change detection means for detecting a speed of change of the light beam direction of the imaging light beam when the light beam direction change device changes the light beam direction; and an imaging device, characterized in that the position of the focus lens is corrected by the focus adjustment means using the control direction detected by the control direction detection means and the change rate detected by the incident light change detection means.

2. 2. The imaging device according to claim 1, wherein the light direction changing device comprises a light bending means for bending the imaging light beam by a mirror, and an imaging device rotating means for rotatably mounting the imaging device.

3. a first calculation means for calculating an error in the position of the focus lens, which is driven and controlled by the focus adjustment means, caused by rotation of the imaging device by the imaging device rotation means; a second calculation means for calculating an error in a control position of the mirror in accordance with the change angle of the mirror; a main subject detection means for detecting the position of a main subject to be photographed on the imaging element; Further provided with 3. The imaging apparatus according to claim 2, wherein exposure control is changed in accordance with the image height of the position of the main subject detected by said main subject detection means and the errors calculated by said first and second calculation means.

4. 4. The imaging device according to claim 3, further comprising image processing to correct the focal length, the aperture value before the change in the exposure control, and the aperture step before and after the change in the exposure control, if the focal length, the aperture value before the change in the exposure control, and the aperture step before and after the change in the exposure control satisfy predetermined conditions.

5. Further, a communication means for communicating with the light direction changing device is provided.

3. The imaging device according to claim 2, wherein information for correcting the position of the focus lens, which is driven and controlled by the focus adjustment means, is obtained from the light direction change device by the communication means.

6. The communication means transmits from the light beam direction changing device, 6. The imaging device of claim 5, further comprising: information about the structure of the light direction changing device; a maximum speed at which the light direction of the imaging light beam is changed by the light direction changing device; a speed at the time of communication; and a change in the light direction after communication.

7. The communication means communicates with the light direction changing device, information regarding the drive control of the focus adjustment means; main subject detection information in the imaging device; exposure control information for the imaging device; the focal length of the focus lens, and Shooting timing 6. The imaging device according to claim 5, wherein at least one of the following is transmitted.

8. 8. The imaging apparatus according to claim 7, wherein the information relating to the drive control of the focus adjustment means includes at least one of a focus distance, focus sensitivity, and a drive direction and speed of the focus lens.

9. 7. The imaging device according to claim 6, wherein the information regarding the structure of the light beam direction changing device includes at least one of the overall size of the mirror, the size of the drive shaft of the mirror, and the size of the portion that bends the imaging beam incident on the mirror.

10. 6. The imaging device according to claim 5, wherein the information communication with the light direction changing device by the communication means is performed immediately before photographing.

11. 3. The imaging device according to claim 2, wherein the focus adjustment means completes the position correction after driving the focus lens in a direction opposite to the specific direction.

12. The specific direction is the direction of gravity, 12. The imaging device according to claim 11, wherein the imaging device rotating means places the imaging device so that the imaging direction faces upward.

13. 12. The imaging apparatus according to claim 11, wherein the necessity of correcting the position of the focus lens is confirmed at least immediately before shooting.

14. 12. The imaging device according to claim 11, wherein the focus adjustment means drives the focus lens in a direction opposite to the specific direction in accordance with a drive speed of the imaging device rotation means immediately before shooting.

15. The specific direction is the direction of gravity, 15. The imaging device according to claim 14, wherein the imaging device rotating means places the imaging device so that the imaging direction faces upward.

16. 2. The imaging device according to claim 1, wherein the imaging device comprises an interchangeable lens including the focus lens and a main body including the imaging element.

17. A motion compensation device that uses a light direction changing means to quickly change the light direction of an imaging light ray so as to keep capturing a main subject at a fixed position on an imaging element of an imaging device, and compensates for subject motion caused by the change in the light direction of the imaging light ray, a main subject position acquisition means for acquiring a position of a main subject from the imaging device; a control means for controlling the light direction changing means in accordance with the acquired position of the main subject; and The imaging device is a main subject position detection means for detecting the position of a main subject; a subject feature pre-registration means for pre-registering a feature to be used when detecting the main subject, among a plurality of feature values ​​that characterize the main subject; a first main subject detection means for detecting the main subject from the detected subjects using only the feature amounts registered by the subject feature amount pre-registration means; a second main subject detection means for detecting the main subject from the detected subjects using the plurality of feature amounts; and a first main subject detecting means for detecting a position of the main subject and a second main subject detecting means for detecting a position of the main subject;

18. 18. The image stabilization device according to claim 17, wherein the plurality of feature amounts are any of facial recognition information indicating the shapes and positions of individual facial organs, information about the shape and pattern of clothing worn by a person and parts thereof, information about the shape and pattern of accessories, information about similarity to a person or a specific animal, information about similarity to the shape of a specific vehicle, information about the color and brightness of the subject area, and movement of the subject in the horizontal, vertical, and optical axis directions.

19. 18. The image stabilization device according to claim 17, wherein the registration in the subject feature pre-registration means is performed by detecting a subject to be photographed in advance by the second main subject detection means, and then registering the subject feature values ​​to be used for detecting the subject to be photographed in accordance with the detection result, with priority assigned to the subject feature values.

20. 20. The image stabilization device according to claim 19, wherein the detection by the second main subject detection means of the subject to be photographed in the pre-photographing is based on any one of a past photographed image photographed by the imaging device, an image obtained by retouching the past photographed image using an image recorded on a recording medium attached to the imaging device, a released video, public information about an event made public by a organizer, and accompanying information about an individual past subject.

21. 18. The image stabilization device according to claim 17, wherein a determination threshold used to determine whether to use the first main subject detection means or the second main subject detection means is changed in accordance with any one of a continuous shooting speed, a sensor frame rate, a readout pixel count, and a readout range.

22. the light beam direction changing means includes a light beam bending means for bending the imaging light beam by a mirror and an imaging device rotating means for rotatably mounting the imaging device, 18. The image blur correction device according to claim 17, wherein the position of the main subject detected by the first main subject detection means or the second main subject detection means is converted from the detected position of the main subject on a screen into real space and the position of the subject is corrected after estimating any one of the following: the image height of the main subject, vibration of the mirror, an error in the control position of the mirror due to a drive delay of the mirror, and an error in the control position of the focus lens caused by rotation of the image capture device by the image capture device rotation means.

23. A control method for an imaging device that continues to capture a main subject at a fixed position on an imaging element by changing the light direction of an imaging light beam at high speed using a light direction changing device, comprising: a focus adjustment step of driving and controlling the position of the focus lens to perform focus adjustment; a control direction detection step of detecting a control direction of the focus lens in the focus adjustment step; an incident light ray change detection step of detecting a change rate of the light ray direction of the imaging light ray when the light ray direction change device changes the light ray direction of the imaging light ray; and a control method for correcting the position of the focus lens in the focus adjustment step using the control direction detected in the control direction detection step and the rate of change detected in the incident light change detection step.

24. A control method for a motion compensation device, which uses a light direction changing means to change the light direction of an imaging light ray at high speed in order to keep capturing a main subject at a fixed position on an imaging element of an imaging device, and performs motion compensation for a subject caused by the change in the light direction of the imaging light ray, a main subject position acquisition step of acquiring a position of a main subject from the imaging device; a control step of controlling the light direction changing means in accordance with the acquired position of the main subject; and In the imaging device, a main subject position detection step for detecting the position of a main subject; a subject feature pre-registration step of pre-registering feature amounts to be used when detecting the main subject, among a plurality of feature amounts that characterize the main subject; a first main subject detection step of detecting the main subject from the detected subjects using only the feature amounts registered in the subject feature amount pre-registration step; a second main subject detection step of detecting the main subject from the detected subjects using the plurality of feature amounts; and a control method for detecting the position of the main subject by switching between the first main subject detection step and the second main subject detection step in accordance with the movement of the detected subject;

25. A program for causing a computer to function as each of the means of the imaging device according to claim 1.

26. A program for causing a computer to function as each of the means of the image stabilization device and the imaging device according to claim 17.

Images