Imaging element, camera device, working method of imaging element and storage medium

Abstract

The imaging element includes a memory and a processor. The memory stores image data obtained by shooting at a first frame rate. The processor outputs the image data at a second frame rate. The processor uses the image data to derive jitter degree information indicating the degree of jitter contained in an image represented by the image data, and outputs the derived jitter degree information at a rate higher than the second frame rate. The first frame rate is a frame rate higher than the second frame rate.

Description

[0001] This application is a divisional application of the Chinese patent application with the application date of April 17, 2020, application number 202080031008.8, and invention name “Imaging element, camera device, working method and procedure of imaging element”. Technical Field

[0002] The technology of the present invention relates to an imaging element, a camera device, a working method of the imaging element, and a program. Background Art

[0003] Japanese Patent Publication No. 2017-188760 discloses a stacked imaging element formed by stacking a memory substrate and a signal processing substrate. The memory substrate has a memory such as a DRAM (Dynamic Random Access Memory) that temporarily stores pixel signals output by the pixel substrate. The signal processing substrate performs various signal processing with respect to the pixel signals stored in the memory substrate. In the stacked imaging element described in Japanese Patent Publication No. 2017-188760, when a moving object is detected in a pixel signal by an analysis unit included in the signal processing substrate, signal processing is performed by a signal processing unit included in the signal processing substrate.

[0004] Japanese Patent Application Laid-Open No. 2015-222925 discloses a shake correction device, which includes a shake detection mechanism, a motion vector detection mechanism, a first shake correction mechanism, a second shake correction mechanism, a first shake correction control mechanism, and a second shake correction control mechanism.

[0005] In the shake correction device described in Japanese Patent Application Laid-Open No. 2015-222925, a shake detection mechanism detects shake based on angular velocity information and generates a shake signal. A motion vector detection mechanism detects a motion vector based on a camera signal generated in the shooting of a subject in a first shooting mode based on an imaging element. The first shake correction mechanism corrects the shake of a subject image optically. The second shake correction mechanism corrects the shake of a subject image electronically. The first shake correction control mechanism determines a first correction amount for correcting the shake of the subject image based on the shake signal, and controls the first shake correction mechanism according to the first correction amount to correct the shake of the subject image. The second shake correction control mechanism determines a second correction amount for correcting the shake of the subject image based on the motion vector, and controls the second shake correction mechanism according to the second correction amount to correct the shake of the subject image.

[0006] Furthermore, in the shake correction device described in Japanese Patent Gazette No. 2015-222925, in the first shooting mode, the first shake correction control mechanism and the second shake correction control mechanism correct the shake of the subject image, and the first shake correction control mechanism corrects the shake of the subject image according to the first correction amount and the second correction amount. Summary of the invention

[0007] One embodiment of the technology involved in the present invention provides an imaging element, a camera device, an operating method and a program of the imaging element, which can improve the real-time performance of the output of jitter degree information relative to the output of image data, compared with the case where the jitter degree information is output at a time interval longer than a period specified by the frame rate for the output of the image data.

[0008] Means for solving technical problems

[0009] The first method involved in the technology of the present invention is an imaging element, which includes: a storage unit, which stores image data obtained by shooting at a first frame rate and is built into the imaging element; a processing unit, which processes the image data and is built into the imaging element; a first output unit, which outputs the image data processed by the processing unit at a second frame rate and is built into the imaging element; and a second output unit, which outputs jitter degree information indicating the degree of jitter contained in an image represented by the image data and is built into the imaging element, the processing unit derives the jitter degree information by using the image data, and the second output unit outputs the jitter degree information derived by the processing unit at a rate higher than the second frame rate, and the first frame rate is a frame rate higher than the second frame rate.

[0010] A second aspect according to the technology of the present invention is the imaging element according to the first aspect, wherein the resolution of image data used in the processing unit is higher than the resolution of image data output by the first output unit.

[0011] A third aspect according to the technology of the present invention is the imaging element according to the first aspect, wherein a resolution of image data used in the processing unit is a maximum resolution.

[0012] A fourth aspect according to the technology of the present invention is the imaging element according to any one of the first to third aspects, wherein the image data used in the processing unit is partial image data representing a portion of the image.

[0013] A fifth aspect of the technology of the present invention is the imaging element according to the fourth aspect, comprising a photosensitive pixel group, wherein the partial image data is image data obtained from a region in the photosensitive pixel group that is in focus.

[0014] A sixth aspect according to the technology of the present invention is the imaging element according to the fifth aspect, wherein the area in focus is an area in which the focus state is continuously maintained in an operation mode in which the focus state is continuously maintained.

[0015] The seventh mode involved in the technology of the present invention is an imaging element described in the fifth mode or the sixth mode, wherein the photosensitive pixel group has multiple phase difference pixels, and the processing unit determines the area to be in focus from the photosensitive pixel group based on the pixel data of at least a part of the multiple phase difference pixels.

[0016] An eighth aspect according to the technology of the present invention is the imaging element according to any one of the first aspect to the seventh aspect, wherein the image data is time-series data obtained by imaging at a first frame rate.

[0017] The 9th method involved in the technology of the present invention is the imaging element described in the 8th method, wherein the first frame data related to a part of the frames in the timing data is output by the first output unit, and the second frame data related to the remaining frames is used by the processing unit to derive the jitter degree information.

[0018] A tenth aspect of the technology of the present invention is the imaging element described in the ninth aspect, wherein the second frame data is frame data representing a plurality of frames, and the processing unit derives the shaking degree information for each periodically determined frame based on the second frame data.

[0019] The 11th mode involved in the technology of the present invention is the imaging element described in the 10th mode, wherein the periodically determined frames are frames determined by a time interval that is longer than a period specified by the first frame rate and is less than a period specified by the second frame rate.

[0020] A twelfth aspect according to the technology of the present invention is the imaging element according to any one of the first aspect to the eleventh aspect, wherein at least the photoelectric conversion element and the storage unit are integrated into a single chip.

[0021] A thirteenth aspect according to the technology of the present invention is the imaging element according to the twelfth aspect, wherein the imaging element is a stacked-type imaging element in which a storage unit is stacked on a photoelectric conversion element.

[0022] The 14th method involved in the technology of the present invention is a camera device, which includes: an imaging element described in any one of the 1st to 13th methods; a vibration sensor that detects the applied vibration; and a correction unit that corrects the shake based on vibration data output from the vibration sensor and the shake degree information output from the second output unit.

[0023] The 15th method involved in the technology of the present invention is a camera device described in the 14th method, wherein the processing unit derives shake degree information based on image data whose shake is corrected by the correction unit, and the camera device also includes a post-stage circuit, which receives the shake degree information output from the second output unit, and the post-stage circuit uses vibration data and a reference output level of the vibration sensor to control the correction unit to correct the shake, and the post-stage circuit changes the reference output level according to the shake degree information output from the second output unit.

[0024] The 16th method involved in the technology of the present invention is a camera device described in the 14th method, wherein the processing unit derives shake degree information based on image data whose shake is corrected by the correction unit, and the imaging element also includes a correction control circuit, which uses vibration data and a reference output level of the vibration sensor to control the correction unit to correct the shake, and the correction control unit changes the reference output level based on the shake degree information output from the second output unit.

[0025] A 17th aspect according to the technology of the present invention is the imaging device according to the 15th aspect or the 16th aspect, wherein the amount of change in the reference output level is limited to a range that does not exceed the degree of reduction in the number of pixels of image data used in the processing unit.

[0026] The 18th method involved in the technology of the present invention is a camera device described in any one of the 15th to 17th methods, wherein the image data is timing data obtained by shooting at a first frame rate, the first frame data related to a part of the frames in the timing data is output by the first output unit, the second frame data related to the remaining frames in the timing data is frame data representing a plurality of frames, and is used to derive jitter degree information for each frame periodically determined by the processing unit, and the plurality of frames are frames obtained by reflecting the changed reference output level in the jitter correction based on the correction unit each time the reference output level is changed.

[0027] A nineteenth aspect according to the technology of the present invention is the imaging device according to any one of the fourteenth to eighteenth aspects, wherein the correction unit corrects the blur optically.

[0028] The 20th mode involved in the technology of the present invention is a camera device described in any one of the 14th to 19th modes, which also includes a control unit that performs at least one of controlling a storage device to store image data output from a first output unit and controlling a display device to display an image based on the image data output from the first output unit.

[0029] The 21st method involved in the technology of the present invention is a working method of an imaging element, wherein the imaging element is built-in with: a storage unit, which stores image data obtained by shooting at a first frame rate; and a processing unit, which processes the image data; a first output unit, which outputs the image data processed by the processing unit at a second frame rate; and a second output unit, the working method of the imaging element includes the following steps: using the image data through the processing unit, deriving jitter degree information indicating the degree of jitter contained in the image represented by the image data; and the second output unit outputting the jitter degree information derived by the processing unit at a rate greater than the second frame rate, and the first frame rate is a frame rate greater than the second frame rate.

[0030] The 22nd method involved in the technology of the present invention is a program for causing a computer to function as a processing unit, a first output unit and a second output unit included in an imaging element, wherein the imaging element is built-in with: a storage unit that stores image data obtained by shooting at a first frame rate; and a processing unit that processes the image data; a first output unit that outputs the image data processed by the processing unit at a second frame rate; and a second output unit, the program comprising the following steps: using the image data by the processing unit, deriving jitter degree information indicating the degree of jitter contained in an image represented by the image data; and the second output unit outputting the jitter degree information derived by the processing unit at a rate greater than the second frame rate, the first frame rate being a frame rate greater than the second frame rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a perspective view showing an example of the appearance of the imaging device according to the first and second embodiments.

[0032] Figure 2 This is a block diagram showing an example of the configuration of the imaging device according to the first embodiment.

[0033] Figure 3A This is a conceptual diagram for explaining the imaging frame rate of the imaging element included in the imaging device according to the first and second embodiments.

[0034] Figure 3B This is a conceptual diagram for explaining the output frame rate of the imaging element included in the imaging device according to the first and second embodiments.

[0035] Figure 4 This is a block diagram showing an example of a stacked structure of an imaging element according to the first embodiment and an example of a connection relationship among the imaging element, a signal processing unit, and a CPU.

[0036] Figure 5 This is a diagram showing an example of arrangement of imaging elements included in the imaging device according to the first embodiment on an imaging plane of each pixel included in the photoelectric conversion element.

[0037] Figure 6 It indicates the relative light of the subject to Figure 5 A conceptual diagram of an example of the incident characteristics of the first phase difference pixel and the second phase difference pixel included in the photoelectric conversion element shown.

[0038] Figure 7 Yes means Figure 5 FIG. 1 is a schematic structural diagram of an example of the structure of a non-phase difference pixel included in the photoelectric conversion element shown.

[0039] Figure 8 This is a block diagram showing an example of the configuration of an electrical system of the imaging element according to the first embodiment.

[0040] Fig. 9 This is a graph showing an example of a change pattern of the output level of the vibration sensor included in the imaging device according to the first embodiment in a state where the reference output level of the vibration sensor is not shifted.

[0041] Fig.10 This is a graph showing an example of a change pattern of the output level of the vibration sensor included in the imaging device according to the first embodiment when the reference output level of the vibration sensor is shifted.

[0042] Fig.11 This is a conceptual diagram showing an example of the structure of a reference output table stored in a storage circuit included in the imaging element according to the first embodiment.

[0043] Fig.12 This is a conceptual diagram showing an example of the correspondence relationship between digital non-phase difference image data and digital phase difference image data obtained by imaging by the imaging device according to the first embodiment.

[0044] Fig.13 This is a conceptual diagram showing an example of time series data stored in the memory 64 by the control circuit included in the imaging element of the imaging device according to the first embodiment.

[0045] Fig.14 This is a conceptual diagram showing an example of the positional relationship among the imaging surface of the imaging element, the AF control target area, and the in-focus state area according to the first embodiment.

[0046] Fig.15 This is a block diagram showing an example of the configuration of a control circuit included in the imaging element according to the first embodiment.

[0047] Fig.16The present invention is a conceptual diagram showing an example of a still image represented by still image data generated as practical image data by a practical image data generating unit of a processing unit included in a control circuit according to the first embodiment, and an example of a live preview image represented by live preview image data generated as practical image data by the practical image data generating unit.

[0048] Fig.17 This is a block diagram showing an example of the function of the Nth analyzing unit of the processing unit included in the control circuit according to the first embodiment.

[0049] Fig.18 This is a block diagram showing an example of the relationship between the control circuit included in the imaging element according to the first embodiment and peripheral devices.

[0050] Fig.19 This is a graph for explaining a method of calculating a changed reference output level from a reference output level, and is a graph showing an example of a relationship between an output level and a temperature.

[0051] Fig. 20 This is a block diagram showing an example of the relationship between a correction control unit, a storage circuit, a CPU, and an optical image stabilization unit of a control circuit included in the imaging element according to the first embodiment.

[0052] Fig.21 This is a flowchart showing an example of a phase difference pixel processing flow according to the first embodiment.

[0053] Fig. 22 This is a flowchart showing an example of a non-phase difference pixel processing flow according to the first embodiment.

[0054] Fig.23 This is a flowchart showing an example of the processing flow of the practical image data generating unit according to the first embodiment.

[0055] Fig.24 This is a flowchart showing an example of the flow of the N-th analytical image data acquisition process according to the first embodiment.

[0056] Fig.25 This is a flowchart showing an example of the processing flow of the Nth analysis unit involved in the first embodiment.

[0057] Fig.26 This is a flowchart showing an example of a processing flow of the correction control unit according to the first embodiment.

[0058] Fig. 27 This is a block diagram showing an example of the configuration of an imaging device according to the second embodiment.

[0059] Fig.28This is a block diagram showing an example of the relationship between the imaging element and the CPU according to the second embodiment.

[0060] Fig.29 This is a conceptual diagram showing an example of a method of installing various programs from a storage medium storing the various programs into a computer in an imaging element.

[0061] Fig.30 This is a block diagram showing an example of a schematic configuration of a smart device in which the imaging element according to the first or second embodiment is incorporated. DETAILED DESCRIPTION

[0062] Hereinafter, an example of an embodiment of an imaging element, an imaging device, an operating method of an imaging element, and a program according to the technology of the present invention will be described with reference to the drawings.

[0063] First, the meanings of terms used in the following description are explained.

[0064] CPU is short for "Central Processing Unit". GPU is short for "Graphics Processing Unit". RAM is short for "Random Access Memory". ROM is short for "Read Only Memory". DRAM is short for "Dynamic Random Access Memory". SRAM is short for "Static Random Access Memory".

[0065] LSI is the abbreviation of "Large-Scale Integration". ASIC is the abbreviation of "Application Specific Integrated Circuit". PLD is the abbreviation of "Programmable Logic Device". FPGA is the abbreviation of "Field-Programmable Gate Array".

[0066] SSD is the abbreviation of "Solid State Drive". USB is the abbreviation of "Universal Serial Bus". HDD is the abbreviation of "Hard Disk Drive". EEPROM is the abbreviation of "Electrically Erasable and Programmable Read Only Memory".

[0067] CCD is the abbreviation of "Charge Coupled Device". CMOS is the abbreviation of "Complementary Metal Oxide Semiconductor". EL is the abbreviation of "Electro-Luminescence". A / D is the abbreviation of "An alog / Digital". I / F is the abbreviation of "Interface". UI is the abbreviation of "User Interface". PC is the abbreviation of "Personal Computer". AF is the abbreviation of "Auto-Focus". AE is the abbreviation of "Automatic Exposure". SoC is the abbreviation of "System-on-a-chip".

[0068] OIS is the abbreviation of “Optical Image Stabilization”. BIS is the abbreviation of “Body Image Stabilization”. EIS is the abbreviation of “Electronic Image Stabilization”.

[0069] [First embodiment of the technology of the present invention]

[0070] As an example, Figure 1 As shown in FIG. 1 , the imaging device 10 is a lens-interchangeable camera and includes an imaging device body 12 and an interchangeable lens 14 that is replaceably mounted on the imaging device body 12 .

[0071] The imaging device 44 is provided on the imaging device body 12. The interchangeable lens 14 has an imaging lens 40. When the interchangeable lens 14 is mounted on the imaging device body 12, subject light representing the subject is transmitted through the imaging lens 40 and is imaged on an imaging surface 44A of the imaging device 44.

[0072] A release button 20 and a dial 22 are provided on the upper surface of the camera body 12. The dial 22 is operated when setting the operation mode of the camera system and the operation mode of the playback system. The release button 20 functions as a camera preparation indication unit and a camera indication unit, and is capable of detecting pressing operations in two stages: a camera preparation indication state and a camera indication state. The camera preparation indication state, for example, refers to a state in which the button is pressed from the standby position to an intermediate position (hereinafter referred to as a "half-pressed position"), and the camera indication state refers to a state in which the button is pressed to a final pressed position (hereinafter referred to as a "fully pressed position") beyond the intermediate position. In addition, hereinafter, the "state in which the button is pressed from the standby position to the half-pressed position" is referred to as the "half-pressed state", and the "state in which the button is pressed from the standby position to the fully pressed position" is referred to as the "fully pressed state".

[0073] In the imaging device 10, a moving image display shooting mode and a recording shooting mode are selectively set as operation modes according to a user's instruction. In each of the moving image display shooting mode and the recording shooting mode, an AF mode is set according to a user's instruction.

[0074] In the shooting mode for displaying moving images, if the AF mode is set, the AE function is used to set the exposure state for each frame, and the AF function is used to control the focus, and shooting for displaying moving images is performed. By shooting for displaying moving images, a live preview image is generated. In addition, the live preview image is usually also called a live view image.

[0075] The shooting mode for recording is roughly divided into a shooting mode for recording moving images and a shooting mode for recording still images, and the shooting mode for recording moving images and the shooting mode for recording still images are selectively set according to the user's instructions. In the camera device 10, in the shooting mode for recording moving images, if the AF mode is set, the AE function is exerted for each frame to set the exposure state, and the AF function is exerted to control the focus, and shooting for recording moving images is performed. The resolution of the moving image obtained by shooting for recording moving images is higher than the resolution of the instant preview image, and is stored in a specified storage device such as a memory card, USB memory, SSD or HDD (hereinafter referred to as the "specified storage device"). In addition, the specified storage device is an example of the "storage device" involved in the technology of the present invention.

[0076] In the still image recording shooting mode, when the AF mode is set, the shooting conditions are adjusted by half-pressing the release button 20, and then, when it is further fully pressed, still image shooting is performed. That is, after the AE function is activated by half-pressing the release button 20 to set the exposure state, the AF function is activated to control the focus, and when the release button 20 is fully pressed, still image shooting is performed. The still image obtained by performing still image shooting is stored in a predetermined storage device.

[0077] In the imaging device 10, the AF mode is roughly divided into a normal AF mode and a continuous AF mode, and the normal AF mode and the continuous AF mode are selectively set according to the user's instruction. The normal AF mode is an AF mode in which focus control is performed in a state where the focus position is fixed during the half-press state. The continuous AF mode is an AF mode in which the focus state with respect to the designated subject is continuously maintained without fixing the focus position during the half-press state.

[0078] As an example, Figure 2 As shown, the imaging device 10 includes a computer 13 , drivers 23 , 26 , a communication I / F 25 , a signal processing circuit 30 , a vibration sensor 32 , a position detection sensor 39 , a storage device 42 , an external I / F 46 , and a UI system device 43 .

[0079] The computer 13 includes a CPU 35, a ROM 36, and a RAM 37, and the CPU 35, the ROM 36, and the RAM 37 are connected to each other via a bus 38. In addition, the drivers 23, 26, the communication I / F 25, the signal processing circuit 30, the vibration sensor 32, the position detection sensor 39, the storage device 42, the external I / F 46, and the UI system device 43 are also connected to the bus 38.

[0080] The CPU 35 controls the entire imaging device 10. Various programs are stored in the ROM 36. The CPU 35 reads various programs from the ROM 36 and expands the read various programs into the RAM 37. The CPU 35 controls the entire imaging device 10 according to the various programs expanded into the RAM 37.

[0081] The storage device 42 is a nonvolatile storage device such as an SSD, an HDD, or an EEPROM. The CPU 35 reads and writes various information to and from the storage device 42.

[0082] The external I / F 46 is a communication device having an FPGA. A predetermined storage device is connected to the external I / F 46. The CPU 35 reads and writes various information to the predetermined storage device via the external I / F 46. In addition, a computing device such as a PC or a server is also connected to the external I / F 46. The external I / F 46 controls the exchange of various information between the CPU 35 and the computing device.

[0083] The UI system device 43 is a device that receives instructions from the user or prompts information to the user. The UI system device 43 includes a receiving device 43A and a display 43B. As an example of the display 43B, a liquid crystal display can be cited. The display 43B may not be a liquid crystal display, but other types of displays such as an organic electroluminescent (EL) display. Under the control of the CPU 35, the display 43B displays character information in addition to various images such as instant preview images and still images. In addition, the display 43B is an example of a "display device" involved in the technology of the present invention. Furthermore, the CPU 35 is an example of a "control unit (control processor)" involved in the technology of the present invention.

[0084] The receiving device 43A includes a hard key unit and a touch panel. The hard key unit includes a release button 20 (see Figure 1 ) and turntable 22 (reference Figure 1 ) multiple hard keys. The touch panel is a transmissive touch panel and overlaps the surface of the display area of ​​the display 43B. The touch panel detects contact based on an indicator such as a finger or a stylus. The CPU 35 operates according to various instructions received by the receiving device 43A.

[0085] The vibration sensor 32 is a device including a gyro sensor, and detects vibration applied to the camera device 10. Examples of the vibration applied to the camera device 10 include vibration applied to the camera device 10 by a user holding the camera device 10, vibration caused by wind to the camera device 10 placed on a support such as a tripod, and vibration applied by a vehicle.

[0086] The gyro sensor detects the rotation of each axis around the pitch axis PA, the yaw axis YA, and the roll axis RA (axis parallel to the optical axis L1) (reference Figure 1 ). The vibration sensor 32 converts the rotation amount around the pitch axis PA and the rotation amount around the yaw axis YA detected by the gyro sensor into a displacement in a two-dimensional plane parallel to the pitch axis PA and the yaw axis YA, thereby detecting the vibration of the camera device 10. And, the vibration sensor 32 outputs vibration data representing the detected vibration to the CPU 35. In addition, the meaning of parallel in the first embodiment involved in the technology of the present invention includes not only the meaning of completely parallel, but also the meaning of approximately parallel including the error allowed in design and manufacturing.

[0087] The imaging lens 40 includes an objective lens 15A and a lens group 15B. The imaging element 44 is located in the subsequent stage of the imaging lens 40. The objective lens 15A and the lens group 15B are arranged in this order along the optical axis L1 from the subject side to the imaging element 44 side.

[0088] The vibration sensor 32 has a built-in temperature sensor 34 . The temperature sensor 34 measures the temperature inside the vibration sensor 32 , and outputs temperature data indicating the measured temperature to the CPU 35 .

[0089] The communication I / F 25, the driver 26, and the signal processing circuit 30 are connected to the imaging element 44. The communication I / F 25 is a communication device having an FPGA, and controls the transmission and reception of various signals between the CPU 35 and the imaging element 44. The driver 26 supplies a timing control signal to the imaging element 44 under the control of the CPU 35. The timing control signal refers to, for example, a vertical synchronization signal and a horizontal synchronization signal. The vertical synchronization signal is a signal that specifies the timing for starting the shooting of one frame. The horizontal synchronization signal is a signal that specifies the timing for starting the shooting of one horizontal line.

[0090] The signal processing circuit 30 is a device including an ASIC. Image data 69 obtained by photographing a subject by the imaging element 44 is input to the signal processing circuit 30. The image data 69 is data representing an image of the subject. The signal processing circuit 30 performs various signal processing on the image data 69 input from the imaging element 44. The various signal processing includes, for example, well-known signal processing such as white balance adjustment, sharpness adjustment, gamma correction, color space conversion processing, and color difference correction.

[0091] In addition, in the first embodiment of the technology of the present invention, a device including an ASIC is exemplified as the signal processing circuit 30, but the technology of the present invention is not limited thereto, and the signal processing circuit 30 may be a device including an ASIC, an FPGA, and / or a PLD. Furthermore, the signal processing circuit 30 may also be a computer including a CPU, a ROM, and a RAM. The CPU may be one or more. Furthermore, the signal processing circuit 30 may be implemented by a combination of a hardware structure and a software structure.

[0092] The imaging lens 40 includes an objective lens 15A and a lens group 15B. The objective lens 15A and the lens group 15B are arranged in this order along the optical axis L1 from the object side to the imaging surface 44A side.

[0093] A movable aperture (not shown) and a mechanical shutter (not shown) are arranged between the lens group 15B and the imaging element 44. The aperture and the mechanical shutter are arranged in sequence along the optical axis L1 from the lens group 15B side to the imaging element 44 side. The aperture operates by receiving power from a driving source such as a motor (not shown), and the aperture opening is changed by the operation of the aperture, thereby adjusting the exposure. The mechanical shutter operates by receiving power from a driving source such as a motor (not shown). The subject light passes through the imaging lens 40 and is imaged on the imaging surface 44A via the mechanical shutter.

[0094] The lens group 15B includes a focusing lens 15B1, a zoom lens (not shown), and an anti-vibration lens 15B2. The focusing lens 15B1, the zoom lens, and the anti-vibration lens 15B2 are arranged in order along the optical axis L1 from the objective lens 15A side to the imaging element 44 side.

[0095] The focusing lens 15B1 and the zoom lens move along the optical axis L1 according to the applied power. The anti-vibration lens 15B2 moves in a direction perpendicular to the optical axis of the anti-vibration lens 15B2. In addition, the meaning of "vertical" in the first embodiment of the technology of the present invention includes not only the meaning of completely vertical, but also the meaning of approximately vertical including the error allowed in design and manufacturing.

[0096] In the AF mode, the focusing lens 15B1 moves along the optical axis L1 under the control of the CPU 35, so that the subject light is imaged on the imaging surface 44A at the focus position corresponding to the subject distance. The "focus position" mentioned here refers to the position of the focusing lens 15B1 on the optical axis L1 in the focused state. In the following, for the sake of convenience, the control of aligning the focusing lens 15B1 with the focus position is referred to as "AF control".

[0097] However, in the imaging device 10, jitter is generated due to vibration (hereinafter, also referred to as "vibration") applied to the imaging device 10. In the first embodiment of the technology involved in the present invention, "jitter" refers to the phenomenon that the optical axis L1 is tilted relative to the reference axis due to vibration, and the subject image obtained by imaging on the imaging surface 44A changes. The "reference axis" mentioned here refers to, for example, the optical axis L1 in a state where no vibration is applied. As the subject image, there can be cited an optical image and an electronic image (image). The subject image changes due to the change in the positional relationship between the optical axis L1 and the imaging surface 44A.

[0098] In order to correct the shake, the camera device 10 includes an optical shake correction unit 29. The optical shake correction unit 29 is an example of a "correction unit" involved in the technology of the present invention. Here, the "correction unit" can be replaced by a "correction device", "correction mechanism", "optical correction device" or "optical correction mechanism". The optical shake correction unit 29 operates under the control of the CPU 35 described later, and corrects the shake optically based on the vibration data output from the vibration sensor 32. In the first embodiment involved in the technology of the present invention, "correction of shake" includes not only the meaning of eliminating shake, but also the meaning of reducing shake.

[0099] In addition, for the sake of convenience, the following description is based on the premise that the optical shake correction unit 29 corrects the shake to such an extent that the shake caused by the vibration applied to the camera device 10 is not visually perceived in the image obtained by capturing the subject by the camera device 10 .

[0100] The optical shake correction unit 29 includes an anti-vibration lens 15B2, an actuator 17, a driver 23, and a position detection sensor 39. As a method for correcting the shake by the optical shake correction unit 29, various well-known methods can be adopted. In the first embodiment of the technology involved in the present invention, OIS is adopted as a method for correcting the shake. OIS refers to a method for correcting the shake by moving the anti-vibration lens 15B2 based on vibration data obtained by detecting vibration by a vibration sensor 32 (described later). Specifically, the shake is corrected by moving the anti-vibration lens 15B2 in a direction to cancel the shake by an amount that cancels the shake.

[0101] An actuator 17 is mounted on the anti-vibration lens 15B2. The actuator 17 is a displacement mechanism equipped with a coil motor, and by driving the coil motor, the anti-vibration lens 15B2 is moved in a direction perpendicular to the optical axis of the anti-vibration lens 15B2. In addition, here, as the actuator 17, a displacement mechanism equipped with a coil motor is exemplified, but the technology of the present invention is not limited to this, and other power sources such as stepping motors or piezoelectric elements can also be used instead of coil motors.

[0102] The actuator 17 is controlled by the driver 23. When the actuator 17 is driven under the control of the driver 23, the position of the anti-vibration lens 15B2 is mechanically changed with respect to the optical axis L1.

[0103] The position detection sensor 39 detects the current position of the anti-vibration lens 15B2 and outputs a position signal indicating the detected current position. Here, as an example of the position detection sensor 39, a device including a Hall element is used. Here, the current position of the anti-vibration lens 15B2 refers to the current position within the two-dimensional plane of the anti-vibration lens. The two-dimensional plane of the anti-vibration lens refers to a two-dimensional plane perpendicular to the optical axis of the anti-vibration lens 15B2. In addition, here, as an example of the position detection sensor 39, a device including a Hall element is used, but the technology of the present invention is not limited to this, and a magnetic sensor or an optical sensor may also be used instead of the Hall element.

[0104] The imaging element 44 is an example of a "stacked imaging element" involved in the technology of the present invention. In the first embodiment involved in the technology of the present invention, the imaging element 44 is a CMOS image sensor. Moreover, here, a CMOS image sensor is exemplified as the imaging element 44, but the technology of the present invention is not limited to this. For example, even if the imaging element 44 is a CCD image sensor, the technology of the present invention is also established.

[0105] In the imaging element 44, by capturing the subject at the imaging frame rate, as an example, Figure 3A As shown, a plurality of image data 69 each representing an image of the subject is generated. Furthermore, in the imaging element 44, the generated plurality of image data 69 is output at an output frame rate. Both the imaging frame rate and the output frame rate are variable frame rates. In addition, the imaging frame rate is an example of the “first frame rate” involved in the technology of the present invention, and the output frame rate is an example of the “second frame rate” involved in the technology of the present invention.

[0106] The image capture frame rate and the output frame rate have a relationship of "image capture frame rate > output frame rate". That is, the image capture frame rate is a frame rate higher than the output frame rate, and is specified by the vertical synchronization signal which is one of the timing control signals. As an example, Figure 3A As shown in FIG. 1 , the image capture frame rate is the frame rate at which 8 frames are captured within a period T. Figure 3B As shown, the output frame rate is a frame rate for outputting two frames in a period T. Specifically, as an example of a camera frame rate, 240fps (frame per second) can be cited, and as an example of an output frame rate, 60fps can be cited. In addition, here, an example is given in which the camera frame rate and the output frame rate have a relationship of "camera frame rate>output frame rate", but even if "camera frame rate=output frame rate", the technology of the present invention is also established.

[0107] As an example, Figure 4As shown, a photoelectric conversion element 61, a processing circuit 62, and a memory 64 are built into the imaging element 44. Here, the processing circuit 62 is an example of a "processor" involved in the technology of the present invention, and the memory 64 is an example of a "storage unit (memory)" involved in the technology of the present invention. The imaging element 44 is an imaging element in which the photoelectric conversion element 61, the processing circuit 62, and the memory 64 are integrated into a single chip. That is, the photoelectric conversion element 61, the processing circuit 62, and the memory 64 are packaged. In the imaging element 44, the processing circuit 62 and the memory 64 are stacked on the photoelectric conversion element 61. Specifically, the photoelectric conversion element 61 and the processing circuit 62 are electrically connected to each other through conductive bumps such as copper (not shown in the figure), and the processing circuit 62 and the memory 64 are also electrically connected to each other through conductive bumps such as copper (not shown in the figure). Here, a three-layer structure of the photoelectric conversion element 61, the processing circuit 62, and the memory 64 is illustrated, but the technology of the present invention is not limited thereto, and the two-layer structure of the memory layer and the photoelectric conversion element 42 may be used, in which the processing circuit 110 and the memory 112 are used as one layer, or the two-layer structure of the processor layer and the memory 112 may be used, in which the processing circuit 110 and the photoelectric conversion element 42 are used as one layer. Furthermore, the memory 112 may be built into the processing circuit 110.

[0108] The processing circuit 62 is, for example, an LSI, and the memory 64 is, for example, a DRAM. However, the technology of the present invention is not limited thereto, and an SRAM may be used as the memory 64 instead of the DRAM.

[0109] The processing circuit 62 is a device including an ASIC and an FPGA, and controls the entire imaging element 44 according to the instruction of the CPU 35. In addition, here, an example is given in which the processing circuit 62 is implemented by a device including an ASIC and an FPGA, but the technology of the present invention is not limited to this, and for example, a device including an ASIC, an FPGA and / or a PLD may also be used. In addition, as the processing circuit 62, a computer including a CPU, a ROM and a RAM may be used. The CPU may be one or more. In addition, the processing circuit 62 may be implemented by a combination of a hardware structure and a software structure.

[0110] The photoelectric conversion element 61 has a plurality of photodiodes arranged in a matrix. An example of the plurality of photodiodes is photodiodes for "4896×3265" pixels.

[0111] A color filter is disposed in each photodiode included in the photoelectric conversion element 61. The color filter includes a G filter corresponding to G (green), an R filter corresponding to R (red), and a B filter corresponding to B (blue), which are most useful for obtaining a brightness signal. The photoelectric conversion element 61 has R pixels, G pixels, and B pixels (refer to Figure 5). The R pixel is a pixel corresponding to a photodiode equipped with an R color filter, the G pixel is a pixel corresponding to a photodiode equipped with a G color filter, and the B pixel is a pixel corresponding to a photodiode equipped with a B color filter.

[0112] The imaging element 44 has a so-called electronic shutter function, and controls the charge accumulation time of each photodiode in the photoelectric conversion element 61 by exercising the electronic shutter function under the control of the CPU 35. The charge accumulation time is a so-called shutter speed.

[0113] In the imaging device 10, still image shooting and moving image shooting are performed by a rolling shutter method. Here, the rolling shutter method is exemplified, but the technology of the present invention is not limited to this, and a global shutter method may be applied instead of the rolling shutter method.

[0114] The processing circuit 62 reads out the image data 69 obtained by photographing the subject by the photoelectric conversion element 61. The image data 69 is the signal charge accumulated in the photoelectric conversion element 61. The processing circuit 62 performs A / D conversion on the analog image data 69 read out from the photoelectric conversion element 61. The processing circuit 62 stores the digital image data 69 obtained by performing A / D conversion on the analog image data 69 in the memory 64. The memory 64 includes a plurality of storage areas, and in each of the plurality of storage areas, for example, the digital image data 69 is stored in the address corresponding to the pixel of the photoelectric conversion element 61 in pixel units for each frame. And the plurality of storage areas are randomly accessed by the processing circuit 62.

[0115] The processing circuit 62 is connected to the CPU 35 and the signal processing circuit 30 . The processing circuit 62 exchanges various signals with the CPU 35 . Furthermore, the processing circuit 62 outputs the digital image data 69 to the signal processing circuit 30 .

[0116] As an example, Figure 5 As shown, in the imaging surface 44A of the photoelectric conversion element 61, R pixels, G pixels, and B pixels are arranged in the row direction (horizontal direction) and the column direction (vertical direction) with a predetermined periodicity. In the first embodiment of the technology of the present invention, R pixels, G pixels, and B pixels are arranged in a periodicity corresponding to the X-Trans (registered trademark) arrangement. Figure 5 In the example shown, an X-Trans arrangement is illustrated, but the technology of the present invention is not limited to this, and the arrangement of R pixels, G pixels, and B pixels may also be a Bayer arrangement, a honeycomb arrangement, or the like.

[0117] exist Figure 5In the example shown, in the first row, the R pixel, the G pixel, and the B pixel are sequentially arranged in a row direction in a circular manner with G pixel, B pixel, R pixel, G pixel, R pixel, and B pixel. Also, in the second row, the R pixel, the G pixel, and the B pixel are sequentially arranged in a row direction in a circular manner with R pixel, G pixel, G pixel, B pixel, G pixel, and G pixel. Also, in the third row, the R pixel, the G pixel, and the B pixel are sequentially arranged in a row direction in a circular manner with B pixel, G pixel, G pixel, R pixel, G pixel, and G pixel. Also, in the fourth row, the R pixel, the G pixel, and the B pixel are sequentially arranged in a row direction in a circular manner with G pixel, R pixel, B pixel, G pixel, B pixel, and R pixel. Also, in the fifth row, the R pixel, the G pixel, and the B pixel are sequentially arranged in a row direction in a circular manner with B pixel, G pixel, G pixel, R pixel, G pixel, and G pixel. Also, in the sixth row, the R pixel, the G pixel, and the B pixel are sequentially arranged in a row direction in a circular manner with R pixel, G pixel, G pixel, B pixel, G pixel, and G pixel. Then, the arrangement pattern of R pixels, G pixels, and B pixels in the first to sixth rows is repeated in the column direction in units of six rows, thereby forming an arrangement pattern of R pixels, G pixels, and B pixels in the entire photoelectric conversion element 61 .

[0118] The photoelectric conversion element 61 is formed to include a photosensitive pixel group. The photosensitive pixel group has two types of photosensitive pixels, namely, phase difference pixels and pixels different from the phase difference pixels, namely, non-phase difference pixels N. Usually, non-phase difference pixels N are also called normal pixels. A plurality of phase difference pixel lines 61A and a plurality of non-phase difference pixel lines 61B are arranged on the camera surface 44A. The phase difference pixel line 61A is a horizontal line containing phase difference pixels. Specifically, the phase difference pixel line 61A is a horizontal line in which phase difference pixels and non-phase difference pixels N are mixed. The non-phase difference pixel line 61B is a horizontal line containing only a plurality of non-phase difference pixels N, that is, a horizontal line composed of a plurality of non-phase difference pixels N.

[0119] As an example, Figure 5 As shown, phase difference pixel lines 61A and a predetermined number of non-phase difference pixel lines 61B are alternately arranged along the column direction on the imaging surface 44A. The "predetermined number of rows" mentioned here refers to, for example, 2 rows. In addition, here, 2 rows are exemplified as the predetermined number of rows, but the technology of the present invention is not limited to this, and the predetermined number of rows can be more than 3 rows, or can be more than a dozen rows, dozens of rows, or hundreds of rows.

[0120] The phase difference pixel line 61A is arranged by skipping 2 rows in the column direction from the first row to the final row. Some pixels of the phase difference pixel line 61A are phase difference pixels. Specifically, the phase difference pixel line 61A is a horizontal line in which phase difference pixels and non-phase difference pixels N are periodically arranged. The phase difference pixels are roughly divided into the first phase difference pixel L and the second phase difference pixel R. In the phase difference pixel line 61A, the first phase difference pixel L and the second phase difference pixel R are alternately arranged as G pixels in the row direction at intervals of several pixels.

[0121] The first phase difference pixels L and the second phase difference pixels R are arranged to appear alternately in the column direction. Figure 5 In the example shown, in the fourth column, the first phase difference pixel L, the second phase difference pixel R, the first phase difference pixel L, and the second phase difference pixel R are arranged in sequence from the first row along the column direction. That is, the first phase difference pixel L and the second phase difference pixel R are arranged alternately from the first row along the column direction. Figure 5 In the example shown, in the 10th column, the second phase difference pixel R, the first phase difference pixel L, the second phase difference pixel R, and the first phase difference pixel L are sequentially arranged along the column direction from the first row. That is, the second phase difference pixel R and the first phase difference pixel L are alternately arranged along the column direction from the first row.

[0122] As an example, Figure 6 As shown, the first phase difference pixel L includes a microlens 19, a light shielding member 21A, and a photodiode PD. In the first phase difference pixel L, the light shielding member 21A is arranged between the microlens 19 and the light receiving surface of the photodiode PD. The left half of the light receiving surface of the photodiode PD in the row direction (the left side when facing the subject from the light receiving surface (in other words, the right side when facing the light receiving surface from the subject)) is shielded by the light shielding member 21A.

[0123] The second phase difference pixel R includes a microlens 19, a light shielding member 21B, and a photodiode PD. In the second phase difference pixel R, the light shielding member 21B is disposed between the microlens 19 and the light receiving surface of the photodiode PD. The right half of the light receiving surface of the photodiode PD in the row direction (the right side when facing the subject from the light receiving surface (in other words, the left side when facing the light receiving surface from the subject)) is shielded by the light shielding member 21B.

[0124] The light beam passing through the exit pupil of the imaging lens 40 is roughly divided into left area passing light 300L and right area passing light 300R. The left area passing light 300L refers to the left half of the light beam passing through the exit pupil of the imaging lens 40 when it faces the subject side from the phase difference pixel side, and the right area passing light 300R refers to the right half of the light beam passing through the exit pupil of the imaging lens 40 when it faces the subject side from the phase difference pixel side. The light beam passing through the exit pupil of the imaging lens 40 is divided left and right by the micro lens 19 and the light shielding members 21A and 21B functioning as a pupil dividing unit, and the first phase difference pixel L receives the left area passing light 300L as subject light, and the second phase difference pixel R receives the right area passing light 300R as subject light. As a result, the imaging element 44 generates first phase difference image data representing a first phase difference image corresponding to the subject image corresponding to the left area transmission light 300L and second phase difference image data representing a second phase difference image corresponding to the subject image corresponding to the right area transmission light 300R.

[0125] In addition, for the sake of convenience, the first phase difference pixel L and the second phase difference pixel R are referred to as “phase difference pixels” when there is no need to distinguish between them. Also, for the sake of convenience, the light shielding members 21A and 21B are referred to as “light shielding members” without being marked with a symbol when there is no need to distinguish between them. Also, the first phase difference image data and the second phase difference image data are referred to as “phase difference image data” when there is no need to distinguish between them.

[0126] As an example, Figure 7 As shown, the non-phase difference pixel is different from the phase difference pixel in that it does not have a light shielding component. The photodiode PD of the non-phase difference pixel receives the left area passing light 300L and the right area passing light 300R as the subject light. As a result, the imaging element 44 generates non-phase difference image data representing a non-phase difference image corresponding to the subject light.

[0127] As an example, Figure 8 As shown, the processing circuit 62 includes a readout circuit 62A, a digital processing circuit 62B, a storage circuit 62C, and a control circuit 62D.

[0128] The readout circuit 62A includes a first readout circuit 62A1 and a second readout circuit 62A2. The first readout circuit 62A1 and the second readout circuit 62A2 are connected to the photoelectric conversion element 61, the digital processing circuit 62B, and the control circuit 62D, respectively. The digital processing circuit 62B, the storage circuit 62C, and the memory 64 are connected to the control circuit 62D, respectively. The control circuit 62D is connected to the signal processing circuit 30 and the CPU 35.

[0129] The image data 69 is roughly divided into analog image data 69A and digital image data 69B. In the following, for the sake of convenience, when it is not necessary to distinguish between the analog image data 69A and the digital image data 69B, they are referred to as "image data 69".

[0130] The analog image data 69A is roughly divided into analog phase difference image data related to phase difference pixels, namely analog phase difference image data 69A1, and analog non-phase difference image data related to non-phase difference pixels N, namely analog non-phase difference image data 69A2. The digital image data 69B is roughly divided into digital phase difference image data 69B1 and digital non-phase difference image data 69B2. The digital phase difference image data 69B1 is image data obtained by digitizing the analog phase difference image data 69A1. The digital non-phase difference image data 69B2 is image data obtained by digitizing the analog non-phase difference image data 69A2. In addition, the analog image data 69A and the digital image data 69B are examples of "image data" involved in the technology of the present invention.

[0131] The CPU 35 supplies various signals such as temperature data, position signal, and vibration data to the control circuit 62D. Figure 2 ) supplies a timing control signal to the control circuit 62D.

[0132] The readout circuit 62A reads out analog image data 69A obtained by photographing the subject at the imaging frame rate in units of horizontal lines. That is, the readout circuit 62A controls the photoelectric conversion element 61 under the control of the control circuit 62D, and reads out the analog image data 69A in units of horizontal lines from the photoelectric conversion element 61. The imaging frame rate is determined by the slave driver 26 (reference Figure 2 ) specifies that the readout timing in units of horizontal lines is determined according to a horizontal synchronization signal which is one of the timing control signals supplied.

[0133] The readout circuit 62A is configured to selectively read out the analog phase difference image data 69A1 and the analog non-phase difference image data 69A2 for all pixels included in the photoelectric conversion element 61. The details will be described later, and the selective readout of the analog phase difference image data 69A1 and the analog non-phase difference image data 69A2 is realized by the first readout circuit 62A1 and the second readout circuit 62A2.

[0134] In addition, here, an example of selectively reading out the analog phase difference image data 69A1 and the analog non-phase difference image data 69A2 for all pixels included in the photoelectric conversion element 61 is given, but the technology of the present invention is not limited to this. For example, the analog phase difference image data 69A1 and the analog non-phase difference image data 69A2 may be selectively read out for a specified portion of all pixels included in the photoelectric conversion element 61.

[0135] From drive 26 (reference Figure 2 ) supplies a vertical synchronization signal to the control circuit 62D, and the control circuit 62D transmits the vertical synchronization signal to the readout circuit 62A. When the vertical synchronization signal is input from the control circuit 62D, the readout circuit 62A starts reading out one frame of analog image data 69A from the photoelectric conversion element 61.

[0136] Specifically, the control circuit 62D will drive the driver 26 (reference Figure 2 ) is transmitted to the first readout circuit 62A1 and the second readout circuit 62A2. When the vertical synchronization signal is input to each of the first readout circuit 62A1 and the second readout circuit 62A2, the analog phase difference image data 69A1 of the phase difference pixel and the analog non-phase difference image data 69A2 of the non-phase difference pixel N are read out independently. The analog phase difference image data 69A1 is read out by the first readout circuit 62A1, and the analog non-phase difference image data 69A2 is read out by the second readout circuit 62A2.

[0137] In more detail, when a vertical synchronization signal is input from the control circuit 62D, the first readout circuit 62A1 starts reading out one frame of analog phase difference image data 69A1 from the photoelectric conversion element 61. Furthermore, the first readout circuit 62A1 reads out the analog phase difference image data 69A1 from each phase difference pixel for each phase difference pixel line 61A according to the horizontal synchronization signal input from the control circuit 62D. When a readout start condition (described later) is satisfied, the second readout circuit 62A2 starts reading out one frame of analog non-phase difference image data 69A2 from the photoelectric conversion element 61. Furthermore, the second readout circuit 62A2 reads out the analog non-phase difference image data 69A2 from each non-phase difference pixel N for each non-phase difference pixel line 61B according to the horizontal synchronization signal input from the control circuit 62D.

[0138] The “readout start condition” referred to here is a condition in which a vertical synchronization signal is input from the control circuit 62D and the readout of one frame of the pseudo phase difference image data 69A1 by the first readout circuit 62A1 is completed.

[0139] The readout circuit 62A performs analog signal processing on the analog image data 69A read out from the photoelectric conversion element 61. The analog signal processing includes well-known processing such as noise elimination processing and analog gain processing. The noise elimination processing is a processing for eliminating noise caused by the deviation of characteristics between pixels included in the photoelectric conversion element 61. The analog gain processing is a processing for applying gain to the analog image data 69A. The analog image data 69A after the analog signal processing is output to the digital processing circuit 62B through the readout circuit 62A.

[0140] The digital processing circuit 62B includes an A / D converter 62B1. The digital processing circuit 62B performs digital signal processing on the analog image data 69A input from the readout circuit 62A. The digital signal processing includes, for example, correlated double sampling, A / D conversion by the A / D converter 62B1, and digital gain processing.

[0141] The analog image data 69A is subjected to correlated double sampling by the digital processing circuit 62B. The analog image data 69A subjected to the correlated double sampling signal processing is subjected to A / D conversion by the A / D converter 62B1, whereby the analog image data 69A is digitized to obtain digital image data 69B. Furthermore, the digital image data 69B is subjected to digital gain processing by the digital processing circuit 62B. The digital gain processing is processing for applying gain to the digital image data 69B.

[0142] The control circuit 62D acquires the digital image data 69B obtained by performing the digital signal processing from the digital processing circuit 62B, and stores the acquired digital image data 69B in the memory 64 .

[0143] The memory 64 is a memory capable of storing a plurality of frames of digital image data 69B. The memory 64 has a plurality of storage areas, and the digital image data 69B is stored in corresponding storage areas of the memory 64 in units of pixels by the control circuit 62D.

[0144] The control circuit 62D can perform random access to the memory 64 and obtain the digital non-phase difference image data 69B2 from the memory 64. The control circuit 62D performs signal processing on the digital non-phase difference image data 69B2 obtained from the memory 64. Examples of the "signal processing" mentioned here include synchronization processing, digital thinning processing, digital addition processing, and data embedding processing.

[0145] The synchronization process is a process of calculating all color information for each pixel based on a mosaic image corresponding to the arrangement of color filters. For example, in the case of an imaging element composed of RGB color filters, all RGB color information is calculated for each pixel based on a mosaic image composed of RGB.

[0146] The digital thinning process is a process of thinning out the pixels included in the digital non-phase difference image data 69B2 in line units. Line units refer to, for example, horizontal line units and / or vertical line units. The digital addition process is, for example, a process of performing addition and averaging on the pixel values ​​of multiple pixels included in the digital non-phase difference image data 69B2. The data embedding process may include a process of embedding specific data into the low-order free bits of the digital non-phase difference image data 69B2. As the "specific data" mentioned here, for example, information that can determine the method of digital thinning processing performed on the digital non-phase difference image data 69B2 or a frame number that can determine the frame may be cited.

[0147] The control circuit 62D outputs the digital phase difference image data 69B1 and the practical image data (described later) as the digital non-phase difference image data 69B2 after signal processing to the signal processing circuit 30 at the output frame rate. The digital phase difference image data 69B1 is transmitted to the CPU 35 via the signal processing circuit 30. The CPU 35 performs AF control based on the digital phase difference image data 69B1. The practical image data is also transmitted to the CPU 35 via the signal processing circuit 30. The CPU 35 performs various processing on the practical image data. The various processing refers to, for example, processing for causing the display 43B to display an image represented by the practical image data and processing for storing the practical image data in a predetermined storage device.

[0148] The storage circuit 62C stores a reference output table 62C1 (reference Fig.11 The details will be described later. The reference output table 62C1 is a table that establishes a corresponding relationship between the temperature of the vibration sensor 32 and the reference output level of the vibration sensor 32 (hereinafter referred to as "reference output level"). In the first embodiment of the technology involved in the present invention, the unit of temperature is "℃", and the output level of the vibration sensor 32 (in Fig.11 and Fig.19 The unit of the reference output level in the example shown is "mdps". The reference output level refers to the output level of the vibration sensor 32 in a vibration-free state. The vibration-free state refers to a state in which the vibration sensor 32 is stationary, that is, a state in which no vibration is applied to the camera device 10. In addition, hereinafter, for the sake of convenience of explanation, the state in which the vibration sensor 32 is moving, that is, the state in which vibration is applied to the camera device 10 is referred to as a "vibration state". Also, in the following explanation, the output level of the vibration sensor 32 (in the vibration-free state) is omitted. Fig.11 and Fig.19 In the example shown, the unit of the reference output level) is indicated.

[0149] As will be described in detail later, the control circuit 62D generates a position indication signal based on the reference output level of the reference output table 62C1 stored in the storage circuit 62C, and the temperature data, position signal, and vibration data supplied from the CPU 35. The control circuit 62D outputs the generated position indication signal to the CPU 35. The position indication signal is an indication of the position of the anti-vibration lens 15B2 (reference position) in the two-dimensional plane of the anti-vibration lens. Figure 2 The CPU 35 controls the optical shake correction unit 29 according to the position instruction signal input from the control circuit 62D. The optical shake correction unit 29 corrects the shake by aligning the position of the anti-vibration lens 15B2 to the position indicated by the position instruction signal.

[0150] However, in the control circuit 62D, the correction amount required for the correction of the shake (hereinafter referred to as the "shake correction amount") is calculated, and the position instruction signal is determined according to the calculated shake correction amount. Fig. 9 As shown, the time integral value (in Fig. 9 In the example shown, the area of ​​the grid area corresponds to the jitter correction amount. In a vibration-free state, the current output level is consistent with the reference output level, but in a vibration state, the current output level is higher or lower than the reference output level.

[0151] As a premise for the control circuit 62D to calculate the jitter correction amount, Fig. 9 In the example shown, although it is ideal that the reference output level is constant, in reality, as an example, Fig.10 As shown, the reference output level is not constant but shifts. As the cause of the reference output level shift, the environment in which the vibration sensor 32 is installed and the time degradation can be cited. When the reference output level shifts, the displacement amount of the reference output level is calculated as a time integral value, so the error of the displacement amount of the reference output level is included in the jitter correction amount. In this case, in a vibrating state, a jitter correction amount larger than the actually required jitter correction amount is calculated. In addition, in a non-vibrating state, since the jitter correction amount is a value other than "0", even if there is no need to correct the jitter, the optical jitter correction unit 29 will operate as if there is jitter.

[0152] Therefore, in the imaging element 44, as an example, Fig.11 As shown, the reference output level of the reference output table 62C1 is optimized, and the jitter correction amount is calculated based on the optimized reference output level. Since the reference output level varies depending on the temperature in the vibration sensor 32, the reference output table 62C1 has a plurality of temperatures and reference output levels in the vibration sensor 32, and a different reference output level is pre-associated with each of the plurality of temperatures.

[0153] exist Fig.11 In the example shown, multiple temperatures are set in a range from 20.0°C to 27.0°C with a gradient of 0.5°C. In the reference output table 62C1, the temperatures and reference output levels that are associated with each other are fixed values. The reference output level to be changed is the reference output level after the optical jitter correction unit 29 corrects the jitter. Fig.11 In the example shown, the change reference output level is associated with each temperature in the range of 23.0°C to 26.0°C.

[0154] The reference output level is used for calculating the time integral value, that is, the jitter correction amount. If the reference output level has not been changed from the vibration sensor 32, the reference output level is used as the reference output level. If the reference output level is changed from the vibration sensor 32, the reference output level is used as the reference output level. Fig.11 In the example shown, the reference output level is used as the use reference output level for each temperature in the range of 20.0°C to 22.5°C and in the range of 26.5°C to 27.0°C. Fig.11 In the example shown, for each temperature in the range of 23.0°C to 26.0°C, the changed reference output level is adopted as the used reference output level.

[0155] As an example, Fig.12 As shown, for each of the time-series digital non-phase difference image data 69B2 obtained by photographing at the imaging frame rate by the photoelectric conversion element 61, the digital phase difference image data 69B1 is associated with each frame in a time-series manner. That is, the digital phase difference image data 69B1 and the digital non-phase difference image data 69B2 are acquired by the control circuit 62D in units of one frame.

[0156] As an example, Fig.13 As shown, the control circuit 62D generates time series data based on the time series digital non-phase difference image data 69B2 obtained by shooting at the imaging frame rate, and stores the generated time series data in the memory 64 in a FIFO manner. The time series data is roughly divided into the first frame data related to a part of the frames and the second frame data related to the remaining frames. The second frame data is frame data representing multiple frames.

[0157] The first frame data is the signal processing image data 69B2a. The signal processing image data 69B2a is the digital non-phase difference image data 69B2 itself, and is associated with the digital phase difference image data 69B1 in the same frame. The second frame data is the first analysis image data 69B2b, the second analysis image data 69B2c, and the third analysis image data 69B2d. The details will be described later. The first analysis image data 69B2b, the second analysis image data 69B2c, and the third analysis image data 69B2d are local image data representing a local image in the digital non-phase difference image data 69B2 of one frame of all pixels of the imaging surface 44A. The "local image data representing a local image" mentioned here refers to, for example, a local image of the subject that converges within the field of view of the imaging device 10.

[0158] The periodically determined frames in the time series data are frames determined at time intervals specified at 60fps. That is, the time intervals between the periodically determined frames are time intervals equivalent to the period specified at 60fps. The "between frames" mentioned here refer to, for example, between the signal processing image data 69B2a adjacent in time, between the first analysis image data 69B2b adjacent in time, between the second analysis image data 69B2c adjacent in time, and between the third analysis image data 69B2d adjacent in time. The signal processing image data 69B2a, the first analysis image data 69B2b, the second analysis image data 69B2c, and the third analysis image data 69B2d are obtained at time intervals specified at 60fps, respectively, and stored in the memory 64 in a FIFO manner.

[0159] In addition, for the sake of convenience, the first analysis image data 69B2b, the second analysis image data 69B2c, and the third analysis image data 69B2d are referred to as "Nth analysis image data" when it is not necessary to distinguish and explain them. Furthermore, the image represented by the Nth analysis image data is referred to as "Nth analysis image". That is, the Nth analysis image is a general term for the first analysis image represented by the first analysis image data 69B2b, the second analysis image represented by the second analysis image data 69B2c, and the third analysis image represented by the third analysis image data 69B2d.

[0160] Here, the Nth analysis image data will be described. As an example, Fig.14As shown, the imaging plane 44A has an AF control target area 61C, which is an image side area that is an AF control target. The AF control target area 61C corresponds to a part of the object side area that is designated as the AF control target in the subject and is followed in the continuous AF mode. The AF control target area 61C may be a region that is controlled by the receiving device 43A (refer to Figure 2 ) may be an area designated by an instruction received, or may be an area determined as a face area by a so-called face detection function.

[0161] The Nth analysis image data is digital non-phase difference image data 69B2 corresponding to the focus state area in the digital non-phase difference image data 69B2 obtained by reading the analog non-phase difference image data 69A2 by the first readout circuit 62A1. The focus state area is an example of a "region that becomes a focus state" involved in the technology of the present invention. The focus state area refers to the image side area that becomes a focus state in the photosensitive pixel group within the AF control object area 61C, that is, the image side area that continues to maintain a focus state in the continuous AF mode. The focus state area is determined by analyzing the digital phase difference image data 69B1 obtained in the same frame as the digital non-phase difference image data 69B2 by the control circuit 62D.

[0162] As an example, Fig.15 As shown, the control circuit 62D includes a processing unit 70, a first output unit 72, a second output unit 74, and a correction control unit 76. The processing unit 70 is an example of a "processing unit (processing circuit)" involved in the technology of the present invention, the first output unit 72 is an example of a "first output unit (first output circuit)" involved in the technology of the present invention, the second output unit 74 is an example of a "second output unit" involved in the technology of the present invention, and the correction control unit 76 is an example of a "correction control unit (correction control circuit)" involved in the technology of the present invention.

[0163] The processing unit 70 processes the image data 69. The processing unit 70 includes a practical image data generating unit 78, a first analyzing unit 80A1, a second analyzing unit 80A2, and a third analyzing unit 80A3. The practical image data generating unit 78 is connected to the first output unit 72. The first analyzing unit 80A1, the second analyzing unit 80A2, and the third analyzing unit 80A3 are connected to the second output unit 74. The second output unit 74 is connected to the correction control unit 76. In addition, for the sake of convenience, the first analyzing unit 80A1, the second analyzing unit 80A2, and the third analyzing unit 80A3 are referred to as "the Nth analyzing unit 80A" when it is not necessary to distinguish and explain them.

[0164] The practical image data generating unit 78 acquires the signal processing image data 69B2a and the digital phase difference image data 69B1 corresponding to the signal processing image data 69B2a from the memory 64. The practical image data generating unit 78 generates the practical image data by performing signal processing on the signal processing image data 69B2a acquired from the memory 64. The practical image data refers to, for example, still image data representing a still image or instant preview image data representing an instant preview image. In the first embodiment of the technology of the present invention, the still image data and the instant preview image data are selectively switched according to the instruction received by the receiving device 43A. In addition, here, the still image data and the instant preview image data are used as the practical image data, but the technology of the present invention is not limited to this, and the practical image data may also be recording animation data representing recording animation.

[0165] The first output unit 72 acquires the practical image data generated by the practical image data generating unit 78 and the digital phase difference image data 69B1 corresponding to the practical image data, and outputs the acquired practical image data and digital phase difference image data 69B1 at the output frame rate.

[0166] The Nth analyzing section 80A acquires the Nth analysis image data from the memory 64, and derives the shake degree information by using the acquired Nth analysis image data. The shake degree information is information indicating the shake degree contained in the image represented by the Nth analysis image data, that is, the Nth analysis image. In other words, the shake degree is an index indicating the effect of the shake correction by the optical shake correction section 29.

[0167] The first analysis unit 80A1 acquires the first analysis image data 69B2b and the digital phase difference image data 69B1 corresponding to the first analysis image data from the memory 64. As will be described in detail later, the first analysis unit 80A1 derives the jitter degree information by analyzing the first analysis image data 69B2b and the digital phase difference image data 69B1 acquired from the memory 64. The jitter degree information derived by the first analysis unit 80A1 is information indicating the jitter degree included in the first analysis image. The jitter degree information derived by the first analysis unit 80A1 corresponds to the offset of the reference output level, and is expressed, for example, by the number of pixels corresponding to the jitter amount that is not corrected by the optical jitter correction unit 29 and remains in the first analysis image.

[0168] The second analysis unit 80A2 acquires the second analysis image data 69B2c and the digital phase difference image data 69B1 corresponding to the second analysis image data from the memory 64. As will be described in detail later, the second analysis unit 80A2 derives the jitter degree information by analyzing the second analysis image data 69B2c and the digital phase difference image data 69B1 acquired from the memory 64. The jitter degree information derived by the second analysis unit 80A2 is information indicating the jitter degree included in the second analysis image. The jitter degree information derived by the second analysis unit 80A2 corresponds to the offset of the reference output level, and is expressed, for example, by the number of pixels corresponding to the jitter amount that is not corrected by the optical jitter correction unit 29 and remains in the second analysis image.

[0169] The third analysis unit 80A3 acquires the third analysis image data 69B2d and the digital phase difference image data 69B1 corresponding to the third analysis image data from the memory 64. As will be described in detail later, the third analysis unit 80A3 derives the jitter degree information by analyzing the third analysis image data 69B2d and the digital phase difference image data 69B1 acquired from the memory 64. The jitter degree information derived by the third analysis unit 80A3 is information indicating the jitter degree included in the third analysis image. The jitter degree information derived by the third analysis unit 80A3 corresponds to the offset of the reference output level, and is expressed, for example, by the number of pixels corresponding to the jitter amount that is not corrected by the optical jitter correction unit 29 and remains in the third analysis image.

[0170] The second output unit 74 acquires the shake degree information derived by the Nth analysis unit 80A from the Nth analysis unit 80A, and outputs the acquired shake degree information to the correction control unit 76 at the same rate as the output frame rate.

[0171] As an example, Fig.16 As shown, when a still image represented by still image data adopted as practical image data is compared with an instant preview image represented by instant preview image data adopted as practical image data, the overall number of pixels is different. That is, the still image is an image of maximum resolution, that is, full resolution, while the instant preview image is an image with a lower resolution than the still image. Fig.16 In the example shown, in the still image, pixels are not thinned out, whereas in the live preview image, pixels are periodically thinned out in both the row direction and the column direction.

[0172] As an example, Fig.17As shown, the Nth analyzing unit 80A acquires the latest Nth analysis image data from the memory 64. The Nth analysis image data acquired from the memory 64 is the partial image data of the highest resolution (full resolution) in the digital non-phase difference image data 69B2 of one frame. The partial image data refers to the digital non-phase difference image data 69B2 corresponding to the in-focus state area in the AF control target area 61C in the digital non-phase difference image data 69B2 of one frame.

[0173] The Nth analysis unit 80A has a storage area 81. In the storage area 81, the Nth analysis image data acquired from the memory 64 is stored in a FIFO manner, and a pair of Nth analysis image data adjacent in time (hereinafter, also simply referred to as a "pair of Nth analysis image data") is stored. That is, every time the Nth analysis unit 80A acquires the Nth analysis image data from the memory 64, the acquired Nth analysis image data is stored in the storage area 81 in a FIFO manner, and the previous Nth analysis image data and the latest Nth analysis image data in the storage area 81 are updated.

[0174] Whenever the storage content in the storage area 81 is updated, the Nth analyzing unit 80A compares a pair of Nth analysis image data in the storage area 81, and derives the jitter degree information based on the comparison result. That is, the Nth analyzing unit 80A determines the number of pixels corresponding to the jitter amount remaining in the latest Nth analysis image data by comparing the pair of Nth analysis image data in the storage area 81, and determines the determined number of pixels as the jitter degree information.

[0175] As an example, Fig.18 As shown, the second output unit 74 acquires the shake degree information from the Nth analysis unit 80A, and outputs the acquired shake degree information to the correction control unit 76 at the output frame rate and the same rate. On the other hand, the first output unit 72 acquires the practical image data and the digital phase difference image data 69B1 from the practical image data generation unit 78 for each frame, and outputs the acquired practical image data and the digital phase difference image data 69B1 to the signal processing circuit 30 at the output frame rate.

[0176] After the signal processing circuit 30 performs various signal processing on the practical image data, the signal processing circuit 30 outputs the practical image data together with the digital phase difference image data 69B1 to the CPU 35. The CPU 35 performs continuous AF control according to the digital phase difference image data 69B1 input from the signal processing circuit 30. In addition, the CPU 35 performs various processing based on the practical image data input from the signal processing circuit 30. For example, if the practical image data is still image data, the CPU 35 stores the still image data in a predetermined storage device via the external I / F 46, or causes the display 43B to display a still image. In addition, if the practical image data is live preview image data, the CPU 35 causes the display 43B to display a live preview image.

[0177] The correction control unit 76 controls the optical shake correction unit 29 to correct the shake using the vibration data and the reference output level in the reference output table 62C1.

[0178] Then, the correction control unit 76 updates the reference output table 62C1. To update the reference output table 62C1, the correction control unit 76 obtains the temperature data and the jitter degree information. That is, the correction control unit 76 obtains the temperature data from the CPU 35 and the jitter degree information from the second output unit 74. Then, the correction control unit 76 determines whether it is necessary to update the reference output table 62C1 (refer to Fig.11 ), and when it is determined that updating is necessary, the reference output table 62C1 is updated.

[0179] In this case, first, the correction control unit 76 obtains the reference output level corresponding to the temperature represented by the temperature data from the reference output table 62C1. Here, if the reference output level corresponding to the temperature represented by the temperature data does not exist in the reference output table 62C1, the correction control unit 76 calculates the reference output level corresponding to the temperature represented by the temperature data by using a linear interpolation method (for example, an interpolation method). Fig.19 In the example shown, the reference output level when the temperature that does not exist in the reference output table 62C1 (hereinafter referred to as "non-existent temperature") is 23.25°C is calculated by linear interpolation based on the specified temperature and the reference output level for the specified temperature to obtain "506.2". The "current temperature" mentioned here refers to the temperature specified in the reference output table 62C1. Fig.19 In the example shown, 23.0° C. and 23.5° C. are shown as the predetermined temperatures, and “ 506 ” for the reference output level of 23.0° C. and “ 507 ” for the reference output level of 23.5° C. are shown as the reference output levels for the predetermined temperatures.

[0180] Next, the correction control unit 76 calculates the change reference output level based on the reference output level obtained from the reference output table 62C1, the jitter degree information obtained from the second output unit 74, and the temperature represented by the temperature data. The change reference output level is calculated by using, for example, a predetermined calculation formula that uses the reference output level, the jitter degree information, the temperature, and the change reference output level limit value as independent variables and the change reference output level as a dependent variable. Here, the change reference output level limit value refers to a value that limits the change amount of the reference output level to the number of thinning pixels (described later).

[0181] When the number of pixels (hereinafter also referred to as the "difference equivalent pixel number") equivalent to the difference between the reference output level and the calculated changed reference output level (hereinafter also referred to as the "output level difference") exceeds the number of pixels to be thinned out relative to the number of pixels of the instant preview image with respect to the number of pixels of the image with the highest resolution (hereinafter also referred to as the "thinning pixel number"), the possibility of visually perceiving jitter equivalent to the output level difference in images such as the instant preview image even after the jitter is corrected increases, compared to the case where the difference equivalent pixel number is less than the thinning pixel number.

[0182] The output level difference is expressed as an increase or decrease in the angular velocity of the gyro sensor included in the vibration sensor 32, so the displayed image is offset only by the increase or decrease in the angular velocity. The optical system (hereinafter referred to as the "optical system") such as the objective lens 15A and the lens group 15B included in the imaging lens 40 determines how many pixels the image is offset due to the reference output level input from the vibration sensor 32.

[0183] In addition, the above-mentioned "thinning pixel number" is an example of "the degree of reduction of the number of pixels of image data" involved in the technology of the present invention. Moreover, here, "thinning" is illustrated as an example of reducing the number of pixels, but the number of pixels can also be reduced by methods other than thinning, such as merging pixels.

[0184] Next, the correction control unit 76 determines whether or not the reference output table 62C1 needs to be updated based on the calculated change reference output level. If it is determined that the reference output table 62C1 needs to be updated, the correction control unit 76 updates the reference output table 62C1.

[0185] The updating of the reference output table 62C1 refers to the addition of a new reference output level change and the change of an existing reference output level change. The new addition of a reference output level change means that when the reference output level change is not determined for a certain temperature, the reference output level change is newly added to the reference output table 62C1. Fig.11In the example shown, since the change reference output levels in the range of 20.0°C to 22.5°C and the range of 26.5°C to 27.0°C are not determined, when the change reference output levels are calculated for the temperatures in these ranges, the calculated new change reference output levels are added.

[0186] The conventional change of the reference output level is to change the reference output level that has been determined for a certain temperature. Fig.11 In the example shown, the change reference output level is determined to be in the range of 23.0° C. to 26.0° C. The change reference output level is calculated for the temperature in this range, and when the calculated new change reference output level is different from the existing change reference output level in the reference output table 62C1, the existing change reference output level is changed to the calculated new change reference output level.

[0187] exist Fig.11 In the example shown, for example, when the change reference output level calculated by the correction control unit 76 for the temperature of 23.0°C is "510", the change reference output level does not need to be changed. However, when the change reference output level calculated by the correction control unit 76 for the temperature of 23.0°C is a value other than "510", the change reference output level for the temperature of 23.0°C is changed to the latest value calculated by the correction control unit 76. Along with this, the use reference output level for the temperature of 23.0°C is also changed to the same value as the updated change reference output level. Furthermore, when the change reference output level for other temperatures is also determined in the reference output table 62C1, the change reference output level for other temperatures is also changed using the linear interpolation method.

[0188] Furthermore, when the change reference output level is calculated for the non-existent temperature, the change reference output level for the existing temperature in the reference output table 62C1 is also changed. Specifically, the change reference output level in the reference output table 62C1 is changed by linear interpolation based on the non-existent temperature and the change reference output level calculated for the non-existent temperature. For example, Fig.19 As shown, the change reference output level of 23.25°C with no temperature is changed from "510.8" to "512", the change reference output level of 23.0°C is changed from "510" to "511.2", and the change reference output level of 23.5°C is changed from "511" to "512.8". In the same way, the change reference output levels of other existing temperatures are also changed.

[0189] Furthermore, for example, in a state where a change reference output level is not determined in the reference output table 62C1, when the change reference output level for a non-existent temperature is calculated, the change reference output level for the adjacent temperature is calculated by linear interpolation based on the non-existent temperature, the reference output level calculated for the non-existent temperature, the change reference output level calculated for the non-existent temperature, the temperature adjacent to the non-existent temperature in the reference output table 62C1 (hereinafter referred to as the "adjacent temperature"), and the reference output level for the adjacent temperature in the reference output table 62C1, and the calculated change reference output level is added to the reference output table 62C1.

[0190] As an example, Fig. 20 As shown, the correction control unit 76 obtains temperature data from the CPU 35, and obtains the use reference output level corresponding to the temperature represented by the obtained temperature data from the reference output table 62C1. In addition, the correction control unit 76 obtains vibration data from the CPU 35, and calculates the shake correction amount based on the obtained vibration data and the use reference output level. In addition, the correction control unit 76 obtains a position signal from the CPU 35, generates a position indication signal based on the obtained position signal and the shake correction amount, and outputs the generated position indication signal to the CPU 35. The CPU 35 controls the optical shake correction unit 29 according to the position indication signal input from the correction control unit 76. That is, in the optical shake correction unit 29, the position of the anti-vibration lens 15B2 is adjusted to the position indicated by the position indication signal, thereby correcting the shake.

[0191] Next, the operation of the imaging device 10 will be described.

[0192] First, refer to Fig.21 The phase difference pixel processing executed by the processing circuit 62 will be described.

[0193] exist Fig.21 In the phase difference pixel processing shown, first, in step ST10, the control circuit 62D determines whether a vertical synchronization signal is received as a timing control signal. In step ST10, if the vertical synchronization signal is not received, the determination is "No", and the phase difference pixel processing is transferred to step ST18. In step ST10, if the vertical synchronization signal is received, the determination is "Yes", and the phase difference pixel processing is transferred to step ST12.

[0194] In step ST12 , the first readout circuit 62A1 reads out the pseudo phase difference image data 69A1 from the phase difference pixels for all phase difference pixel lines 61A corresponding to one frame, and then the phase difference pixel processing proceeds to step ST14 .

[0195] In step ST14 , the digital processing circuit 62B performs digital signal processing on the analog phase difference image data 69A1 read out by the first readout circuit 62A1 , thereby converting the analog phase difference image data 69A1 into digital phase difference image data 69B1 .

[0196] In the next step ST16 , the control circuit 62D acquires the digital phase difference image data 69B1 from the digital processing circuit 62B, and stores the acquired digital phase difference image data 69B1 in the memory 64 , and then the phase difference pixel processing is transferred to step ST18 .

[0197] In step ST18, the control circuit 62D determines whether a condition for ending the phase difference pixel processing (hereinafter referred to as "phase difference pixel processing ending condition") is satisfied. The phase difference pixel processing ending condition may be, for example, a condition that the receiving device 43A (see Figure 2 ) receives the condition of the instruction to end the phase difference pixel processing. In step ST18, if the phase difference pixel processing end condition is not met, it is determined as "no", and the phase difference pixel processing is transferred to step ST10. In step ST18, if the phase difference pixel processing end condition is met, it is determined as "yes", and the phase difference pixel processing is ended.

[0198] Next, refer to Fig. 22 The non-phase difference pixel processing executed by the processing circuit 62 will be described.

[0199] exist Fig. 22 In the non-phase difference pixel processing shown, first, in step ST30, it is determined whether the readout of the analog phase difference image data 69A1 for one frame by the first readout circuit 62A1 is completed. In step ST30, if the readout of the analog phase difference image data 69A1 for one frame by the first readout circuit 62A1 is not completed, it is determined as "No", and the non-phase difference pixel processing is transferred to step ST38. In step ST30, if the readout of the analog phase difference image data 69A1 for one frame by the first readout circuit 62A1 is completed, it is determined as "Yes", and the non-phase difference pixel processing is transferred to step ST32.

[0200] In step ST32 , the second readout circuit 62A2 reads out the pseudo non-phase difference image data 69A2 from the non-phase difference pixels N for all the non-phase difference pixel lines 61B for one frame, and then the non-phase difference pixel processing proceeds to step ST34 .

[0201] In step ST34 , the digital processing circuit 62B performs digital signal processing on the analog non-phase difference image data 69A2 read out by the second readout circuit 62A2 , thereby converting the analog non-phase difference image data 69A2 into digital non-phase difference image data 69B2 .

[0202] In the next step ST36 , the control circuit 62D acquires the digital non-phase difference image data 69B2 from the digital processing circuit 62B, and stores the acquired digital non-phase difference image data 69B2 in the memory 64 , and then the non-phase difference pixel processing is transferred to step ST38 .

[0203] In step ST38, the control circuit 62D determines whether the condition for ending the non-phase difference pixel processing (hereinafter referred to as "non-phase difference pixel processing ending condition") is satisfied. As the non-phase difference pixel processing ending condition, for example, the receiving device 43A (refer to Figure 2 ) receives the condition of the instruction to end the non-phase difference pixel processing. In step ST38, if the non-phase difference pixel processing end condition is not met, it is determined as "no", and the non-phase difference pixel processing is transferred to step ST30. In step ST38, if the non-phase difference pixel processing end condition is met, it is determined as "yes", and the non-phase difference pixel processing is ended.

[0204] Next, refer to Fig.23 The practical image data generating unit process executed by the processing circuit 62 will be described.

[0205] exist Fig.23 In the practical image data generation unit process shown, first, in step ST50, the practical image data generation unit 78 determines whether the digital non-phase difference image data 69B2 in the memory 64 is updated. In step ST50, if the digital non-phase difference image data 69B2 in the memory 64 is not updated, it is determined as "No", and the practical image data generation unit process transfers to step ST60. In step ST50, if the digital non-phase difference image data 69B2 in the memory 64 is updated, it is determined as "Yes", and the practical image data generation unit process transfers to step ST52.

[0206] In step ST52 , the practical image data generating unit 78 acquires the signal processing image data and the digital phase difference image data 69B1 from the memory 64 , and then the practical image data generating unit process shifts to step ST54 .

[0207] In step ST54 , the practical image data generating unit 78 generates practical image data by performing signal processing on the signal processing image data acquired in step ST52 , and then the processing of the practical image data generating unit shifts to step ST56 .

[0208] In step ST56, the first output unit 72 determines whether the timing to output the practical image data (the practical image data output timing) has arrived. The practical image data output timing is a timing specified by the output frame rate. In step ST56, if the practical image data output timing has not arrived, the determination is "No", and the determination of step ST56 is performed again. In step ST56, if the practical image data output timing has arrived, the determination is "Yes", and the processing of the practical image data generation unit is transferred to step ST58.

[0209] In step ST58 , the first output unit 72 acquires the practical image data and the digital phase difference image data 69B1 from the practical image data generating unit 78 , and outputs the acquired practical image data and the digital phase difference image data 69B1 to the signal processing circuit 30 .

[0210] In the signal processing circuit 30, various signal processing is performed on the practical image data input from the imaging element 44, and the practical image data after various signal processing is output to the CPU 35. The CPU 35 displays the image represented by the practical image data on the display 43B, or stores the practical image data in a predetermined storage device via the external I / F 46. In addition, the digital phase difference image data 69B1 is also output from the signal processing circuit 30 to the CPU 35, and the digital phase difference image data 69B1 is used by the CPU 35 for AF control.

[0211] In the next step ST60, the control circuit 62D determines whether the condition for terminating the practical image data generating unit processing (hereinafter referred to as "the practical image data generating unit processing terminating condition") is satisfied. The practical image data generating unit processing terminating condition may be, for example, a condition that the receiving device 43A (see Figure 2 ) receives the condition of an instruction to end the practical image data generating unit processing. In step ST60, if the practical image data generating unit processing end condition is not satisfied, it is determined as "No", and the practical image data generating unit processing is transferred to step ST50. In step ST60, if the practical image data generating unit processing end condition is satisfied, it is determined as "Yes", and the practical image data generating unit processing ends.

[0212] Next, refer to Fig.24 The Nth analysis image data acquisition process executed by the processing circuit 62 will be described.

[0213] exist Fig.24In the Nth analysis image data acquisition process shown, first, in step ST80, the control circuit 62D determines whether the digital non-phase difference image data 69B2 in the memory 64 is updated. In step ST80, if the digital non-phase difference image data 69B2 in the memory 64 is not updated, it is determined as "No", and the Nth analysis image data acquisition process is transferred to step ST88. In step ST80, if the digital non-phase difference image data 69B2 in the memory 64 is updated, it is determined as "Yes", and the Nth analysis image data acquisition process is transferred to step ST82.

[0214] In step ST82 , the control circuit 62D acquires the digital phase difference image data 69B1 from the memory 64 , and determines the in-focus state area based on the acquired digital phase difference image data 69B1 , and then the Nth analysis image data acquisition process moves to step ST84 .

[0215] In step ST84, the control circuit 62D acquires the digital phase difference image data 69B1 acquired in step ST82 and the digital non-phase difference image data 69B2 of the same frame from the memory 64. Furthermore, the control circuit 62D acquires the digital non-phase difference image data 69B2 corresponding to the focus state area determined in step ST82 from the digital non-phase difference image data 69B2 acquired from the memory 64 as the Nth analysis image data, and then the Nth analysis image data acquisition process is transferred to step ST86.

[0216] In step ST86 , the control circuit 62D stores the N-th analytical image data acquired in step ST84 in the memory 64 , and then the N-th analytical image data acquisition process shifts to step ST88 .

[0217] In step ST88, the control circuit 62D determines whether a condition for terminating the Nth analysis image data acquisition process (hereinafter referred to as "the Nth analysis image data acquisition process termination condition") is satisfied. The Nth analysis image data acquisition process termination condition may be, for example, a condition that the receiving device 43A (see Figure 2 ) receives the condition of an instruction to end the Nth analysis image data acquisition process. In step ST88, if the Nth analysis image data acquisition process end condition is not satisfied, it is determined as "No", and the Nth analysis image data acquisition process is transferred to step ST80. In step ST88, if the Nth analysis image data acquisition process end condition is satisfied, it is determined as "Yes", and the Nth analysis image data acquisition process is ended.

[0218] Next, refer to Fig.25 The Nth analysis unit process executed by the processing circuit 62 will be described.

[0219] exist Fig.25 In the Nth analysis unit process shown in FIG. 1 , first, in step ST90, the Nth analysis unit 80A executes Fig.24 In step ST86 shown in FIG. 1 , it is determined whether the N-th analysis image data in memory 64 is updated. In step ST90, if the N-th analysis image data in memory 64 is not updated, it is determined as "No", and the N-th analysis unit process shifts to step ST104. In step ST90, if the N-th analysis image data in memory 64 is updated, it is determined as "Yes", and the N-th analysis unit process shifts to step ST92.

[0220] In step ST92 , the Nth analyzing unit 80A acquires the Nth analysis image data from the memory 64 , and stores the acquired Nth analysis image data in the storage area 81 in a FIFO manner. Then, the Nth analyzing unit process shifts to step ST94 .

[0221] In step ST94, the Nth analyzing unit 80A determines whether a pair of Nth analysis image data is stored in the storage area 81. In step ST94, if a pair of Nth analysis image data is not stored in the storage area 81, the determination is "No", and the Nth analyzing unit process shifts to step ST104. In step ST94, if a pair of Nth analysis image data is stored in the storage area 81, the determination is "Yes", and the Nth analyzing unit process shifts to step ST96.

[0222] In step ST96 , the N-th analyzing unit 80A compares a pair of N-th analysis image data, and then the N-th analyzing unit process shifts to step ST98 .

[0223] In step ST98 , the Nth analyzing unit 80A derives the shaking degree information based on the comparison result of the pair of Nth analysis image data in step ST96 , and then the Nth analyzing unit process shifts to step ST100 .

[0224] In step ST100, the second output unit 74 determines whether the timing for outputting the jitter degree information (jitter degree information output timing) has arrived. The jitter degree information output timing is a timing specified at the same rate as the output frame rate. In step ST100, if the jitter degree information output timing has not arrived, the determination of "No" is performed again in step ST100. In step ST100, if the jitter degree information output timing has arrived, the determination of "Yes" is performed, and the processing of the Nth analysis unit is transferred to step ST102.

[0225] In step ST102 , the second output unit 74 acquires the shake degree information derived in step ST98 , and outputs the acquired shake degree information to the correction control unit 76 , and then the Nth analysis unit proceeds to step ST104 .

[0226] In step ST104, the control circuit 62D determines whether a condition for terminating the Nth analysis unit process (hereinafter referred to as "Nth analysis unit process termination condition") is satisfied. The Nth analysis unit image data acquisition process termination condition may be, for example, a condition that the receiving device 43A (see Figure 2 ) receives the condition of the instruction to end the Nth analysis unit processing. In step ST104, if the end condition of the Nth analysis unit processing is not met, it is determined as "No", and the Nth analysis unit processing is transferred to step ST90. In step ST104, if the end condition of the Nth analysis unit processing is met, it is determined as "Yes", and the Nth analysis unit processing is ended.

[0227] Next, refer to Fig.26 The correction control unit processing executed by the processing circuit 62 will be described. Note that, for the sake of convenience, the description here is based on the premise that the change reference output level in the reference output table 62C1 is not determined (does not exist).

[0228] exist Fig.26 In the correction control unit process shown, first, in step ST120, the correction control unit 76 determines whether the jitter degree information output from the second output unit 74 is received. In step ST120, if the jitter degree information output from the second output unit 74 is not received, the correction control unit process is transferred to step ST136. In step ST120, if the jitter degree information output from the second output unit 74 is received, the correction control unit process is transferred to step ST122.

[0229] In step ST122 , the correction control unit 76 acquires the latest temperature data measured by the temperature sensor 34 from the CPU 35 , and then the correction control unit process shifts to step ST123 .

[0230] In step ST123 , the correction control unit 76 determines the reference output level of the temperature represented by the temperature data acquired in step ST122 , ie, the current temperature, from the reference output table 62C1 , and then the correction control unit process shifts to step ST124 .

[0231] In step ST124 , the correction control unit 76 calculates the amount of change in the reference output level determined in step ST123 using the jitter degree information received in step ST120 , and then the correction control unit process shifts to step ST126 .

[0232] In step ST126, the correction control unit 76 changes the reference output level according to the change amount calculated in step ST124, thereby generating a changed reference output level. The reference output table 62C1 is thereby updated as described above. The changed reference output level generated in step ST126 is adopted as the used reference output level.

[0233] In the next step ST126 , the correction control unit 76 acquires the use reference output level corresponding to the current temperature from the reference output table 62C1 updated in step ST126 , and then the correction control unit process shifts to step ST130 .

[0234] In step ST130 , the correction control unit 76 acquires the vibration data and the position signal from the CPU 35 , and then the correction control unit process shifts to step ST132 .

[0235] In step ST132 , the correction control unit 76 generates a position indication signal using the use reference output level acquired in step ST128 , the vibration data acquired in step ST130 , and the position signal acquired in step ST130 , and then the correction control unit processing transfers to step ST134 .

[0236] In step ST134 , the correction control unit 76 outputs the position instruction signal generated in step ST132 to the CPU 35 , and then the correction control unit process shifts to step ST136 .

[0237] In step ST136, the control circuit 62D determines whether a condition for ending the correction control unit process (hereinafter referred to as "correction control unit process ending condition") is satisfied. The correction control unit process ending condition may be, for example, a correction control unit process ending condition caused by the receiving device 43A (see Figure 2 ) receives the condition of the instruction to end the correction control unit processing. In step ST136, if the correction control unit processing end condition is not met, it is determined as "no", and the correction control unit processing is transferred to step ST120. In step ST136, if the correction control unit processing end condition is met, it is determined as "yes", and the correction control unit processing ends.

[0238] Whenever the reference output level is changed in step ST126 included in the correction control unit process, the changed reference output level, i.e., the changed reference output level, is reflected in the correction of the jitter by the optical jitter correction unit 29 by executing the processes of steps ST128 to ST134. Thus, the changed reference output level, i.e., the changed reference output level, is reflected in the correction of the jitter by the optical jitter correction unit 29 for each frame determined periodically. Fig.25In step ST92 of the N-th analyzing section process shown in FIG. 1 , the N-th analyzing section 80A acquires and uses the N-th analyzing image data obtained after the blurring correction.

[0239] That is, the first analysis image data 69B2b, the second analysis image data 69B2c, and the third analysis image data 69B2d are image data of each frame periodically determined, and for each of these image data, the changed reference output level, i.e., the changed reference output level, is reflected in the correction of the jitter by the optical jitter correction section 29. Furthermore, the changed reference output level is reflected in the correction of the jitter by the optical jitter correction section 29 for each frame, and the first analysis image data 69B2b obtained after the jitter correction is used by the first analysis section 80A1 to derive the jitter degree information. Furthermore, the changed reference output level, i.e., the changed reference output level, is reflected in the correction of the jitter by the optical jitter correction section 29 for each frame, and the second analysis image data 69B2c obtained after the jitter correction is used by the second analysis section 80A2 to derive the jitter degree information. Furthermore, the changed reference output level, i.e., the changed reference output level, is reflected in the jitter correction by the optical jitter correction unit 29 for each frame, and the third analysis image data 69B2d obtained after the jitter correction is used by the third analysis unit 80A3 to derive the jitter degree information. In addition, here, the description is given on the premise that the changed reference output level in the reference output table 62C1 is not determined, but even if the changed reference output level is determined, it is sufficient to reflect the updated changed reference output level in the jitter correction by the optical jitter correction unit 29.

[0240] In this way, by reflecting the changed reference output level in the correction of the jitter by the optical jitter correction section 29, the first analysis section 80A1 derives the jitter degree information from the first analysis image data 69B2b for which the jitter is corrected by the optical jitter correction section 29. Furthermore, the second analysis section 80A2 derives the jitter degree information from the second analysis image data 69B2c for which the jitter is corrected by the optical jitter correction section 29. Furthermore, the third analysis section 80A3 derives the jitter degree information from the third analysis image data 69B2d for which the jitter is corrected by the optical jitter correction section 29.

[0241] As described above, in the camera device 10, the first output unit 72 outputs the practical image data at the output frame rate. Furthermore, the Nth analysis unit 80A derives the jitter degree information from the Nth analysis image data, and the second output unit 74 outputs the jitter degree information at the same rate as the output frame rate. That is, there is no difference between the period of outputting the practical image data and the period of outputting the jitter degree information. Therefore, compared with the case where the jitter degree information is output at a time interval longer than the period specified by the output frame rate, the real-time performance of the output of the jitter degree information relative to the output of the practical image data can be improved.

[0242] Furthermore, the highest resolution is used as the resolution of the Nth analysis image data in imaging device 10. Therefore, compared with a case where the resolution of the Nth analysis image data analyzed by Nth analysis unit 80A is lower than the highest resolution, it is possible to derive high-precision blur information.

[0243] Furthermore, in the imaging device 10, the Nth analysis image data is partial image data representing a portion of the digital non-phase difference image data 69B2. Therefore, compared with the case where the Nth analysis unit 80A analyzes image data of all pixels of the digital non-phase difference image data 69B2, the processing load for analysis in the Nth analysis unit 80A can be reduced.

[0244] Furthermore, in the imaging device 10, the Nth analysis image data is the digital non-phase difference image data 69B2 corresponding to the in-focus area of ​​the digital non-phase difference image data 69B2. Therefore, compared with the case where the digital non-phase difference image data 69B2 corresponding to the image side area in the non-focus state is analyzed by the Nth analysis unit 80A, it is possible to derive high-precision shake degree information.

[0245] In the imaging device 10, the focus state area is determined by analyzing the digital phase difference image data 69B1 obtained in the same frame as the digital non-phase difference image data 69B2 by the control circuit 62D. Therefore, compared with the case where the shake degree information is derived by analyzing the digital non-phase difference image data 69B2, the shake degree information can be derived at a higher speed.

[0246] In the imaging device 10, the processing unit 70 generates useful image data based on time series data obtained by imaging at the imaging frame rate and derives the shake degree information. Therefore, the generation of useful image data and the derivation of shake degree information can be performed in parallel.

[0247] Furthermore, in the imaging device 10, the first output unit 72 outputs the practical image data, and the N-th analysis image data is used to derive the shake degree information by the N-th analysis unit 80A. Therefore, compared with the case where the first output unit 72 is not used and only the N-th analysis unit 80A performs output of the practical image data and derives the shake degree information, the real-time performance of the deriving of the shake degree information based on the output of the practical image data by the first output unit 72 can be improved.

[0248] In the imaging device 10, the Nth analyzing unit 80A derives the shake degree information for each frame periodically determined (each frame at a period of 60 fps) based on the Nth analysis image data. Therefore, more shake degree information can be derived compared to the case where the shake degree information is derived only for a single frame.

[0249] Furthermore, in the imaging device 10, the time interval between frames of the time series data is the same as the period specified by the imaging frame rate (240fps). Furthermore, the practical image data is outputted by the first output unit 72 at the output frame rate (60fps). Furthermore, the time interval between frames of the first analytical image data is the same as the period specified by the output frame rate, the time interval between frames of the second analytical image data is also the same as the period specified by the output frame rate, and the time interval between frames of the third analytical image data is also the same as the period specified by the output frame rate. Therefore, compared with the case where the time interval between frames of the first analytical image data is a time interval longer than the period specified by the output frame rate, the frame rate required as the output frame rate of the practical image data can be ensured, and the acquisition frequency of the Nth analytical image data can be increased.

[0250] Furthermore, in the imaging device 10, the reference output level is changed according to the shake degree information by the correction control unit 76 built into the imaging element 44. Furthermore, the correction control unit 76 controls the optical shake correction unit 29 to correct the shake according to the vibration data and the changed reference output level. Therefore, the accuracy of the shake correction can be improved compared to the case where the reference output level is fixed.

[0251] Furthermore, in the imaging device 10, the change amount of the reference output level is limited within a range whose upper limit is the number of thinned pixels. This makes it possible to suppress the change of the reference output level from being visually perceived, compared with a case where the reference output level is changed without limit.

[0252] Furthermore, in the imaging device 10, each of the first analysis image data 69B2b, the second analysis image data 69B2c, and the third analysis image data 69B2d is image data obtained by reflecting the changed reference output level in the correction of the shake by the optical shake correction section 29 every time the reference output level is changed. Therefore, compared with the case where the shake degree information is derived by analyzing the image data obtained by reflecting the fixed reference output level in the correction of the shake by the optical shake correction section 29, it is possible to derive the shake degree information with high accuracy.

[0253] Furthermore, in the image pickup device 10, the optical image stabilization unit 29 optically corrects the image stabilization. Therefore, the load for image processing can be reduced compared to the case of using EIS.

[0254] Furthermore, in the camera device 10, the imaging element 44 is an imaging element in which the photoelectric conversion element 61, the processing circuit 62, and the memory 64 are integrated into a single chip. Thus, compared with an imaging element in which the photoelectric conversion element 61, the processing circuit 62, and the memory 64 are not integrated into a single chip, the portability of the imaging element 44 is improved. Furthermore, compared with an imaging element in which the photoelectric conversion element 61, the processing circuit 62, and the memory 64 are not integrated into a single chip, the degree of freedom in design can be improved. Furthermore, compared with an imaging element in which the photoelectric conversion element 61, the processing circuit 62, and the memory 64 are not integrated into a single chip, it is also possible to contribute to the miniaturization of the camera device body 12.

[0255] Furthermore, in the imaging device 10, a stacked imaging element in which a memory 64 is stacked on a photoelectric conversion element 61 is used as the imaging element 44. As a result, since the wiring connecting the photoelectric conversion element 42 and the memory 112 can be shortened, the wiring delay can be reduced. As a result, the transmission speed of the image data 69 from the photoelectric conversion element 61 to the memory 64 can be increased compared to the case where the photoelectric conversion element 61 and the memory 64 are not stacked. The increase in the transmission speed also contributes to the speed-up of the processing in the entire processing circuit 62. In addition, compared to the case where the photoelectric conversion element 61 and the memory 64 are not stacked, the degree of freedom of design can also be increased. In addition, compared to the case where the photoelectric conversion element 61 and the memory 64 are not stacked, it can also contribute to the miniaturization of the imaging device body 12.

[0256] In the imaging device 10, a live preview image based on the practical image data is displayed on the display 43B. Furthermore, the practical image data is stored in a predetermined storage device. Therefore, the versatility of the practical image data can be improved.

[0257] In the first embodiment, the imaging element 44 is exemplified as an imaging element in which the photoelectric conversion element 61, the processing circuit 62, and the memory 64 are integrated into a single chip, but the technology of the present invention is not limited to this. For example, at least the photoelectric conversion element 61 and the memory 64 among the photoelectric conversion element 61, the processing circuit 62, and the memory 64 can be integrated into a single chip.

[0258] Furthermore, in the first embodiment described above, the highest resolution is used as the resolution of the Nth analysis image data, but the technology of the present invention is not limited to this. For example, the resolution of the Nth analysis image data may be a resolution higher than the resolution of the practical image data output by the first output unit 72. In this case, compared with the case where the resolution of the Nth analysis image data is a resolution lower than the resolution of the practical image data output by the first output unit 72, it is possible to derive high-precision shake degree information.

[0259] Furthermore, in the first embodiment, the example of maintaining the focus state by the continuous AF mode is given, but the technology of the present invention is also applicable in the normal AF mode. Furthermore, the shake degree information can also be derived by using the digital non-phase difference image data 69B2 obtained in the non-focus state.

[0260] Furthermore, in the above-mentioned first embodiment, the image data respectively input into the practical image data generating unit 78 and the Nth analysis unit 80A are image data of different frames, but the technology of the present invention is not limited to this, and the same frame of image data can also be used in the practical image data generating unit 78 and the Nth analysis unit 80A.

[0261] Furthermore, in the first embodiment, an example is given of outputting the jitter degree information at the same rate as the output frame rate, but the jitter degree information may be output at a rate higher than the output frame rate, that is, at a period shorter than the period specified by the output frame rate.

[0262] Furthermore, in the first embodiment, each of the time interval between frames of the signal processing image data and the time interval between frames of the Nth analysis image data is the same as the period specified by the output frame rate, but the technology of the present invention is not limited thereto. For example, each of the time interval between frames of the signal processing image data and the time interval between frames of the Nth analysis image data may be a time interval that is longer than the period specified by the imaging frame rate and shorter than the period specified by the output frame rate.

[0263] Furthermore, in the above-mentioned embodiment, an example is given in which the signal processing image data is skipped by three frames and input to the practical image data generating unit 78, and the image data of three frames between the signal processing image data is input to the Nth analyzing unit 80A as the Nth analysis image data, but the technology of the present invention is not limited to this. For example, the acquisition of the signal processing image data by the practical image data generating unit 78 and the acquisition of the Nth analysis image data by the Nth analyzing unit 80A may be performed alternately every one frame.

[0264] Furthermore, in the first embodiment described above, three analysis image data, namely, the first analysis image data 69B2b, the second analysis image data 69B2c, and the third analysis image data 69B2d, are exemplified, but the technology of the present invention is not limited thereto. For example, the number of analysis image data may be changed according to the difference between the imaging frame rate and the output frame rate. For example, when the imaging frame rate is 300 fps and the output frame rate is 60 fps, four analysis image data (first to fourth analysis image data) may be provided.

[0265] In addition, in the first embodiment, an example of determining the focus state area based on the digital phase difference image data 69B1 is given for explanation, but the technology of the present invention is not limited to this. For example, when the imaging device 10 is equipped with a contrast AF function, the focus state area determination information for determining the position of the focus state area is sent from the CPU 35 to the control circuit 62D, and the control circuit 62D determines the focus state area based on the focus state area determination information.

[0266] Furthermore, in the first embodiment described above, a thinned-out image obtained by thinning out pixels is exemplified as a live preview image, but the method of reducing the amount of data is not limited to this, and the amount of data may be reduced by merging pixels through image processing and compressing. Furthermore, in the first embodiment described above, an example of thinning out pixels from full-resolution image data is given as an example, but image data may also be obtained by thinning out the photosensitive pixels of the photoelectric conversion element 61.

[0267] Furthermore, in the first embodiment, OIS is exemplified as a method for correcting shake, but the technology of the present invention is not limited thereto. For example, BIS or EIS may be used instead of OIS, or two or more correction methods among OIS, BIS and EIS may be combined.

[0268] [Second embodiment of the technology of the present invention]

[0269] In the first embodiment described above, the correction control unit 76 is built into the imaging element 44. However, in the second embodiment of the present invention, the correction control unit 76 is not built into the imaging element. In addition, the same reference numerals are attached to the same components as those in the first embodiment, and the description thereof is omitted.

[0270] As an example, Fig. 27 As shown, the imaging device 100 of the second embodiment of the present invention is different from the imaging device 10 of the first embodiment in that it has an imaging element 440 instead of the imaging element 44. In addition, the imaging device 100 is different from the imaging device 10 in that the CPU 35, which is a subsequent circuit of the imaging element 440, operates as a correction control unit 35A. In addition, the imaging device 100 is different from the imaging device 10 in that the storage device 42 stores a reference output table 62C1.

[0271] The program P is stored in the ROM 36, and the CPU 35 reads the program P from the ROM 36, expands it to the RAM 37, and executes the expanded program P, thereby operating as the correction control unit 35A. The correction control unit 35A has the same function as the correction control unit 76 described in the first embodiment. Therefore, the correction control unit 35A updates the reference output table 62C1 stored in the storage device 42, and obtains the use reference output level from the reference output table 62C1.

[0272] As an example, Fig.28 As shown, the imaging element 440 is different from the imaging element 44 in that it includes a control circuit 462D instead of the control circuit 62D. The control circuit 462D is different from the control circuit 62D in that it does not include the correction control unit 76. The second output unit 74 outputs the shake degree information to the correction control unit 35A of the CPU 35.

[0273] Thus, in the imaging device 100, the correction control unit 76 is not provided in the imaging element 440, and the CPU 35, which is a subsequent circuit of the imaging element 440, operates as the correction control unit 35A. Therefore, the processing load required for the imaging element 440 can be reduced compared with the imaging element 44 of the first embodiment.

[0274] Furthermore, in the imaging device 100, the correction control unit 35A changes the reference output level based on the shake degree information, as in the first embodiment described above. Furthermore, the correction control unit 35A controls the optical shake correction unit 29 to correct the shake based on the vibration data and the changed reference output level. Therefore, the accuracy of shake correction can be improved compared to the case where the reference output level is fixed.

[0275] Also, in the imaging device 100, the change amount of the reference output level is limited within a range whose upper limit is the number of thinned pixels, as in the first embodiment, thereby preventing the change of the reference output level from being visually perceived, compared with the case where the reference output level is changed without limit.

[0276] Furthermore, in the imaging device 100, as in the first embodiment, each of the first analysis image data 69B2b, the second analysis image data 69B2c, and the third analysis image data 69B2d is image data obtained by reflecting the changed reference output level in the correction of the shake by the optical shake correction section 29 every time the reference output level is changed. Therefore, compared with the case where the shake degree information is derived by analyzing the image data obtained by reflecting the fixed reference output level in the correction of the shake by the optical shake correction section 29, it is possible to derive the shake degree information with high accuracy.

[0277] Furthermore, in each of the above-mentioned embodiments, an example of a method in which the processing circuit 62 is implemented by a device including an ASIC and an FPGA is given for explanation, but the technology of the present invention is not limited to this. For example, the above-mentioned phase difference pixel processing, non-phase difference pixel processing, practical image data generation unit processing, Nth analysis image data acquisition processing, Nth analysis unit processing and correction control unit processing (hereinafter, when there is no need to distinguish and explain these, referred to as "processing within the camera device") can be implemented by the software structure of the computer. In addition, processing within the camera device is an example of "specific processing" involved in the technology of the present invention.

[0278] In this case, for example, Fig.29 As shown, various programs for causing the computer 852 built in the imaging element 540 to execute the above-mentioned in-camera processing are stored in the storage medium 900.

[0279] The various programs refer to a phase difference pixel processing program 902, a non-phase difference pixel processing program 904, a practical image data generation unit processing program 906, an N-th analysis image data acquisition processing program 908, an N-th analysis unit processing program 910, and a correction control unit processing program 912. The phase difference pixel processing program 902 is a program for causing the computer 852 to perform the above-mentioned phase difference pixel processing. The various programs are an example of a "program" involved in the technology of the present invention. The non-phase difference pixel processing program 904 is a program for causing the computer 852 to perform the above-mentioned non-phase difference pixel processing. The practical image data generation unit processing program 906 is a program for causing the computer 852 to perform the above-mentioned practical image data generation unit processing. The N-th analysis image data acquisition processing program 908 is a program for causing the computer 852 to perform the above-mentioned N-th analysis image data acquisition processing. The N-th analysis unit processing program 910 is a program for causing the computer 852 to perform the above-mentioned N-th analysis unit processing. The correction control unit processing program 912 is a program for causing the computer 852 to perform the above-mentioned correction control unit processing.

[0280] As an example, Fig.29 As shown, the computer 852 is an example of a "computer" involved in the technology of the present invention, and includes a CPU 852A, a ROM 852B, and a RAM 852C. Various programs stored in the storage medium 900 are installed in the computer 852. The CPU 852A executes processing in the imaging device according to various programs.

[0281] Here, one CPU is illustrated as CPU 852A, but the technology of the present invention is not limited thereto, and a GPU may be used instead of CPU 852A, or multiple CPUs may be used. In addition, storage medium 900 is a non-temporary storage medium. As an example of storage medium 900, any portable storage medium such as SSD or USB memory may be cited.

[0282] exist Fig.29 In the example shown, various programs are stored in the storage medium 900, but the technology of the present invention is not limited to this. For example, various programs may be pre-stored in ROM 852B, CPU 852A reads various programs from ROM 852B, expands them to RAM 852C, and executes the expanded various programs.

[0283] Furthermore, various programs may be stored in a storage unit such as another computer or a server device connected to the computer 852 via a communication network (not shown), and downloaded to the computer 852 in response to a request from the imaging device 10. In this case, the downloaded various programs are executed by the CPU 852A of the computer 852.

[0284] Furthermore, the computer 852 may be provided outside the imaging element 44. In this case, the computer 852 only needs to control the processing circuit 62 according to the program.

[0285] As hardware resources for executing the processing in the imaging device, various processors shown below can be used. As a processor, for example, a general-purpose processor, i.e., a CPU, can be cited, which functions as a hardware resource for executing the processing in the imaging device by executing software, i.e., a program, as described above. Furthermore, as a processor, for example, a dedicated circuit as a processor can be cited, and the processor has a circuit structure specially designed for executing specific processing, such as an FPGA, a PLD, or an ASIC.

[0286] The hardware resource for executing the processing in the camera device may be composed of one of these various processors, or may be composed of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Furthermore, the hardware resource for executing the processing in the camera device may be a single processor.

[0287] As an example of a system composed of one processor, there is a first method in which a processor is composed of a combination of one or more CPUs and software, such as a client or server computer, and the processor functions as a hardware resource for executing the processing in the camera device. There is a second method in which a processor is used, such as a SoC, to realize the functions of the entire system including multiple hardware resources for executing the processing in the camera device by using one IC chip. In this way, the processing in the camera device is realized by using one or more of the above-mentioned various processors as hardware resources.

[0288] Furthermore, as the hardware configuration of these various processors, more specifically, a circuit combining circuit elements such as semiconductor elements can be used.

[0289] Furthermore, in the above-mentioned embodiments, a lens-interchangeable camera is exemplified as the imaging device 10, but the technology of the present invention is not limited thereto. Fig.30 As an example, Fig.30 The smart device 950 shown is an example of a camera device involved in the technology of the present invention. The imaging element 44 (440) is mounted on the smart device 950. Even the smart device 950 constructed in this way can obtain the same function and effect as the camera device 10, 100 described in the above embodiments. In addition, the technology of the present invention is not limited to the smart device 950, but can also be applied to a personal computer or a wearable terminal device.

[0290] Furthermore, in each of the above embodiments, the display 43B is exemplified, but the technology of the present invention is not limited thereto. For example, a separate display attached to the imaging device body 12 may be used as the "display device" involved in the technology of the present invention.

[0291] Furthermore, the above-mentioned various processes are merely examples, and therefore, it is of course possible to delete unnecessary steps, add new steps, or switch the processing order without departing from the scope of the subject matter.

[0292] The recorded contents and illustrated contents shown above are detailed descriptions of the parts involved in the technology of the present invention, and are only an example of the technology of the present invention. For example, the descriptions related to the above-mentioned structure, function, action and effect are descriptions related to an example of the structure, function, action and effect of the parts involved in the technology of the present invention. Therefore, without departing from the scope of the main purpose of the technology of the present invention, it is of course possible to delete unnecessary parts of the recorded contents and illustrated contents shown above, or to add new elements, or to replace them. In addition, in order to avoid complication and to make it easy to understand the parts involved in the technology of the present invention, in the recorded contents and illustrated contents shown above, descriptions related to technical common sense, etc. that do not need to be specifically explained in terms of the technology that can implement the present invention are omitted.

[0293] In this specification, "A and / or B" has the same meaning as "at least one of A and B". That is, "A and / or B" may mean only A, only B, or a combination of A and B. Furthermore, in this specification, when "and / or" is added to express three or more items, the same concept as "A and / or B" may also be applied.

[0294] All documents, patent applications, and technical standards described in this specification are incorporated herein by reference to the same extent as if each document, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[0295] The following additional notes are further disclosed regarding the above-mentioned embodiment.

[0296] (Note 1)

[0297] An imaging element having:

[0298] Memory; and

[0299] processor,

[0300] The memory stores image data obtained by shooting at a first frame rate.

[0301] The processor performs the following processing:

[0302] Process the image data.

[0303] and outputting the processed image data at a frame rate that is lower than the first frame rate, i.e., a second frame rate,

[0304] By using the image data stored in the memory, shake degree information indicating the degree of shake included in the image represented by the image data is derived, and the derived shake degree information is output at a rate higher than the second frame rate.

Claims

1. An imaging element, comprising: processor; and Memory, The memory stores image data obtained by shooting at a first frame rate. the processor processes the image data and outputs the processed image data at a second frame rate, the processor derives, by using the image data, shaking degree information indicating a shaking degree included in an image represented by the image data, outputting the derived jitter degree information at a frame rate higher than the second frame rate, The first frame rate is a frame rate higher than the second frame rate.

2. The imaging element according to claim 1, wherein The image data used in the processor has a higher resolution than the image data output by the processor.

3. The imaging element according to claim 1, wherein The resolution of the image data used in the processor is a maximum resolution.

4. The imaging element according to any one of claims 1 to 3, wherein The image data used in the processor is partial image data representing a partial image.

5. The imaging element according to claim 4, comprising a photosensitive pixel group, The local image data is image data obtained from a region in the photosensitive pixel group that is in a focused state.

6. The imaging element according to claim 5, wherein: The area in the focused state is an area in which the focused state is continuously maintained in an operation mode in which the focused state is continuously maintained.

7. The imaging element according to claim 5 or 6, wherein: The photosensitive pixel group comprises a plurality of phase difference pixels, The processor determines, based on pixel data of at least a portion of the phase difference pixels among the plurality of phase difference pixels, a region in the photosensitive pixel group that is in the focus state.

8. The imaging element according to any one of claims 1 to 3, 5 and 6, wherein: The image data is time-series data obtained by capturing images at the first frame rate.

9. The imaging element according to claim 8, wherein: The processor outputs the first frame data related to a portion of the frames in the time series data, and the second frame data related to the remaining frames is used for the processor to derive the jitter degree information.

10. The imaging element according to claim 9, wherein The second frame data is frame data representing a plurality of frames, The processor derives the jitter degree information for each frame determined periodically based on the second frame data.

11. The imaging element according to claim 10, wherein The periodically determined frames are frames determined at a time interval that is longer than a period defined by the first frame rate and is equal to or less than a period defined by the second frame rate.

12. The imaging element according to any one of claims 1 to 3, 5, 6, 9 to 11, wherein: At least the photoelectric conversion element and the memory are integrated into a single chip.

13. A camera device, comprising: The imaging element according to any one of claims 1 to 12; a vibration sensor to detect the applied vibration; and A correction device corrects the shake based on the vibration data output from the vibration sensor and the shake degree information output from the processor.

14. The imaging device according to claim 13, wherein: The processor derives the shaking degree information based on the image data after the shaking is corrected by the correction device, The camera device further includes a post-stage circuit, which receives the shaking degree information output from the processor. The subsequent circuit controls the correction device to correct the jitter using the vibration data and the reference output level of the vibration sensor. The subsequent stage circuit changes the reference output level according to the jitter degree information output from the processor.

15. The imaging device according to claim 13, wherein: The processor derives the shaking degree information based on the image data after the shaking is corrected by the correction device, The imaging element further includes a correction control circuit that uses the vibration data and a reference output level of the vibration sensor to control the correction device to correct the jitter. The correction control circuit changes the reference output level according to the jitter degree information output from the processor.

16. The imaging device according to claim 14 or 15, wherein: The amount of change in the reference output level is limited within a range that does not exceed the degree of reduction in the number of pixels of the image data used in the processor.

17. The imaging device according to claim 14 or 15, wherein: The image data is time-series data obtained by shooting at the first frame rate. The processor outputs first frame data related to a portion of frames in the time series data, The second frame data related to the remaining frames in the time series data is frame data representing a plurality of frames, and is used for the processor to derive the jitter degree information for each frame determined periodically. The plurality of frames are frames in which each time the reference output level is changed, the changed reference output level is reflected in the jitter correction by the correction device.

18. The imaging device according to any one of claims 13 to 15, wherein: The correction device corrects the jitter optically.

19. The camera device according to any one of claims 13 to 15, further comprising: A control processor performs at least one of control to cause a storage device to store the image data output from the processor and control to cause a display device to display an image based on the image data output from the processor.

20. A method for operating an imaging element, the imaging element comprising: a memory for storing image data obtained by photographing at a first frame rate; and a processor for processing the image data and outputting the processed image data at a second frame rate, the method comprising the following steps: deriving, by using the image data, shaking degree information indicating the degree of shaking included in an image represented by the image data; and outputting the derived jitter degree information at a frame rate higher than the second frame rate, The first frame rate is a frame rate higher than the second frame rate.

21. A storage medium storing a program for causing a computer to execute specific processing, the computer being suitable for an imaging element having a memory and a processor, the memory storing image data obtained by photographing at a first frame rate, the processor processing the image data and outputting the processed image data at a second frame rate, The specific process includes the following steps: deriving, by using the image data, shaking degree information indicating the degree of shaking included in an image represented by the image data; and outputting the derived jitter degree information at a frame rate higher than the second frame rate, The first frame rate is a frame rate higher than the second frame rate.

Images