Imaging device, driving method, and computer readable medium

Abstract

Provided is an image pickup apparatus, a driving method, and a computer-readable medium that can suppress hand shake and can suppress discontinuity of a moving subject when the moving subject is picked up. An image pickup apparatus includes an image pickup sensor and at least one processor, and the processor is configured to execute: image pickup processing for acquiring a plurality of first frames picked up at a first frame rate in a first exposure mode by the image pickup sensor; correction processing for performing electronic shake correction based on an amount of shake applied to the image pickup apparatus in the plurality of first frames; and generation processing for generating video data of a second frame having a second frame rate lower than the first frame rate by synthesizing the plurality of first frames, and in the first exposure mode, the processor sets m to a positive integer, and makes an exposure time in an mth first frame constituting the second frame shorter than an exposure time in an m+1th first frame.

Description

TECHNICAL FIELD

[0001] The technology of the present application relates to an image pickup apparatus, a driving method, and a computer-readable medium. BACKGROUND

[0002] In the image pickup apparatus described in Patent Literature 1, the image pickup element is configured to be able to change the frame rate at the time of image acquisition, and the hand-shake sensor is configured to detect the amount of camera shake by detecting physical vibration. In a case where the amount of shake from the hand-shake sensor exceeds a prescribed value, the image pickup apparatus sets the frame rate at the time of image acquisition in the image pickup element to a value higher than the normal frame rate, and controls to perform image acquisition at the high-speed frame rate. Then, the image pickup apparatus synthesizes the images of the plurality of frames thus obtained.

[0003] In the image processing apparatus described in Patent Literature 2, a motion vector operation section calculates a motion vector that associates image blocks of partial regions of image frames that constitute an image signal of a first frame rate, based on motion in the image, between a plurality of image frames. A synthesis section generates one image frame by synthesizing the plurality of image frames by adding the image blocks associated between the plurality of image frames by the motion vector, and outputs the one image frame as an image signal of a second frame rate that is lower than the first frame rate.

[0004] The image acquisition apparatus described in Patent Literature 3 is characterized by including: an exposure time calculation section that calculates an exposure time of an image pickup mechanism based on an aperture value corresponding to an aperture and a brightness of an object; a prescribed time calculation section that calculates a prescribed time based on a focal length of a lens and a prescribed coefficient; a photographing method determination section that, in a case where the exposure time is equal to or longer than the prescribed time, acquires a plurality of object images and synthesizes the plurality of object images, thereby determining to perform multiple-exposure photographing of acquiring a synthesized image of the object; a motion detection section that detects motion between the plurality of object images in the multiple-exposure photographing; and a prescribed coefficient calculation section that updates the prescribed coefficient based on the detected amount of motion.

[0005] PRIOR ART DOCUMENTS

[0006] PATENT LITERATURE

[0007] Patent Literature 1: Japanese Patent Application Laid-Open No. 2006-33123

[0008] Patent Literature 2: Japanese Patent Application Laid-Open No. 2006-319784

[0009] Patent Literature 3: Japanese Patent Application Laid-Open No. 2007-104029 SUMMARY

[0010] PROBLEMS TO BE SOLVED BY THE INVENTION

[0011] One embodiment of the present invention provides a camera device, driving method, and computer-readable medium that can suppress hand shakiness and prevent the movement of the subject from becoming discontinuous when shooting a rapidly moving subject.

[0012] means for solving technical problems

[0013] To achieve the above objectives, the camera device of the present invention includes a camera sensor and at least one processor. The processor is configured to perform the following processes: camera processing, acquiring a plurality of first frames captured at a first frame rate in a first exposure mode using the camera sensor; correction processing, performing electronic jitter correction based on the amount of jitter applied to the camera device in the plurality of first frames; and generation processing, generating video data of a second frame having a second frame rate lower than the first frame rate by combining the plurality of first frames. In the first exposure mode, the processor sets m to a positive integer such that the exposure time in the m-th first frame constituting the second frame is shorter than the exposure time in the (m+1)-th first frame.

[0014] The preferred processor performs the following processing: set the unit time corresponding to the shutter speed as P, set the second frame rate as V, set the value obtained by dividing the first frame rate by the second frame rate as N, set n as a positive integer, and set the largest n that satisfies P < 1 / V - (n-1) / (V×N) as m, in the first exposure mode, the second frame is generated by compositing the m to Nth first frames from the 1st to the Nth first frames, and the exposure time in the mth first frame is shorter than the exposure time of the (m+1)th first frame.

[0015] Ideally, the total exposure time in each of the m-th to N-th first frames should be equal to the unit time.

[0016] Preferably, the exposure time in the m-th first frame is shorter than the frame duration of the first frame, and the exposure time in each of the (m+1)-Nth first frames is equal to the frame duration of the first frame.

[0017] The exposure time in each of the m-th to N-th first frames is preferably continuous in time.

[0018] The preferred processor performs the following processing: when m≥2, acquire at least one of the first frames from the 1st to the (m-1th)th frames that are not used in the generation of the video data.

[0019] The preferred processor uses multiple first frames acquired for jitter correction.

[0020] The preferred camera sensor has phase difference pixels, and the processor performs focusing based on the signals of the phase difference pixels contained in the acquired first frame.

[0021] The preferred processor is capable of executing a second exposure mode, which exposes the image by distributing the exposure time across each of the m-th to N-th first frames.

[0022] The preferred processor can selectively execute the first exposure mode and the second exposure mode.

[0023] The preferred processor performs the following processing: detects the moving speed of the subject; if the moving speed is above a specified value, selects the first exposure mode; if the moving speed is below the specified value, selects the second exposure mode.

[0024] The preferred processor performs the following processing: if the jitter is less than a specified value, select the first exposure mode; if the jitter is greater than or equal to the specified value, select the second exposure mode.

[0025] Preferably, a jitter detection sensor is used to detect rotational jitter in the tilt direction and translational jitter in the cross direction that intersects the rotation axis in the tilt direction. The processor performs translational jitter correction on multiple first frames and rotational jitter correction on second frames.

[0026] The preferred processor performs the following processing: when M is set to a positive integer, N is set to a multiple of M, and M < N, jitter correction is performed on multiple first frames based on the first jitter amount that ends jitter correction at 1 / (V×N) seconds; jitter correction is performed on the composite frame that combines the first frames into M frames based on the second jitter amount that ends jitter correction at 1 / (V×M) seconds; and the second frame is generated by combining the composite frames into N / M frames.

[0027] The preferred processor performs the following processing: In terms of jitter amount, rotational jitter in the tilt direction is used to correct jitter in the second frame, and translational jitter in the cross direction that intersects the rotation axis in the tilt direction is used to correct jitter in multiple first frames.

[0028] The driving method of the camera device of the present invention is a driving method of a camera device equipped with a camera sensor, comprising: a camera step, acquiring a plurality of first frames captured at a first frame rate in a first exposure mode by means of the camera sensor; a correction step, performing electronic jitter correction based on the jitter amount applied to the camera device in the plurality of first frames; and a generation step, generating video data of a second frame having a second frame rate lower than the first frame rate by combining the plurality of first frames, wherein in the first exposure mode, m is set to a positive integer such that the exposure time in the m-th first frame constituting the second frame is shorter than the exposure time in the (m+1)-th first frame.

[0029] The camera program of the present invention is a camera program that enables a camera device equipped with a camera sensor to operate. It is configured to perform the following processes: camera processing, acquiring a plurality of first frames captured at a first frame rate in a first exposure mode by the camera sensor; correction processing, performing electronic jitter correction based on the amount of jitter applied to the camera device in the plurality of first frames; and generation processing, generating video data of a second frame having a second frame rate lower than the first frame rate by combining the plurality of first frames. In the first exposure mode, m is set to a positive integer such that the exposure time in the m-th first frame constituting the second frame is shorter than the exposure time in the (m+1)-th first frame. Attached Figure Description

[0030] Figure 1 This is a schematic perspective view showing an example of the front surface side of a camera device.

[0031] Figure 2 This is a schematic perspective view showing an example of the rear side of a camera device.

[0032] Figure 3 This is a diagram illustrating an example of the internal structure of a camera device.

[0033] Figure 4 This is a block diagram illustrating an example of the functional structure of a processor and an image processing unit.

[0034] Figure 5 This is a block diagram illustrating an example of the structure of the correction calculation unit.

[0035] Figure 6 This is a diagram illustrating an example of the structure of a signal processing unit.

[0036] Figure 7 This is a diagram illustrating an example of the structure of a signal processing unit.

[0037] Figure 8 This is a diagram illustrating an example of jitter correction processing.

[0038] Figure 9 This is a diagram illustrating an example of frame composition processing.

[0039] Figure 10 This is a diagram illustrating an example of the structure of a camera sensor.

[0040] Figure 11 This is a diagram illustrating an example of the camera sensor's camera action.

[0041] Figure 12 This represents an example of camera action when m=1.

[0042] Figure 13 An example of camera action when m=2.

[0043] Figure 14 This diagram illustrates an example of the second exposure mode involved in the second embodiment.

[0044] Figure 15 This is a flowchart illustrating an example of exposure mode selection processing.

[0045] Figure 16 This is a flowchart illustrating an example of exposure mode selection processing.

[0046] Figure 17 This diagram illustrates an example of the camera operation involved in the third embodiment.

[0047] Figure 18 This diagram illustrates an example of the structure of the image processing unit according to the fourth embodiment.

[0048] Figure 19 This is a diagram illustrating the jitter correction involved in the fourth embodiment.

[0049] Figure 20 This diagram illustrates an example of the structure of the image processing unit according to the fifth embodiment.

[0050] Figure 21 This is a diagram illustrating the jitter correction involved in the fifth embodiment. Detailed Implementation

[0051] An example of an embodiment of the technology of the present invention will be described with reference to the accompanying drawings.

[0052] First, let me explain the terms used in the following explanation.

[0053] In the following explanations, "IC" is an abbreviation for "Integrated Circuit". "CPU" is an abbreviation for "Central Processing Unit". "ROM" is an abbreviation for "Read Only Memory". "RAM" is an abbreviation for "Random Access Memory". "CMOS" is an abbreviation for "Complementary Metal Oxide Semiconductor".

[0054] FPGA stands for Field-Programmable Gate Array. PLD stands for Programmable Logic Device. ASIC stands for Application-Specific Integrated Circuit. OVF stands for Optical View Finder. EVF stands for Electronic View Finder. JPEG stands for Joint Photographic Experts Group. DSP stands for Digital Signal Processor.

[0055] In this invention, the term "equal" includes not only the case of complete equality, but also the case of substantial equality within the scope of the technical field to which this invention pertains. Furthermore, in this invention, the term "orthogonal" includes not only the case of being orthogonal at a 90° angle, but also the case of substantial intersection within the scope of the technical field to which this invention pertains.

[0056] [First Implementation Method]

[0057] As a first embodiment of the imaging device, an interchangeable-lens digital camera will be used as an example to describe the technology of the present invention. However, the technology of the present invention is not limited to interchangeable-lens cameras, but can also be applied to digital cameras with integrated lenses.

[0058] (Structure of the camera device)

[0059] Figure 1 This shows an example of the front side of the camera device 10. For example... Figure 1 As shown, the imaging device 10 is an interchangeable-lens digital camera. The imaging device 10 consists of a main body 11 and an imaging lens 12 replaceably mounted on the main body 11. The imaging lens 12 is mounted via a camera-side mount 11A and a lens-side mount 12A (see reference). Figure 3 It is mounted on the front surface 11C side of the main body 11. The camera lens 12 is an example of a lens involved in the technology of this invention.

[0060] A turntable 13 and a release button 14 are provided on the upper surface of the main body 11. The turntable 13 is operated when setting the operation mode, etc. The operation modes of the camera device 10 include, for example, still image shooting mode, moving image shooting mode, and image display mode. When starting still image shooting or moving image shooting, the user operates the release button 14.

[0061] Figure 1 Z-axis A shown Z Corresponding to the optical axis of imaging lens 12. X-axis A X and Y-axis A Y They are orthogonal to each other and perpendicular to the Z-axis A. Z Orthogonal. X-axis A X and Y-axis A Y This corresponds to the pitch axis and yaw axis involved in the technology of this invention. In the following description, the Z-axis A will be used as the reference axis. Z The direction of rotation is called the tilt direction. Furthermore, the rotation around the X-axis A... X The direction of rotation is called the pitch direction. Furthermore, the rotation around the Y-axis A... Y The direction of rotation is called the yaw direction. Furthermore, the X-axis A... X The direction is called the X direction, and the Y-axis A is... Y The direction is called the Y direction. Additionally, the Z-axis A... Z This is an example of a "rotation axis" involved in the technology of this invention. Furthermore, the X and Y directions are examples of "intersecting directions that intersect the rotation axis" involved in the technology of this invention.

[0062] Figure 2 This shows an example of the rear side of the camera device 10. For example... Figure 2 As shown, a viewfinder eyepiece 18 comprising a display 15, indicator keys 16, and a viewfinder 17 is provided on the back 11D of the main body 11. The viewfinder can be an optical viewfinder or an electronic viewfinder. The display 15 displays images based on the captured image signals and various menu screens. The indicator keys 16 receive various instructions.

[0063] Figure 3 This illustrates an example of the internal structure of the imaging device 10. The main body 11 and the imaging lens 12 are electrically connected by contact between an electrical contact 11B provided on the camera-side bayonet 11A and an electrical contact 12B provided on the lens-side bayonet 12A.

[0064] The imaging lens 12 includes an objective lens 30, a focusing lens 31, a rear lens 32, and an aperture 33. Each component is aligned along the optical axis (i.e., the Z-axis A) of the imaging lens 12. ZThe objective lens 30, aperture 33, focusing lens 31, and rear lens 32 are arranged sequentially from the objective lens side. The objective lens 30, focusing lens 31, and rear lens 32 constitute the imaging optical system. The type, number, and arrangement order of the lenses constituting the imaging optical system are not limited to... Figure 3 Example shown.

[0065] Furthermore, the camera lens 12 has a lens drive control unit 34. The lens drive control unit 34 is composed of, for example, a CPU, RAM, and ROM. The lens drive control unit 34 is electrically connected to the processor 40 in the main body 11 via electrical contacts 12B and 11B.

[0066] The lens drive control unit 34 drives the focusing lens 31 and the aperture 33 based on control signals sent from the processor 40. For focusing the imaging lens 12, the lens drive control unit 34 controls the drive of the focusing lens 31 based on focusing control signals sent from the processor 40. The processor 40 performs, for example, vertex adjustment using a phase difference method.

[0067] The main body 11 is equipped with an image sensor 20, a processor 40, an image processing unit 41, an operation unit 42, a shake detection sensor 44, a memory 45, and a display 15. The image sensor 20, the image processing unit 41 (which is part of the processor 40), the operation unit 42, the shake detection sensor 44, the memory 45, and the display 15 are controlled by the processor 40. The processor 40 is, for example, composed of a CPU, RAM, and ROM. In this case, the processor 40 performs various processes based on the camera program 45A stored in the memory 45. Alternatively, the processor 40 may be composed of an assembly of multiple IC chips. The image sensor 20 is, for example, a CMOS image sensor.

[0068] The jitter detection sensor 44 detects the amount of jitter applied to the camera device 10. The jitter detection sensor 44 is, for example, a 5-axis jitter detection sensor that detects jitter in the roll, yaw, pitch, X, and Y directions respectively. Hereinafter, jitter in the roll direction will be referred to as rotational jitter. Jitter in the yaw and pitch directions will be referred to as angular jitter. Jitter in the X and Y directions will be referred to as translational jitter.

[0069] The jitter detection sensor 44 is, for example, composed of a gyroscope sensor 44A and an accelerometer sensor 44B (see reference). Figure 4The gyroscope sensor 44A detects rotational and angular jitter. The accelerometer sensor 44B detects translational jitter. The jitter detection sensor 44 outputs the detected jitter detection signal to the processor 40. The jitter detection sensor 44 is one example of a jitter detection unit according to the technology of this invention. Alternatively, the jitter detection sensor 44 can be disposed within the camera lens 12. In this case, the processor 40 can obtain the jitter detection value from the jitter detection sensor 44 within the camera lens 12 via electrical contacts 12B and 11B.

[0070] The image processing unit 41 is, for example, configured as a DSP. The image processing unit 41 performs various image processing operations on the image signal to generate image data in a predetermined file format (e.g., JPEG format). Furthermore, when capturing moving images, the image processing unit 41 performs electronic shake correction processing and frame synthesis processing, as described later.

[0071] The display 15 displays images based on the image data generated by the image processing unit 41. These images include still images, moving images, and live view images. Live view images are images that are displayed in real-time on the display 15 by sequentially outputting the image data generated by the image processing unit 41 to the display 15.

[0072] Image data generated by the image processing unit 41 can be stored in the internal memory (not shown) built into the main body 11 or on a storage medium (e.g., a memory card) that can be attached to or removed from the main body 11.

[0073] The operating unit 42 includes the aforementioned rotary dial 13, release button 14, and indicator key 16 (see reference). Figure 1 and Figure 2 The processor 40 controls various parts within the main body 11 and the lens drive control unit 34 within the camera lens 12 according to the operation of the operation unit 42.

[0074] (Processor architecture)

[0075] Figure 4 This is an example of the functional structure of the processor 40 and the image processing unit 41, a part of the processor 40. The processor 40 performs processing via a camera program 45A stored in the memory 45, thereby implementing various functional units. For example... Figure 4 As shown, for example, the processor 40 implements the main control unit 50, the camera control unit 51, the exposure control unit 52, and the correction amount calculation unit 53.

[0076] The main control unit 50 controls the operation of the camera device 10 based on the instruction signal input from the operation unit 42. The camera control unit 51 controls the camera sensor 20 to perform camera processing, causing the camera sensor 20 to perform camera actions. The camera control unit 51 drives the camera sensor 20 in still image camera mode or moving image camera mode.

[0077] The user can select between still image shooting mode and moving image shooting mode via the operation unit 42. Furthermore, the user can set the exposure value, including shutter speed and aperture value, via the operation unit 42. The main control unit 50 supplies the unit time P, set using the operation unit 42, corresponding to the shutter speed (in seconds), to the exposure control unit 52. For example, the unit time P is equal to the shutter speed. The main control unit 50 inputs the aperture value A, set using the operation unit 42, to the lens drive control unit 34.

[0078] Furthermore, the user can select the automatic exposure mode by operating the operation unit 42. When the automatic exposure mode is selected, the main control unit 50 calculates the optimal values ​​for shutter speed and aperture by using the brightness of the image signal obtained from the image sensor 20 and the calculation of the program graph.

[0079] Exposure control unit 52 determines the exposure start timing of image sensor 20 based on the unit time P supplied from main control unit 50, and supplies the determined exposure start timing to image control unit 51. Lens drive control unit 34 adjusts the aperture 33 (reference aperture) based on the aperture value A supplied from main control unit 50. Figure 3 Adjust the opening diameter of the opening.

[0080] The correction calculation unit 53 calculates a correction amount, which is used to correct jitter in the image signal output from the camera sensor 20 based on the jitter detection signals output from each of the gyroscope sensor 44A and accelerometer sensor 44B included in the jitter detection sensor 44. The correction calculation unit 53 supplies the calculated correction amount to the image processing unit 41.

[0081] Gyroscope sensor 44A is an angular velocity sensor that detects rotational and angular jitter, outputting an angular velocity signal as the detected value. Gyroscope sensor 44A outputs an angular velocity signal B representing the rotational jitter. R And the angular velocity signal B representing angular jitter. Y Angular velocity signal B P Angular velocity signal B Y Indicates angular jitter in the yaw direction. Angular velocity signal B P This indicates angular jitter in the pitch direction.

[0082] Accelerometer 44B outputs an acceleration signal as the detected value for translational jitter. Accelerometer 44B outputs an acceleration signal B representing the translational jitter in the X direction. SX And the acceleration signal B representing translational jitter in the Y direction. SY .

[0083] In this embodiment, the jitter detection axes are five axes: roll, yaw, pitch, X, and Y. Conversely, the jitter correction axes are three axes: roll, X, and Y. Therefore, for the yaw and pitch directions, it is not possible to correct the jitter based on the angular velocity signal B, which represents angular jitter. Y Angular velocity signal B P This is used to directly correct angular jitter. In this embodiment, angular jitter in the yaw direction is corrected by including translational jitter in the X direction, and angular jitter in the pitch direction is corrected by including translational jitter in the Y direction.

[0084] The correction calculation unit 53 is based on the angular velocity signal B. R The unit calculates the rotational amount Cr used to correct rotational jitter. Furthermore, the correction amount calculation unit 53 is based on the acceleration signal B. SX Acceleration signal B SY and angular velocity signal B Y Angular velocity signal B P Calculate the displacement Cs used to correct translational jitter.

[0085] The image processing unit 41 includes a shake correction unit 46 that performs electronic shake correction processing when capturing moving images, and a frame combining unit 47 that combines multiple frames whose shake has been corrected. A frame, in this context, refers to an image represented by an image signal output from the camera sensor 20 during one frame. Shake correction processing is one example of the correction processing involved in the present invention.

[0086] The jitter correction unit 46 corrects translational and rotational jitter by changing the cropping region, which is formed by partially cropping an image from the imaging area of ​​the camera sensor 20, between frames. The change in the cropping region includes displacement and rotation of the cropping region. Therefore, the jitter correction unit 46 needs to perform calculations such as projection transformation or affine transformation on the signals contained in the cropping region.

[0087] The frame synthesis unit 47 performs frame synthesis processing to generate video data at a second frame rate lower than the first frame rate corresponding to the drive frequency of the camera sensor 20 by synthesizing multiple frames whose jitter has been corrected by the jitter correction unit 46. Frame synthesis processing is one example of the generation processing involved in the technology of this invention.

[0088] (Structure of the correction calculation unit)

[0089] Figures 5-7 This illustrates an example of the structure of the correction calculation unit 53. For example... Figure 5 As shown, the correction calculation unit 53 consists of signal processing units 61-65, adder 66, adder 67, rotation calculation unit 68, and displacement calculation unit 69.

[0090] The signal processing unit 61 will process the angular velocity signal B representing the rotational jitter. R The correction quantity V, converted into angular information R And output it to the rotation calculation unit 68. For example... Figure 6 As shown, the signal processing unit 61 includes, for example, a subtractor 61A, a high-pass filter (hereinafter referred to as HPF) 61B, a multiplier 61C, and an integrator 61D.

[0091] Subtractor 61A receives the angular velocity signal B R Subtract zero-point correction value Z R Offset correction is then performed. Zero-point correction value Z R This is the output value from gyroscope sensor 44A when gyroscope sensor 44A is stationary. HPF61B removes residual DC components that cannot be completely removed by offset correction using subtractor 61A.

[0092] Multiplier 61C multiplies the output signal from HPF61B by the gain value G. R Gain correction is then performed. Gain value G R This value is determined based on the focal length of the camera lens 12 and / or the sensitivity of the gyroscope sensor 44A. Additionally, in the case of rotational jitter, the gain value G... R It is independent of the focal length of the camera lens 12. The integrator 62D generates a correction amount V representing angle information by integrating the output signal from the multiplier 62C. R It is then output to the rotation calculation unit 68.

[0093] The rotation calculation unit 68 calculates the rotation amount based on the correction amount V input from the signal processing unit 61. R Calculate the rotation amount Cr, and input the calculated rotation amount Cr into the jitter correction unit 46 (reference). Figure 4 The jitter correction unit 46 rotates the shearing region by an amount equivalent to the angle corresponding to the input rotation amount Cr.

[0094] The signal processing unit 62 will process the acceleration signal B, which represents the translational jitter in the X direction. SX Converted into a correction quantity V representing position information SX And output it. The signal processing unit 63 outputs the angular velocity signal B, which represents the angular jitter in the yaw direction. Y The correction quantity V, converted into angular information YAnd output. Adder 66 will take the correction amount V output from signal processing unit 62. SX The correction amount V output from the signal processing unit 63 Y Add them together and use the sum as the total correction amount V. S1 Output to displacement calculation unit 69.

[0095] The signal processing unit 64 will process the acceleration signal B, which represents the translational jitter in the Y direction. SY Converted into a correction quantity V representing position information SY And output it. The signal processing unit 65 will output the angular velocity signal B, which represents the pitch direction angle jitter. P The correction quantity V, converted into angular information P And output. Adder 67 will take the correction amount V output from signal processing unit 64. SY The correction amount V output from the signal processing unit 65 P Add them together and use the sum as the total correction amount V. S2 Output to displacement calculation unit 69.

[0096] like Figure 7 As shown, the signal processing unit 62 includes, for example, a subtractor 62A, an HPF 62B, a multiplier 62C, a first integrator 62D, and a second integrator 62E. This is to process the acceleration signal B, which is the second derivative value of the position information. SX The signal processing unit 62 has two integrators, a first integrator 62D and a second integrator 62E, which are converted into position information.

[0097] and Figure 6 Similarly, in the signal processing unit 61 shown, the subtractor 62A processes the acceleration signal B... SX Subtract zero-point correction value Z SX Offset correction is then performed. Zero-point correction value Z SX This is the output value from the accelerometer 44B when it is stationary. The HPF62B removes the residual DC component from the offset correction based on subtractor 62A. Multiplier 62C multiplies the output signal from the HPF62B by the gain value G. SX Gain correction is then performed. Gain value G SX The value is determined based on the focal length of the camera lens 12 and / or the sensitivity of the accelerometer sensor 44B.

[0098] The first integrator 62D outputs the integrated value obtained by integrating the output signal from the multiplier 62C. The second integrator 62E generates a correction quantity V representing position information by integrating the integrated value output from the first integrator 62D. SX And output it.

[0099] The signal processing unit 63 includes, for example, a subtractor 63A, an HPF 63B, a multiplier 63C, and an integrator 63D. The structure of the signal processing unit 63 is similar to... Figure 6 The structure of the signal processing unit 61 shown is the same.

[0100] Signal processing unit 64 includes, for example, a subtractor 64A, an HPF 64B, a multiplier 64C, a first integrator 64D, and a second integrator 64E. The structure of signal processing unit 64 is the same as that of signal processing unit 62.

[0101] The signal processing unit 65 includes, for example, a subtractor 65A, an HPF 65B, a multiplier 65C, and an integrator 65D. The structure of the signal processing unit 65 is similar to... Figure 6 The structure of the signal processing unit 61 shown is the same.

[0102] The displacement calculation unit 69 is based on the total correction amount V input from the adder 66. S1 The total correction amount V input from adder 67 S2 The displacement Cs, representing the displacement in the X and Y directions, is calculated and input into the jitter correction unit 46 (reference). Figure 4 The jitter correction unit 46 moves the shearing region by a distance and direction corresponding to the input displacement Cs.

[0103] (Jitter correction processing)

[0104] Figure 8 This illustrates an example of jitter correction processing based on jitter correction unit 46. Figure 8 In the image sensor 20, the symbol 21 represents the imaging area in the light-receiving surface 20A of the image sensor 20. The imaging area 21 is, for example, an effective pixel area. Furthermore, the symbol CA represents a cropped region, a region of the image partially cropped from the overall image corresponding to the imaging area 21. The cropped region CA is smaller than the imaging area 21.

[0105] In addition, image cropping is performed, for example, by storing the overall image data corresponding to the camera area 21 in the memory, and then reading the data corresponding to the cropping area CA from the overall image data stored in the memory.

[0106] Within the imaging area 21, the jitter correction unit 46 displaces and / or rotates the shearing region CA. Specifically, the jitter correction unit 46 displaces the shearing region CA in the X and Y directions based on the displacement amount Cs input from the displacement amount calculation unit 69, and rotates the shearing region CA in the tilt direction based on the rotation amount Cr input from the rotation amount calculation unit 68. Figure 8 The two-dot dashed line represents an example of the shear region CA after displacement and rotation.

[0107] Each time a frame of image signal is output from the camera sensor 20, the jitter correction unit 46 changes the clipping region CA. That is, the jitter correction unit 46 corrects translational jitter and rotational jitter by changing the clipping region CA between frames.

[0108] (Frame composition processing)

[0109] Figure 9 An example of frame synthesis processing based on frame synthesis unit 47 is described below. The camera sensor 20 outputs an image signal at a first frame rate. The jitter correction unit 46 changes the clipping region CA every frame and clips the image from the clipping region CA. Hereinafter, the frame acquired at the first frame rate is referred to as the first frame.

[0110] Figure 9 This example illustrates how a second frame is synthesized by combining four first frames. The first frames F1(1) to F1(4) are four consecutive frames of images cropped (i.e., jitter-corrected) by the jitter correction unit 46. The frame synthesis unit 47 generates a second frame F2 by adding frames F1(1) to F1(4) with each corresponding pixel.

[0111] In this example, the second frame F2 is generated by synthesizing four first frames F1(1) to F1(4), therefore the frame rate of the second frame F2 (hereinafter referred to as the second frame rate) is 1 / 4 of the first frame rate. Video data at the second frame rate is generated from multiple second frames F2. For example, the first frame rate is 240 fps (frames per second), and the second frame rate is 60 fps.

[0112] The second frame F2, which constitutes the video data, is generated by synthesizing multiple first frames F1(1) to F1(4) that have undergone jitter correction. Therefore, the effect of jitter is suppressed and the image becomes bright.

[0113] In the case of electronic image stabilization (EIS), there are technical challenges, such as the difficulty in correcting for jitter generated during the exposure time of each frame. However, in the case of video data shot at a high first frame rate of 240fps, the exposure time in each frame is very short, thus minimizing the amount of jitter generated during the exposure time. Therefore, it is difficult to generate large-scale jitter between frames, thereby improving the accuracy of EIS as a whole for the video data. The synthesized 60fps second frame rate video data becomes an image with suppressed jitter compared to video data shot at 60fps without synthesis.

[0114] Additionally, the brightness of the second frame F2 constituting the video data can be adjusted within the image processing unit 41 by gain adjustment.

[0115] (Structure of a camera sensor)

[0116] Figure 10 An example illustrating the structure of the camera sensor 20. Figure 10 The image sensor 20 shown is a CMOS image sensor. The image sensor 20 has an image capture area 21, a vertical scanning circuit 22, a line memory 23, a horizontal scanning circuit 24, and an output amplifier 25.

[0117] In the imaging area 21, a plurality of pixels 26 are arranged in a two-dimensional matrix along the X and Y directions. Furthermore, in the imaging area 21, a plurality of row selection lines L1 and a plurality of row reset lines L2 are routed along the X direction, and a plurality of column signal lines L3 are routed along the Y direction. Pixels 26 are connected to the row selection lines L1, row reset lines L2, and column signal lines L3. Hereinafter, the plurality of pixels 26 arranged along the X direction will sometimes be referred to simply as a "row". Additionally, a portion of the plurality of pixels 26 may be phase difference pixels used for focusing.

[0118] Pixel 26 has a photodiode D1, an amplifier transistor M1, a pixel selection transistor M2, and a reset transistor M3. The photodiode D1 generates a signal charge corresponding to the amount of incident light by performing photoelectric conversion, and accumulates the generated signal charge. The amplifier transistor M1 generates a voltage (hereinafter referred to as the pixel signal S) corresponding to the amount of signal charge accumulated in the photodiode D1.

[0119] Pixel selection transistor M2, controlled by vertical scan circuit 22 via row selection line L1, outputs the pixel signal S generated by amplifier transistor M1 to column signal line L3. Reset transistor M3, controlled by vertical scan circuit 22 via row reset line L2, discards the signal charge accumulated in photodiode D1 onto the power line. Hereinafter, discarding signal charge from photodiode D1 is referred to as resetting pixel 26.

[0120] In addition, in a CMOS image sensor, reading out the pixel signal S from pixel 26 is a non-destructive readout that does not affect the signal charge accumulated in pixel 26. Therefore, after reading out the pixel signal S, pixel 26 needs to be reset.

[0121] The vertical scanning circuit 22 generates a row selection signal SEL and a reset signal RST based on a timing signal input from the camera control unit 51. During signal readout, the vertical scanning circuit 22 assigns the row selection signal SEL to the row selection line L1, thereby outputting the pixel signal S from the pixel 26 connected to the row selection line L1 to the column signal line L3.

[0122] Furthermore, during the reset operation, the vertical scan circuit 22 resets the pixel 26 connected to the row reset line L2 by assigning a reset signal RST to the row reset line L2. For example, the reset of the pixel 26 connected to the row reset line L2 of the nth row is performed during the period when the pixel signal S is read from the pixel 26 connected to the row selection line L1 of the (n+1)th row.

[0123] Line memory 23 stores pixel signals S output from a row of pixels 26. Line memory 23 is composed of capacitors, etc. Line memory 23 is connected to horizontal output line 24A via transistor 29, which acts as a switch. Output amplifier 25 is connected to the end of horizontal output line 24A. Horizontal scanning circuit 24 sequentially outputs the row of pixel signals S stored in line memory 23 to horizontal output line 24A by performing a horizontal scan that sequentially selects transistor 29. The pixel signals S output to horizontal output line 24A are then output as image signals to external image processing unit 41 via output amplifier 25.

[0124] The operation of the vertical scanning circuit 22, the line memory 23, and the horizontal scanning circuit 24 is controlled by the camera control unit 51 (see reference). Figure 4 The camera control unit 51 controls the vertical scanning circuit 22 to sequentially select the row selection line L1, and simultaneously outputs the pixel signal S. Furthermore, the camera control unit 51 controls the vertical scanning circuit 22 to sequentially select the row reset line L2, and simultaneously resets the pixel 26. Thus, the camera control unit 51 controls the reading and resetting of the image signal from the camera sensor 20 through "sequential readout mode" and "sequential reset mode".

[0125] In motion picture camera mode, the camera control unit 51 drives the camera sensor 20 at a first frame rate supplied from the main control unit 50. The camera control unit 51 causes the camera sensor 20 to repeatedly perform image signal readout and reset operations at a period corresponding to the first frame rate.

[0126] Furthermore, the structure of the camera sensor 20 is not limited to... Figure 10 The structure shown. For example, an A / D converter can also be set in the camera sensor 20.

[0127] (Camera action)

[0128] Figure 11 This illustrates an example of the camera action of the camera sensor 20 in motion image camera mode. Figure 11 In this context, T1 represents the first frame period as the reciprocal of the first frame rate. T2 represents the second frame period as the reciprocal of the second frame rate. For example, when the first frame rate is 240fps, T1 = 1 / 240 seconds. When the second frame rate is 60fps, T2 = 1 / 60 seconds.

[0129] The first frame period T1 is defined by the period of the line selection signal SEL input from the vertical scanning circuit 22 on the same line selection line L1. That is, an image signal is output from the camera sensor 20 according to each first frame period T1.

[0130] In this embodiment, the second frame period T2 includes four first frames F1(1) to F1(4). F1(n) represents the nth first frame after the start of the second frame period T2. Here, n is a positive integer.

[0131] The symbols E1 to E4 represent the exposure time in each of the first frames F1(1) to F1(4). The exposure times E1 to E4 are the time from the input reset signal RST to the input line selection signal SEL in each line (i.e., the charge accumulation time).

[0132] In this embodiment, the exposure periods are staggered row by row. That is, the camera sensor 20 is exposed using an electronic shutter in a focal plane manner.

[0133] exist Figure 11 In this case, the exposure times E1 through E4 are all the same, equal to the first frame period T1. That is, Figure 11 The exposure times E1 to E4 shown are the longest exposure times when the electronic shutter is fully open.

[0134] The first frames F1(1) to F1(4) are jittered by the jitter correction unit 46 and then synthesized by the frame synthesis unit 47 to become the second frame F2.

[0135] A second frame F2 is generated in each second frame period T2. The second frame F2 is a composite of four images of the first frames F1(1) to F1(4) taken with the same exposure time, so the brightness is equivalent to that of the images taken with four times the exposure time.

[0136] Figure 11 The camera action shown is performed when the unit time P corresponding to the aforementioned shutter speed is the maximum settable value. In this case, the unit time P is equal to the second frame period T2.

[0137] (Exposure control)

[0138] Next, the control of exposure time based on exposure control unit 52 will be explained. Exposure control unit 52 determines the exposure start timing based on the unit time P supplied from main control unit 50. Specifically, exposure control unit 52 determines the exposure start timing so that the total exposure time in each of the first frame is equal to the unit time P. Based on the determined exposure start timing, exposure control unit 52 controls the reset timing of camera sensor 20 via camera control unit 51.

[0139] More specifically, when the largest n satisfying the following equation (1) is set to m, the exposure control unit 52 determines the exposure start timing (i.e., the reset timing) so that the exposure time in the mth first frame F1(m) is shorter than the exposure time in the (m+1)th first frame F1(m+1).

[0140] P<1 / V-(n-1) / (V×N)···(1)

[0141] Here, V represents the second frame rate. N is the value obtained by dividing the first frame rate by the second frame rate. That is, N represents the number of frames in the first frame of the second frame period T2. n is a positive integer.

[0142] The exposure control unit 52 makes the total exposure time of each of the m-th first frame F1(m) to the N-th first frame F1(N) equal to the unit time P. Furthermore, it is preferable that the exposure control unit 52 makes the exposure time of the m-th first frame F1(m) shorter than the exposure time of each of the (m+1)-N-th first frames F1(m+1) to F1(N). Specifically, the exposure time of the m-th first frame F1(m) is shorter than the first frame period T1, and the exposure time of each of the (m+1)-N-th first frames F1(m+1) is equal to the first frame period T1. In this embodiment, the exposure times of each of the m-th first frame F1(m) to the N-th first frame F1(N) are continuous in time.

[0143] After the jitter correction unit 46 corrects the jitter in the m-th to N-th first frames F1(1) to F1(N), the images are combined by the frame synthesis unit 47 to form the second frame F2 constituting the video data. That is, when m≥2, the first first frame F1(1) to the (m-1)-th first frame F1(m-1) are not used to generate video data.

[0144] For example, if the first frame rate is 240fps and the second frame rate is 60fps, then V = 240fps and N = 4. In this case, for example, if P = 1 / 70 of a second, then m = 1. And if P = 1 / 100, then m = 2.

[0145] Figure 12 This is an example of the camera action when m=1. When m=1, the exposure control unit 52 delays the timing of the reset signal RST generated by the vertical scanning circuit 22 in the first frame F1(1) (i.e., the reset timing), so that the exposure time E1 in the first frame F1(1) is shorter than the first frame period T1.

[0146] In the first frame F1(1), the accumulation is at pixel 26 (reference). Figure 10 The signal charge in the first frame F1(1) is discarded upon reset. After the signal charge is discarded upon reset, pixel 26 resumes the accumulation of signal charge. That is, the reset timing in the first frame F1(1) corresponds to the exposure start timing. In this example, the period from the start of the frame period of the first frame F1(1) to the reset (in Figure 12 The period indicated by the diagonal shading in the text is called the shading period. Additionally, during the shading period, pixel 26 can be repeatedly reset.

[0147] Thus, when m=1, the second frame F2 is generated based on the four first frames F1(1) to F1(4). In this example, the exposure times E1 to E4 are continuous in time.

[0148] Figure 13 This is an example of the camera action when m=2. When m=2, the exposure control unit 52 delays the timing of the reset of the vertical scan circuit 22 in the second first frame F1(2), so that the exposure time E2 in the second frame F1(2) is shorter than the first frame period T1.

[0149] In this example, the first first frame F1(1) is discarded after being read from the camera sensor 20, or discarded within the camera sensor 20. That is, the first first frame F1(1) is not used to generate video data. In this example, the frame period of the first first frame F1(1) and the frame period of the second first frame F1(2) from the start to the reset period (in Figure 13 The period indicated by the diagonal shading in the text is called the shading period. Additionally, during the shading period, pixel 26 can be repeatedly reset.

[0150] Thus, when m=2, the second frame F2 is generated based on the three first frames F1(2) to F1(4). In this example, the exposure times E2 to E4 are continuous in time.

[0151] The same applies to the cases where m=3 and m=4. When m=3, the exposure control unit 52 makes the exposure time E3 in the third first frame F1(3) shorter than the first frame period T1. When m=4, the exposure time E4 in the fourth first frame F1(4) is made shorter than the first frame period T1.

[0152] As described above, in the exposure mode of this embodiment, the exposure time in each of the plurality of first frames is temporally continuous, regardless of the length of the unit time P (see reference). Figures 11-13In contrast, consider distributing the exposure time across each of the multiple first frames. In this case, the exposure time of each first frame is discontinuous and discrete, which could lead to discontinuous motion of the subject when shooting a rapidly moving subject. In the exposure mode of this embodiment, the exposure time in each of the multiple first frames is temporally continuous, thus suppressing the effects of camera shake and preventing discontinuous motion of the subject.

[0153] [Second Implementation]

[0154] Next, the second embodiment will be described. In the second embodiment, in addition to the first exposure mode, a second exposure mode in which the exposure time is distributed across each of the plurality of first frames can also be executed for the exposure mode involved in the first embodiment.

[0155] In the second exposure mode, the exposure control unit 52 sets the value obtained by dividing the unit time P by the number of frames N of the first frame in the second frame period T2 as the exposure time in each of the plurality of first frames. In this embodiment, as Figure 14 As shown, in the second exposure mode, exposure times E1 through E4 are all equal. However, there can be differences between the exposure times.

[0156] In the second exposure mode, the exposure times E1 to E4 are discrete. Therefore, as mentioned above, when shooting fast-moving subjects, the movement of the subject may become discontinuous. However, in the second exposure mode, the unit time P is shorter, and the exposure times E1 to E4 are also shorter, while the number of the first frame of the generated video data remains constant, thus achieving a high level of jitter suppression.

[0157] For example, the preferred exposure control unit 52 uses the operation unit 42 to allow the user to set a first exposure mode and a second exposure mode, and switches between them based on the setting information supplied from the main control unit 50.

[0158] Furthermore, the exposure control unit 52 can select an appropriate exposure mode from the first exposure mode and the second exposure mode based on the detected value of the subject's moving speed or the amount of jitter.

[0159] Figure 15 This is an example of selection processing that chooses the exposure mode based on the detected speed of the moving subject. For example... Figure 15 As shown, the exposure control unit 52 first determines whether motion image recording based on the camera sensor 20 has started (step S10). At this time, as the initial exposure mode, for example, the second exposure mode is used for motion image recording.

[0160] When the exposure control unit 52 determines that motion image capture has started (step S10: Yes), it detects the moving speed of the subject (step S11). For example, the exposure control unit 52 detects the moving speed of the subject by calculating the moving vector of the subject between the first frames. Alternatively, the exposure control unit 52 can detect the moving speed of the subject by calculating the moving vector of the subject between the second frames.

[0161] Next, the exposure control unit 52 determines whether the moving speed of the detected subject is above or below a predetermined value (step S12). If the exposure control unit 52 determines that the moving speed of the subject is above or below the predetermined value (step S12: Yes), a first exposure mode is selected (step S13). On the other hand, if the exposure control unit 52 determines that the moving speed of the subject is below the predetermined value (step S12: No), a second exposure mode is selected (step S14). The image sensor 20 exposes itself using the exposure mode selected by the exposure control unit 52.

[0162] After step S13 or S14, the exposure control unit 52 determines whether the moving image capture has ended (step S15). If the exposure control unit 52 determines that the moving image capture has not ended (step S15: No), the process returns to step S11 and step S11 is executed again. On the other hand, if the exposure control unit 52 determines that the moving image capture has ended (step S15: Yes), the process ends.

[0163] As described above, when the subject is moving at a fast speed, the first exposure mode suitable for shooting fast-moving subjects is selected. For example, when a person is walking, or when a user holding the camera device 10 is walking, the second exposure mode is selected.

[0164] Figure 16 This is an example of exposure mode selection based on jitter level. For example... Figure 16 As shown, the exposure control unit 52 first determines whether motion image capture based on the camera sensor 20 has started (step S20). At this time, as the initial exposure mode, for example, the second exposure mode is used for motion image capture.

[0165] When the exposure control unit 52 determines that motion image capture has started (step S20: Yes), it acquires the jitter amount (step S21). For example, the exposure control unit 52 acquires the detected value of the jitter amount from the jitter detection sensor 44. In addition, the exposure control unit 52 can acquire the rotation amount calculated by the rotation amount calculation unit 68 and / or the displacement amount calculated by the displacement amount calculation unit 69 from the correction amount calculation unit 53 as the jitter amount.

[0166] Next, the exposure control unit 52 determines whether the acquired jitter amount is above a predetermined value (step S22). If the exposure control unit 52 acquires rotation and displacement as jitter amounts, it determines whether either or both of these amounts are above a predetermined value. If the exposure control unit 52 determines that the jitter amount is above a predetermined value (step S22: Yes), it selects a second exposure mode (step S23). On the other hand, if the exposure control unit 52 determines that the jitter amount is below a predetermined value (step S22: No), it selects a first exposure mode (step S24). The camera sensor 20 exposes using the exposure mode selected by the exposure control unit 52.

[0167] After step S23 or S24, the exposure control unit 52 determines whether the moving image capture has ended (step S25). If the exposure control unit 52 determines that the moving image capture has not ended (step S25: No), the process returns to step S21 and step S21 is executed again. On the other hand, if the exposure control unit 52 determines that the moving image capture has ended (step S25: Yes), the process ends.

[0168] As described above, when the amount of shaking applied to the camera device 10 is large, a second exposure mode suitable for shaking suppression is selected.

[0169] [Third Implementation Method]

[0170] Next, the third embodiment will be described. In the first embodiment, when m≥2, the first first frame F1(1) to the (m-1)th first frame F1(m-1) are discarded and not used to generate video data. In contrast, in the third embodiment, when m≥2, at least one of the first first frame F1(1) to the (m-1)th first frame F1(m-1) is acquired, and the acquired first frame is used for shake correction and / or focus adjustment. In addition, shake correction and focus adjustment are just examples; the acquired first frame may also be used for other functions such as scene recognition and subject recognition. The aspect that the first first frame F1(1) to the (m-1)th first frame F1(m-1) is not used to generate video data is the same as in the embodiment described above.

[0171] Figure 17 This represents an example of camera action when m=2. When m=2, the first frame F1(1) is used for shake correction or focus adjustment. When the first frame F1(1) is used for shake correction, the shake correction unit 46 performs shake correction based on the motion vector obtained from the difference between frames.

[0172] Furthermore, when the first frame F1(1) is used for shake correction, the main control unit 50 controls the lens drive control unit 34 based on the signal of the phase difference pixel contained in the first frame F1(1), thereby performing phase difference focusing.

[0173] The same applies to cases where m ≥ 3. For example, when m = 3, the first frame F1(1) and the second frame F1(2) are used for shake correction and / or focus adjustment.

[0174] As described above, in this embodiment, the first frame, which is not used to generate video data, is effectively utilized.

[0175] [Fourth Implementation Method]

[0176] Next, the fourth embodiment will be described. In the first embodiment, translational jitter and rotational jitter are corrected for the first frame synthesized by the frame synthesis unit 47. In contrast, in the fourth embodiment, translational jitter is corrected for the first frame, and rotational jitter is corrected for the second frame synthesized by the frame synthesis unit 47.

[0177] Figure 18 This illustrates the structure of the image processing unit 41 according to the fourth embodiment. For example... Figure 18 As shown, in this embodiment, the image processing unit 41 includes a first jitter correction unit 46A, a second jitter correction unit 46B, and a frame synthesis unit 47. The first jitter correction unit 46A is disposed at the front end of the frame synthesis unit 47. The second jitter correction unit 46B is disposed at the rear end of the frame synthesis unit 47. A displacement amount Cs is input to the first jitter correction unit 46A from the displacement calculation unit 69. A rotation amount Cr is input to the second jitter correction unit 46B from the rotation calculation unit 68.

[0178] Figure 19 Explain the jitter correction involved in the fourth embodiment. For example... Figure 19 As shown, the first jitter correction unit 46A performs translation jitter correction on the first frame based on the displacement Cs input from the displacement calculation unit 69. The frame synthesis unit 47 generates a second frame by synthesizing multiple first frames that have undergone translation jitter correction. The second jitter correction unit 46B performs rotation jitter correction on the second frame based on the rotation Cr input from the rotation calculation unit 68.

[0179] Translation and rotation jitter corrections require computational processing such as projection transformations or affine transformations. In particular, rotation jitter correction requires significantly more processing time than translation jitter correction, potentially preventing completion of the process within the first frame. In this embodiment, by performing rotation jitter correction on the second frame, which has a longer frame duration, rotation jitter correction can be reliably performed.

[0180] Furthermore, the first jitter correction unit 46A and the second jitter correction unit 46B can be configured to correct jitter components in specific frequency regions, respectively. Moreover, the frequency region of translational jitter corrected by the first jitter correction unit 46A can be different from the frequency region of rotational jitter corrected by the second jitter correction unit 46B. Additionally, the detection frequency region of translational jitter detected by the jitter detection sensor 44 can be different from the detection frequency region of rotational jitter detected by the jitter detection sensor 44.

[0181] [Fifth Implementation]

[0182] Next, the fifth embodiment will be described. In the fourth embodiment, the image processing unit 41 has one frame combining unit. In contrast, in the fifth embodiment, the image processing unit 41 has multiple frame combining units.

[0183] Figure 20 This illustrates the structure of the image processing unit 41 according to the fifth embodiment. For example... Figure 20 As shown, in this embodiment, the image processing unit 41 includes a first jitter correction unit 46A, a second jitter correction unit 46B, a first frame synthesis unit 47A, and a second frame synthesis unit 47B. The first jitter correction unit 46A is disposed before the first frame synthesis unit 47A. The second jitter correction unit 46B is disposed after the first frame synthesis unit 47A. The second frame synthesis unit 47B is disposed after the second jitter correction unit 46B. A displacement amount Cs is input to the first jitter correction unit 46A from the displacement calculation unit 69. A rotation amount Cr is input to the second jitter correction unit 46B from the rotation calculation unit 68.

[0184] When M < N (where M is a positive integer and N is a multiple of M), the first frame compositing unit 47A generates a composite frame by compositing M first frames. Similarly to the first embodiment, N is the value obtained by dividing the first frame rate by the second frame rate. The second frame compositing unit 47B generates a second frame by compositing N / M composite frames. For example, when N = 4 and M = 2, the first frame compositing unit 47A generates a composite frame by compositing 2 first frames, and the second frame compositing unit 47B generates a second frame by compositing 2 composite frames.

[0185] Figure 21 This explains the jitter correction involved in the fifth embodiment. For example... Figure 21 As shown, the first jitter correction unit 46A performs translation jitter correction on the first frame based on the displacement Cs input from the displacement calculation unit 69. The first frame synthesis unit 47A generates a composite frame by synthesizing multiple first frames that have undergone translation jitter correction. The second jitter correction unit 46B performs rotation jitter correction on the composite frame based on the rotation Cr input from the rotation calculation unit 68. The second frame synthesis unit 47B generates a second frame by synthesizing multiple composite frames that have undergone rotation jitter correction.

[0186] Furthermore, the first jitter correction unit 46A performs jitter correction on the first frame based on a first jitter amount applied to the camera device 10 that ends jitter correction within a first frame period (i.e., 1 / (V×N) seconds). The second jitter correction unit 46B performs jitter correction on the composite frame based on a second jitter amount applied to the camera device 10 that ends jitter correction within N / M times the first frame period (i.e., 1 / (V×M) seconds). In this embodiment, the first jitter amount is the amount of translational jitter, and the second jitter amount is the amount of rotational jitter.

[0187] In this embodiment, for example, it is preferable to set the first frame rate to 240fps and the second frame rate to 30fps. In this case, V=30, N=8, for example, M=2.

[0188] [Variation Example]

[0189] In the above embodiments, the amount of jitter applied to the camera device 10 is detected by the accelerometer 44B, but the amount of jitter can also be detected based on the motion vector obtained from the difference between frames. Furthermore, the amount of jitter can also be detected by calculating the spatial frequency, etc., from an image analysis of a single frame.

[0190] Furthermore, in the above embodiments, the image processing unit 41 is part of the processor 40, but the image processing unit 41 may be provided separately from the processor 40. The processor 40 and the image processing unit 41 are examples of processors according to the technology of the present invention.

[0191] Furthermore, the above embodiments can be combined with each other as long as they do not create contradictions.

[0192] In the above embodiments, the hardware structure of the control unit, taking processor 40 as an example, can use various processors as shown below. Among these various processors, in addition to a CPU, a commonly used processor that functions as an execution software (program), processors such as FPGAs, whose circuit structure can be changed after manufacturing, are also included. An FPGA includes dedicated circuitry, which is a processor with a circuit structure specifically designed to execute specific processes such as PLDs or ASICs.

[0193] The control unit can consist of one of these various processors, or it can consist of a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs or a combination of a CPU and an FPGA). Furthermore, multiple control units can be composed of a single processor.

[0194] Several examples can be considered regarding a single processor constituting multiple control units. In the first example, represented by client and server computers, there exists a combination of one or more CPUs and software to form a single processor, which functions as multiple control units. In the second example, represented by System-on-Chip (SOC), there exists a processor that uses a single IC chip to implement the overall system functionality, including multiple control units. Thus, the control unit can utilize one or more of the aforementioned processors to construct the hardware structure.

[0195] Furthermore, as the hardware structure of these various processors, more specifically, they can use circuits composed of circuit elements such as semiconductor elements.

[0196] The descriptions and illustrations above are detailed explanations of the parts involved in the technology of this invention, and are merely one example of the technology of this invention. For example, the descriptions related to the above structure, function, effect, and effect are examples of the structure, function, effect, and effect of the parts involved in the technology of this invention. Therefore, without departing from the technical spirit of this invention, unnecessary parts can be deleted from the descriptions and illustrations above, or new elements can be added or replaced. Furthermore, in order to avoid complexity and to facilitate understanding of the parts involved in the technology of this invention, descriptions related to technical common sense that do not require special explanation in aspects that enable the implementation of this invention have been omitted from the descriptions and illustrations above.

[0197] All documents, patent applications and technical standards set forth in this specification are incorporated herein by reference to the same extent as each specific and separately described document, patent application and technical standard incorporated herein by reference.

[0198] Symbol Explanation

[0199] 10-Camera unit, 11-Main body, 11A-Camera side mount, 11B-Electrical contact, 11C-Front end, 11D-Rear end, 12-Camera lens, 12A-Lens side mount, 12B-Electrical contact, 13-Operation dial, 14-Release button, 15-Display, 16-Indicator key, 18-Viewfinder eyepiece, 20-Camera sensor, 20A-Light receiving surface, 21-Camera area, 22-Vertical scanning circuit, 23-Line memory, 24-Horizontal scanning circuit, 24A-Horizontal output line, 25-Input line Output amplifier, 26-pixel, 29-transistor, 30-objective lens, 31-focusing lens, 32-rear lens, 34-lens drive control unit, 40-processor, 41-image processing unit, 42-operation unit, 44-shake detection sensor, 44A-gyroscope sensor, 44B-accelerometer sensor, 45-memory, 45A-camera program, 46-shake correction unit, 46A-first shake correction unit, 46B-second shake correction unit, 47-frame synthesis unit, 47A-first frame synthesis unit, 47B- Second frame synthesis unit, 50-main control unit, 51-camera control unit, 52-exposure control unit, 53-correction calculation unit, 61, 62, 63, 64, 65-signal processing units, 61A, 62A, 63A, 64A, 65A-subtractors, 61B, 62B, 63B, 64B, 65B-HPF, 61C, 62C, 63C, 64C, 65C-multipliers, 61D, 63D, 65D-integrators, 62D, 64D-first integrators, 62E, 64E-second integrators , 66, 67 - Adder, 68 - Rotation calculation unit, 69 - Displacement calculation unit, D1 - Photodiode, E1~E4 - Exposure time, F1(1)~F1(4) - First frame, F2 - Second frame, L1 - Row selection line, L2 - Row reset line, L3 - Column signal line, M1 - Amplifier transistor, M2 - Pixel selection transistor, M3 - Reset transistor, P - Unit time, RST - Reset signal, S - Pixel signal, SEL - Row selection signal, T1 - First frame period, T2 - Second frame period.

Claims

1. A camera device comprising a camera sensor and at least one processor, wherein the camera device, The processor is configured to perform the following processes: The camera processing acquires multiple first frames captured at a first frame rate in a first exposure mode using the camera sensor. Correction processing, in the plurality of first frames, performing electronic jitter correction based on the amount of jitter applied to the camera device; and The generation process involves combining the multiple first frames to generate video data of a second frame with a frame rate lower than that of the first frames. In the first exposure mode, the processor sets m to a positive integer, such that the exposure time in the m-th first frame constituting the second frame is shorter than the exposure time in the (m+1)-th first frame.

2. The camera device according to claim 1, wherein, The processor performs the following processing: Let P be the unit time corresponding to the shutter speed, V be the second frame rate, N be the value obtained by dividing the first frame rate by the second frame rate, and n be a positive integer. If we define the largest n that satisfies P < 1 / V - (n-1) / (V×N) as m, In the first exposure mode, the second frame is generated by combining the m-th to N-th first frames from the 1st to the Nth first frames, and the exposure time of the m-th first frame is shorter than the exposure time of the (m+1)-th first frame.

3. The camera device according to claim 2, wherein, The total exposure time in each of the m-th to N-th first frames is equal to the unit time.

4. The camera device according to claim 3, wherein, The exposure time in the m-th first frame is shorter than the frame period of the first frame, and the exposure time in each of the (m+1)-Nth first frames is equal to the frame period of the first frame.

5. The camera device according to claim 4, wherein, The exposure times in each of the m-th to N-th first frames are temporally continuous.

6. The camera device according to any one of claims 2 to 5, wherein, The processor performs the following processing: When m≥2, at least one of the first frames from the first to the (m-1)th frames that are not used in the generation of the video data is obtained.

7. The camera device according to claim 6, wherein, The processor uses the acquired plurality of first frames for the jitter correction.

8. The camera device according to claim 6, wherein, The camera sensor has phase difference pixels. The processor performs focusing based on the signals of the phase difference pixels contained in the acquired first frame.

9. The camera device according to claim 2, wherein, The processor is capable of executing a second exposure mode, which exposes the image by distributing the exposure time across each of the m-th to N-th first frames.

10. The camera device according to claim 9, wherein, The processor can selectively execute the first exposure mode and the second exposure mode.

11. The camera device according to claim 10, wherein, The processor performs the following processing: The moving speed of the subject is detected. If the moving speed is above a specified value, the first exposure mode is selected. If the moving speed is below the specified value, the second exposure mode is selected.

12. The camera device according to claim 10, wherein, The processor performs the following processing: If the jitter is less than a specified value, the first exposure mode is selected; if the jitter is greater than or equal to a specified value, the second exposure mode is selected.

13. The camera device according to claim 1, comprising: The vibration detection sensor detects rotational vibration in the tilt direction and translational vibration in a direction intersecting the rotation axis in the tilt direction. The processor performs translation jitter correction on the plurality of first frames and rotation jitter correction on the second frames.

14. The camera device according to claim 2, wherein, The processor performs the following processing: When M is set to a positive integer, N is set to a multiple of M, and M < N, Based on the first jitter amount that ends jitter correction at 1 / (V×N) seconds, jitter correction is performed on the plurality of first frames. Based on the second jitter amount, which ends jitter correction at 1 / (V×M) seconds, jitter correction is performed on the composite frame formed by combining the first frame into M frames. The second frame is generated by combining the synthesized frames into N / M frames.

15. The camera device according to claim 1, wherein, The processor performs the following processing: Of the jitter amount, the rotational jitter in the tilt direction is used to correct jitter in the second frame. The jitter amount includes translational jitter in the direction of intersection with the rotation axis that intersects the tilt direction, which corrects jitter in the plurality of first frames.

16. A driving method for a camera device equipped with a camera sensor, comprising: The camera process involves acquiring multiple first frames captured at a first frame rate in a first exposure mode using the camera sensor. The correction process involves performing electronic jitter correction based on the amount of jitter applied to the camera device in the plurality of first frames; and The generation process involves combining the multiple first frames to generate video data of a second frame with a frame rate lower than that of the first frames. In the first exposure mode, m is set to a positive integer such that the exposure time of the mth first frame constituting the second frame is shorter than the exposure time of the (m+1)th first frame.

17. A computer-readable medium having recorded a camera program that causes a camera device equipped with a camera sensor to operate, the camera program being configured to perform the following processing: The camera processing acquires multiple first frames captured at a first frame rate in a first exposure mode using the camera sensor. Correction processing, in the plurality of first frames, performing electronic jitter correction based on the amount of jitter applied to the camera device; and The generation process involves combining the multiple first frames to generate video data of a second frame with a frame rate lower than that of the first frames. In the first exposure mode, m is set to a positive integer such that the exposure time of the mth first frame constituting the second frame is shorter than the exposure time of the (m+1)th first frame.

Images