Image pickup element, image pickup device, image pickup element operation method, and program

Abstract

To provide an imaging element, an imaging apparatus, an operation method of the imaging element and a program which output second image data to which vibration information is given, the vibration information synchronized with first image data obtained by being captured by the imaging element.SOLUTION: An imaging element includes a memory 64 that stores first image data (digital image data) obtained by being captured by the imaging element and is incorporated in the imaging element, and a processing circuit 62 that is configured to perform image data processing on the first image data and is incorporated in the imaging element. The processing circuit 62 is configured to receive vibration information related to a vibration exerted on the imaging element within a frame output period defined by a first frame rate, and output second image data (vibration information embedded image data) obtained by assigning the vibration information to a specific position set in the first image data within the frame output period.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The technology of the present disclosure relates to an imaging element, an imaging device, an operation method for an imaging element, and a program. [Background technology]

[0002] Japanese Patent Application Laid-Open Publication No. 2018-011098 discloses a video processing device that performs processing to record video data generated by imaging and shake information related to shakes that occurred during imaging. The video processing device includes: an assigning means that assigns playback valid information indicating whether playback is possible or playback invalid information indicating whether playback is impossible to the video data as playback identification information indicating whether the video data can be played back; and a processing means that generates and records recording data including the video data, shake information, and playback identification information; the processing means generates first data including video data, shake information, and playback invalid information for a first period before the start of recording in accordance with recording start vibration information; generates second data including video data, shake information, and playback valid information for a second period from the start of recording to the end of recording in accordance with a recording end instruction; generates third data including video data, shake information, and playback invalid information for a third period after the end of recording; and generates recording data that integrates the first, second, and third data.

[0003] Japanese Patent Publication No. 2012-124614 discloses an imaging device that images a subject, generates moving image data, and records the data on a recording medium, and that is characterized by comprising: a device operation recording means that records operation information of the imaging device when recording the moving image; a camerawork determination means that determines the camerawork type corresponding to the operation information; a frame range identification means that determines the frame range in which the camerawork type has been determined from the moving image data; a camerawork management means that correlates the camerawork type with the frame range and records them as camerawork information; a target camerawork identification means that identifies a camerawork type that represents the moving image from the camerawork information according to a predetermined method; and a camerawork expression information creation means that creates information that represents the camerawork type identified by the target camerawork identification means when displaying a list of representative images of the moving image.

[0004] Japanese Patent Application Laid-Open Publication No. 2016-208483 discloses an imaging system and an aerial photography system using the same, which has a synchronization holding means consisting of a Gyro sensor, holding means for holding Gyro values from the Gyro sensor, sequence detection means for receiving a horizontal synchronization signal from an image sensor and obtaining a sequence H number for the frame of this synchronization signal, and pairing holding means for pairing and holding at least one pair of this sequence H number and Gyro value, and which identifies the position of image blur using the image from the image sensor, the camera parameters of the image sensor, and blur angle information from the Gyro value, and returns the video data at that position to its original position, thereby correcting image blur and obtaining high-quality video. Summary of the Invention

[0005] One embodiment of the technology of the present disclosure provides an imaging element, an imaging device, an operating method of the imaging element, and a program that can output second image data to which vibration information synchronized with first image data obtained by capturing an image using the imaging element is added. [Means for solving the problem]

[0006] A first aspect of the technology of the present disclosure is an imaging element comprising: a memory that stores first image data obtained by imaging by the imaging element and is built into the imaging element; and a first processor that performs image data processing on the first image data and is built into the imaging element, wherein the first processor receives vibration information related to vibrations applied to the imaging element within a frame output period defined by a first frame rate, and outputs second image data to which the vibration information has been added at a specific position set in the first image data within the frame output period.

[0007] A second aspect of the technology of the present disclosure is an imaging element according to the first aspect, in which the first image data is pixel line data consisting of multiple lines, and the specific position is a position set in at least one line of pixel line data.

[0008] A third aspect according to the technique of the present disclosure is the image sensor according to the second aspect, in which the specific position is a position provided in each of the pixel line data of the plurality of lines.

[0009] A fourth aspect of the technology of the present disclosure is an imaging element according to the second aspect, in which the first processor receives partial area designation information that designates a partial area of the first image data, and receives vibration information during the exposure period of a pixel line that corresponds to the partial area among multiple pixel lines included in the imaging element, and the specific position is a position set in the pixel line data that corresponds to the partial area.

[0010] A fifth aspect of the technology of the present disclosure is an imaging element according to the second aspect, in which the first processor receives vibration information during an exposure period of a central pixel line located at the center of the imaging element among multiple pixel lines included in the imaging element, and the specific position is a position set in the pixel line data corresponding to the central pixel line.

[0011] A sixth aspect of the technology of the present disclosure is an imaging element according to the fourth or fifth aspect, in which the specific position is a position set in pixel line data for a pixel line, among multiple pixel lines included in the imaging element, that was exposed during a period closest to the period during which vibration information was acquired.

[0012] A seventh aspect of the technique of the present disclosure is the image sensor according to any one of the second to sixth aspects, in which the specific position is the beginning or end of pixel line data.

[0013] An eighth aspect of the technology of the present disclosure is an imaging element according to any one of the first to seventh aspects, in which the first processor determines whether or not to add vibration information to the first image data depending on the value of the vibration information.

[0014] A ninth aspect according to the technique of the present disclosure is the image sensor according to the eighth aspect, wherein the first processor adds vibration information to the first image data when the vibration information exceeds a threshold value.

[0015] A tenth aspect of the technology of the present disclosure is an imaging element according to any one of the first to ninth aspects, in which the vibration information is at least one of angular velocity, acceleration, integral value of angle, integral value of acceleration, and shake correction amount.

[0016] An eleventh aspect of the technology of the present disclosure is an imaging element according to any one of the first to tenth aspects, in which the imaging element is an imaging element in which at least a photoelectric conversion element and a memory are integrated into a single chip.

[0017] A twelfth aspect according to the technique of the present disclosure is the imaging element according to the eleventh aspect, in which the imaging element is a stacked imaging element in which a photoelectric conversion element and a memory are stacked.

[0018] A thirteenth aspect of the technology of the present disclosure is an imaging element according to any one of the first to twelfth aspects, in which the first processor performs shake correction processing on the second image data based on vibration information assigned to the second image data within a frame output period.

[0019] A fourteenth aspect of the technique of the present disclosure is the image sensor according to the thirteenth aspect, wherein the first processor performs the shake correction process using a mean value, a median value, or a mode value of the vibration information.

[0020] A 15th aspect of the technology of the present disclosure is an imaging element according to any one of the first to fourteenth aspects, in which a first processor performs image data processing on first image data captured at a second frame rate higher than the first frame rate, and the image data processing includes an acquisition process for acquiring the first image data, a reception process for receiving vibration information, a generation process for generating second image data by adding the vibration information to the first image data, and a shake correction process for correcting shake in the second image data based on the vibration information added to the second image data.

[0021] A 16th aspect of the technology of the present disclosure is an image sensor according to the 15th aspect, in which the first processor generates and outputs one frame of third image data by combining multiple frames of second image data after shake correction processing within a frame output period.

[0022] A 17th aspect of the technology of the present disclosure is an imaging device comprising an imaging element described in any one of the first to sixteenth aspects, and a second processor provided downstream of the imaging element to which second image data is input from the imaging element, wherein the second image data includes a plurality of lines of pixel line data, each of which has vibration information attached thereto, and the second processor obtains specific subject position information indicating the position of a specific subject based on the second image data, and performs shake correction processing on the second image data based on the vibration information attached to pixel line data corresponding to the specific subject position information among the plurality of lines of pixel line data.

[0023] An 18th aspect of the technology of the present disclosure is a method for operating an imaging element that includes a memory that stores first image data obtained by imaging by the imaging element and is built into the imaging element, and a processor that processes the first image data and is built into the imaging element, the method including receiving vibration information related to vibrations applied to the imaging element within a frame output period defined by a first frame rate, and outputting second image data in which the vibration information is added to the first image data within the frame output period.

[0024] A 19th aspect of the technology of the present disclosure is a program for causing a computer applied to an imaging element, the computer including a memory that stores first image data obtained by imaging by the imaging element and is built into the imaging element, and a processor that processes the first image data and is built into the imaging element, to execute processing including receiving vibration information regarding vibrations applied to the imaging element within a frame output period defined by a first frame rate, and outputting second image data in which the vibration information has been added to the first image data within the frame output period. [Brief explanation of the drawings]

[0025] [Figure 1] FIG. 1 is a perspective view showing an example of the appearance of an imaging device according to first to fifth embodiments. [Figure 2] 1 is a block diagram showing an example of the configuration of an imaging device according to first to third embodiments. [Figure 3A] 2 is a conceptual diagram illustrating an imaging frame rate of an imaging element included in the imaging device according to the first to fifth embodiments. FIG. [Figure 3B] 2 is a conceptual diagram illustrating the output frame rate of an image sensor included in the image pickup device according to the first to fifth embodiments. FIG. [Figure 4] 2 is a block diagram showing an example of the electrical configuration of the imaging device main body according to the first embodiment. FIG. [Figure 5] 1 is a block diagram showing an example of a layered structure of an imaging element included in an imaging device according to any one of the first to fifth embodiments, and an example of the connection relationship between the imaging element, a signal processing unit, and a controller. [Figure 6] 1 is a diagram showing an example of the arrangement on an imaging surface of pixels included in a photoelectric conversion element of an imaging element included in an imaging device according to any one of the first to fifth embodiments. [Figure 7] 2 is a block diagram showing an example of the configuration of an electrical system of an imaging element included in the imaging device according to the first embodiment. FIG. [Figure 8]3 is a conceptual diagram showing an example of vibration information embedded image data generated by an imaging element included in the imaging device according to the first embodiment. FIG. [Figure 9] 10 is a time chart showing an example of an exposure period, vibration information acquisition timing, pixel line data acquisition timing, vibration information embedding timing, shake correction processing timing, and shake corrected image data storage timing within an imaging element included in an imaging device according to the first embodiment. [Figure 10A] 10 is a flowchart showing an example of the flow of a shake-corrected image data generation process according to the first embodiment. [Figure 10B] 10 is a flowchart showing an example of the flow of a shake-corrected image data generation process according to the first embodiment. [Figure 11] 10 is a conceptual diagram illustrating partial area designation information input by a user in an imaging device according to a second embodiment. FIG. [Figure 12] FIG. 10 is a block diagram showing an example of the electrical configuration of an imaging device main body according to a second embodiment. [Figure 13] FIG. 10 is a block diagram showing an example of the configuration of an electrical system of an imaging element included in an imaging device according to a second embodiment. [Figure 14] 10 is a conceptual diagram illustrating a pixel line including a partial area indicated by partial area designation information in vibration information embedded image data generated by an imaging element included in an imaging device according to a second embodiment. FIG. [Figure 15] 10 is a conceptual diagram illustrating a central pixel line in vibration information embedded image data generated by an imaging element included in an imaging device according to a second embodiment. FIG. [Figure 16] 10 is a flowchart showing an example of the flow of a shake-corrected image data generation process according to the second embodiment. [Figure 17] FIG. 11 is a block diagram showing an example of the configuration of an electrical system of an imaging element included in an imaging device according to a third embodiment. [Figure 18] 10 is a conceptual diagram illustrating vibration information embedded image data generated by an imaging element included in an imaging device according to a third embodiment. FIG. [Figure 19] 11 is a flowchart showing an example of the flow of a shake-corrected image data generation process according to the third embodiment. [Figure 20] FIG. 10 is a block diagram showing an example of the configuration of an imaging device according to a fourth embodiment. [Figure 21] FIG. 10 is a block diagram showing an example of the configuration of an electrical system of an imaging element included in an imaging device according to a fourth embodiment. [Figure 22] FIG. 13 is a conceptual diagram illustrating an example of the flow of a composite image data generation process according to the fourth embodiment. [Figure 23A] 13 is a flowchart showing an example of the flow of a composite image data generation process according to the fourth embodiment. [Figure 23B] 13 is a flowchart showing an example of the flow of a composite image data generation process according to the fourth embodiment. [Figure 24] FIG. 13 is a conceptual diagram showing an example of how a shake-corrected image data generation program is executed in an imaging device according to a fifth embodiment. [Figure 25] 13 is a conceptual diagram illustrating a pixel line including a specific subject indicated by specific subject position information in vibration information embedded image data generated by an imaging element included in an imaging device according to a fifth embodiment. FIG. [Figure 26] FIG. 11 is a conceptual diagram showing an example of how a shake-corrected image data generation program is installed from a storage medium storing the program to a computer in an imaging apparatus according to a fifth embodiment. [Figure 27] FIG. 10 is a conceptual diagram showing an example of how a program is installed from a storage medium on which the program is stored to a computer in an imaging element. DETAILED DESCRIPTION OF THE INVENTION

[0026] An example of an embodiment of an imaging device according to the technique of the present disclosure will now be described with reference to the accompanying drawings.

[0027] First, the meanings of terms used in the following description will be explained.

[0028] CPU is an abbreviation for "Central Processing Unit." RAM is an abbreviation for "Random Access Memory." ROM is an abbreviation for "Read Only Memory." DRAM is an abbreviation for "Dynamic Random Access Memory." SRAM is an abbreviation for "Static Random Access Memory."

[0029] LSI is an abbreviation for "Large-Scale Integrated Circuit." IC is an abbreviation for "Integrated Circuit." ASIC is an abbreviation for "Application Specific Integrated Circuit." PLD is an abbreviation for "Programmable Logic Device." FPGA is an abbreviation for "Field-Programmable Gate Array."

[0030] SSD is an abbreviation for "Solid State Drive." USB is an abbreviation for "Universal Serial Bus."

[0031] CCD is an abbreviation for "Charge Coupled Device". CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor". EL is an abbreviation for "Electro-Luminescence". A / D is an abbreviation for "Analog / Digital". I / F is an abbreviation for "Interface". UI is an abbreviation for "User Interface". AF is an abbreviation for "Auto-Focus". AE is an abbreviation for "Automatic Exposure". SoC is an abbreviation for "System-on-a-chip". OIS is an abbreviation for "Optical Image Stabilizer".

[0032] [First embodiment] 1, the imaging device 10 is an interchangeable lens camera. The imaging device 10 includes an imaging device body 12 and an interchangeable lens 14 that is interchangeably attached to the imaging device body 12.

[0033] The imaging device body 12 is provided with an imaging element 44. When the interchangeable lens 14 is attached to the imaging device body 12, subject light representing the subject passes through the interchangeable lens 14 and forms an image on the imaging surface 44A of the imaging element 44.

[0034] A release button 20 and a dial 22 are provided on the top surface of the imaging device body 12. The dial 22 is operated when setting the operation mode of the imaging system and the operation mode of the playback system, etc. The release button 20 functions as an imaging preparation instruction unit and an imaging instruction unit, and is capable of detecting two pressing stages: an imaging preparation instruction state and an imaging instruction state. The imaging preparation instruction state refers to a state in which the button is pressed from the standby position to an intermediate position (half-pressed position), for example, and the imaging instruction state refers to a state in which the button is pressed beyond the intermediate position to a final pressed position (fully-pressed position). Note that, hereinafter, the "state in which the button is pressed from the standby position to the half-pressed position" is referred to as the "half-pressed state," and the "state in which the button is pressed from the standby position to the fully-pressed position" is referred to as the "fully-pressed state." Depending on the configuration of the imaging device, the imaging preparation instruction state may be a state in which the operating user's finger is in contact with the release button 20, and the imaging instruction state may be a state in which the operating user's finger has moved from a state in which the operating user's finger is in contact with the release button 20 to a state in which the operating user's finger is released.

[0035] In the imaging device 10, an imaging mode or a playback mode is selectively set as an operating mode in response to a user instruction. The imaging modes are broadly divided into a display moving image imaging mode and a recording imaging mode. In each of the display moving image imaging mode and the recording imaging mode, an AF mode is set in response to a user instruction.

[0036] In the image capture mode for display moving images, when the AF mode is set as the focus adjustment setting of the interchangeable lens 14, the AE function is activated to set the exposure state for each frame, and the AF function is activated to control focus, thereby capturing an image for the display moving images. A live view image is generated by capturing an image for the display moving images. Note that the live view image is also generally referred to as a through image. Note that the AE function and AF function do not necessarily have to be performed for each frame.

[0037] The recording imaging mode is broadly divided into a moving image recording imaging mode and a still image recording imaging mode, and the moving image recording imaging mode and the still image recording imaging mode are selectively set in response to a user instruction. In the imaging device 10, when the AF mode is set in the moving image recording imaging mode, the AE function is activated to set the exposure state for each frame, and the AF function is activated to control focus, thereby capturing moving images to be recorded. The moving images obtained by capturing moving images to be recorded are recorded on a predetermined storage medium such as a memory card or USB memory.

[0038] In the still image recording imaging mode, when the AF mode is set, the imaging conditions are adjusted by pressing the release button 20 halfway, and then the still image is captured by pressing it all the way down. In other words, when the release button 20 is pressed halfway, the AE function is activated to set the exposure, and then the AF function is activated to control focus, and when the release button 20 is pressed all the way down, the still image is captured.

[0039] As an example, as shown in FIG. 2, the interchangeable lens 14 has an imaging lens 40. The imaging lens 40 is equipped with an objective lens 40A, a focus lens 40B, and an aperture 40C. The objective lens 40A, the focus lens 40B, and the aperture 40C are arranged in this order along the optical axis L1 from the subject side to the imaging device body 12 side. The aperture 40C is operated by receiving power from a drive source (not shown) such as a motor. This changes the aperture size of the aperture 40C. Changing the aperture size of the aperture 40C adjusts the exposure.

[0040] The focus lens 40B is attached to a slide mechanism (not shown) and moves along the optical axis L1 in response to power provided by a drive source (not shown), such as a motor. In AF mode, the focus lens 40B moves along the optical axis L1 under the control of the controller 46 to reach a focus position corresponding to the subject distance. The "focus position" here refers to the position of the focus lens 40B on the optical axis L1 when the subject is in focus. When the focus lens 40B reaches the focus position corresponding to the subject distance, the subject light is imaged on the imaging surface 44A of the image sensor 44 by the imaging lens 40.

[0041] The imaging device body 12 includes a mechanical shutter 42 and an imaging element 44. The mechanical shutter 42 operates by receiving power from a drive source (not shown) such as a motor. When the interchangeable lens 14 is attached to the imaging device body 12, subject light passes through the imaging lens 40 and is focused on the imaging surface 44A of the imaging element 44 via the mechanical shutter 42.

[0042] The imaging device main body 12 is equipped with a vibration sensor 47. The vibration sensor 47 is a device including a gyro sensor, and detects, as vibration, an angular velocity given to the imaging element 44. Examples of vibrations given to the imaging element 44 include vibrations given to the imaging element 44 by a user holding the imaging device 10, vibrations caused by wind on the imaging device 10 placed on a support such as a tripod, and vibrations given from a vehicle.

[0043] The gyro sensor detects angular velocities around each of the pitch axis PA, yaw axis YA, and roll axis RA (see FIG. 1). The vibration sensor 47 converts the angular velocities around the pitch axis PA and the yaw axis YA detected by the gyro sensor into angular velocities in a two-dimensional plane parallel to the pitch axis PA and the yaw axis YA, thereby detecting vibrations applied to the imaging element 44. Note that the meaning of "parallel" in the first embodiment of the technology disclosed herein includes not only perfect parallelism but also approximately parallelism that includes tolerances allowed in design and manufacturing.

[0044] The vibration sensor 47 is connected to the controller 46 via a communication line 58, and detects vibrations applied to the image sensor 44 under the control of the controller 46. The vibration sensor 47 is also connected to the image sensor 44 via a communication line 52, and outputs vibration information indicating the detected vibrations to the image sensor 44. Here, angular velocity is used as an example of the vibration information. Note that the vibration information may be data such as angular velocity, or may be information that has been processed by some kind of processing.

[0045] The imaging device main body 12 includes a controller 46, a UI device 48, and a signal processing unit 50. The controller 46 and the signal processing unit 50 are each realized by an LSI. Furthermore, since the controller 46 and the signal processing unit 50 are each located after the imaging element 44, they can also be considered as subsequent circuits of the imaging element 44.

[0046] The controller 46 controls the entire imaging device 10. The UI device 48 is a device that presents information to the user and receives instructions from the user. The UI device 48 is connected to the controller 46, and the controller 46 acquires various types of information from the UI device 48 and controls the UI device 48.

[0047] The image pickup device 44 is connected to the controller 46 via a communication line 57, and under the control of the controller 46, captures an image of the subject to generate image data representing the image of the subject.

[0048] The image sensor 44 is connected to a signal processing unit 50 via a communication line 53. The signal processing unit 50 is a device including an ASIC. The controller 46 is connected to the signal processing unit 50 via a communication line 60.

[0049] Image data is input to the signal processing unit 50 from the image sensor 44 via a communication line 53. The signal processing unit 50 performs various signal processing on the image data. The various signal processing includes, for example, well-known signal processing such as white balance adjustment, sharpness adjustment, gamma correction, color space conversion processing, and color difference correction.

[0050] In the first embodiment, a device including an ASIC is exemplified as the signal processing unit 50, but the technology of the present disclosure is not limited to this, and the signal processing unit 50 may be a device including an ASIC, an FPGA, and / or a PLD. Furthermore, the signal processing unit 50 may be a computer including a CPU, a ROM, and a RAM. There may be one or more CPUs. Furthermore, the signal processing unit 50 may be realized by a combination of a hardware configuration and a software configuration.

[0051] The imaging element 44 is an example of a "stacked imaging element" according to the technology of the present disclosure. In the present embodiment, the imaging element 44 is a CMOS image sensor. Although a CMOS image sensor is exemplified as the imaging element 44 here, the technology of the present disclosure is not limited thereto, and the technology of the present disclosure also applies even if the imaging element 44 is a CCD image sensor, for example.

[0052] The image sensor 44 captures an image of the subject at the image capture frame rate, thereby generating multiple frames of image data, each representing an image of the subject, as shown in FIG. 3A as an example. The multiple frames of image data generated by the image sensor 44 are output at the output frame rate. Both the image capture frame rate and the output frame rate are variable frame rates. The output frame rate is an example of a "first frame rate" according to the technology of the present disclosure, and the image capture frame rate is an example of a "second frame rate" according to the technology of the present disclosure.

[0053] The imaging frame rate and the output frame rate have a relationship of "imaging frame rate ≥ output frame rate." In other words, the imaging frame rate is a frame rate that is higher than the output frame rate. For example, as shown in FIG. 3A, the imaging frame rate is a frame rate at which eight frames are captured within a period T, and the output frame rate is a frame rate at which two frames are output within a period T, as shown in FIG. 3B. Specifically, an example of the imaging frame rate is 240 fps (frames per second), and an example of the output frame rate is 60 fps.

[0054] 4, the controller 46 includes a CPU 46A, a storage 46B, a memory 46C, a first communication I / F 46D1, and a second communication I / F 46D2. The CPU 46A, the storage 46B, the memory 46C, the first communication I / F 46D1, and the second communication I / F 46D2 are connected to one another via a bus line 88.

[0055] The storage 46B stores a control program for the imaging device 10. The CPU 46A reads the control program from the storage 46B and loads the read control program into the memory 46C. The CPU 46A controls the entire imaging device 10 in accordance with the control program loaded into the memory 46C.

[0056] The first communication I / F 46D1 and the second communication I / F 46D2 are each a communication device having an FPGA. The first communication I / F 46D1 and the second communication I / F 46D2 employ, for example, the PCI-e connection standard. The first communication I / F 46D1 is connected to the signal processing unit 50 via a communication line 60. Image data that has been subjected to various signal processes by the signal processing unit 50 is input to the first communication I / F 46D1 via the communication line 60. The first communication I / F 46D1 transfers the image data input from the signal processing unit 50 to the CPU 46A.

[0057] The second communication I / F 46D2 is connected to the imaging element 44 via a communication line 57. The CPU 46A controls the imaging element 44 via the second communication I / F 46D2. The second communication I / F 46D2 is also connected to the vibration sensor 47 via a communication line 58. The CPU 46A controls the vibration sensor 47 via the second communication I / F 46D2.

[0058] The UI device 48 includes a touch panel display 26 and a reception device 84. The display 32 is connected to a bus line 88. An example of the display 32 is a liquid crystal display. The display 32 may not be a liquid crystal display, but may be another type of display such as an organic EL display. Under the control of the CPU 46A, the display 32 displays various images such as live view images and still images, as well as text information.

[0059] The accepting device 84 includes a hard key unit 25 and a touch panel 34. The hard key unit 25 is a plurality of hard keys including a release button 20 and a dial 22. The touch panel 34 is a transparent touch panel that is overlaid on the surface of the display area of the display 32. The touch panel 34 detects contact with a pointing object such as a finger or a stylus pen. The hard key unit 25 and the touch panel 34 are connected to a bus line 88, and the CPU 46A operates in accordance with various instructions accepted by the hard key unit 25 and the touch panel 34, respectively.

[0060] 5, the imaging element 44 incorporates a photoelectric conversion element 61, a processing circuit 62, and a memory 64. The imaging element 44 is an imaging element in which the photoelectric conversion element 61, the processing circuit 62, and the memory 64 are integrated into a single chip. That is, the photoelectric conversion element 61, the processing circuit 62, and the memory 64 are integrated into a single package. In the imaging element 44, the processing circuit 62 and the memory 64 are stacked on the photoelectric conversion element 61. Specifically, the photoelectric conversion element 61 and the processing circuit 62 are electrically connected to each other by conductive bumps (not shown) such as copper, and the processing circuit 62 and the memory 64 are also electrically connected to each other by conductive bumps (not shown) such as copper.

[0061] The processing circuit 62 is, for example, an LSI, and the memory 64 is, for example, a DRAM. However, the technology of the present disclosure is not limited to this, and SRAM may be adopted as the memory 64 instead of DRAM.

[0062] The processing circuit 62 is a device including an ASIC and an FPGA, and controls the entire image sensor 44 according to instructions from the controller 46. Note that, although an example in which the processing circuit 62 is realized by a device including an ASIC and an FPGA is given here, the technology of the present disclosure is not limited thereto, and may be, for example, a device including an ASIC, an FPGA, and / or a PLD. Furthermore, a computer including a CPU, storage, and memory may be adopted as the processing circuit 62. There may be one or more CPUs. Furthermore, the processing circuit 62 may be realized by a combination of a hardware configuration and a software configuration.

[0063] The photoelectric conversion element 61 has a large number of photodiodes arranged in a matrix. An example of the plurality of photodiodes is photodiodes for "4896 x 3265" pixels.

[0064] A color filter is arranged on each photodiode included in the photoelectric conversion element 61. The color filters include a G filter corresponding to G (green), which contributes most to obtaining a luminance signal, an R filter corresponding to R (red), and a B filter corresponding to B (blue). The photoelectric conversion element 61 has R pixels, G pixels, and B pixels (see FIG. 6). The R pixels are pixels corresponding to photodiodes in which the R filters are arranged, the G pixels are pixels corresponding to photodiodes in which the G filters are arranged, and the B pixels are pixels corresponding to photodiodes in which the B filters are arranged.

[0065] The image sensor 44 has a so-called electronic shutter function, and by operating the electronic shutter function under the control of the controller 46, it controls the charge accumulation time of each photodiode in the photoelectric conversion element 61. The charge accumulation time refers to the so-called shutter speed.

[0066] The imaging device 10 uses a rolling shutter system to capture still images and moving images. In the still image recording imaging mode, still image capture is achieved by activating the electronic shutter function and operating the mechanical shutter 42 (see FIG. 2). In the still image recording imaging mode, continuous shooting can also be achieved by activating the electronic shutter function without operating the mechanical shutter 42. In the moving image recording imaging mode, moving image capture is also achieved by activating the electronic shutter function without operating the mechanical shutter 42. Furthermore, in the display moving image imaging mode, live view image capture is also achieved by activating the electronic shutter function without operating the mechanical shutter 42. Note that while the rolling shutter system is exemplified here as a system for realizing the electronic shutter function, the technology of the present disclosure is not limited to this, and a global shutter system may be applied instead of the rolling shutter system.

[0067] The processing circuit 62 reads out image data obtained by capturing an image of a subject with the photoelectric conversion element 61. The image data is signal charges accumulated in the photoelectric conversion element 61. The processing circuit 62 performs A / D conversion on the analog image data read out from the photoelectric conversion element 61. The processing circuit 62 stores digital image data obtained by performing A / D conversion on the analog image data in the memory 64. Here, the processing circuit 62 is an example of a "first processor" according to the technology of the present disclosure, and the memory 64 is an example of a "memory" according to the technology of the present disclosure. The digital image data is an example of "first image data" according to the technology of the present disclosure.

[0068] The processing circuit 62 is connected to the signal processing unit 50 via a communication line 53. The processing circuit 62 is also connected to the second communication I / F 46D2 of the controller 46 via a communication line 57.

[0069] The memory 64 has a plurality of storage areas, each of which stores digital image data for one frame, for example. The processing circuit 62 randomly accesses the plurality of storage areas.

[0070] As an example, as shown in Fig. 6, on the imaging surface 44A of the photoelectric conversion element 61, R pixels, G pixels, and B pixels are arranged with a predetermined periodicity in each of the row direction (horizontal direction) and column direction (vertical direction). In the first embodiment, the R pixels, G pixels, and B pixels are arranged with a periodicity corresponding to the X-Trans (registered trademark) array. Note that although the example shown in Fig. 6 illustrates an X-Trans array, the technology of the present disclosure is not limited to this, and the array of the R pixels, G pixels, and B pixels may be a Bayer array, a honeycomb array, or the like.

[0071] The image sensor 44 has a plurality of pixel lines 61A. Each pixel line 61A has R pixels, G pixels, and B pixels arranged along the row direction. The plurality of pixel lines 61A are an example of the "plurality of pixel lines" according to the technology of the present disclosure.

[0072] As an example, as shown in FIG. 7, the processing circuit 62 includes a readout circuit 62A, a digital processing circuit 62B, a control circuit 62C, and an output circuit 62D.

[0073] The readout circuit 62A is connected to the photoelectric conversion element 61, the digital processing circuit 62B, and the control circuit 62C. The memory 64 is connected to the control circuit 62C. The output circuit 62D is also connected to the control circuit 62C. The control circuit 62C is connected to the controller 46 via a communication line 57, and is connected to the vibration sensor 47 via a communication line 52. The output circuit 62D is connected to the signal processing unit 50 via a communication line 53.

[0074] The controller 46 supplies timing control signals to the control circuit 62C via a communication line 57. The timing control signals include a vertical synchronization signal that defines the exposure period in units of one frame.

[0075] The readout circuit 62A reads out the analog image data obtained by capturing an image of a subject at the image capturing frame rate in units of horizontal pixel lines. That is, under the control of the control circuit 62C, the readout circuit 62A controls the photoelectric conversion element 61 to read out the analog image data from the photoelectric conversion element 61 in units of horizontal pixel lines.

[0076] The readout circuit 62A performs analog signal processing on the analog image data read out from the photoelectric conversion element 61. Analog signal processing includes well-known processes such as correlated double sampling, analog gain processing, and noise cancellation. Correlated double sampling is a process for reducing thermal noise in the analog image data. Analog gain processing is a process for applying gain to the analog image data. Noise cancellation is a process for canceling noise caused by variations in characteristics between pixels included in the photoelectric conversion element 61. The analog image data that has undergone this analog signal processing is output by the readout circuit 62A to the digital processing circuit 62B.

[0077] The digital processing circuit 62B includes an A / D converter 62B1. The digital processing circuit 62B performs digital signal processing on the analog image data input from the readout circuit 62A. The digital signal processing includes, for example, A / D conversion by the A / D converter 62B1 and digital gain processing.

[0078] The analog image data is subjected to A / D conversion by the A / D converter 62B1, whereby the analog image data is digitized to obtain digital image data. The digital image data is then subjected to digital gain processing by the digital processing circuit 62B. Digital gain processing refers to processing that applies gain to the digital image data.

[0079] The control circuit 62C acquires the digital image data obtained by the digital signal processing from the digital processing circuit 62B in units of horizontal pixel lines. Here, the analog signal processing performed by the readout circuit 62A and the digital signal processing performed by the digital processing circuit 62B are an example of "image data processing" according to the technique of the present disclosure.

[0080] The controller 46 supplies a vertical synchronization signal to the vibration sensor 47 via the communication line 58. The vibration sensor 47 receives the vertical synchronization signal and detects the angular velocity applied to the image sensor 44 during the exposure period of one frame of image data for each horizontal pixel line. The vibration sensor 47 supplies the angular velocity to the control circuit 62C. The angular velocity is an example of "vibration information" according to the technology of the present disclosure.

[0081] The control circuit 62C includes a vibration information embedding unit 62C1 and a shake correction unit 62C2. The vibration information embedding unit 62C1 receives vibration information from the vibration sensor 47 within a frame output period defined by the output frame rate. The vibration information embedding unit 62C1 then embeds vibration information at a specific position in the digital image data within the frame output period and outputs the information to the memory 64.

[0082] As an example, as shown in Figure 8, one frame of digital image data consists of N lines of pixel line data, including first pixel line data, second pixel line data, ..., Nth pixel line data. Here, the one horizontal pixel line constituting the top line of the photoelectric conversion elements 61 is referred to as the first pixel line, and the pixel data read out from the first pixel line is referred to as the first pixel line data. The one horizontal pixel line constituting the second line of the photoelectric conversion elements 61 is referred to as the second pixel line, and the pixel data read out from the second pixel line is referred to as the second pixel line data. Similarly, hereinafter, the Nth horizontal pixel line, which is the bottommost line, is referred to as the Nth pixel line, and the pixel data read out from the Nth pixel line is referred to as the Nth pixel line data.

[0083] Note that when there is no need to distinguish between the first, second, ..., Nth pixel lines, they will simply be referred to as "pixel lines." Furthermore, when there is no need to distinguish between the first, second, ..., Nth pixel line data, they will simply be referred to as "pixel line data." Here, pixel line data is an example of "pixel line data" according to the technology of the present disclosure.

[0084] The vibration information embedding unit 62C1 embeds pixel line vibration information at the beginning of the pixel line data. Note that the pixel line vibration information refers to information indicating vibration acquired by the vibration sensor 47 during the exposure period of one pixel line, and more specifically, refers to vibration information input from the vibration sensor 47 to the image sensor 44 during the exposure period of one pixel line. A bit area for embedding corresponding pixel line vibration information is provided in advance at the beginning of each pixel line data. Note that the beginning of each pixel line data is an example of a "specific position" according to the technology of the present disclosure.

[0085] The vibration information embedding unit 62C1 stores the pixel line data in which the pixel line vibration information is embedded in a corresponding storage area of the memory 64.

[0086] The memory 64 is a memory capable of storing vibration information-embedded image data of multiple frames. The pixel line data in which pixel line vibration information is embedded is stored in the memory 64 on a frame-by-frame basis as vibration information-embedded image data. The vibration information-embedded image data is an example of the "second image data" according to the technology of the present disclosure.

[0087] The control circuit 62C can randomly access the memory 64, and during a frame output period, reads vibration information-embedded image data from the memory 64 and supplies it to the shake correction unit 62C2. Note that the frame output period is an example of a "frame output period" according to the technology of the present disclosure. Here, the frame output period refers to the time interval between frames whose output timing, defined by the output frame rate, is adjacent on the time axis.

[0088] The shake correction unit 62C2 reads N pieces of pixel line vibration information from the vibration information-embedded image data and calculates the average value of the pixel line vibration information. The shake correction unit 62C2 performs a known shake correction process on the vibration information-embedded image data based on the calculated average value. The shake correction unit 62C2 stores the image data after the shake correction process in the memory 64 as shake-corrected image data within the frame output period. Note that the shake correction process is an example of the "shake correction process" according to the technology of the present disclosure.

[0089] The control circuit 62C reads out the shake-corrected image data from the memory 64 within the frame output period, and outputs it to the output circuit 62D. The output circuit 62D outputs the input shake-corrected image data to the signal processing unit .

[0090] 9, the vertical synchronization signal falls, for example, four times within one frame output period defined by the output frame rate. The exposure period of the image sensor 44 starts in synchronization with the falling edges of the vertical synchronization signal. Therefore, according to the first embodiment, four frames of digital image data are acquired within one frame output period.

[0091] The photoelectric conversion element 61 is reset in synchronization with the falling edge of the vertical synchronization signal, and exposure begins. Since the imaging device 10 employs a rolling shutter system, exposure begins for each horizontal pixel line at a fixed time interval. That is, within the exposure period for one frame shown in the shaded area of Figure 9, the topmost region 90 extending along the time axis represents the exposure period for the first pixel line, and the bottommost region 92 represents the exposure period for the Nth pixel line.

[0092] At the center of a predetermined exposure period, the vibration sensor 47 acquires vibration information for each horizontal pixel line. The vibration sensor 47 sequentially acquires vibration information for N lines corresponding to the first pixel line to the Nth pixel line at regular time intervals. The vibration sensor 47 outputs the vibration information for N lines to the vibration information embedding unit 62C1.

[0093] When the predetermined exposure period ends, the readout circuit 62A reads out analog pixel line data for each horizontal pixel line from the photoelectric conversion element 61. The readout circuit 62A performs analog signal processing on the analog pixel line data and outputs the pixel line data after the analog signal processing to the digital processing circuit 62B.

[0094] The digital processing circuit 62B performs digital signal processing on the analog pixel line data to convert it into digital pixel line data, and then outputs the pixel line data after the digital signal processing to the vibration information embedding unit 62C1.

[0095] The vibration information embedding unit 62C1 embeds corresponding pixel line vibration information at the beginning of the pixel line data. The vibration information embedding unit 62C1 sequentially stores the pixel line data with embedded pixel line vibration information in the memory 64. When the pixel line data with embedded pixel line vibration information for all pixel lines has been stored, the shake correction unit 62C2 reads out one frame of vibration information-embedded image data. The shake correction unit 62C2 calculates the average value of the pixel line vibration information for N lines and performs shake correction processing on the vibration information-embedded image data using the average value. The shake correction unit 62C2 stores the shake-corrected image data generated by the shake correction processing in the memory 64.

[0096] The output circuit 62D outputs the shake-corrected image data generated at the end of each frame output period to the downstream signal processing unit 50. That is, of the shake-corrected image data for four frames generated in each frame output period, only the shake-corrected image data for one frame generated at the end of each frame output period is output to the signal processing unit 50 via the output circuit 62D. The shake-corrected image data for the remaining three frames is not output but is deleted from the memory 64. Note that, although an example is given here in which the shake-corrected image data for one frame generated at the end of the frame output period is output to the signal processing unit 50, the technology of the present disclosure is not limited to this, and the shake-corrected image data for a predetermined other frame may also be output to the signal processing unit 50.

[0097] Next, the operation of the imaging element 44 according to the first embodiment will be described.

[0098] 10A and 10B show an example of the flow of the shake-corrected image data generation process executed by the processing circuit 62 of the image sensor 44 within a frame output period.

[0099] 10A and 10B, first, in step ST10, the vibration information embedding unit 62C1 determines whether or not the exposure start timing has arrived. Exposure of the image sensor 44 begins in synchronization with the falling edge of the vertical synchronization signal output from the controller 46. If the exposure start timing has not arrived in step ST10, the determination is negative, and the shake-corrected image data generation processing proceeds to step ST26 shown in FIG. 10B. If the exposure start timing has arrived in step ST10, the determination is positive, and the shake-corrected image data generation processing proceeds to step ST11.

[0100] In step ST11, the vibration information embedding unit 62C1 sets a variable n to 1. The variable n represents the number of pixel lines included in the imaging element 44. Thereafter, the shake-corrected image data generation process proceeds to step ST12.

[0101] In step ST12, the vibration information embedding unit 62C1 determines whether the variable n is 2 or greater. If the variable n is 1 in step ST12, the determination is negative, and the shake-corrected image data generation process proceeds to step ST13. If the variable n is 2 or greater in step ST12, the determination is positive, and the shake-corrected image data generation process proceeds to step ST14. First, the case where the variable n is 1 will be described.

[0102] In step ST13, the vibration information embedding unit 62C1 controls the readout circuit 62A to start exposure of the n-th pixel line. That is, when n=1, exposure of the first pixel line starts in step ST13. Thereafter, the shake-corrected image data generation process proceeds to step ST18.

[0103] In step ST18, the vibration information embedding unit 62C1 determines whether or not the variable n is equal to the number N of pixel lines included in the image sensor 44. If the variable n is 1, the variable n is not equal to the number N of pixel lines, and therefore the determination in step ST18 is negative, and the shake-corrected image data generation process proceeds to step ST19.

[0104] In step ST19, the vibration information embedding unit 62C1 adds 1 to the variable n, thereby increasing the variable n by 1. That is, the variable n is incremented to 2. Thereafter, the shake-corrected image data generation process proceeds to step ST12. Below, a description will be given of the case where the variable n is 2 or greater.

[0105] In step ST12, the variable n is equal to or greater than 2, so the determination is affirmative, and the shake-corrected image data generation process proceeds to step ST14.

[0106] In step ST14, the vibration information embedding unit 62C1 starts exposing the n-th pixel line, after which the shake-corrected image data generation process proceeds to step ST15.

[0107] In step ST15, the vibration information embedding unit 62C1 acquires the (n-1)th pixel line data by controlling the readout circuit 62A and the digital processing circuit 62B. After that, the shake-corrected image data generation process proceeds to step ST16.

[0108] In step ST16, the vibration information embedding unit 62C1 acquires the (n-1)th pixel line vibration information from the vibration sensor 47. After that, the shake-corrected image data generation process proceeds to step ST17.

[0109] In step ST17, the vibration information embedding unit 62C1 embeds the (n-1)th pixel line vibration information into the (n-1)th pixel line data and stores it in the memory 64. Thereafter, the shake corrected image data generation process proceeds to step ST18.

[0110] In step ST18, the vibration information embedding unit 62C1 determines whether the variable n is equal to the number N of pixel lines included in the image sensor 44. If the variable n is not equal to the number N of pixel lines in step ST18, the determination is negative, and the shake-corrected image data generation process proceeds to step ST19. If the variable n is equal to the number N of pixel lines in step ST18, the determination is positive, and the shake-corrected image data generation process proceeds to step ST20.

[0111] In step ST19, the vibration information embedding unit 62C1 adds 1 to the variable n, thereby incrementing the variable n by 1. Thereafter, the shake-corrected image data generation process proceeds to step ST12. Therefore, the shake-corrected image data generation process repeats steps ST12 to ST17 while incrementing the variable n by 1 from 2 to N, and when the variable n reaches the number of pixel lines N, the shake-corrected image data generation process proceeds to step ST20.

[0112] In step ST20, the vibration information embedding unit 62C1 acquires the n-th pixel line data by controlling the readout circuit 62A and the digital processing circuit 62B. After that, the shake-corrected image data generation process proceeds to step ST21.

[0113] In step ST21, the vibration information embedding unit 62C1 acquires the n-th pixel line vibration information from the vibration sensor 47. After that, the shake-corrected image data generation process proceeds to step ST22.

[0114] In step ST22, the vibration information embedding unit 62C1 embeds the n-th pixel line vibration information into the n-th pixel line data and stores it in the memory 64. Thereafter, the shake-corrected image data generation process proceeds to step ST23 shown in FIG. 10B.

[0115] 10B, the shake correction unit 62C2 reads out one frame of vibration information embedded image data from the memory 64. Thereafter, the shake corrected image data generation process proceeds to step ST24.

[0116] In step ST24, the shake correction unit 62C2 calculates the average value of pixel line vibration information included in the vibration information embedded image data read out in step ST23. After that, the shake corrected image data generation process proceeds to step ST25.

[0117] In step ST25, the shake correction unit 62C2 performs shake correction processing on the vibration information embedded image data based on the average value calculated in step ST24 to generate shake-corrected image data. After that, the shake-corrected image data generation processing proceeds to step ST26.

[0118] In step ST26, the shake correction unit 62C2 stores the shake-corrected image data generated in step ST25 in the memory 64. Thereafter, the shake-corrected image data generation process proceeds to step ST27.

[0119] In step ST27, the shake correction unit 62C2 determines whether the output timing of the shake corrected image data has arrived. The output timing of the shake corrected image data is the timing specified by the frame output period. If the output timing of the shake corrected image data has arrived, the determination is affirmative, and the shake corrected image data generation process proceeds to step ST28. If the output timing of the shake corrected image data has not arrived, the determination is negative, and the shake corrected image data generation process proceeds to step ST10.

[0120] In step ST28, the shake correction unit 62C2 reads out the shake corrected image data from the memory 64, and outputs it via the output circuit 62D to the signal processing unit 50. Thereafter, the shake corrected image data generation process proceeds to step ST29.

[0121] In step ST29, the shake correction unit 62C2 determines whether or not a condition for terminating the shake corrected image data generation process (hereinafter referred to as the "shake corrected image data generation process termination condition") has been satisfied. An example of the shake corrected image data generation process termination condition is that an instruction to terminate the shake corrected image data generation process has been accepted by the acceptance device 84 (see FIG. 4). In step ST29, if the shake corrected image data generation process termination condition is not satisfied, the determination is negative, and the shake corrected image data generation process proceeds to step ST10. In step ST29, if the shake corrected image data generation process termination condition is satisfied, the determination is positive, and the shake corrected image data generation process ends.

[0122] As described above, the imaging element 44 according to the first embodiment includes a processing circuit 62 that performs analog signal processing and digital signal processing on analog image data obtained by capturing an image with the imaging element 44, and a memory 64 that stores digital image data converted from the analog image data by the processing circuit 62. The processing circuit 62 receives vibration information indicating vibrations applied to the imaging element 44 within a frame output period defined by the output frame rate. The processing circuit 62 embeds pixel line vibration information at the beginning of each of multiple lines of pixel line data constituting the digital image data, and stores the pixel line data with the embedded pixel line vibration information in the memory 64. The processing circuit 62 stores four frames of vibration information-embedded image data in the memory 64 during one frame output period and outputs the one frame of vibration information-embedded image data stored at the end of the frame output period. Therefore, this configuration makes it possible to output vibration information-embedded image data including vibration information synchronized with the digital image data.

[0123] Furthermore, according to the image sensor 44 of the first embodiment, the processing circuit 62 embeds corresponding pixel line vibration information in each of the multiple lines of pixel line data. Therefore, compared to a case where pixel line vibration information is not embedded in the pixel line data of each pixel line, the pixel line data and vibration information can be synchronized more accurately.

[0124] Furthermore, according to the image sensor 44 of the first embodiment, the processing circuit 62 embeds corresponding pixel line vibration information at the beginning of multiple lines of pixel line data. Therefore, compared to when pixel line vibration information is not embedded at the beginning of the pixel line data, the pixel line vibration information can be easily extracted from the pixel line data.

[0125] Furthermore, according to the imaging element 44 of the first embodiment, the imaging element 44 is an imaging element in which at least the photoelectric conversion element 61 and the memory 64 are integrated into a single chip. This can contribute to a smaller imaging element 44 than when the photoelectric conversion element 61 and the memory 64 are not integrated into a single chip.

[0126] Furthermore, according to the imaging element 44 of the first embodiment, the imaging element 44 is a stacked type imaging element in which the photoelectric conversion element 61 and the memory 64 are stacked. Therefore, compared to when the photoelectric conversion element 61 and the memory 64 are not stacked, the wiring connecting the photoelectric conversion element 61 and the memory 64 can be made shorter, thereby reducing wiring delay and increasing the transfer speed of image data.

[0127] Furthermore, according to the image sensor 44 of the first embodiment, the shake correction unit 62C2 of the control circuit 62C performs shake correction processing on the vibration information embedded image data within a frame output period based on the pixel line vibration information added to the vibration information embedded image data. Therefore, because the shake correction processing is performed within the image sensor 44, the image sensor 44 can be made smaller than when the shake correction processing is performed in a separate downstream circuit.

[0128] Furthermore, according to the image sensor 44 of the first embodiment, the shake correction unit 62C2 performs shake correction processing using the average value of the pixel line vibration information, which allows for more accurate shake correction processing than when the average value of the pixel line vibration information is not used.

[0129] In the first embodiment, angular velocity is used as the vibration information, but the technology of the present disclosure is not limited to this. The vibration information may be at least one of acceleration, an integral value of angle, an integral value of acceleration, and a shake correction amount. Therefore, compared to when the vibration information is a single value, it is possible to embed optimal vibration information in accordance with the specifications of the shake correction process.

[0130] In the first embodiment, the vibration information embedding unit 62C1 embeds pixel line vibration information at the beginning of each pixel line data, but the technology of the present disclosure is not limited to this. A bit area for embedding pixel line vibration information at the end of the pixel line data may be prepared in advance, and the vibration information embedding unit 62C1 may embed pixel line vibration information at the end of each pixel line data. In this case, the same effect as when pixel line vibration information is embedded at the beginning of each pixel line data can be obtained.

[0131] Furthermore, in the first embodiment, a bit area for embedding pixel line vibration information is provided at the beginning of each of the multiple lines of pixel line data, but the technology of the present disclosure is not limited to this. It is sufficient that a bit area for embedding pixel line vibration information is provided at the beginning of at least one line of pixel line data among the multiple lines of pixel line data. In this case, too, the pixel line data and vibration information can be synchronized more accurately than when vibration information is not embedded in at least one line of pixel line data.

[0132] In the first embodiment, the shake correction unit 62C2 performs shake correction processing using the average value of the vibration information, but the technology of the present disclosure is not limited to this. The shake correction unit 62C2 may perform shake correction processing using the median or mode of the vibration information instead of the average value of the vibration information. In this case, the same effect as when performing shake correction processing using the average value of the pixel line vibration information can be obtained.

[0133] Furthermore, in this first embodiment, each pixel line is composed of one horizontal pixel line, but the technology of the present disclosure is not limited to this, and each pixel line may be composed of multiple horizontal pixel lines.

[0134] Furthermore, in the first embodiment, the vibration sensor 47 acquires vibration information for each horizontal pixel line at the center of a predetermined exposure period. However, the technology of the present disclosure is not limited to this. The vibration sensor 47 may acquire vibration information asynchronously with the exposure period of each pixel line. In this case, the vibration information embedding unit 62C1 may embed vibration information at the beginning of the pixel line data for the pixel line exposed for the period closest to the period for which the vibration information was acquired. In this case, the vibration information can be embedded in pixel line data whose exposure period for the pixel line more accurately matches the exposure period for the vibration information, compared to when the vibration information is not embedded in the pixel line data for the pixel line having the exposure period closest to the period for which the vibration information was acquired. Note that, because the exposure period for each pixel line and the period for which the vibration information was acquired are asynchronous, in this case, vibration information does not need to be embedded in all pixel line data.

[0135] [Second embodiment] As shown in FIG. 11 as an example, in the imaging device 10 according to the second embodiment, a user touches a partial area of a live-view image displayed on the touch panel display 26 with a finger, a stylus pen, or other indicator to accurately specify an area for shake correction processing. The controller 46 outputs partial area designation information to the imaging element 44. The partial area designation information is information indicating the position of the partial area designated by the user via the touch panel 34 in the digital image data displayed on the display 32 as a live-view image, and is, for example, coordinates indicating the position of at least one pixel constituting the partial area. For example, if the upper left pixel of the image represented by the digital image data is taken as the origin, the partial area designation information is the coordinates (X, Y) of the pixel designated by the user (see FIG. 14). The imaging element 44 performs shake correction processing based on pixel line vibration information acquired during exposure of the pixel line corresponding to the position of the partial area designated by the partial area designation information.

[0136] The configuration of the imaging element 44 according to this second embodiment is the same as that of the imaging element 44 according to the first embodiment described above, so the same components as those described in the first embodiment are given the same symbols and their description will be omitted.

[0137] 12, in the imaging device 10 according to the second embodiment, in the recording imaging mode, digital image data acquired by the processing circuit 62 is output to the signal processing unit 50 via the output circuit 62D at the output frame rate. The signal processing unit 50 performs various signal processes on the digital image data and outputs it to the CPU 46A via the first communication I / F 46D1. The CPU 46A displays the input digital image data on the touch panel display 26 as a live view image.

[0138] The user determines the angle of view by looking at the live view image displayed on the touch panel display 26, and then touches a partial area of the live view image using a pointer (see FIG. 11). As the partial area, an area of a specific subject displayed in the live view image that the user wishes to capture particularly clearly is selected. For example, if the specific subject is a person, an area including the person's face is selected as the partial area.

[0139] The touch panel 34 detects contact with a partial area of the live view image displayed on the display 32. The touch panel 34 outputs information specifying the position of the detected contact as partial area designation information to the CPU 46A. The CPU 46A outputs the partial area designation information to the image sensor 44 via the second communication I / F 46D2.

[0140] As an example, as shown in FIG. 13, the control circuit 62C receives partial area designation information input from the controller 46 and stores it in the memory 64.

[0141] Thereafter, the user causes the imaging device 10 to capture an image at the determined angle of view. As in the first embodiment, the processing circuit 62 captures an image at the imaging frame rate and acquires digital image data in units of horizontal pixel lines. Furthermore, the control circuit 62C receives vibration information in units of horizontal pixel lines that indicates vibrations applied to the imaging element 44, and the vibration information embedding unit 62C1 embeds pixel line vibration information at the beginning of each of the multiple lines of pixel line data that make up the digital image data. The vibration information embedding unit 62C1 sequentially stores the pixel line data with the embedded pixel line vibration information in the memory 64.

[0142] When all pixel line data has been stored in memory 64, the shake correction unit 62C2 reads one frame of vibration information-embedded image data and partial area designation information from memory 64. The shake correction unit 62C2 identifies the pixel line corresponding to the partial area designated by the user based on the Y coordinate of the coordinates (X, Y) included in the partial area designation information.

[0143] 14, when the coordinates (X, Y) included in the partial area designation information correspond to the fifth pixel line of the vibration information-embedded image data, the shake correction unit 62C2 reads out the fifth pixel line vibration information embedded at the beginning of the fifth pixel line data. Here, the fifth pixel line vibration information is vibration information acquired by the vibration sensor 47 during the exposure period of the fifth pixel line corresponding to the partial area designated by the user. The shake correction unit 62C2 performs shake correction processing on the vibration information-embedded image data based on the fifth pixel line vibration information, and stores the shake-corrected image data generated by the shake correction processing in the memory 64.

[0144] In the second embodiment, the user specifies a partial area before capturing an image. However, there may be cases where the user does not specify a partial area. In this case, as shown in FIG. 15, the control circuit 62C identifies a central pixel line located at the center of the image sensor 44 from among the multiple pixel lines included in the image sensor 44. In the example shown in FIG. 15, one frame of digital image data is composed of 15 lines of pixel line data, including the first pixel line data through the fifteenth pixel line data, so the eighth pixel line is identified as the central pixel line. The control circuit 62C reads out eighth pixel line vibration information embedded at the beginning of the eighth pixel line data. Here, the eighth pixel line data is vibration information acquired by the vibration sensor 47 during the exposure period of the eighth pixel line, which is the central pixel line.

[0145] The shake correction unit 62C2 performs shake correction processing on the vibration information embedded image data based on the eighth pixel line vibration information, and stores in the memory 64 the shake corrected image data generated by the shake correction processing.

[0146] Next, the operation of the image sensor 44 according to the second embodiment will be described. The shake-corrected image data generation process executed by the processing circuit 62 of the image sensor 44 within a frame output period will be described with reference to FIGS. 10A and 16.

[0147] The processes of steps ST10 to ST22 included in the shake-corrected image data generation process shown in Fig. 10A are the same as the processes described in the first embodiment, and therefore their description will be omitted. Also, the processes of step ST23 and steps ST26 to ST29 included in the shake-corrected image data generation process shown in Fig. 16 are the same as the processes described in the first embodiment, and therefore their description will be omitted. Only the differences from the first embodiment will be described below.

[0148] 16, the shake correction unit 62C2 determines whether or not a partial area has been designated by the user. If a partial area has been designated, partial area designation information is stored in the memory 64, and the control circuit 62C can determine whether or not a partial area has been designated by checking whether or not the partial area designation information exists. If a partial area has been designated in step ST30, the determination is affirmative, and the shake corrected image data generation process proceeds to step ST31. If a partial area has not been designated in step ST30, the determination is negative, and the shake corrected image data generation process proceeds to step ST33.

[0149] In step ST31, the shake correction unit 62C2 reads out the partial area designation information from the memory 64. After that, the shake corrected image data generation process proceeds to step ST32.

[0150] In step ST32, the shake correction unit 62C2 identifies a pixel line including a pixel specified by the partial area designation information. The partial area designation information is, for example, coordinates (X, Y) of a pixel specified by the user, and the shake correction unit 62C2 identifies a pixel line corresponding to the partial area specified by the user based on the Y coordinate of the coordinates (X, Y). Then, the shake-corrected image data generation process proceeds to step ST34.

[0151] On the other hand, in step ST33, the shake correction unit 62C2 identifies a central pixel line located in the center of the image sensor 44. Thereafter, the shake corrected image data generation process proceeds to step ST34.

[0152] In step ST34, the shake correction unit 62C2 performs shake correction processing on the vibration information embedded image data using pixel line vibration information included in the pixel line data of the identified pixel line. That is, if a partial area is specified in step ST30, a pixel line including the pixel specified by the partial area designation information is specified in step ST32. Therefore, the shake correction unit 62C2 performs shake correction processing using pixel line vibration information included in the pixel line data of the pixel line including the pixel specified by the partial area designation information. On the other hand, if a partial area is not specified in step ST30, a central pixel line is specified in step ST33. Therefore, the shake correction unit 62C2 performs shake correction processing using pixel line vibration information included in the pixel line data of the central pixel line. Thereafter, the shake corrected image data generation processing proceeds to step ST26.

[0153] As described above, in the image sensor 44 according to the second embodiment, the control circuit 62C receives partial area designation information that designates a partial area of digital image data. The control circuit 62C reads out pixel line vibration information embedded at the beginning of pixel line data for a pixel line corresponding to the partial area among the plurality of pixel lines included in the image sensor 44, and uses the information for shake correction processing. The pixel line vibration information used for shake correction processing is pixel line vibration information acquired by the vibration sensor 47 during the exposure period of the pixel line corresponding to the partial area. Therefore, shake correction processing can be performed based on pixel line vibration information corresponding to the pixel line intended by the user among the plurality of pixel lines included in the image sensor 44.

[0154] Furthermore, with the image sensor 44 according to the second embodiment, if partial area designation information is not received, the control circuit 62C receives pixel line vibration information for the exposure period of a central pixel line located at the center of the image sensor 44, among the multiple pixel lines included in the image sensor 44. The control circuit 62C reads out the pixel line vibration information embedded at the beginning of the pixel line data for the central pixel line and uses it for shake correction processing. Therefore, the vibration information and digital image data can be synchronized more accurately than when pixel line vibration information for the exposure period of the central pixel line is not received.

[0155] In the second embodiment, the partial area designation information is the coordinates (X, Y) of at least one pixel that constitutes the partial area, but the technology of the present disclosure is not limited to this. The partial area designation information may be the number of pixels in each of the X-axis direction and the Y-axis direction from the upper left pixel of the image represented by the digital image data to at least one pixel that constitutes the partial area.

[0156] In addition, in this second embodiment, the partial area designation information is expressed with the upper left pixel of the image represented by the digital image data as the origin, but the technology disclosed herein is not limited to this, and another pixel of the image may be used as the origin, or the pixel in the center of the image may be used as the origin.

[0157] Furthermore, in the second embodiment, one pixel is designated as the partial region, but the technology of the present disclosure is not limited to this, and the partial region may include two or more pixels. As a result, if there are multiple pixel lines corresponding to the partial region, the shake correction unit 62C2 may perform shake correction processing on the vibration information embedded image data using the average value, median value, or mode value of pixel line vibration information included in the pixel line data of the multiple pixel lines corresponding to the partial region.

[0158] [Third embodiment] 17, in an image sensor 44 according to the third embodiment, a control circuit 62C includes a comparison unit 62C3. The comparison unit 62C3 determines whether or not to embed vibration information in pixel line data, depending on the value of the vibration information. Other configurations of the image sensor 44 according to the third embodiment are the same as those of the image sensor 44 according to the first embodiment. Therefore, the same components as those described in the first embodiment are designated by the same reference numerals, and their description will be omitted.

[0159] The comparison unit 62C3 has a predetermined threshold value for comparison with the vibration information. The threshold value may be a fixed value or a variable value. In the third embodiment, the vibration information is an angular velocity indicating the vibration applied to the image sensor 44, so the comparison unit 62C3 has a predetermined threshold value for the angular velocity. Furthermore, the greater the optical zoom magnification of the image capture device 10, the greater the shaking of the image data due to the vibration applied to the image sensor 44. Therefore, the threshold value may be determined according to the optical zoom magnification of the image capture device 10.

[0160] As an example, as shown in FIG. 18, the threshold value is an absolute value including positive and negative values. The comparison unit 62C3 acquires vibration information detected by the vibration sensor 47. The comparison unit 62C3 compares the acquired vibration information with the threshold value for each horizontal pixel line. When the vibration information exceeds the threshold value, the comparison unit 62C3 outputs the vibration information to the vibration information embedding unit 62C1. The vibration information embedding unit 62C1 embeds the vibration information input from the comparison unit 62C3 into the corresponding pixel line data.

[0161] 18, the comparison unit 62C3 compares the vibration information with a threshold value for each horizontal pixel line. The first pixel line vibration information to the third pixel line vibration information acquired by the vibration sensor 47 during exposure of the first pixel line to the third pixel line are smaller than the threshold value. In this case, the comparison unit 62C3 determines that the pixel line vibration information is smaller than the threshold value and does not output the pixel line vibration information to the vibration information embedding unit 62C1.

[0162] On the other hand, the fourth pixel line vibration information, which is vibration information acquired by the vibration sensor 47 during exposure of the fourth pixel line, is greater than the threshold. The comparison unit 62C3 determines that the fourth pixel line vibration information is greater than the threshold and outputs the fourth pixel line vibration information to the vibration information embedding unit 62C1. The vibration information embedding unit 62C1 embeds the input fourth pixel line vibration information into the fourth pixel line data and stores the fourth pixel line data with the embedded fourth pixel line vibration information in a corresponding storage area of the memory 64.

[0163] Similarly, sixth pixel line vibration information, which is vibration information acquired by the vibration sensor 47 during exposure of the sixth pixel line, is also greater than the threshold. The comparison unit 62C3 determines that the sixth pixel line vibration information is greater than the threshold and outputs the sixth pixel line vibration information to the vibration information embedding unit 62C1. The vibration information embedding unit 62C1 embeds the input sixth pixel line vibration information into the sixth pixel line data and stores the sixth pixel line data with the embedded sixth pixel line vibration information in a corresponding storage area of the memory 64.

[0164] Therefore, in the vibration information embedded image data according to the third embodiment, pixel line vibration information is embedded only in pixel line data of pixel lines for which vibration information greater than the threshold value was acquired during exposure.

[0165] For all pixel lines, the comparison unit 62C3 compares the vibration information with a threshold value, and when the vibration information embedding unit 62C1 stores pixel line data with embedded vibration information greater than the threshold value in the memory 64, the shake correction unit 62C2 reads out one frame of vibration information-embedded image data from the memory 64. The shake correction unit 62C2 calculates the average value of the pixel line vibration information embedded in the vibration information-embedded image data and performs shake correction processing on the vibration information-embedded image data using the average value. The shake correction unit 62C2 stores the shake-corrected image data generated by the shake correction processing in the memory 64.

[0166] Next, the operation of the image sensor 44 according to the third embodiment will be described. Here, the shake-corrected image data generation process executed by the processing circuit 62 of the image sensor 44 within a frame output period will be described with reference to FIGS. 19 and 10B.

[0167] The processes of steps ST10 to ST16 and steps ST18 to ST22 included in the shake-corrected image data generation process shown in Fig. 19 are the same as the processes shown in Fig. 10A described in the first embodiment, and therefore their description will be omitted. Also, the processes of steps ST23 to ST29 included in the shake-corrected image data generation process shown in Fig. 10B are the same as the processes described in the first embodiment, and therefore their description will be omitted. Only the differences from the first embodiment will be described below.

[0168] In step ST40, the comparison unit 62C3 compares the (n-1)th pixel line vibration information with a threshold value. After that, the shake-corrected image data generation process proceeds to step ST41.

[0169] In step ST41, the comparison unit 62C3 determines whether the (n-1)th pixel line vibration information is greater than the threshold value. If the (n-1)th pixel line vibration information is greater than the threshold value in step ST41, the determination is affirmative, and the shake-corrected image data generation process proceeds to step ST42. If the (n-1)th pixel line vibration information is equal to or less than the threshold value in step ST41, the determination is negative, and the shake-corrected image data generation process proceeds to step ST43.

[0170] In step ST42, the comparison unit 62C3 outputs the (n-1)th pixel line vibration information to the vibration information embedding unit 62C1, and the vibration information embedding unit 62C1 embeds the input (n-1)th pixel line vibration information into the (n-1)th pixel line data. After that, the shake-corrected image data generation process proceeds to step ST43.

[0171] In step ST43, the vibration information embedding unit 62C1 stores the (n-1)th pixel line data in the memory 64. After that, the shake-corrected image data generation process proceeds to step ST18.

[0172] As described above, according to the image sensor 44 of the third embodiment, the control circuit 62C determines whether or not to embed vibration information into digital image data depending on the value of the vibration information. Therefore, the processing load for embedding vibration information can be reduced compared to when vibration information is always embedded in digital image data.

[0173] Furthermore, according to the image sensor 44 of the third embodiment, the control circuit 62C embeds vibration information into the corresponding pixel line data when the vibration information exceeds a predetermined threshold. Therefore, since only the vibration information that exceeds the threshold is embedded into the pixel line data, the processing load for embedding the vibration information can be reduced compared to when vibration information is embedded into the pixel line data of all pixel lines.

[0174] 18, the vibration information is depicted as continuous analog data, but the technology of the present disclosure is not limited to this. The vibration sensor 47 may acquire one or more vibration information values for each horizontal pixel line in synchronization with the exposure time of the pixel line, and the comparator 62C3 may compare the acquired vibration information values with a threshold value.

[0175] Furthermore, in the third embodiment, the threshold value is a value relative to the angular velocity applied to the image sensor 44, but the technology of the present disclosure is not limited to this. When acceleration, an integral value of an angle, an integral value of acceleration, or a shake correction amount is used as vibration information, a value corresponding to each type is set as the threshold value.

[0176] [Fourth embodiment] 20 , an image capture device 10 according to the fourth embodiment includes an optical shake correction mechanism 96. The optical shake correction mechanism 96 corrects shake in an image captured by the image capture device 10 by displacing the imaging lens 40 in a direction that cancels out vibrations imparted to the image capture device 10. Here, a so-called OIS is used as the optical shake correction mechanism 96. The other configuration of the image capture device 10 according to the fourth embodiment is the same as that of the image capture device 10 according to the first embodiment described above, and therefore the same components as those described in the first embodiment are designated by the same reference numerals and their description will be omitted.

[0177] 21, an image sensor 44 according to the fourth embodiment differs from the image sensor 44 according to the first embodiment in that a control circuit 62C includes a comparison unit 62C3 and an image synthesis unit 62C4. The other configuration of the image sensor 44 according to the fourth embodiment is the same as that of the image sensor 44 according to the first embodiment, and therefore the same components as those described in the first embodiment are denoted by the same reference numerals and their description will be omitted.

[0178] In the image sensor 44 according to the fourth embodiment, in a still image recording imaging mode, the processing circuit 62 acquires multiple frames of image data at a high-speed imaging frame rate that is higher than the output frame rate. The processing circuit 62 performs image data processing, including image shake correction, on each of the multiple frames of image data, and then combines the multiple frames of image data to generate one frame of image data having a required exposure. In the example shown in FIG. 21, four frames of image data are combined to generate one frame of combined image data, so the high-speed imaging frame rate is set to one-quarter of the imaging frame rate used to generate one frame of image data having a standard exposure. Here, the high-speed imaging frame rate is an example of a "second frame rate" according to the technology of the present disclosure.

[0179] The control circuit 62C performs image data processing. As will be described in detail later, the image data processing includes an acquisition process, a reception process, a generation process, and a shake correction process. Here, the acquisition process is an example of an "acquisition process" according to the technology of the present disclosure, the reception process is an example of an "reception process" according to the technology of the present disclosure, the generation process is an example of a "generation process" according to the technology of the present disclosure, and the shake correction process is an example of a "shake correction process" according to the technology of the present disclosure.

[0180] The acquisition process is a process of acquiring digital image data from the digital processing circuit 62B. The reception process is a process of receiving vibration information input from the vibration sensor 47 on a frame-by-frame basis. The generation process is a process of embedding the input vibration information in image data to generate vibration information-embedded image data. The shake correction process is a process of performing shake correction on the embedded image data based on the vibration information.

[0181] The comparison unit 62C3 has a predetermined threshold value. The comparison unit 62C3 compares the input vibration information with the threshold value. In the fourth embodiment, the vibration information is an angular velocity indicating the vibration applied to the image sensor 44, so the comparison unit 62C3 is provided with a threshold value for the angular velocity in advance. As shown in FIG. 22 as an example, the threshold value is an absolute value that includes positive and negative values.

[0182] 22, the input vibration information is smaller than the threshold value in the first, second, and fourth frames. In this case, the comparison unit 62C3 does not output the vibration information to the vibration information embedding unit 62C1. The vibration information embedding unit 62C1 stores the digital image data of the first, second, and fourth frames in the memory 64 as first, second, and fourth frame image data without embedding the corresponding vibration information.

[0183] In the example shown in Fig. 22, in the third frame, a large vibration is applied to the image sensor 44. The image data for the third frame is image data obtained by imaging with the image sensor 44 in a state in which an anti-vibration lens (not shown) included in the imaging lens 40 is displaced by the optical shake correction mechanism 96 in the direction opposite to the direction of the vibration applied to the imaging sensor 44. In Fig. 22, the hatched areas shown to the right and below the third frame image data indicate the amount of displacement of the imaging lens 40 by the optical shake correction mechanism 96.

[0184] Furthermore, since the input vibration information exceeds the threshold value in the third frame, the comparison unit 62C3 outputs the vibration information to the vibration information embedding unit 62C1. Here, the vibration information embedding unit 62C1 performs a generation process. The generation process is a process of embedding the input vibration information in the third frame image data to generate third frame vibration information-embedded image data. The vibration information embedding unit 62C1 stores the third frame vibration information-embedded image data in the memory 64.

[0185] The shake correction unit 62C2 performs shake correction processing. The shake correction processing is processing in which the third frame vibration information-embedded image data is read from the memory 64 and shake correction is performed on the third frame vibration information-embedded image data. In other words, the shake correction processing is processing in which shake on the third frame image data is corrected based on the vibration information embedded in the third frame vibration information-embedded image data. The shake correction unit 62C2 overwrites and stores the third frame shake-corrected image data generated by the shake correction processing in the storage area of the third frame vibration information-embedded image data in the memory 64. Hereinafter, when there is no need to distinguish between the first, second, and fourth frame image data and the third frame shake-corrected image data, they will simply be referred to as "image data for synthesis."

[0186] The image synthesis unit 62C4 generates one frame of synthesized image data by reading and synthesizing the image data for synthesis from the memory 64 within the frame output period. The synthesized image data is image data for one frame having the required exposure amount. Here, the synthesized image data is an example of the "third image data" according to the technology of the present disclosure.

[0187] The output circuit 62D reads the composite image data from the memory 64 and outputs it to the signal processing unit 50.

[0188] Next, the operation of the image sensor 44 according to the fourth embodiment will be described. Here, the composite image data generation process executed by the processing circuit 62 of the image sensor 44 within a frame output period will be described with reference to FIGS. 23A and 23B.

[0189] 23A and 23B, first, in step ST50, the comparison unit 62C3 determines whether or not the image capture timing has arrived. If the image capture timing has not arrived in step ST50, the determination is negative, and the composite image data generation processing proceeds to step ST64. If the image capture timing has arrived in step ST50, the determination is positive, and the composite image data generation processing proceeds to step ST51.

[0190] In step ST51, the comparison section 62C3 sets the variable n to 1. After that, the composite image data generation process proceeds to step ST52.

[0191] In step ST52, the comparison unit 62C3 controls the readout circuit 62A and the digital processing circuit 62B to acquire the n-th frame image data. The control circuit 62C outputs the acquired n-th frame image data to the vibration information embedding unit 62C1. Thereafter, the composite image data generation process proceeds to step ST53.

[0192] In step ST53, comparison unit 62C3 acquires vibration information detected by vibration sensor 47 during exposure for the n-th frame image data. Hereinafter, the vibration information detected by vibration sensor 47 during exposure for the n-th frame image data will be referred to as "n-th frame vibration information." Thereafter, the composite image data generation process proceeds to step ST54.

[0193] In step ST54, the comparison unit 62C3 determines whether the n-th frame vibration information is greater than a predetermined threshold. If the n-th frame vibration information is equal to or less than the threshold in step ST54, the determination is negative, and the composite image data generation process proceeds to step ST56. If the n-th frame vibration information is greater than the threshold in step ST54, the determination is positive, and the composite image data generation process proceeds to step ST55.

[0194] In step ST55, the comparison unit 62C3 outputs the n-th frame vibration information to the vibration information embedding unit 62C1. The vibration information embedding unit 62C1 embeds the n-th frame vibration information into the n-th frame image data to generate n-th frame vibration information-embedded image data. Thereafter, the composite image data generation process proceeds to step ST56.

[0195] In step ST56, the vibration information embedding unit 62C1 stores the nth frame image data or the nth frame vibration information-embedded image data in the memory 64. If it is determined in step ST54 that the nth frame vibration information is equal to or less than the threshold, vibration information is not embedded in the nth frame image data, and so the vibration information embedding unit 62C1 stores the nth frame image data that does not include vibration information in the memory 64. On the other hand, if it is determined in step ST54 that the nth frame vibration information exceeds the threshold, vibration information is embedded in the nth frame image data in step ST55, and so the vibration information embedding unit 62C1 stores the nth frame vibration information-embedded image data in the memory 64. Thereafter, the composite image data generation process proceeds to step ST57.

[0196] In step ST57, the shake correction unit 62C2 reads out the n-th frame image data or the n-th frame vibration information embedded image data stored in the memory 64 in step ST56 from the memory 64. Thereafter, the composite image data generation process proceeds to step ST58.

[0197] In step ST58, the shake correction unit 62C2 determines whether the image data read out from the memory 64 includes vibration information. In step ST58, if the image data read out from the memory 64 is the nth frame image data, the determination is negative because the image data does not include vibration information, and the composite image data generation process proceeds to step ST61. In step ST58, if the image data read out from the memory 64 is the nth frame vibration information-embedded image data, the determination is positive because the image data includes vibration information, and the composite image data generation process proceeds to step ST59.

[0198] In step ST59, the shake correction unit 62C2 performs shake correction processing on the n-th frame vibration information-embedded image data based on the vibration information included in the n-th frame vibration information-embedded image data. After that, the composite image data generation processing proceeds to step ST60.

[0199] In step ST60, the shake correction unit 62C2 stores the n-th frame shake-corrected image data generated by performing the shake correction process in step ST59 in the memory 64. Thereafter, the composite image data generation process proceeds to step ST61.

[0200] In step ST61, the processing circuit 62 determines whether or not the variable n is 4. If the variable n is 4 in step ST61, the determination is affirmative, and the composite image data generation process proceeds to step ST63. If the variable n is not 4 in step ST61, the determination is negative, and the composite image data generation process proceeds to step ST62.

[0201] In step ST62, the processing circuit 62 increments the variable n by 1. Thereafter, the composite image data generation process proceeds to step ST52. Therefore, the processes from step ST52 to step ST60 are repeatedly executed while changing the variable n from 1 to 4.

[0202] In step ST63, the image synthesis unit 62C4 reads out four frames of synthesis image data from the memory 64. Thereafter, the synthesis image data generation process proceeds to step ST64.

[0203] In step ST64, the image synthesis unit 62C4 performs image synthesis processing on the four frames of synthesis image data. After that, the synthesis image data generation processing proceeds to step ST65.

[0204] In step ST65, the image synthesis section 62C4 stores the synthetic image data generated in step ST64 in the memory 64. After that, the synthetic image data generation process proceeds to step ST66.

[0205] In step ST66, the image synthesis unit 62C4 reads out the synthetic image data from the memory 64 and outputs the read synthetic image data via the output circuit 62D to the signal processing unit 50. Thereafter, the synthetic image data generation process proceeds to step ST67.

[0206] In step ST67, the image synthesis unit 62C4 determines whether or not a condition for terminating the synthetic image data generation process (hereinafter referred to as the "synthetic image data generation process termination condition") has been satisfied. An example of the synthetic image data generation process termination condition is that an instruction to terminate the synthetic image data generation process has been accepted by the acceptance device 84 (see FIG. 4). In step ST67, if the synthetic image data generation process termination condition has not been satisfied, the determination is negative, and the synthetic image data generation process proceeds to step ST50. In step ST67, if the synthetic image data generation process termination condition has been satisfied, the determination is positive, and the synthetic image data generation process ends.

[0207] As described above, according to the image sensor 44 of the fourth embodiment, the processing circuit 62 performs image data processing on digital image data captured at a high-speed imaging frame rate that is higher than the output frame rate. The image data processing includes an acquisition process for acquiring digital image data, a reception process for receiving vibration information, a generation process for generating vibration-information-embedded image data by embedding the vibration information in the digital image data, and a shake correction process for correcting shake in the vibration-information-embedded image data based on the vibration information embedded in the vibration-information-embedded image data. Therefore, image data processing including shake correction processing can be performed on digital image data obtained by capturing images at a high-speed imaging frame rate.

[0208] Furthermore, in the image sensor 44 according to the fourth embodiment, the processing circuit 62 generates and outputs one frame of composite image data by combining multiple frames of composite image data after shake correction processing within the frame output period. Therefore, it is possible to output image data of higher quality than when multiple frames of composite image data are not combined.

[0209] 22, the vibration information is depicted as continuous analog data, but the technology of the present disclosure is not limited to this. The vibration sensor 47 may acquire one or more vibration information values on a frame-by-frame basis in synchronization with the exposure time of each frame, and the comparison unit 62C3 may compare the acquired vibration information values with a threshold value.

[0210] In the fourth embodiment, the processing circuit 62 is provided with a comparison unit 62C3 having a predetermined threshold, and when the comparison unit 62C3 determines that the vibration information exceeds the threshold, the vibration information embedding unit 62C1 embeds the vibration information into the corresponding digital image data, but the technology of the present disclosure is not limited to this. As in the first embodiment, the vibration information embedding unit 62C1 may acquire vibration information on a frame-by-frame basis and embed the acquired vibration information into the corresponding digital image data regardless of the magnitude of the vibration information.

[0211] Furthermore, in the fourth embodiment, the threshold value is a value relative to the angular velocity applied to the image sensor 44, but the technology of the present disclosure is not limited to this. When acceleration, an integral value of an angle, an integral value of acceleration, or a shake correction amount is used as vibration information, a value corresponding to each type is set as the threshold value.

[0212] Furthermore, in the fourth embodiment, the imaging device 10 is provided with the optical shake correction mechanism 96, but the technology of the present disclosure is not limited to this, and the optical shake correction mechanism 96 does not have to be provided.

[0213] [Fifth embodiment] The imaging device 10 according to the fifth embodiment includes an imaging element 44 similar to the imaging element 44 according to the first embodiment. However, the control circuit 62C of the imaging element 44 according to the fifth embodiment does not include a shake correction unit 62C2 (see FIG. 7), and the processing circuit 62 does not perform shake correction processing based on vibration information. The processing circuit 62 outputs the vibration information embedded image data to the controller 46 via the downstream signal processing unit 50. The other configurations of the imaging element 44 according to the fifth embodiment are the same as those of the imaging element 44 according to the first embodiment, and therefore description thereof will be omitted.

[0214] As an example, as shown in FIG. 24 , in an imaging device 10 according to the fifth embodiment, a shake-corrected image data generation program is stored in storage 46B. CPU 46A reads the shake-corrected image data generation program from storage 46B and executes the read shake-corrected image data generation program on memory 46C. CPU 46A operates as an image data acquisition unit 46A1, a specific subject detection unit 46A2, a specific subject position information acquisition unit 46A3, and a shake correction unit 46A4 in accordance with the shake-corrected image data generation program executed on memory 46C. CPU 46A executes a shake-corrected image data generation process by operating as image data acquisition unit 46A1, specific subject detection unit 46A2, specific subject position information acquisition unit 46A3, and shake correction unit 46A4. Here, CPU 46A is an example of a “second processor” according to the technology of the present disclosure.

[0215] The image data acquisition unit 46A1 acquires the vibration information-embedded image data stored in the memory 64 of the image sensor 44 via the output circuit 62D, the signal processing unit 50, and the first communication I / F 46D1. As shown in Fig. 25 as an example, the vibration information-embedded image data includes multiple lines of pixel line data, and vibration information acquired during the exposure period of each pixel line is embedded at the beginning of each of the multiple lines of pixel line data. The vibration information-embedded image data and the vibration information have been described in the first embodiment above, so a description thereof will be omitted here.

[0216] The specific subject detection unit 46A2 detects a specific subject based on the vibration information embedded image data. Here, pattern matching is adopted as a method for detecting a specific subject. Note that the specific subject here is an example of a "specific subject" according to the technology of the present disclosure.

[0217] The specific subject position information acquisition unit 46A3 acquires, as specific subject position information, position information of an area including the specific subject detected by the specific subject detection unit 46A2. In the example shown in FIG. 25, a person's face is detected as the specific subject by the specific subject detection unit 46A2, and a rectangular area surrounding the specific subject is indicated by hatching. The specific subject position information acquisition unit 46A3 acquires, as specific subject position information, position information of the area indicated by hatching. The specific subject position information is the coordinates of two pixels on the diagonal of the rectangular area, for example, the upper left and lower right pixels. Here, the specific subject position information is an example of "specific subject position information" according to the technology of the present disclosure.

[0218] The shake correction unit 46A4 acquires the vibration information embedded image data and the specific subject position information, and detects which pixel line among the plurality of pixel lines the position of the specific subject corresponds to based on the specific subject position information. For example, in the example shown in Fig. 25, the position of the specific subject corresponds to the third pixel line to the seventh pixel line.

[0219] In this case, the shake correction unit 46A4 acquires the third to seventh pixel line vibration information embedded at the beginning of the third to seventh pixel line data from the vibration information embedded image data, and calculates the average value of the third to seventh pixel line vibration information. The shake correction unit 46A4 performs shake correction processing on the vibration information embedded image data based on the calculated average value.

[0220] As described above, the imaging device 10 according to the fifth embodiment includes an imaging element 44 and a CPU 46A that is provided downstream of the imaging element 44 and receives vibration information-embedded image data from the imaging element 44. The vibration information-embedded image data includes multiple lines of pixel line data, with vibration information embedded at the beginning of each of the multiple lines of pixel line data. The specific subject position information acquisition unit 46A3 acquires specific subject position information indicating the position of a specific subject based on the vibration information-embedded image data. The shake correction unit 46A4 performs shake correction processing on the vibration information-embedded image data based on vibration information embedded in pixel lines of the multiple lines of pixel line data that correspond to the specific subject position information. Therefore, because the shake correction processing is performed by the CPU 46A in the downstream circuit, the load on the processing circuit 62 can be reduced compared to when the shake correction processing is performed by the processing circuit 62 within the imaging element 44.

[0221] In the fifth embodiment, pattern matching is used as a method for detecting a specific subject, but the technology of the present disclosure is not limited to this. The specific subject detection unit 46A2 may detect a specific subject using image contrast or machine learning.

[0222] Furthermore, in the fifth embodiment, the specific subject position information is the coordinates of two pixels on a diagonal line of a rectangular area, for example, the coordinates of the upper left and lower right pixels, but the technology of the present disclosure is not limited to this. The specific subject position information may also be the coordinates of the lower left and upper right pixels on a diagonal line of a rectangular area. Furthermore, the specific subject position information may also be the coordinates of the upper left pixel of the rectangular area and the number of pixels on each of two adjacent sides of the four sides of the rectangular area.

[0223] Furthermore, in the fifth embodiment, the specific subject position information may be position information of any shape surrounding the specific subject, such as a circle or an ellipse, instead of a rectangle. When the specific subject position information is position information of a shape represented by a circle, the specific subject position information may be information including the coordinates of the pixel corresponding to the center of the circle and the number of pixels corresponding to the radius.

[0224] In the fifth embodiment, the shake-corrected image data generation program is stored in the storage 46B, but the technology of the present disclosure is not limited to this. As an example, as shown in Fig. 26, a shake-corrected image data generation program for causing a computer 98 built in the controller 46 to execute the shake-corrected image data generation process described above is stored in a storage medium 100.

[0225] 26, the computer 98 includes a CPU 46A, a storage 46B, and a memory 46C. A shake-corrected image data generation program stored in a storage medium 100 is installed in the computer 98. The CPU 46A executes the above-described shake-corrected image data generation process in accordance with the shake-corrected image data generation program.

[0226] Here, a single CPU is exemplified as the CPU 46A, but the technology of the present disclosure is not limited to this, and multiple CPUs may be employed instead of the CPU 46A. Note that an example of the storage medium 100 is any portable storage medium such as a flash memory card, an SSD, or a USB memory.

[0227] Alternatively, the shake-corrected image data generation program may be stored in a storage unit of another computer or server device connected to the computer 98 via a communications network (not shown), and the shake-corrected image data generation program may be downloaded to the computer 98 in response to a request from the imaging device 10. In this case, the downloaded shake-corrected image data generation program is executed by the CPU 46A of the computer 98.

[0228] In the above embodiments, the vibration information is embedded at the beginning of pixel line data or at a specific position in digital image data, but the technology of the present disclosure is not limited to this. As long as the vibration information is assigned or added to a bit area that is provided in advance at a specific position in pixel line data or digital image data, the method and format of the vibration information are not particularly limited.

[0229] In addition, in each of the above embodiments, PCI-e is exemplified as the communication standard for the first communication I / F 46D1 and the second communication I / F 46D2, but the technology of the present disclosure is not limited to this. The first communication I / F 46D1 and the second communication I / F 46D2 may adopt other communication standards, including, for example, MIPI, LVDS, SATA, or SLVS-EC. Furthermore, the first communication I / F 46D1 and the second communication I / F 46D2 may adopt different communication standards.

[0230] In addition, in the above-described embodiments, the processing circuit 62 is implemented by a device including an ASIC and an FPGA, but the technology of the present disclosure is not limited to this. For example, the above-described imaging process may be implemented by a software configuration using a computer.

[0231] 27, the processing circuit 62 of the image sensor 44 has a built-in computer 102. A program for executing the shake-corrected image data generation process and the composite image data generation process is stored in a storage medium 104.

[0232] The computer 102 includes a CPU 102A, a storage 102B, and a memory 102C. A program stored in a storage medium is installed in the computer 102. The CPU 102A executes the above-described shake-corrected image data generation process and composite image data generation process in accordance with the program.

[0233] The CPU 102A is not limited to a single CPU, and multiple CPUs may be used. Note that an example of the storage medium 104 is any portable storage medium such as a flash memory card, an SSD, or a USB memory.

[0234] Instead of storing the program in the storage medium 104, the program may be stored in advance in, for example, the storage 102B. The CPU 102A may read the program from the storage 102B and execute the program on the memory 102C. Furthermore, the storage 102B and the memory 102C may be realized by the same medium, for example, a memory.

[0235] Alternatively, the program may be stored in a storage unit of another computer or server device connected to the computer 102 via a communication network (not shown), and the program may be downloaded to the computer 102 in response to a request from the imaging device 10. In this case, the downloaded program is executed by the CPU 102A of the computer 102.

[0236] The computer may also be provided outside the image sensor 44. In this case, the computer may control the processing circuit 62 according to a program.

[0237] The following various processors can be used as hardware resources for executing the shake-corrected image data generation process and composite image data generation process (hereinafter referred to as "various processes") described in the above embodiments. Examples of processors include, for example, a CPU, which is a general-purpose processor that functions as a hardware resource for executing various processes by executing software, i.e., a program, as described above. Examples of processors also include dedicated electrical circuits, such as FPGAs, PLDs, or ASICs, which are processors with a circuit configuration designed specifically for executing specific processes.

[0238] The hardware resources that execute various processes may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resources that execute various processes may be a single processor.

[0239] Examples of a single processor include: first, a form in which one processor is configured by combining one or more CPUs and software, as typified by computers such as client and server, and this processor functions as a hardware resource that executes processing within the imaging device; second, a form in which a processor is used that realizes the functions of the entire system, including multiple hardware resources that execute various processes, on a single IC chip, as typified by SoC; and thus, processing within the imaging device is realized using one or more of the various processors described above as hardware resources.

[0240] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements.

[0241] Furthermore, in the above embodiments, an interchangeable lens camera is exemplified as the imaging device 10, but the technology of the present disclosure is not limited to this. For example, the technology of the present disclosure may be applied to a smart device. The smart device is equipped with the imaging element 44 described in the above embodiments. A smart device configured in this manner can also achieve the same effects and advantages as the imaging device 10 described in the above embodiments. Note that the technology of the present disclosure is not limited to smart devices, but can also be applied to personal computers or wearable terminal devices.

[0242] Furthermore, the various processes described above are merely examples, and it goes without saying that unnecessary steps may be deleted, new steps may be added, or the order of processes may be changed, without departing from the spirit of the invention.

[0243] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0244] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0245] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. An imaging element, a memory that stores first image data obtained by capturing an image with the imaging element; and a first processor that performs image data processing on the first image data, the first image data is pixel line data consisting of a plurality of lines, The first processor receiving vibration information relating to vibrations applied to the image sensor within a frame output period defined by a first frame rate, the vibration information being vibration information during an exposure period of a central pixel line located at the center of the image sensor among a plurality of pixel lines included in the image sensor; outputting second image data in which the vibration information is added to a specific position that is provided in the first image data and that is provided in pixel line data of at least one line within the frame output period; The specific position is a position provided in the pixel line data corresponding to the center pixel line. Image sensor.

2. An imaging element, a memory that stores first image data obtained by capturing an image with the imaging element; and a first processor that performs image data processing on the first image data, the first image data is pixel line data consisting of a plurality of lines, The first processor receiving partial area designation information that designates a partial area of the first image data; receiving vibration information relating to vibrations applied to the image sensor within a frame output period defined by a first frame rate, the vibration information being vibration information during an exposure period of a pixel line corresponding to the partial area among a plurality of pixel lines included in the image sensor; outputting second image data in which the vibration information is added to a specific position provided in the first image data within the frame output period; The specific position is a position provided in the pixel line data corresponding to the partial area, and is a position provided in the pixel line data for a pixel line that was exposed during a period closest to the period during which the vibration information was acquired, among a plurality of pixel lines included in the imaging element. Image sensor.

3. An imaging element, a memory that stores first image data obtained by capturing an image with the imaging element; and a first processor that performs image data processing on the first image data, the first image data is pixel line data consisting of a plurality of lines, The first processor receiving vibration information relating to vibration applied to the imaging element within a frame output period defined by a first frame rate; outputting second image data in which the vibration information is added to a specific position provided in the first image data within the frame output period; The specific position is a position provided in at least one line of pixel line data, and is at the beginning or end of the pixel line data. Image sensor.

4. An imaging element, a memory that stores first image data obtained by capturing an image with the imaging element; and a first processor that performs image data processing on the first image data, The first processor receiving vibration information relating to vibration applied to the imaging element within a frame output period defined by a first frame rate; outputting second image data in which the vibration information is added to a specific position provided in the first image data within the frame output period; performing the image data processing on the first image data captured at a second frame rate higher than the first frame rate; The image data processing includes: an acquisition process for acquiring the first image data; a reception process for receiving the vibration information; a generation process of generating the second image data by adding the vibration information to the first image data; and a shake correction process for correcting shake on the second image data based on the vibration information added to the second image data. Image sensor.

5. The first processor generates and outputs one frame of third image data by synthesizing the second image data of the plurality of frames after the shake correction processing within the frame output period. The imaging device according to claim 4 .

6. The first processor determines whether or not to add the vibration information to the first image data according to a value of the vibration information. The imaging device according to claim 1 .

7. The imaging device according to claim 6 , wherein the first processor adds the vibration information to the first image data when the vibration information exceeds a threshold value.

8. The vibration information is at least one of angular velocity, acceleration, integral value of angle, integral value of acceleration, and shake correction amount. The imaging device according to claim 1 .

9. The imaging element is an imaging element in which at least a photoelectric conversion element and the memory are integrated into one chip. The imaging device according to claim 1 .

10. The imaging element is a stacked imaging element in which the photoelectric conversion element and the memory are stacked. The imaging device according to claim 9 .

11. The first processor performs a shake correction process on the second image data based on the vibration information added to the second image data within the frame output period. The imaging device according to claim 1 .

12. The first processor performing the shake correction process using an average value, a median value, or a mode value of the vibration information; The imaging device according to claim 11.

13. The imaging device according to any one of claims 1 to 12; a second processor provided downstream of the imaging element and receiving the second image data from the imaging element; the second image data includes pixel line data for a plurality of lines; the vibration information is assigned to each of the plurality of lines of pixel line data, The second processor acquiring specific subject position information indicating a position of the specific subject based on the second image data; performing a shake correction process on the second image data based on the vibration information assigned to pixel line data corresponding to the specific subject position information among the plurality of lines of pixel line data; Imaging device.

14. A method for operating an imaging device, the method including: a memory that stores first image data obtained by imaging by the imaging device; and a processor that performs image data processing on the first image data, the first image data is pixel line data consisting of a plurality of lines, receiving vibration information relating to vibrations applied to the image sensor within a frame output period defined by a first frame rate, the vibration information being for an exposure period of a central pixel line located at the center of the image sensor among a plurality of pixel lines included in the image sensor; and outputting, within the frame output period, second image data to which the vibration information is added at a specific position that is provided in the first image data and that is provided in pixel line data of at least one line; The specific position is a position provided in the pixel line data corresponding to the center pixel line. How the image sensor works.

15. A method for operating an imaging element, comprising: a memory for storing first image data obtained by imaging with the imaging element; and a processor for performing image data processing on the first image data, the first image data is pixel line data consisting of a plurality of lines, receiving partial area designation information that designates a partial area of the first image data; receiving vibration information relating to vibrations applied to the image sensor within a frame output period defined by a first frame rate, the vibration information being for an exposure period of a pixel line corresponding to the partial area among a plurality of pixel lines included in the image sensor; and outputting second image data to which the vibration information is added at a specific position provided in the first image data within the frame output period; The specific position is a position provided in the pixel line data corresponding to the partial area, and is a position provided in the pixel line data for a pixel line that was exposed during a period closest to the period during which the vibration information was acquired, among a plurality of pixel lines included in the imaging element. How the image sensor works.

16. A method for operating an imaging element, comprising: a memory for storing first image data obtained by imaging with the imaging element; and a processor for performing image data processing on the first image data, the first image data is pixel line data consisting of a plurality of lines, receiving vibration information relating to vibrations applied to the imaging element within a frame output period defined by a first frame rate; and outputting second image data to which the vibration information is added at a specific position provided in the first image data within the frame output period; The specific position is a position provided in at least one line of pixel line data, and is at the beginning or end of the pixel line data. How the image sensor works.

17. A method for operating an imaging element, comprising: a memory for storing first image data obtained by imaging with the imaging element; and a processor for performing image data processing on the first image data, receiving vibration information relating to vibrations applied to the imaging element within a frame output period defined by a first frame rate; outputting second image data to which the vibration information is added at a specific position provided in the first image data within the frame output period; and performing the image data processing on the first image data captured at a second frame rate higher than the first frame rate; The image data processing includes: an acquisition process for acquiring the first image data; a reception process for receiving the vibration information; a generation process of generating the second image data by adding the vibration information to the first image data; and a shake correction process for correcting shake on the second image data based on the vibration information added to the second image data. How the image sensor works.

18. A program for causing a computer included in an imaging device to execute processing, the program including: a memory that stores first image data obtained by imaging with an imaging device; and a computer that performs image data processing on the first image data, the first image data is pixel line data consisting of a plurality of lines, The process comprises: receiving vibration information relating to vibrations applied to the image sensor within a frame output period defined by a first frame rate, the vibration information being for an exposure period of a central pixel line located at the center of the image sensor among a plurality of pixel lines included in the image sensor; and outputting, within the frame output period, second image data to which the vibration information is added at a specific position that is provided in the first image data and that is provided in pixel line data of at least one line; The specific position is a position provided in the pixel line data corresponding to the center pixel line. program.

19. A program for causing a computer included in an imaging element to execute processing, the program including: a memory for storing first image data obtained by imaging with the imaging element; and a computer for performing image data processing on the first image data, the first image data is pixel line data consisting of a plurality of lines, The process comprises: receiving partial area designation information that designates a partial area of the first image data; receiving vibration information relating to vibrations applied to the image sensor within a frame output period defined by a first frame rate, the vibration information being for an exposure period of a pixel line corresponding to the partial area among a plurality of pixel lines included in the image sensor; and outputting second image data to which the vibration information is added at a specific position provided in the first image data within the frame output period; The specific position is a position provided in the pixel line data corresponding to the partial area, and is a position provided in the pixel line data for a pixel line that was exposed during a period closest to the period during which the vibration information was acquired, among a plurality of pixel lines included in the imaging element. program.

20. A program for causing a computer included in an imaging element to execute processing, the program including: a memory for storing first image data obtained by imaging with the imaging element; and a computer for performing image data processing on the first image data, the first image data is pixel line data consisting of a plurality of lines, The process comprises: receiving vibration information relating to vibrations applied to the imaging element within a frame output period defined by a first frame rate; and outputting second image data to which the vibration information is added at a specific position provided in the first image data within the frame output period; The specific position is a position provided in at least one line of pixel line data, and is at the beginning or end of the pixel line data. program.

21. A program for causing a computer included in an imaging element to execute processing, the program including: a memory for storing first image data obtained by imaging with the imaging element; and a computer for performing image data processing on the first image data, The process comprises: receiving vibration information relating to vibrations applied to the imaging element within a frame output period defined by a first frame rate; outputting second image data to which the vibration information is added at a specific position provided in the first image data within the frame output period; and performing the image data processing on the first image data captured at a second frame rate higher than the first frame rate; The image data processing includes: an acquisition process for acquiring the first image data; a reception process for receiving the vibration information; a generation process of generating the second image data by adding the vibration information to the first image data; and a shake correction process for correcting shake on the second image data based on the vibration information added to the second image data. program.

Images