Image pickup element, image pickup device, image pickup method and program

Abstract

To provide an imaging element, an imaging apparatus, an imaging method, and a program, capable of outputting a smooth moving image as compared to a case where an image that is captured with a shorter exposure time than the exposure time corresponding to a frame rate when being output is output as it is.SOLUTION: An imaging element incorporates: a reading section that reads image data captured at a first frame rate; a storage section that stores the image data; a processing section that performs processing; and an output section that outputs the processed image data at a second frame rate lower than the first frame rate. The reading section reads image data of each of a plurality of frames in parallel. The storage section stores in parallel each of a plurality of pieces of the image data read in parallel by the reading section. The processing section performs generation processing to generate output image data for 1 frame using the image data of each of a plurality of frames stored in the storage section.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The technology of the present disclosure relates to an imaging element, an imaging device, an imaging method, and a program. [Background technology]

[0002] Japanese Patent Application Laid-Open Publication No. 2015-126043 discloses an electronic device including a first semiconductor chip having a sensor unit in which a plurality of sensors are arranged, and a second semiconductor chip having a signal processing unit that processes signals acquired by the sensor. In the electronic device described in Japanese Patent Application Laid-Open Publication No. 2015-126043, the first semiconductor chip and the second semiconductor chip are stacked.

[0003] In the electronic device described in JP 2015-126043 A, signals read from each sensor row are subjected to predetermined signal processing including AD conversion in parallel for each sensor column, and the digitized image data is transferred to a memory unit via a pipeline. In this way, a large amount of information can be processed while the electronic device as a whole can achieve low power consumption, and as a result, it is said that degradation of sensor performance due to thermal noise can also be suppressed.

[0004] Japanese Patent Application Laid-Open Publication No. 2017-225084 discloses an image sensor in which a plurality of pixels are roughly divided into a first pixel region and a second region. In the image sensor described in Japanese Patent Application Laid-Open Publication No. 2017-225084, the first pixel region is a pixel region that captures an image with a first exposure time and generates a first image signal. The second region is a pixel region that captures an image with a second exposure time that is longer than the first exposure time and generates a second image signal from which information about the subject is extracted. Summary of the Invention

[0005] One embodiment of the technology disclosed herein provides an imaging element, an imaging device, an imaging method, and a program that can output smoother moving images than when images captured with an exposure time shorter than the exposure time corresponding to the frame rate at the time of output are output as is. [Means for solving the problem]

[0006] An imaging element according to a first aspect includes a readout unit that reads out image data for each frame obtained by imaging a subject at a first frame rate and is built into the imaging element, a memory unit that stores the image data read out by the readout unit and is built into the imaging element, a processing unit that processes the image data and is built into the imaging element, and an output unit that outputs the image data processed by the processing unit at a second frame rate and is built into the imaging element, wherein the first frame rate is higher than the second frame rate, the readout unit reads out image data for each of a plurality of frames in parallel within an output period defined by the second frame rate as a period during which one frame of image data is output, the memory unit stores in parallel each of the image data read out in parallel by the readout unit, the processing unit performs a generation process to generate one frame of output image data using the image data for each of the plurality of frames stored in the memory unit, and the output unit outputs the output image data generated by the generation process at the second frame rate.

[0007] This allows smoother moving images to be output compared to when images captured with an exposure time shorter than the exposure time corresponding to the frame rate at the time of output are output as is.

[0008] In the image sensor according to the second aspect, the first frame rate is changed in conjunction with the exposure time.

[0009] This makes it possible to respond to changes in the brightness of the subject.

[0010] In the image sensor according to the third aspect, the first frame rate increases as the exposure time decreases.

[0011] This makes it possible to handle relatively bright subjects.

[0012] In the image sensor according to the fourth aspect, exposure for imaging is resumed after the start of exposure and the readout process of image data for at least one pixel by the readout section is completed.

[0013] This allows the non-exposure time between the previous exposure and the next exposure to be relatively short, regardless of the exposure time.

[0014] In the imaging device according to the fifth aspect, the readout section changes the readout speed of the image data in accordance with the number of frames for which image data is read out in parallel.

[0015] This allows image data to be processed without delay.

[0016] In the imaging element according to the sixth aspect, the readout section changes the readout speed of the image data in accordance with the number of frames from which image data is read out in parallel and the number of AD conversion circuits that AD convert the readout image data.

[0017] This allows image data to be processed smoothly even when there is a limit to the number of AD conversion circuits.

[0018] In the imaging element according to the seventh aspect, the reading unit changes the amount of data when AD converting image data in accordance with the number of frames from which image data is read in parallel and the number of AD conversion circuits that AD convert the read image data.

[0019] This allows image data to be processed smoothly even when there is a limit to the number of AD conversion circuits.

[0020] In the imaging device according to the eighth aspect, the storage section includes a plurality of storage areas that individually store each of the plurality of image data.

[0021] This allows multiple image data to be stored in parallel.

[0022] In the imaging element according to the ninth aspect, the generation process is a process of generating image data for one frame by averaging, on a pixel-by-pixel basis, at least a portion of the image data for each of a plurality of frames stored in the storage unit.

[0023] This makes it possible to prevent overexposure and deterioration of image quality.

[0024] In the image sensor according to the tenth aspect, the generation process generates output image data for one frame by combining partial image data, which is a part of the image data, from a plurality of image data.

[0025] This allows a portion of the output image data to be output that is newer in time.

[0026] An imaging element according to an eleventh aspect is an imaging element in which at least a photoelectric conversion element and a storage unit are integrated into one chip.

[0027] This makes it possible to improve the portability of the imaging element compared to an imaging element in which the photoelectric conversion element and the storage unit are not integrated into a single chip.

[0028] The imaging element according to the twelfth aspect is a stacked imaging element in which a storage section is stacked on a photoelectric conversion element.

[0029] This reduces the load on the processing between the photoelectric conversion element and the storage unit compared to an imaging element in which the photoelectric conversion element and the storage unit are not stacked.

[0030] An imaging device according to a thirteenth aspect includes an imaging element according to any one of the first to twelfth aspects, and a control unit that controls a display unit to display an image based on image data output by the output unit.

[0031] This allows the user to view an image based on the plurality of image data output by the output unit.

[0032] An imaging method according to a fourteenth aspect includes the steps of reading out image data for each frame obtained by imaging a subject at a first frame rate, storing the read-out image data, processing the image data, and outputting the processed image data at a second frame rate lower than the first frame rate, wherein the reading out step reads out image data for each of a plurality of frames in parallel within an output period defined by the second frame rate as the period during which one frame of image data is output; the storing step stores each of the image data read out in parallel in parallel; the processing step generates output image data for one frame using the image data for each of the stored frames; and the outputting step outputs the generated output image data at the second frame rate.

[0033] This allows smoother moving images to be output compared to when images captured with an exposure time shorter than the exposure time corresponding to the frame rate at the time of output are output as is.

[0034] A program according to a fifteenth aspect causes a computer to execute the following steps: reading image data for each frame obtained by capturing an image of a subject at a first frame rate; storing the read image data; processing the image data; and outputting the processed image data at a second frame rate lower than the first frame rate; the reading step reads image data for each of a plurality of frames in parallel within an output period defined by the second frame rate as the period during which one frame of image data is output; the storing step stores each of the image data read in parallel in parallel; the processing step generates output image data for one frame using the image data for each of the stored frames; and the output step outputs the generated output image data at the second frame rate.

[0035] This allows smoother moving images to be output compared to when images captured with an exposure time shorter than the exposure time corresponding to the frame rate at the time of output are output as is.

[0036] An image sensor according to an embodiment of the present disclosure includes a memory that stores image data and is built into the image sensor, and a processor that reads image data for each frame obtained by capturing an image of a subject at a first frame rate, processes the image data, and outputs the image data processed by the processing unit at a second frame rate that is lower than the first frame rate, The image sensor reads out image data for each of the multiple frames in parallel within an output period defined by a second frame rate as a period during which one frame of image data is output, stores each of the image data read out in parallel in a memory in parallel, generates output image data for one frame using the image data for each of the multiple frames that have been stored, and outputs the generated output image data at the second frame rate. [Brief explanation of the drawings]

[0037] [Figure 1] 1 is a block diagram showing an example of a hardware configuration of an imaging apparatus according to a first embodiment. [Figure 2A] 1 is a schematic diagram showing an example of the schematic configuration of an imaging element included in an imaging device according to a first embodiment. [Figure 2B] FIG. 2 illustrates multiple storage areas of a memory. [Figure 3] 2 is a block diagram showing an example of the main configuration of an imaging element included in the imaging device according to the first embodiment. FIG. [Figure 4A] FIG. 2 is a conceptual diagram showing in simplified form the exposure, readout, storage, and output operations of an imaging element. [Figure 4B] FIG. 10 is a detailed explanatory diagram showing an example of the operation from exposure to reset of the imaging element. [Figure 5A] FIG. 1 is a schematic diagram illustrating how a moving image appears in the prior art. [Figure 5B] FIG. 1 is a schematic diagram illustrating how a moving image appears in the prior art. [Figure 5C] FIG. 1 is a schematic diagram illustrating how a moving image appears in the prior art. [Figure 5D]FIG. 1 is a schematic diagram illustrating how a moving image appears in the prior art. [Figure 6] 3 is a conceptual diagram showing an example of the operation of exposure, readout, storage, and output of the image sensor according to the first embodiment. FIG. [Figure 7] 5 is a flowchart showing an example of imaging processing according to the first embodiment. [Figure 8] 10 is a flowchart showing an example of exposure readout processing according to the first embodiment. [Figure 9] 10 is a flowchart showing an example of an output image generation process according to the first embodiment. [Figure 10A] 2A to 2C are schematic diagrams illustrating how a moving image output from an imaging element according to the first embodiment appears. [Figure 10B] 2A to 2C are schematic diagrams illustrating how a moving image output from an imaging element according to the first embodiment appears. [Figure 10C] 2A to 2C are schematic diagrams illustrating how a moving image output from an imaging element according to the first embodiment appears. [Figure 10D] 2A to 2C are schematic diagrams illustrating how a moving image output from an imaging element according to the first embodiment appears. [Figure 11A] 4 is a conceptual diagram showing an operation in which an AD conversion column according to the first embodiment AD converts one piece of image data. FIG. [Figure 11B] 4 is a conceptual diagram showing an operation in which an AD conversion column according to the first embodiment AD converts two pieces of image data. FIG. [Figure 11C] 4 is a conceptual diagram showing an operation in which the AD conversion column according to the first embodiment AD converts three pieces of image data. FIG. [Figure 11D] 4 is a conceptual diagram showing an operation in which an AD conversion column according to the first embodiment AD converts four pieces of image data. FIG. [Figure 12] 10A and 10B are schematic diagrams illustrating an example of a problem that occurs when the exposure time of an image sensor is short. [Figure 13] 13 is a schematic diagram showing an example of AD conversion processing of the image sensor according to the second embodiment that solves the problem shown in FIG. 12. FIG. [Figure 14] 10 is a flowchart showing an example of exposure readout processing according to the second embodiment. [Figure 15] 11 is a flowchart showing an example of exposure readout processing according to the third embodiment. [Figure 16] 13 is a schematic diagram showing an example of a readout process and output according to the fourth embodiment. FIG. [Figure 17] FIG. 13 is a schematic diagram showing an example of a readout process and output according to the fifth embodiment. [Figure 18] FIG. 13 is a schematic diagram showing an example of a readout process and an output according to a modification of the fifth embodiment. [Figure 19] 13 is a flowchart showing an example of an output image generation process according to the fifth embodiment. [Figure 20] 10A and 10B are conceptual diagrams showing an example of how a program is installed from a storage medium storing the program to an image sensor according to each embodiment. [Figure 21] FIG. 1 is a block diagram illustrating an example of a schematic configuration of a smart device incorporating an imaging element according to each embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0038] Hereinafter, an example of an embodiment of the technology of the present disclosure will be described with reference to the drawings.

[0039] First, let us explain the terms used in the following explanation. In the following explanation, "AD" is an abbreviation for "Analog-to-Digital." "OVF" is an abbreviation for "Optical View Finder." "EVF" is an abbreviation for "Electronic View Finder." "AE" is an abbreviation for "Auto Exposure." "CMOS" is an abbreviation for "Complementary Metal Oxide Semiconductor." "CCD" is an abbreviation for "Charge Coupled Device." "LSI" is an abbreviation for "Large-Scale Integration." "CPU" is an abbreviation for "Central Processing Unit." "ROM" is an abbreviation for "Read Only Memory." "RAM" is an abbreviation for "Random Access Memory." "I / F" is an abbreviation for "Interface." "ASIC" is an abbreviation for "Application Specific Integrated Circuit." "PLD" is an abbreviation for "Programmable Logic Device." "FPGA" is an abbreviation for "Field Programmable Gate Array." "SSD" is an abbreviation for "Solid State Drive." "USB" is an abbreviation for "Universal Serial Bus." "CD-ROM" is an abbreviation for "Compact Disc Read Only Memory." "IC" is an abbreviation for "Integrated Circuit." "OVF" is an abbreviation for "Optical Viewfinder." "EVF" is an abbreviation for "Electronic Viewfinder." "HDD" is an abbreviation for "Hard Disc Drive." "DRAM" is an abbreviation for "Dynamic Random Access Memory." "SRAM" is an abbreviation for "Static Random Access Memory." "PC" is an abbreviation for "Personal Computer." "fps" is an abbreviation for "frames per second."

[0040] [First embodiment] An example of an embodiment of an imaging device according to the technique of the present disclosure will now be described with reference to the accompanying drawings.

[0041] 1, the imaging device 10 is an interchangeable lens camera. The imaging device 10 includes an imaging device body 12 and an interchangeable lens 14 that is interchangeably attached to the imaging device body 12. The interchangeable lens 14 includes an imaging lens 18 having a focus lens 16 that can be manually moved along an optical axis L1.

[0042] The imaging device body 12 is also provided with a Hybrid Finder (registered trademark) 21. The Hybrid Finder 21 here refers to a finder that selectively uses, for example, an optical viewfinder (hereinafter referred to as "OVF") and an electronic viewfinder (hereinafter referred to as "EVF"). The EVF includes a second display 80.

[0043] The interchangeable lens 14 is interchangeably attached to the imaging device body 12. The lens barrel of the interchangeable lens 14 is provided with a focus ring 22 that is used in manual focus mode. Manual rotation of the focus ring 22 moves the focus lens 16 along the optical axis L1. The focus lens 16 stops at a focus position that corresponds to the subject distance. In this embodiment, the "focus position" refers to the position on the optical axis L1 of the focus lens 16 when in focus.

[0044] A touch panel display 30 is provided on the back surface of the imaging device body 12. The touch panel display 30 includes a liquid crystal display (hereinafter referred to as the "first display") 40 and a touch panel 42. The first display 40 or the second display 80 is an example of a "display unit" according to the technology of the present disclosure.

[0045] The first display 40 displays images, text information, etc. The first display 40 is used to display a live view image (through image), which is an example of a continuous frame image obtained by capturing continuous frames in imaging mode. The first display 40 is also used to display a still image, which is an example of a single frame image obtained by capturing a single frame when an instruction to capture a still image is given. Furthermore, the first display 40 is also used to display a playback image in playback mode, a menu screen, etc.

[0046] The touch panel 42 is a transmissive touch panel, and is overlaid on the surface of the display area of ​​the first display 40. The touch panel 42 detects contact with a pointing object such as a finger or a stylus pen, for example.

[0047] The imaging device 10 includes a mount 44 provided on the imaging device body 12, and a mount 46 on the interchangeable lens 14 side that corresponds to the mount 44. The interchangeable lens 14 is interchangeably attached to the imaging device body 12 by coupling the mount 44 to the mount 46.

[0048] The imaging lens 18 includes an aperture 47. The aperture 47 is disposed closer to the imaging device body 12 than the focus lens 16, and is connected to a motor 49. The aperture 47 operates by receiving power from the motor 49 to adjust the exposure.

[0049] The interchangeable lens 14 includes a slide mechanism 48 and a motor 50. The slide mechanism 48 moves the focus lens 16 along the optical axis L1 when the focus ring 22 is operated. The focus lens 16 is attached to the slide mechanism 48 so that it can slide along the optical axis L1. The slide mechanism 48 is also connected to a motor 50, and the slide mechanism 48 receives power from the motor 50 to slide the focus lens 16 along the optical axis L1.

[0050] The motors 49 and 50 are connected to the imaging device body 12 via mounts 44 and 46, and their drive is controlled in accordance with commands from the imaging device body 12. In this embodiment, stepping motors are used as an example of the motors 49 and 50. Therefore, the motors 49 and 50 operate in synchronization with pulsed power in response to commands from the imaging device body 12.

[0051] The imaging device 10 is a digital camera that records still images and moving images obtained by capturing an image of a subject. The imaging device main body 12 includes an operation unit 54, an external interface (I / F) 63, and a subsequent circuit 90. The subsequent circuit 90 is a circuit that receives data sent from the imaging element 20. In this embodiment, an IC is used as the subsequent circuit 90. An example of an IC is an LSI.

[0052] The subsequent circuit 90 includes a CPU 52, an I / F 56, a primary storage unit 58, a secondary storage unit 60, an image processing unit 62, a first display control unit 64, a second display control unit 66, a position detection unit 70, and a device control unit 74. In the present embodiment, a single CPU is exemplified as the CPU 52, but the technology of the present disclosure is not limited to this, and multiple CPUs may be employed instead of the CPU 52. The CPU 52 is an example of a "control unit (control processor)" according to the technology of the present disclosure.

[0053] In this embodiment, the image processing unit 62, the first display control unit 64, the second display control unit 66, the position detection unit 70, and the device control unit 74 are each implemented by an ASIC. However, the technology of the present disclosure is not limited to this. For example, at least one of a PLD and an FPGA may be used instead of the ASIC. Also, at least two of an ASIC, a PLD, and an FPGA may be used. Also, a computer including a CPU, a ROM, and a RAM may be used. The CPU may be singular or plural. Also, at least one of the image processing unit 62, the first display control unit 64, the second display control unit 66, the position detection unit 70, and the device control unit 74 may be implemented by a combination of hardware and software.

[0054] The CPU 52, I / F 56, primary memory unit 58, secondary memory unit 60, image processing unit 62, first display control unit 64, second display control unit 66, operation unit 54, external I / F 63, and touch panel 42 are connected to each other via a bus 68.

[0055] The CPU 52 controls the entire imaging device 10. In the imaging device 10 according to this embodiment, in autofocus mode, the CPU 52 performs focus control by driving and controlling the motor 50 so that the contrast value of the image obtained by imaging is maximized. In addition, in autofocus mode, the CPU 52 calculates AE information, which is a physical quantity indicating the brightness of the image obtained by imaging. In imaging mode, the CPU 52 derives an exposure time (shutter speed) and an aperture value according to the brightness of the image indicated by the AE information. Then, the CPU 52 sets the exposure state by controlling the relevant components so that the derived exposure time and aperture value are achieved.

[0056] The primary storage unit 58 refers to a volatile memory, such as a RAM, and the secondary storage unit 60 refers to a non-volatile memory, such as a flash memory or a HDD.

[0057] An imaging program 60A is stored in the secondary storage unit 60. The CPU 52 reads out the imaging program 60A from the secondary storage unit 60 and loads the read imaging program 60A into the primary storage unit 58. The CPU 52 executes processes from imaging to display in accordance with the imaging program 60A loaded into the primary storage unit 58.

[0058] The operation unit 54 is a user interface that is operated by the user when giving various instructions to the subsequent circuit 90. Various instructions received by the operation unit 54 are output as operation signals to the CPU 52. The CPU 52 executes processing in accordance with the operation signals input from the operation unit 54.

[0059] The position detection unit 70 is connected to the CPU 52. The position detection unit 70 is connected to the focus ring 22 via the mounts 44 and 46, detects the rotation angle of the focus ring 22, and outputs rotation angle information indicating the rotation angle as a result of the detection to the CPU 52. The CPU 52 executes processing according to the rotation angle information input from the position detection unit 70.

[0060] When the imaging mode is set, subject light passes through the imaging lens 18 including the focus lens 16 and the mechanical shutter 72 to form an image on the light receiving surface of the color imaging element 20 .

[0061] The device control unit 74 is connected to the CPU 52. The device control unit 74 is also connected to the image sensor 20 and the mechanical shutter 72. The device control unit 74 is further connected to the motors 49 and 50 of the imaging lens 18 via the mounts 44 and 46. The device control unit 74 controls the image sensor 20, the mechanical shutter 72, and the motors 49 and 50 under the control of the CPU 52.

[0062] The imaging element 20 is an example of a "stacked imaging element" according to the technology of the present disclosure. The imaging element 20 is, for example, a CMOS image sensor. As shown in FIG. 2A as an example, the imaging element 20 includes a photoelectric conversion element 92, a processing circuit 94, and a memory 96. Preferably, the imaging element 20 includes at least the photoelectric conversion element 92 and the memory 96 integrated on a single chip. By integrating at least the photoelectric conversion element 92 and the memory 96 on a single chip, the portability of the imaging element can be improved compared to imaging elements that do not include the photoelectric conversion element 92 and the memory 96 integrated on a single chip. FIG. 2A illustrates an imaging element 20 in which the photoelectric conversion element 92, the processing circuit 94, and the memory 96 are stacked and integrated on a single chip. Specifically, the photoelectric conversion element 92 and the processing circuit 94 are electrically connected to each other by conductive bumps (not shown) made of copper or the like, and the processing circuit 94 and the memory 96 are also electrically connected to each other by conductive bumps (not shown) made of copper or the like.

[0063] The imaging element 20 is a stacked type imaging element in which a memory 96 is stacked on a photoelectric conversion element 92. By stacking the memory 96 on the photoelectric conversion element 92, the load on processing between the photoelectric conversion element 92 and the memory 96 is reduced compared to an imaging element in which the photoelectric conversion element 92 and the memory 96 are not stacked.

[0064] 3, the processing circuit 94 is an example of a "processor" according to the technology of the present disclosure, and includes a photoelectric conversion element drive circuit 94A, an AD conversion circuit 94B, an image processing circuit 94C, and an output circuit 94D. The processing circuit 94 operates under the control of the CPU 52 via the device control unit 74.

[0065] The photoelectric conversion element drive circuit 94A is connected to the photoelectric conversion element 92 and the AD conversion circuit 94B. The memory 96 is connected to the AD conversion circuit 94B and the image processing circuit 94C. The image processing circuit 94C is connected to the output circuit 94D. The output circuit 94D is connected to the I / F 56 of the subsequent circuit 90.

[0066] The photoelectric conversion element driving circuit 94A controls the photoelectric conversion element 92 under the control of the CPU 52 and reads captured image data from the photoelectric conversion element 92. "Captured image data" here refers to analog image data representing a subject. The AD conversion circuit 94B digitizes the analog image data read by the photoelectric conversion element driving circuit 94A through AD conversion and stores the digitized image data in the memory 96. The memory 96 is capable of storing multiple frames of image data. The image processing circuit 94C processes the image data stored in the memory 96. The photoelectric conversion element driving circuit 94A is an example of a "reading unit" according to the technology of the present disclosure. The memory 96 is an example of a "storage unit" according to the technology of the present disclosure. The image processing circuit 94C is an example of a "processing unit" according to the technology of the present disclosure. The output circuit 94D is an example of an "output unit" according to the technology of the present disclosure.

[0067] The processing circuit 94 is, for example, an LSI, and the memory 96 is, for example, a RAM. In this embodiment, a DRAM is used as an example of the memory 96, but the technology of the present disclosure is not limited to this, and an SRAM may also be used.

[0068] In this embodiment, the processing circuit 94 is implemented by an ASIC. However, the technology of the present disclosure is not limited to this. For example, instead of an ASIC, a PLD and an FPGA may be used. At least one of the above may be employed. At least two of the above may be employed. A computer including a CPU, a ROM, and a RAM may be employed. The CPU may be single or multiple. The processing circuit 94 may be realized by a combination of hardware and software.

[0069] The photoelectric conversion element 92 has a plurality of photosensors (hereinafter also referred to as pixels) arranged in a matrix. In this embodiment, photodiodes are used as an example of the photosensors. Also, an example of the plurality of photosensors is photodiodes for "4896 columns x 3265 rows" of pixels. Note that, hereinafter, a row of pixels may be referred to as a line.

[0070] The photoelectric conversion element 92 includes color filters. The color filters include a G filter corresponding to G (green), which contributes most to obtaining a luminance signal, an R filter corresponding to R (red), and a B filter corresponding to B (blue). In this embodiment, the G filters, R filters, and B filters are arranged with a predetermined periodicity in the row direction (horizontal direction) and column direction (vertical direction) for the multiple photodiodes of the photoelectric conversion element 92. This allows the imaging device 10 to perform demosaic processing of R, G, and B signals according to a repetitive pattern. Note that demosaic processing refers to the process of calculating all color information for each pixel from a mosaic image corresponding to the color filter array of a single-chip color imaging element. For example, in the case of an imaging element including color filters of three colors (RGB), demosaic refers to the process of calculating all RGB color information for each pixel from a mosaic image consisting of RGB.

[0071] Although a CMOS image sensor is exemplified as the imaging element 20 here, the technology of the present disclosure is not limited to this, and the technology of the present disclosure also applies even if the photoelectric conversion element 92 is a CCD image sensor, for example.

[0072] The image sensor 20 has a so-called electronic shutter function, and by operating the electronic shutter function under the control of the device control unit 74, it controls the charge accumulation time of each photodiode in the photoelectric conversion element 92. The charge accumulation time refers to the so-called exposure time.

[0073] The imaging device 10 uses a rolling shutter system to capture still images and moving images such as live view images. Still image capture is achieved by activating the electronic shutter function and operating the mechanical shutter 72, while moving image capture is achieved by activating the electronic shutter function without operating the mechanical shutter 72.

[0074] The processing circuit 94 is controlled by the CPU 52 via the device control unit 74. The processing circuit 94 reads out analog image data for each frame obtained by capturing an image of a subject using the photoelectric conversion element 92. The analog image data is data based on signal charges accumulated in the photoelectric conversion element 92. The processing circuit 94 performs AD conversion on the analog image data read out from the photoelectric conversion element 92. The processing circuit 94 stores the digital image data obtained by performing AD conversion on the analog image data in the memory 96. The processing circuit 94 acquires digital image data from the memory 96, processes the acquired image data, and outputs it to the I / F 56 of the subsequent circuit 90 as output image data. Hereinafter, analog or digital image data will be simply referred to as "image data."

[0075] The first frame rate is a frame rate relating to the time from when exposure is started in the photoelectric conversion element 92 to when one frame of image data captured by the exposure is read from the photoelectric conversion element 92, the read image data is AD converted, and the AD converted image data is stored in the memory 96. The second frame rate is a frame rate relating to the time from when exposure is started in the photoelectric conversion element 92 to when one frame of image data is read from the photoelectric conversion element 92, the read image data is AD converted, and the AD converted image data is stored in the memory 96. The first frame rate is a frame rate related to the time required to output the image signal to the outside of the image sensor 20. Here, "outside of the image sensor 20" refers to, for example, the I / F 56 of the subsequent circuit 90. The first frame rate is a frame rate higher than the second frame rate.

[0076] In the first embodiment, 60 fps (frames per second) is used as an example of the second frame rate. However, the technology of the present disclosure is not limited to this, and the second frame rate can be changed as long as the relationship "second frame rate < first frame rate" is satisfied. Furthermore, the first frame rate can be changed within a range that does not fall below the second frame rate. Hereinafter, the output period for one frame output at the second frame rate will be simply referred to as the "output period." When the second frame rate is 60 fps as described above, the output period is 1 / 60th of a second (16.667 milliseconds).

[0077] One exposure of the image sensor 20 obtains one frame of image data. In the first embodiment, exposure and image data readout processing are performed for each line using a rolling shutter method. When exposure of one line is completed, the charges of that line are read out, AD converted, and the AD converted image data is stored in memory 96 and reset. The process from reading to resetting is referred to as the image data readout processing.

[0078] Before describing the detailed operation of the image sensor 20 according to the first embodiment, a problem that occurs when the first frame rate and the second frame rate are the same will be described.

[0079] As an example, as shown in Figure 4A, we will explain the case where readout processing and output are performed at the same first and second frame rates of 60 fps. The horizontal axis of Figure 4A represents time, and we will assume that the brightness of the subject gradually increases from frame 1 to frame 4, and the exposure times gradually shorten from T1 to T2, T3, and T4 to prevent overexposure. The exposure time T1 of the first frame is, for example, 1 / 60 of a second, but the exposure time T2 of the second frame is shorter than T1. The exposure time T3 of the third frame is shorter than T2, and the exposure time T4 of the fourth frame is shorter than T3.

[0080] In Figure 4A, the start of exposure of the first through Nth lines of the photoelectric conversion element 92, i.e., the reset for exposure, is indicated by a single diagonal line. However, as shown in more detail in Figure 4B, the operations of exposure after reset, readout, AD conversion, storage in memory, and reset are performed for each line. Since the storage of image data in memory 96 occurs after AD conversion is completed, it does not overlap with the exposure time in terms of time. However, for the sake of explanation, the thick diagonal line is used in the same period as the diagonal line indicating the start of exposure. Furthermore, the output of the stored image data is also shown in the same period with a thick diagonal line. In the following explanation, for simplicity, the operation shown in Figure 4B will be represented as in Figure 4A.

[0081] 4A, when the exposure time is T1, the first line is exposed for the time T1, and after exposure, the image data is read out and AD converted, and stored in memory 96 as image data for one line. The second line is exposed for the time T1 later than the first line, and after exposure, the image data is read out and AD converted, and stored in memory 96 as image data for one line. This read process is performed sequentially up to the final Nth line, and the image data for all lines is stored in memory 96 as image data for the first frame. This operation is repeated for each frame.

[0082] Since the first frame rate remains unchanged even if the exposure time is shortened, for example, after the first line is read out and reset after the exposure for the second frame T2, it waits for a time TA until the exposure for the third frame starts. Similarly, after the first line is read out and reset after the exposure for the third frame T3, it waits for a time TB until the exposure for the fourth frame starts. After the first line is read out and reset after exposure for the exposure time T4 for the fourth frame, a waiting time of time TC occurs before exposure for the fifth frame starts.

[0083] Thus, the shorter the exposure time, the longer the waiting time until exposure of the next frame begins. This waiting time occurs across all lines from the first line to the final Nth line. Therefore, when capturing an image of a moving subject, the subject moves during this waiting time. Therefore, if frames 2 through 4 are displayed frame by frame, the resulting images will be discontinuous, as shown in Figures 5A through 5C. If this is output as a moving image, the subject's movement will appear jerky, as if it were discontinuous, as shown in Figure 5D.

[0084] In order to display smooth movement in moving images, it is possible to bring the exposure time of one frame closer to the output period. Bringing the exposure time of one frame closer to the output period is also called widening the time aperture. Widening the time aperture shortens the above-mentioned waiting time, so the movement of the subject in one frame image connects with the movement of the subject in the next frame, making the subject appear to move continuously and smoothly. However, in order to prevent overexposure, the exposure time is set shorter as the subject becomes brighter, so widening the time aperture can be difficult.

[0085] The operation of the image sensor 20 according to the first embodiment, which solves the above problems, will be described below. In the first embodiment, the first frame rate is changed to a frame rate higher than the second frame rate in conjunction with the exposure time. This makes it possible to respond to changes in the brightness of the subject. More specifically, the first frame rate is changed to a higher value as the exposure time becomes shorter. This makes it possible to respond to relatively bright subjects. However, the first frame rate is changed to a value higher than the second frame rate only when the exposure time is shorter than the output period. Changing the first frame rate to a value higher than the second frame rate means exposing multiple times within the output period, i.e., more than once.

[0086] As an example, assume that the subject becomes brighter over time, as shown in FIG. 6. The horizontal axis in FIG. 6 represents time. The second frame rate is 60 fps. As the subject gradually becomes brighter, the exposure times for exposures E1 to E6 gradually shorten, from T1 to T6. However, in this embodiment, unlike the method shown in FIG. 4A, the exposure and readout processes are performed continuously without any waiting time. As a result, image data for multiple frames may be read out in parallel during one output period. For example, during the output period for the third frame shown in the figure, up to three frames of image data for exposures E3, E4, and E5 are read out in parallel and stored. In other words, the photoelectric conversion element drive circuit 94A reads out image data for each of the captured frames in parallel during the output period defined by the second frame rate, which is the period during which one frame of image data is output.

[0087] The exposure for capturing an image is resumed after the photoelectric conversion element drive circuit 94A has completed the readout process of one line of image data after the start of the previous exposure. In particular, it is preferable to resume the exposure without waiting after the readout process of the image data from the previous exposure is completed. This makes it possible to approximate a state of constant exposure, and regardless of the exposure time, the non-exposure time between the previous exposure and the next exposure can be relatively short. Note that, although the image data is read out one line at a time in the first embodiment, it may also be read out one pixel at a time.

[0088] As shown in FIG. 2B, the memory 96 has a first area 96A, a second area 96B, a third area 96C, etc., which are multiple storage areas that individually store each piece of image data. Hereinafter, "storage area" will also be referred to as "area." As shown in FIG. 6, the image data obtained by the first exposure E1 (exposure time T1) is read out by a photoelectric conversion element drive circuit 94A and AD converted by an AD conversion circuit 94B. The AD converted image data is then stored in the memory 9 The image data is stored in the first area 96A of the image processing unit 96. The stored image data is output by the output circuit 94D as output image data for the first frame, and is displayed, for example, on the first display 40. This allows the user to view an image based on the plurality of image data output by the output circuit 94D.

[0089] After the image data of the first line is read, the image data is reset and the readout process is completed. When the readout process is completed, the second exposure E2 (exposure time T2) starts without any waiting time. This is called the exposure and readout process being performed consecutively. The above process is performed for each line.

[0090] The image data obtained by exposure E2 is read out, A / D converted, and then stored in a second area 96B of the memory 96, which is different from the first area 96A. After the image data obtained by exposure E2 is read out, the image data is reset, and exposure E3 begins. The reading of the image data obtained by exposure E3 overlaps in time with the middle of the reading of exposure E2, as shown in FIG. 6. Therefore, the image data obtained by exposure E3 is stored in a first area 96A of the memory 96, which is different from the second area 96B where the storage process of the image data obtained by exposure E2 is being executed. In other words, the memory 96 stores each of the image data read out in parallel by the photoelectric conversion element drive circuit 94A in parallel.

[0091] As described above, exposure and readout processes are performed continuously from exposure E1 to exposure E6. The storage process in memory 96 is performed by selecting different areas from the first area 96A, second area 96B, third area 96C, etc. provided in memory 96 so that the storage processes are performed in parallel. By performing exposure and readout processes continuously, it is possible to capture multiple images, i.e., more than one image, within one output period.

[0092] The output image data for the second frame is image data obtained by exposure E2. The output image data for the third frame is image data obtained by combining image data obtained by exposure E3 and image data obtained by exposure E4. The output image data for the fourth frame is image data obtained by combining image data obtained by exposure E5 and image data obtained by exposure E6. The image processing circuit 94C performs a generation process to generate output image data for one frame by combining the image data of each of the multiple frames stored in the memory 96.

[0093] The process of generating output image data by combining image data can be performed using a known method. For example, the image processing circuit 94C generates image data for one frame by averaging at least a portion of the image data for each of multiple frames stored in the memory 96 on a pixel-by-pixel basis. For example, if one of the image data to be averaged contains noise, the noise is reduced by averaging, thereby preventing image quality degradation. Furthermore, while simple addition may result in increased pixel values ​​and cause overexposure, averaging can prevent overexposure. The combining process can be performed on at least a portion of the common pixels of the multiple image data. The combined output image data is stored in the memory 96. Note that the output image data may also be stored in a storage device other than the memory 96.

[0094] The output circuit 94D outputs the output image data generated by the image processing circuit 94C and stored in the memory 96 to the subsequent circuit 90 at the second frame rate. The CPU 52 stores the output image data in the primary storage unit 58 and causes the first display control unit 64 to display the image data on the first display 40.

[0095] The above processing will be explained using a flowchart. First, the imaging processing performed by the CPU 52 of the subsequent circuit 90 will be explained using FIG. 7. In step S10, the CPU 52 and stores the exposure time in the primary storage unit 58. Next, in step S11, the CPU 52 outputs the stored exposure time to the device control unit 74. Next, in step S12, the CPU 52 determines whether or not the timing for outputting a vertical synchronization signal conforms to the second frame rate. If the determination in step S12 is negative, the process proceeds to step S15. If the determination in step S12 is positive, the process proceeds to step S13, where the CPU 52 outputs a vertical synchronization signal to the device control unit 74. Next, in step S14, the CPU 52 controls the first display control unit 64 to display the output image data input from the image sensor 20 on the first display 40. Thereafter, the imaging process proceeds to step S15.

[0096] In step S15, the CPU 52 determines whether or not the imaging end condition is met. If the determination is affirmative, the CPU 52 outputs to the device control unit 74 that the imaging end condition is met, and then ends the imaging process. If the determination is affirmative, for example, it means that the user issues an instruction to end imaging from the operation unit 54. If the determination is negative in step S15, the process returns to step S10. The above processing is performed by the CPU 52 executing the imaging program 60A.

[0097] Next, the exposure readout process executed by the image sensor 20 under the control of the CPU 52 will be described with reference to FIG. 8. First, in step S16, the photoelectric conversion element drive circuit 94A acquires the exposure time input from the CPU 52 to the device control unit 74. Next, in step S17, the photoelectric conversion element drive circuit 94A controls the photoelectric conversion element 92 to perform exposure for the acquired exposure time. Next, in step S18, the photoelectric conversion element drive circuit 94A reads out image data obtained by exposure. The AD conversion circuit 94B then performs AD conversion processing on the read-out image data and stores the AD-converted image data in the memory 96. Next, in step S19, the photoelectric conversion element drive circuit 94A determines whether the exposure end condition is satisfied. The exposure end condition being satisfied means, for example, that the CPU 52 has input information indicating that the image capture end condition is satisfied to the device control unit 74. If the determination is affirmative, the photoelectric conversion element drive circuit 94A and the AD conversion circuit 94B terminate the exposure readout process. If the determination is negative, the process returns to step S16.

[0098] Next, the output image generation process executed by the image sensor 20 under the control of the CPU 52 will be described with reference to FIG. 9. In step S20, the image processing circuit 94C determines whether a vertical synchronization signal has been input from the CPU 52 to the device control unit 74. If the determination is negative, step S20 is repeated. If the determination is positive, the process proceeds to step S21, where the image processing circuit 94C determines whether there are multiple pieces of image data that can be output stored in the memory 96 at that time. If there are multiple pieces of image data that can be output, the determination is positive and the process proceeds to step S22, where the image processing circuit 94C combines the multiple pieces of image data to generate one piece of output image data. The generated output image data is stored in the memory 96 or another storage unit. If the determination is negative in step S21, the process proceeds to step S23, where the image processing circuit 94C generates one piece of output image data from one piece of image data. The generated output image data is stored in the memory 96 or another storage unit.

[0099] Next, in step S24, the output circuit 94D outputs the generated output image data to the I / F 56 of the subsequent circuit 90. Next, in step S25, the image processing circuit 94C determines whether or not the output image generation end condition is satisfied. The output image generation end condition is satisfied when, for example, the CPU 52 inputs to the device control unit 74 that the imaging end condition is satisfied. If the determination in step S25 is positive, the image processing circuit 94C terminates the output image generation process. If the determination is negative, the process returns to step S20.

[0100] As explained above, one frame of output image data is generated using multiple image data. By doing so, it is possible to obtain an image that captures the trajectory (afterimage) of the object's movement during one output period. For example, the second frame shown in FIG. 6 has an exposure time of T2, and the object captured during exposure time T2 is shown in FIG. 10A. The third frame shown in FIG. 6 is a combination of images from exposures E3 and E4, resulting in the image shown in FIG. 10B, which captures the trajectory of the object's movement over the total exposure time T3 and T4. Similarly, the fourth frame is a combination of images from exposures E5 and E6, resulting in the image shown in FIG. 10C, which captures the trajectory of the object's movement over the total exposure time T5 and T6. When these are viewed as a moving image, the object appears to move smoothly, as shown in FIG. 10D.

[0101] When the image processing circuit 94C synthesizes output image data, it is preferable to add and synthesize image data whose total exposure time within one output frame is as close as possible to the output period. By doing so, the trajectory of the subject's movement within the time period corresponding to the output period can be synthesized into one output frame, resulting in a more natural moving image.

[0102] The output image data obtained by the above processing is stored in the memory 96 or other storage unit, and is also displayed on the first display 40 as a live view image.

[0103] The image sensor 20 according to the first embodiment can synthesize image data with a total exposure time closer to the output period, thereby outputting smoother moving images than when images captured with an exposure time shorter than the exposure time corresponding to the frame rate at the time of output are output as is.

[0104] [Second embodiment] As the exposure time becomes shorter, the number of exposures per output period increases. Therefore, the number of image data read in parallel per output period increases. Meanwhile, the number of AD conversion columns performing AD conversion may be limited. If the number of image data read in parallel per output period exceeds the number of image data that can be simultaneously AD converted, AD conversion stalls, preventing parallel readout of image data. Therefore, in this embodiment, the photoelectric conversion element driving circuit 94A changes the image data readout speed according to the number of image data read in parallel. In other words, the image data readout speed is changed according to the number of frames of image data read in parallel. The readout speed refers to the speed at which the readout process is performed. This allows image data to be processed smoothly.

[0105] Specifically, in this embodiment, the photoelectric conversion element driving circuit 94A changes the readout speed of the image data according to the number of frames in which the image data is read out in parallel, i.e., the number of image data, and the number of AD conversion circuits that AD convert the readout image data, i.e., the number of AD conversion columns.

[0106] 11A, in the second embodiment, the image sensor 20 has a total of eight AD conversion columns T0, T1, T2, T3, B0, B1, B2, and B3 as an AD conversion circuit 94B. The image sensor 20 simultaneously uses two AD conversion columns as a pair to perform AD conversion processing on two lines in parallel. There are four pairs of AD conversion columns: AD conversion columns T0 and B0, AD conversion columns T1 and B1, AD conversion columns T2 and B2, and AD conversion columns T3 and B3. Note that, hereinafter, when there is no need to particularly distinguish between the eight AD conversion columns T0 to T3 and B0 to B3, they will be referred to as the AD conversion circuit 94B.

[0107] As shown in Fig. 11A, when AD conversion is performed on only the image data of the first image, one pair of AD conversion columns T0 and B0 out of eight AD conversion columns is used to AD convert two lines in parallel. Also, as shown in Fig. 11B, when AD conversion is performed on two image data of the first image and the second image, When AD conversion is performed, two sets of AD conversion columns, T0 and B0, and T1 and B1, are used to AD convert two lines of each image data in parallel. Similarly, as shown in Figure 11C, when AD converting three sets of image data for the first through third images, a set of AD conversion columns, T2 and B2, is added to the example shown in Figure 11B, resulting in three sets of AD conversion columns, where the three sets of image data are AD converted in parallel. Similarly, as shown in Figure 11D, when AD converting four sets of image data for the first through fourth images, a set of AD conversion columns, T3 and B3, is added, where the four sets of image data are AD converted in parallel.

[0108] By using the AD conversion column as described above, it is possible to perform AD conversion processing on a maximum of 8 lines (4 frames) of image data in parallel. In other words, it is possible to perform read processing on a maximum of 8 lines (4 frames) of image data in parallel.

[0109] As an example, consider a case where the exposure time continuously changes from T4 to T5, T6, T7, and T8, each of which is shorter than a predetermined first threshold, as shown in Figure 12. Exposure times T1 to T3 are equal to or greater than the first threshold. The first threshold can be set as the exposure time that allows for the acquisition of a number of images in one output period that can be subjected to AD conversion processing in parallel.

[0110] In this case, the readout processes of image data for exposure times T4 to T8 overlap during the time TX indicated by the dotted line frame in Figure 12, and it is necessary to perform readout processes for 10 lines, i.e., 5 frames of image data. However, as mentioned above, the AD conversion column is limited to AD conversion processes for 8 lines, i.e., 4 frames of image data, so it is not possible to perform AD conversion processes on 5 frames of image data in parallel.

[0111] Therefore, the readout speed of the image data is reduced and readout processing is performed. In this way, image data for five or more frames is readout processing performed in parallel. Specifically, in the case described with reference to FIG. 12, image data exposed during the exposure time from T1 to T3 is processed at the normal AD conversion processing speed. On the other hand, image data exposed during the exposure time from T4 to T8, which is shorter than a predetermined first threshold, is processed at a slower AD conversion processing speed than normal.

[0112] Here, slowing down the AD conversion processing speed of image data means that AD conversion is performed intermittently, rather than continuously, for each line of one frame of image data. Four sets of AD conversion columns can simultaneously AD convert up to eight lines at a given time, and the number of image data that can be AD converted in parallel is limited to four. Therefore, for example, when AD converting five frames of image data, four sets of AD conversion columns are assigned to each line of the five frames of image data in order for AD conversion processing. Therefore, at some point, some image data will not be AD converted. As a result, the AD conversion processing speed for one image data item appears to be slower.

[0113] Specifically, as shown in FIG. 13 , five frames of image data to be AD converted in parallel are designated as first data to fifth data. In the first AD conversion process 1, the first line L1 (hereinafter, the first line will be referred to as L1, the second line as L2, etc.) and L2 of the first to fourth data are AD converted. The fifth data is not AD converted. In the next AD conversion process 2, L3 and L4 of the second to fourth data and L1 and L2 of the fifth data are AD converted. The first data is not AD converted. In the next AD conversion process 3, L5 and L6 of the third data and the fourth data, and L3 and L4 of the fifth data and the first data are AD converted. The second data is not AD converted. When such processes are performed, the AD conversion process time per piece of image data appears longer. Therefore, the readout process time for each piece of image data is also longer than the normal readout process time, but five frames of image data can be read out in parallel.

[0114] The exposure readout process according to the second embodiment will be described with reference to FIG. 14. In FIG. 14, steps that execute the same processes as those in FIG. 8 described in the first embodiment are assigned the same step numbers, and descriptions thereof will be omitted. Steps S16 and S17 are the same as those in the flow described in FIG. 8. Next, in step S32, the AD conversion circuit 94B determines whether the exposure time is shorter than a predetermined first threshold. If the determination is affirmative, the process proceeds to step S34. In step S34, the AD conversion circuit 94B performs AD conversion processing by reducing the AD conversion processing speed. Thereafter, the exposure readout process proceeds to step S19. On the other hand, if the determination is negative in step S32, the process proceeds to step S36, in which the AD conversion circuit 94B performs AD conversion processing at the normal AD conversion processing speed. Thereafter, the exposure readout process proceeds to step S19. Steps S19 and thereafter are the same as those in the flow described in FIG. 8.

[0115] As explained above, when reading out more image data than can be AD converted in parallel, the AD conversion processing speed for each image data item can be set lower than the normal AD conversion processing speed, thereby enabling the image data to be read out in parallel.

[0116] Note that the numbers of AD conversion columns T0 to T3 and B0 to B3 are merely examples and are not limited to eight. Naturally, increasing the number of AD conversion columns makes it possible to AD convert more frames of image data. However, it is not desirable to increase the number of AD conversion columns unnecessarily, as this increases costs. For this reason, in this embodiment, in order to process image data smoothly, the number of AD conversion columns is limited to an appropriate number, thereby reducing the image data readout speed.

[0117] According to the second embodiment described above, even if there is a limit to the number of AD conversion columns, image data can be processed without delay.

[0118] [Third embodiment] As explained in the second embodiment, there is a limit to the number of AD conversion columns that perform AD conversion processing in the readout process. In this embodiment, the amount of data when AD converting image data is changed depending on the number of frames from which image data is read in parallel and the number of AD conversion columns that AD convert the readout image data. Specifically, when the number of frames from which image data is read in parallel increases, the conversion bit precision, which is the amount of data when AD converting image data, is reduced. Reducing the conversion bit precision reduces the processing time for AD conversion processing, allowing more frames of image data to be AD converted within the same time. The conversion bit precision is the number of bits of image data processed in one AD conversion process.

[0119] For example, if the exposure time is longer than the second threshold (exposure time > second threshold), the conversion bit precision is set to 14 bits. If the exposure time is equal to or less than the second threshold and longer than the third threshold (third threshold < exposure time ≦ second threshold), the conversion bit precision is set to 12 bits. If the exposure time is equal to or less than the third threshold (exposure time ≦ third threshold), the conversion bit precision is set to 10 bits. The second threshold is, for example, one-third of one output period. The third threshold is, for example, one-fifth of one output period.

[0120] The shorter the exposure time, the greater the number of frames from which image data is read in parallel. In this embodiment, as described above, the shorter the exposure time, the lower the conversion bit precision of the AD conversion process. That is, the conversion bit precision of the AD conversion process is changed according to the number of frames from which image data is read in parallel. The smaller the conversion bit precision, the less data is processed, and therefore the shorter the AD conversion process time.

[0121] Note that the values ​​of the conversion bit precision described above are merely examples, and the specific values ​​are not limited to those described above. The number of frames of image data that need to be AD converted in parallel depending on the exposure time depends on the number and processing capacity of the AD conversion columns, as well as the second frame rate. Furthermore, the extent to which the AD conversion processing time can be shortened by changing the conversion bit precision depends on the processing capacity of the AD conversion columns. Therefore, taking these values ​​into consideration, the conversion bit precision of the AD conversion processing, as well as the second and third thresholds for changing the conversion bit precision, are appropriately set. Furthermore, the number of thresholds is not limited to two. For example, two conversion bit precisions may be specified using one threshold.

[0122] The exposure readout process for changing the conversion bit precision of the AD conversion described above will be described with reference to FIG. 15. In FIG. 15, steps for executing the same processes as those in FIG. 8 described in the first embodiment are assigned the same step numbers, and descriptions thereof will be omitted. Steps S16 and S17 are the same as those in FIG. 8. Next, in step S52, the AD conversion circuit 94B determines whether the exposure time is equal to or less than a predetermined second threshold. If the exposure time is equal to or less than the second threshold, the process proceeds to step S54. On the other hand, if the exposure time is longer than the second threshold, the process proceeds to step S58.

[0123] In step S58, the AD conversion circuit 94B performs AD conversion processing with a conversion bit precision of 14 bits on the image data read out by the photoelectric conversion element drive circuit 94A. The AD conversion circuit 94B stores the AD converted image data in the memory 96, and the exposure readout processing proceeds to step S19.

[0124] In step S54, the AD conversion circuit 94B determines whether the exposure time is equal to or less than a predetermined third threshold. If the exposure time is equal to or less than the third threshold, the process proceeds to step S56. In step S56, the AD conversion circuit 94B performs AD conversion processing on the image data read out by the photoelectric conversion element drive circuit 94A with a conversion bit precision of 10 bits. The AD conversion circuit 94B stores the AD converted image data in the memory 96, and the exposure readout process proceeds to step S19.

[0125] On the other hand, if it is determined in step S54 that the exposure time is not equal to or less than the third threshold, the process proceeds to step S60. In step S60, the AD conversion circuit 94B performs AD conversion processing on the image data read out by the photoelectric conversion element drive circuit 94A with a conversion bit precision of 12 bits. The AD conversion circuit 94B stores the AD converted image data in the memory 96, and the exposure readout process proceeds to step S19. Steps from S19 onwards are the same as those described in FIG. 8 of the first embodiment.

[0126] According to the third embodiment described above, even if the number of AD conversion columns is limited, image data can be processed smoothly. Moreover, unlike the second embodiment, the image data read speed is not reduced from the normal speed, so there is no concern about an increase in rolling distortion.

[0127] [Fourth embodiment] When capturing images for a long period of time, the amount of image data to be stored increases, which puts a strain on the storage capacity of the memory 96. In this embodiment, the CPU 52 deletes the image data used to synthesize the output image data from the image data stored in the memory 96 after the output image data has been output. This allows the memory 96 to be used more efficiently.

[0128] As an example, as shown in FIG. 16, the output image data output as the first frame is created based on image data D1 obtained and stored in exposure E1. Therefore, the image data D1 is deleted after the first frame is output. Note that in FIG. 16, the deletion process is indicated by a hatched box. Similarly, the output image data output as the second frame is created based on image data D1 obtained and stored in exposure E1. Image data D2 based on exposure E2 used in the creation is deleted after the output image data for the second frame is output. The output image data for the third frame is a combination of image data D3 and image data D4 based on exposures E3 and E4. Therefore, image data D3 and image data D4 are deleted after the third frame is output. Similarly, image data D5 and image data D6 used for the output image data for the fourth frame are deleted after the fourth frame is output. Image data D5 and image data D6 were stored in the first area 96A of memory 96 from which image data D1 was deleted, and in the second area 96B of memory 96 from which image data D2 was deleted, respectively. By deleting image data from memory 96 after output, the memory 96 can be used more efficiently.

[0129] [Fifth embodiment] When the exposure time is shortened and image data from multiple exposures is combined to create one output image, a time lag may occur in the output image depending on the time at which the images are combined to create the output image data. The time lag is the difference between the time of exposure and the time of output. If image data exposed at a time further away from the output timing is used to combine the images, the time lag in the output image will be greater.

[0130] In this embodiment, partial image data, which is a portion of image data, is combined from multiple image data. Specifically, consider a case where image data that was exposed before the output period or whose exposure began before the output period is output as output image data. In this case, the image processing circuit 94C generates output image data by combining partial image data of image data that was exposed closer to the time of output with the remaining image data of the output image data that has been output up to that point, from the middle of outputting the output image data. The image data that was exposed closer to the time of output is, for example, image data that is being newly stored during output. Furthermore, the partial image data to be combined is partial image data whose pixel positions correspond to the remaining image data of the output image data that has been output up to that point. This method makes it possible to output output image data that is partly new in time.

[0131] 17, of the image data D1 stored in memory 96 based on exposure E1, partial image data 4A from the first line to the nth line is being output as the third frame. When the output of partial image data 4A is complete, image data D2 based on exposure E2 that can be output in the third frame is being stored in memory 96, so the image processing circuit 94C combines the image data D2 with the image data D1. Note that n is an integer greater than or equal to 2 and less than N, where N is the number of lines.

[0132] Specifically, the image processing circuit 94C generates partial image data 4D by combining partial image data 4B from the (n+1)th line to the Nth line (final line) of image data D1 based on exposure E1 with partial image data 4C from the (n+1)th line to the Nth line of image data D2 based on exposure E2. The output circuit 94D outputs the generated partial image data 4D following the partial image data 4A. The combining is performed by, for example, averaging.

[0133] As explained above, by combining and synthesizing partial image data having common pixels from image data D1 and temporally newer image data D2, it is possible to output output image data having a portion that is temporally new as the third frame.

[0134] [Variations] In the fifth embodiment, instead of combining the output image data with newly stored image data midway, it is possible to replace the output image data with the stored image data during output.

[0135] 18, for the third frame, partial image data 4A of image data D1 based on exposure E1 is output, and then partial image data 4C of image data D2 based on exposure E2 is output instead. The partial image data 4A and partial image data 4C are as described above. This allows a portion of the output image data to be temporally new as the third frame.

[0136] The output image generation process of the fifth embodiment and its modified examples will be described with reference to FIG. 19. In FIG. 19, steps that execute the same processes as those in FIG. 9 described in the first embodiment are assigned the same step numbers, and their description will be omitted. Steps S20, S21, S22, and S23 are the same as those in the flow described in FIG. 9. Next, in step S70, the output circuit 94D starts outputting the output image data. Next, in step S72, the image processing circuit 94C determines whether there is new image data that can be output from the middle. If the determination is positive, the output image generation process proceeds to step S74, where the image processing circuit 94C combines partial image data that is part of the new image data with the new image data, or replaces the partial image data with the new image data to newly generate the remaining partial image data for output. Next, in step S76, the output circuit 94D outputs the generated remaining partial image data.

[0137] On the other hand, if the determination in step S72 is negative, the output image generation process proceeds to step S76, and the output circuit 94D outputs the remaining output image data. After step S76, the output image generation process proceeds to step S25. Steps S25 and onward are the same as the flow described in FIG. 9.

[0138] According to the fifth embodiment described above, it is possible to output output image data in which a portion of the image data is newer in terms of time.

[0139] 20 , various programs 200 for causing a computer 20A built into the image sensor 20 to execute the above-described imaging process, exposure and readout process, and output image generation process are stored in a storage medium 210. The computer 20A includes a CPU 20A1, a ROM 20A2, and a RAM 20A3. The program 200 from the storage medium 210 is installed in the computer 20A. The CPU 20A1 of the computer 20A executes the above-described imaging process, exposure and readout process, output image generation process, etc. in accordance with the program 200. Here, a single CPU is illustrated as the CPU 20A1, but the technology of the present disclosure is not limited to this, and multiple CPUs may be used instead of the CPU 20A1.

[0140] An example of the storage medium 210 is any portable storage medium such as an SSD or a USB memory.

[0141] Alternatively, the program 200 may be stored in a storage unit of another computer or server device connected to the computer 20A via a communication network (not shown), and the program 200 may be downloaded in response to a request from the imaging device 10, etc. In this case, the downloaded program 200 is executed by the computer 20A.

[0142] The computer 20A may also be provided outside the image sensor 20. In this case, the computer 20A may control the processing circuit 94 in accordance with the program 200.

[0143] The hardware resources for executing the various processes described in the above embodiments may include the following processors. The processor may, for example, execute software, i.e., a program, as described above, to perform the various processes according to the technology of the present disclosure. Examples of processors include CPUs, which are general-purpose processors that function as hardware resources for executing programs, and dedicated electrical circuits, such as FPGAs, PLDs, and ASICs, which are processors with circuit configurations specifically designed to execute specific processes.

[0144] The hardware resources that execute various processes according to the technology of the present disclosure may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resources that execute various processes according to the technology of the present disclosure may be a single processor.

[0145] Examples of configurations using a single processor include, first, a configuration in which one processor is configured using a combination of one or more CPUs and software, as typified by computers such as client and server, and this processor functions as a hardware resource that executes various processes related to the technology of the present disclosure. Second, a configuration in which a processor is used that realizes the functions of an entire system including multiple hardware resources that execute various processes related to the technology of the present disclosure on a single IC chip, as typified by SoC (System-on-a-chip). In this way, various processes related to the technology of the present disclosure are realized using one or more of the above-mentioned various processors as hardware resources.

[0146] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements.

[0147] Furthermore, in the above-described embodiments, an interchangeable lens camera has been described as an example of the imaging device 10, but the technology of the present disclosure is not limited thereto. For example, the technology of the present disclosure may be applied to a smart device 300 shown in FIG. 21. The smart device 300 shown in FIG. 21 is an example of an imaging device according to the technology of the present disclosure. The smart device 300 is equipped with the imaging element 20 described in the above-described embodiments. The smart device 300 configured in this manner can also achieve the same effects and advantages as the imaging device 10 described in the above-described embodiments. Note that the technology of the present disclosure is not limited to the smart device 300, but can also be applied to a PC or a wearable terminal device.

[0148] In addition, in each of the above embodiments, the first display 40 and the second display 80 are exemplified as display devices, but the technology of the present disclosure is not limited to this. For example, a separate display attached to the imaging device body 12 may be used as the "display unit (display)" according to the technology of the present disclosure.

[0149] Furthermore, the imaging process, exposure readout process, and output image generation process described in each of the above embodiments are merely examples, and it goes without saying that unnecessary steps may be deleted, new steps may be added, or the order of the processes may be changed, without departing from the spirit of the invention.

[0150] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. An imaging element, a processor that reads image data for each frame obtained by capturing an image of a subject at a first frame rate, processes the image data, and outputs the processed image data at a second frame rate; a memory for storing the image data; the first frame rate is a frame rate higher than the second frame rate, The processor: determining a readout speed of the image data based on the number of frames for which the image data is read out in parallel and the number of AD conversion circuits for AD-converting the readout image data; When the number of frames exceeds an upper limit of the number of frames that can be simultaneously AD converted due to the number of AD conversion circuits, the readout speed of the image data is set to a readout speed lower than the readout speed that would be set if the upper limit was not exceeded; reading out the image data of a plurality of frames in parallel within an output period defined by the second frame rate as a period during which the image data for one frame is output; the memory concurrently stores the image data read out by the processor; The processor performs a generation process to generate output image data for one frame using the image data of the plurality of frames stored in the memory, and outputs the output image data generated by the generation process at the second frame rate. Image sensor.

2. The image sensor according to claim 1 , wherein the first frame rate is changed in conjunction with an exposure time.

3. The image sensor according to claim 2 , wherein the first frame rate increases as the exposure time decreases.

4. 4. The imaging device according to claim 1, wherein exposure for imaging is resumed after the processor has completed a readout process of the image data for at least one pixel after the exposure has started.

5. The imaging device according to claim 1 , wherein the processor changes a readout speed of the image data in accordance with the number of frames for which the image data is read out in parallel.

6. The imaging element according to claim 5, wherein the processor changes the readout speed of the image data in accordance with the number of frames for which the image data is read out in parallel and the number of AD conversion circuits for AD converting the readout image data.

7. The imaging element according to claim 5, wherein the processor changes the amount of data when AD converting the image data in accordance with the number of frames from which the image data is read in parallel and the number of AD conversion circuits that AD convert the read image data.

8. The imaging device according to claim 1 , wherein the memory includes a plurality of storage areas for individually storing the image data of the plurality of frames.

9. 9. The imaging element according to claim 1, wherein the generation process is a process of generating image data for one frame by averaging at least a portion of the image data of the plurality of frames stored in the memory on a pixel-by-pixel basis.

10. 10. The imaging device according to claim 1, wherein the generation process generates output image data for one frame by combining partial image data, which is a part of the image data, from a plurality of the image data.

11. 11. The imaging device according to claim 1, wherein at least the photoelectric conversion element and the memory are integrated into a single chip.

12. The imaging device according to claim 11, wherein the imaging device is a stacked type imaging device in which the memory is stacked on the photoelectric conversion element.

13. The imaging device according to any one of claims 1 to 12; a control processor that controls a display to display an image based on the output image data output by the processor; An imaging device including:

14. reading out image data for each frame obtained by capturing an image of a subject at a first frame rate; storing the image data; performing processing on the image data; determining a readout speed of the image data; outputting the processed image data at a second frame rate lower than the first frame rate, the determining step determines the readout speed based on the number of frames from which the image data is read out in parallel and the number of AD conversion circuits that perform AD conversion on the readout image data; When the number of frames exceeds an upper limit of the number of frames that can be simultaneously AD converted due to the number of AD conversion circuits, the readout speed of the image data is set to a readout speed lower than the readout speed that would be set if the upper limit was not exceeded; the reading step includes reading the image data of a plurality of frames in parallel within an output period defined by the second frame rate as a period during which the image data for one frame is output; the storing step includes storing the image data that have been read out in parallel in parallel; The processing step generates output image data for one frame using the stored image data for a plurality of frames, The outputting step outputs the generated output image data at the second frame rate.

15. On the computer, a step of reading image data for each frame obtained by capturing an image of a subject at a first frame rate; storing the image data; performing processing on the image data; determining a read speed of the image data; and a procedure of outputting the processed image data at a second frame rate lower than the first frame rate, the determining step determines the readout speed based on the number of frames from which the image data is read out in parallel and the number of AD conversion circuits that perform AD conversion on the readout image data; When the number of frames exceeds an upper limit of the number of frames that can be simultaneously AD converted due to the number of AD conversion circuits, the readout speed of the image data is set to a readout speed lower than the readout speed that would be set if the upper limit was not exceeded; the reading step includes reading the image data of a plurality of frames in parallel within an output period defined by the second frame rate as a period during which the image data for one frame is output; the step of storing includes storing the image data that have been read out in parallel in parallel; The procedure for performing the processing includes generating output image data for one frame using the image data of a plurality of stored frames, The step of outputting includes outputting the generated output image data at the second frame rate. Do, program.

Images