Camera device, driving method of camera device and storage medium

Abstract

The present invention provides an imaging device, a driving method, and a storage medium that can appropriately determine whether a pixel group including phase difference pixels is set to a short exposure or a long exposure. The imaging device includes: a processor; and an imaging element, a column signal line for reading a signal extends in a first direction, a first pixel group and a second pixel group are arranged in the first direction, the first pixel group includes a plurality of phase difference pixels and a plurality of imaging pixels arranged in a second direction intersecting the first direction, the second pixel group includes a plurality of imaging pixels arranged in the second direction, the processor is configured to perform exposure time setting processing, the exposure time setting processing makes one of the first exposure time for the first pixel group to be exposed and the second exposure time for the second pixel group to be exposed shorter than the other, and in the setting processing, it is configured to determine which of the first exposure time and the second exposure time is to be shorter than the other based on information of a subject photographed by the imaging element.

Description

Technical Field

[0001] The technology of the present invention relates to a camera device, a driving method of the camera device and a storage medium. Background Art

[0002] The image processing device described in Patent Document 1 includes a synthesis processing unit that synthesizes first image data and second image data, wherein the first image data is based on a first signal charge read out from unit pixels on a first row of a solid-state imaging element exposed during a first exposure period, and the second image data is based on a second signal charge read out from unit pixels on a second row of a solid-state imaging element different from the first row and exposed during a second exposure period that is included in the first exposure period and is shorter than the first exposure period.

[0003] In the image sensor described in Patent Document 2, a method for reading out signal charges accumulated in an image sensor from an area sensor in which a plurality of image sensors are arranged in a matrix is ​​as follows. First, a plurality of exposure times of different time lengths are set, and then these exposure times are respectively allocated to each row of the area sensor. Next, the signal charges accumulated in the image sensor during the allocated exposure time are read out in units of rows of the area sensor. Then, the read signal charges are synthesized in units of screens of the area sensor.

[0004] Previous technical literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2013-121130

[0007] Patent Document 2: International Publication No. 2006 / 049098 Summary of the invention

[0008] Technical issues to be solved by the invention

[0009] One embodiment of the technology according to the present invention provides an imaging device, a driving method, and a storage medium capable of appropriately determining whether a pixel group including phase difference pixels is to be exposed for a short time or for a long time.

[0010] Means for solving technical problems

[0011] In order to achieve the above-mentioned purpose, the camera device of the present invention comprises: a processor; and a camera element, wherein a column signal line for reading out a signal extends in a first direction, a first pixel group and a second pixel group are arranged in the first direction, the first pixel group includes a plurality of phase difference pixels and a plurality of camera pixels arranged in a second direction intersecting the first direction, and the second pixel group includes a plurality of camera pixels arranged in the second direction, the processor is configured to perform exposure time setting processing, the exposure time setting processing makes one of a first exposure time for exposing the first pixel group and a second exposure time for exposing the second pixel group shorter than the other, and in the setting processing, it is configured to determine which of the first exposure time and the second exposure time is to be shorter than the other based on information of a subject photographed by the camera element.

[0012] The processor is preferably configured to execute a process of detecting brightness of a region of interest (ROI: Region Of Interest) in an imaging area as information by the imaging element, and determining the first exposure time and the second exposure time based on the detected brightness.

[0013] The processor is preferably configured to execute the following processing: when the brightness is greater than or equal to the first threshold, the first exposure time is made shorter than the second exposure time, and when the brightness is less than the first threshold, the second exposure time is made shorter than the first exposure time.

[0014] The processor is preferably configured to execute a process of detecting a moving speed of the subject as information and determining the first exposure time and the second exposure time based on the detected moving speed.

[0015] The processor is preferably configured to execute processing such that when the moving speed is equal to or greater than a second threshold, the first exposure time is made shorter than the second exposure time, and when the moving speed is less than the second threshold, the second exposure time is made shorter than the first exposure time.

[0016] The processor is preferably configured to execute processing for reading out signals from one of the first pixel group and the second pixel group and then reading out signals from the other of the first pixel group and the second pixel group within a frame period.

[0017] In the camera device of the present invention, the processor is configured to perform the following processing: detecting the position of the subject in the camera area of ​​the camera element related to the first direction, and when the detected position is on one direction side in the first direction, after reading out the signal from one of the first pixel group and the second pixel group from the other direction side, the signal is read out from the other of the first pixel group and the second pixel group from the one direction side; when the detected position is on the other direction side, after reading out the signal from one of the first pixel group and the second pixel group from the one direction side, the signal is read out from the other of the first pixel group and the second pixel group from the other direction side.

[0018] The processor is preferably configured to execute a synthesis process for generating a video signal by synthesizing a signal read out from the first pixel group and a signal read out from the second pixel group.

[0019] The processor is preferably configured to execute the synthesis process after performing phase processing for reducing a phase difference between a signal read out from the first pixel group and a signal read out from the second pixel group.

[0020] The processor is preferably configured to perform phase processing on the signal read out from the second pixel group.

[0021] The processor is preferably configured to perform electronic vibration reduction processing on the video signal generated by the synthesis processing.

[0022] The imaging device of the present invention comprises: a processor; and an imaging element, wherein a column signal line for reading out a signal extends in a first direction, a first pixel group and a second pixel group are arranged in the first direction, the first pixel group includes a plurality of phase difference pixels and a plurality of imaging pixels arranged in a second direction intersecting the first direction, the second pixel group includes a plurality of imaging pixels arranged in a second direction, the processor is configured to selectively execute a first mode and a second mode, wherein:

[0023] The first mode includes:

[0024] a first setting process for making one of a first exposure time for exposing the first pixel group and a second exposure time for exposing the second pixel group shorter than the other; and a first synthesis process for generating a first video signal by synthesizing a signal read out from the first pixel group and a signal read out from the second pixel group,

[0025] The second mode includes:

[0026] A second setting process is to make one of the first exposure time and the second exposure time during the first frame and the first exposure time and the second exposure time during the second frame after the first frame shorter than the other; and a second synthesis process is to generate a second video signal by synthesizing the signal read out from the first pixel group and the second pixel group during the first frame and the signal read out from the first pixel group and the second pixel group during the second frame, so that the set upper limit value of the frame rate in the first mode is higher than the set upper limit value of the frame rate in the second mode.

[0027] The imaging device of the present invention comprises: a processor; and an imaging element, wherein a column signal line for reading out a signal extends in a first direction, a first pixel group and a second pixel group are arranged in the first direction, each of which includes a plurality of imaging pixels arranged in a second direction intersecting the first direction, the first pixel group includes a plurality of first phase difference pixels having a first opening area, and the second pixel group includes a plurality of second phase difference pixels having a second opening area smaller than the first opening area, and the processor is configured to perform an exposure time setting process, wherein the exposure time setting process causes a first exposure time for which the first pixel group is exposed to be shorter than a second exposure time for which the second pixel group is exposed to be exposed.

[0028] The processor is preferably configured to execute a synthesis process for generating a video signal by synthesizing a signal read out from the first pixel group and a signal read out from the second pixel group.

[0029] In the driving method of the camera device of the present invention, the camera device includes a camera element, wherein a column signal line for reading a signal extends in a first direction, a first pixel group and a second pixel group are arranged in the first direction, the first pixel group includes a plurality of phase difference pixels and a plurality of camera pixels arranged in a second direction intersecting the first direction, and the second pixel group includes a plurality of camera pixels arranged in the second direction, and the driving method of the camera device includes: an exposure time setting process, wherein one of a first exposure time for exposing the first pixel group and a second exposure time for exposing the second pixel group is shorter than the other, and in the setting process, based on information of a subject captured by the camera element, it is determined which of the first exposure time and the second exposure time is to be shorter than the other.

[0030] The program of the present invention causes an imaging device to operate, wherein the imaging device includes an imaging element, wherein a column signal line for reading out a signal extends in a first direction, a first pixel group and a second pixel group are arranged in the first direction, the first pixel group includes a plurality of phase difference pixels and a plurality of imaging pixels arranged in a second direction intersecting the first direction, and the second pixel group includes a plurality of imaging pixels arranged in the second direction, the program causes the imaging device to perform an exposure time setting process, the exposure time setting process causes one of a first exposure time for which the first pixel group is exposed and a second exposure time for which the second pixel group is exposed to be shorter than the other, and in the setting process, based on information about a subject captured by the imaging element, it is determined which of the first exposure time and the second exposure time is to be shorter than the other. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a schematic perspective view showing an example of the front side of the imaging device.

[0032] Figure 2 This is a schematic perspective view showing an example of the back side of the imaging device.

[0033] Figure 3 This is a diagram showing an example of the internal structure of the imaging device.

[0034] Figure 4 This is a block diagram showing an example of the functional structure of a processor.

[0035] Figure 5 This is a diagram showing an example of the structure of an imaging sensor.

[0036] Figure 6 This is a diagram showing an example of the structure of an imaging pixel.

[0037] Figure 7 This is a diagram showing an example of the structure of a phase difference pixel.

[0038] Figure 8 This is a diagram showing an example of a pixel arrangement of an imaging sensor.

[0039] Fig. 9 A diagram for explaining focus control and exposure control.

[0040] Fig.10 This is a diagram showing an example of imaging timing when the brightness of the AF area is equal to or greater than the first threshold value.

[0041] Fig.11 This is a diagram showing an example of imaging timing when the brightness of the AF area is smaller than the first threshold value.

[0042] Fig.12 This is a flowchart illustrating an example of the exposure control process.

[0043] Fig.13 It is a diagram schematically showing HDR synthesis processing.

[0044] Fig.14 This is a diagram showing an example of a relationship between the coefficient α and the pixel brightness.

[0045] Fig.15 It is a diagram schematically showing the signal levels of the PG1 signal and the PG2 signal with respect to the amount of incident light.

[0046] Fig.16 This is a flowchart for explaining an example of exposure control according to the second embodiment.

[0047] Fig.17 This is a diagram showing an example of imaging timing according to the third embodiment.

[0048] Fig.18 This is a diagram for explaining an example in which the directions of signal readout of the first pixel group and the second pixel group are changed according to the position of the subject.

[0049] Fig.19 It is a diagram schematically showing the HDR synthesis processing including the phase processing involved in the fourth embodiment.

[0050] Fig. 20 This is a diagram showing an example of imaging timing in the second mode according to the fifth embodiment.

[0051] Fig.21 This is a diagram showing an example of a pixel arrangement of an imaging sensor according to the sixth embodiment.

[0052] Fig. 22 This is a diagram showing an example of the structure of the first phase difference pixel.

[0053] Fig.23 It is a diagram showing an example of the structure of the second phase difference pixel. DETAILED DESCRIPTION

[0054] An example of an embodiment according to the technology of the present invention will be described with reference to the drawings.

[0055] First, the terms used in the following description are explained.

[0056] In the following description, “IC” is the abbreviation of “Integrated Circuit”. “CPU” is the abbreviation of “Central Processing Unit”. “ROM” is the abbreviation of “Read Only Memory”. “RAM” is the abbreviation of “Random Access Memory”. “CMOS” is the abbreviation of “Complementary Metal Oxide Semiconductor”. “HDR” is the abbreviation of “High Dynamic Range”. “AF” is the abbreviation of “Auto Focus”. EEPROM is the abbreviation of “Electrically Erasable Programmable Read-Only Memory”.

[0057] “FPGA” is the abbreviation of “Field-Programmable Gate Array”. “PLD” is the abbreviation of “Programmable Logic Device”. “ASIC” is the abbreviation of “Application Specific Integrated Circuit”. “JPEG” is the abbreviation of “Joint Photographic Experts Group”.

[0058] In the present invention, the term “equal” includes not only completely equal but also substantially equal in the sense of including an error generally allowed in the technical field to which the technology of the present invention belongs.

[0059] [First embodiment]

[0060] As a first embodiment of the imaging device, a lens-interchangeable digital camera is taken as an example to explain the technology of the present invention. In addition, the technology of the present invention is not limited to the lens-interchangeable type, and can also be applied to a lens-integrated digital camera.

[0061] (Structure of the imaging device)

[0062] Figure 1 FIG. 2 shows an example of the front surface side of the imaging device 10. Figure 1As shown in FIG. 1 , the imaging device 10 is a lens-interchangeable digital camera. The imaging device 10 is composed of a main body 11 and an imaging lens 12 that is replaceably mounted on the main body 11. The imaging lens 12 is mounted via a camera-side mount 11A and a lens-side mount 12A (see FIG. 1 ). Figure 3 ) and is mounted on the front surface 11C side of the main body 11. The imaging lens 12 is an example of a lens involved in the technology of the present invention.

[0063] A dial 13 and a release button 14 are provided on the upper surface of the main body 11. The dial 13 is operated when setting an operation mode, etc. The operation modes of the imaging device 10 include, for example, a still image capturing mode, a moving image capturing mode, and an image display mode. When starting still image capturing or moving image capturing, the release button 14 is operated by the user.

[0064] Figure 2 FIG. 2 shows an example of the back side of the imaging device 10. Figure 2 As shown, a display 15, an indication key 16, and a viewfinder eyepiece portion 18 of a viewfinder (not shown) are provided on the back side 11D of the main body 11. The viewfinder can be an optical viewfinder or an electronic viewfinder. An image based on a video signal obtained by shooting and various menu screens are displayed on the display 15. The indication key 16 receives various instructions.

[0065] Figure 3 An example of the internal structure of the imaging device 10 is shown. The body 11 and the imaging lens 12 are electrically connected by contact between an electrical contact 11B provided on the camera side mount 11A and an electrical contact 12B provided on the lens side mount 12A.

[0066] The imaging lens 12 includes an objective lens 30, a focusing lens 31, a rear lens 32, and an aperture 33. The components are arranged in order from the objective lens side along the optical axis LA of the imaging lens 12 to form the objective lens 30, the aperture 33, the focusing lens 31, and the rear lens 32. The objective lens 30, the focusing lens 31, and the rear lens 32 constitute an imaging optical system. The type, number, and arrangement order of the lenses constituting the imaging optical system are not limited to Figure 3 Example shown.

[0067] Furthermore, the imaging lens 12 has a lens drive control unit 34. The lens drive control unit 34 is composed of, for example, a CPU, a RAM, and a ROM. Furthermore, the ROM includes an erasable EEPROM, a flash memory, and the like. The lens drive control unit 34 is electrically connected to a processor 40 in the main body 11 via the electrical contacts 12B and the electrical contacts 11B.

[0068] The lens drive control unit 34 drives the focus lens 31 and the aperture 33 based on the control signal sent from the processor 40. In order to focus the imaging lens 12, the lens drive control unit 34 controls the drive of the focus lens 31 based on the control signal for focusing sent from the processor 40. The processor 40 performs phase difference focusing.

[0069] The main body 11 is provided with an image sensor 20, a processor 40, an operation unit 42, a memory 45, a display 15, and a shake detection sensor 47. The operations of the image sensor 20, the memory 45, and the display 15 are controlled by the processor 40. The processor 40 is composed of, for example, a CPU, a RAM, and a ROM. In this case, the processor 40 performs various processes based on a program 45A stored in the memory 45. In addition, the processor 40 may be composed of a collection of multiple IC chips. The image sensor 20 is, for example, a CMOS image sensor. The image sensor 20 is an example of an "image sensor" involved in the technology of the present invention.

[0070] The display 15 is based on the image processing unit 52 (refer to Figure 4 ) to display an image. The image includes a still image, a moving image, and a live view image. The live view image is an image displayed in real time on the display 15 by sequentially outputting the image data generated by the image processing unit 52 to the display 15.

[0071] The video signal generated by the image processing unit 52 can be stored as image data in an internal memory (not shown) built into the main body 11 or in a storage medium (for example, a memory card) removable from the main body 11 .

[0072] The operation unit 42 includes the aforementioned dial 13, the release button 14 and the indication key 16 (see Figure 1 and Figure 2 The processor 40 controls each unit in the main body 11 and the lens drive control unit 34 in the imaging lens 12 according to the operation of the operation unit 42 .

[0073] The shake detection sensor 47 detects the amount of shake applied to the imaging device 10. The shake detection sensor 47 is, for example, a five-axis shake detection sensor that detects shake in the roll direction, yaw direction, pitch direction, X direction, and Y direction. The X direction and the Y direction are directions orthogonal to the optical axis LA.

[0074] The shake detection sensor 47 is composed of, for example, a gyro sensor (not shown) for detecting rotational shake and angular shake and an acceleration sensor (not shown) for detecting translational shake. The shake detection sensor 47 outputs a detection signal of the detected shake to the processor 40. In addition, the shake detection sensor 47 may be provided in the imaging lens 12. In this case, the processor 40 may obtain a detection value of the shake amount from the shake detection sensor 47 in the imaging lens 12 via the electrical contact 12B and the electrical contact 11B.

[0075] (Processor Structure)

[0076] Figure 4 FIG. 4 shows an example of the functional structure of the processor 40. The processor 40 implements various functional units by executing processing according to the program 45A stored in the memory 45. Figure 4 As shown, for example, the processor 40 realizes a main control section 50 , an imaging control section 51 , an image processing section 52 , a focus detection section 53 , a brightness detection section 54 , and a correction amount calculation section 55 .

[0077] The main control unit 50 controls the overall operation of the imaging device 10 based on the instruction signal input from the operation unit 42. The imaging control unit 51 controls the imaging sensor 20 to perform imaging processing for causing the imaging sensor 20 to perform imaging operations. The imaging control unit 51 drives the imaging sensor 20 in a still image imaging mode or a moving image imaging mode. In addition, the imaging control unit 51 implements an HDR function that generates an HDR (High Dynamic Range) image with an expanded dynamic range in the still image imaging mode or the moving image imaging mode.

[0078] The user can select a still image capture mode and a moving image capture mode through the operation unit 42. Furthermore, the user can set an exposure value including a shutter speed and an aperture value by operating the operation unit 42. Furthermore, the user can select on / off of the HDR function by operating the operation unit 42.

[0079] Furthermore, the user can select an automatic focus detection (hereinafter referred to as AF) mode by operating the operation unit 42. When the AF mode is selected, the main control unit 50 controls the position of the focus lens 31 based on the focus detection result detected by the focus detection unit 53.

[0080] The focus detection unit 53 detects the phase difference pixels ZL and ZR (reference Figure 7 and Figure 8 ) output signal to perform focus detection. In addition, focus detection refers to detecting the position of the focus lens 31 that is focused on the subject.

[0081] The focus detection unit 53 then generates a focus image based on the AF area 21A (reference AF area 21A) set in the imaging area 21 of the imaging sensor 20. Fig. 9 ) contains phase difference pixels ZL, ZR (reference Figure 7 ) output by the main control unit 50 to perform focus detection. For example, the position and size of the AF area 21A in the imaging area 21 can be set by the user operating the operating unit 42. In addition, the position and size of the AF area 21A can also be set based on the result of subject detection (for example, face detection) performed by the main control unit 50 using the video signal. In addition, the AF area 21A is an example of a "region of interest" involved in the technology of the present invention. In addition, when shooting with a fixed focus, it can also be another subject area that is different from the focus area and is determined by the user or the imaging device.

[0082] The brightness detection unit 54 detects the brightness of the AF area 21A (an example of the region of interest) based on the video signal output from the imaging sensor 20. The main control unit 50 performs exposure time setting processing based on the brightness of the AF area 21A detected by the brightness detection unit 54.

[0083] The correction amount calculation section 55 calculates a correction amount for electronic shake correction (ie, electronic vibration reduction processing) based on the shake amount detection value output from the shake detection sensor 47. The correction amount calculation section 55 outputs the calculated correction amount to the image processing section 52.

[0084] The image processing unit 52 generates image data in a predetermined file format (e.g., JPEG format, etc.) by performing various image processing on the video signal. The image data output from the image processing unit 52 is recorded in, for example, the memory 45. The video signal constituting the image data is output from the image processing unit 52 to the display 15, and the image is displayed on the display 15.

[0085] The image processing unit 52 includes an HDR synthesis unit 56 and a shake correction unit 57. The HDR synthesis unit 56 operates when the HDR function is turned on, and the shake correction unit 57 operates when the moving image capture mode is selected.

[0086] When the HDR function is turned on, the HDR synthesis unit 56 generates an HDR image by synthesizing a pair of video signals having different exposure times output from the imaging sensor 20 .

[0087] In the motion image recording mode, the shake correction unit 57 performs electronic anti-shake processing on the video signal based on the correction amount input from the correction amount calculation unit 55. For example, the electronic anti-shake processing is performed by changing the cropping area based on the correction amount for each frame of the video signal. When the HDR function is turned on, the shake correction unit 57 performs electronic anti-shake processing on the video signal (i.e., HDR image) generated by the HDR synthesis unit 56. As a result, the electronic anti-shake processing can be simplified and the electronic anti-shake processing in the processor 40 can be accelerated. In addition, the shake correction unit 57 preferably performs electronic anti-shake processing on the HDR image that has been subjected to de-mosaic processing.

[0088] (Structure of image sensor)

[0089] Figure 5 An example of the structure of the imaging sensor 20 is shown. Figure 5 The imaging sensor 20 shown is a CMOS type image sensor and includes an imaging area 21 , a vertical scanning circuit 22 , a line memory 23 , a horizontal scanning circuit 24 , and an output amplifier 25 .

[0090] In the imaging region 21, a plurality of pixels 26 are arranged in a two-dimensional matrix along the X and Y directions. In the imaging region 21, a plurality of row selection lines L1 and a plurality of row reset lines L2 are wired along the X direction, and a plurality of column signal lines L3 are wired along the Y direction.

[0091] The column signal line L3 extends in the Y direction. The row selection line L1 and the row reset line L2 extend in the X direction intersecting the Y direction. The Y direction is an example of the "first direction" involved in the technology of the present invention. The X direction is an example of the "second direction" involved in the technology of the present invention.

[0092] The pixels 26 are connected to the row selection line L1, the row reset line L2, and the column signal line L3. Hereinafter, the plurality of pixels 26 arranged in the X direction are sometimes referred to as a "row". Although the details will be described later, some of the plurality of pixels 26 are phase difference pixels for focusing.

[0093] The pixel 26 includes a photodiode D1, an amplifier transistor M1, a pixel selection transistor M2, and a reset transistor M3. The photodiode D1 generates a signal charge corresponding to the amount of incident light by photoelectrically converting the incident light, and accumulates the generated signal charge. The amplifier transistor M1 generates a voltage corresponding to the charge amount of the signal charge accumulated in the photodiode D1 (hereinafter referred to as a pixel signal S).

[0094] The pixel selection transistor M2 is controlled by the vertical scanning circuit 22 via the row selection line L1, and outputs the pixel signal S generated by the amplifier transistor M1 to the column signal line L3. The reset transistor M3 is controlled by the vertical scanning circuit 22 via the row reset line L2, and discards the signal charge accumulated in the photodiode D1 to the power supply line. Hereinafter, discarding the signal charge from the photodiode D1 is referred to as resetting the pixel 26.

[0095] The vertical scanning circuit 22 generates a row selection signal SEL and a reset signal RST based on a vertical synchronization signal input from the imaging control unit 51. In a signal readout operation, the vertical scanning circuit 22 applies the row selection signal SEL to the row selection line L1, thereby outputting a pixel signal S from the pixel 26 connected to the row selection line L1 to the column signal line L3.

[0096] Furthermore, during the reset operation, the vertical scanning circuit 22 supplies the reset signal RST to the row reset line L2 , thereby resetting the pixels 26 connected to the row reset line L2 .

[0097] The line memory 23 stores pixel signals S output from one row of pixels 26. The line memory 23 is composed of a capacitor or the like. The line memory 23 is connected to the horizontal output line 24A via a transistor 29 as a switch. The output amplifier 25 is connected to the end of the horizontal output line 24A. The horizontal scanning circuit 24 sequentially outputs the pixel signals S for one row stored in the line memory 23 to the horizontal output line 24A by performing horizontal scanning by sequentially selecting the transistors 29. The pixel signals S output to the horizontal output line 24A are output to the external image processing unit 52 as video signals via the output amplifier 25.

[0098] The operations of the vertical scanning circuit 22, the line memory 23, and the horizontal scanning circuit 24 are controlled by the imaging control unit 51 (see Figure 4 The imaging control unit 51 controls the vertical scanning circuit 22 to sequentially select the row selection line L1 row by row and output the pixel signal S. Furthermore, the imaging control unit 51 controls the vertical scanning circuit 22 to reset the pixel 26.

[0099] In the moving image capture mode, the image capture control unit 51 performs the image capture at a predetermined frame period T (reference Fig.10 and Fig.11 ) drives the imaging sensor 20. The imaging control unit 51 causes the imaging sensor 20 to repeatedly execute the operation of reading and resetting the video signal at a frame period T.

[0100] In addition, the structure of the imaging sensor 20 is not limited to Figure 5 For example, an A / D converter may be provided in the imaging sensor 20 .

[0101] (Pixel structure)

[0102] The plurality of pixels 26 arranged in the imaging region 21 include imaging pixels N and phase difference pixels ZL and ZR for imaging. Figure 6 An example of the structure of the imaging pixel N is shown. Figure 7 An example of the structure of the phase difference pixels ZL and ZR is shown. The phase difference pixels ZL and ZR each receive one of the light beams split in the X direction around the principal ray.

[0103] like Figure 6 As shown, the imaging pixel N is configured to include a photodiode D1 as a photoelectric conversion element, a color filter CF, and a microlens ML. The color filter CF is arranged between the photodiode D1 and the microlens ML.

[0104] The color filter CF transmits light of any one color of R (red), G (green), or B (blue). The microlens ML converges the light beam LF incident from the exit pupil EP of the imaging lens 12 at substantially the center of the photodiode D1 via the color filter CF.

[0105] like Figure 7 As shown, each of the phase difference pixels ZL and ZR includes a photodiode D1, a light shielding layer SF, and a microlens ML. The microlens ML converges the light beam LF incident from the exit pupil EP of the imaging lens 12 substantially at the center of the photodiode D1, similarly to the imaging pixel N.

[0106] The light shielding layer SF is formed of a metal film or the like, and is disposed between the photodiode D1 and the microlens ML. The light shielding layer SF shields a portion of the light beam LF incident on the photodiode D1 via the microlens ML.

[0107] In the phase difference pixel ZL, the light shielding layer SF shields the negative side (first side) in the X direction with the center of the photodiode D1 as a reference. That is, in the phase difference pixel ZL, the light shielding layer SF allows the light beam LF from the exit pupil EP1 on the first side to be incident on the photodiode D1, and shields the light beam LF from the exit pupil EP2 on the positive side (second side) in the X direction.

[0108] In the phase difference pixel ZR, the light shielding layer SF shields the positive side (second side) in the X direction with the center of the photodiode D1 as a reference. That is, in the phase difference pixel ZR, the light shielding layer SF allows the light beam LF from the exit pupil EP2 on the second side to be incident on the photodiode D1, and shields the light beam LF from the exit pupil EP1 on the negative side (first side) in the X direction.

[0109] (Pixel arrangement)

[0110] Figure 8An example of a pixel arrangement of the imaging sensor 20 is shown. Figure 8 The color arrangement of the color filters CF shown is a so-called Bayer arrangement. The Bayer arrangement is a color arrangement in which G color filters CF are arranged in two diagonal pixels among four pixels of 2×2, and R and B color filters CF are arranged in the other two pixels. Figure 8 "R" in represents an imaging pixel N provided with a color filter CF having R. "G" represents an imaging pixel N provided with a color filter CF having G. "B" represents an imaging pixel N provided with a color filter CF having B. The color arrangement of the color filter CF is not limited to the Bayer arrangement, and may be other color arrangements.

[0111] Phase difference pixels ZL and ZR are arranged in the imaging area 21 by replacing a part of the imaging pixels N arranged in the Bayer method. In the present embodiment, the phase difference pixels ZL and ZR are arranged to replace a part of the imaging pixels N of the color filter CF provided with B. The phase difference pixels ZL and ZR are arranged in the X direction. In the present embodiment, one of the phase difference pixels ZL and ZR is arranged every 2 pixels (that is, every other pixel) in the X direction. That is, one imaging pixel N is arranged between the phase difference pixel ZL and the phase difference pixel ZR.

[0112] Phase difference pixels ZL and ZR are arranged every 16 pixels in the Y direction. A plurality of imaging pixels N (G and B) are arranged between the phase difference pixels ZL arranged in the Y direction. A plurality of imaging pixels N (G and B) are arranged between the phase difference pixels ZR arranged in the Y direction.

[0113] In addition, the arrangement pattern of the phase difference pixels ZL and ZR is not limited to Figure 8 For example, phase difference pixels ZL and ZR are arranged adjacent to each other in the X direction.

[0114] The plurality of pixels 26 included in the imaging area 21 (see Figure 5 ) is divided into a first pixel group PG1 and a second pixel group PG2. The first pixel group PG1 includes a plurality of phase difference pixels ZL, ZR and a plurality of imaging pixels N arranged in the X direction. The second pixel group PG2 includes a plurality of imaging pixels N arranged in the X direction. The first pixel group PG1 and the second pixel group PG2 are alternately arranged in the Y direction.

[0115] In this embodiment, when the row address is set to n (here, n is a natural number including 0), the first pixel group PG1 includes a row with a row address of "4n" and a row with a row address of "4n+1". In addition, the second pixel group PG2 includes a row with a row address of "4n+2" and a row with a row address of "4n+3". The phase difference pixels ZL and ZR exist in the row with a row address of "16n". That is, the phase difference pixels ZL and ZR are only included in the first pixel group PG1.

[0116] Fig. 9 The focus control and exposure control by the main control unit 50 will be described. Fig. 9 As shown in FIG. 1 , the focus detection unit 53 performs focus detection based on the signals (hereinafter referred to as phase difference pixel signals) output from the phase difference pixels ZL and ZR included in the AF area 21A set in the imaging area 21, thereby generating focus information. The focus information is information indicating the position of the focus lens 31 focused on the subject. Specifically, the focus detection unit 53 generates the focus information based on the phase difference between the phase difference pixel signal output from the phase difference pixel ZL and the phase difference pixel signal output from the phase difference pixel ZR, and outputs the generated focus information to the main control unit 50.

[0117] The brightness detection unit 54 generates brightness information indicating the brightness of the AF area 21A based on the signal output from the imaging pixel N included in the AF area 21A (hereinafter referred to as the imaging pixel signal). The brightness detection unit 54 calculates the brightness value for each pixel based on the imaging pixel signal, and calculates the average brightness value in the AF area 21A. The brightness detection unit 54 outputs the calculated average brightness value to the main control unit 50 as brightness information. In addition, the brightness information is an example of the "information of the subject" involved in the technology of the present invention.

[0118] The main control unit 50 controls the position of the focus lens 31 via the lens drive control unit 34 based on the focus information input from the focus detection unit 53. Furthermore, the main control unit 50 controls the exposure time of the first pixel group PG1 and the second pixel group PG2 via the imaging control unit 51 based on the brightness information input from the brightness detection unit 54. Specifically, the main control unit 50 makes one of the first exposure time E1 for which the first pixel group PG1 is exposed and the second exposure time E2 for which the second pixel group PG2 is exposed shorter than the other based on the brightness information.

[0119] (Camera timing)

[0120] Fig.10 and Fig.11 An example of the imaging timing of the imaging sensor 20 when the HDR function is turned on in the moving image imaging mode is shown.

[0121] The frame period T of moving image pickup is defined by a vertical synchronization signal VD supplied from the image pickup control unit 51 to the vertical scanning circuit 22. The vertical scanning circuit 22 supplies a row selection signal SEL and a reset signal RST to the first pixel group PG1 and the second pixel group PG2 based on the vertical synchronization signal VD.

[0122] Specifically, the vertical scanning circuit 22 supplies the row selection signal SEL to the first pixel group PG1 and the second pixel group PG2 in synchronization with the vertical synchronization signal VD. In this embodiment, the vertical scanning circuit 22 selects the row selection line L1 while changing the row address in the order of 0, 1, 2, ... (refer to Figure 5 ) and supplies a row selection signal SEL to the selected row selection line L1.

[0123] The vertical scanning circuit 22 supplies the reset signal RST to the first pixel group PG1 and the second pixel group PG2 according to their respective exposure times. The first exposure time E1 is the time from the input of the reset signal RST to the input of the row selection signal SEL in the first pixel group PG1 (i.e., the charge accumulation time). The second exposure time E2 is the time from the input of the reset signal RST to the input of the row selection signal SEL in the second pixel group PG2. The exposure period is staggered by row. That is, the imaging sensor 20 is exposed by an electronic shutter of a focal plane method.

[0124] In addition, Fig.10 and Fig.11 In the figure, the hatched period is the light shielding period. During the light shielding period, the reset signal RST is periodically supplied to the first pixel group PG1 and the second pixel group PG2. Also, during the light shielding period, the reset signal RST is stably supplied to the first pixel group PG1 and the second pixel group PG2.

[0125] Fig.10 An example of the imaging timing when the brightness of the AF area 21A is equal to or greater than an arbitrary first threshold value is shown. In this case, the main control unit 50 controls the reset timing so that E1<E2. Fig.11 An example of the imaging timing when the brightness of the AF area 21A is lower than the first threshold value is shown. In this case, the main control unit 50 controls the reset timing so that E2<E1.

[0126] (Exposure Control)

[0127] Next, refer to Fig.12 The flowchart shown in FIG. 1 illustrates an example of the flow of exposure control when the HDR function is turned on in the moving image capture mode. First, the main control unit 50 determines whether there is an instruction to start moving image capture based on an operation of the operation unit 42 (step S10).

[0128] When the main control unit 50 determines that there is an instruction to start moving image capture (step S10: Yes), it controls the image capture control unit 51 to cause the image sensor 20 to perform an image capture operation at a predetermined frame cycle T, and acquires brightness information of the AF area 21A from the brightness detection unit 54 (step S11). In addition, the main control unit 50 acquires brightness information from the brightness detection unit 54 at each frame cycle T.

[0129] The main control unit 50 determines whether the brightness of the AF area 21A is greater than or equal to the first threshold value based on the acquired brightness information (step S12). When the main control unit 50 determines that the brightness of the AF area 21A is greater than or equal to the first threshold value (step S12: Yes), the main control unit 50 controls the reset timing so that E1<E2 in the next frame cycle T (step S13). On the other hand, when the main control unit 50 determines that the brightness of the AF area 21A is less than the first threshold value (step S12: No), the main control unit 50 controls the reset timing so that E2<E1 in the next frame cycle T (step S14).

[0130] Next, the main control unit 50 determines whether there is an instruction to end the motion image shooting based on the operation of the operation unit 42 (step S15). When the main control unit 50 determines that there is no instruction to end the motion image shooting (step S15: No), the process is transferred to step S11 to reacquire the brightness information. The process from step S11 to step S15 is repeated every frame period T until it is determined in step S15 that there is an instruction to end the motion image shooting. Then, when the main control unit 50 determines that there is an instruction to end the motion image shooting (step S15: Yes), the process ends.

[0131] As described above, when the brightness of the AF area 21A is greater than or equal to the first threshold value, the main control unit 50 makes the first exposure time E1 for the first pixel group PG1 including the phase difference pixels ZL and ZR to be exposed shorter than the second exposure time E2 for the second pixel group PG2 not including the phase difference pixels ZL and ZR to be exposed. In this way, when the brightness of the AF area 21A is greater than or equal to the first threshold value, the main control unit 50 makes the first exposure time E1 shorter than the second exposure time E2 so that overexposure due to signal saturation of the phase difference pixels ZL and ZR does not occur, and thus phase difference information with further improved accuracy can be obtained.

[0132] On the other hand, when the brightness of the AF area 21A is less than the first threshold value, the main control unit 50 makes the second exposure time E2 for the second pixel group PG2 not including the phase difference pixels ZL and ZR to be exposed shorter than the first exposure time E1 for the first pixel group PG1 including the phase difference pixels ZL and ZR to be exposed. In this way, when the brightness of the AF area 21A is less than the first threshold value, the main control unit 50 makes the first exposure time E1 longer than the second exposure time E2 so that blackening due to small signals of the phase difference pixels ZL and ZR does not occur, and phase difference information with better accuracy can be obtained.

[0133] Therefore, when the first exposure time E1 and the second exposure time E2 are different in order to expand the dynamic range, by performing the above-mentioned exposure control, it is possible to obtain accurate phase difference information regardless of the brightness of the AF area 21A. Thus, even when the dynamic range is expanded, it is possible to perform focusing with good accuracy using the phase difference method.

[0134] (HDR synthesis processing)

[0135] Fig.13 The HDR synthesis process by the HDR synthesis unit 56 is schematically shown. The HDR synthesis unit 56 generates an HDR image by adding signals output from the first pixel group PG1 and the second pixel group PG2 adjacent to each other in the Y direction.

[0136] Specifically, the HDR synthesis unit 56 adds the signal output from the first pixel group PG1 (hereinafter referred to as the PG1 signal) and the signal output from the second pixel group PG2 (hereinafter referred to as the PG2 signal) for each corresponding pair of pixels. In addition, a corresponding pair of pixels refers to a pair of pixels located at the same address in the X direction and only two pixels are located at different addresses in the Y direction.

[0137] For example, in HDR synthesis, only the imaging pixel signal is used instead of the phase difference pixel signal. Therefore, for the phase difference pixels ZL and ZR, in the first pixel group PG1, the signal obtained by performing pixel complementation using the imaging pixel signal of the imaging pixel N existing around the phase difference pixels ZL and ZR is used for HDR synthesis.

[0138] In addition to the imaging pixel signals, the phase difference pixel signals may be used for HDR synthesis. Since the phase difference pixels ZL and ZR receive less light than the imaging pixel N, the HDR synthesis unit 56 may perform gain correction on the phase difference pixel signals and use them for HDR synthesis.

[0139] The HDR synthesis unit 56 may synthesize the PG1 signal and the PG2 signal in the state of analog signals, or synthesize them after A / D conversion to digital signals. Furthermore, the HDR synthesis unit 56 may synthesize the PG1 signal and the PG2 signal in the state of RAW data, or synthesize them after performing de-mosaic processing.

[0140] In addition, in the present embodiment, the HDR synthesis unit 56 adds a value obtained by multiplying the PG1 signal by the coefficient α and a value obtained by multiplying the PG2 signal by the coefficient 1-α. Here, the coefficient α is a value satisfying 0≤α≤1.

[0141] Fig.14 is a diagram showing an example of a relationship between coefficient α and pixel brightness. Fig.14 Based on the relationship shown in FIG. 1 , the coefficient α is calculated for each pair of pixels to be added, and the PG1 signal and the PG2 signal are added.

[0142] When E1<E2, the HDR synthesis unit 56 determines the coefficient α using the relational expression F1 in which the coefficient α increases as the brightness increases. On the other hand, when E2<E1, the HDR synthesis unit 56 determines the coefficient α using the relational expression F2 in which the coefficient α decreases as the brightness increases. In either case, the higher the brightness, the greater the addition ratio of the signal on the short exposure side. The brightness of a pixel is based on the brightness information (R, G, B signal values) of the pixel on the side with the shorter exposure time, for example.

[0143] Fig.15 The signal levels of the PG1 signal and the PG2 signal relative to the amount of incident light are schematically shown. When E1 < E2, the PG1 signal has an effect of improving the sensitivity of the area with a large amount of incident light, and the PG2 signal has an effect of improving the sensitivity of the area with a small amount of incident light. On the other hand, when E2 < E1, the PG1 signal has an effect of improving the sensitivity of the area with a small amount of incident light, and the PG2 signal has an effect of improving the sensitivity of the area with a large amount of incident light. As a result, the dynamic range of the HDR image generated by adding the PG1 signal and the PG2 signal is expanded.

[0144] As described above, according to the technique of the present invention, it is possible to appropriately determine whether to set the pixel group (the first pixel group PG1 in this embodiment) including the phase difference pixels to short exposure or long exposure.

[0145] [Second embodiment]

[0146] Next, the second embodiment is described. In the first embodiment, in the moving image shooting mode and when the HDR function is turned on, the main control unit 50 performs exposure control based on the "brightness of the AF area 21A" as the information of the subject. In contrast, in the second embodiment, the main control unit 50 performs exposure control based on the "moving speed of the subject" as the information of the subject.

[0147] In the present embodiment, the main control unit 50 detects the moving speed of the subject by, for example, obtaining a moving vector of the subject between frames based on a video signal output from the imaging area 21 of the imaging sensor 20 at each frame period T. Alternatively, the main control unit 50 may detect the moving speed of the subject based on a video signal output from the AF area 21A.

[0148] Fig.16 An example of exposure control according to the second embodiment is described below. First, the main control unit 50 determines whether there is an instruction to start moving image capture based on an operation of the operation unit 42 (step S20).

[0149] When the main control unit 50 determines that there is an instruction to start motion image capture (step S20: Yes), it controls the image capture control unit 51 to cause the image sensor 20 to perform image capture at a predetermined frame period T, and detects the moving speed of the subject based on the video signal output from the image sensor 20 (step S21).

[0150] The main control unit 50 determines whether the moving speed is greater than or equal to an arbitrary second threshold value based on the detected moving speed of the subject (step S22). When the main control unit 50 determines that the moving speed is greater than or equal to the second threshold value (step S22: Yes), the main control unit 50 controls the reset timing so that E1<E2 in the next frame cycle T (step S23). On the other hand, when the main control unit 50 determines that the moving speed is less than the second threshold value (step S22: No), the main control unit 50 controls the reset timing so that E2<E1 in the next frame cycle T (step S24).

[0151] Next, the main control unit 50 determines whether there is an instruction to end the motion image shooting based on the operation of the operation unit 42 (step S25). When the main control unit 50 determines that there is no instruction to end the motion image shooting (step S25: No), the process is transferred to step S21 to re-detect the moving speed of the subject. The process from step S21 to step S25 is repeated every frame period T until it is determined in step S25 that there is an instruction to end the motion image shooting. Then, when the main control unit 50 determines that there is an instruction to end the motion image shooting (step S25: Yes), the process ends.

[0152] As described above, when the moving speed of the subject is greater than or equal to the second threshold, the main control unit 50 makes the first exposure time E1 for the first pixel group PG1 including the phase difference pixels ZL and ZR to be exposed shorter than the second exposure time E2 for the second pixel group PG2 not including the phase difference pixels ZL and ZR to be exposed. On the other hand, when the moving speed of the subject is less than the second threshold, the main control unit 50 makes the second exposure time E2 for the second pixel group PG2 not including the phase difference pixels ZL and ZR to be exposed shorter than the first exposure time E1 for the first pixel group PG1 including the phase difference pixels ZL and ZR to be exposed.

[0153] Thus, in this embodiment, when the moving speed of the subject is fast, the exposure time is shortened to obtain the phase difference information. On the other hand, when the moving speed of the subject is slow, the exposure time is lengthened to obtain the phase difference information in a fully exposed state.

[0154] [Third embodiment]

[0155] Next, the third embodiment is described. In the first embodiment, as Fig.10 and Fig.11 As shown, when the main control unit 50 reads out the video signal from the image sensor 20, it selects the row address in the order of 0, 1, 2, ... and reads out the signal. That is, in the first embodiment, the main control unit 50 reads out the signal alternately from the first pixel group PG1 and the second pixel group PG2. In contrast, in the third embodiment, the main control unit 50 reads out the signal from one of the first pixel group PG1 and the second pixel group PG2, and then reads out the signal from the other of the first pixel group PG1 and the second pixel group PG2.

[0156] Fig.17 An example of imaging timing according to the third embodiment is shown. Fig.17 As shown, in this embodiment, first, the row selection signal SEL is supplied to the first pixel group PG1 with row addresses "4n" and "4n+1" to perform signal reading. Then, the row selection signal SEL is supplied to the second pixel group PG2 with row addresses "4n+2" and "4n+3" to perform signal reading. In addition, the signal reading from the first pixel group PG1 and the signal reading from the second pixel group PG2 are continuously performed in the same frame period T.

[0157] In the present embodiment, similarly to the signal readout, the resetting is performed by resetting the first pixel group PG1 and then resetting the second pixel group PG2 .

[0158] In addition, Fig.17In contrast to the example shown, the main control unit 50 may read out signals from the first pixel group PG1 after reading out signals from the second pixel group PG2. However, since the phase difference pixels ZL and ZR are included only in the first pixel group PG1, it is preferred to read out signals from the first pixel group PG1 before the second pixel group PG2. By reading out signals from the first pixel group PG1 first, the phase difference information can be acquired as soon as possible after the start of the frame period.

[0159] And, in Fig.17 In the example shown, the main control unit 50 reads signals in the same direction in the Y direction for both the first pixel group PG1 and the second pixel group PG2. Specifically, when the direction of increasing the row address is set as the positive direction and the direction of decreasing the row address is set as the negative direction, Fig.17 In the example shown, the main control unit 50 reads signals in the positive direction for both the first pixel group PG1 and the second pixel group PG2. In addition, the positive direction in the Y direction is an example of "one direction in the first direction" involved in the technology of the present invention. And, the negative direction in the Y direction is an example of "another direction in the first direction" involved in the technology of the present invention.

[0160] The signal readout directions may be opposite in the first pixel group PG1 and the second pixel group PG2 . Also, the signal readout direction may be changed according to the position of the subject in the imaging region 21 .

[0161] Fig.18 An example of changing the direction of signal readout of the first pixel group PG1 and the second pixel group PG2 according to the position of the subject is described. In this example, the main control unit 50 performs subject detection (for example, face detection) based on the video signal output from the imaging sensor 20, and detects the position of the subject in the imaging area 21. Then, the main control unit 50 determines in which area of ​​the positive direction side and the negative direction side in the Y direction the subject is located with respect to the virtual line C located at the center of the imaging area 21 in the Y direction.

[0162] When the position of the subject is more toward the positive direction side than the virtual line C, the main control unit 50 reads out the first pixel group PG1 from the negative direction side of the Y direction to the positive direction signal, and then reads out the second pixel group PG2 from the positive direction side of the Y direction to the negative direction signal. On the other hand, when the position of the subject is more toward the negative direction side than the virtual line C, the main control unit 50 reads out the first pixel group PG1 from the positive direction side of the Y direction to the negative direction signal, and then reads out the second pixel group PG2 from the negative direction side of the Y direction to the positive direction signal.

[0163] Thus, by changing the direction of signal reading of the first pixel group PG1 and the second pixel group PG2 according to the position of the subject, the time difference between the PG1 signal and the PG2 signal read from the vicinity of the subject is reduced, thereby improving the image quality of the subject in the HDR image.

[0164] In addition, Fig.18 In the example shown, the first pixel group PG1 is read out before the second pixel group PG2, but the second pixel group PG2 may be read out before the first pixel group PG1. Furthermore, when the subject intersects with the virtual line C, the position of the subject may be determined based on the area on the positive direction side and the negative direction side where the center of gravity of the subject is located relative to the virtual line C. Furthermore, when the subject intersects with the virtual line C, the position of the subject may be determined based on the area of ​​the subject on the positive direction side and the negative direction side relative to the virtual line C being larger.

[0165] [Fourth embodiment]

[0166] Next, the fourth embodiment is described. In the first embodiment, the center of gravity of the first pixel group PG1 and the second pixel group PG2 are offset by 2 pixels in the Y direction. That is, the PG1 signal read out from the first pixel group PG1 and the PG2 signal read out from the second pixel group PG2 are phase-shifted in the Y direction. In the fourth embodiment, in the image processing unit 52, after performing phase processing for reducing the phase deviation between the PG1 signal and the PG2 signal, the PG1 signal and the PG2 signal are synthesized.

[0167] Fig.19 HDR synthesis processing including phase processing is schematically shown. In the present embodiment, the image processing unit 52 includes a low-pass filter processing unit 58. The low-pass filter processing unit 58 performs low-pass filter processing on the image generated by the PG1 signal (hereinafter referred to as the PG1 image) and the PG2 image of the image generated by the PG2 signal (hereinafter referred to as the PG2 image). The PG1 image and the PG2 image are, for example, images subjected to demosaic processing.

[0168] In this embodiment, the HDR synthesis unit 56 synthesizes the PG1 image and the PG2 image processed by the low-pass filter to generate the HDR image. Since the PG2 image is blurred by the low-pass filter, the phase deviation with respect to the PG1 image is reduced.

[0169] In this embodiment, the PG2 image is subjected to a low-pass filter process. This is because the PG2 image is not the “main image” in either the case where the subject is bright or dim. Fig.12As shown in FIG. 1 , when the brightness of the AF area 21A is greater than the first threshold, E1<E2, so the second pixel group PG2 is overexposed. On the contrary, when the brightness of the AF area 21A is less than the first threshold, E2<E1, so the second pixel group PG2 is underexposed. In this way, the second pixel group PG2 is overexposed when the subject is bright, and is underexposed when the subject is dark. Therefore, in either case, the PG2 image read out from the second pixel group PG2 is not the main image, and the PG1 image becomes the main image.

[0170] As described above, in the present embodiment, the synthesis process is performed after the low-pass filter process is performed on the image that is not the main image between the PG1 image and the PG2 image, so that the phase deviation can be reduced while suppressing the degradation of the image quality.

[0171] In addition, the phase deviation between the PG1 signal and the PG2 signal can also be reduced by performing pixel complementation processing on the PG2 image instead of low-pass filter processing. By performing pixel complementation processing on the PG2 image, the pixel signal of the pixel position corresponding to the PG1 image can be calculated. In this way, phase processing is a concept that includes low-pass filter processing and pixel complementation processing.

[0172] [Fifth embodiment]

[0173] Next, the fifth embodiment will be described. As described above, in the first embodiment, the main control unit 50 executes the first mode, which includes: a first setting process of making one of the first exposure time E1 and the second exposure time E2 shorter than the other; and a first synthesis process of generating an HDR image (first video signal) by synthesizing the PG1 signal and the PG2 signal.

[0174] In the fifth embodiment, the main control unit 50 can execute a second mode of changing the exposure time for each frame period in addition to the first mode. The second mode includes a second setting process and a second synthesis process. In the second setting process, the main control unit 50 makes one of the first exposure time E1 and the second exposure time E2 during the first frame period and the first exposure time and the second exposure time during the second frame period after the first frame period shorter than the other. In the second synthesis process, the main control unit 50 generates an HDR image (a second video signal) by synthesizing the signal read out from the first pixel group PG1 and the second pixel group PG2 during the first frame period and the signal read out from the first pixel group PG1 and the second pixel group PG2 during the second frame period.

[0175] Fig. 20An example of the imaging timing in the second mode is shown. In the second mode, the first exposure time E1 and the second exposure time E2 are equal in each of the first frame period and the second frame period. That is, in the second mode, the main control unit 50 does not perform the exposure time setting process based on the brightness information of the AF area 21A.

[0176] Furthermore, the first exposure time E1 and the second exposure time E2 are different in the first frame period and the second frame period. In the present embodiment, the first exposure time E1 and the second exposure time E2 in the first frame period are shorter than the first exposure time E1 and the second exposure time E2 in the second frame period. The first frame period and the second frame period are alternately repeated in each frame period T.

[0177] In the second mode, since the HDR image is generated by synthesizing the PG1 signal and the PG2 signal of two frames, the resolution of the HDR image is high but the frame rate is low. In contrast, in the first mode, since the HDR image is generated by synthesizing the PG1 signal and the PG2 signal for each frame, the resolution of the HDR image is low although the frame rate is high. Therefore, the main control unit 50 preferably sets the upper limit value of the frame rate in the first mode to be higher than the upper limit value of the frame rate in the second mode.

[0178] Furthermore, it is preferable that the main control unit 50 allows the user to select the first mode and the second mode by operating the operation unit 42. Thus, the user can select an appropriate mode depending on which of the resolution and the frame rate is more important.

[0179] [Sixth embodiment]

[0180] Next, the sixth embodiment will be described. In the first embodiment, as Figure 8 As shown, of the first pixel group PG1 and the second pixel group PG2, only the first pixel group PG1 includes the phase difference pixels ZL and ZR. In the sixth embodiment, the first pixel group PG1 includes the first phase difference pixels Z1L and Z1R, and the second pixel group PG2 includes the second phase difference pixels Z2L and Z2R.

[0181] Fig.21 An example of the pixel arrangement of the imaging sensor 20 according to the sixth embodiment is shown. The first phase difference pixels Z1L and Z1R and the second phase difference pixels Z2L and Z2R are alternately arranged every 18 pixels in the Y direction. That is, the first phase difference pixels Z1L and Z1R are included in the first pixel group PG1, and the second phase difference pixels Z2L and Z2R are included in the second pixel group PG2.

[0182] Fig. 22An example of the structure of the first phase difference pixels Z1L and Z1R is shown. The structures of the first phase difference pixels Z1L and Z1R are similar to those of the phase difference pixels ZL and ZR of the first embodiment (see Figure 7 Since about 50% of the light receiving surface of the photodiode D1 is shielded by the light shielding layer SF, the first phase difference pixels Z1L and Z1R each have an opening area (hereinafter referred to as the first opening area) that is about 50% of the light receiving area of ​​the photodiode D1.

[0183] Fig.23 An example of the structure of the second phase difference pixel Z2L, Z2R is shown. In the second phase difference pixel Z2L, Z2R, for example, more than 50% of the area of ​​the light receiving surface of the photodiode D1 is shielded by the light shielding layer SF. That is, the second phase difference pixel Z2L, Z2R has a second opening area that is smaller than the first opening area of ​​the first phase difference pixel Z1L, Z1R.

[0184] In this embodiment, the main control unit 50 makes the first exposure time E1 of the first pixel group PG1 shorter than the second exposure time E2 of the second pixel group PG2, regardless of the brightness of the AF area 21A. That is, the exposure time of the first phase difference pixels Z1L and Z1R is shorter than the exposure time of the second phase difference pixels Z2L and Z2R.

[0185] Thus, in the present embodiment, in the second pixel group PG2 for long exposure, the phase difference pixel signals are obtained through the second phase difference pixels Z2L and Z2R with small aperture areas, so the phase difference pixel signals are less likely to be overexposed. In addition, in the first pixel group PG1 for short exposure, the phase difference pixel signals are obtained through the first phase difference pixels Z1L and Z1R with large aperture areas, so the phase difference pixel signals are less likely to turn black. Therefore, in the present embodiment, appropriate phase difference information without overexposure and turning black can be obtained.

[0186] Furthermore, in this embodiment, the phase difference pixel signal is unlikely to become a high level that causes overexposure or a low level that causes blackening, so when the phase difference pixel signal is used in addition to the imaging pixel signal to generate an HDR image, the image quality is improved.

[0187] [Modifications]

[0188] In each of the above-mentioned embodiments, the HDR synthesis process of synthesizing the PG1 signal and the PG2 signal is performed inside the camera device 10, but the PG1 signal and the PG2 signal may be output as RAW data to the outside of the camera device 10. Furthermore, the PG1 signal and the PG2 signal output from the camera device 10 may be read into an external device such as a personal computer, and the HDR synthesis process may be performed using software installed in the external device.

[0189] Furthermore, in each of the above-mentioned embodiments, the imaging sensor 20 may be composed of the imaging region 21, the vertical scanning circuit 22, the line memory 23, the horizontal scanning circuit 24, and the output amplifier 25, but the imaging sensor 20 may be composed of only the imaging region 21. The vertical scanning circuit 22, the line memory 23, the horizontal scanning circuit 24, and the output amplifier 25 may be provided in the processor 40.

[0190] Furthermore, the above-mentioned embodiments can be combined with each other as long as no contradiction occurs.

[0191] In the above-mentioned embodiments, as the hardware structure of the control unit, the processor 40 is used as an example, and various processors shown below can be used. Among the above-mentioned various processors, in addition to the CPU which is a common processor that functions by executing software (program), there are also processors such as FPGA whose circuit structure can be changed after manufacturing. FPGA includes a dedicated circuit, which is a processor having a circuit structure specially designed to execute specific processing such as PLD or ASIC.

[0192] The control unit may be composed of one of these various processors, or may be composed of a combination of two or more processors of the same type or different types (for example, a combination of multiple FPGAs or a combination of a CPU and an FPGA). Furthermore, multiple control units may be composed of one processor.

[0193] There are multiple examples of multiple control units composed of one processor. In the first example, as represented by computers such as clients and servers, there is a method in which one processor is composed of a combination of one or more CPUs and software, and the processor functions as multiple control units. In the second example, as represented by a system on chip (System On Chip: SOC), there is a method in which a processor that realizes the functions of the entire system including multiple control units by one IC chip is used. In this way, the control unit can use one or more of the above-mentioned various processors to form a hardware structure.

[0194] Furthermore, as the hardware configuration of these various processors, more specifically, a circuit formed by combining circuit elements such as semiconductor elements can be used.

[0195] The recorded contents and illustrated contents shown above are detailed descriptions of the part involved in the technology of the present invention, and are only an example of the technology of the present invention. For example, the description related to the above-mentioned structure, function, action and effect is a description related to an example of the structure, function, action and effect of the part involved in the technology of the present invention. Therefore, without departing from the technical gist of the present invention, unnecessary parts can be deleted from the recorded contents and illustrated contents shown above, or new elements can be added or replaced. In addition, in order to avoid complexity and to make it easy to understand the part involved in the technology of the present invention, the description related to the technical common sense that does not need to be specifically explained in terms of the technology that can implement the present invention is omitted in the recorded contents and illustrated contents shown above.

[0196] All documents, patent applications, and technical standards described in this specification are incorporated herein by reference to the same extent as if each individual document, patent application, or technical standard was specifically and individually described as being incorporated by reference.

[0197] Explanation of symbols

[0198] 10-imaging device, 11-body, 11A-camera side mount, 11B-electrical contact, 11C-front surface, 11D-back surface, 12-imaging lens, 12A-lens side mount, 12B-electrical contact, 13-turn dial, 14-release button, 15-AF display, 16-indicator key, 18-viewfinder eyepiece, 20-imaging sensor, 21-imaging area, 21A-AF area, 22-vertical scanning circuit, 23 -Line memory, 24-Horizontal scanning circuit, 24A-Horizontal output line, 25-Output amplifier, 26-Pixel, 29-Transistor, 30-Objective lens, 31-Focus lens, 32-Rear lens, 33-Aperture, 34-Lens drive control unit, 40-Processor, 42-Operation unit, 45-Memory, 45A-Program, 47-Shake detection sensor, 50-Main control unit, 51-Camera control unit, 52-Image processing unit , 53-focus detection unit, 54-brightness detection unit, 55-correction amount calculation unit, 56-HDR synthesis unit, 57-jitter correction unit, 58-low-pass filter processing unit, C-virtual line, CF-color filter, D1-photodiode, E1-first exposure time, E2-second exposure time, EP-exit pupil, L1-row selection line, L2-row reset line, L3-column signal line, LA-optical axis, LF-light beam, M1-amplifier transistor, M2-pixel selection transistor, M3-reset transistor, ML-microlens, N-camera pixel, PG1-first pixel group, PG2-second pixel group, RST-reset signal, S-pixel signal, SEL-row selection signal, SF-light shielding layer, T-frame period, VD-vertical synchronization signal, ZL, ZR-phase difference pixel, Z1L, Z1R-first phase difference pixel, Z2L, Z2R-second phase difference pixel.

Claims

1. A camera device comprising: processor; and An imaging element, wherein a column signal line for reading out a signal extends in a first direction, a first pixel group and a second pixel group are arranged in the first direction, the first pixel group includes a plurality of phase difference pixels and a plurality of imaging pixels arranged in a second direction intersecting the first direction, and the second pixel group includes a plurality of imaging pixels arranged in the second direction, The processor is configured to execute exposure time setting processing, wherein the exposure time setting processing makes one of a first exposure time for which the first pixel group is exposed and a second exposure time for which the second pixel group is exposed shorter than the other. In the setting process, it is configured to determine which of the first exposure time and the second exposure time is to be made shorter than the other based on information of the subject imaged by the imaging element.

2. The imaging device according to claim 1, in, The processor is configured to execute the following processing: detecting the brightness of the area of ​​interest in the imaging area by the imaging element as the information, and determining the first exposure time and the second exposure time based on the detected brightness.

3. The imaging device according to claim 2, in, The processor is configured to perform the following processing: When the brightness is equal to or greater than a first threshold, the first exposure time is made shorter than the second exposure time. When the brightness is lower than the first threshold, the second exposure time is made shorter than the first exposure time.

4. The imaging device according to claim 1, in, The processor is configured to perform the following processing: As the information, a moving speed of the subject is detected, and the first exposure time and the second exposure time are determined based on the detected moving speed.

5. The imaging device according to claim 4, in, The processor is configured to perform the following processing: When the moving speed is equal to or greater than a second threshold, the first exposure time is made shorter than the second exposure time, When the moving speed is lower than the second threshold, the second exposure time is made shorter than the first exposure time.

6. The imaging device according to any one of claims 1 to 5, in, The processor is configured to perform the following processing: In a frame period, after reading out a signal from one of the first pixel group and the second pixel group, reading out a signal from the other of the first pixel group and the second pixel group.

7. The imaging device according to claim 6, in, The processor is configured to perform the following processing: detecting a position of a subject in an imaging area of ​​the imaging element with respect to a first direction, When the detected position is one direction side in the first direction, after reading out signals from one of the first pixel group and the second pixel group from the other direction side, reading out signals from the other of the first pixel group and the second pixel group from the one direction side, When the detected position is the other direction side, after reading signals from one of the first pixel group and the second pixel group from the one direction side, signals are read from the other of the first pixel group and the second pixel group from the other direction side.

8. The imaging device according to any one of claims 1 to 5, in, The processor is configured to execute a synthesis process for generating a video signal by synthesizing a signal read out from the first pixel group and a signal read out from the second pixel group.

9. The imaging device according to claim 8, in, The processor is configured to execute the synthesis process after performing a phase process for reducing a phase difference between a signal read out from the first pixel group and a signal read out from the second pixel group.

10. The imaging device according to claim 9, in, The processor is configured to perform the phase processing on the signal read out from the second pixel group.

11. The imaging device according to claim 8, in, The processor is configured to perform electronic vibration reduction processing on the video signal generated by the synthesis processing.

12. A camera device comprising: processor; and An imaging element, wherein a column signal line for reading out a signal extends in a first direction, a first pixel group and a second pixel group are arranged in the first direction, the first pixel group includes a plurality of phase difference pixels and a plurality of imaging pixels arranged in a second direction intersecting the first direction, and the second pixel group includes a plurality of imaging pixels arranged in the second direction, The processor is configured to selectively execute a first mode and a second mode. in, The first mode includes: A first setting process is to make one of a first exposure time for the first pixel group to be exposed and a second exposure time for the second pixel group to be exposed shorter than the other; and a first synthesis process of synthesizing a signal read out from the first pixel group and a signal read out from the second pixel group to generate a first video signal; The second mode includes: A second setting process is performed to make one of the first exposure time and the second exposure time during a first frame period and the first exposure time and the second exposure time during a second frame period after the first frame period shorter than the other; and a second synthesis process for generating a second video signal by synthesizing the signal read out from the first pixel group and the second pixel group during the first frame period and the signal read out from the first pixel group and the second pixel group during the second frame period, The setting upper limit value of the frame rate in the first mode is set to be higher than the setting upper limit value of the frame rate in the second mode.

13. A camera device comprising: processor; and An imaging element, wherein a column signal line for reading out a signal extends in a first direction, and a first pixel group and a second pixel group are arranged in the first direction, each of which includes a plurality of imaging pixels arranged in a second direction intersecting the first direction. The first pixel group includes a plurality of first phase difference pixels having a first opening area. The second pixel group includes a plurality of second phase difference pixels having a second opening area smaller than the first opening area. The processor is configured to execute exposure time setting processing, wherein the exposure time setting processing makes a first exposure time for which the first pixel group is exposed shorter than a second exposure time for which the second pixel group is exposed.

14. The imaging device according to claim 13, in, The processor is configured to execute a synthesis process for generating a video signal by synthesizing a signal read out from the first pixel group and a signal read out from the second pixel group.

15. A method for driving an imaging device, the imaging device comprising an imaging element, in, A column signal line for reading out a signal extends in a first direction, a first pixel group and a second pixel group are arranged in the first direction, the first pixel group includes a plurality of phase difference pixels and a plurality of imaging pixels arranged in a second direction intersecting the first direction, the second pixel group includes a plurality of imaging pixels arranged in the second direction, and the driving method of the imaging device includes: The exposure time setting process makes one of a first exposure time for exposing the first pixel group and a second exposure time for exposing the second pixel group shorter than the other, In the setting process, it is determined whether one of the first exposure time and the second exposure time is to be made shorter than the other based on information of the subject imaged by the imaging element.

16. A computer-readable storage medium storing a program, wherein the program causes an imaging device to operate, the imaging device comprising an imaging element. in, A column signal line for reading out a signal extends in a first direction, a first pixel group and a second pixel group are arranged in the first direction, the first pixel group includes a plurality of phase difference pixels and a plurality of imaging pixels arranged in a second direction intersecting the first direction, and the second pixel group includes a plurality of imaging pixels arranged in the second direction. The program causes the imaging device to execute exposure time setting processing, wherein the exposure time setting processing makes one of a first exposure time for which the first pixel group is exposed and a second exposure time for which the second pixel group is exposed shorter than the other. In the setting process, it is determined whether one of the first exposure time and the second exposure time is to be made shorter than the other based on information of the subject imaged by the imaging element.

Images