Image control device, imaging device, image control method, and image control program

Abstract

To provide an imaging control apparatus, an imaging apparatus, an imaging control method, and an imaging control program.SOLUTION: A system control unit 11 reads a pixel signal from a group G1 when an exposure period EX is finished, and when the reading is finished, performs reading a pixel signal from a group G2 and imaging a subject by the group G1 (exposure period EX (G1)). Then, the system control unit 11 reads a pixel signal from the group G1 after the finish of the imaging, and displays a live view image LV1 on the basis of the pixel signal.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an imaging control device, an imaging device, an imaging control method, and an imaging control program. [Background Art]

[0002] Patent Literature 1 describes an image processing apparatus including: a reading unit that reads, as region image data for each of a plurality of divided regions of a memory unit, image data captured by an imaging element provided with a plurality of photoelectric conversion elements and transferred to the memory unit on which optical noise is superimposed, and rereads data in a predetermined region after reading of each piece of the region image data is completed; and an output unit that outputs corrected image data obtained by correcting, for each of the plurality of regions, captured image data captured by the imaging element and stored in the memory unit in accordance with optical noise determined according to the data reread by the reading unit.

[0003] Patent Literature 2 describes an imaging apparatus that, when an imaging instruction is issued, exposes the light receiving surface of an imaging element, reads out imaging signals from the imaging element divided into five field periods after the end of this exposure, reads out imaging signals from a field including phase difference detection pixels in the first field period, reads out imaging signals from a field including only imaging pixels in the next field period, displays a post-view image based on the imaging signal read out in the first field period, and updates the post-view image based on the imaging signal read out in the next field period. [Prior Art Documents] [Patent Documents]

[0004] [Patent Document 1] International Publication No. 2020 / 021887 [Patent Document 2] International Publication No. 2018 / 088119 [Summary of the Invention] [Means for Solving the Problems]

[0005] The technology disclosed herein is as follows:

[0006] (1) An imaging control device for controlling an image sensor (image sensor 5) having a plurality of pixel rows (pixel rows 62) in which a plurality of pixels (pixels 61), each including a photoelectric conversion unit (photoelectric conversion unit 61A) and a charge holding unit (charge holding unit 61B) that holds the charge transferred from the photoelectric conversion unit, are arranged in one direction (row direction X), Equipped with a processor (system control unit 11), The above processor is The above-mentioned multiple pixel rows (group G1, group G2, group G3, and group G4) are exposed, and a first control is performed to transfer the charge accumulated in the photoelectric conversion section of the above-mentioned multiple pixel rows by the exposure to the charge holding section. A second control is performed to read out a signal corresponding to the charge held in the charge holding unit of some of the pixel rows (group G1) among the charge holding units that have held charge as a result of the first control described above. Based on the charge-holding unit that holds the charge as described in the first control, a third control is performed which reads out a signal corresponding to the charge held in the charge-holding unit of the pixel rows other than the aforementioned partial pixel rows (any of group G2, group G3, and group G4), and exposes the partial pixel rows from which the signal was read in the second control. An imaging control device that performs a fourth control to read out a signal corresponding to the charge held in the charge holding section of some of the pixel rows from which the signal was read out by the second control described above.

[0007] (2) (1) The imaging control device described above, The above processor is an imaging control device that performs display control of a live view image based on the results of one or both of the second control and the fourth control.

[0008] (3) The imaging control device described in (1) or (2), The above processor is an imaging control device that, in the above third control, performs exposure of some pixel rows by shifting the exposure period of all or some of the above pixel rows.

[0009] (4) An imaging control device according to any one of (1) to (3), The above image sensor includes a pixel row for phase difference detection (second pixel row), which is the pixel row containing the pixels for phase difference detection. Some of the above pixel rows include the above-mentioned pixel rows for phase difference detection, The above processor is an imaging control device that derives an evaluation value for focus adjustment based on the signal read out from the charge holding unit of the phase difference detection pixel row by the above second control.

[0010] (5) (4) The imaging control device described above, The above processor is an imaging control device that determines whether to execute or not execute the above third control based on the above evaluation value.

[0011] (6) (5) The imaging control device described above, The above processor is an imaging control device that, if the above evaluation value is greater than or equal to a threshold, does not execute the above third control.

[0012] (7) The imaging control device described in (5) or (6), The above processor, when the above third control is not executed, performs a fifth control in place of the third and fourth controls, which reads out a signal corresponding to the charge held in the charge holding unit of the pixel row other than the above some pixel row among the charge holding units that hold charge by the above first control.

[0013] (8) The imaging control device described in (5) or (6), The above processor is an imaging control device that, when the above third control is not executed, performs the above first control instead of the above third control and the above fourth control.

[0014] (9) The imaging control apparatus according to (4), wherein said processor controls a signal readout speed in said fourth control based on said evaluation value.

[0015] (10) The imaging control apparatus according to (9), wherein when said evaluation value is equal to or greater than a threshold value, said processor makes a signal readout speed in said fourth control faster than a signal readout speed in said third control.

[0016] (11) The imaging control apparatus according to any one of (1) to (8), wherein said processor makes a signal readout speed in said fourth control faster than a signal readout speed in said third control.

[0017] (12) The imaging control apparatus according to any one of (1) to (11), wherein after performing said second control, said processor performs, at least once, a process of performing said third control and said fourth control in this order.

[0018] (13) The imaging control apparatus according to (12), wherein when readout of signals from charge holding portions of said pixel rows other than said part of pixel rows is completed by said third control, said processor starts said first control while omitting said fourth control to be performed subsequent to said third control, or in combination with said fourth control.

[0019] (14) The imaging control apparatus according to (12), wherein The above processor performs the above process n times, where n is multiple, and after the nth time the above process is performed, it performs the first control and control to read a signal corresponding to the charge held in the charge holding part of the pixel row other than the above some pixel row for which signal readout is incomplete (control to execute a drive shown by the line ROg4), and is an imaging control device.

[0020] (15) An imaging control device as described in any of (1) to (14), An imaging device (digital camera 100) comprising the above-mentioned image sensor.

[0021] (16) An imaging control method for controlling an image sensor having multiple pixel rows in which multiple pixels, each containing a photoelectric conversion unit and a charge holding unit that holds the charge transferred from the photoelectric conversion unit, are arranged in one direction, The first control is performed to expose the above-mentioned multiple pixel rows and transfer the charge accumulated in the photoelectric conversion section of the above-mentioned multiple pixel rows by the exposure to the charge holding section. A second control is performed to read out a signal corresponding to the charge held in the charge holding unit of some of the pixel rows, among the charge holding units that have held charge as a result of the first control described above. In the first control described above, a third control is performed to read out a signal corresponding to the charge held in the charge holding unit of the pixel rows other than the aforementioned partial pixel rows, and to expose the aforementioned partial pixel rows from which the signal was read out in the second control described above. An imaging control method that performs a fourth control to read out a signal corresponding to the charge held in the charge holding part of the part of the pixel row from which the signal was read out by the second control described above.

[0022] (17) An imaging control program for controlling an image sensor having multiple pixel rows in which multiple pixels, each containing a photoelectric conversion unit and a charge holding unit that holds the charge transferred from the photoelectric conversion unit, are arranged in one direction, The first control is performed to expose the above-mentioned multiple pixel rows and transfer the charge accumulated in the photoelectric conversion section of the above-mentioned multiple pixel rows by the exposure to the charge holding section. A second control is performed to read out a signal corresponding to the charge held in the charge holding unit of some of the pixel rows, among the charge holding units that have held charge as a result of the first control described above. In the first control described above, a third control is performed to read out a signal corresponding to the charge held in the charge holding unit of the pixel rows other than the aforementioned partial pixel rows, and to expose the aforementioned partial pixel rows from which the signal was read out in the second control described above. An imaging control program that causes the processor to execute a step of performing a fourth control, which involves reading a signal corresponding to the charge held in the charge holding part of some of the pixel rows from which the signal was read by the second control described above. [Brief explanation of the drawing]

[0023] [Figure 1] This figure shows a schematic configuration of a digital camera 100, which is one embodiment of the imaging device of the present invention. [Figure 2] Figure 1 is a schematic plan view showing the general configuration of the image sensor 5. [Figure 3] Figure 2 is a schematic plan view showing the general configuration of pixels 61 in the image sensor 5. [Figure 4] Figure 3 is a schematic cross-sectional view of line AA of pixel 61. [Figure 5] Figure 1 is a timing chart showing the operation of the digital camera 100 in imaging mode. [Figure 6] Figure 1 is a timing chart showing a first modified example of the operation of the digital camera 100 in imaging mode. [Figure 7] Figure 1 shows a timing chart illustrating a second modified example of the operation of the digital camera 100 in imaging mode. [Figure 8] Figure 1 shows a timing chart illustrating a third modified example of the operation of the digital camera 100 in imaging mode. [Figure 9] Figure 1 shows a timing chart illustrating a fourth modified example of the operation of the digital camera 100 in imaging mode. [Figure 10]Figure 1 shows a timing chart illustrating a fifth modified example of the operation of the digital camera 100 in imaging mode. [Figure 11] Figure 1 shows a timing chart illustrating a sixth modified example of the operation of the digital camera 100 in imaging mode. [Figure 12] This shows the appearance of the Smartphone 200. [Figure 13] Figure 12 is a block diagram showing the configuration of the smartphone 200. [Modes for carrying out the invention]

[0024] Figure 1 is a diagram showing the schematic configuration of a digital camera 100, which is one embodiment of the imaging device of the present invention. The digital camera 100 shown in Figure 1 comprises a lens device 40 having an imaging lens 1, an aperture 2, a lens drive unit 8 that drives the imaging lens 1, an aperture drive unit 9 that drives the aperture 2, and a lens control unit 4 that controls the lens drive unit 8 and the aperture drive unit 9, and a main body 100A.

[0025] The main unit 100A comprises an image sensor 5, a system control unit 11 that provides overall control of the entire electrical control system of the digital camera 100, an operation unit 14, a display device 22, a memory 16 including RAM (Random Access Memory) and ROM (Read Only Memory), a memory control unit 15 that controls data storage in the memory 16 and data reading from the memory 16, a digital signal processing unit 17, and an external memory control unit 20 that controls data storage in the storage medium 21 and data reading from the storage medium 21.

[0026] The lens device 40 may be detachable from the main body 100A, or it may be integrated with the main body 100A. The imaging lens 1 includes a focus lens that is movable in the optical axis direction. This focus lens is a lens for adjusting the focus of the imaging optical system, which includes the imaging lens 1 and the aperture 2, and is composed of a single lens or multiple lenses. As the focus lens moves in the optical axis direction, the position of the principal point of the focus lens changes along the optical axis direction, thereby changing the focal position on the subject side. As the focus lens, a liquid lens whose principal point position in the optical axis direction can be changed by electrical control may be used.

[0027] The lens control unit 4 of the lens device 40 controls the lens drive unit 8 based on the lens drive signal transmitted from the system control unit 11 to change the position of the principal point of the focus lens included in the imaging lens 1. The lens control unit 4 of the lens device 40 controls the aperture drive unit 9 based on the drive control signal transmitted from the system control unit 11 to change the aperture amount (F number) of the aperture 2.

[0028] The image sensor 5 captures an image of the subject through an imaging optical system including an imaging lens 1 and an aperture 2. The image sensor 5 has a light-receiving surface 60 (see Figure 2) in which multiple pixels are arranged in two dimensions, and the imaging optical system converts the image of the subject formed on this light-receiving surface 60 into a group of pixel signals using these multiple pixels and outputs it. For example, a CMOS (complementary metal-oxide semiconductor) image sensor is used for the image sensor 5. The image sensor 5 is driven by a driver (not shown), which is controlled by a system control unit 11.

[0029] The system control unit 11 provides overall control for the digital camera 100. Its hardware structure consists of various processors that execute programs, including an image capture control program. The programs executed by the system control unit 11 are stored in the ROM of the memory 16.

[0030] Various types of processors include CPUs (Central Processing Units), which are general-purpose processors that execute programs and perform various processes; Programmable Logic Devices (PLDs), such as FPGAs (Field Programmable Gate Arrays), whose circuit configurations can be changed after manufacturing; and dedicated electrical circuits, such as ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform particular processes. More specifically, the structure of these various types of processors is an electrical circuit that combines circuit elements such as semiconductor elements.

[0031] The system control unit 11 may be composed of one of various processors, or it may be composed of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs or a combination of a CPU and an FPGA).

[0032] The system control unit 11 drives the image sensor 5 and lens device 40 according to the imaging control program, and outputs the subject image captured through the imaging optical system of the lens device 40 as a group of pixel signals. The system control unit 11 and the memory 16 constitute the imaging control device. The group of pixel signals output from the image sensor 5 is processed by the digital signal processing unit 17 to generate image data that is suitable for display on the display device 22 or suitable for storage on the storage medium 21.

[0033] The system control unit 11 receives instruction signals from the user through the operation unit 14. The operation unit 14 includes a touch panel integrated with the display surface 22b, as well as various buttons and other controls.

[0034] The display device 22 comprises a display surface 22b composed of an organic EL (electroluminescence) panel or a liquid crystal panel, and a display controller 22a that controls the display on the display surface 22b.

[0035] The memory control unit 15, the digital signal processing unit 17, the external memory control unit 20, and the display controller 22a are interconnected by a control bus 24 and a data bus 25, and are controlled by commands from the system control unit 11.

[0036] Figure 2 is a schematic plan view showing the general configuration of the image sensor 5 shown in Figure 1. Figure 3 is a schematic plan view showing the general configuration of the pixels 61 in the image sensor 5 shown in Figure 2. Figure 4 is a schematic cross-sectional view of the AA line of the pixels 61 shown in Figure 3.

[0037] The image sensor 5 comprises a light-receiving surface 60 in which multiple pixel rows 62, each consisting of multiple pixels 61 arranged in the row direction X, are arranged in the column direction Y which is orthogonal to the row direction X; a drive circuit 63 for driving the pixels 61 arranged on the light-receiving surface 60; and a signal processing circuit 64 for processing the pixel signals read out to the signal lines from each pixel 61 of the pixel rows 62 arranged on the light-receiving surface 60.

[0038] The multiple pixels 61 include a phase difference detection pixel that receives one of a pair of light beams that have passed through two different parts aligned in the row direction X of the pupil region of the imaging optical system and detects a signal corresponding to the amount of light received, a phase difference detection pixel that receives the other of the pair of light beams and detects a signal corresponding to the amount of light received, and a normal pixel that receives both of the pair of light beams and detects a signal corresponding to the amount of light received.

[0039] The pixel row 62 includes a first pixel row containing only normal pixels and a second pixel row containing both phase-difference detection pixels and normal pixels. For example, the second pixel row is discretely arranged at equal intervals in the column direction Y. Note that in the image sensor 5, phase-difference detection pixels are not mandatory, and all pixels 61 may consist of normal pixels.

[0040] In the following, in Figure 2, the upper end of the light-receiving surface 60 in the column direction Y will be referred to as the upper end, and the lower end of the light-receiving surface 60 in the column direction Y will be referred to as the lower end. This upper end constitutes one end of the light-receiving surface 60, and this lower end constitutes the other end of the light-receiving surface 60.

[0041] The pixels 61 arranged on the light-receiving surface 60 are classified into multiple groups. For example, let M be a non-negative integer, let n be the number of groups set on the light-receiving surface 60 (here, n=4), and let k be 1 to n. In the example in Figure 2, the pixel row 62 that is (n×M+k) from the top edge of the light-receiving surface 60 out of all the pixel rows 62 arranged on the light-receiving surface 60 is designated as group Gk. As shown in Figure 2, the pixels 61 arranged on the light-receiving surface 60 are classified into group G1, group G2, group G3, and group G4. This grouping is just one example and is not limited to this.

[0042] As shown in Figure 3, the pixel 61 includes a photoelectric conversion unit 61A, a charge holding unit 61B, a charge transfer unit 61C, a floating diffusion 61D, and a readout circuit 61E.

[0043] The photoelectric conversion unit 61A receives light that has passed through the imaging optical system of the lens device 40 and generates and stores an electric charge corresponding to the amount of light received. The photoelectric conversion unit 61A is composed of a photodiode or the like.

[0044] The charge transfer unit 61C transfers the charge accumulated in the photoelectric conversion unit 61A to the charge holding unit 61B. The charge transfer unit 61C is composed of an impurity region in the semiconductor substrate and an electrode formed above this impurity region.

[0045] The voltage applied to the electrodes constituting the charge transfer unit 61C is controlled by the drive circuit 63, thereby transferring charge from the photoelectric conversion unit 61A to the charge holding unit 61B.

[0046] The charge holding section 61B holds the charge transferred from the photoelectric conversion section 61A by the charge transfer section 61C. The charge holding section 61B is composed of impurity regions within the semiconductor substrate.

[0047] The floating diffusion 61D is used to convert electric charge into a signal, and the charge held in the charge holding unit 61B is transferred to it.

[0048] The readout circuit 61E is a circuit that reads out a signal corresponding to the potential of the floating diffusion 61D as a pixel signal to the signal line 65. The readout circuit 61E is driven by the drive circuit 63. The image signal described above is formed by the collection of pixel signals.

[0049] As shown in Figure 4, a P-well layer 71 is formed on the surface of the N-type substrate 70, and a photoelectric conversion section 61A is formed on the surface portion of the P-well layer 71.

[0050] The photoelectric conversion section 61A is composed of an N-type impurity layer 73 and a P-type impurity layer 74 formed thereon. The semiconductor substrate is composed of an N-type substrate 70 and a P-well layer 71.

[0051] On the surface of the P-well layer 71, a charge-holding portion 61B made of an N-type impurity layer is formed, slightly separated from the photoelectric conversion portion 61A.

[0052] A transfer electrode 76 is formed above the region 75 of the P-well layer 71 between the charge holding portion 61B and the photoelectric conversion portion 61A, via an oxide film (not shown).

[0053] The region 75 and the transfer electrode 76 constitute the charge transfer section 61C. In the example in Figure 3, the transfer electrode 76 is formed above the charge holding section 61B, but the transfer electrode 76 only needs to be formed at least above the region 75.

[0054] By controlling the potential of the transfer electrode 76 to form a channel in region 75, the charge accumulated in the photoelectric conversion unit 61A can be transferred to the charge holding unit 61B. The potential of the transfer electrode 76 is controlled by the drive circuit 63.

[0055] A floating diffusion layer 61D, consisting of an N-type impurity layer, is formed on the surface of the P-well layer 71, slightly separated from the charge-holding portion 61B.

[0056] A readout electrode 72 is formed above the P-well layer 71 between the charge-holding portion 61B and the floating diffusion 61D, via an oxide film (not shown).

[0057] By controlling the potential of the readout electrode 72 and forming a channel in the region between the charge holding section 61B and the floating diffusion 61D, the charge held in the charge holding section 61B can be transferred to the floating diffusion 61D. The potential of the readout electrode 72 is controlled by the drive circuit 63.

[0058] In the example shown in Figure 4, the readout circuit 61E consists of a reset transistor 77 for resetting the potential of the floating diffusion 61D, an output transistor 78 for converting the potential of the floating diffusion 61D into a pixel signal and outputting it, and a selection transistor 79 for selectively reading the pixel signal output from the output transistor 78 to the signal line 65. The configuration of the readout circuit is just one example and is not limited to this. The readout circuit 61E may also be shared by multiple pixels 61.

[0059] Pixel 61 is provided with a light-shielding film (not shown), and areas other than the photoelectric conversion unit 61A are shielded from light by this light-shielding film.

[0060] The structure of pixel 61 shown in Figures 3 and 4 is an example and is not limited thereto.

[0061] The drive circuit 63 shown in Figure 2 independently drives the transfer electrode 76, read electrode 72, and read circuit 61E of each pixel 61 for each pixel row 62, performing resets of each photoelectric conversion unit 61A included in the pixel row 62 (discharge of charge accumulated in the photoelectric conversion unit 61A) and reading out the pixel signal to the signal line 65 according to the charge accumulated in each photoelectric conversion unit 61A.

[0062] Furthermore, the drive circuit 63 simultaneously drives the charge transfer units 61C of all pixels 61 to simultaneously transfer charge from the photoelectric conversion unit 61A of each pixel 61 to the charge holding unit 61B. The drive circuit 63 is controlled by the system control unit 11.

[0063] The photoelectric conversion unit 61A is reset by resetting the floating diffusion 61D with the reset transistor 77 while the charge transfer unit 61C is in a state where charge can be transferred and a channel is formed in the semiconductor substrate below the read electrode 72.

[0064] Therefore, once the reading of the pixel signal corresponding to the charge held by the charge holding unit 61B has been completed, it is possible to reset the photoelectric conversion unit 61A that transfers charge to the charge holding unit 61B (in other words, to start exposure of the photoelectric conversion unit 61A).

[0065] The signal processing circuit 64 shown in Figure 2 performs correlated double sampling on the pixel signals read from each pixel 61 of the pixel row 62 to the signal line 65, converts the pixel signals after correlated double sampling into digital signals, and outputs them to the data bus 25 (see Figure 1). The signal processing circuit 64 is controlled by the system control unit 11. The digital signal processing unit 17 performs signal processing such as demosaicing and gamma correction on the group of pixel signals output from the image sensor 5 to the data bus 25 to generate image data.

[0066] The system control unit 11 can drive the image sensor 5 using global reset drive, global shutter drive, rolling reset drive, rolling shutter drive, and rolling readout drive, respectively.

[0067] Global reset drive is a drive that simultaneously resets the photoelectric conversion unit 61A of each pixel 61 to be exposed, and simultaneously starts the exposure of each pixel 61.

[0068] Global shutter drive is a drive that simultaneously transfers the charge accumulated in the photoelectric conversion unit 61A of each pixel 61 by exposure initiated at each pixel 61 by global reset drive to the charge holding unit 61B, thereby simultaneously ending exposure at each pixel 61.

[0069] The rolling reset drive is a drive that sequentially performs the process of resetting each photoelectric conversion unit 61A of a plurality of pixel rows 62 to be exposed, and starting exposure of each photoelectric conversion unit 61A, while changing the pixel row 62.

[0070] The rolling shutter drive is a drive that sequentially performs the process of transferring charge from the photoelectric conversion unit 61A of a pixel row 62, where exposure has been started by the rolling reset drive, to the charge holding unit 61B of the same pixel row 62, thereby ending the exposure of that pixel row 62, while changing the pixel row 62.

[0071] When rolling reset drive and rolling shutter drive are performed, the exposure periods of all of the multiple pixel rows 62 to be exposed are slightly shifted. However, it is also possible to shift the exposure periods of only some of the multiple pixel rows to be exposed by using rolling reset drive and rolling shutter drive.

[0072] For example, consider a configuration in which a signal processing circuit 64 is further positioned above the light-receiving surface 60. In this configuration, since there are two signal processing circuits 64, it becomes possible to read pixel signals simultaneously from two pixel rows 62. Therefore, if two adjacent pixel rows 62 from among the multiple pixel rows 62 to be exposed are paired, the process of starting the exposure of a pair is performed sequentially, changing the pair each time, and the process of ending the exposure of a pair that has started exposure is also performed sequentially, changing the pair each time. In this way, the exposure period is made the same for each pair, while the exposure period is shifted for each pair.

[0073] The rolling readout drive is a drive that sequentially reads out the pixel signal corresponding to the charge held in each charge holding unit 61B for each pixel row 62.

[0074] When the digital camera 100 is set to imaging mode, the system control unit 11 continuously performs imaging for live view image display (hereinafter referred to as LV imaging) by, for example, a set of rolling reset drive, rolling shutter drive, and rolling readout drive. Alternatively, the system control unit 11 may perform LV imaging by a set of global reset drive, global shutter drive, and rolling readout drive.

[0075] Then, when the system control unit 11 receives an instruction to perform imaging for storage (hereinafter referred to as "main imaging") for storing still image data in the storage medium 21 during the execution of this set (hereinafter referred to as "imaging instruction"), it performs control to execute global reset drive and global shutter drive to perform main imaging. The digital signal processing unit 17 shown in Figure 1 processes the pixel signal group output from the image sensor 5 by main imaging to generate image data, and stores this image data in the storage medium 21.

[0076] Figure 5 is a timing chart showing the operation of the digital camera 100 shown in Figure 1 during the imaging mode. In Figure 5, the horizontal axis represents time.

[0077] The upper and middle sections of Figure 5 show the driving timing of the photoelectric conversion unit 61A and the charge holding unit 61B for each pixel row 62 of the image sensor 5. In the upper and middle sections of Figure 5, the vertical axis indicates the position in the column direction Y of the pixel row 62.

[0078] The solid and dashed lines in the upper part of Figure 5 indicate the timing of the global reset drive and global shutter drive.

[0079] The solid line in the middle section of Figure 5 indicates the timing at which charge is retained in the charge holding unit 61B by the global shutter drive.

[0080] The dashed line in the middle of Figure 5 indicates the timing at which the pixel signal is read out from the charge holding unit 61B by the rolling readout drive.

[0081] The lower part of Figure 5 shows the drawing state of the display surface 22b. In the lower part of Figure 5, the vertical axis indicates the position in the column direction Y of the display pixel row on the display surface 22b. The solid lines shown in the lower part of Figure 5 indicate the timing of drawing on the display pixel row of the display surface 22b.

[0082] When the system control unit 11 is set to imaging mode, it controls the system to continuously perform LV imaging and display the live view image on the display surface 22b. If an imaging command is issued while LV imaging is in progress, the system control unit 11 terminates the LV imaging that is currently being performed at the time the imaging command is received.

[0083] Then, at time t1, the system control unit 11 performs a global reset drive indicated by the line GR, simultaneously resetting the photoelectric conversion unit 61A in all pixel rows 62 formed on the light-receiving surface 60. As a result, exposure starts at the same time in all pixel rows 62 formed on the light-receiving surface 60.

[0084] Subsequently, after a predetermined exposure time has elapsed, the system control unit 11 controls the system to execute a global shutter drive, indicated by the line GS, at time t2. This global shutter drive simultaneously transfers charge from the photoelectric conversion unit 61A to the charge holding unit 61B in all pixel rows 62 formed on the light-receiving surface 60, and the charge is held in the charge holding unit 61B as indicated by the line ST. As a result, exposure is completed at the same time in all pixel rows 62 formed on the light-receiving surface 60. In Figure 5, the period enclosed by the line GR and the line GS is shown as the exposure period EX of this image.

[0085] The system control unit 11 performs control to execute a global shutter drive indicated by the linear GS, and then performs control to execute a rolling readout drive indicated by the linear ROg1. Through this rolling readout drive, a pixel row 62 in one of groups G1, G2, G3, and G4 (here, group G1) is selected sequentially from the upper end to the lower end of the light-receiving surface 60, and the pixel signal is read out from the charge holding section 61B of the selected pixel row 62. The group of pixel signals read out from group G1 by the rolling readout drive indicated by the linear ROg1 is referred to as the pixel signal group IMG1.

[0086] The system control unit 11 causes the digital signal processing unit 17 to process the pixel signals sequentially read out from pixel row 62 of group G1 by a rolling readout drive indicated by the line ROg1. When a line image is generated by this processing, the system control unit 11 controls the display of that line image on the display surface 22b. Through this control, as shown by the line DR1, the live view image (so-called post-view image PV) of the subject captured during this imaging (exposure period EX) is displayed on the display surface 22b.

[0087] When the rolling readout drive indicated by the line ROg1 ends at time t3, the system control unit 11 controls group G1 to execute a global reset drive indicated by the line GRg1. This global reset drive simultaneously resets the photoelectric conversion unit 61A in all pixel rows 62 of group G1, and exposure starts at the same time in all pixel rows 62 of group G1. At time t3, the reading of pixel signals from all charge holding units 61B in group G1 has been completed by the rolling readout drive indicated by the line ROg1. Therefore, at time t3, exposure can start simultaneously in all pixels 61 of group G1.

[0088] Furthermore, at time t3, the system control unit 11 performs control to execute a rolling readout drive indicated by the line ROg2. Through this rolling readout drive, a pixel row 62 in any one of groups G2, G3, and G4 (here, group G2) is selected sequentially from the upper end to the lower end of the light-receiving surface 60, and the pixel signal is read out from the charge-holding section 61B of the selected pixel row 62. The group of pixel signals read out from group G2 by the rolling readout drive indicated by the line ROg2 is referred to as the pixel signal group IMG2.

[0089] When the rolling readout drive indicated by the line ROg2 ends at time t4, the system control unit 11 controls group G1 to execute the global shutter drive indicated by the line GSg1. This global shutter drive simultaneously transfers charge from the photoelectric conversion unit 61A to the charge holding unit 61B in all pixel rows 62 of group G1, and the charge is held in the charge holding unit 61B as indicated by the line STg1. As a result, exposure ends at the same time in all pixel rows 62 of group G1. In Figure 5, the period enclosed by the lines GRg1 and GSg1 is shown as the exposure period EX(G1).

[0090] Furthermore, at time t4, the system control unit 11 performs control to execute a global shutter drive indicated by the line GSg1, and then performs control to execute a rolling readout drive indicated by the line ROG1. In this rolling readout drive, pixel rows 62 of group G1 are selected sequentially from the upper end to the lower end of the light-receiving surface 60, and pixel signals are read out from the charge-holding section 61B of the selected pixel rows 62.

[0091] The system control unit 11 causes the digital signal processing unit 17 to process the pixel signals sequentially read out from pixel row 62 of group G1 by a rolling readout drive indicated by the line ROG1 which started at time t4. When a line image is generated by this processing, the system control unit 11 controls the display of the line image on the display surface 22b. As a result of this control, the live view image LV1 of the subject captured by group G1 during the period F2 between time t3 and time t4 is displayed on the display surface 22b, as shown by the line DR2. In Figure 5, the period between time t2 and time t3 is shown as period F1.

[0092] When the rolling readout drive indicated by the line ROG1, which started at time t4, ends at time t5, the system control unit 11 controls group G1 to execute a global reset drive indicated by the line GRg1. This global reset drive simultaneously resets the photoelectric conversion unit 61A in all pixel rows 62 of group G1, and exposure starts at the same time in all pixel rows 62 of group G1.

[0093] Furthermore, at time t5, the system control unit 11 performs control to execute a rolling readout drive indicated by the linear ROg3. In this rolling readout drive, a pixel row 62 in either group G3 or group G4 (here, group G3) is selected sequentially from the upper end to the lower end of the light-receiving surface 60, and the pixel signal is read out from the charge-holding unit 61B of the selected pixel row 62. The group of pixel signals read out from group G3 by the rolling readout drive indicated by the linear ROg3 is referred to as the pixel signal group IMG3.

[0094] When the rolling readout drive indicated by the line ROg3 ends at time t6, the system control unit 11 controls group G1 to execute the global shutter drive indicated by the line GSg1. This global shutter drive simultaneously transfers charge from the photoelectric conversion unit 61A to the charge holding unit 61B in all pixel rows 62 of group G1, and the charge is held in the charge holding unit 61B as indicated by the line STg1. As a result, exposure ends at the same time in all pixel rows 62 of group G1.

[0095] Furthermore, at time t6, the system control unit 11 performs control to execute a global shutter drive indicated by the line GSg1, and then performs control to execute a rolling readout drive indicated by the line ROG1. In this rolling readout drive, pixel rows 62 of group G1 are selected sequentially from the upper end to the lower end of the light-receiving surface 60, and pixel signals are read out from the charge-holding section 61B of the selected pixel rows 62.

[0096] The system control unit 11 causes the digital signal processing unit 17 to process the pixel signals sequentially read out from pixel row 62 of group G1 by a rolling readout drive indicated by the line ROG1 which started at time t6. When a line image is generated by this processing, the system control unit 11 controls the display of the line image on the display surface 22b. As a result of this control, the live view image LV2 of the subject captured during the period F4 between time t5 and time t6 is displayed on the display surface 22b, as shown by the line DR3. In Figure 5, the period between time t4 and time t5 is shown as period F3.

[0097] When the rolling readout drive indicated by the line ROG1, which started at time t6, ends at time t7, the system control unit 11 controls group G1 to execute a global reset drive indicated by the line GRg1. This global reset drive simultaneously resets the photoelectric conversion unit 61A in all pixel rows 62 of group G1, and exposure starts at the same time in all pixel rows 62 of group G1.

[0098] Furthermore, at time t7, the system control unit 11 performs control to execute a rolling readout drive indicated by the linear ROg4. In this rolling readout drive, pixel rows 62 of group G4 are selected sequentially from the upper end to the lower end of the light-receiving surface 60, and pixel signals are read out from the charge-holding section 61B of the selected pixel rows 62. The group of pixel signals read out from group G4 by the rolling readout drive indicated by the linear ROg4 is referred to as the pixel signal group IMG4.

[0099] When the rolling readout drive indicated by the line ROg4 ends at time t8, the system control unit 11 controls group G1 to execute the global shutter drive indicated by the line GSg1. This global shutter drive simultaneously transfers charge from the photoelectric conversion unit 61A to the charge holding unit 61B in all pixel rows 62 of group G1, and the charge is held in the charge holding unit 61B as indicated by the line STg1. As a result, exposure ends at the same time in all pixel rows 62 of group G1.

[0100] Furthermore, at time t8, the system control unit 11 performs control to execute a global shutter drive indicated by the line GSg1, and then performs control to execute a rolling readout drive indicated by the line ROG1. In this rolling readout drive, pixel rows 62 of group G1 are selected sequentially from the upper end to the lower end of the light-receiving surface 60, and pixel signals are read out from the charge-holding section 61B of the selected pixel rows 62.

[0101] The system control unit 11 causes the digital signal processing unit 17 to process the pixel signals sequentially read out from pixel row 62 of group G1 by the rolling readout drive indicated by the line ROG1 which started at time t8. When a line image is generated by this processing, the system control unit 11 controls the display surface 22b to display the line image. As a result of this control, the live view image LV3 of the subject captured during the period F6 between time t7 and time t8 is displayed on the display surface 22b, as shown by the line DR4. In Figure 5, the period between time t6 and time t7 is shown as period F5. Also, the period between time t9, when the rolling readout drive indicated by the line ROG1 which started at time t8 ends, and time t8 is shown as period F7.

[0102] At time t8, the digital signal processing unit 17 processes the pixel signal groups IMG1, IMG2, IMG3, and IMG4 read between time t2 and time t8 to generate image data and store it in the storage medium 21.

[0103] In Figure 5, the control that executes the drive indicated by the straight line GR and the straight line GS constitutes the first control. The control that executes the drive indicated by the straight line ROg1 during period F1 constitutes the second control. The control that executes the drive indicated by the straight lines ROg2, GRg1, and GSg1 during period F2, the control that executes the drive indicated by the straight lines ROg3, GRg1, and GSg1 during period F4, and the control that executes the drive indicated by the straight lines ROg4, GRg1, and GSg1 during period F6 each constitute the third control. The control that executes the drive indicated by the straight line ROG1 during period F3, the control that executes the drive indicated by the straight line ROG1 during period F5, and the control that executes the drive indicated by the straight line ROG1 during period F7 each constitute the fourth control.

[0104] As described above, with the digital camera 100, even if the total number of pixels 61 formed on the light-receiving surface 60 is large and it takes time to complete the readout of pixel signals from all pixels 61, the post-view image PV can be displayed before the readout of all pixel signals is completed, specifically a little after the time t2 shown in Figure 5. Therefore, the state of the subject captured during the exposure period EX can be checked instantly, improving convenience.

[0105] Furthermore, according to the digital camera 100, even after the post-view image PV is displayed, it is possible to use group G1, which has already completed reading out the pixel signals, to perform imaging to generate a live view image (exposure for periods F2, F4, and F6 in Figure 5). This imaging allows the live view image to be displayed and even updated before the reading out of all the pixel signals obtained in the main imaging is completed. As a result, the user can continue to check the state of the subject they want to photograph using the post-view image PV and the live view image that is displayed afterward, enabling them to capture the appropriate shutter opportunity and take good pictures.

[0106] In each of the periods F1, F3, and F5 shown in Figure 5, the system control unit 11 preferably reads out pixel signals from a group that includes a second pixel row containing a pixel for phase difference detection. In other words, in the above description, it is preferable that the second pixel row is included in group G1.

[0107] In each of the periods F1, F3, and F5, the system control unit 11 reads pixel signals from the second pixel row, and using the pixel signals read from the phase difference detection pixels, it can derive evaluation values ​​for focus adjustment (values ​​for evaluating the amount of deviation between the main subject and the focal position, for example, the amount of defocus). This allows for the deriving of evaluation values ​​in each of the following periods, for example, from time t3 to time t5, from time t5 to time t7, and from time t7 to time t9, and enables control of the focus lens and prediction of the subject position based on these evaluation values. As a result, focus adjustment can be performed quickly and with high accuracy before the start of the next main image capture. In particular, in continuous shooting mode, where the main image capture is performed multiple times in succession according to the imaging instruction, the quality of the captured image can be improved by deriving the above evaluation values ​​at a high frequency.

[0108] Alternatively, instead of periods F1, F3, and F5, pixel signals from a group including the second pixel row may be read out during at least one of periods F2 and F4. This also allows for fast and highly accurate focus adjustment during the next image capture.

[0109] In the operation shown in Figure 5, the display of the post-view image PV may be omitted. Even in this case, the subject's condition can be checked immediately after the main image is captured using the live view image LV1 and live view image LV2.

[0110] Furthermore, in the operation shown in Figure 5, the display of the live view image LV1 may be omitted. Doing so allows the post-view image PV to be displayed for a longer period, making it possible to examine the subject captured in detail.

[0111] Furthermore, in the operation shown in Figure 5, the display of live view images LV1, LV2, and LV3 may be omitted. Even in this case, by using the captured image data acquired by group G1 during periods F2, F4, and F6, for example, subject detection processing and evaluation value derivation processing, it becomes possible to improve the accuracy of predicting the movement of a specific subject or to improve the accuracy of control that keeps the focus on a specific subject.

[0112] Furthermore, as shown in Figure 5, even when displaying post-view image PV, live view image LV1, live view image LV2, and live view image LV3, it is possible to improve the accuracy of predicting the movement of a specific subject or the accuracy of controlling the camera to maintain focus on a specific subject by using the captured image data acquired by group G1 during periods F2, F4, and F6, for example, to perform subject detection processing and evaluation value derivation processing.

[0113] In the above explanation, it is assumed that the pixels 61 formed on the light-receiving surface 60 are classified into four groups, but it is sufficient for the pixels 61 to be classified into two or more groups. For example, if group G4 does not exist on the light-receiving surface 60, the process shown in Figure 5 should be modified by removing the process from time t7 onwards.

[0114] Alternatively, for example, groups G1 and G3 may be merged to form the first group, and groups G2 and G4 may be merged to form the second group. In this case, the system control unit 11 may perform control during period F1 to read pixel signals from the first group and display a post-view image, during period F2 to perform exposure for the first group and read pixel signals from the second group, and during period F3 to read pixel signals from the first group to display a live view image, perform subject detection processing, or perform evaluation value derivation processing.

[0115] The following describes variations in the operation of the digital camera 100 in its imaging mode.

[0116] Figure 6 is a timing chart showing a first modified example of the operation of the digital camera 100 shown in Figure 1 during the imaging mode. The timing chart shown in Figure 6 is the same as the timing chart shown in Figure 5, except that the global reset drive, which was shown by the line GRg1 in Figure 5, has been changed to a rolling reset drive shown by the line RRg1, the global shutter drive, which was shown by the line GSg1 in Figure 5, has been changed to a rolling shutter drive shown by the line RSg1, the line STg1 shown in Figure 5 has been removed, and the rolling readout drive, which was shown by the line ROG1 in Figure 5, has been changed to a rolling readout drive shown by the line rog1.

[0117] At time t3, the system control unit 11 performs control to execute a rolling reset drive indicated by the line RRg1. In this rolling reset drive, the pixel rows 62 of group G1 are sequentially selected from the top side, and the photoelectric conversion unit 61A in the selected pixel rows 62 is reset.

[0118] When the rolling reset drive is completed at time t4, the system control unit 11 controls the system to execute the rolling shutter drive indicated by the line RSg1. In this rolling shutter drive, the pixel rows 62 of group G1 are sequentially selected from the top, and the charge accumulated in the photoelectric conversion unit 61A in the selected pixel rows 62 is transferred to the charge holding unit 61B. When the charge is transferred to the charge holding unit 61B by this rolling shutter drive, the pixel signal corresponding to that charge is read out by the rolling readout drive indicated by the line rog1, and the live view image LV1 is displayed based on this pixel signal.

[0119] At time t5, the system control unit 11 controls the group G1 to perform a rolling reset drive, and at time t6, once this rolling reset drive is completed, it controls the group G1 to perform a rolling shutter drive. When charge is transferred to the charge holding unit 61B by this rolling shutter drive, a pixel signal corresponding to that charge is read out by the rolling readout drive indicated by the line rog1, and the live view image LV2 is displayed based on this pixel signal.

[0120] At time t7, the system control unit 11 controls the group G1 to perform a rolling reset drive, and at time t8, once this rolling reset drive is completed, it controls the group G1 to perform a rolling shutter drive. When charge is transferred to the charge holding unit 61B by this rolling shutter drive, a pixel signal corresponding to that charge is read out by the rolling readout drive indicated by the line rog1, and the live view image LV3 is displayed based on this pixel signal.

[0121] As described above, the process shown in Figure 6 makes it possible to make the time required from the end of exposure for each pixel row 62 of group G1, indicated by the straight line RSg1, to the start of display of the line image corresponding to each pixel row 62, uniform for all pixel rows 62 of group G1. Therefore, the quality of live view image LV1, live view image LV2, and live view image LV3 can be improved.

[0122] In the process shown in Figure 6, the controls that execute the drives indicated by the lines ROg2, RRg1, and RSg1 in periods F2 and F3, the controls that execute the drives indicated by the lines ROg3, RRg1, and RSg1 in periods F4 and F5, and the controls that execute the drives indicated by the lines ROg4, RRg1, and RSg1 in periods F6 and F7 each constitute the third control. In addition, the control that executes the drive indicated by line rog1 constitutes the fourth control.

[0123] Figure 7 is a timing chart showing a second modified example of the operation of the digital camera 100 in imaging mode as shown in Figure 1. The timing chart shown in Figure 7 differs from that in Figure 5 in the processing from time t3 onwards. Furthermore, the second modified example assumes that the second pixel row is included in the group (group G1) that first reads out the pixel signals after this image capture.

[0124] The system control unit 11 acquires the pixel signals of the phase difference detection pixels from the pixel signals read out from group G1 by the rolling readout drive that started in period F1, and derives an evaluation value for focus adjustment based on these pixel signals.

[0125] Then, the system control unit 11 decides whether to execute or not execute the controls from time t3 onwards in Figure 5 (i.e., the third and fourth controls) based on the derived evaluation value. Specifically, if the evaluation value is less than the threshold, the system control unit 11 decides to execute the controls from time t3 onwards in Figure 5, and executes these controls. The operation in this case is as shown in Figure 5.

[0126] On the other hand, if the evaluation value is above a threshold, the system control unit 11 decides not to execute the control from time t3 onwards in Figure 5, and instead of the third and fourth control as shown in Figure 7, it performs a rolling readout drive indicated by the straight line RO. In this rolling readout drive, pixel rows 62 included in groups G2, G3, and G4 are selected sequentially from the upper end to the lower end of the light-receiving surface 60, and the pixel signal is read out from the charge-holding section 61B of the selected pixel row 62.

[0127] The fifth control involves executing a rolling readout drive indicated by a linear RO. The pixel signal group read out from groups G2, G3, and G4 by the rolling readout drive indicated by a linear RO is referred to as pixel signal group IMG5.

[0128] When the rolling readout drive, indicated by the linear RO, is completed, the digital signal processing unit 17 processes the pixel signal group IMG1 and pixel signal group IMG5 read from the image sensor 5 to generate image data and store it in the storage medium 21. When the rolling readout drive, indicated by the linear RO, is completed, the system control unit 11 starts the next main image capture.

[0129] If the evaluation value derived based on the pixel signal read out from group G1 during period F1 is large, it is estimated that the degree of blur in the image data obtained from this imaging is large. Therefore, in such cases, instead of reading out the pixel signals from groups G2, G3, and G4 in separate periods and performing imaging with group G1 in between, the time until the next imaging can be performed can be shortened by reading out the pixel signals from groups G2, G3, and G4 all at once. As a result, it is possible to capture an appropriate shutter opportunity and obtain good images.

[0130] In the example shown in Figure 7, rolling readout drive, indicated by the linear RO, is performed when the evaluation value derived based on the pixel signal read from group G1 is greater than or equal to a threshold.

[0131] As a variation of this, the system control unit 11 may, if the evaluation value is greater than or equal to a threshold, stop reading out pixel signals from groups G2, G3, and G4 and perform control to perform the next main imaging (i.e., first control).

[0132] Figure 8 is a timing chart showing a third modified example of the operation of the digital camera 100 in imaging mode as shown in Figure 1. The timing chart in Figure 8 differs from that in Figure 7 in that at time t3, the system control unit 11 stops reading out pixel signals from groups G2, G3, and G4, and then performs control to start the next main image capture.

[0133] In Figure 8, after the exposure period EX, which begins after time t3, if the evaluation value derived based on the pixel signals read from group G1 exposed during this exposure period EX is less than the threshold, the processing shown in Figure 5 from time t3 onwards is performed. If this evaluation value is greater than or equal to the threshold, the next main image is taken. By doing so as shown in Figure 8, the time until the next main image can be taken can be shortened, allowing for the capture of an appropriate shutter opportunity and the production of a good image.

[0134] Figure 9 is a timing chart showing a fourth modified example of the operation of the digital camera 100 in imaging mode as shown in Figure 1. The timing chart in Figure 9 differs from that in Figure 5 in that the lengths of periods F3, F5, and F7 are shorter than the lengths of periods F1, F2, F4, and F6.

[0135] In other words, in the timing chart shown in Figure 9, the readout speed of the pixel signal by the rolling readout drive indicated by linear ROG1 is faster than the readout speeds of the pixel signal by the rolling readout drive indicated by linear ROg1, linear ROg2, linear ROg3, and linear ROg4, respectively.

[0136] The readout speed of the pixel signals from the image sensor 5 can be changed by changing the number of conversion bits set in the AD (Analog to Digital) converter included in the signal processing circuit 64, or by changing the clock frequency of the AD converter. The readout speed can also be changed by changing the number of pixels 61 from which the pixel signals are read out of group G1.

[0137] The process shown in Figure 9 reduces the time required to read out all pixel signals obtained in the main image capture, allowing for a faster start to the next main image capture. In continuous shooting mode, the continuous shooting speed can be increased.

[0138] Furthermore, the control of speeding up the pixel signal readout rate during periods F3, F5, and F7, as shown in Figure 9, may be performed continuously, or it may be performed only when the evaluation value described above is above a threshold.

[0139] In other words, the system control unit 11 may perform the above-mentioned speed-up control if the evaluation value derived based on the pixel signals read out from group G1 including the phase difference detection pixels during period F1 is greater than or equal to a threshold value, and if this evaluation value is less than the threshold value, it may not perform the above-mentioned speed-up control and may set the readout speed of the pixel signals in periods F3, F5, and F7 to be the same as the readout speed of the pixel signals in periods F1, F2, F4, and F6.

[0140] This approach shortens the time until the next image capture is possible if the subject image captured in the main image capture is significantly blurred. On the other hand, if the subject image captured in the main image capture is not significantly blurred, the post-view image and the live view image can be displayed under the same conditions, or the live view image itself can be enhanced in quality, allowing for clear observation of the subject during imaging.

[0141] Figure 10 is a timing chart showing a fifth modified example of the operation of the digital camera 100 shown in Figure 1 during the imaging mode. The timing chart shown in Figure 10 is the same as the timing chart shown in Figure 5, except that the next main image capture starts during period F7.

[0142] In the timing chart shown in Figure 10, when the rolling readout drive indicated by the line ROg4 is completed at time t8, the system control unit 11 performs control to execute the rolling readout drive indicated by the line ROG1 in parallel with control to execute the next main imaging (i.e., the first control).

[0143] In Figure 10, the operation after the end of the exposure period EX, which starts after time t8, is the same as the operation from time t2 to time t9. According to the process shown in Figure 10, the time until the next main image can be shortened, and in continuous shooting mode, the continuous shooting speed can be increased. Note that in Figure 10, the drives shown by the linear GRg1 and linear GSg1 in period F6 and the drive shown by the linear ROG1 in period F7 are not mandatory and can be omitted.

[0144] Figure 11 is a timing chart showing a sixth modified example of the operation of the digital camera 100 shown in Figure 1 during the imaging mode. The timing chart shown in Figure 11 is the same as the timing chart shown in Figure 5, except that the exposure of group G1 in period F6 and the readout of the pixel signal obtained from that exposure in period F7 are omitted, and the next main imaging starts at time t7.

[0145] In the timing chart shown in Figure 11, when the rolling readout drive indicated by the line ROG1 ends at time t7, the system control unit 11 performs control to execute the rolling readout drive indicated by the line ROg4 (i.e., the fourth control) in parallel with control to execute the next main image (i.e., the first control). The operation after the end of the exposure period EX, which starts at time t7, is the same as the operation from time t2 to time t8. According to the process shown in Figure 11, the time until the next main image can be shortened, and in continuous shooting mode, the continuous shooting speed can be increased.

[0146] Next, we will describe the configuration of a smartphone, which is another embodiment of the imaging device of the present invention.

[0147] Figure 12 shows the external appearance of the smartphone 200. The smartphone 200 shown in Figure 12 has a flat casing 201, and one side of the casing 201 is equipped with a display input unit 204 which is an integrated display panel 202 as a display unit and an operation panel 203 as an input unit.

[0148] Furthermore, such a housing 201 includes a speaker 205, a microphone 206, an operating unit 207, and a camera unit 208. However, the configuration of the housing 201 is not limited to this; for example, a configuration in which the display unit and input unit are independent, or a configuration having a folding structure or a sliding mechanism, can also be adopted.

[0149] Figure 13 is a block diagram showing the configuration of the smartphone 200 shown in Figure 12.

[0150] As shown in Figure 13, the main components of a smartphone include a wireless communication unit 210, a display input unit 204, a call unit 211, an operation unit 207, a camera unit 208, a storage unit 212, an external input / output unit 213, a GNSS (Global Navigation Satellite System) receiver unit 214, a motion sensor unit 215, a power supply unit 216, and a main control unit 220.

[0151] Furthermore, the main function of the smartphone 200 is to provide a wireless communication function that performs mobile wireless communication via a base station device BS (not shown) and a mobile communication network NW (not shown).

[0152] The wireless communication unit 210 performs wireless communication with base station equipment BS connected to the mobile communication network NW, in accordance with instructions from the main control unit 220. Using this wireless communication, it sends and receives various file data such as voice data and image data, email data, etc., and receives web data or streaming data, etc.

[0153] The display input unit 204 is a so-called touch panel that, under the control of the main control unit 220, displays images (still images and moving images) or text information to visually convey information to the user and detects user operations on the displayed information, and comprises a display panel 202 and an operation panel 203.

[0154] The display panel 202 uses LCD (Liquid Crystal Display), OELD (Organic Electro-Luminescence Display), etc., as display devices.

[0155] The operation panel 203 is a device that is visibly mounted on the display surface of the display panel 202 and detects one or more coordinates operated by the user's finger or stylus. When this device is operated by the user's finger or stylus, it outputs a detection signal generated by the operation to the main control unit 220. The main control unit 220 then detects the operation position (coordinates) on the display panel 202 based on the received detection signal.

[0156] As shown in Figure 13, the display panel 202 and operation panel 203 of the smartphone 200, which is illustrated as one embodiment of the imaging device of the present invention, together constitute a display input unit 204, but the operation panel 203 is positioned to completely cover the display panel 202.

[0157] When such an arrangement is adopted, the operation panel 203 may also be equipped with a function to detect user operations in areas outside the display panel 202. In other words, the operation panel 203 may be equipped with a detection area for the overlapping portion that overlaps with the display panel 202 (hereinafter referred to as the display area) and a detection area for the outer edge portion that does not overlap with the display panel 202 (hereinafter referred to as the non-display area).

[0158] The size of the display area and the size of the display panel 202 may be made to match perfectly, but it is not necessary for them to match. Furthermore, the operation panel 203 may have two sensitive areas: an outer edge portion and an inner portion. The width of the outer edge portion is designed appropriately according to the size of the housing 201, etc.

[0159] Furthermore, the position detection methods used in the control panel 203 include matrix switch methods, resistive film methods, surface acoustic wave methods, infrared methods, electromagnetic induction methods, and capacitive methods, and any of these methods can be adopted.

[0160] The communication unit 211 is equipped with a speaker 205 or a microphone 206, and converts the user's voice input through the microphone 206 into audio data that can be processed by the main control unit 220 and outputs it to the main control unit 220, or decodes audio data received by the wireless communication unit 210 or the external input / output unit 213 and outputs it from the speaker 205.

[0161] Furthermore, as shown in Figure 12, for example, the speaker 205 can be mounted on the same side as the display input unit 204, and the microphone 206 can be mounted on the side of the housing 201.

[0162] The operation unit 207 is a hardware key using a key switch or the like, which receives instructions from the user. For example, as shown in Figure 12, the operation unit 207 is mounted on the side of the casing 201 of the smartphone 200 and is a push-button type switch that turns on when pressed with a finger or the like, and turns off when the finger is released due to a restoring force such as a spring.

[0163] The memory unit 212 stores the control program and control data of the main control unit 220, application software, address data associated with the name or telephone number of the communication partner, sent and received email data, web data downloaded by web browsing, downloaded content data, and also temporarily stores streaming data. The memory unit 212 is composed of an internal memory unit 217 built into the smartphone and an external memory unit 218 with a removable external memory slot.

[0164] The internal storage units 217 and external storage units 218 that constitute the storage unit 212 are implemented using storage media such as flash memory type, hard disk type, multimedia card micro type, card type memory (for example, MicroSD® memory), RAM (Random Access Memory), and ROM (Read Only Memory).

[0165] The external input / output unit 213 serves as an interface for all external devices connected to the smartphone 200, and is intended for direct or indirect connection to other external devices via communication (e.g., Universal Serial Bus (USB), IEEE 1394, Bluetooth (registered trademark), RFID (Radio Frequency Identification), Infrared Data Association (IrDA) (registered trademark), UWB (Ultra Wideband) (registered trademark), ZigBee (registered trademark), etc.) or network (e.g., Ethernet (registered trademark), Wireless LAN (Local Area Network), etc.).

[0166] External devices that can be connected to the Smartphone 200 include, for example, wired / wireless headsets, wired / wireless external chargers, wired / wireless data ports, memory cards connected via card sockets, SIM (Subscriber Identity Module Card) / UIM (User Identity Module Card) cards, external audio / video equipment connected via audio / video I / O (Input / Output) terminals, wirelessly connected external audio / video equipment, wired / wireless connected smartphones, wired / wireless connected personal computers, wired / wireless connected personal computers, earphones, etc.

[0167] The external input / output unit 213 can transmit data received from such external devices to the various internal components of the smartphone 200, or enable data from inside the smartphone 200 to be transmitted to external devices.

[0168] The GNSS receiver 214 receives GNSS signals transmitted from GNSS satellites ST1 to STn in accordance with instructions from the main control unit 220, performs positioning calculation processing based on the received GNSS signals, and detects the position of the smartphone 200, consisting of its latitude, longitude, and altitude. When the GNSS receiver 214 can obtain position information from the wireless communication unit 210 or the external input / output unit 213 (for example, wireless LAN), it can also use that position information to detect the position.

[0169] The motion sensor unit 215 includes, for example, a 3-axis acceleration sensor, and detects the physical movement of the smartphone 200 according to the instructions of the main control unit 220. By detecting the physical movement of the smartphone 200, the direction of movement or acceleration of the smartphone 200 is detected. The detection results are output to the main control unit 220.

[0170] The power supply unit 216 supplies power stored in a battery (not shown) to each part of the smartphone 200 according to the instructions of the main control unit 220.

[0171] The main control unit 220 is equipped with a microprocessor and operates according to the control program and control data stored in the memory unit 212, and comprehensively controls each part of the smartphone 200. The microprocessor of the main control unit 220 has the same functions as the system control unit 11. In addition, the main control unit 220 is equipped with a mobile communication control function that controls each part of the communication system for voice communication or data communication via the wireless communication unit 210, and an application processing function.

[0172] The application processing function is realized by the operation of the main control unit 220 according to the application software stored in the memory unit 212. Examples of application processing functions include an infrared communication function that controls the external input / output unit 213 to communicate data with a counterpart device, an email function that sends and receives emails, and a web browsing function that displays web pages.

[0173] Furthermore, the main control unit 220 is equipped with image processing functions, such as displaying video on the display input unit 204 based on image data (still image or moving image data) such as received data or downloaded streaming data.

[0174] The image processing function refers to the function in which the main control unit 220 decodes the above image data, applies image processing to the decoded result, and displays the image on the display input unit 204.

[0175] Furthermore, the main control unit 220 performs display control for the display panel 202 and operation detection control to detect user operations through the operation unit 207 and the operation panel 203.

[0176] By executing display control, the main control unit 220 displays software keys such as icons or scroll bars for launching application software, or displays a window for composing an email.

[0177] A scroll bar is a software key that accepts commands to move the display portion of an image, such as a large image that does not fit within the display area of ​​the display panel 202.

[0178] Furthermore, by executing operation detection control, the main control unit 220 detects user operations through the operation unit 207, accepts operations on the icons and input of strings into the input fields of the windows through the operation panel 203, or accepts requests to scroll the displayed image through the scroll bar.

[0179] Furthermore, by executing operation detection control, the main control unit 220 determines whether the operation position on the operation panel 203 is in the overlapping portion (display area) that overlaps with the display panel 202 or in the outer edge portion (non-display area) that does not overlap with the display panel 202, and has a touch panel control function that controls the display position of the sensitive area of ​​the operation panel 203 or the software key.

[0180] Furthermore, the main control unit 220 can detect gesture operations on the operation panel 203 and execute pre-set functions in response to the detected gesture operations.

[0181] Gesture control refers to operations that differ from traditional simple touch operations, such as drawing a path with a finger or other object, specifying multiple locations simultaneously, or combining these to draw a path from at least one of multiple locations.

[0182] The camera unit 208 includes the lens device 40, image sensor 5, and digital signal processing unit 17 shown in Figure 1. In the smartphone 200, the image control device is composed of the main control unit 220 and the internal storage unit 217.

[0183] The image data generated by the camera unit 208 can be stored in the storage unit 212 or output via the external input / output unit 213 or the wireless communication unit 210.

[0184] In the smartphone 200 shown in Figure 13, the camera unit 208 is mounted on the same side as the display input unit 204, but the mounting position of the camera unit 208 is not limited to this, and it may also be mounted on the back of the display input unit 204.

[0185] Furthermore, the camera unit 208 can be used for various functions of the smartphone 200. For example, images acquired by the camera unit 208 can be displayed on the display panel 202, or images from the camera unit 208 can be used as one of the inputs for the operation panel 203.

[0186] Furthermore, when the GNSS receiver 214 detects a position, it can also detect the position by referring to the image from the camera unit 208. Moreover, by referring to the image from the camera unit 208, it is possible to determine the optical axis direction of the camera unit 208 of the smartphone 200, or to determine the current usage environment, either without using the 3-axis accelerometer or in combination with the 3-axis accelerometer. Of course, the image from the camera unit 208 can also be used within the application software.

[0187] In addition, the still image or video image data can be supplemented with location information acquired by the GNSS receiver 214, audio information acquired by the microphone 206 (which may be converted to text by the main control unit, etc.), posture information acquired by the motion sensor 215, etc., and stored in the storage unit 212 or output via the external input / output unit 213 or the wireless communication unit 210. Even with a smartphone 200 configured as described above, the same effects as a digital camera 100 can be obtained. [Explanation of Symbols]

[0188] 1 imaging lens 2 apertures 4. Lens control unit 5 Image sensor 8. Lens drive unit 9. Aperture drive unit 11 System Control Unit 14,207 Operation section 15 Memory Control Unit 16 memory 17 Digital signal processing unit 20 External memory control unit 21 Storage medium 22a Display Controller 22b Display surface 22 Display device 24 control bus 25 Data Bus 40 Lens device 60 Photosensitive area 61A Photoelectric conversion unit 61B Charge holding section 61C Charge Transfer Section 61D Floating Diffusion 61E Circuit 61 pixels 62 pixel rows 63 Drive Circuit 64 Signal Processing Circuits 65 signal line 70 N-type substrate 71 P-well layer 72 electrode 73 N-type impurity layer 74 P-type impurity layer 75 areas 76 Transfer electrodes 77 Reset Transistor 78 Output transistors 79 Selective Transistors 100A Main Unit 100 Digital Cameras GR, GS, GRg1, GSg1, RRg1, RSg1, RO Straight line ST, ROg1, ROg2, ROg3, ROg4, ROG1, rog1 Straight line DR1, DR2, DR3, DR4 straight line 200 Smartphones 201 cabinet 202 Display Panel 203 Control Panel 204 Display Input Section 205 Speakers 206 Microphone 208 Camera Department 210 Wireless Communication Section 211 Telephone section 212 Storage section 213 External input / output section 214 GNSS receiver 215 Motion sensor unit 216 Power supply section 217 Internal storage 218 External storage unit 220 Main Control Unit

Claims

1. An imaging control device for controlling an image sensor having a plurality of pixel rows in which a plurality of pixels, each including a photoelectric conversion unit and a charge holding unit that holds the charge transferred from the photoelectric conversion unit, are arranged in one direction, Equipped with a processor, The aforementioned processor, A first control is performed to expose the plurality of pixel rows and transfer the charge accumulated in the photoelectric conversion unit of the plurality of pixel rows by the exposure to the charge holding unit. A second control is performed to read out a signal corresponding to the charge held in the charge holding unit of some of the pixel rows, among the charge holding units that have held charge by the first control. In the charge holding section that holds charge according to the first control, a third control is performed which reads out a signal corresponding to the charge held in the charge holding section of the pixel rows other than the certain pixel rows, and exposes the certain pixel rows from which the signal was read in the second control. A fourth control is performed to read a signal corresponding to the charge held in the charge holding section of some of the pixel rows from which the signal was read by the second control, The exposure in the first control is for the present imaging, and the exposure in the third control is for imaging for displaying the live view image. The aforementioned pixel row includes a pixel row for phase difference detection which includes a pixel row for phase difference detection, The aforementioned processor, Based on the signal read from the charge holding unit of the phase difference detection pixel row by the second control, an evaluation value for focus adjustment that varies according to the degree of blur of the captured image data is derived. If the evaluation value is below the threshold and it is estimated that the degree of blurring is not large, the execution of the third control is decided; if the evaluation value is above the threshold and it is estimated that the degree of blurring is large, the execution of the third control is decided not to be performed, and / or, If the evaluation value is greater than or equal to the threshold and the degree of blurring is estimated to be large, the image capture control device controls the signal readout speed in the fourth control to be faster than when the evaluation value is less than the threshold and the degree of blurring is estimated to be small.

2. The imaging control device according to claim 1, The processor is an imaging control device that performs display control of a live view image based on the result of one or both of the second and fourth controls.

3. The imaging control device according to claim 1, The processor is an imaging control device that, in the third control, performs exposure of some of the pixel rows by shifting the exposure period of all or some of the pixel rows.

4. The imaging control device according to claim 1, The processor, when the third control is not executed, performs a fifth control in place of the third and fourth controls, which reads out a signal corresponding to the charge held in the charge holding unit of the pixel rows other than the certain pixel rows among the charge holding units that hold charge by the first control.

5. The imaging control device according to claim 1, The processor is an imaging control device that, when the third control is not performed, performs the first control instead of the third and fourth controls.

6. The imaging control device according to claim 1, The processor is an imaging control device that, when the evaluation value is greater than or equal to a threshold, makes the signal readout speed in the fourth control faster than the signal readout speed in the third control.

7. The imaging control device according to claim 1, The processor is an imaging control device that makes the signal readout speed in the fourth control faster than the signal readout speed in the third control.

8. The imaging control device according to claim 1, The processor is an imaging control device that performs the second control, followed by the third and fourth controls in that order at least once.

9. The imaging control device according to claim 8, The imaging control device wherein, when the processor completes the reading of signals from the charge holding units of the pixel rows other than the certain pixel rows by the third control, it omits the fourth control which is performed following the third control, or starts the first control in conjunction with the fourth control.

10. The imaging control device according to claim 8, The processor performs the above process n times, where n is a plurality of times, and after the nth time the above process, it performs the first control and control to read out a signal corresponding to the charge held in the charge holding unit for pixel rows other than the certain pixel rows for which signal readout is incomplete.

11. An imaging control device according to any one of claims 1 to 10, An imaging device comprising the aforementioned image sensor.

12. An imaging control method for controlling an image sensor having multiple pixel rows in which multiple pixels, each including a photoelectric conversion unit and a charge holding unit that holds the charge transferred from the photoelectric conversion unit, are arranged in one direction, A first control is performed to expose the plurality of pixel rows and transfer the charge accumulated in the photoelectric conversion unit of the plurality of pixel rows by the exposure to the charge holding unit. A second control is performed to read out a signal corresponding to the charge held in the charge holding unit of some of the pixel rows, among the charge holding units that have held charge by the first control. In the charge holding section that holds charge according to the first control, a third control is performed which reads out a signal corresponding to the charge held in the charge holding section of the pixel rows other than the certain pixel rows, and exposes the certain pixel rows from which the signal was read in the second control. A fourth control is performed to read a signal corresponding to the charge held in the charge holding section of some of the pixel rows from which the signal was read by the second control, The exposure in the first control is for the present imaging, and the exposure in the third control is for imaging for displaying the live view image. The aforementioned pixel row includes a pixel row for phase difference detection which includes a pixel row for phase difference detection, Based on the signal read from the charge holding unit of the phase difference detection pixel row by the second control, an evaluation value for focus adjustment that varies according to the degree of blur of the captured image data is derived. If the evaluation value is below the threshold and it is estimated that the degree of blurring is not large, the execution of the third control is decided; if the evaluation value is above the threshold and it is estimated that the degree of blurring is large, the execution of the third control is decided not to be performed, and / or, An imaging control method in which, if the evaluation value is greater than or equal to the threshold and the degree of blurring is estimated to be large, the signal readout speed in the fourth control is controlled to be faster than when the evaluation value is less than the threshold and the degree of blurring is estimated to be small.

13. An imaging control program for controlling an image sensor having multiple pixel rows in which multiple pixels, each containing a photoelectric conversion unit and a charge holding unit that holds the charge transferred from the photoelectric conversion unit, are arranged in one direction, A first control is performed to expose the plurality of pixel rows and transfer the charge accumulated in the photoelectric conversion unit of the plurality of pixel rows by the exposure to the charge holding unit. A second control is performed to read out a signal corresponding to the charge held in the charge holding unit of some of the pixel rows, among the charge holding units that have held charge by the first control. In the charge holding section that holds charge according to the first control, a third control is performed which reads out a signal corresponding to the charge held in the charge holding section of the pixel rows other than the certain pixel rows, and exposes the certain pixel rows from which the signal was read in the second control. A fourth control is performed to read a signal corresponding to the charge held in the charge holding section of some of the pixel rows from which the signal was read by the second control, The exposure in the first control is for the present imaging, and the exposure in the third control is for imaging for displaying the live view image. The aforementioned pixel row includes a pixel row for phase difference detection which includes a pixel row for phase difference detection, Based on the signal read from the charge holding unit of the phase difference detection pixel row by the second control, an evaluation value for focus adjustment that varies according to the degree of blur of the captured image data is derived. If the evaluation value is below the threshold and it is estimated that the degree of blurring is not large, the execution of the third control is decided; if the evaluation value is above the threshold and it is estimated that the degree of blurring is large, the execution of the third control is decided not to be performed, and / or, An imaging control program that causes the processor to execute a step to control the signal readout speed in the fourth control so that it is faster than when the evaluation value is below the threshold and the degree of blur is estimated to be large.

Images