Image control device, imaging device, image control method, image control program

Abstract

The present invention provides an imaging control device, an imaging device, an imaging control method, and an imaging control program. [Solution] The system control unit 11 starts exposure of each pixel 61 at time t1, performs a first control at time t2 to transfer the charge accumulated in the photoelectric conversion unit 61A of each pixel to the charge holding unit 61B, performs a second control to read out a signal corresponding to the charge transferred to the charge holding unit 61B by the first control, and before time t1, captures an image of the subject with the image sensor 5 to acquire captured image data, and controls the operating conditions of the imaging unit 50 during the period in which the second control is performed based on the captured image data.

Description

Technical Field

[0001] The present invention relates to an imaging control device, an imaging device, an imaging control method, and an imaging control program.

Background Art

[0002] Patent Document 1 describes an imaging device that simultaneously resets the signals of all pixels in a photoelectric conversion unit, determines an exposure time by transferring the signals from the photoelectric conversion unit to an accumulation unit after a predetermined time, and then sequentially reads out the signals from the pixels, and an incident light amount suppression means for suppressing the amount of light incident on the imaging device during the signal readout operation from each pixel of the imaging device.

[0003] Patent Document 2 describes an imaging device having a detection unit that detects pixels in which the output level of an imaging signal is close to saturation and the output level of the imaging signal decreases due to an increase in the amount of signal charge held in a charge holding unit due to the influence of a high-brightness subject.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Means for Solving the Problems

[0005] The technology of the present disclosure is as follows.

[0006] (1) An imaging control device for controlling an imaging unit including an imaging device in which a plurality of pixels including a photoelectric conversion unit and a charge holding unit for holding charges transferred from the photoelectric conversion unit are two-dimensionally arranged, comprising a processor, wherein the processor At the first timing, exposure of the above-mentioned multiple pixels is started, and at the second timing, a first control is performed to transfer the charge accumulated in the photoelectric conversion section of the above-mentioned multiple pixels by the exposure to the charge holding section. A second control is performed to read out a signal corresponding to the charge transferred to the charge holding unit by the first control described above. An imaging control device that captures an image of a subject using the image sensor during a second period different from the first period during which the first control described above is performed, thereby acquiring image data, and performs a third control to control the operating conditions of the imaging unit during the period during which the second control described above is performed, based on the image data described above.

[0007] (2) (1) The imaging control device described above, The first period and the second period described above are non-overlapping in the imaging control device.

[0008] (3) (2) The imaging control device described above, The above processor is an imaging control device that acquires the brightness distribution of a subject captured by the image sensor based on the above captured image data, and controls the above operating conditions based on the above brightness distribution.

[0009] (4) (3) The imaging control device described above, The second period mentioned above is the period prior to the first timing mentioned above. The above processor is an imaging control device that, in the second period, causes the image sensor to image the subject at an exposure value lower than the exposure value in the first period, and acquires the image data.

[0010] (5) (3) The imaging control device described above, The above processor is In the imaging during the second period described above, control is performed to read signals from groups when the multiple pixels are divided into multiple groups, and the image imaging data composed of the image data corresponding to the groups is acquired. An imaging control device that causes a first group, which is part of the above-mentioned multiple groups, to take images at a first exposure value, and causes a second group, which is lower than the first exposure value, to take images at a second exposure value.

[0011] (6) (5) The imaging control device described above, The above processor is Based on the image data corresponding to the first group described above, the display of the live view image is controlled. An imaging control device that acquires the brightness distribution based on the image data corresponding to the second group described above.

[0012] (7) An imaging control device according to any one of (3) to (6), The above processor is Based on the above brightness distribution, high-brightness regions where the brightness of the subject being imaged exceeds a threshold are detected. During the second period described above, multiple imaging operations were performed to acquire multiple images of the above-mentioned image data. An imaging control device that, when it is determined that the high-brightness region detected based on the above-mentioned multiple imaging image data has moved between the above-mentioned multiple imaging cycles, changes the above-mentioned operating conditions from the first condition to the second condition.

[0013] (8) An imaging control device according to any one of (1) to (7), The imaging unit includes a first optical element capable of changing the amount of light incident on the image sensor. The above operating conditions are conditions relating to the operation of the first optical element, in the imaging control device.

[0014] (9) (8) The imaging control device described above, The above-described third control includes an imaging control device that controls the first optical element to reduce the amount of incident light during the period in which the second control is performed to less than the amount of incident light to the image sensor during the period in which the first control is performed.

[0015] (10) An imaging control device according to any one of (1) to (9), where the imaging unit includes a focus lens, and the operation conditions are conditions related to the operation of the focus lens. The imaging control device.

[0016] (11) An imaging control device according to (10), where the third control includes control to stop driving the focus lens. The imaging control device.

[0017] (12) An imaging control device according to any one of (1) to (11), where the operation conditions are conditions related to the readout operation of signals from the plurality of charge holding units. The imaging control device.

[0018] (13) An imaging control device according to (12), where the third control includes control to set the readout speed of the signal from the charge holding unit to a first speed and control to set the readout speed to a second speed higher than the first speed. The imaging control device.

[0019] (14) An imaging control device according to (12), where the operation conditions are conditions related to the readout order of signals from the plurality of charge holding units. The imaging control device.

[0020] (15) An imaging control device according to (14), where the processor acquires the luminance distribution of the subject imaged by the imaging device based on the captured image data, detects a high-luminance region based on the luminance distribution, and when it is determined that the high-luminance region exists, determines the readout order of the signals based on the positions of the pixels where the high-luminance region is imaged. The imaging control device.

[0021] (16) An imaging control device for controlling an imaging unit comprising an imaging sensor having a plurality of pixels arranged in a two-dimensional manner, each including a photoelectric conversion unit and a charge holding unit that holds the charge transferred from the photoelectric conversion unit, Equipped with a processor, The above processor is A first imaging control is performed to cause the image sensor to capture the subject at a first exposure value. A second imaging control is performed to cause the image sensor to capture the subject at a second exposure value lower than the first exposure value mentioned above. An imaging control device that, based on the imaging image data acquired from the image sensor by the second imaging control described above, acquires the luminance distribution of the subject being imaged by the image sensor, and performs specific control based on the luminance distribution.

[0022] (17) The imaging control device described in (16), The above processor is an imaging control device that detects high-luminance regions where the luminance is above a threshold based on the above luminance distribution, and performs the above-mentioned specific control when such high-luminance regions exist.

[0023] (18) (17) The imaging control device described above, The above processor is The above second imaging control is performed multiple times to acquire multiple images of the above-mentioned image data. An imaging control device that performs the specific control described above when it is determined that the high-brightness region detected based on the above-mentioned multiple imaging image data has moved between the above-mentioned multiple imaging cycles.

[0024] (19) An imaging control device for controlling an imaging unit comprising an imaging sensor having a plurality of pixels arranged in a two-dimensional manner, each including a photoelectric conversion unit and a charge holding unit that holds the charge transferred from the photoelectric conversion unit, Equipped with a processor, The above processor is The above image sensor is used to perform imaging control to capture an image of the subject. In the imaging control described above, when the multiple pixels are divided into multiple groups, control is performed to read signals from the groups, and imaging image data consisting of image data corresponding to the groups is obtained. For the first group, which is part of the above multiple groups, imaging is performed at a first exposure value, and for the second group, which is part of the above multiple groups excluding the first group, imaging is performed at a second exposure value lower than the first exposure value. An imaging control device that acquires the luminance distribution of a subject captured by the image sensor based on the above-mentioned captured image data, and performs specific control based on the above-mentioned luminance distribution.

[0025] (20) (19) The imaging control device described above, The above processor is Based on the image data corresponding to the first group described above, the display of the live view image is controlled. An imaging control device that acquires the brightness distribution based on the image data corresponding to the second group described above.

[0026] (twenty one) An imaging control device as described in any of (1) to (20), An imaging device comprising the above-mentioned imaging unit.

[0027] (twenty two) An imaging control method for controlling an imaging unit comprising an imaging sensor having a plurality of pixels arranged in a two-dimensional manner, each including a photoelectric conversion unit and a charge holding unit that holds the charge transferred from the photoelectric conversion unit, At the first timing, exposure of the above-mentioned multiple pixels is started, and at the second timing, a first control is performed to transfer the charge accumulated in the photoelectric conversion section of the above-mentioned multiple pixels by the exposure to the charge holding section. A second control is performed to read out a signal corresponding to the charge transferred to the charge holding unit by the first control described above. An imaging control method comprising: capturing an image of a subject with the image sensor during a second period different from the first period during which the first control described above is performed, acquiring image data; and performing a third control to control the operating conditions of the imaging unit during the period during which the second control described above is performed, based on the image data described above.

[0028] (twenty three) An imaging control program for controlling an imaging unit comprising an imaging sensor having a plurality of pixels arranged in a two-dimensional manner, each including a photoelectric conversion unit and a charge holding unit that holds the charge transferred from the photoelectric conversion unit, At the first timing, exposure of the above-mentioned multiple pixels is started, and at the second timing, a first control is performed to transfer the charge accumulated in the photoelectric conversion section of the above-mentioned multiple pixels by the exposure to the charge holding section. A second control is performed to read out a signal corresponding to the charge transferred to the charge holding unit by the first control described above. An imaging control program that causes a processor to execute a step of: capturing an image of a subject with the image sensor during a second period different from the first period during which the first control described above is performed, acquiring image data, and performing a third control to control the operating conditions of the imaging unit during the period during which the second control described above is performed, based on the image data described above. [Brief explanation of the drawing]

[0029] [Figure 1] This figure shows a schematic configuration of a digital camera 100, which is one embodiment of the imaging device of the present invention. [Figure 2] Figure 1 is a schematic plan view showing the general configuration of the image sensor 5. [Figure 3] Figure 2 is a schematic plan view showing the general configuration of pixels 61 in the image sensor 5. [Figure 4] Figure 3 is a schematic cross-sectional view of the AA line of pixel 61. [Figure 5] Figure 1 is a timing chart showing the operation of the digital camera 100 in imaging mode. [Figure 6] This is a diagram showing the appearance of the Smartphone 200. [Figure 7]Figure 6 shows the configuration of smartphone 200, which is a block diagram. [Modes for carrying out the invention]

[0030] Figure 1 is a diagram showing the schematic configuration of a digital camera 100, which is one embodiment of the imaging device of the present invention. The digital camera 100 shown in Figure 1 comprises a lens device 40 having an imaging lens 1, an aperture 2, a lens drive unit 8 that drives the imaging lens 1, an aperture drive unit 9 that drives the aperture 2, and a lens control unit 4 that controls the lens drive unit 8 and the aperture drive unit 9, and a main body 100A.

[0031] The main unit 100A comprises an image sensor 5, a system control unit 11 that provides overall control of the entire electrical control system of the digital camera 100, an operation unit 14, a display device 22, a memory 16 including RAM (Random Access Memory) and ROM (Read Only Memory), a memory control unit 15 that controls data storage in the memory 16 and data reading from the memory 16, a digital signal processing unit 17, and an external memory control unit 20 that controls data storage in the storage medium 21 and data reading from the storage medium 21.

[0032] The lens device 40 may be detachable from the main body 100A, or it may be integrated with the main body 100A. The imaging lens 1 includes a focus lens that is movable in the optical axis direction. This focus lens is a lens for adjusting the focus of the imaging optical system, which includes the imaging lens 1 and the aperture 2, and is composed of a single lens or multiple lenses. As the focus lens moves in the optical axis direction, the position of the principal point of the focus lens changes along the optical axis direction, thereby changing the focal position on the subject side. As the focus lens, a liquid lens whose principal point position in the optical axis direction can be changed by electrical control may be used.

[0033] The aperture 2 is an optical element capable of changing the amount of light incident on the image sensor 5. The lens device 40 may also include an ND (Neutral Density) filter as an optical element capable of changing the amount of light incident on the image sensor 5. The ND filter may be capable of changing the amount of light incident on the image sensor 5 by electrically controlling the light transmittance, or it may be capable of changing the amount of light incident on the image sensor 5 by being configured to move back and forth in relation to the optical path (reducing the light transmittance when inserted into the optical path). The aperture 2 and the ND filter each constitute the first optical element.

[0034] The lens control unit 4 of the lens device 40 controls the lens drive unit 8 based on the lens drive signal transmitted from the system control unit 11 to change the position of the principal point of the focus lens included in the imaging lens 1. The lens control unit 4 of the lens device 40 controls the aperture drive unit 9 based on the drive control signal transmitted from the system control unit 11 to change the aperture amount (F number) of the aperture 2. If the lens device 40 includes an ND filter, the lens control unit 4 electrically controls the transmittance of the ND filter or inserts or removes the ND filter from the optical path in response to a command from the system control unit 11.

[0035] The image sensor 5 captures an image of the subject through an imaging optical system that includes an imaging lens 1 and an aperture 2 (and an ND filter). The image sensor 5 has a light-receiving surface 60 (see Figure 2) in which multiple pixels are arranged in two dimensions, and the imaging optical system converts the image of the subject formed on this light-receiving surface 60 into a pixel signal using these multiple pixels and outputs it. For example, a CMOS (complementary metal-oxide semiconductor) image sensor is used for the image sensor 5. The imaging unit 50 is composed of the image sensor 5 and the lens device 40.

[0036] The system control unit 11 provides overall control for the digital camera 100. Its hardware structure consists of various processors that execute programs, including an image capture control program. The programs executed by the system control unit 11 are stored in the ROM of the memory 16.

[0037] Various types of processors include CPUs (Central Processing Units), which are general-purpose processors that execute programs and perform various processes; Programmable Logic Devices (PLDs), such as FPGAs (Field Programmable Gate Arrays), whose circuit configurations can be changed after manufacturing; and dedicated electrical circuits, such as ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform particular processes. More specifically, the structure of these various types of processors is an electrical circuit that combines circuit elements such as semiconductor elements.

[0038] The system control unit 11 may be composed of one of various processors, or it may be composed of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs or a combination of a CPU and an FPGA).

[0039] The system control unit 11 drives the image sensor 5 and the lens device 40 according to the imaging control program, and outputs the subject image captured through the imaging optical system of the lens device 40 as an image signal. The system control unit 11 and the memory 16 constitute an imaging control device that controls the imaging unit 50. The image signal output from the image sensor 5 is processed by the digital signal processing unit 17 to generate imaging image data that is suitable for display on the display device 22 or suitable for storage on the storage medium 21.

[0040] The system control unit 11 receives instruction signals from the user through the operation unit 14. The operation unit 14 includes a touch panel integrated with the display surface 22b, as well as various buttons and other controls.

[0041] The display device 22 comprises a display surface 22b composed of an organic EL (electroluminescence) panel or a liquid crystal panel, and a display controller 22a that controls the display on the display surface 22b.

[0042] The memory control unit 15, the digital signal processing unit 17, the external memory control unit 20, and the display controller 22a are interconnected by a control bus 24 and a data bus 25, and are controlled by commands from the system control unit 11.

[0043] Figure 2 is a schematic plan view showing the general configuration of the image sensor 5 shown in Figure 1. Figure 3 is a schematic plan view showing the general configuration of the pixels 61 in the image sensor 5 shown in Figure 2. Figure 4 is a schematic cross-sectional view of the AA line of the pixels 61 shown in Figure 3.

[0044] The image sensor 5 comprises a light-receiving surface 60 in which multiple pixel rows 62, each consisting of multiple pixels 61 arranged in the row direction X, are arranged in the column direction Y which is orthogonal to the row direction X; a drive circuit 63 for driving the pixels 61 arranged on the light-receiving surface 60; and a signal processing circuit 64 for processing the pixel signals read out to the signal lines from each pixel 61 of the pixel rows 62 arranged on the light-receiving surface 60.

[0045] The multiple pixels 61 include a phase difference detection pixel that receives one of a pair of light beams that have passed through two different parts aligned in the row direction X of the pupil region of the imaging optical system and detects a signal corresponding to the amount of light received, a phase difference detection pixel that receives the other of the pair of light beams and detects a signal corresponding to the amount of light received, and a normal pixel that receives both of the pair of light beams and detects a signal corresponding to the amount of light received.

[0046] Each pixel row 62 includes a first pixel row containing only normal pixels and a second pixel row containing both phase-difference detection pixels and normal pixels. The second pixel row is discretely arranged at equal intervals in the column direction Y. In the image sensor 5, phase-difference detection pixels are not mandatory, and all pixels 61 may consist of normal pixels.

[0047] In the following, in Figure 2, the upper end of the light-receiving surface 60 in the column direction Y will be referred to as the upper end, and the lower end of the light-receiving surface 60 in the column direction Y will be referred to as the lower end. This upper end constitutes one end of the light-receiving surface 60, and this lower end constitutes the other end of the light-receiving surface 60.

[0048] As shown in Figure 3, the pixel 61 includes a photoelectric conversion unit 61A, a charge holding unit 61B, a charge transfer unit 61C, a floating diffusion 61D, and a readout circuit 61E.

[0049] The photoelectric conversion unit 61A receives light that has passed through the imaging optical system of the lens device 40 and generates and stores an electric charge corresponding to the amount of light received. The photoelectric conversion unit 61A is composed of a photodiode or the like.

[0050] The charge transfer unit 61C transfers the charge accumulated in the photoelectric conversion unit 61A to the charge holding unit 61B. The charge transfer unit 61C is composed of an impurity region in the semiconductor substrate and an electrode formed above this impurity region.

[0051] The voltage applied to the electrodes constituting the charge transfer unit 61C is controlled by the drive circuit 63, thereby transferring charge from the photoelectric conversion unit 61A to the charge holding unit 61B.

[0052] The charge holding section 61B holds the charge transferred from the photoelectric conversion section 61A by the charge transfer section 61C. The charge holding section 61B is composed of impurity regions within the semiconductor substrate.

[0053] The floating diffusion 61D is used to convert electric charge into a signal, and the charge held in the charge holding unit 61B is transferred to it.

[0054] The readout circuit 61E is a circuit that reads out a signal corresponding to the potential of the floating diffusion 61D as a pixel signal to the signal line 65. The readout circuit 61E is driven by the drive circuit 63.

[0055] As shown in Figure 4, a P-well layer 71 is formed on the surface of the N-type substrate 70, and a photoelectric conversion section 61A is formed on the surface portion of the P-well layer 71.

[0056] The photoelectric conversion section 61A is composed of an N-type impurity layer 73 and a P-type impurity layer 74 formed thereon. The semiconductor substrate is composed of an N-type substrate 70 and a P-well layer 71.

[0057] On the surface of the P-well layer 71, a charge-holding portion 61B made of an N-type impurity layer is formed, slightly separated from the photoelectric conversion portion 61A.

[0058] A transfer electrode 76 is formed above the region 75 of the P-well layer 71 between the charge holding portion 61B and the photoelectric conversion portion 61A, via an oxide film (not shown).

[0059] The region 75 and the transfer electrode 76 constitute the charge transfer section 61C. In the example in Figure 3, the transfer electrode 76 is formed above the charge holding section 61B, but the transfer electrode 76 only needs to be formed at least above the region 75.

[0060] By controlling the potential of the transfer electrode 76 to form a channel in region 75, the charge accumulated in the photoelectric conversion unit 61A can be transferred to the charge holding unit 61B. The potential of the transfer electrode 76 is controlled by the drive circuit 63.

[0061] A floating diffusion layer 61D, consisting of an N-type impurity layer, is formed on the surface of the P-well layer 71, slightly separated from the charge-holding portion 61B.

[0062] A readout electrode 72 is formed above the P-well layer 71 between the charge-holding portion 61B and the floating diffusion 61D, via an oxide film (not shown).

[0063] By controlling the potential of the readout electrode 72 and forming a channel in the region between the charge holding section 61B and the floating diffusion 61D, the charge held in the charge holding section 61B can be transferred to the floating diffusion 61D. The potential of the readout electrode 72 is controlled by the drive circuit 63.

[0064] In the example shown in Figure 4, the readout circuit 61E consists of a reset transistor 77 for resetting the potential of the floating diffusion 61D, an output transistor 78 for converting the potential of the floating diffusion 61D into a pixel signal and outputting it, and a selection transistor 79 for selectively reading the pixel signal output from the output transistor 78 to the signal line 65. The configuration of the readout circuit is just one example and is not limited to this. The readout circuit 61E may also be shared by multiple pixels 61.

[0065] Pixel 61 is provided with a light-shielding film (not shown), and areas other than the photoelectric conversion unit 61A are shielded from light by this light-shielding film.

[0066] The structure of pixel 61 shown in Figures 3 and 4 is an example and is not limited thereto.

[0067] The drive circuit 63 shown in Figure 2 independently drives the transfer electrode 76, read electrode 72, and read circuit 61E of each pixel 61 for each pixel row 62 to reset each photoelectric conversion unit 61A included in the pixel row 62 (discharge of charge accumulated in the photoelectric conversion unit 61A), read out the pixel signal to the signal line 65 according to the charge accumulated in each photoelectric conversion unit 61A, and so on.

[0068] Furthermore, the drive circuit 63 simultaneously drives the charge transfer units 61C of all pixels 61 to simultaneously transfer charge from the photoelectric conversion unit 61A of each pixel 61 to the charge holding unit 61B. The drive circuit 63 is controlled by the system control unit 11.

[0069] The photoelectric conversion unit 61A is reset by resetting the floating diffusion 61D with the reset transistor 77 while the charge transfer unit 61C is in a state where charge can be transferred and a channel is formed in the semiconductor substrate below the read electrode 72.

[0070] Therefore, once the reading of the pixel signal corresponding to the charge held by the charge holding unit 61B has been completed, it is possible to reset the photoelectric conversion unit 61A that transfers charge to the charge holding unit 61B (in other words, to start exposure of the photoelectric conversion unit 61A).

[0071] The signal processing circuit 64 shown in Figure 2 performs correlated double sampling on the pixel signals read from each pixel 61 of the pixel row 62 to the signal line 65, converts the pixel signals after correlated double sampling into digital signals, and outputs them to the data bus 25 (see Figure 1). The signal processing circuit 64 is controlled by the system control unit 11. The digital signal processing unit 17 performs signal processing such as demosaicing and gamma correction on the group of pixel signals output from the image sensor 5 to the data bus 25 to generate image data.

[0072] The system control unit 11 can drive the image sensor 5 with a global reset drive, a global shutter drive, a rolling reset drive, a rolling shutter drive, a first rolling readout drive, and a second rolling readout drive.

[0073] Global reset drive is a drive that simultaneously resets the photoelectric conversion unit 61A of each pixel 61 formed on the light-receiving surface 60 of the image sensor 5, and simultaneously starts exposure of each pixel 61.

[0074] Global shutter drive is a drive that simultaneously transfers the charge accumulated in the photoelectric conversion unit 61A of each pixel 61 by exposure initiated at each pixel 61 by global reset drive to the charge holding unit 61B, thereby simultaneously ending exposure at each pixel 61.

[0075] The rolling reset drive is a drive that sequentially performs the process of resetting each photoelectric conversion unit 61A in a pixel row 62 and starting exposure for each photoelectric conversion unit 61A, while changing the pixel row 62.

[0076] Rolling shutter drive is a drive that sequentially performs the process of transferring charge from the photoelectric conversion unit 61A of the exposed pixel row 62 to the charge holding unit 61B of the same pixel row 62 to end the exposure of that pixel row 62, while changing the pixel row 62.

[0077] The first rolling readout drive is a drive that sequentially reads out the pixel signals corresponding to the charges held in each charge holding unit 61B by the global shutter drive, for each pixel row 62.

[0078] The second rolling readout drive is a drive that sequentially reads out pixel signals corresponding to the charge held in the charge holding section 61B of the pixel row 62 by the rolling shutter drive, while changing the pixel row 62.

[0079] When the digital camera 100 is set to imaging mode, the system control unit 11 continuously performs imaging for live view image display (hereinafter referred to as LV imaging) by, for example, a set of rolling reset drive, rolling shutter drive, and second rolling readout drive. Alternatively, the system control unit 11 may perform LV imaging by a set of global reset drive, global shutter drive, and first rolling readout drive.

[0080] Then, when the system control unit 11 receives an instruction to perform imaging for storage of still image data to be stored in the storage medium 21 (hereinafter referred to as an imaging instruction) during the execution of this set, it performs imaging for storage by a set of global reset drive, global shutter drive, and first rolling readout drive. The digital signal processing unit 17 shown in Figure 1 processes the pixel signal group output from the image sensor 5 by this imaging for storage to generate image data, and stores this image data in the storage medium 21.

[0081] Figure 5 is a timing chart showing the operation of the digital camera 100 in imaging mode as shown in Figure 1. In Figure 5, the horizontal axis represents time. The upper part of Figure 5 shows the display synchronization signal VD supplied to the display controller 22a.

[0082] The middle section of Figure 5 shows the drive timing of the photoelectric conversion unit 61A and the charge holding unit 61B for each pixel row 62 of the image sensor 5. In the middle section of Figure 5, the vertical axis indicates the position in the column direction Y of the pixel row 62.

[0083] The straight line RR shown in the middle of Figure 5 indicates the timing at which each photoelectric conversion unit 61A included in the pixel row 62 is reset by the rolling reset drive.

[0084] The straight line RS shown in the middle of Figure 5 indicates the timing at which exposure of each photoelectric conversion unit 61A included in the pixel row 62 is terminated by the rolling shutter drive.

[0085] The period enclosed by the linear RR and the linear RS to its right indicates the exposure period (LV1, LV2) of the image sensor 5 during LV imaging.

[0086] The straight line GR shown in the middle of Figure 5 indicates the timing at which each photoelectric conversion unit 61A included in the pixel row 62 is reset by the global reset drive.

[0087] The straight line GS shown in the middle of Figure 5 indicates the timing at which charge is transferred from each photoelectric conversion unit 61A included in the pixel row 62 to the charge holding unit 61B by global shutter driving.

[0088] The period enclosed by the straight line GR and the straight line GS represents the exposure period EX of the image sensor 5 during memory imaging.

[0089] The straight line ST shown in the middle of Figure 5 indicates the timing at which charge is held in the charge holding unit 61B.

[0090] The straight line RO1 shown in the middle of Figure 5 indicates the timing at which a pixel signal corresponding to the charge held in the charge holding unit 61B is output from the image sensor 5 by the first rolling readout drive.

[0091] The straight line RO2 shown in the middle of Figure 5 indicates the timing at which the pixel signal corresponding to the charge held in the charge holding unit 61B is output from the image sensor 5 by the second rolling readout drive.

[0092] The lower part of Figure 5 shows the drawing state of the display surface 22b. In the lower part of Figure 5, the vertical axis indicates the position in the column direction Y of the display pixel row on the display surface 22b.

[0093] The straight line DR shown in the lower part of Figure 5 indicates the timing at which drawing occurs on the display pixel row of the display surface 22b.

[0094] When the system control unit 11 is set to imaging mode, it repeatedly executes a set of a rolling reset drive indicated by linear RR, a rolling shutter drive indicated by linear RS, and a second rolling readout drive indicated by linear RO2 at predetermined intervals.

[0095] When a pixel signal is output from pixel row 62 by the drive indicated by the straight line RO2 of this set, line data is generated based on this pixel signal, and a line image based on this line data is drawn on the display pixel row corresponding to this pixel row 62.

[0096] In Figure 5, “lv1” indicates the period during which the live view image obtained during exposure period LV1 is displayed. In Figure 5, “lv2” indicates the period during which the live view image obtained during exposure period LV2 is displayed.

[0097] If an imaging instruction is given while the above setup for LV imaging is being performed, the system control unit 11 terminates the setup that was being executed at the time the imaging instruction was received, and then at time t1, performs a global reset drive indicated by the line GR, simultaneously resetting the photoelectric conversion unit 61A in all pixel rows 62. As a result, exposure starts at the same time in all pixel rows 62. After a predetermined exposure time has elapsed, the system control unit 11 performs a global shutter drive indicated by the line GS at time t2. This drive simultaneously transfers charge from the photoelectric conversion unit 61A to the charge holding unit 61B in all pixel rows 62, and the charge is held in the charge holding unit 61B as indicated by the line ST. As a result, exposure ends at the same time in all pixel rows 62.

[0098] In Figure 5, the period enclosed by the lines GR and GS is shown as the exposure period EX for memory imaging. When the global shutter is driven at time t2, the charge generated in each photoelectric conversion unit 61A during the exposure period EX is transferred to the charge holding unit 61B (line ST in Figure 5).

[0099] Time t1 constitutes the first timing, and time t2 constitutes the second timing. The global reset drive and global shutter drive performed between time t1 and time t2 constitute the first control or first imaging control. The exposure period EX constitutes the first period.

[0100] The system control unit 11 performs a global shutter drive indicated by the line GS, and then performs a first rolling readout drive indicated by the line RO1. In this first rolling readout drive, the system control unit 11 sequentially selects pixel rows 62 from the upper end to the lower end of the light-receiving surface 60, and reads out pixel signals from the selected pixel rows 62.

[0101] The first rolling readout drive, indicated by the linear RO1, constitutes the second control. The period during which the first rolling readout drive, indicated by the linear RO1, is performed (i.e., the period during which the second control is performed) will be referred to below as the signal readout period. The pixel signals output from the image sensor 5 during this signal readout period are processed by the digital signal processing unit 17 to become image data, which is then stored in the storage medium 21.

[0102] The signal readout period increases as the number of pixels in the captured image data (the number of pixels 61 that are subject to signal readout) increases. Although the charge holding section 61B is shielded from light, if light is shone on the light-shielding film above the charge holding section 61B during the signal readout period, the influence of that light may cause leakage charge to be generated below the area that was shone with light, charge that was not present during the exposure period EX. This leakage charge tends to increase as the signal readout period lengthens, because light is shone for a longer period of time. Also, the amount of leakage charge tends to increase with increasing light intensity.

[0103] In this embodiment, the system control unit 11 determines whether or not leakage charge that may affect the image quality of the captured image data is generated before the signal readout period ends.

[0104] Specifically, the system control unit 11 captures an image of the subject using the image sensor 5 during a period different from the exposure period EX (for example, an exposure period LV2) to acquire captured image data, and then performs a third control to control the operating conditions of the imaging unit 50 during the signal readout period based on this captured image data.

[0105] The system control unit 11, based on the image data obtained during exposure period LV2, determines that leakage charge that could affect the image quality of the image data obtained during exposure period EX is generated. In this case, it changes the operating conditions of the imaging unit 50 during the signal readout period (i.e., controls the operating conditions of the imaging unit 50) compared to when it determines that no leakage charge that could affect the image quality of the image data obtained during exposure period EX is generated. Changing the operating conditions of the imaging unit 50 constitutes a specific control.

[0106] The operating conditions for the imaging unit 50 include conditions related to the operation of the aperture 2 (specifically, setting the F-number), conditions related to the operation of the ND filter (specifically, setting the light transmittance), conditions related to the operation of the focus lens (specifically, setting the position of the principal point of the focus lens), and conditions related to the readout operation of the pixel signal from the charge holding unit 61B of each pixel 61 (specifically, setting the readout speed of the pixel signal or setting the readout order of the pixel signal).

[0107] For example, consider a case where the exposure value during exposure period EX (a value determined by a combination of exposure time, imaging sensitivity, and the F-number of aperture 2 (and / or the transmittance of the ND filter)) is manually set by the user. In this case, the system control unit 11 sets the exposure value during LV imaging lower than the exposure value set by the user. Methods to lower the exposure value include shortening the exposure time, lowering the imaging sensitivity, increasing the F-number of aperture 2, and lowering the transmittance of the ND filter.

[0108] The system control unit 11 then acquires, for example, the image data output from the image sensor 5 during LV imaging with an exposure period of LV2, and obtains the brightness distribution of the subject being imaged by the image sensor 5 based on this image data. Based on this brightness distribution, the system control unit 11 detects high-brightness regions in the subject being imaged by the image sensor 5 where the brightness is above a threshold.

[0109] Exposure period LV2 constitutes the second period. This second period is different from the first period in which imaging for memory is performed, and does not overlap with the first period. The control for performing imaging during exposure period LV2 constitutes the second imaging control. Note that the first and second periods may partially overlap. For example, the start timing of exposure period EX may coincide with the end timing of the lowermost pixel row 62 in exposure period LV2.

[0110] The system control unit 11 determines that the subject being imaged contains high-luminance regions if, in the imaged image data obtained during imaging with exposure period LV2, there is a region (hereinafter referred to as a saturation region) where pixels whose pixel values ​​have reached a saturation value are clustered over a predetermined area or larger. If there is no saturation region in the imaged image data, the system control unit 11 determines that the subject being imaged does not contain high-luminance regions.

[0111] If the system control unit 11 determines that the subject being imaged does not contain high-brightness areas, it determines that there is no possibility of leakage charge occurring in the signal readout period immediately after the exposure period EX, which could affect the image quality of the captured image data. If the system control unit 11 determines that the subject being imaged contains high-brightness areas, it determines that there is a possibility of leakage charge occurring in the signal readout period immediately after the exposure period EX, which could affect the image quality of the captured image data. The following describes an example of controlling the operating conditions of the imaging unit 50 during the signal readout period.

[0112] (Example of controlling the F-number during the signal readout period) If the system control unit 11 determines that the subject being imaged does not contain high-brightness areas, it maintains the F-number of aperture 2 during the signal readout period immediately following the exposure period EX at the same value as set during the exposure period EX. If it determines that the subject being imaged contains high-brightness areas, it sets the F-number of aperture 2 during the signal readout period immediately following the exposure period EX to a value higher than the value set during the exposure period EX.

[0113] As a result, during the signal readout period, the amount of incident light on the light-receiving surface 60 of the image sensor 5 decreases compared to the exposure period EX, thereby reducing the amount of leakage charge and improving the quality of the image data obtained during storage imaging.

[0114] (Example of ND filter control during signal readout period) If the system control unit 11 determines that the subject being imaged does not contain high-luminance areas, it maintains the light transmittance of the ND filter during the signal readout period immediately following the exposure period EX at the value set during the exposure period EX. If the system control unit 11 determines that the subject being imaged contains high-luminance areas, it sets the light transmittance of the ND filter during the signal readout period immediately following the exposure period EX to a value lower than the value set during the exposure period EX.

[0115] As a result, the amount of incident light on the light-receiving surface 60 of the image sensor 5 decreases during the signal readout period, thereby reducing the amount of leakage charge and improving the quality of the image data obtained during storage imaging.

[0116] (Example of focusing lens control during signal readout period) If the system control unit 11 determines that the subject being imaged does not contain a high-brightness region, it sets the position of the principal point of the focus lens to unfixed during the signal readout period immediately following the exposure period EX. In other words, the system control unit 11 controls the position of the principal point of the focus lens in preparation for the start of the next exposure.

[0117] If the system control unit 11 determines that the subject being imaged contains a high-brightness region, it maintains the position of the principal point of the focus lens in the signal readout period immediately following the exposure period EX as it was during the exposure period EX. In other words, if the system control unit 11 determines that the subject being imaged contains a high-brightness region, it stops driving the focus lens during the signal readout period.

[0118] In this way, the imaging range of the high-brightness region imaged on the light-receiving surface 60 can be fixed to the same size as during the exposure period EX during the signal readout period. In other words, it is possible to prevent the imaging range of this high-brightness region from becoming larger during the signal readout period. By keeping the imaging range from becoming larger, the number of pixels 61 into which a large amount of leakage charge is introduced can be reduced, thereby improving the quality of the captured image data.

[0119] (Example of controlling the signal readout speed during the signal readout period) If the system control unit 11 determines that the subject being imaged does not contain high-brightness areas, it sets the signal readout speed during the signal readout period immediately following the exposure period EX to the default first speed. If it determines that the subject being imaged contains high-brightness areas, it sets the signal readout speed during the signal readout period immediately following the exposure period EX to a second speed that is faster than the first speed.

[0120] This shortens the signal readout period and reduces the amount of leakage charge, thereby improving the quality of image data obtained during storage imaging. The signal readout speed can be changed by changing the number of conversion bits set in the AD (Analog to Digital) converter included in the signal processing circuit 64, or by changing the clock frequency of the AD converter.

[0121] (Example of controlling the signal readout order during the signal readout period) If the system control unit 11 determines that the subject being imaged does not contain high-brightness regions, it sets the signal readout order in the signal readout period immediately following the exposure period EX to a predetermined order (for example, an order of reading from the top to the bottom). If the system control unit 11 determines that the subject being imaged contains high-brightness regions, it determines the signal readout order in the signal readout period immediately following the exposure period EX based on the position of the pixel 61 in which the high-brightness region is imaged.

[0122] For example, if the system control unit 11 determines that a high-brightness area is imaged at the lower end of the light-receiving surface 60, it sets the system control unit 11 to sequentially read signals from the pixel row 62 at the lower end of the light-receiving surface 60 toward the upper end. Also, if the system control unit 11 determines that a high-brightness area is imaged in the central part of the light-receiving surface 60, it sets the system control unit 11 to sequentially read signals from the pixel row 62 in the central part of the light-receiving surface 60 toward the lower end, and after reading the signal from the pixel row 62 at the lower end, it sets the system control unit 11 to sequentially read signals from the pixel row 62 at the upper end of the light-receiving surface 60 toward the pixel row 62 in the central part.

[0123] In this way, the signal from pixel 61, which may have a large amount of leakage charge, is read out first, thus shortening the time that the charge is held in the charge holding section 61B of that pixel 61. As a result, the amount of leakage charge caused by the presence of high-brightness regions can be reduced, and the quality of the image data obtained in imaging for storage can be improved.

[0124] In the above description, the system control unit 11 detects high-brightness regions of the subject being imaged based on the image data obtained from LV imaging during exposure period LV2. As a variation of this, the system control unit 11 may also detect high-brightness regions based on the image data obtained from each of the multiple LV imaging sessions performed before exposure period EX.

[0125] For example, the system control unit 11 detects high-brightness regions based on the image data obtained from LV imaging during exposure period LV1, and detects high-brightness regions based on the image data obtained from LV imaging during exposure period LV2. If the system control unit 11 determines that the subject contains high-brightness regions based on the image data obtained from LV imaging during exposure period LV1, and also determines that the subject contains high-brightness regions based on the image data obtained from LV imaging during exposure period LV2, it determines whether the high-brightness regions of the subject have moved between the LV imaging during exposure period LV1 and the LV imaging during exposure period LV2.

[0126] Specifically, the system control unit 11 compares the location of the region where the pixel value is saturated in the image data obtained from LV imaging during exposure period LV1 with the location of the region where the pixel value is saturated in the image data obtained from LV imaging during exposure period LV2. The system control unit 11 determines that the high-luminance region of the subject has moved if these locations are separated by a distance threshold, and determines that the high-luminance region of the subject has not moved if these locations are not separated by a distance threshold.

[0127] If the system control unit 11 determines that the high-luminance region of the subject is moving, it determines that there is a possibility of leakage charge occurring in the signal readout period immediately after the exposure period EX that could affect the image quality of the captured image data. If the system control unit 11 determines that the high-luminance region of the subject is not moving, it determines that there is no possibility of leakage charge occurring in the signal readout period immediately after the exposure period EX that could affect the image quality of the captured image data.

[0128] The fact that the position of the high-brightness region during imaging in exposure period LV1 does not change from the position of the high-brightness region during imaging in exposure period LV2 suggests that the position of the high-brightness region during imaging in exposure period EX is also likely to remain unchanged. Furthermore, in exposure period EX, where a higher exposure value is set than during LV imaging, the pixel value of the pixel 61 that images this high-brightness region will reach its saturation value. Therefore, even if leakage charge is added to the pixel 61 that images the high-brightness region during the signal readout period after exposure period EX, the pixel value of that pixel 61 is already saturated, so there will be little effect on image quality.

[0129] Therefore, if the position of the high-brightness region does not change between multiple imaging sessions performed before the exposure period EX, the image quality can be avoided by maintaining the same F-number of aperture 2, transmittance of the ND filter, principal point position of the focus lens, and signal readout operation during the signal readout period as they were during the exposure period EX.

[0130] On the other hand, if the position of the high-brightness region changes between multiple imaging sessions performed before the exposure period EX, there is a high probability that the high-brightness region will shift between the exposure period EX and the subsequent signal readout period. Therefore, the impact of leakage charge on image quality during the signal readout period will be greater than when the high-brightness region does not shift.

[0131] Therefore, if the position of the high-brightness region changes between multiple imaging cycles performed before the exposure period EX, the image quality of the image data obtained from the storage imaging can be improved by changing at least one of the following during the signal readout period: the F-number of aperture 2, the transmittance of the ND filter, the principal point position of the focus lens, and the signal readout operation, from the exposure period EX, to reduce the amount of light incident on the light-receiving surface 60.

[0132] In the explanation so far, LV imaging for acquiring imaging image data to detect high-brightness regions has been performed with the same exposure for all pixels 61. However, for example, the exposure may be different for the first row of pixels which includes the pixels for phase difference detection and the second row of pixels which includes only normal pixels.

[0133] Specifically, during LV imaging with exposure period LV2, the system control unit 11 controls the sequential reading of signals from each group when multiple pixels 61 are divided into multiple groups (for example, two groups: a first group containing only the first row of pixels and a second group containing the second row of pixels), thereby acquiring image data composed of image data corresponding to each group.

[0134] The system control unit 11 causes the first group, which includes only the first row of pixels, to take images with a first exposure value (for example, the same value as the exposure value during exposure period EX), and the second group, which includes the second row of pixels, to take images with a second exposure value lower than the first exposure value. The exposure value is changed by changing the exposure time or the imaging sensitivity. The system control unit 11 then controls the display of the live view image based on the image data corresponding to the first group, acquires the luminance distribution described above based on the image data corresponding to the second group, and detects high-luminance regions based on that luminance distribution.

[0135] By doing this, the live view image will be an image captured with the user-defined exposure, thus improving the quality of the live view image. The signal read from the second pixel row is mainly used for focus control, so lowering the exposure value does not affect the live view image. Therefore, when capturing with exposure period LV2, the exposure value of the second pixel row is lowered to detect high-brightness areas, thereby improving the image quality of both the live view image and the image data obtained from the storage image.

[0136] Next, we will describe the configuration of a smartphone, which is another embodiment of the imaging device of the present invention.

[0137] Figure 6 shows the external appearance of the smartphone 200. The smartphone 200 shown in Figure 6 has a flat casing 201, and one side of the casing 201 is equipped with a display input unit 204 which is an integrated display panel 202 as a display unit and an operation panel 203 as an input unit.

[0138] Furthermore, such a housing 201 includes a speaker 205, a microphone 206, an operating unit 207, and a camera unit 208. However, the configuration of the housing 201 is not limited to this; for example, a configuration in which the display unit and input unit are independent, or a configuration having a folding structure or a sliding mechanism, can also be adopted.

[0139] Figure 7 is a block diagram showing the configuration of the smartphone 200 shown in Figure 6.

[0140] As shown in Figure 7, the main components of the smartphone include a wireless communication unit 210, a display input unit 204, a call unit 211, an operation unit 207, a camera unit 208, a storage unit 212, an external input / output unit 213, a GNSS (Global Navigation Satellite System) receiver unit 214, a motion sensor unit 215, a power supply unit 216, and a main control unit 220.

[0141] Furthermore, the main function of the smartphone 200 is to provide a wireless communication function that performs mobile wireless communication via a base station device BS (not shown) and a mobile communication network NW (not shown).

[0142] The wireless communication unit 210 performs wireless communication with base station equipment BS connected to the mobile communication network NW, in accordance with instructions from the main control unit 220. Using this wireless communication, it sends and receives various file data such as voice data and image data, email data, etc., and receives web data or streaming data, etc.

[0143] The display input unit 204 is a so-called touch panel that, under the control of the main control unit 220, displays images (still images and moving images) or text information to visually convey information to the user and detects user operations on the displayed information, and comprises a display panel 202 and an operation panel 203.

[0144] The display panel 202 uses LCD (Liquid Crystal Display), OELD (Organic Electro-Luminescence Display), etc., as display devices.

[0145] The operation panel 203 is a device that is visibly mounted on the display surface of the display panel 202 and detects one or more coordinates operated by the user's finger or stylus. When this device is operated by the user's finger or stylus, it outputs a detection signal generated by the operation to the main control unit 220. The main control unit 220 then detects the operation position (coordinates) on the display panel 202 based on the received detection signal.

[0146] As shown in Figure 7, the display panel 202 and operation panel 203 of the smartphone 200, which is illustrated as one embodiment of the imaging device of the present invention, together constitute a display input unit 204, with the operation panel 203 positioned to completely cover the display panel 202.

[0147] When such an arrangement is adopted, the operation panel 203 may also be equipped with a function to detect user operations in areas outside the display panel 202. In other words, the operation panel 203 may be equipped with a detection area for the overlapping portion that overlaps with the display panel 202 (hereinafter referred to as the display area) and a detection area for the outer edge portion that does not overlap with the display panel 202 (hereinafter referred to as the non-display area).

[0148] The size of the display area and the size of the display panel 202 may be made to match perfectly, but it is not necessary for them to match. Furthermore, the operation panel 203 may have two sensitive areas: an outer edge portion and an inner portion. The width of the outer edge portion is designed appropriately according to the size of the housing 201, etc.

[0149] Furthermore, the position detection methods used in the control panel 203 include matrix switch methods, resistive film methods, surface acoustic wave methods, infrared methods, electromagnetic induction methods, and capacitive methods, and any of these methods can be adopted.

[0150] The communication unit 211 is equipped with a speaker 205 or a microphone 206, and converts the user's voice input through the microphone 206 into audio data that can be processed by the main control unit 220 and outputs it to the main control unit 220, or decodes audio data received by the wireless communication unit 210 or the external input / output unit 213 and outputs it from the speaker 205.

[0151] Furthermore, as shown in Figure 6, for example, the speaker 205 can be mounted on the same side as the display input unit 204, and the microphone 206 can be mounted on the side of the housing 201.

[0152] The operation unit 207 is a hardware key using a key switch or the like, which receives instructions from the user. For example, as shown in Figure 6, the operation unit 207 is mounted on the side of the casing 201 of the smartphone 200 and is a push-button type switch that turns on when pressed with a finger or the like, and turns off when the finger is released due to a restoring force such as a spring.

[0153] The memory unit 212 stores the control program and control data of the main control unit 220, application software, address data associated with the name or telephone number of the communication partner, sent and received email data, web data downloaded by web browsing, downloaded content data, and also temporarily stores streaming data. The memory unit 212 is composed of an internal memory unit 217 built into the smartphone and an external memory unit 218 with a removable external memory slot.

[0154] The internal storage units 217 and external storage units 218 that constitute the storage unit 212 are implemented using storage media such as flash memory type, hard disk type, multimedia card micro type, card type memory (for example, MicroSD® memory), RAM (Random Access Memory), and ROM (Read Only Memory).

[0155] The external input / output unit 213 serves as an interface for all external devices connected to the smartphone 200, and is intended for direct or indirect connection to other external devices via communication (e.g., Universal Serial Bus (USB), IEEE 1394, Bluetooth (registered trademark), RFID (Radio Frequency Identification), Infrared Data Association (IrDA) (registered trademark), UWB (Ultra Wideband) (registered trademark), ZigBee (registered trademark), etc.) or network (e.g., Ethernet (registered trademark), Wireless LAN (Local Area Network), etc.).

[0156] External devices that can be connected to the Smartphone 200 include, for example, wired / wireless headsets, wired / wireless external chargers, wired / wireless data ports, memory cards connected via card sockets, SIM (Subscriber Identity Module Card) / UIM (User Identity Module Card) cards, external audio / video equipment connected via audio / video I / O (Input / Output) terminals, wirelessly connected external audio / video equipment, wired / wireless connected smartphones, wired / wireless connected personal computers, wired / wireless connected personal computers, earphones, etc.

[0157] The external input / output unit 213 can transmit data received from such external devices to the various internal components of the smartphone 200, or enable data from inside the smartphone 200 to be transmitted to external devices.

[0158] The GNSS receiver 214 receives GNSS signals transmitted from GNSS satellites ST1 to STn in accordance with instructions from the main control unit 220, performs positioning calculation processing based on the received GNSS signals, and detects the position of the smartphone 200, consisting of its latitude, longitude, and altitude. When the GNSS receiver 214 can obtain position information from the wireless communication unit 210 or the external input / output unit 213 (for example, wireless LAN), it can also use that position information to detect the position.

[0159] The motion sensor unit 215 includes, for example, a 3-axis acceleration sensor, and detects the physical movement of the smartphone 200 according to the instructions of the main control unit 220. By detecting the physical movement of the smartphone 200, the direction of movement or acceleration of the smartphone 200 is detected. The detection results are output to the main control unit 220.

[0160] The power supply unit 216 supplies power stored in a battery (not shown) to each part of the smartphone 200 according to the instructions of the main control unit 220.

[0161] The main control unit 220 is equipped with a microprocessor and operates according to the control program and control data stored in the memory unit 212, and comprehensively controls each part of the smartphone 200. The microprocessor of the main control unit 220 has the same functions as the system control unit 11. In addition, the main control unit 220 is equipped with a mobile communication control function that controls each part of the communication system for voice communication or data communication via the wireless communication unit 210, and an application processing function.

[0162] The application processing function is realized by the operation of the main control unit 220 according to the application software stored in the memory unit 212. Examples of application processing functions include an infrared communication function that controls the external input / output unit 213 to communicate data with a counterpart device, an email function that sends and receives emails, and a web browsing function that displays web pages.

[0163] Furthermore, the main control unit 220 is equipped with image processing functions, such as displaying video on the display input unit 204 based on image data (still image or moving image data) such as received data or downloaded streaming data.

[0164] The image processing function refers to the function in which the main control unit 220 decodes the above image data, applies image processing to the decoded result, and displays the image on the display input unit 204.

[0165] Furthermore, the main control unit 220 performs display control for the display panel 202 and operation detection control to detect user operations through the operation unit 207 and the operation panel 203.

[0166] By executing display control, the main control unit 220 displays software keys such as icons or scroll bars for launching application software, or displays a window for composing an email.

[0167] A scroll bar is a software key that accepts commands to move the display portion of an image, such as a large image that does not fit within the display area of ​​the display panel 202.

[0168] Furthermore, by executing operation detection control, the main control unit 220 detects user operations through the operation unit 207, accepts operations on the icons and input of strings into the input fields of the windows through the operation panel 203, or accepts requests to scroll the displayed image through the scroll bar.

[0169] Furthermore, by executing operation detection control, the main control unit 220 determines whether the operation position on the operation panel 203 is in the overlapping portion (display area) that overlaps with the display panel 202 or in the outer edge portion (non-display area) that does not overlap with the display panel 202, and has a touch panel control function that controls the display position of the sensitive area of ​​the operation panel 203 or the software key.

[0170] Furthermore, the main control unit 220 can detect gesture operations on the operation panel 203 and execute pre-set functions in response to the detected gesture operations.

[0171] Gesture control refers to operations that differ from traditional simple touch operations, such as drawing a path with a finger or other object, specifying multiple locations simultaneously, or combining these to draw a path from at least one of multiple locations.

[0172] The camera unit 208 includes the imaging unit 50 and the digital signal processing unit 17 shown in Figure 1. In the smartphone 200, the imaging control device is composed of the main control unit 220 and the internal storage unit 217.

[0173] The image data generated by the camera unit 208 can be stored in the storage unit 212 or output via the external input / output unit 213 or the wireless communication unit 210.

[0174] In the smartphone 200 shown in Figure 7, the camera unit 208 is mounted on the same side as the display input unit 204, but the mounting position of the camera unit 208 is not limited to this, and it may also be mounted on the back of the display input unit 204.

[0175] Furthermore, the camera unit 208 can be used for various functions of the smartphone 200. For example, images acquired by the camera unit 208 can be displayed on the display panel 202, or images from the camera unit 208 can be used as one of the inputs for the operation panel 203.

[0176] Furthermore, when the GNSS receiver 214 detects a position, it can also detect the position by referring to the image from the camera unit 208. Moreover, by referring to the image from the camera unit 208, it is possible to determine the optical axis direction of the camera unit 208 of the smartphone 200, or to determine the current usage environment, either without using the 3-axis accelerometer or in combination with the 3-axis accelerometer. Of course, the image from the camera unit 208 can also be used within the application software.

[0177] In addition, still image or video image data can be supplemented with location information acquired by the GNSS receiver 214, audio information acquired by the microphone 206 (which may be converted to text by the main control unit, etc.), posture information acquired by the motion sensor 215, etc., and stored in the storage unit 212 or output via the external input / output unit 213 or wireless communication unit 210. Even with a smartphone 200 configured as described above, the quality of image data obtained from imaging for storage can be improved. [Explanation of symbols]

[0178] 1 imaging lens 2 apertures 4. Lens control unit 5 Image sensor 8. Lens drive unit 9. Aperture drive unit 11 System Control Unit 14,207 Operation section 15 Memory Control Unit 16 memory 17 Digital signal processing unit 20 External memory control unit 21 Storage medium 22a Display Controller 22b Display surface 22 Display device 24 control bus 25 Data Bus 40 Lens device 50 Imaging Unit 60 Photosensitive area 61A Photoelectric conversion unit 61B Charge holding section 61C Charge Transfer Section 61D Floating Diffusion 61E Circuit 61 pixels 62 pixel rows 63 Drive Circuit 64 Signal Processing Circuits 65 signal line 70 N-type substrate 71 P-well layer 72 electrode 73 N-type impurity layer 74 P-type impurity layer 75 areas 76 Transfer electrodes 77 Reset Transistor 78 Output transistors 79 Selective Transistors 100A Main Unit 100 Digital Cameras 200 Smartphones 201 cabinet 202 Display Panel 203 Control Panel 204 Display Input Section 205 Speakers 206 Microphone 208 Camera Department 210 Wireless Communication Section 211 Telephone section 212 Storage section 213 External input / output section 214 GNSS receiver 215 Motion sensor unit 216 Power supply section 217 Internal storage 218 External storage unit 220 Main Control Unit

Claims

1. An imaging control device for controlling an imaging unit comprising an imaging sensor having a plurality of pixels arranged in a two-dimensional manner, each including a photoelectric conversion unit and a charge holding unit that holds the charge transferred from the photoelectric conversion unit, Equipped with a processor, The aforementioned processor, A first control is performed to start exposure of the plurality of pixels and transfer the charge accumulated in the photoelectric conversion unit of the plurality of pixels by the exposure to the charge holding unit. A second control is performed to read a signal corresponding to the charge transferred to the charge holding unit by the first control. In a second period that does not overlap with the first period during which the first control is performed, the image sensor is used to image a subject and image data is acquired. Based on the image data, the brightness distribution of the subject imaged by the image sensor is acquired, and a third control is performed to control the operating conditions of the imaging unit during the period during which the second control is performed, based on the brightness distribution. In the imaging during the second period, control is performed to read signals from groups when the plurality of pixels are divided into a plurality of groups, and the image imaging data composed of image data corresponding to the group is obtained. An imaging control device that causes a first group, which is part of the aforementioned plurality of groups, to take images at a first exposure value, and causes a second group, which is part of the plurality of groups excluding the first group, to take images at a second exposure value lower than the first exposure value.

2. The imaging control device according to claim 1, The aforementioned processor, Based on the image data corresponding to the first group, the display control of the live view image is performed. An imaging control device that acquires the luminance distribution based on the image data corresponding to the second group.

3. The imaging control device according to claim 1, The aforementioned processor, Based on the luminance distribution, a high-luminance region in the subject being imaged where the luminance is above a threshold is detected. During the second period, multiple imaging operations are performed to acquire multiple image data images. An imaging control device that, when it is determined that the high-brightness region detected based on the plurality of imaging image data has moved between the plurality of imaging cycles, changes the operating condition from the first condition to the second condition.

4. The imaging control device according to claim 1, The imaging unit includes a first optical element capable of changing the amount of light incident on the image sensor. The aforementioned operating conditions are conditions relating to the operation of the first optical element, in an imaging control device.

5. The imaging control device according to claim 4, The third control includes an imaging control device that controls the first optical element to reduce the amount of incident light during the period in which the second control is performed to less than the amount of incident light to the image sensor during the period in which the first control is performed.

6. The imaging control device according to claim 1, The imaging unit includes a focusing lens, The aforementioned operating conditions are conditions relating to the operation of the focus lens, in an imaging control device.

7. The imaging control device according to claim 6, The third control includes an image capture control device that stops the driving of the focus lens.

8. The imaging control device according to claim 1, The aforementioned operating conditions are conditions relating to the readout operation of signals from the plurality of charge holding units, in an imaging control device.

9. The imaging control device according to claim 8, The third control includes a control that sets the readout speed of the signal from the charge holding unit to a first speed, and a control that sets the readout speed to a second speed that is faster than the first speed, in an imaging control device.

10. The imaging control device according to claim 8, The aforementioned operating conditions are conditions relating to the order in which signals are read from the plurality of charge holding units, in an imaging control device.

11. The imaging control device according to claim 10, The processor detects high-luminance regions based on the luminance distribution, and if it determines that the high-luminance regions exist, it determines the signal readout order based on the position of the pixel where the high-luminance regions are imaged, thereby providing an imaging control device.

12. An imaging control device for controlling an imaging unit comprising an imaging sensor having a plurality of pixels arranged in a two-dimensional manner, each including a photoelectric conversion unit and a charge holding unit that holds the charge transferred from the photoelectric conversion unit, Equipped with a processor, The aforementioned processor, A first imaging control is performed to cause the image sensor to image the subject at a first exposure value. A second imaging control is performed to cause the image sensor to image the subject at a second exposure value lower than the first exposure value. An imaging control device that, based on the imaging image data acquired from the image sensor by the second imaging control, acquires the luminance distribution of the subject being imaged by the image sensor, and performs specific control based on the luminance distribution.

13. The imaging control device according to claim 12, The processor is an imaging control device that detects high-luminance regions where the luminance is above a threshold based on the luminance distribution, and performs the specific control described above when such high-luminance regions exist.

14. The imaging control device according to claim 13, The aforementioned processor, The second imaging control is performed multiple times to acquire multiple images of the captured image data. An imaging control device that performs the specific control described above when it is determined that the high-brightness region detected based on the plurality of imaging image data has moved between the plurality of imaging cycles.

15. An imaging control device for controlling an imaging unit comprising an imaging sensor having a plurality of pixels arranged in a two-dimensional manner, each including a photoelectric conversion unit and a charge holding unit that holds the charge transferred from the photoelectric conversion unit, Equipped with a processor, The aforementioned processor, The image sensor is used to perform imaging control to capture an image of the subject. In the imaging control described above, when the plurality of pixels are divided into a plurality of groups, control is performed to read signals from the groups, and imaging image data consisting of image data corresponding to the groups is obtained. For the first group, which is part of the aforementioned plurality of groups, imaging is performed at a first exposure value, and for the second group, which is part of the plurality of groups excluding the first group, imaging is performed at a second exposure value lower than the first exposure value. An imaging control device that acquires the luminance distribution of a subject captured by the image sensor based on the captured image data, and performs specific control based on the luminance distribution.

16. The imaging control device according to claim 15, The aforementioned processor, Based on the image data corresponding to the first group, the display control of the live view image is performed. An imaging control device that acquires the brightness distribution based on the image data corresponding to the second group.

17. An imaging control device according to any one of claims 1 to 16, An imaging device comprising the aforementioned imaging unit.

18. An imaging control method for controlling an imaging unit comprising an imaging sensor having a plurality of pixels arranged in a two-dimensional manner, each including a photoelectric conversion unit and a charge holding unit that holds the charge transferred from the photoelectric conversion unit, A first control is performed to start exposure of the plurality of pixels and transfer the charge accumulated in the photoelectric conversion unit of the plurality of pixels by the exposure to the charge holding unit. A second control is performed to read a signal corresponding to the charge transferred to the charge holding unit by the first control. In a second period that does not overlap with the first period during which the first control is performed, the image sensor is used to image a subject and image data is acquired. Based on the image data, the brightness distribution of the subject imaged by the image sensor is acquired, and a third control is performed to control the operating conditions of the imaging unit during the period during which the second control is performed, based on the brightness distribution. In the imaging during the second period, control is performed to read signals from groups when the plurality of pixels are divided into a plurality of groups, and the image imaging data composed of image data corresponding to the group is obtained. An imaging control method comprising: causing a first group, which is a part of the aforementioned plurality of groups, to perform imaging at a first exposure value; and causing a second group, which is the plurality of groups excluding the first group, to perform imaging at a second exposure value lower than the first exposure value.

19. An imaging control program for controlling an imaging unit comprising an imaging sensor having a plurality of pixels arranged in a two-dimensional manner, each including a photoelectric conversion unit and a charge holding unit that holds the charge transferred from the photoelectric conversion unit, A first control is performed to start exposure of the plurality of pixels and transfer the charge accumulated in the photoelectric conversion unit of the plurality of pixels by the exposure to the charge holding unit. A second control is performed to read a signal corresponding to the charge transferred to the charge holding unit by the first control. In a second period that does not overlap with the first period during which the first control is performed, the image sensor is used to image a subject and image data is acquired. Based on the image data, the brightness distribution of the subject imaged by the image sensor is acquired, and a third control is performed to control the operating conditions of the imaging unit during the period during which the second control is performed, based on the brightness distribution. In the imaging during the second period, control is performed to read signals from groups when the plurality of pixels are divided into a plurality of groups, and the image imaging data composed of image data corresponding to the group is obtained. An imaging control program that causes the processor to execute a step of performing imaging at a first exposure value for a first group which is part of the aforementioned plurality of groups, and performing imaging at a second exposure value which is lower than the first exposure value for a second group which is part of the plurality of groups excluding the first group.

Images