Imaging device

Abstract

To suppress image discontinuities between blocks caused by, for example, differences in imaging conditions for each block.SOLUTION: An imaging device includes an imaging element for which imaging conditions can be set for each block containing a plurality of pixels, and an imaging condition setting unit that sets a first imaging condition for the block in a first region where first light from a first subject enters, sets a second imaging condition for the block in a second region where second light from a second subject enters, and sets imaging conditions different from the first and second imaging conditions for the block in a third region between the first and second regions. The imaging condition setting unit sets the number of imaging conditions to be set in the third region based on a set value set based on the first imaging condition and a set value set based on the second imaging condition.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an imaging device. [Background technology]

[0002] BACKGROUND ART Imaging devices equipped with image processing technology that generates an image from a signal from an imaging element are known (see Patent Document 1). There has been a demand for improved image quality. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2006-197192 Summary of the Invention

[0004] According to a first aspect, an imaging device includes an imaging element capable of setting imaging conditions for each block including a plurality of pixels, and an imaging condition setting unit that sets a first imaging condition for the block in a first region where a first light from a first subject is incident, sets a second imaging condition for the block in a second region where a second light from a second subject is incident, and sets imaging conditions different from the first imaging condition and the second imaging condition for the block in a third region between the first region and the second region, and the imaging condition setting unit sets the number of imaging conditions to be set in the third region based on a setting value set by the first imaging condition and a setting value set by the second imaging condition. [Brief explanation of the drawings]

[0005] [Figure 1] 1 is a block diagram illustrating the configuration of a camera according to a first embodiment. [Figure 2] FIG. 1 is a cross-sectional view of a stacked imaging element. [Figure 3] 1 is a diagram illustrating a pixel array and a unit area of ​​an imaging chip. FIG. [Figure 4] FIG. 2 is a diagram illustrating a circuit in a unit area. [Figure 5] FIG. 2 is a diagram schematically illustrating an image of a subject formed on an imaging element of a camera. [Figure 6] FIG. 10 is a diagram illustrating an example of a setting screen for imaging conditions. [Figure 7] FIG. 7(a) is a diagram illustrating a predetermined range in a live view image, and FIG. 7(b) is an enlarged view of the predetermined range. [Figure 8] FIG. 8 is a diagram illustrating image data corresponding to FIG. 7(b). [Figure 9] FIG. 9(a) is a diagram illustrating an example of a region of interest in a live view image, and FIG. 9(b) is an enlarged view of a pixel of interest and a reference pixel Pr. [Figure 10] Figure 10(a) is a diagram illustrating an example of the arrangement of photoelectric conversion signals output from pixels, Figure 10(b) is a diagram explaining the interpolation of image data of the G color component, and Figure 10(c) is a diagram illustrating image data of the G color component after interpolation. [Figure 11] FIG. 11(a) is a diagram showing image data of the R color component extracted from FIG. 10(a), FIG. 11(b) is a diagram explaining the interpolation of the color difference component Cr, and FIG. 11(c) is a diagram explaining the interpolation of image data of the color difference component Cr. [Figure 12] FIG. 12(a) is a diagram showing image data of the B color component extracted from FIG. 10(a), FIG. 12(b) is a diagram explaining the interpolation of the color difference component Cb, and FIG. 12(c) is a diagram explaining the interpolation of image data of the color difference component Cb. [Figure 13] FIG. 2 is a diagram illustrating an example of the positions of focus detection pixels on an imaging surface. [Figure 14] FIG. 2 is an enlarged view of a partial area of ​​a focus detection pixel line. [Figure 15] FIG. 10 is an enlarged view of a focus point. [Figure 16] FIG. 16(a) is a diagram illustrating a template image representing an object to be detected, and FIG. 16(b) is a diagram illustrating a live view image and a search range. [Figure 17] 10 is a flowchart illustrating a process flow for setting imaging conditions for each region and capturing an image. [Figure 18]18(a) to 18(c) are diagrams showing a plurality of blocks according to the second embodiment, including a part of the predetermined range in FIG. 7(b). [Figure 19] 19(a) to 19(c) are diagrams illustrating examples of the arrangement of the first imaging region and the second imaging region on the imaging surface of the imaging element. [Figure 20] FIG. 20 is a block diagram illustrating the configuration of an imaging system according to an eleventh modification. [Figure 21] FIG. 10 is a diagram illustrating the supply of a program to a mobile device. [Figure 22] FIG. 10 is a block diagram illustrating the configuration of a camera according to a third embodiment. [Figure 23] FIG. 13 is a diagram schematically illustrating the correspondence between each block and a plurality of correction units in the third embodiment. [Figure 24] FIG. 2 is a cross-sectional view of a stacked imaging element. [Figure 25] 3A and 3B are diagrams schematically showing processing of first image data and second image data in relation to image processing. [Figure 26] 5A and 5B are diagrams schematically showing processing of first image data and second image data in relation to focus detection processing. [Figure 27] 5A and 5B are diagrams schematically showing processing of first image data and second image data in subject detection processing. [Figure 28] 10 is a diagram schematically showing processing of first image data and second image data in relation to setting of imaging conditions such as exposure calculation processing. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0006] ---First embodiment--- A digital camera will be described as an example of an electronic device incorporating an image processing device according to the first embodiment. Camera 1 (FIG. 1) is configured to be able to capture images under different conditions for each region of the imaging surface of image sensor 32a. Image processing unit 33 performs appropriate processing for each region with different imaging conditions. Details of camera 1 will be described with reference to the drawings.

[0007] <Camera description> 1 is a block diagram illustrating the configuration of a camera 1 according to the first embodiment. In FIG. 1, camera 1 has an imaging optical system 31, an imaging unit 32, an image processing unit 33, a control unit 34, a display unit 35, an operation member 36, and a recording unit 37.

[0008] The imaging optical system 31 guides a light beam from the subject field to the imaging unit 32. The imaging unit 32 includes an imaging element 32a and a driving unit 32b, and photoelectrically converts the image of the subject formed by the imaging optical system 31. The imaging unit 32 can capture images under the same conditions across the entire imaging surface of the imaging element 32a, or can capture images under different conditions for different regions of the imaging surface of the imaging element 32a. Details of the imaging unit 32 will be described later. The driving unit 32b generates a driving signal required for the imaging element 32a to perform accumulation control. Imaging instructions for the imaging unit 32, such as the charge accumulation time, are sent from the control unit 34 to the driving unit 32b.

[0009] The image processing unit 33 includes an input unit 33a, a correction unit 33b, and a generation unit 33c. The input unit 33a receives image data acquired by the imaging unit 32. The correction unit 33b performs pre-processing to correct the input image data. Details of the pre-processing will be described later. The generation unit 33c performs image processing on the input image data and the pre-processed image data to generate an image. The image processing includes, for example, color interpolation processing, pixel defect correction processing, edge enhancement processing, noise reduction processing, white balance adjustment processing, gamma correction processing, display brightness adjustment processing, saturation adjustment processing, etc. Furthermore, the generation unit 33c generates an image to be displayed on the display unit 35.

[0010] The control unit 34 is configured by, for example, a CPU, and controls the overall operation of the camera 1. For example, the control unit 34 performs a predetermined exposure calculation based on the photoelectric conversion signal acquired by the imaging unit 32, and determines exposure conditions such as the charge accumulation time (exposure time) of the image sensor 32a required for proper exposure, the aperture value of the imaging optical system 31, and ISO sensitivity, and instructs the driving unit 32b. The control unit 34 also determines image processing conditions for adjusting saturation, contrast, sharpness, etc., depending on the imaging scene mode set in the camera 1 and the type of subject element detected, and instructs the image processing unit 33. Detection of subject elements will be described later.

[0011] The control unit 34 includes an object detection unit 34a, a setting unit 34b, an imaging control unit 34c, and a lens movement control unit 34d. These are realized in software by the control unit 34 executing a program stored in a non-volatile memory (not shown), but they may also be configured using an ASIC or the like.

[0012] The object detection unit 34a performs known object recognition processing to detect subject elements such as people (human faces), animals (animal faces) such as dogs and cats, plants, vehicles such as bicycles, automobiles, and trains, buildings, stationary objects, landscapes such as mountains and clouds, and predetermined specific objects from the image data acquired by the imaging unit 32. The setting unit 34b divides the image data acquired by the imaging unit 32 into multiple regions including the subject elements detected as described above.

[0013] The setting unit 34b further sets imaging conditions for the multiple regions. The imaging conditions include the exposure conditions (charge accumulation time, gain, ISO sensitivity, frame rate, etc.) and the image processing conditions (for example, white balance adjustment parameters, gamma correction curve, display brightness adjustment parameters, saturation adjustment parameters, etc.). It is possible to set the same imaging conditions for all of the multiple regions, or to set different imaging conditions for each of the multiple regions.

[0014] The imaging control unit 34c applies the imaging conditions set for each region by the setting unit 34b to control the imaging unit 32 (image sensor 32a) and image processing unit 33. This makes it possible for the imaging unit 32 to capture images under different exposure conditions for each of the multiple regions, and for the image processing unit 33 to perform image processing under different image processing conditions for each of the multiple regions. The number of pixels constituting each region may be any number, and may be, for example, 1000 pixels or 1 pixel. Furthermore, the number of pixels may differ between regions.

[0015] The lens movement control unit 34d controls an automatic focus adjustment (AF) operation that focuses on a corresponding subject at a predetermined position (called a focus point) on the imaging screen. Focusing increases the sharpness of the subject image. That is, the image captured by the imaging optical system 31 is adjusted by moving the focus lens of the imaging optical system 31 in the optical axis direction. Based on the calculation result, the lens movement control unit 34d sends a drive signal to the lens drive mechanism 31m of the imaging optical system 31 to move the focus lens of the imaging optical system 31 to a focused position, for example, a signal to adjust the subject image with the focus lens of the imaging optical system 31. In this way, the lens movement control unit 34d functions as a moving unit that moves the focus lens of the imaging optical system 31 in the optical axis direction based on the calculation result. The process performed by the AF calculation unit 34d for AF operation is also called focus detection processing. Details of the focus detection processing will be described later.

[0016] The display unit 35 plays back and displays images generated by the image processing unit 33, images that have been subjected to image processing, and images that have been read out by the recording unit 37. The display unit 35 also displays an operation menu screen, a setting screen for setting imaging conditions, and the like.

[0017] The operation members 36 are made up of various operation members such as a release button, a menu button, etc. The operation members 36 send operation signals corresponding to each operation to the control unit 34. The operation members 36 also include a touch operation member provided on the display surface of the display unit 35.

[0018] The recording unit 37 records image data and the like on a recording medium such as a memory card (not shown) in response to an instruction from the control unit 34. The recording unit 37 also reads out image data recorded on the recording medium in response to an instruction from the control unit 34.

[0019] <Explanation of stacked image sensor> As an example of the above-described imaging element 32a, a stacked imaging element 100 will be described. FIG. 2 is a cross-sectional view of the imaging element 100. The imaging element 100 includes an imaging chip 111, a signal processing chip 112, and a memory chip 113. The imaging chip 111 is stacked on the signal processing chip 112. The signal processing chip 112 is stacked on the memory chip 113. The imaging chip 111 and the signal processing chip 112, and the signal processing chip 112 and the memory chip 113 are electrically connected by connecting portions 109. The connecting portions 109 are, for example, bumps or electrodes. The imaging chip 111 captures a light image from a subject and generates image data. The imaging chip 111 outputs the image data from the imaging chip 111 to the signal processing chip 112. The signal processing chip 112 performs signal processing on the image data output from the imaging chip 111. The memory chip 113 has multiple memories and stores the image data. The imaging element 100 may be configured with an imaging chip and a signal processing chip. When the imaging element 100 is configured with an imaging chip and a signal processing chip, a storage unit for storing image data may be provided in the signal processing chip or may be provided separately from the imaging element 100.

[0020] As shown in Figure 2, incident light is mainly incident in the positive direction of the Z axis, as indicated by the white arrow. Also, as shown by the coordinate axes, the left direction on the paper, perpendicular to the Z axis, is the positive X axis, and the direction towards you on the paper, perpendicular to the Z axis and X axis, is the positive Y axis. In the following figures, the coordinate axes in Figure 2 are used as the reference and are displayed so that the orientation of each figure can be understood.

[0021] The imaging chip 111 is, for example, a CMOS image sensor. Specifically, the imaging chip 111 is a back-illuminated CMOS image sensor. The imaging chip 111 has a microlens layer 101, a color filter layer 102, a passivation layer 103, a semiconductor layer 106, and a wiring layer 108. In the imaging chip 111, the microlens layer 101, the color filter layer 102, the passivation layer 103, the semiconductor layer 106, and the wiring layer 108 are arranged in this order along the positive direction of the Z axis.

[0022] The microlens layer 101 has a plurality of microlenses L. The microlenses L focus incident light onto the photoelectric conversion units 104 (described later). The color filter layer 102 has a plurality of color filters F. The color filter layer 102 has a plurality of types of color filters F with different spectral characteristics. Specifically, the color filter layer 102 has a first filter (R) with spectral characteristics that mainly transmits red light, second filters (Gb, Gr) with spectral characteristics that mainly transmit green light, and a third filter (B) with spectral characteristics that mainly transmit blue light. The color filter layer 102 has the first, second, and third filters arranged in a Bayer array, for example. The passivation layer 103 is made of a nitride film or an oxide film and protects the semiconductor layer 106.

[0023] The semiconductor layer 106 has a photoelectric conversion unit 104 and a readout circuit 105. The semiconductor layer 106 has a plurality of photoelectric conversion units 104 between a first surface 106a, which is a light incident surface, and a second surface 106b opposite the first surface 106a. The semiconductor layer 106 has a plurality of photoelectric conversion units 104 arranged in the X-axis direction and the Y-axis direction. The photoelectric conversion units 104 have a photoelectric conversion function of converting light into electric charges. The photoelectric conversion units 104 also accumulate electric charges resulting from photoelectric conversion signals. The photoelectric conversion units 104 are, for example, photodiodes. The semiconductor layer 106 has a readout circuit 105 closer to the second surface 106b than the photoelectric conversion units 104. The semiconductor layer 106 has a plurality of readout circuits 105 arranged in the X-axis direction and the Y-axis direction. The readout circuit 105 is configured with a plurality of transistors, and reads out image data generated from charges photoelectrically converted by the photoelectric conversion unit 104, and outputs the image data to the wiring layer .

[0024] The wiring layer 108 has a plurality of metal layers. The metal layers are, for example, Al wiring, Cu wiring, etc. The wiring layer 108 outputs image data read by the readout circuit 105. The image data is output from the wiring layer 108 to the signal processing chip 112 via a connection unit 109.

[0025] The connecting portion 109 may be provided for each photoelectric conversion portion 104. Alternatively, the connecting portion 109 may be provided for each of a plurality of photoelectric conversion portions 104. When the connecting portion 109 is provided for each of a plurality of photoelectric conversion portions 104, the pitch of the connecting portions 109 may be larger than the pitch of the photoelectric conversion portions 104. Alternatively, the connecting portions 109 may be provided in a peripheral region of the region in which the photoelectric conversion portions 104 are arranged.

[0026] The signal processing chip 112 has a plurality of signal processing circuits. The signal processing circuits perform signal processing on the image data output from the imaging chip 111. The signal processing circuits are, for example, an amplifier circuit that amplifies the signal value of the image data, a correlated double sampling circuit that performs noise reduction processing on the image data, and an analog-to-digital (A / D) conversion circuit that converts analog signals into digital signals. A signal processing circuit may be provided for each photoelectric conversion unit 104.

[0027] Furthermore, a signal processing circuit may be provided for each of the multiple photoelectric conversion units 104. The signal processing chip 112 has multiple through electrodes 110. The through electrodes 110 are, for example, through-silicon electrodes. The through electrodes 110 connect the circuits provided in the signal processing chip 112 to each other. The through electrodes 110 may also be provided in a peripheral region of the imaging chip 111 or in the memory chip 113. Note that some of the elements constituting the signal processing circuit may be provided in the imaging chip 111. For example, in the case of an analog-to-digital conversion circuit, a comparator that compares an input voltage with a reference voltage may be provided in the imaging chip 111, and circuits such as a counter circuit and a latch circuit may be provided in the signal processing chip 112.

[0028] The memory chip 113 has a plurality of storage units. The storage units store image data that has undergone signal processing by the signal processing chip 112. The storage units are, for example, volatile memories such as DRAMs. A storage unit may be provided for each photoelectric conversion unit 104. Alternatively, a storage unit may be provided for each of the plurality of photoelectric conversion units 104. The image data stored in the storage units is output to a downstream image processing unit.

[0029] FIG. 3 is a diagram illustrating the pixel array of the imaging chip 111 and the unit area 131. In particular, it shows the imaging chip 111 observed from the back side (imaging surface) side. For example, 20 million or more pixels are arranged in a matrix in the pixel area. In the example of FIG. 3, four pixels, 2 pixels by 2 pixels adjacent to each other, form one unit area 131. The grid lines in the figure show the concept of adjacent pixels being grouped to form the unit area 131. The number of pixels forming the unit area 131 is not limited to this, and may be around 1,000, for example, 32 pixels by 32 pixels, or it may be more or less than that, or even just 1 pixel.

[0030] As shown in the partially enlarged view of the pixel region, unit region 131 in Fig. 3 contains a so-called Bayer array consisting of four pixels: green pixels Gb and Gr, a blue pixel B, and a red pixel R. The green pixels Gb and Gr are pixels that have a green filter as the color filter F and receive light in the green wavelength band of incident light. Similarly, the blue pixel B is a pixel that has a blue filter as the color filter F and receives light in the blue wavelength band, and the red pixel R is a pixel that has a red filter as the color filter F and receives light in the red wavelength band.

[0031] In this embodiment, multiple blocks are defined such that each block includes at least one unit area 131. That is, the smallest unit of one block is one unit area 131. As described above, the smallest possible number of pixels forming one unit area 131 is one pixel. Therefore, when one block is defined in pixel units, the smallest possible number of pixels that can define one block is one pixel. Each block can control the pixels included in that block using different control parameters. All unit areas 131 within that block, i.e., all pixels within that block, are controlled under the same imaging conditions. In other words, photoelectric conversion signals with different imaging conditions can be acquired between a pixel group included in one block and a pixel group included in another block. Examples of control parameters include frame rate, gain, thinning rate, number of rows or columns to be added together for adding photoelectric conversion signals, charge accumulation time or number of accumulations, number of digitization bits (word length), etc. The image sensor 100 can freely perform thinning not only in the row direction (X-axis direction of the image sensor 111) but also in the column direction (Y-axis direction of the image sensor 111). Furthermore, the control parameters may be parameters for image processing.

[0032] Fig. 4 is a diagram illustrating the circuitry in a unit area 131. In the example of Fig. 4, one unit area 131 is formed by four pixels, 2 pixels x 2 pixels adjacent to each other. As mentioned above, the number of pixels included in the unit area 131 is not limited to this, and may be 1000 pixels or more, or may be as small as 1 pixel. The two-dimensional positions of the unit area 131 are indicated by symbols A to D.

[0033] The reset transistors (RST) of the pixels included in the unit region 131 are configured to be able to be turned on and off individually for each pixel. In Fig. 4, a reset wiring 300 is provided to turn on and off the reset transistor of pixel A, and a reset wiring 310 is provided to turn on and off the reset transistor of pixel B, separately from the reset wiring 300. Similarly, a reset wiring 320 is provided to turn on and off the reset transistor of pixel C, separately from the reset wirings 300 and 310. A dedicated reset wiring 330 is also provided for another pixel D to turn on and off its reset transistor.

[0034] The transfer transistors (TX) of the pixels included in the unit region 131 are also configured to be able to be turned on and off individually for each pixel. In Fig. 4, a transfer wiring 302 that turns on and off the transfer transistor of pixel A, a transfer wiring 312 that turns on and off the transfer transistor of pixel B, and a transfer wiring 322 that turns on and off the transfer transistor of pixel C are provided separately. A dedicated transfer wiring 332 is also provided for another pixel D to turn on and off its transfer transistor.

[0035] Furthermore, the selection transistors (SEL) of the pixels included in the unit region 131 are configured to be able to be turned on and off individually for each pixel. In Fig. 4, a selection wiring 306 that turns on and off the selection transistor of pixel A, a selection wiring 316 that turns on and off the selection transistor of pixel B, and a selection wiring 326 that turns on and off the selection transistor of pixel C are provided separately. A dedicated selection wiring 336 is also provided for another pixel D to turn on and off the selection transistor.

[0036] The power supply wiring 304 is commonly connected to pixels A to D included in the unit area 131. Similarly, the output wiring 308 is commonly connected to pixels A to D included in the unit area 131. Furthermore, the power supply wiring 304 is commonly connected among a plurality of unit areas, but the output wiring 308 is provided individually for each unit area 131. The load current source 309 supplies a current to the output wiring 308. The load current source 309 may be provided on the imaging chip 111 side or on the signal processing chip 112 side.

[0037] By individually turning on and off the reset transistors and transfer transistors of the unit area 131, it is possible to control charge accumulation, including the accumulation start time, accumulation end time, and transfer timing, for pixels A to D included in the unit area 131. Furthermore, by individually turning on and off the selection transistors of the unit area 131, it is possible to output the photoelectric conversion signals of each of pixels A to D via a common output wiring 308.

[0038] Here, a so-called rolling shutter method is known, which controls charge accumulation in a regular order for rows and columns for pixels A to D included in unit area 131. When pixels are selected for each row using the rolling shutter method and then columns are specified, photoelectric conversion signals are output in the order of "ABCD" in the example of Figure 4.

[0039] By configuring the circuit based on the unit area 131 in this way, it is possible to control the charge accumulation time for each unit area 131. In other words, it is possible to output photoelectric conversion signals at different frame rates between the unit areas 131. Furthermore, by causing the unit areas 131 included in some blocks of the imaging chip 111 to accumulate charge (imaging) while the unit areas 131 included in other blocks are rested, it is possible to cause imaging to be performed only in predetermined blocks of the imaging chip 111 and output the corresponding photoelectric conversion signals. Furthermore, it is also possible to switch the blocks that perform charge accumulation (imaging) between frames (blocks that are subject to accumulation control) to cause different blocks of the imaging chip 111 to sequentially perform imaging and output photoelectric conversion signals.

[0040] As described above, output wiring 308 is provided corresponding to each unit area 131. Since the imaging element 100 has the imaging chip 111, signal processing chip 112, and memory chip 113 stacked on top of each other, by using electrical connections between the chips using connection parts 109 for these output wiring 308, it is possible to route the wiring without increasing the size of each chip in the planar direction.

[0041] <Block control of image sensor> In this embodiment, imaging conditions can be set for each of a plurality of blocks in the image sensor 32a. The imaging control unit 34c of the control unit 34 associates the plurality of regions with the blocks and causes imaging to be performed under the imaging conditions set for each region.

[0042] 5 is a diagram showing a model of an image of a subject formed on the image sensor 32a of the camera 1. Before an image capture command is issued, the camera 1 photoelectrically converts the subject image to obtain a live view image. A live view image is an image for monitoring that is repeatedly captured at a predetermined frame rate (e.g., 60 fps).

[0043] Before the setting unit 34b divides the regions, the control unit 34 sets the same imaging conditions for the entire imaging chip 111 (i.e., the entire imaging screen). The same imaging conditions refer to setting common imaging conditions for the entire imaging screen, and are considered to be the same even if there is a variation of less than about 0.3 stops in the Apex value, for example. The imaging conditions that are set to be the same for the entire imaging chip 111 are determined based on exposure conditions corresponding to the photometric value of the subject brightness or exposure conditions that are manually set by the user.

[0044] 5, an image including a person 61a, a car 62a, a bag 63a, a mountain 64a, and clouds 65a and 66a is formed on the imaging surface of the imaging chip 111. The person 61a is holding the bag 63a with both hands. A car 62a is parked to the right and rear of the person 61a.

[0045] <Area division> Based on the live view image, the control unit 34 divides the screen of the live view image into multiple regions as follows: First, the object detection unit 34a detects subject elements from the live view image. The subject elements are detected using a known subject recognition technology. In the example of FIG. 5, the object detection unit 34a detects a person 61a, a car 62a, a bag 63a, a mountain 64a, a cloud 65a, and a cloud 66a as subject elements.

[0046] Next, the setting unit 34b divides the screen of the live view image into regions including the above-mentioned subject elements. In this embodiment, the region including the person 61a is referred to as a first region 61, the region including the car 62a is referred to as a second region 62, the region including the bag 63a is referred to as a third region 63, the region including the mountain 64a is referred to as a fourth region 64, the region including the cloud 65a is referred to as a fifth region 65, and the region including the cloud 66a is referred to as a sixth region 66. In the above description, control unit 34 detects subject elements and divides regions using a live view image captured by image sensor 32a. However, in the case of a camera equipped with a photometric sensor capable of capturing subject images, such as a single-lens reflex camera, control unit 34 may detect subject elements and divide regions using a live view image captured by the photometric sensor.

[0047] <Setting imaging conditions for each block> When the setting unit 34b divides the screen into a plurality of regions, the control unit 34 causes the display unit 35 to display a setting screen such as the one shown in Fig. 6. In Fig. 6, a live view image 60a is displayed, and a setting screen 70 for setting imaging conditions is displayed to the right of the live view image 60a.

[0048] The setting screen 70 lists, from top to bottom, frame rate, shutter speed (TV), and gain (ISO) as examples of setting items for imaging conditions. The frame rate is the number of frames of a live view image acquired per second or a moving image recorded by the camera 1. The gain is the ISO sensitivity. Setting items for imaging conditions other than those exemplified in FIG. 6 may be added as appropriate. If not all setting items fit within the setting screen 70, other setting items may be displayed by scrolling the setting items up and down.

[0049] In this embodiment, control unit 34 sets an area selected by the user from among the areas divided by setting unit 34b as a target for setting (changing) imaging conditions. For example, in camera 1 capable of touch operation, the user taps on the display surface of display unit 35 on which live view image 60a is displayed a display position of a main subject for which the user wishes to set (change) imaging conditions. For example, when the user taps on the display position of person 61a, control unit 34 sets first area 61 including person 61a in live view image 60a as a target area for setting (changing) imaging conditions, and displays the outline of first area 61 with emphasis.

[0050] In FIG. 6, a first area 61, which is displayed with an emphasized outline (bold, bright, in a different color, with a dashed line, blinking, etc.), indicates an area for which imaging conditions are to be set (changed). In the example of FIG. 6, a live view image 60a is displayed with an emphasized outline of the first area 61. In this case, the first area 61 is the area for which imaging conditions are to be set (changed). For example, in a camera 1 that is capable of touch operation, when the user taps on the shutter speed (TV) display 71, the control unit 34 displays the current setting value of the shutter speed for the emphasized area (first area 61) on the screen (reference numeral 68). In the following explanation, the camera 1 will be described on the assumption that touch operations are performed, but the imaging conditions may also be set (changed) by operating buttons and the like that constitute the operation member .

[0051] When the user taps the upper icon 71a or the lower icon 71b for the shutter speed (TV), the setting unit 34b increases or decreases the shutter speed display 68 from the current setting value in accordance with the tap operation, and sends an instruction to the imaging unit 32 (FIG. 1) to change the imaging conditions for the unit area 131 (FIG. 3) of the imaging element 32a corresponding to the highlighted area (first area 61) in accordance with the tap operation. The confirmation icon 72 is an operation icon for finalizing the set imaging conditions. The setting unit 34b also sets (changes) the frame rate and gain (ISO) in the same way as it sets (changes) the shutter speed (TV).

[0052] Although the setting unit 34b has been described as setting the imaging conditions based on a user operation, the setting unit 34b is not limited to this. The setting unit 34b may set the imaging conditions based on the judgment of the control unit 34, without based on a user operation. For areas that are not highlighted (areas other than the first area 61), the imaging conditions that have been set are maintained, except for some blocks, as will be described later.

[0053] Instead of highlighting the outline of the region that is the target of setting (changing) the imaging conditions, the control unit 34 may brighten the entire target region, increase the contrast of the entire target region, or blink the entire target region. The target region may also be surrounded by a frame. The frame surrounding the target region may be a double frame or a single frame, and the display mode of the frame, such as the line type, color, and brightness, may be changed as appropriate. The control unit 34 may also display an arrow or other symbol near the target region that indicates the region that is the target of setting the imaging conditions. The control unit 34 may darken areas other than the target region that is the target of setting (changing) the imaging conditions, or reduce the contrast of areas other than the target region.

[0054] <Setting imaging conditions for blocks including the boundary of the region> When the imaging conditions for at least some regions are set to be different from the imaging conditions for other regions as described above, the setting unit 34b sets the imaging conditions for blocks including the boundaries of the regions as follows. Fig. 7(a) is a diagram illustrating a predetermined range 80 that includes the boundary between a first region 61 corresponding to a person and a fourth region 64 corresponding to a mountain in a live view image 60a. Fig. 7(b) is an enlarged view of the predetermined range 80 in Fig. 7(a). In Fig. 7(b), the predetermined range 80 includes a plurality of blocks 81 to 89. The white areas in Figure 7(b) indicate areas corresponding to people. The diagonally shaded and hatched areas in Figure 7(b) indicate areas corresponding to mountains. Blocks 82, 85, and 87 include the boundary B1 between the first region 61 and the fourth region 64. That is, in Figure 7(b), of the areas corresponding to mountains, the areas corresponding to mountains in blocks 82, 85, and 87 where the boundary B1 exists are shaded.

[0055] When the first area 61 corresponding to a person is set to the first imaging condition and the fourth area 64 corresponding to a mountain is set to the fourth imaging condition, a question arises as to whether the blocks 82, 85, and 87 including the boundary B1 between the first area 61 and the fourth area 64 should be set to the first imaging condition or the fourth imaging condition.

[0056] However, if the imaging conditions for blocks 82, 85, and 87, including boundary B1, are set to the first imaging conditions, the first imaging conditions may not be suitable for imaging the shaded areas of blocks 82, 85, and 87, i.e., the fourth region within each block 82, 85, and 87. For example, in an extreme case, blown-out highlights or crushed shadows may occur in the image data corresponding to the shaded areas. Conversely, if the imaging conditions for blocks 82, 85, and 87 are set to the fourth imaging conditions, the fourth imaging conditions may not be suitable for imaging the white areas of blocks 82, 85, and 87, i.e., the first region within each block 82, 85, and 87. In an extreme case, blown-out highlights or crushed shadows may occur in the image data corresponding to the white areas. Blowed-out highlights refers to the loss of gradation in data in high-brightness areas of an image due to overexposure. Crushed shadows refers to the loss of gradation in data in low-brightness areas of an image due to underexposure.

[0057] Therefore, in this embodiment, the setting unit 34b sets the imaging conditions for blocks where boundary portions of regions exist as follows. In the following description, among blocks within the same subject (same region), blocks that include boundaries with adjacent regions are referred to as boundary blocks, and blocks that do not include boundaries are referred to as main blocks. That is, in the example of FIG. 7(b), blocks 82, 85, and 87 are boundary blocks, and the remaining blocks 81, 83, 84, 86, 88, and 89 are main blocks. For ease of explanation, the region to which the boundary blocks 82, 85, and 87 belong is referred to as boundary region 67. That is, boundary region 67 is a region where a boundary block including a boundary (boundary B1) between two regions (first region 61 and fourth region 64) exists, and includes part of the peripheral portion of the first region 61 and part of the peripheral portion of the fourth region 64. In FIG. 7(b) and FIG. 8, which will be described later, the region surrounded by a thick dashed line is boundary region 67.

[0058] The setting unit 34b sets an imaging condition (for example, imaging condition C) between the imaging condition of the main block of one region (for example, imaging condition A) and the imaging condition of the main block of the other region (for example, imaging condition B) as the imaging condition of the block where the boundary between the two regions exists. In other words, the setting unit 34b sets the imaging condition of the boundary block so that gradation information is not lost from the signals from the pixels of the boundary block.

[0059] 7, the setting unit 34b calculates, as a seventh imaging condition, an imaging condition between the first imaging condition set for the main blocks 81 and 84 in the first region 61 and the fourth imaging condition set for the main blocks 83, 86, 88, and 89 in the fourth region 64. The setting unit 34b sets the calculated seventh imaging condition as the imaging condition for the boundary blocks 82, 85, and 87. That is, the setting unit 34b sets the imaging condition for the first region 61 excluding the boundary region 67 to the first imaging condition, sets the imaging condition for the fourth region 64 excluding the boundary region 67 to the fourth imaging condition, and sets the imaging condition for the boundary region 67 to the seventh imaging condition.

[0060] For example, if only the ISO sensitivity differs between the first and fourth imaging conditions, and the ISO sensitivity of the first imaging condition is 100 and the ISO sensitivity of the fourth imaging condition is 800, the setting unit 34b sets the ISO sensitivity of the seventh imaging condition to a value between 100 and 800, for example, 400.

[0061] Also, for example, if only the shutter speed differs between the first and fourth imaging conditions, and the shutter speed of the first imaging condition is 1 / 1000 seconds and the shutter speed of the fourth imaging condition is 1 / 100 seconds, the setting unit 34b sets the shutter speed of the seventh imaging condition to a value between 1 / 1000 seconds and 1 / 100 seconds, for example, 1 / 500 seconds. As described above, when the ISO sensitivity or shutter speed is set as the seventh imaging condition to a value between the first imaging condition and the fourth imaging condition, it may be set to a value exactly halfway between the first imaging condition and the fourth imaging condition.

[0062] Furthermore, if only the frame rate differs between the first and fourth imaging conditions (the charge accumulation time is the same), and the frame rate of the first imaging condition is 30 fps and the frame rate of the fourth imaging condition is 60 fps, the setting unit 34b sets the frame rate of the seventh imaging condition to a value between 30 fps and 60 fps, for example, 45 fps.

[0063] In this way, the setting unit 34b sets the imaging conditions for the other boundary blocks in the same way. Note that if the imaging conditions (e.g., the first and fourth imaging conditions) for multiple areas (e.g., the first area 61 and the fourth area 64) related to the boundary block are the same, the setting unit 34b sets the imaging condition (e.g., the first imaging condition) as the imaging condition for the boundary block. As described above, for example, if the difference between the imaging conditions of two regions related to a boundary block is about 0.3 steps or less in Apex value, the imaging conditions of the two regions are considered to be the same. Therefore, if the imaging conditions of multiple regions related to the boundary block differ within such a variation range of about 0.3 steps in Apex value, the imaging condition of the boundary block may be set to any imaging condition within this range of variation.

[0064] In the above description, when setting the seventh capturing condition, the setting unit 34b calculated a value that was intermediate or approximately intermediate between the first and fourth capturing conditions as the seventh capturing condition. However, the seventh capturing condition may be set closer to the first or fourth capturing condition as long as no blown-out highlights or crushed shadows occur. In this case, it is desirable to set the seventh capturing condition so that the first and second correction processes described below and subsequent image processing, etc., are performed favorably. For example, the seventh capturing condition may be set as in any of the following (a) to (d).

[0065] (a) It is desirable to set the seventh imaging condition so as to approach the imaging condition for the area having a higher importance as a subject. For example, since the first area 61 corresponds to the person 61a who is the main subject and the fourth area 64 corresponds to the mountain 64a in the background, it is desirable to set the seventh imaging condition so as to approach the first imaging condition set for the main block for the first area 61 corresponding to the person 61a who is the main subject, within a range that does not cause blown-out highlights or crushed shadows. For example, if a face can be recognized by face recognition, it is desirable to set the imaging conditions of the boundary block close to the imaging conditions set for the main block for the area to which the recognized face belongs.

[0066] (b) For example, for each boundary block, the number of pixels corresponding to the first region 61 within the boundary block may be compared with the number of pixels corresponding to the fourth region 64, and the imaging conditions for the boundary block may be adjusted to be closer to the imaging conditions for the region with the larger number of pixels within a range that does not result in blown-out highlights or crushed shadows.

[0067] (c) For example, as shown in Fig. 5, the area of ​​the fourth region 64 on the image is larger than the area of ​​the first region 61. In this case, the imaging conditions of the boundary block between the first region 61 and the fourth region 64 may be set to be closer to the fourth imaging conditions set for the main block of the fourth region having a larger area, within a range that does not cause blown-out highlights or crushed shadows.

[0068] (d) For example, in order to increase the number of gradations of the signal from the pixel, the sensitivity of the first imaging condition of the first region 61 may be compared with the sensitivity of the fourth imaging condition of the fourth region 64, and the imaging condition of the boundary block may be set to the higher sensitivity side within a range that does not cause whiteout.

[0069] In the above description, when setting the seventh capturing condition, the setting unit 34b calculated a value between the first capturing condition and the fourth capturing condition as the seventh capturing condition. However, if blown-out highlights or crushed shadows do not occur in blocks including the boundary between regions, the seventh capturing condition may be set to the same as the first capturing condition or the fourth capturing condition. In this case, the seventh capturing condition may be set as in any of the following (e) to (h).

[0070] (e) Since the first region 61 corresponds to the person 61a who is the main subject, and the fourth region 64 corresponds to the mountain 64a which is the background, the seventh imaging condition may be the same as the first imaging condition set in the main block for the first region 61 corresponding to the person 61a who is the main subject, as long as no blown-out highlights or crushed shadows occur.

[0071] (f) For example, for each boundary block, the number of pixels corresponding to the first region 61 within the boundary block is compared with the number of pixels corresponding to the fourth region 64, and if no whiteout or blackout occurs, the imaging conditions for the boundary block may be made the same as the imaging conditions for the region with the larger number of pixels.

[0072] (g) For example, as shown in Fig. 5, the area of ​​the fourth region 64 on the image is larger than the area of ​​the first region 61. In this case, as long as blown-out highlights and crushed shadows do not occur, the imaging conditions for the boundary block between the first region 61 and the fourth region may be set to the same as the fourth imaging conditions set for the main block of the fourth region having the larger area.

[0073] (h) For example, if no whiteout occurs, in order to increase the number of gradations of the signal from the pixel, the sensitivity of the first imaging condition of the first region 61 may be compared with the sensitivity of the fourth imaging condition of the fourth region 64, and the imaging condition of the boundary block may be set to the imaging condition with the higher sensitivity between the first imaging condition and the fourth imaging condition.

[0074] Note that if blown-out highlights occur, the signal value from that pixel cannot be referenced, but even if crushed shadows occur, it may be possible to reference the signal value from that pixel. Therefore, when setting the seventh imaging condition, if blown-out highlights or crushed shadows occur in boundary blocks regardless of whether the first imaging condition, the fourth imaging condition, or an imaging condition between the two is set, it is desirable to set the seventh imaging condition so that crushed shadows are allowed to occur in boundary blocks and so that blown-out highlights do not occur.

[0075] As described above, after the imaging conditions for each region have been set, when a release button (not shown) constituting the operation member 36 or a display (release icon) instructing the start of imaging is operated, the control unit 34 controls the imaging unit 32 to capture images under the imaging conditions set for each of the divided regions. Then, the image processing unit 33 performs image processing on the image data acquired by the imaging unit 32. As described above, the image processing can be performed under different image processing conditions for each region.

[0076] After the image processing by the image processing unit 33, the recording unit 37 receives an instruction from the control unit 34 and records the image data after the image processing on a recording medium such as a memory card (not shown). This completes the series of imaging processes.

[0077] <First correction process> The correction section 33b of the image processing section 33 performs a first correction process as one of pre-processing processes performed before image processing, focus detection processing, subject detection (detection of subject elements) processing, and processing for setting imaging conditions.

[0078] As described above, in this embodiment, the setting unit 34b divides the area of ​​the image capture screen and sets the image capture conditions for each area. Furthermore, the setting unit 34b sets the image capture conditions for the boundary blocks as described above. Therefore, even for the same subject, the image capture conditions may differ between the portion captured in the boundary block and the portion captured in the main block. Therefore, in this embodiment, the signals from the pixels belonging to the boundary blocks are subjected to the following correction process to obtain signals similar to those captured under the same imaging conditions as those in the main block. This correction process is called the first correction process. The first correction process is performed to reduce discontinuities that occur in the processed image due to the existence of parts with different imaging conditions within the same region.

[0079] FIG. 8 is a diagram illustrating image data corresponding to FIG. 7(b). In FIG. 8, blocks 81 to 89 are each composed of four pixels (2 pixels x 2 pixels). Of the pixels shown in FIG. 8, subject light from person 61a is incident on white pixels 81a to 81d, 82a to 82c, 84a to 84d, 85a, and 87a, while subject light from mountain 64a is incident on shaded pixels 82d, 83a to 83d, 85b to 85d, 86a to 86d, 87b to 87d, 88a to 88d, and 89a to 89d. That is, subject light from person 61a is incident on each pixel of blocks 81 and 84, but not from mountain 64a. Therefore, blocks 81 and 84 are main blocks for the first region. Similarly, subject light from mountain 64a is incident on each pixel of blocks 83, 86, 88, and 89, but subject light from person 61a is not incident on each pixel of blocks 83, 86, 88, and 89. Therefore, blocks 83, 86, 88, and 89 are main blocks for the fourth region.

[0080] In block 82, subject light from person 61a is incident on pixels 82a, 82b, and 82c, while subject light from mountain 64a is incident on pixel 82d. In block 85, subject light from person 61a is incident on pixel 85a, while subject light from mountain 64a is incident on pixels 85b, 85c, and 85d. In block 87, subject light from person 61a is incident on pixel 87a, while subject light from mountain 64a is incident on pixels 87b, 87c, and 87d. Therefore, blocks 82, 85, and 87 are boundary blocks for the first and fourth regions.

[0081] Correction unit 33b determines from which subject area light is incident on each pixel belonging to the boundary block. Specifically, correction unit 33b calculates the position of the boundary on the imaging surface of image sensor 32a based on the detection result of the subject element by object detection unit 34a. Correction unit 33b then extracts a boundary block based on the calculated boundary position, and calculates from which subject element subject light is incident on each pixel belonging to the extracted boundary block.

[0082] 7 and 8, for example, correction unit 33b calculates the position of boundary B1 between first region 61 and fourth region 64 based on the detection results of subject elements by object detection unit 34a. Correction unit 33b then extracts blocks 82, 85, and 87 as boundary blocks based on the calculated position of boundary B1. Based on the calculated position of boundary B1, correction unit 33b then calculates that subject light from person 61a is incident on pixels 82a, 82b, and 82c in boundary block 82, pixel 85a in boundary block 85, and pixel 87a in boundary block 87. Based on the calculated position of boundary B1, correction unit 33b also calculates that subject light from mountain 64a is incident on pixel 82d in boundary block 82, pixels 85b, 85c, and 85d in boundary block 85, and pixels 87b, 87c, and 87d in boundary block 87.

[0083] Pixels 82a, 82b, and 82c of the boundary block 82 capture the subject light from the person 61a under a seventh imaging condition, which is an imaging condition between the first and fourth imaging conditions. On the other hand, each pixel of blocks 81 and 84, which are main blocks for the first region 61, captures the subject light from the person 61a under the first imaging condition. Therefore, the corrector 33b performs a first correction process on the signals from the pixels 82a, 82b, and 82c so as to obtain signals similar to those obtained when the subject light from the person 61a is captured under the first imaging condition. Similarly, pixel 82d of boundary block 82 captures an image of subject light from mountain 64a under the seventh imaging condition. Meanwhile, pixels in blocks 83, 86, 88, and 89, which are main blocks for fourth region 64, capture an image of subject light from mountain 64a under the fourth imaging condition. Therefore, the corrector 33b performs a first correction process on the signal from the pixel 82d so as to obtain a signal similar to that obtained when the subject light from the mountain 64a is captured under the fourth imaging condition.

[0084] For example, if only the ISO sensitivity differs between the first and fourth imaging conditions, with the ISO sensitivity of the first imaging condition being 100, the ISO sensitivity of the fourth imaging condition being 800, and the ISO sensitivity of the seventh imaging condition being 400, the first correction process is performed as follows: That is, as the first correction process, the corrector 33b multiplies the signals from pixel 82a, pixel 82b, and pixel 82c by 100 / 400, and multiplies the signal from pixel 82d by 800 / 400.

[0085] For example, if only the shutter speed differs between the first and fourth imaging conditions, and the shutter speed of the first imaging condition is 1 / 1000 seconds, the shutter speed of the fourth imaging condition is 1 / 100 seconds, and the shutter speed of the seventh imaging condition is 1 / 500 seconds, the first correction process is performed as follows: That is, as the first correction process, the corrector 33b multiplies the signals from pixel 82a, pixel 82b, and pixel 82c by (1 / 1000) / (1 / 500)=1 / 2, and multiplies the signal from pixel 82d by (1 / 100) / (1 / 500)=5.

[0086] For example, if only the frame rate differs between the first and fourth imaging conditions, with the frame rate under the first imaging condition being 30 fps, the frame rate under the fourth imaging condition being 60 fps, and the frame rate under the seventh imaging condition being 45 fps, the first correction process is performed as follows: Specifically, as the first correction process, the correction unit 33b thins out some of the frame image signals from pixels 82a, 82b, and 82c having a frame rate of 45 fps and converts them into frame image signals having a frame rate of 30 fps. This frame rate conversion is performed by selecting, from the frame image signals from pixels 82a to 82c, a frame image signal whose generation timing is closest to the 30 fps frame image signal under the first imaging condition. Alternatively, the frame rate conversion may be performed by interpolating a frame image signal synchronized with the generation timing of the 30 fps frame image signal under the first imaging condition based on the frame image signals from pixels 82a to 82c that are generated before and after the generation timing of the 30 fps frame image signal under the first imaging condition. Furthermore, as a first correction process, correction unit 33b converts frame image signals from pixel 82d with a frame rate of 45 fps into frame image signals with a frame rate of 60 fps. This frame rate conversion is performed, for example, by combining frame image signals read out successively from pixel 82d, i.e., by interpolating and calculating new frame image signals based on the previous and next frame image signals, thereby increasing the number of frame image signals.

[0087] In this way, the correction unit 33b performs the first correction process on signals from each pixel of all boundary blocks as needed. That is, the correction unit 33b performs the first correction process on signals from a pixel belonging to a boundary block if the imaging conditions applied to the main block for the same subject element as the pixel in question and the imaging conditions applied to the boundary block are different. However, if the imaging conditions applied to the main block for the same subject element as the pixel in question and the imaging conditions applied to the boundary block are the same, there is no need to perform the first correction process, and therefore the correction unit 33b does not perform the first correction process. As described above, even if there are some differences in the imaging conditions, they are considered to be the same imaging conditions. In addition, if the discontinuity that occurs in the image can be alleviated by performing image processing to reduce sharpness and contrast on image data obtained by capturing an image in a boundary block or a main block adjacent to the boundary block, the first correction process does not need to be performed.

[0088] <Second correction process> The corrector 33b of the image processor 33 further performs the following second correction process as needed before the image processing, focus detection process, subject detection (subject element detection) process, and process for setting the imaging conditions. Note that the corrector 33b performs the second correction process after the first correction process that has been performed as needed as described above. In the second correction process, signals from pixels in the boundary block corrected by the first correction process are processed as signals obtained by imaging under the imaging conditions set for the main block, rather than the imaging conditions set for the boundary block. For example, when performing the second correction process, signals from pixels 82a, 82b, and 82c in boundary block 82 corrected by the first correction process are processed by correction unit 33b as signals obtained by imaging under the first imaging condition, rather than the seventh imaging condition. Similarly, a signal from pixel 82d in boundary block 82 corrected by the first correction process is processed by correction unit 33b as a signal obtained by imaging under the fourth imaging condition, rather than the seventh imaging condition.

[0089] 1. When performing image processing When the image processing performed on the image data acquired by applying different imaging conditions between the divided regions is predetermined image processing, the correction unit 33b of the image processing unit 33 performs a second correction process on the image data located at the boundary between the regions as pre-processing of the image processing. The predetermined image processing is a process of calculating image data at a target position to be processed in the image by referring to image data at multiple reference positions around the target position, and examples of this include pixel defect correction processing, color interpolation processing, edge enhancement processing, and noise reduction processing.

[0090] The second correction process is performed to alleviate discontinuities that occur in an image after image processing due to differences in imaging conditions between divided regions. Generally, when a position of interest is located at a boundary between divided regions, image data to which the same imaging conditions as those of the image data of the position of interest are applied and image data to which imaging conditions different from those of the image data of the position of interest are applied may be mixed at multiple reference positions around the position of interest. In this embodiment, the second correction process is performed as follows based on the idea that it is preferable to calculate image data of the position of interest by referring to image data of the reference positions that have been subjected to the second correction process so as to reduce differences between the image data due to differences in imaging conditions, rather than calculating image data of the position of interest by directly referring to image data of the reference positions to which different imaging conditions are applied.

[0091] FIG. 9(a) is an enlarged view of a region of interest 90 at the boundary between the first region 61 and the fourth region 64 in the live view image 60a of FIG. 7(a). Image data from pixels on the image sensor 32a corresponding to the first region 61 to which the first imaging condition is applied is shown in white, and image data from pixels on the image sensor 32a corresponding to the fourth region 64 to which the fourth imaging condition is applied is shown in shaded. In FIG. 9(a), image data from a pixel of interest P is located in the first region 61, near the boundary 91 between the first region 61 and the fourth region 64, i.e., in the boundary portion. The pixels (eight pixels in this example) surrounding the pixel of interest P, which are included in the region of interest 90 (e.g., 3 × 3 pixels) centered around the pixel of interest P, are designated as reference pixels Pr. FIG. 9(b) is an enlarged view of the pixel of interest P and reference pixels Pr1 to Pr8. The position of the pixel of interest P is the position of interest, and the positions of the reference pixels Pr1 to Pr8 surrounding the pixel of interest P are reference positions. The first imaging condition is applied to the reference pixels Pr1 to Pr6 and the target pixel P corresponding to the first region 61, and the fourth imaging condition is applied to the reference pixels Pr7 and Pr8 corresponding to the fourth region 64. In the following description, the reference pixels Pr1 to Pr8 will be collectively referred to as Pr.

[0092] The generation unit 33c of the image processing unit 33 normally performs image processing by directly referencing the image data of the reference pixels Pr without performing the second correction processing. However, when the imaging conditions (referred to as first imaging conditions) applied to the pixel of interest P differ from the imaging conditions (referred to as fourth imaging conditions) applied to the reference pixels Pr surrounding the pixel of interest P, the correction unit 33b performs the second correction processing on the image data of the reference pixels Pr under the fourth imaging condition as shown in the following (Example 1) to (Example 3). Then, the generation unit 33c performs image processing to calculate image data of the pixel of interest P by referring to the image data of the reference pixels Pr after the second correction processing.

[0093] (Example 1) For example, if only the ISO sensitivity differs between the first and fourth imaging conditions, and the ISO sensitivity under the first imaging condition is 100 and the ISO sensitivity under the fourth imaging condition is 800, the correction unit 33b of the image processing unit 33 multiplies the image data of reference pixels Pr7 and Pr8 under the fourth imaging condition, among the image data of reference pixel Pr, by 100 / 800 as the second correction process, thereby reducing the difference between the image data due to the difference in imaging conditions. Note that, when the amount of light incident on the target pixel P is the same as the amount of light incident on the reference pixel Pr, the difference in image data becomes small, but when the amount of light incident on the target pixel P is originally different from the amount of light incident on the reference pixel Pr, the difference in image data may not become small. The same applies to the examples described later.

[0094] (Example 2) For example, when only the shutter speed differs between the first and fourth imaging conditions, and the shutter speed under the first imaging condition is 1 / 1000 seconds and the shutter speed under the fourth imaging condition is 1 / 100 seconds, the correction unit 33b of the image processing unit 33 multiplies the image data of reference pixels Pr7 and Pr8 under the fourth imaging condition, among the image data of reference pixel Pr, by (1 / 1000) / (1 / 100)=1 / 10 as a second correction process, thereby reducing the difference between the image data due to the difference in imaging conditions.

[0095] (Example 3) For example, when only the frame rate differs between the first and fourth imaging conditions (the charge accumulation time is the same), and the frame rate under the first imaging condition is 30 fps and that under the fourth imaging condition is 60 fps, the correction unit 33b of the image processing unit 33 performs the second correction process by adopting, for the image data under the fourth imaging condition (60 fps) among the image data of the reference pixel Pr, image data of a frame image whose acquisition start timing is close to that of the frame image acquired under the first imaging condition (30 fps). This reduces the difference between the image data due to the difference in imaging conditions. In addition, the second correction process may involve interpolating and calculating image data for a frame image whose acquisition start timing is close to that of a frame image acquired under the first imaging condition (30 fps) based on multiple adjacent frame images acquired under the fourth imaging condition (60 fps).

[0096] On the other hand, the correction unit 33b of the image processing unit 33 does not perform the second correction process on the image data of the reference pixel Pr when the imaging condition (referred to as the first imaging condition) applied to the pixel of interest P is the same as the imaging condition (referred to as the fourth imaging condition) applied to all reference pixels Pr surrounding the pixel of interest P. In other words, the generation unit 33c performs image processing to calculate the image data of the pixel of interest P by directly referring to the image data of the reference pixel Pr. As mentioned above, even if there are some differences in the imaging conditions, they are considered to be the same imaging conditions.

[0097] <Image processing example> Image processing involving the second correction processing will be illustrated. (1) Pixel defect correction processing In this embodiment, pixel defect correction processing is one of the image processing processes performed during image capture. Generally, pixel defects occur in the imaging element 32a, which is a solid-state imaging element, during or after manufacturing, and the imaging element 32a may output image data of an abnormal level. Therefore, the generation unit 33c of the image processing unit 33 corrects the image data output from the pixel where the pixel defect occurred, so that the image data at the pixel position where the pixel defect occurred is not noticeable.

[0098] An example of pixel defect correction processing will be described. The generation unit 33c of the image processing unit 33 designates, for example, a pixel at the position of a pixel defect recorded in a non-volatile memory (not shown in advance) in an image of one frame as a target pixel P (processing target pixel), and uses pixels (in this example, 8 pixels) around the target pixel P included in a target region 90 (for example, 3×3 pixels) centered on the target pixel P as reference pixels Pr.

[0099] The generation unit 33c of the image processing unit 33 calculates the maximum value and the minimum value of the image data in the reference pixels Pr, and performs Max, Min filter processing to replace the image data output from the target pixel P with the above maximum value or minimum value when the image data output from the target pixel P exceeds these maximum values or minimum values. Such processing is performed for all pixel defects whose position information is recorded in a non-volatile memory (not shown).

[0100] In the present embodiment, when a pixel to which a fourth imaging condition different from the first imaging condition applied to the target pixel P is included in the reference pixels Pr, the correction unit 33b of the image processing unit 33 performs second correction processing on the image data to which the fourth imaging condition is applied. Thereafter, the generation unit 33c of the image processing unit 33 performs the above-described Max, Min filter processing.

[0101] (2) Color interpolation processing In the present embodiment, color interpolation processing is one of the image processes performed at the time of imaging. As illustrated in FIG. 3, in the imaging chip 111 of the image sensor 100, green pixels Gb, Gr, blue pixels B, and red pixels R are arranged in a Bayer array. Since the generation unit 33c of the image processing unit 33 lacks image data of color components different from the color components of the color filter F arranged at each pixel position, color interpolation processing is performed to generate image data of the lacking color components by referring to the image data at the surrounding pixel positions.

[0102] An example of color interpolation processing will be described. FIG. 10(a) is a diagram illustrating the arrangement of image data output from the image sensor 32a. Corresponding to each pixel position, it has a color component of either R, G, or B according to the rule of the Bayer array. <G color interpolation> First, we will explain general G color interpolation. The generation unit 33c of the image processing unit 33, which performs G color interpolation, sequentially sets the positions of the R color component and the B color component as a focus position, and generates G color component image data at the focus position by referencing four G color component image data at reference positions surrounding the focus position. For example, when generating G color component image data at the focus position indicated by the bold frame in FIG. 10(b) (second row, second column counting from the upper left position; similarly, the focus position will be represented by counting from the upper left position), the generation unit 33c of the image processing unit 33 references four G color component image data G1 to G4 located near the focus position (second row, second column). For example, the generation unit 33c of the image processing unit 33 sets (aG1+bG2+cG3+dG4) / 4 as the G color component image data at the focus position (second row, second column). Note that a to d are weighting coefficients set according to the distance between the reference position and the focus position and the image structure.

[0103] Next, G color interpolation according to this embodiment will be described. In FIGS. 10(a) to 10(c), the first imaging condition is applied to the region to the left and above the thick line, and the fourth imaging condition is applied to the region to the right and below the thick line. Note that the first imaging condition and the fourth imaging condition are different in FIGS. 10(a) to 10(c). Furthermore, the image data G1 to G4 of the G color component in FIG. 10(b) are reference positions for image processing of the pixel at the target position (second row, second column). In FIG. 10(b), the first imaging condition is applied to the target position (second row, second column). Of the reference positions, the first imaging condition is applied to the image data G1 to G3. Furthermore, of the reference positions, the fourth imaging condition is applied to the image data G4. Therefore, the correction unit 33b of the image processing unit 33 performs a second correction process on the image data G4. Thereafter, the generation unit 33c of the image processing unit 33 calculates the image data of the G color component at the target position (second row, second column).

[0104] The generation unit 33c of the image processing unit 33 generates image data of the G color component at the positions of the B color component and the R color component in Figure 10(a), thereby obtaining image data of the G color component at each pixel position, as shown in Figure 10(c).

[0105] <R color interpolation> FIG. 11(a) is a diagram obtained by extracting the image data of the R color component from FIG. 10(a). The generation unit 33c of the image processing unit 33 calculates the image data of the color difference component Cr shown in FIG. 11(b) based on the image data of the G color component shown in FIG. 10(c) and the image data of the R color component shown in FIG. 11(a).

[0106] First, the interpolation of the general color difference component Cr will be described. When the generation unit 33c of the image processing unit 33 generates the image data of the color difference component Cr at the target position indicated by the thick frame (the second row and the second column) in FIG. 11(b), for example, it refers to the image data Cr1 to Cr4 of the four color difference components located near the target position (the second row and the second column). The generation unit 33c of the image processing unit 33 uses, for example, (eCr1 + fCr2 + gCr3 + hCr4) / 4 as the image data of the color difference component Cr at the target position (the second row and the second column). Here, e to h are weight coefficients provided according to the distance between the reference position and the target position and the image structure.

[0107] Similarly, when the generation unit 33c of the image processing unit 33 generates the image data of the color difference component Cr at the target position indicated by the thick frame (the second row and the third column) in FIG. 11(c), for example, it refers to the image data Cr2, Cr4 to Cr6 of the four color difference components located near the target position (the second row and the third column). The generation unit 33c of the image processing unit 33 uses, for example, (qCr2 + rCr4 + sCr5 + tCr6) / 4 as the image data of the color difference component Cr at the target position (the second row and the third column). Here, q to t are weight coefficients provided according to the distance between the reference position and the target position and the image structure. In this way, the image data of the color difference component Cr is generated for each pixel position.

[0108] Next, the interpolation of the color difference component Cr according to this embodiment will be described. In FIGS. 11(a) to 11(c), for example, the first imaging condition is applied to the region to the left and above the bold line, and the fourth imaging condition is applied to the region to the right and below the bold line. Note that the first imaging condition and the fourth imaging condition are different in FIGS. 11(a) to 11(c). In FIG. 11(b), the position indicated by the bold frame (second row, second column) is the target position of the color difference component Cr. Furthermore, the color difference component image data Cr1 to Cr4 in FIG. 11(b) are reference positions for image processing of the pixel at the target position (second row, second column). In FIG. 11(b), the first imaging condition is applied to the target position (second row, second column). Of the reference positions, the first imaging condition is applied to image data Cr1, Cr3, and Cr4. Of the reference positions, the fourth imaging condition is applied to image data Cr2. Therefore, the correction unit 33b of the image processing unit 33 performs a second correction process on the image data Cr2. Thereafter, the generation unit 33c of the image processing unit 33 calculates image data of the color difference component Cr at the target position (second row, second column). 11(c), the position indicated by the bold frame (second row, third column) is the target position of the color difference component Cr. Also, the image data Cr2, Cr4, Cr5, and Cr6 of the color difference components in FIG. 11(c) are reference positions for image processing of the pixel at the target position (second row, third column). In FIG. 11(c), the fourth imaging condition is applied to the target position (second row, third column). Of the reference positions, the first imaging condition is applied to the image data Cr4 and Cr5. Also, of the reference positions, the fourth imaging condition is applied to the image data Cr2 and Cr6. Therefore, the correction unit 33b of the image processing unit 33 performs the second correction process on the image data Cr4 and Cr5. Then, the generation unit 33c of the image processing unit 33 calculates the image data of the color difference component Cr at the target position (second row, third column).

[0109] The generation unit 33c of the image processing unit 33 obtains image data of the color difference component Cr at each pixel position, and then adds the image data of the G color component shown in Figure 10(c) corresponding to each pixel position, thereby obtaining image data of the R color component at each pixel position.

[0110] FIG. 12(a) is a diagram obtained by extracting the image data of the B color component from FIG. 10(a). The generation unit 33c of the image processing unit 33 calculates the image data of the color difference component Cb shown in FIG. 12(b) based on the image data of the G color component shown in FIG. 10(c) and the image data of the B color component shown in FIG. 12(a).

[0111] First, the interpolation of the general color difference component Cb will be described. When the generation unit 33c of the image processing unit 33 generates the image data of the color difference component Cb at the attention position indicated by the thick frame (the third row and the third column) in FIG. 12(b), for example, it refers to the image data Cb1 to Cb4 of the four color difference components located in the vicinity of the attention position (the third row and the third column). The generation unit 33c of the image processing unit 33 uses, for example, (uCb1 + vCb2 + wCb3 + xCb4) / 4 as the image data of the color difference component Cb at the attention position (the third row and the third column). Here, u to x are weighting coefficients provided according to the distance between the reference position and the attention position and the image structure.

[0112] Similarly, when the generation unit 33c of the image processing unit 33 generates the image data of the color difference component Cb at the attention position indicated by the thick frame (the third row and the fourth column) in FIG. 12(c), for example, it refers to the image data Cb2, Cb4 to Cb6 of the four color difference components located in the vicinity of the attention position (the third row and the fourth column). The generation unit 33c of the image processing unit uses, for example, (yCb2 + zCb4 + αCb5 + βCb6) / 4 as the image data of the color difference component Cb at the attention position (the third row and the fourth column). Here, y, z, α, and β are weighting coefficients provided according to the distance between the reference position and the attention position and the image structure. Thus, the image data of the color difference component Cb is generated for each pixel position.

[0113] Next, the interpolation of the color difference component Cb according to this embodiment will be described. In FIGS. 12(a) to 12(c), for example, the first imaging condition is applied to the region to the left and above the bold line, and the fourth imaging condition is applied to the region to the right and below the bold line. Note that the first imaging condition and the fourth imaging condition are different in FIGS. 12(a) to 12(c). In FIG. 12(b), the position indicated by the bold frame (third row, third column) is the target position of the color difference component Cb. Furthermore, the color difference component image data Cb1 to Cb4 in FIG. 12(b) are reference positions for image processing of the pixel at the target position (third row, third column). In FIG. 12(b), the fourth imaging condition is applied to the target position (third row, third column). Of the reference positions, the first imaging condition is applied to the image data Cb1 and Cb3. Furthermore, of the reference positions, the fourth imaging condition is applied to the image data Cb2 and Cb4. Therefore, the correction unit 33b of the image processing unit 33 performs second correction processing on the data Cb1 and Cb3, respectively. Thereafter, the generation unit 33c of the image processing unit 33 calculates image data of the color difference component Cb at the target position (third row, third column). 12(c), the position indicated by a bold frame (third row, fourth column) is the target position of the chrominance component Cb. Furthermore, the image data Cb2, Cb4 to Cb6 of the chrominance components in FIG. 12(c) are reference positions for performing image processing on the pixel at the target position (third row, fourth column). In FIG. 12(c), the fourth imaging condition is applied to the target position (third row, fourth column). Furthermore, the fourth imaging condition is applied to the image data Cb2, Cb4 to Cb6 at all reference positions. Therefore, the generation unit 33c of the image processing unit 33 calculates the image data of the chrominance component Cb at the target position (third row, fourth column) by referring to the image data Cb2, Cb4 to Cb6 at the reference positions that have not been subjected to the second correction process by the correction unit 33b of the image processing unit 33.

[0114] The generation unit 33c of the image processing unit 33 obtains image data of the color difference component Cb at each pixel position, and then adds the image data of the G color component shown in Figure 10(c) corresponding to each pixel position, thereby obtaining image data of the B color component at each pixel position. In the above-described "G color interpolation," for example, when generating image data for the G color component at the position of interest indicated by the bold frame (second row, second column) in FIG. 10(b), the four pieces of image data G1 to G4 for the G color component located near the position of interest are referenced. However, the number of pieces of image data for the G color component to be referenced may be changed depending on the image structure. For example, if the images near the position of interest have similarity in the vertical direction (for example, a vertical stripe pattern), the interpolation process is performed using only the image data above and below the position of interest (G1 and G2 in FIG. 10(b)). Also, if the images near the position of interest have similarity in the horizontal direction (for example, a horizontal stripe pattern), the interpolation process is performed using only the image data to the left and right of the position of interest (G3 and G4 in FIG. 10(b)). In these cases, the image data G4 to be corrected by the correction unit 33b may or may not be used.

[0115] (3) Edge enhancement processing An example of contour enhancement processing will be described. The generation unit 33c of the image processing unit 33 performs a known linear filter operation using a kernel of a predetermined size centered on a pixel of interest P (a pixel to be processed) in one frame of image, for example. When the kernel size of a sharpening filter, which is an example of a linear filter, is N×N pixels, the position of the pixel of interest P is the focus position, and the positions of (N2-1) reference pixels Pr surrounding the pixel of interest P are reference positions. The kernel size may be N×M pixels.

[0116] The generation unit 33c of the image processing unit 33 performs a filter process to replace the image data at the target pixel P with the result of a linear filter operation, for example, by shifting the target pixel from left to right on each horizontal line from the top horizontal line of the frame image to the bottom horizontal line.

[0117] In this embodiment, the correction unit 33b of the image processing unit 33 performs a second correction process on the image data to which the fourth imaging condition has been applied when the reference pixels Pr include pixels to which a fourth imaging condition different from the first imaging condition applied to the pixel of interest P has been applied. Thereafter, the generation unit 33c of the image processing unit 33 performs the linear filter process described above.

[0118] (4) Noise reduction processing An example of noise reduction processing will be described. For example, the generation unit 33c of the image processing unit 33 performs a known linear filter operation on one frame of image using a kernel of a predetermined size centered on a pixel of interest P (a pixel to be processed). When the kernel size of a smoothing filter, which is an example of a linear filter, is N×N pixels, the position of the pixel of interest P is the target position, and the positions of (N2-1) reference pixels Pr surrounding the pixel of interest P are reference positions. The kernel size may be N×M pixels.

[0119] The generation unit 33c of the image processing unit 33 performs a filter process to replace the image data at the target pixel P with the result of a linear filter operation, for example, by shifting the target pixel from left to right on each horizontal line from the top horizontal line of the frame image to the bottom horizontal line.

[0120] In this embodiment, the correction unit 33b of the image processing unit 33 performs a second correction process on the image data to which the fourth imaging condition has been applied when the reference pixels Pr include pixels to which a fourth imaging condition different from the first imaging condition applied to the pixel of interest P has been applied. Thereafter, the generation unit 33c of the image processing unit 33 performs the linear filter process described above.

[0121] 2. When performing focus detection processing The lens movement control unit 34d of the control unit 34 performs focus detection processing using signal data (image data) corresponding to a predetermined position (focus point) on the imaging screen. When different imaging conditions are set between the divided areas and the focus point for AF operation is located on the boundary between the divided areas, the lens movement control unit 34d of the control unit 34 performs second correction processing on the signal data for focus detection of at least one area as preprocessing for the focus detection processing.

[0122] The second correction process is performed to prevent a decrease in the accuracy of the focus detection process due to differences in imaging conditions between the regions of the imaging screen divided by the setting unit 34b. For example, when signal data for focus detection of a focus point for detecting an amount of image shift (phase difference) in an image is located at the boundary between the divided regions, signal data to which different imaging conditions have been applied may be mixed in the signal data for focus detection. In the present embodiment, the second correction process is performed as follows, based on the idea that it is preferable to detect the amount of image shift (phase difference) using signal data that has been subjected to the second correction process so as to reduce differences between signal data due to differences in imaging conditions, rather than detecting the amount of image shift (phase difference) using signal data to which different imaging conditions have been applied as is.

[0123] <Example of focus detection processing> An example of focus detection processing involving second correction processing will be described below. The AF operation of this embodiment, for example, focuses on a subject corresponding to a focus point selected by a user from among multiple focus points on the imaging screen. The lens movement control unit 34d (generator) of the control unit 34 calculates the defocus amount of the imaging optical system 31 by detecting the image shift amount (phase difference) between multiple subject images formed by light beams passing through different pupil regions of the imaging optical system 31. The lens movement control unit 34d of the control unit 34 moves the focus lens of the imaging optical system 31 to a position where the defocus amount is zero (below the allowable value), i.e., to the in-focus position, thereby adjusting the focus of the imaging optical system 31.

[0124] FIG. 13 is a diagram illustrating the positions of focus detection pixels on the imaging surface of the image sensor 32a. In this embodiment, focus detection pixels are arranged discretely along the X-axis direction (horizontal direction) of the imaging chip 111. In the example of FIG. 13, 15 focus detection pixel lines 160 are arranged at predetermined intervals. The focus detection pixels that make up the focus detection pixel line 160 output photoelectric conversion signals for focus detection. In the imaging chip 111, normal imaging pixels are provided at pixel positions other than those of the focus detection pixel line 160. The imaging pixels output photoelectric conversion signals for live view images and recording.

[0125] Fig. 14 is an enlarged view of a portion of the focus detection pixel line 160 corresponding to the focus point 80A shown in Fig. 13. Fig. 14 illustrates red pixels R, green pixels G (Gb, Gr), and blue pixels B, as well as focus detection pixels S1 and S2. The red pixels R, green pixels G (Gb, Gr), and blue pixels B are arranged according to the Bayer array rules described above.

[0126] The square regions illustrated for the red pixel R, green pixel G (Gb, Gr), and blue pixel B represent the light-receiving regions of the imaging pixels. Each imaging pixel receives a light beam that passes through the exit pupil of the imaging optical system 31 (FIG. 1). That is, the red pixel R, green pixel G (Gb, Gr), and blue pixel B each have a square mask opening, and light that passes through these mask openings reaches the light-receiving portion of the imaging pixel.

[0127] The shape of the light receiving regions (mask openings) of the red pixel R, green pixel G (Gb, Gr), and blue pixel B is not limited to a rectangle, and may be, for example, a circle.

[0128] The semicircular regions illustrated for the focus detection pixels S1 and S2 indicate the light receiving regions of the focus detection pixels. That is, the focus detection pixel S1 has a semicircular mask opening on the left side of the pixel position in FIG. 14, and light passing through this mask opening reaches the light receiving portion of the focus detection pixel S1. On the other hand, the focus detection pixel S2 has a semicircular mask opening on the right side of the pixel position in FIG. 14, and light passing through this mask opening reaches the light receiving portion of the focus detection pixel S2. In this way, the focus detection pixels S1 and S2 each receive a pair of light beams that pass through different regions of the exit pupil of the imaging optical system 31 (FIG. 1).

[0129] The positions of the focus detection pixel lines 160 on the imaging chip 111 are not limited to the positions illustrated in Fig. 13. The number of focus detection pixel lines 160 is also not limited to the example in Fig. 13. Furthermore, the shape of the mask openings in the focus detection pixels S1 and S2 is not limited to semicircular, and may be rectangular, for example, formed by horizontally dividing the square-shaped light-receiving regions (mask openings) in the imaging pixels R, G, and B.

[0130] Furthermore, the focus detection pixel line 160 on the imaging chip 111 may be formed by arranging focus detection pixels along the Y-axis direction (vertical direction) of the imaging chip 111. Image sensors in which imaging pixels and focus detection pixels are two-dimensionally arranged as in Fig. 14 are well known, and detailed illustration and description of these pixels will be omitted.

[0131] 14, a configuration in which each of the focus detection pixels S1 and S2 receives one of a pair of light beams for focus detection, known as a 1PD structure, has been described. Alternatively, a configuration in which each of the focus detection pixels receives both of a pair of light beams for focus detection, known as a 2PD structure, may be used. The 2PD structure makes it possible to use the photoelectric conversion signals obtained by the focus detection pixels as photoelectric conversion signals for recording.

[0132] The lens movement control unit 34d of the control unit 34 detects the image shift amount (phase difference) between a pair of images formed by a pair of light beams passing through different regions of the imaging optical system 31 (FIG. 1) based on the photoelectric conversion signals for focus detection output from the focus detection pixel S1 and the focus detection pixel S2. Then, the lens movement control unit 34d calculates the defocus amount based on the image shift amount (phase difference). This type of defocus amount calculation using the pupil division phase difference method is well known in the field of cameras, so a detailed description will be omitted.

[0133] It is assumed that focus point 80A (FIG. 13) has been selected by the user at a position corresponding to attention area 90 at the boundary between first area 61 and fourth area 64 in live view image 60a illustrated in FIG. 7(a). FIG. 15 is an enlarged view of focus point 80A. White pixels indicate that the first imaging condition is applied, and shaded pixels indicate that the fourth imaging condition is applied. The position surrounded by a frame 170 in FIG. 15 corresponds to focus detection pixel line 160 (FIG. 13).

[0134] The lens movement control unit 34d of the control unit 34 normally performs focus detection processing without performing the second correction processing, using signal data from the focus detection pixels indicated by the frame 170 as is. However, if the signal data surrounded by the frame 170 contains a mixture of signal data to which the first imaging condition has been applied and signal data to which the fourth imaging condition has been applied, the lens movement control unit 34d of the control unit 34 performs the second correction processing as shown in the following (Example 1) to (Example 3) on the signal data under the fourth imaging condition out of the signal data surrounded by the frame 170. Then, the lens movement control unit 34d of the control unit 34 performs focus detection processing using the signal data after the second correction processing.

[0135] (Example 1) For example, if only the ISO sensitivity differs between the first and fourth imaging conditions, and the ISO sensitivity of the first imaging condition is 100 and the ISO sensitivity of the fourth imaging condition is 800, the lens movement control unit 34d of the control unit 34 multiplies the signal data of the fourth imaging condition by 100 / 800 as the second correction process, thereby reducing the difference between the signal data due to the difference in imaging conditions. Note that, when the amount of light incident on a pixel to which the first imaging condition is applied is the same as the amount of light incident on a pixel to which the fourth imaging condition is applied, the difference in signal data becomes small, but when the amount of light incident on a pixel to which the first imaging condition is applied is different from the amount of light incident on a pixel to which the fourth imaging condition is applied, the difference in signal data may not become small.

[0136] (Example 2) For example, if only the shutter speed differs between the first and fourth imaging conditions, and the shutter speed of the first imaging condition is 1 / 1000 second and the shutter speed of the fourth imaging condition is 1 / 100 second, the lens movement control unit 34d of the control unit 34 multiplies the signal data of the fourth imaging condition by 1 / 1000 / 1 / 100=1 / 10 as the second correction process, thereby reducing the difference between the signal data due to the difference in imaging conditions.

[0137] (Example 3) For example, when only the frame rate differs between the first and fourth imaging conditions (the charge accumulation time is the same), and the frame rate of the first imaging condition is 30 fps and the frame rate of the fourth imaging condition is 60 fps, the lens movement control unit 34d of the control unit 34 performs the second correction process to adopt, for the signal data under the fourth imaging condition (60 fps), the signal data of the frame image whose acquisition start timing is close to that of the frame image acquired under the first imaging condition (30 fps), thereby reducing the difference between the signal data due to the difference in imaging conditions. The second correction process may involve interpolating and calculating signal data for a frame image whose acquisition start timing is close to that of a frame image acquired under the first imaging condition (30 fps) based on multiple adjacent frame images acquired under the fourth imaging condition (60 fps).

[0138] On the other hand, the lens movement control unit 34d of the control unit 34 does not perform the second correction process when the same imaging conditions are applied to the signal data enclosed by the frame 170. In other words, the lens movement control unit 34d of the control unit 34 performs focus detection processing using the signal data from the focus detection pixels indicated by the frame 170 as is.

[0139] As mentioned above, even if there are some differences in the imaging conditions, they are considered to be the same imaging conditions. In addition, in the above example, an example was described in which the second correction processing is performed on signal data of the fourth imaging condition among the signal data using the first imaging condition, but the second correction processing may also be performed on signal data of the first imaging condition among the signal data using the fourth imaging condition. The lens movement control unit 34d of the control unit 34 may determine whether to perform the second correction processing on the signal data under the first imaging condition or the signal data under the fourth imaging condition based on, for example, the ISO sensitivity. When the ISO sensitivity differs between the first and fourth imaging conditions, it is desirable to perform the second correction processing on the signal data obtained under the imaging condition with the lower ISO sensitivity unless the signal data obtained under the imaging condition with the higher ISO sensitivity is saturated. In other words, when the ISO sensitivity differs between the first and fourth imaging conditions, it is desirable to perform the second correction processing on the darker signal data so as to reduce the difference with the brighter signal data.

[0140] Furthermore, the second correction process may be performed on the signal data under the first imaging condition and the signal data under the fourth imaging condition, respectively, to reduce the difference between the two sets of signal data after the second correction process.

[0141] The above explanation has exemplified focus detection processing using the pupil division phase difference method, but the same can also be done using a contrast detection method in which the focus lens of the imaging optical system 31 is moved to the in-focus position based on the contrast of the subject image.

[0142] When the contrast detection method is used, the control unit 34 performs a known focus evaluation value calculation based on signal data output from the imaging pixels of the image sensor 32a corresponding to the focus point at each position of the focus lens while moving the focus lens of the imaging optical system 31. The position of the focus lens that maximizes the focus evaluation value is then determined as the in-focus position.

[0143] The control unit 34 normally performs focus evaluation value calculations using signal data output from imaging pixels corresponding to the focus point without performing the second correction process. However, when the signal data corresponding to the focus point contains a mixture of signal data to which the first imaging condition has been applied and signal data to which the fourth imaging condition has been applied, the control unit 34 performs the second correction process described above on the signal data under the fourth imaging condition among the signal data corresponding to the focus point. The control unit 34 then performs focus evaluation value calculations using the signal data after the second correction process. In the above example, the focus adjustment process is performed after the second correction process, but the focus adjustment may be performed using the image data obtained by the first correction process without performing the second correction process.

[0144] 3. When performing subject detection processing 16(a) is a diagram illustrating an example of a template image representing an object to be detected, and FIG. 16(b) is a diagram illustrating an example of a live-view image 60(a) and a search range 190. The object detection unit 34a of the control unit 34 detects an object (for example, a bag 63a, which is one of the subject elements in FIG. 5) from the live-view image and generates a template image 180 of the detected object. The object detection unit 34a of the control unit 34 may use the entire range of the live-view image 60a as the range for detecting the object, or may use only a portion of the live-view image 60a as the search range 190 to reduce the load on the detection process.

[0145] When different imaging conditions are applied between the divided areas and the search range 190 includes the boundary between the divided areas, the object detection unit 34a of the control unit 34 performs a second correction process on the image data of at least one area within the search range 190 as preprocessing for the subject detection process.

[0146] The second correction process is performed to suppress a decrease in accuracy of the detection process of subject elements due to differences in imaging conditions between the regions of the captured image divided by setting unit 34b. Generally, when search range 190 used to detect subject elements includes the boundaries of the divided regions, image data to which different imaging conditions have been applied may be mixed in the image data of search range 190. In this embodiment, the second correction process is performed as follows, based on the idea that it is preferable to detect subject elements using image data that has been subjected to the second correction process so as to suppress differences between image data due to differences in imaging conditions, rather than detecting subject elements using image data to which different imaging conditions have been applied as is.

[0147] A case will be described where a bag 63a belonging to a person 61a is detected in a live view image 60a illustrated in Fig. 5. The object detection unit 34a of the control unit 34 sets a search range 190 in the vicinity of an area including the person 61a. Note that the area 61 including the person 61a may also be set as the search range.

[0148] If the search range 190 is not divided by two areas with different imaging conditions, the object detection unit 34a of the control unit 34 performs the subject detection process using the image data that constitutes the search range 190 as is, without performing the second correction process. However, if the image data of the search range 190 includes a mixture of image data to which the first imaging condition has been applied and image data to which the fourth imaging condition has been applied, the object detection unit 34a of the control unit 34 performs the second correction process as in (Example 1) to (Example 3) described above as cases in which focus detection process is performed on the image data of the fourth imaging condition out of the image data of the search range 190. Then, the object detection unit 34a of the control unit 34 performs the subject detection process using the image data after the second correction process. As mentioned above, even if there are some differences in the imaging conditions, they are considered to be the same imaging conditions. In addition, in the above example, an example was described in which the second correction processing is performed on image data of the fourth imaging condition among the image data using the first imaging condition, but the second correction processing may also be performed on image data of the first imaging condition among the image data using the fourth imaging condition.

[0149] The second correction process for the image data in the search range 190 described above may be applied to a search range used to detect a specific subject such as a person's face or to an area used to determine the captured scene.

[0150] Furthermore, the second correction process for the image data in the search range 190 described above may be applied not only to the search range used in the pattern matching method using a template image, but also to the search range when detecting features based on the color, edges, etc. of the image.

[0151] Furthermore, this method may be applied to a moving object tracking process in which a known template matching process is performed using image data of multiple frames acquired at different times, thereby searching for an area similar to a tracking target in an earlier acquired frame image from a later acquired frame image. In this case, if image data to which the first imaging condition is applied and image data to which the fourth imaging condition is applied are mixed in the search range set in the later acquired frame image, the control unit 34 performs the second correction process as described above in (Example 1) to (Example 3) on the image data of the fourth imaging condition among the image data in the search range. Then, the control unit 34 performs the tracking process using the image data after the second correction process.

[0152] The same applies to the case where a known motion vector is detected using image data of multiple frames acquired at different times. When image data to which the first imaging condition is applied and image data to which the fourth imaging condition is applied are mixed in the detection area used for detecting the motion vector, the control unit 34 performs the second correction processing as described above in (Example 1) to (Example 3) on the image data of the fourth imaging condition among the image data of the detection area used for detecting the motion vector. Then, the control unit 34 detects the motion vector using the image data after the second correction processing. In the above example, the subject detection process is performed after the second correction process, but the subject detection may be performed using the image data obtained by the first correction process without performing the second correction process.

[0153] 4. Setting imaging conditions When dividing the area of ​​the imaging screen and setting different imaging conditions between the divided areas, the setting unit 34b of the control unit 34 performs new photometry to determine exposure conditions, and then performs a second correction process on the image data of at least one area as pre-processing for setting the exposure conditions.

[0154] The second correction process is performed to suppress a decrease in accuracy of the process for determining exposure conditions due to differences in imaging conditions between the regions of the imaging screen divided by the setting unit 34b. For example, if the photometric range set in the center of the imaging screen includes the boundaries of the divided regions, image data to which different imaging conditions have been applied may be mixed in the image data of the photometric range. In this embodiment, the second correction process is performed as follows, based on the idea that it is preferable to perform exposure calculation processing using image data that has been subjected to the second correction process so as to suppress differences between image data due to differences in imaging conditions, rather than performing exposure calculation processing using image data to which different imaging conditions have been applied as is.

[0155] If the photometric range is not divided into multiple regions with different imaging conditions, the setting unit 34b of the control unit 34 performs exposure calculation processing using the image data that constitutes the photometric range as is without performing the second correction processing. However, if the image data of the photometric range includes a mixture of image data to which the first imaging condition is applied and image data to which the fourth imaging condition is applied, the setting unit 34b of the control unit 34 performs the second correction processing as in (Example 1) to (Example 3) described above for the cases in which focus detection processing and subject detection processing are performed on the image data of the photometric range under the fourth imaging condition. Then, the setting unit 34b of the control unit 34 performs exposure calculation processing using the image data after the second correction processing. As mentioned above, even if there are some differences in the imaging conditions, they are considered to be the same imaging conditions. In addition, in the above example, an example was described in which the second correction processing is performed on image data of the fourth imaging condition among the image data using the first imaging condition, but the second correction processing may also be performed on image data of the first imaging condition among the image data using the fourth imaging condition.

[0156] This is not limited to the photometric range used when performing the exposure calculation process described above, but also applies to the photometric (colorimetric) range used when determining a white balance adjustment value, the photometric range used when determining whether or not to emit fill light from a light source that emits fill light, and even the photometric range used when determining the amount of fill light emitted from the light source.

[0157] Furthermore, when the readout resolution of the photoelectric conversion signal is made different between the divided regions of the imaging screen, the same can be applied to the regions used to determine the imaging scene when determining the readout resolution for each region. In the above example, the shooting conditions were set after the second correction process was performed, but the shooting conditions may be set using the image data obtained by the first correction process without performing the second correction process.

[0158] <Explanation of the flowchart> Fig. 17 is a flowchart illustrating the flow of processing for setting imaging conditions for each area and capturing an image. When the main switch of camera 1 is turned on, control unit 34 starts a program that executes the processing shown in Fig. 17. In step S10, control unit 34 causes display unit 35 to start live view display, and the process proceeds to step S20.

[0159] Specifically, the control unit 34 instructs the imaging unit 32 to start acquiring live view images, and causes the acquired live view images to be sequentially displayed on the display unit 35. As described above, at this point, the same imaging conditions are set for the entire area of ​​the imaging chip 111, i.e., the entire screen. If a setting is made to perform AF operation during live view display, the lens movement control unit 34d of the control unit 34 controls the AF operation to focus on a subject element corresponding to a predetermined focus point by performing focus detection processing. The lens movement control unit 34d performs focus detection processing after the first correction processing and the second correction processing, or after the first correction processing or the second correction processing, as necessary. Furthermore, if the setting to perform AF operation during live view display is not made, the lens movement control section 34d of the control section 34 will perform AF operation when an AF operation command is issued later.

[0160] In step S20, the object detection unit 34a of the control unit 34 detects subject elements from the live view image, and the process proceeds to step S30. The object detection unit 34a performs the first correction process and the second correction process, or the first correction process or the second correction process, and then performs the subject detection process, as necessary. In step S30, the setting unit 34b of the control unit 34 divides the screen of the live view image into areas including subject elements, and the process proceeds to step S40.

[0161] In step S40, the control unit 34 displays the regions on the display unit 35. The control unit 34 highlights the region among the divided regions that is the target of setting (changing) the imaging conditions. Furthermore, the control unit 34 displays an imaging condition setting screen 70 on the display unit 35, as shown in FIG. 6, and the process proceeds to step S50. In addition, when the user taps the display position of another main subject on the display screen with his / her finger, the control unit 34 changes the area including the main subject to the area to be targeted for setting (changing) the imaging conditions and highlights it.

[0162] In step S50, the control unit 34 determines whether or not an AF operation is necessary. For example, if the focus adjustment state has changed due to movement of the subject, if the position of the focus point has been changed by a user operation, or if an AF operation is instructed by a user operation, the control unit 34 makes a positive determination in step S50 and proceeds to step S70. If the focus adjustment state has not changed, the position of the focus point has not been changed by a user operation, or an AF operation is not instructed by a user operation, the control unit 34 makes a negative determination in step S50 and proceeds to step S60.

[0163] In step S70, the control unit 34 performs an AF operation and returns to step S40. The lens movement control unit 34d performs the first correction process and the second correction process, or performs the first correction process or the second correction process and then performs a focus detection process, which is an AF operation, as necessary. After returning to step S40, the control unit 34 repeats the same process as described above based on the live view image acquired after the AF operation.

[0164] In step S60, the setting unit 34b of the control unit 34 sets the imaging conditions for the highlighted area in response to a user operation. Furthermore, if an area is set with imaging conditions different from those for other areas, the setting unit 34b of the control unit 34 sets the imaging conditions for the boundary block as described above. The display transition and imaging condition setting of the display unit 35 in response to a user operation in step S60 are as described above. The setting unit 34b of the control unit 34 performs the first correction process and the second correction process, or the first correction process or the second correction process, as necessary, and then performs the exposure calculation process. After step S60 is executed, the process proceeds to step S80.

[0165] In step S80, the control unit 34 determines whether or not an image capture instruction has been issued. If a release button (not shown) constituting the operation member 36 or a display icon instructing image capture has been operated, the control unit 34 makes an affirmative decision in step S80 and proceeds to step S90. If an image capture instruction has not been issued, the control unit 34 makes a negative decision in step S80 and returns to step S20.

[0166] In step S90, the control unit 34 performs a predetermined imaging process. That is, the imaging control unit 34c controls the imaging element 32a to perform imaging under the imaging conditions set for each of the above-mentioned regions, and the process proceeds to step S100.

[0167] In step S100, the imaging control unit 34c of the control unit 34 sends an instruction to the image processing unit 33 to perform predetermined image processing on the image data obtained by the imaging, and then the process proceeds to step S110. The image processing includes the pixel defect correction processing, color interpolation processing, edge enhancement processing, and noise reduction processing. In addition, the correction unit 33b of the image processing unit 33 performs the first correction process and the second correction process, or the first correction process or the second correction process, on image data located at the boundary of the region as necessary, and then performs image processing.

[0168] In step S110, the control unit 34 sends an instruction to the recording unit 37 to record the image data after image processing on a recording medium (not shown), and the process proceeds to step S120.

[0169] In step S120, the control unit 34 determines whether or not an end operation has been performed. If an end operation has been performed, the control unit 34 makes an affirmative decision in step S120 and ends the processing in Fig. 17. If an end operation has not been performed, the control unit 34 makes a negative decision in step S120 and returns to step S20. If the processing returns to step S20, the control unit 34 repeats the processing described above.

[0170] In the above description, the stacked type imaging element 100 is exemplified as the imaging element 32a, but as long as imaging conditions can be set for each of multiple blocks in the imaging element (imaging chip 111), it does not necessarily have to be configured as a stacked type imaging element.

[0171] According to the first embodiment described above, the following advantageous effects can be obtained. (1) The camera 1 includes an imaging unit 32 capable of setting imaging conditions for each block having a plurality of pixels; a control unit 34 (setting unit 34b) that sets a first imaging condition for a block in the first region 61 into which a first light from a person 61a is incident, a fourth imaging condition for a block in the fourth region 64 into which a second light from a mountain 64a is incident, and a seventh imaging condition for a block in the boundary region 67 into which the first and second light are incident; an image processing unit 33 (correction unit 33b) that corrects signals from pixels in the blocks in the boundary region 67 based on the first and fourth imaging conditions; and an image processing unit 33 (generation unit 33c) that generates an image using the signals corrected by the correction unit 33b, signals from pixels in the blocks in the first region 61, and signals from pixels in the blocks in the fourth region 64. This allows appropriate processing to be performed on each region with different imaging conditions. That is, images can be generated appropriately using image data generated for each region. For example, discontinuities and strangeness that appear in the generated image due to differences in imaging conditions for each region can be suppressed. Furthermore, since the imaging conditions for the boundary blocks can be set appropriately, it is possible to prevent whiteout or blackout of image data from pixels in the boundary blocks, and it is possible to generate image data appropriately.

[0172] (2) The boundary region 67 has boundary blocks 82, 85, and 87 onto which a first light from the person 61a and a second light from the mountain 64a are incident. The correction unit 33b corrects signals from some of the pixels in the boundary blocks 82, 85, and 87 onto which the first and second light are incident, based on a first imaging condition, and corrects signals from other of the pixels in the boundary blocks 82, 85, and 87 onto which the first and second light are incident, based on a fourth imaging condition. This allows appropriate processing to be performed on each region with different imaging conditions. That is, images can be generated appropriately using image data generated for each region. For example, discontinuities and strangeness that appear in the generated image due to differences in imaging conditions for each region can be suppressed.

[0173] (3) The setting unit 34b sets the seventh imaging condition based on the first imaging condition and the fourth imaging condition. This allows the imaging condition for the boundary block to be set appropriately, preventing whiteout and blackout of image data from pixels in the boundary block and enabling appropriate generation of image data.

[0174] (4) The setting unit 34b sets the seventh imaging condition so that gradation information is not lost from signals from pixels in the boundary blocks 82, 85, and 87 of the boundary region 67. This allows the imaging condition for the boundary blocks to be appropriately set, thereby preventing blown-out highlights and crushed shadows in the image data from the pixels in the boundary blocks and enabling appropriate generation of image data.

[0175] (5) The camera 1 includes an imaging unit 32 capable of setting imaging conditions for each block having a plurality of pixels; a control unit 34 (setting unit 34b) that sets a first imaging condition for a block of the first region 61 into which a first light from a person 61a is incident via the imaging optical system 31, a fourth imaging condition for a block of the fourth region 64 into which a second light from a mountain 64a is incident via the imaging optical system 31, and a seventh imaging condition for a block of the boundary region 67 into which the first and second light are incident via the imaging optical system 31; an image processing unit 33 (correction unit 33b) that corrects signals from pixels in the blocks of the boundary region 67 based on the first and fourth imaging conditions; and a control unit 34 (lens movement control unit 34d) that generates a signal for driving the imaging optical system 31 based on the signal corrected by the correction unit 33b, the signal from the pixels in the block of the first region 61, and the signal from the pixels in the block of the fourth region. This allows appropriate processing to be performed in each area with different imaging conditions. That is, the defocus amount can be appropriately detected based on the focus detection signal data generated in each area. For example, it is possible to prevent a decrease in focus detection accuracy due to differences in imaging conditions between areas.

[0176] (6) The camera 1 includes an imaging unit 32 capable of setting imaging conditions for each block having a plurality of pixels; a control unit 34 (setting unit 34b) that sets a first imaging condition for the block of the first region 61 into which a first light from the person 61a is incident, a fourth imaging condition for the block of the fourth region 64 into which a second light from the mountain 64a is incident, and a seventh imaging condition for the block of the boundary region 67 into which the first and second light are incident; an image processing unit 33 (correction unit 33b) that corrects signals from pixels in the block of the boundary region 67 based on the first and fourth imaging conditions; and a control unit 34 (object detection unit 34a) that detects the subjects of the person 61a and the mountain 64a based on image data based on the signals corrected by the correction unit 33b, image data based on signals from pixels in the block of the first region 61, and image data based on signals from pixels in the block of the fourth region 64. This allows appropriate processing to be performed for each area with different imaging conditions. That is, subject elements can be appropriately detected based on the image data generated for each area. For example, it is possible to prevent a decrease in detection accuracy due to differences in imaging conditions for each area.

[0177] (7) The camera 1 includes an imaging unit 32 capable of setting imaging conditions for each block having a plurality of pixels, a control unit 34 (setting unit 34b) that sets a first imaging condition for a block in the first region 61 where a first light from a person 61a is incident, a fourth imaging condition for a block in the fourth region 64 where a second light from a mountain 64a is incident, and a seventh imaging condition for a block in the boundary region 67 where the first and second light are incident, and an image processing unit 33 (correction unit 33b) that corrects signals from pixels in the blocks in the boundary region 67 based on the first and fourth imaging conditions. This allows appropriate processing to be performed for each area with different imaging conditions. That is, imaging conditions can be set appropriately based on the image data generated for each area. For example, it is possible to prevent a decrease in the accuracy of exposure condition setting due to differences in imaging conditions for each area.

[0178] (8) The setting unit 34b sets the seventh imaging condition based on the signal corrected by the correction unit 33b, the signal from the pixel in the block of the first region 61, and the signal from the pixel in the block of the fourth region. That is, after performing the first and second correction processes described above, the setting unit 34b performs new photometry and determines the exposure condition. This allows appropriate processing to be performed for each area with different imaging conditions. That is, imaging conditions can be set appropriately based on the image data generated for each area. For example, it is possible to prevent a decrease in the accuracy of exposure condition setting due to differences in imaging conditions for each area.

[0179] The system may be configured to switch between Mode 1, in which the second correction process is performed as preprocessing, and Mode 2, in which the second correction process is not performed as preprocessing. When Mode 1 is selected, the control unit 34 performs the preprocessing described above and then performs image processing and other processes. On the other hand, when Mode 2 is selected, the control unit 34 performs image processing and other processes without performing the preprocessing described above. For example, if a face detected as a subject element has a shadow, and the image is captured under different imaging conditions for the area including the shadowed portion of the face and the area including the non-shadowed portion of the face so that the brightness of the shadowed portion of the face is approximately the same as the brightness of the non-shadowed portion of the face, if the second correction process is performed on the generated image and then color interpolation is performed, unintended color interpolation may occur in the shadowed portion due to the difference in the imaging conditions set. By configuring the system to switch between Mode 1 and Mode 2 so that color interpolation can be performed using image data as is without performing the second correction process, unintended color interpolation can be avoided.

[0180] ---Second embodiment--- With reference to Fig. 18, a digital camera will be described as an example of an electronic device incorporating an image processing device according to the second embodiment. In the following description, the same components as those in the first embodiment are denoted by the same reference numerals, and differences will be mainly described. Points that are not particularly described are the same as those in the first embodiment. The present embodiment differs from the first embodiment mainly in the method of setting imaging conditions for a main block adjacent to a region block.

[0181] FIG. 18(a) is a diagram showing multiple blocks including a portion of the predetermined range 80 in the first embodiment. To the right of block 86 is block 92, into which subject light from mountain 64a is incident. Although not shown, to the right of block 92 are multiple blocks into which subject light from mountain 64a is incident. In FIG. 18(a), the white background schematically indicates the portion corresponding to person 61a, and the shaded portion schematically indicates the portion corresponding to mountain 64a. That is, the white background corresponds to the first region 61, and the shaded portion corresponds to the fourth region 64. In FIG. 18, the region surrounded by a thick dashed line is boundary region 67.

[0182] In the first embodiment described above, the setting unit 34b sets the imaging condition of block 84, which is the main block of the first region 61, to the first imaging condition, and sets the imaging condition of block 86, which is the main block of the fourth region 64, to the fourth imaging condition. Then, the setting unit 34b sets the imaging condition of block 85, which is the boundary block between the first region 61 and the fourth region 64, to the seventh imaging condition, which is between the first imaging condition and the fourth imaging condition. That is, in the first embodiment, the imaging condition in a certain region is the same except for the boundary block.

[0183] In contrast, in the second embodiment, the setting unit 34b sets the imaging condition of block 84, which is the main block of the first region 61, to the first imaging condition, and sets the imaging condition of block 92, which is the main block of the fourth region 64, to the fourth imaging condition. As shown in FIG. 18(b), the setting unit 34b sets the imaging condition of boundary block 85 to the eighth imaging condition between the first and fourth imaging conditions, and sets the imaging condition of block 86, which is the main block adjacent to boundary block 85, to the ninth imaging condition between the eighth and fourth imaging conditions. In FIG. 18(b), the higher the density of the diagonal hatching on boundary block 85 and main block 86, the closer the set imaging condition is to the first imaging condition. That is, in this embodiment, the imaging condition of a main block in a certain region is set in a stepwise manner so that the difference between the imaging condition of the main block in that region and the imaging condition of the other region becomes smaller near the boundary with the other region.

[0184] For example, if only the ISO sensitivity differs between the first and fourth imaging conditions, and the ISO sensitivity of the first imaging condition is 100 and the ISO sensitivity of the fourth imaging condition is 800, the setting unit 34b sets the ISO sensitivity of the eighth imaging condition to a value between 100 and 800, for example, 200. Then, the setting unit 34b sets the ISO sensitivity of the ninth imaging condition to a value between 200 and 800, for example, 400.

[0185] In this way, the setting unit 34b similarly sets the imaging conditions for the other boundary blocks of the first region 61 and the fourth region 64, and for the main blocks of the fourth region 64 adjacent to the boundary blocks.

[0186] That is, in this embodiment, when two areas for which different imaging conditions are set are adjacent, the setting unit 34b sets the imaging conditions for the boundary block and the main block adjacent to the boundary block so that one imaging condition gradually approaches the other imaging condition within a range that does not cause whiteout or blackout. In the above description, the main block 84 of the first region 61 and the main block 86 of the fourth region 64 exist as blocks adjacent to the boundary block 85. In this embodiment, the imaging conditions (first imaging conditions) for the main block 84 of the first region 61 are the same as the imaging conditions for the other main blocks in the first region 61, and the imaging conditions (ninth imaging conditions) for the main block 86 of the fourth region 64 are different from the imaging conditions (fourth imaging conditions) for the other main blocks (block 92, etc.) in the fourth region 64. That is, in this embodiment, it is desirable to set the same imaging conditions for the blocks corresponding to the person 61a, who is the main subject, except for the boundary block. Therefore, in this embodiment, in the fourth region 64 corresponding to the mountain 64a, which is less important as a subject than the person 61a, the imaging conditions for the block 86 adjacent to the boundary block 85 are set different from the imaging conditions (fourth imaging conditions) for the other main blocks (block 92, etc.) in the fourth region 64. In the following description, a main block (main block 86) of one of two adjacent regions (fourth region 64) for which imaging conditions (ninth imaging conditions) are set so as to be closer to the imaging conditions (first or eighth imaging conditions) set for the main block 84 of the other region (first region 61) or the boundary block (boundary block 85) with the other region (first region 61) compared to another main block (main block 92) of that region (fourth region 64) will be referred to as a quasi-boundary block. In addition, in this embodiment, the region including the boundary block 85 and quasi-boundary block 86 will be referred to as a boundary region 67A.

[0187] In this way, it is desirable to set the imaging conditions in stages by providing a quasi-boundary block in the area (fourth area 64) that is less important as a subject of two adjacent areas (first area 61 and fourth area 64). When two areas that have been set to different imaging conditions are adjacent to each other with a boundary block in between, the setting unit 34b determines in which area to provide a quasi-boundary block, i.e., in which area's main block the imaging conditions are to be set in stages, based on, for example, the following condition: Note that the following conditions (a) to (c) are merely examples, and do not exclude other conditions. (a) For example, if it is determined based on the result of subject recognition that one region contains a face, the setting unit 34b determines that one region is a more important subject region than the other region, and provides a quasi-boundary block in the other region. (b) For example, if it is determined based on the result of focus detection that the distance to the subject in one region is closer than the distance to the subject in the other region, the setting unit 34b determines that one region is a more important subject region than the other region, and provides a quasi-boundary block in the other region. (c) For example, if it is determined based on the result of subject element detection that one region is located closer to the center of the imaging surface than the other region, the setting unit 34b determines that one region is a more important subject region than the other region, and provides a quasi-boundary block in the other region.

[0188] In the above description, the ninth imaging condition for the quasi-boundary block 86 is set closer to the first imaging condition than the fourth imaging condition for the main blocks of the fourth region 64, and the imaging condition for the main block 92 adjacent to the right of the quasi-boundary block 86 is set to the fourth imaging condition. However, as shown in Figure 18(c), the imaging condition for the main block 92 adjacent to the quasi-boundary block 86 may be set to the tenth imaging condition between the fourth and ninth imaging conditions. In other words, the main block 92 adjacent to the quasi-boundary block 86 may also be set as a quasi-boundary block to constitute the boundary region 67A. In FIG. 18(c), the higher the density of the diagonal hatching applied to the boundary block 85, the quasi-boundary block 86, and the quasi-boundary block 92, the closer the set imaging conditions are to the first imaging conditions.

[0189] That is, within the boundary region 67A, the imaging conditions of each block 86, 92, 93 may be set in stages so as to gradually approach the fourth imaging condition from the first imaging condition side, in the order of the quasi-boundary block 86 to the right of the quasi-boundary block 86, the quasi-boundary block 92 to the right of the quasi-boundary block 86, and the main block 93 to the right of the quasi-boundary block 92. That is, of two adjacent regions (first region 61 and fourth region 64), as one region (first region 61) approaches the other region (fourth region 64), the imaging conditions of the main blocks of the other region (fourth region 64) may gradually approach the imaging conditions of the one region (first region 61). In this way, of two adjacent regions, the imaging conditions of the main blocks of the other region may be set in a gradational manner.

[0190] In the example shown in FIG. 18(b), the imaging conditions for the main blocks of the fourth region 64 are changed by one step in main block 86, and in the example shown in FIG. 18(c), the imaging conditions for the main blocks of the fourth region 64 are changed by two steps, between main block 86 and main block 92. However, the imaging conditions may be changed by three or more steps. The number of steps by which the imaging conditions for the main blocks of the fourth region 64 are changed may be set appropriately based on, for example, the area of ​​the fourth region 64 or the difference between the first imaging condition and the fourth imaging condition. For example, the number of steps may be increased as the area of ​​the fourth region 64 increases. Furthermore, the number of steps may be increased as the difference between the first imaging condition and the fourth imaging condition increases.

[0191] As described above, when the imaging conditions of the main block are set in stages, the first correction process and the second correction process do not need to be performed unless discontinuity in the image due to differences in the imaging conditions is a problem.

[0192] According to the second embodiment described above, the following advantages are obtained in addition to the advantages of the first embodiment. (1) The boundary region 67A further includes a quasi-boundary block 86 onto which one of the first light from the person 61a and the second light from the mountain 64a is incident. The setting unit 34b sets an eighth imaging condition to the boundary block 85 onto which the first and second light are incident, and sets a ninth imaging condition to the quasi-boundary block 86 onto which one of the first light and the second light is incident. This makes it possible to more appropriately set the imaging conditions for blocks near the boundary of the region, thereby enabling more appropriate generation of image data.

[0193] ---Modifications of the first and second embodiments--- The following modifications are also within the scope of the present invention, and one or more of the modifications may be combined with the above-described embodiment. (Modification 1) 19(a) to 19(c) are diagrams illustrating the arrangement of the first and second imaging regions on the imaging surface of the image sensor 32a. In the example of Fig. 19(a), the first imaging region is made up of even-numbered columns, and the second imaging region is made up of odd-numbered columns. In other words, the imaging surface is divided into even-numbered columns and odd-numbered columns.

[0194] 19(b), the first imaging area is made up of odd-numbered rows, and the second imaging area is made up of even-numbered rows, i.e., the imaging surface is divided into odd-numbered rows and even-numbered rows.

[0195] 19(c), the first imaging area is made up of blocks in even rows and odd columns, and blocks in odd rows and even columns. The second imaging area is made up of blocks in even rows and even columns, and blocks in odd rows and odd columns. In other words, the imaging surface is divided into a checkerboard pattern.

[0196] 19(a) to 19(c), a first image based on the photoelectric conversion signal read out from the first imaging region and a second image based on the photoelectric conversion signal read out from the second imaging region are generated using the photoelectric conversion signals read out from the image sensor 32a that has captured one frame. According to Modification 1, the first image and the second image are captured at the same angle of view and include a common subject image.

[0197] In Modification 1, the control unit 34 uses the first image for display and the second image for detection. Specifically, the control unit 34 causes the display unit 35 to display the first image as a live view image. The control unit 34 also causes the object detection unit 34a to perform subject detection processing using the second image, the lens movement control unit 34a to perform focus detection processing using the second image, and the setting unit 34b to perform exposure calculation processing using the second image.

[0198] In Modification 1, the imaging conditions set for the first imaging area for capturing the first image are referred to as the first imaging conditions, and the imaging conditions set for the second imaging area for capturing the second image are referred to as the second imaging conditions. The control unit 34 may make the first imaging conditions and the second imaging conditions different.

[0199] 1. As an example, the control unit 34 sets the first imaging conditions to conditions suitable for display by the display unit 35. For example, the first imaging conditions set for the first imaging area are made the same throughout the first imaging area on the imaging screen. On the other hand, the control unit 34 sets the second imaging conditions set for the second imaging area to conditions suitable for focus detection processing, subject detection processing, and exposure calculation processing. The second imaging conditions are made the same throughout the second imaging area on the imaging screen. Note that if the conditions suitable for the focus detection process, the subject detection process, and the exposure calculation process are different, the control unit 34 may set different second imaging conditions for the second imaging area for each frame. For example, the second imaging conditions for the first frame may be conditions suitable for the focus detection process, the second imaging conditions for the second frame may be conditions suitable for the subject detection process, and the second imaging conditions for the third frame may be conditions suitable for the exposure calculation process. In these cases, the second imaging conditions for each frame are set to be the same throughout the entire second imaging area of ​​the imaging screen.

[0200] 2. As another example, the control unit 34 may set different first imaging conditions for the first imaging area depending on the area. The setting unit 34b of the control unit 34 sets different first imaging conditions for each area including subject elements divided by the setting unit 34b. On the other hand, the control unit 34 sets the same second imaging conditions for the entire second imaging area of ​​the imaging screen. The control unit 34 sets the second imaging conditions to conditions suitable for focus detection processing, subject detection processing, and exposure calculation processing. However, if the conditions suitable for focus detection processing, subject detection processing, and exposure calculation processing are different, the imaging conditions set for the second imaging area may be different for each frame.

[0201] 3. As another example, the control unit 34 may set the first imaging condition for the first imaging area to be the same for the entire first imaging area of ​​the imaging screen, while setting the second imaging condition for the second imaging area to be different within the imaging screen. For example, the setting unit 34b may set different second imaging conditions for each area including divided subject elements. Even in this case, if the conditions suitable for the focus detection process, subject detection process, and exposure calculation process are different, the imaging condition set for the second imaging area may be different for each frame.

[0202] 4. As yet another example, the control unit 34 varies the first imaging condition set for the first imaging region on the imaging screen, and varies the second imaging condition set for the second imaging region on the imaging screen. For example, the setting unit 34b sets different first imaging conditions for each region including the divided subject elements, and sets different second imaging conditions for each region including the divided subject elements.

[0203] 19(a) to 19(c), the area ratio between the first and second imaging regions may be made different. For example, based on a user operation or a determination of the control unit 34, the control unit 34 may set the ratio of the first imaging region higher than that of the second imaging region, set the ratio of the first imaging region to the second imaging region equal as illustrated in FIGS. 19(a) to 19(c), or set the ratio of the first imaging region lower than that of the second imaging region. By making the area ratio between the first and second imaging regions different, it is possible to make the first image higher in resolution than the second image, make the resolutions of the first and second images equal, or make the second image higher in resolution than the first image.

[0204] (Variation 2) In the above-described embodiment, when performing image processing, in the case where the imaging condition applied at the position of interest (hereinafter referred to as the first imaging condition) differs from the imaging condition applied at the reference position around the position of interest (hereinafter referred to as the fourth imaging condition), the corrector 33b of the image processing unit 33 corrects the image data of the fourth imaging condition (image data of the fourth imaging condition among the image data of the reference position) based on the first imaging condition, in the second correction process. That is, by performing the second correction process on the image data of the fourth imaging condition at the reference position, discontinuity of the image based on the difference between the first imaging condition and the fourth imaging condition is alleviated.

[0205] Instead, in Modification 2, the corrector 33b of the image processor 33 may correct the image data under the first imaging condition (the image data under the first imaging condition among the image data of the target position and the image data of the reference position) based on the fourth imaging condition. In this case as well, it is possible to reduce discontinuity in the image due to the difference between the first and fourth imaging conditions.

[0206] Alternatively, the corrector 33b of the image processor 33 may correct both the image data under the first imaging condition and the image data under the fourth imaging condition. That is, the second correction process may be performed on the image data of the position of interest under the first imaging condition, the image data under the first imaging condition among the image data of the reference position, and the image data under the fourth imaging condition among the image data of the reference position, thereby mitigating discontinuity of the images due to the difference between the first imaging condition and the fourth imaging condition. For example, in the above (Example 1), the image data of the reference pixel Pr under the first imaging condition (ISO sensitivity 100) is multiplied by 400 / 100 as the second correction process, and the image data of the reference pixel Pr under the fourth imaging condition (ISO sensitivity 800) is multiplied by 400 / 800 as the second correction process. This reduces the difference between the image data due to the difference in imaging conditions. The pixel data of the target pixel is subjected to a second correction process in which it is multiplied by 100 / 400 after color interpolation. This second correction process changes the pixel data of the target pixel after color interpolation to a value similar to that obtained when captured under the first imaging condition. Furthermore, in the above (Example 1), the degree of the second correction process may be changed depending on the distance from the boundary between the first and fourth regions. This reduces the rate at which the image data increases or decreases due to the second correction process compared to the case of the above (Example 1), thereby reducing noise caused by the second correction process. While the above description of the above (Example 1) has been given, the same applies to the above (Example 2).

[0207] According to the second modification, similarly to the above-described embodiment, it is possible to perform appropriate image processing on image data generated for areas with different imaging conditions.

[0208] (Variation 3) In the above-described embodiment, when the second correction process is performed on the image data, the corrected image data is obtained by performing a calculation based on the difference between the first and fourth imaging conditions. Instead of the calculation, the corrected image data may be obtained by referring to a correction table. For example, the corrected image data is read by inputting the first and fourth imaging conditions as arguments. Alternatively, the correction coefficients may be read by inputting the first and fourth imaging conditions as arguments.

[0209] (Variation 4) In the second correction process of the above-described embodiment, upper and lower limits of the corrected image data may be set. By setting upper and lower limit values, it is possible to restrict the correction so as not to make more correction than necessary. The upper and lower limit values ​​may be determined in advance, or, if a photometry sensor is provided in addition to the image sensor 32a, they may be determined based on the output signal from the photometry sensor.

[0210] (Variation 5) In the above embodiment, an example has been described in which the setting unit 34b of the control unit 34 detects subject elements based on a live view image and divides the screen of the live view image into areas including the subject elements. In Modification 5, if the control unit 34 is provided with a photometry sensor separate from the image sensor 32a, the control unit 34 may divide the areas based on an output signal from the photometry sensor.

[0211] The control unit 34 divides the image into a foreground and a background based on the output signal from the photometry sensor. Specifically, the control unit 34 divides the live view image acquired by the image sensor 32b into a foreground region corresponding to an area determined to be the foreground based on the output signal from the photometry sensor, and a background region corresponding to an area determined to be the background based on the output signal from the photometry sensor.

[0212] The control unit 34 further arranges the first and second imaging regions at positions corresponding to the foreground region on the imaging surface of the image sensor 32a, as illustrated in Figures 19(a) to 19(c). On the other hand, the control unit 34 arranges only the first imaging region on the imaging surface of the image sensor 32a at positions corresponding to the background region on the imaging surface of the image sensor 32a. The control unit 34 uses the first image for display and the second image for detection.

[0213] According to the fifth modification, the output signal from the photometry sensor can be used to perform area division of the live view image acquired by the image sensor 32b. Also, for the foreground area, a first image for display and a second image for detection can be obtained, and for the background area, only the first image for display can be obtained.

[0214] (Variation 6) In Modification 6, the generation unit 33c of the image processing unit 33 performs contrast adjustment processing as an example of second correction processing. That is, the generation unit 33c reduces discontinuity in the image due to the difference between the first and fourth imaging conditions by varying the gradation curve (gamma curve).

[0215] For example, assume that only the ISO sensitivity differs between the first and fourth imaging conditions, with the ISO sensitivity of the first imaging condition being 100 and the ISO sensitivity of the fourth imaging condition being 800. The generation unit 33c reduces the value of the image data of the fourth imaging condition among the image data at the reference position to 1 / 8 by flattening the gradation curve.

[0216] Alternatively, the generating unit 33c may increase the value of the image data of the target position and the image data of the reference position under the first imaging condition by eight times by increasing the gradation curve.

[0217] According to the sixth modification, similarly to the above-described embodiment, it is possible to appropriately perform image processing on image data generated in regions with different imaging conditions. For example, it is possible to suppress discontinuities and unnaturalness that appear in the processed image due to differences in imaging conditions at the boundary between regions.

[0218] (Variation 7) In the seventh modification, the image processing unit 33 performs the image processing (for example, noise reduction processing) described above so as not to impair the contours of the subject elements. Generally, smoothing filter processing is employed when performing noise reduction. When a smoothing filter is used, while the noise reduction effect is achieved, the boundaries of the subject elements may become blurred.

[0219] Therefore, the generation unit 33c of the image processing unit 33 compensates for the blurring of the boundaries of the above-mentioned subject elements by, for example, performing a contrast adjustment process in addition to or together with the noise reduction process. In the seventh modification, the generation unit 33c of the image processing unit 33 sets an S-shaped curve as the density conversion (gradation conversion) curve (so-called S-shaped conversion). By performing contrast adjustment using the S-shaped conversion, the generation unit 33c of the image processing unit 33 stretches the gradation portions of the bright data and dark data, respectively, to increase the number of gradations of the bright data (and dark data), and compresses the image data of intermediate gradations to reduce the number of gradations. This reduces the amount of image data with medium brightness in the image and increases the data classified as either bright or dark, thereby making it possible to compensate for the blurring of the boundaries of the subject elements.

[0220] According to the seventh modification, the blurring of the boundaries of the subject elements can be compensated for by making the contrast between light and dark in the image clearer.

[0221] (Variation 8) In the eighth modification, the generation unit 33c of the image processing unit 33 changes the white balance adjustment gain so as to alleviate discontinuity in the image based on the difference between the first and fourth imaging conditions.

[0222] For example, when the imaging conditions applied when capturing an image at a focus position (referred to as the first imaging conditions) differ from the imaging conditions applied when capturing an image at a reference position around the focus position (referred to as the fourth imaging conditions), the generation unit 33c of the image processing unit 33 changes the white balance adjustment gain so that the white balance of the image data of the fourth imaging conditions among the image data of the reference position approaches the white balance of the image data acquired under the first imaging conditions.

[0223] In addition, the generation unit 33c of the image processing unit 33 may change the white balance adjustment gain so that the white balance of the image data of the first imaging condition among the image data of the reference position and the image data of the target position approaches the white balance of the image data acquired under the fourth imaging condition.

[0224] According to variant example 8, by aligning the white balance adjustment gain for image data generated in areas with different imaging conditions to the adjustment gain for one of the areas with different imaging conditions, it is possible to alleviate discontinuity in the image due to the difference between the first and fourth imaging conditions.

[0225] (Variation 9) A plurality of image processing units 33 may be provided, and image processing may be performed in parallel. For example, image processing may be performed on image data captured in area A of the imaging unit 32 while image processing is performed on image data captured in area B of the imaging unit 32. The plurality of image processing units 33 may perform the same image processing, or may perform different image processing. In other words, the same parameters or the like may be applied to the image data of area A and area B to perform similar image processing, or different parameters or the like may be applied to the image data of area A and area B to perform different image processing.

[0226] In the case where a plurality of image processing units 33 are provided, image processing may be performed by one image processing unit on image data to which the first imaging condition is applied, and image processing may be performed by another image processing unit on image data to which the fourth imaging condition is applied. The number of image processing units is not limited to two, and may be the same as the number of imaging conditions that can be set. That is, each image processing unit is responsible for image processing for each region to which a different imaging condition is applied. According to the ninth modification, imaging under different imaging conditions for each region and image processing for image data of images obtained for each region can be performed in parallel.

[0227] (Variation 10) In the above description, the camera 1 has been used as an example, but it may also be configured as a highly functional mobile phone 250 (FIG. 21) equipped with a camera function like a smartphone, or a mobile device such as a tablet terminal.

[0228] (Variation 11) In the above-described embodiment, the camera 1 is described as an example in which the imaging unit 32 and the control unit 34 are configured as a single electronic device. Instead, for example, the imaging unit 32 and the control unit 34 may be provided separately, and the imaging system 1B may be configured in which the imaging unit 32 is controlled by the control unit 34 via communication. An example in which an imaging device 1001 including an imaging unit 32 is controlled by a control device 1002 including a control unit 34 will be described below with reference to FIG.

[0229] Fig. 20 is a block diagram illustrating the configuration of an imaging system 1B according to Modification 11. In Fig. 20, the imaging system 1B is configured with an imaging device 1001 and a display device 1002. The imaging device 1001 includes a first communication unit 1003 in addition to the imaging optical system 31 and imaging unit 32 described in the above embodiment. The display device 1002 includes a second communication unit 1004 in addition to the image processing unit 33, control unit 34, display unit 35, operation member 36, and recording unit 37 described in the above embodiment.

[0230] The first communication unit 1003 and the second communication unit 1004 can perform two-way image data communication using, for example, well-known wireless communication technology or optical communication technology. The imaging device 1001 and the display device 1002 may be connected by a wired cable, and the first communication unit 1003 and the second communication unit 1004 may perform bidirectional image data communication.

[0231] In the imaging system 1B, the control unit 34 controls the imaging unit 32 by performing data communication via the second communication unit 1004 and the first communication unit 1003. For example, by transmitting and receiving predetermined control data between the imaging device 1001 and the display device 1002, the display device 1002 divides the screen into a plurality of areas based on the image as described above, sets different imaging conditions for each divided area, and reads out photoelectric conversion signals photoelectrically converted in each area.

[0232] According to variant example 11, a live view image acquired by the imaging device 1001 and transmitted to the display device 1002 is displayed on the display unit 35 of the display device 1002, allowing the user to remotely control the imaging device 1001 from the display device 1002 located away from the imaging device 1001. The display device 1002 can be configured, for example, by a high-performance mobile phone 250 such as a smartphone. The imaging device 1001 can be configured by an electronic device including the stacked imaging element 100 described above. Although an example has been described in which the control unit 34 of the display device 1002 is provided with the object detection unit 34a, the setting unit 34b, the imaging control unit 34c, and the lens movement control unit 34d, some of the object detection unit 34a, the setting unit 34b, the imaging control unit 34c, and the lens movement control unit 34d may also be provided in the imaging device 1001.

[0233] (Variation 12) The program can be supplied to a mobile device such as the above-mentioned camera 1, high-function mobile phone 250, or tablet terminal by transmitting the program from a personal computer 205 storing the program to the mobile device via infrared communication or short-range wireless communication, as illustrated in FIG. 21, for example.

[0234] The program may be supplied to the personal computer 205 by inserting a recording medium 204 such as a CD-ROM storing the program into the personal computer 205, or by loading the program into the personal computer 205 via a communication line 201 such as a network. When the program is supplied via the communication line 201, the program is stored in a storage device 203 of a server 202 connected to the communication line.

[0235] The program can also be transmitted directly to the mobile device via a wireless LAN access point (not shown) connected to communication line 201. Furthermore, a recording medium 204B such as a memory card storing the program may be set in the mobile device. In this way, the program can be supplied as a computer program product in various forms, such as via a recording medium or a communication line.

[0236] ---Third embodiment--- 22 to 28, a digital camera will be described as an example of an electronic device equipped with an image processing device according to the third embodiment. In the following description, the same components as those in the first and second embodiments are denoted by the same reference numerals, and differences will be mainly described. Points not specifically described are the same as those in the first and second embodiments. This embodiment differs from the first and second embodiments mainly in that, instead of providing the image processing unit 33 of the first embodiment, the imaging unit 32A further includes an image processing unit 32c having the same functions as the image processing unit 33 of the first and second embodiments.

[0237] Fig. 22 is a block diagram illustrating the configuration of a camera 1C according to the third embodiment. In Fig. 22, camera 1C has an imaging optical system 31, an imaging unit 32A, a control unit 34, a display unit 35, an operation member 36, and a recording unit 37. The imaging unit 32A further includes an image processing unit 32c having the same function as the image processing unit 33 of the first embodiment.

[0238] The image processing unit 32c includes an input unit 321, a correction unit 322, and a generation unit 323. Image data from the imaging element 32a is input to the input unit 321. The correction unit 322 performs preprocessing to correct the input image data. The preprocessing performed by the correction unit 322 is the same as the preprocessing performed by the correction unit 33b in the first embodiment. The generation unit 323 performs image processing on the input image data and the image data after the preprocessing to generate an image. The image processing performed by the generation unit 323 is the same as the image processing performed by the generation unit 33c in the first embodiment.

[0239] Fig. 23 is a diagram schematically showing the correspondence between each block in this embodiment and multiple correction units 322. In Fig. 23, one square of the imaging chip 111, which is represented by a rectangle, represents one block 111a. Similarly, one square of the image processing chip 114, which will be described later and is represented by a rectangle, represents one correction unit 322.

[0240] In this embodiment, a correction unit 322 is provided for each block 111a. In other words, a correction unit 322 is provided for each block, which is the smallest unit of an area on the imaging surface where imaging conditions can be changed. For example, in FIG. 23, the hatched blocks 111a correspond to the hatched correction units 322. In FIG. 23, the hatched correction units 322 perform preprocessing on image data from pixels included in the hatched blocks 111a. Each correction unit 322 performs preprocessing on image data from pixels included in the corresponding block 111a. This allows pre-processing of image data to be performed in parallel by multiple correction units 322, thereby reducing the processing load on the correction units 322 and enabling appropriate images to be generated in a short period of time from image data generated in areas with different imaging conditions. In the following description, when explaining the relationship between a certain block 111a and the pixels included in that block 111a, that block 111a may be referred to as the block 111a to which that pixel belongs. Also, the block 111a may be referred to as a unit segment, and a collection of multiple blocks 111a, that is, a collection of multiple unit segments, may be referred to as a composite segment.

[0241] 24 is a cross-sectional view of the stacked imaging element 100A. The stacked imaging element 100A includes a back-illuminated imaging chip 111, a signal processing chip 112, a memory chip 113, and an image processing chip 114 that performs the above-mentioned pre-processing and image processing. That is, the above-mentioned image processing unit 32c is provided in the image processing chip 114. The imaging chip 111, signal processing chip 112, memory chip 113, and image processing chip 114 are stacked and electrically connected to one another by conductive bumps 109 such as Cu.

[0242] A plurality of bumps 109 are arranged on the opposing surfaces of the memory chip 113 and the image processing chip 114. These bumps 109 are aligned with each other, and the memory chip 113 and the image processing chip 114 are pressed together, whereby the aligned bumps 109 are bonded together and electrically connected.

[0243] <First correction process> As in the first embodiment, in the third embodiment, after the setting unit 34b divides the area of ​​the imaging screen, it is possible to set (change) imaging conditions for an area selected by the user or an area determined by the control unit 34. When the control unit 34 sets different imaging conditions for the divided areas, it causes the correction unit 322 of the image processing unit 32c to perform a first correction process as one of pre-processing steps as necessary.

[0244] Correction unit 322 of image processing unit 32c determines from which subject area light is incident on each pixel belonging to the boundary block. Specifically, correction unit 322 calculates the position of the boundary on the imaging surface of image sensor 32a based on the detection result of the subject element by object detection unit 34a. Correction unit 322 then extracts a boundary block based on the calculated boundary position, and calculates from which subject element subject light is incident on each pixel belonging to the extracted boundary block.

[0245] 7 and 8, the correction unit 322 calculates the position of the boundary B1 between the first region 61 and the fourth region 64 based on the detection results of the subject elements by the object detection unit 34a. Then, the correction unit 322 extracts blocks 82, 85, and 87 as boundary blocks based on the calculated position of the boundary B1. Then, based on the calculated position of the boundary B1, the correction unit 322 calculates that subject light from the person 61a is incident on pixels 82a, 82b, and 82c in the boundary block 82, pixel 85a in the boundary block 85, and pixel 87a in the boundary block 87. Furthermore, based on the calculated position of the boundary B1, the correction unit 322 calculates that subject light from the mountain 64a is incident on pixels 82d in the boundary block 82, pixels 85b, 85c, and 85d in the boundary block 85, and pixels 87b, 87c, and 87d in the boundary block 87.

[0246] Pixels 82a, 82b, and 82c of the boundary block 82 capture the subject light from the person 61a under a seventh imaging condition, which is an imaging condition between the first and fourth imaging conditions. On the other hand, each pixel of blocks 81 and 84, which are main blocks for the first region 61, captures the subject light from the person 61a under the first imaging condition. Therefore, the correction unit 322 performs a first correction process on the signals from the pixels 82a, 82b, and 82c so as to obtain signals similar to those obtained when the subject light from the person 61a is captured under the first imaging condition. Similarly, pixel 82d of boundary block 82 captures an image of subject light from mountain 64a under the seventh imaging condition. Meanwhile, pixels in blocks 83, 86, 88, and 89, which are main blocks for fourth region 64, capture an image of subject light from mountain 64a under the fourth imaging condition. Therefore, the correction unit 322 performs a first correction process on the signal from the pixel 82d so as to obtain a signal similar to that obtained when the subject light from the mountain 64a is captured under the fourth imaging condition.

[0247] For example, if only the ISO sensitivity differs between the first and fourth imaging conditions, with the ISO sensitivity of the first imaging condition being 100, the ISO sensitivity of the fourth imaging condition being 800, and the ISO sensitivity of the seventh imaging condition being 400, the first correction process is performed as follows: That is, as the first correction process, the correction unit 322 multiplies the signals from pixels 82a, 82b, and 82c by 100 / 400, and multiplies the signal from pixel 82d by 800 / 400.

[0248] For example, if only the shutter speed differs between the first and fourth imaging conditions, and the shutter speed of the first imaging condition is 1 / 1000 seconds, the shutter speed of the fourth imaging condition is 1 / 100 seconds, and the shutter speed of the seventh imaging condition is 1 / 500 seconds, the first correction process is performed as follows: That is, as the first correction process, the correction unit 322 multiplies the signals from pixels 82a, 82b, and 82c by (1 / 1000) / (1 / 500)=1 / 2, and multiplies the signal from pixel 82d by (1 / 100) / (1 / 500)=5.

[0249] For example, if only the frame rate differs between the first and fourth imaging conditions, with the frame rate under the first imaging condition being 30 fps, the frame rate under the fourth imaging condition being 60 fps, and the frame rate under the seventh imaging condition being 45 fps, the first correction process is performed as follows: Specifically, as the first correction process, the correction unit 322 thins out some of the frame image signals from pixels 82a, 82b, and 82c having a frame rate of 45 fps and converts them into frame image signals having a frame rate of 30 fps. This frame rate conversion is performed by selecting, from the frame image signals from pixels 82a-82c, a frame image signal whose generation timing is closest to the 30 fps frame image signal under the first imaging condition. Alternatively, the frame rate conversion may be performed by interpolating a frame image signal synchronized with the generation timing of the 30 fps frame image signal under the first imaging condition based on the frame image signals from pixels 82a-82c that are generated before and after the generation timing of the 30 fps frame image signal under the first imaging condition. Furthermore, as a first correction process, the correction unit 322 converts the frame image signal having a frame rate of 45 fps from pixel 82d into a frame image signal having a frame rate of 60 fps. This frame rate conversion is performed, for example, by combining frame image signals read out successively from pixel 82d, i.e., by interpolating and calculating a new frame image signal based on the previous and next frame image signals, thereby increasing the number of frame image signals.

[0250] In this way, the correction unit 322 performs the first correction process on signals from each pixel of all boundary blocks as needed. That is, the correction unit 322 performs the first correction process on signals from a pixel belonging to a boundary block if the imaging conditions applied to the main block for the same subject element as the pixel in question and the imaging conditions applied to the boundary block are different. However, if the imaging conditions applied to the main block for the same subject element as the pixel in question and the imaging conditions applied to the boundary block are the same, there is no need to perform the first correction process, and therefore the correction unit 322 does not perform the first correction process. As described above, even if there are some differences in the imaging conditions, they are considered to be the same imaging conditions.

[0251] <Second correction process> The control unit 34 further causes the correction unit 322 to perform the following second correction process as necessary before the image processing, focus detection process, subject detection (subject element detection) process, and process for setting the imaging conditions. In the second correction process, the signals from the pixels of the boundary block corrected by the first correction process are processed as signals obtained by imaging using the imaging conditions set for the main block, rather than the imaging conditions set for the boundary block.

[0252] 1. When performing image processing 1-1. When the imaging conditions of the pixel of interest P and the multiple reference pixels Pr surrounding the pixel of interest P are the same In this case, in the image processing unit 32c, the correction unit 322 does not perform the second correction process, and the generation unit 323 performs image processing using image data of a plurality of reference pixels Pr that have not undergone the second correction process.

[0253] 1-2. When the imaging conditions of the pixel of interest P and at least one of the reference pixels Pr surrounding the pixel of interest P are different The imaging conditions applied to the pixel of interest P are defined as the first imaging conditions, the imaging conditions applied to some of the reference pixels Pr are defined as the first imaging conditions, and the imaging conditions applied to the remaining reference pixels Pr are defined as the second imaging conditions. In this case, the correction unit 322 corresponding to the block 111a to which the reference pixel Pr to which the second imaging condition is applied belongs performs the second correction processing on the image data of the reference pixel Pr to which the second imaging condition is applied, as shown in the following (Example 1) to (Example 3). Then, the generation unit 323 performs image processing to calculate image data of the target pixel P by referring to the image data of the reference pixel Pr to which the first imaging condition is applied and the image data of the reference pixel Pr after the second correction processing.

[0254] (Example 1) For example, if only the ISO sensitivity differs between the first and second imaging conditions, and the ISO sensitivity under the first imaging condition is 100 and the ISO sensitivity under the second imaging condition is 800, the correction unit 322 corresponding to the block 111a to which the reference pixel Pr to which the second imaging condition is applied multiplies the image data of the reference pixel Pr by 100 / 800 as the second correction process, thereby reducing the difference between the image data due to the difference in imaging conditions.

[0255] (Example 2) For example, if only the shutter speed differs between the first and second imaging conditions, and the shutter speed under the first imaging condition is 1 / 1000 second and the shutter speed under the second imaging condition is 1 / 100 second, the correction unit 322 corresponding to the block 111a to which the reference pixel Pr to which the second imaging condition is applied multiplies the image data of the reference pixel Pr by 1 / 1000 / 1 / 100=1 / 10 as the second correction process, thereby reducing the difference between the image data due to the difference in imaging conditions.

[0256] (Example 3) The correction unit 322 corresponding to the block 111a to which the reference pixel Pr to which the second imaging condition is applied belongs performs the second correction process by adopting, for example, when only the frame rate differs between the first imaging condition and the second imaging condition (the charge accumulation time is the same) and the frame rate of the first imaging condition is 30 fps and the frame rate of the second imaging condition is 60 fps, the image data of the reference pixel Pr, i.e., the image data of the frame image acquired under the first imaging condition (30 fps) and the acquisition start timing of which are close to each other, thereby reducing the difference between the image data due to the difference in imaging conditions. The second correction process may involve interpolating and calculating image data for a frame image whose acquisition start timing is close to that of a frame image acquired under the first imaging condition (30 fps) based on multiple adjacent frame images acquired under the second imaging condition (60 fps).

[0257] The same applies when the imaging conditions applied to the pixel of interest P are the second imaging conditions and the imaging conditions applied to the reference pixels Pr surrounding the pixel of interest P are the first imaging conditions. That is, in this case, the correction unit 322 corresponding to the block 111a to which the reference pixel Pr to which the first imaging conditions are applied belongs performs the second correction process on the image data of the reference pixel Pr as in the above-mentioned (Example 1) to (Example 3).

[0258] As mentioned above, even if there are some differences in the imaging conditions, they are considered to be the same imaging conditions.

[0259] The generation unit 323 performs image processing such as pixel defect correction processing, color interpolation processing, contour enhancement processing, and noise reduction processing, similar to the generation unit 33c of the image processing unit 33 in the first embodiment, based on the image data of the reference pixel Pr to which the same imaging conditions as those of the target pixel P have been applied and the image data of the reference pixel Pr to which the second correction processing has been performed by the correction unit 322.

[0260] 25 is a diagram schematically illustrating processing of image data (hereinafter referred to as first image data) from each pixel included in a partial region of the imaging surface to which the first and third imaging conditions are applied (hereinafter referred to as first imaging region 141), and image data (hereinafter referred to as second image data) from each pixel included in a partial region of the imaging surface to which the second and third imaging conditions are applied (hereinafter referred to as second imaging region 142). The first imaging condition is an imaging condition set for a main block in the first imaging region 141, and the second imaging condition is an imaging condition set for a main block in the second imaging region 142. The third imaging condition is an imaging condition set for a boundary block including the boundary between the first imaging region 141 and the second imaging region 142.

[0261] Each pixel in the main block of the first imaging region 141 outputs a portion of the first image data captured under the first imaging condition, and each pixel in the boundary block belonging to the first imaging region 141 outputs the remaining portion of the first image data captured under the third imaging condition. Each pixel in the main block of the second imaging region 142 outputs a portion of the second image data captured under the second imaging condition, and each pixel in the boundary block belonging to the second imaging region 142 outputs the remaining portion of the second image data captured under the third imaging condition. The first image data is output to a correction unit 322 provided in the processing chip 114, which corresponds to the block 111a to which the pixel that generated the first image data belongs. In the following description, the multiple correction units 322 corresponding to the multiple blocks 111a to which the pixel that generated the first image data belongs will be referred to as a first processing unit 151. The first processing unit 151 performs the first correction processing and / or the second correction processing on the first image data as necessary.

[0262] Similarly, the second image data is output to a correction unit 322 provided in the processing chip 114 that corresponds to the block 111a to which the pixel that generated the second image data belongs. In the following description, the multiple correction units 322 that respectively correspond to the multiple blocks 111a to which the pixels that generated the respective second image data belong will be referred to as second processing units 152. The second processing unit 152 performs the first correction processing and / or the second correction processing on the second image data as necessary. Note that, when the second correction process is performed, it is assumed that the signals output from the pixels of the boundary blocks have been corrected by the first correction process. That is, when the second correction process is performed, the first image data is treated as if it had been obtained by capturing images under the first imaging conditions, and the second image data is treated as if it had been obtained by capturing images under the second imaging conditions.

[0263] In the above-described second correction process, for example, when the pixel of interest P is included in the first imaging region 141, the second image data from the reference pixel Pr included in the second imaging region 142 is subjected to the above-described second correction process by the second processing unit 152 as shown in Fig. 25. Note that the second processing unit 152 receives, for example, from the first processing unit 151, information 181 about the first imaging condition required to reduce the difference between the image data due to the difference in imaging condition. Similarly, for example, when the pixel of interest P is included in the second imaging region 142, the first image data from the reference pixel Pr included in the first imaging region 141 is subjected to the second correction processing described above by the first processing unit 151. Note that the first processing unit 151 receives information on the second imaging conditions required to reduce differences between image data due to differences in imaging conditions from the second processing unit 152.

[0264] When the pixel of interest P and the reference pixel Pr are included in the first imaging region 141, the first processing unit 151 does not perform the second correction process on the first image data from the reference pixel Pr. Similarly, when the pixel of interest P and the reference pixel Pr are included in the second imaging region 142, the second processing unit 152 does not perform the second correction process on the second image data from the reference pixel Pr. Alternatively, both the image data under the first imaging condition and the image data under the second imaging condition may be corrected by the first processing unit 151 and the second processing unit 152. That is, by performing the second correction process on the image data of the position of interest under the first imaging condition, the image data under the first imaging condition among the image data of the reference position, and the image data under the second imaging condition among the image data of the reference position, respectively, discontinuity of the images based on the difference between the first imaging condition and the second imaging condition may be alleviated. For example, in the above (Example 1), the image data of the reference pixel Pr under the first imaging condition (ISO sensitivity 100) is multiplied by 400 / 100 as the second correction process, and the image data of the reference pixel Pr under the second imaging condition (ISO sensitivity 800) is multiplied by 400 / 800 as the second correction process. This reduces the difference between the image data due to the difference in imaging conditions. The pixel data of the target pixel is subjected to a second correction process in which it is multiplied by 100 / 400 after color interpolation. This second correction process changes the pixel data of the target pixel after color interpolation to a value similar to that obtained when captured under the first imaging condition. Furthermore, in the above (Example 1), the degree of the second correction process may be changed depending on the distance from the boundary between the first and second regions. This reduces the rate at which the image data increases or decreases due to the second correction process compared to the case of the above (Example 1), thereby reducing noise caused by the second correction process. While the above description of the above (Example 1) has been given, the same applies to the above (Example 2).

[0265] The generation unit 323 performs image processing such as pixel defect correction processing, color interpolation processing, contour enhancement processing, and noise reduction processing based on the image data from the first processing unit 151 and the second processing unit 152, and outputs the image data after image processing.

[0266] When the pixel of interest P is located in the second imaging region 142, the first processing unit 151 may perform the second correction process on the first image data from all pixels included in the first imaging region 141, or may perform the second correction process only on the first image data from pixels included in the first imaging region 141 that have the potential to be used for interpolation of the pixel of interest P in the second imaging region 142. Similarly, when the pixel of interest P is located in the first imaging region 141, the second processing unit 152 may perform the second correction process on the second image data from all pixels included in the second imaging region 142, or may perform the second correction process only on the second image data from pixels included in the second imaging region 142 that have the potential to be used for interpolation of the pixel of interest P in the first imaging region 141.

[0267] 2. When performing focus detection processing As in the first embodiment, the lens movement control unit 34d of the control unit 34 performs focus detection processing using signal data (image data) corresponding to a predetermined position (focus point) on the imaging screen. Note that when different imaging conditions are set between the divided areas and the focus point for AF operation is located on the boundary between the divided areas, that is, when the focus point is divided into two areas, a first area and a second area, in this embodiment, as described in 2-2 below, the lens movement control unit 34d of the control unit 34 causes the correction unit 322 to perform second correction processing on the signal data for focus detection of at least one area. Note that, when the second correction process is performed, it is assumed that the signals output from each pixel of the boundary block have been corrected by the first correction process. That is, when the second correction process is performed, the first signal data from each pixel of the first imaging region 141 is treated as having been obtained by imaging under the first imaging conditions, and the second signal data from each pixel of the second imaging region 142 is treated as having been obtained by imaging under the second imaging conditions.

[0268] 2-1. When signal data from pixels within frame 170 in FIG. 15 does not include a mixture of signal data to which the first imaging condition is applied and signal data to which the second imaging condition is applied In this case, the correction unit 322 does not perform the second correction process, and the lens movement control unit 34d of the control unit 34 performs focus detection processing using signal data from the focus detection pixels indicated by the frame 170 as is.

[0269] 2-2. When signal data from pixels within frame 170 in FIG. 15 includes a mixture of signal data to which the first imaging condition is applied and signal data to which the second imaging condition is applied In this case, the lens movement control unit 34d of the control unit 34 causes the correction unit 322 corresponding to the block 111a to which the pixel to which the second imaging condition is applied belongs, among the pixels in the frame 170, to perform the second correction processing as shown in the following (Example 1) to (Example 3). Then, the lens movement control unit 34d of the control unit 34 performs focus detection processing using the signal data of the pixel to which the first imaging condition is applied and the signal data after the second correction processing.

[0270] (Example 1) For example, if only the ISO sensitivity differs between the first and second imaging conditions, and the ISO sensitivity of the first imaging condition is 100 and the ISO sensitivity of the second imaging condition is 800, the correction unit 322 corresponding to the block 111a to which the pixel to which the second imaging condition is applied multiplies the signal data of the second imaging condition by 100 / 800 as the second correction process, thereby reducing the difference between the signal data due to the difference in imaging conditions.

[0271] (Example 2) For example, if only the shutter speed differs between the first and second imaging conditions, and the shutter speed of the first imaging condition is 1 / 1000 second and the shutter speed of the second imaging condition is 1 / 100 second, the correction unit 322 corresponding to the block 111a to which the pixel to which the second imaging condition is applied multiplies the signal data of the second imaging condition by 1 / 1000 / 1 / 100=1 / 10 as a second correction process, thereby reducing the difference between the signal data due to the difference in imaging conditions.

[0272] (Example 3) The correction unit 322 corresponding to the block 111a to which the pixel to which the second imaging condition is applied belongs performs the second correction process by, for example, adopting signal data of a frame image acquired under the first imaging condition (30 fps) and having an acquisition start timing close to that of a frame image acquired under the first imaging condition (30 fps) when only the frame rate differs between the first imaging condition and the second imaging condition (the charge accumulation time is the same), thereby reducing the difference between the signal data due to the difference in imaging conditions. The second correction process may involve interpolating and calculating signal data for a frame image whose acquisition start timing is close to that of a frame image acquired under the first imaging condition (30 fps) based on multiple adjacent frame images acquired under the second imaging condition (60 fps).

[0273] As mentioned above, even if there are some differences in the imaging conditions, they are considered to be the same imaging conditions. In the above example, the second correction process is performed on the signal data under the second imaging condition among the signal data. However, the second correction process may be performed on the signal data under the first imaging condition among the signal data.

[0274] Furthermore, by performing a second correction process on the signal data under the first imaging condition and the signal data under the second imaging condition, the difference between the two sets of signal data after the second correction process may be reduced.

[0275] FIG. 26 is a diagram schematically showing the processing of the first signal data and the second signal data in the focus detection processing.

[0276] Each pixel in the main block of the first imaging region 141 outputs a portion of the first signal data captured under the first imaging condition, and each pixel in the boundary block belonging to the first imaging region 141 outputs the remaining portion of the first signal data captured under the third imaging condition. Each pixel in the main block of the second imaging region 142 outputs a portion of the second signal data captured under the second imaging condition, and each pixel in the boundary block belonging to the second imaging region 142 outputs the remaining portion of the second signal data captured under the third imaging condition. The first signal data from the first imaging region 141 is output to the first processing unit 151. Similarly, the second signal data from the second imaging region 142 is output to the second processing unit 152.

[0277] The first processing unit 151 performs the first correction processing and / or the second correction processing on the first image data as needed, and the second processing unit 152 performs the first correction processing and / or the first correction processing and / or the second correction processing on the second image data as needed. As described above, when performing the second correction process, all of the first signal data is treated as having been obtained by imaging with the first imaging conditions applied, and all of the second signal data is treated as having been obtained by imaging with the second imaging conditions applied.

[0278] In the above-described second correction process, when the second correction process is performed on signal data under the second imaging condition among the signal data, thereby reducing the difference between the signal data after the second correction process and the signal data under the first imaging condition, the second processing unit 152 performs the process. The second processing unit 152 performs the above-described second correction process on the second signal data from pixels included in the second imaging region 142. Note that the second processing unit 152 receives, for example, from the first processing unit 151, information 181 about the first imaging condition required to reduce the difference between the signal data due to the difference in imaging condition. In addition, when performing the second correction processing on the signal data under the second imaging condition among the signal data to reduce the difference between the signal data after the second correction processing and the signal data under the first imaging condition, the first processing unit 151 does not perform the second correction processing on the first signal data.

[0279] Furthermore, when performing a second correction process on signal data under the first imaging condition among the signal data to reduce the difference between the signal data after the second correction process and the signal data under the first imaging condition, the first processing unit 151 performs the process. The first processing unit 151 performs the above-described second correction process on the first signal data from pixels included in the first imaging region 141. The first processing unit 151 receives information about the second imaging condition required to reduce the difference between the signal data due to the difference in imaging condition from the second processing unit 152. In addition, when performing the second correction processing on the signal data under the first imaging condition among the signal data to reduce the difference between the signal data after the second correction processing and the signal data under the first imaging condition, the second processing unit 152 does not perform the second correction processing on the second signal data.

[0280] Furthermore, when performing second correction processing on the signal data under the first imaging condition and the signal data under the second imaging condition among the signal data to reduce the difference between both sets of signal data after the second correction processing, the first processing unit 151 and the second processing unit 152 perform processing. The first processing unit 151 performs the above-mentioned second correction processing on the first signal data from pixels included in the first imaging region 141, and the second processing unit 152 performs the above-mentioned second correction processing on the second signal data from pixels included in the second imaging region 142.

[0281] The lens movement control unit 34d performs focus detection processing based on signal data from the first processing unit 151 and the second processing unit 152, and based on the calculation results, outputs a drive signal to move the focus lens of the imaging optical system 31 to the in-focus position.

[0282] 3. When performing subject detection processing When different imaging conditions are set between the divided areas and the search range 190 includes the boundaries of the divided areas, in this embodiment, as described in 3-2 below, the object detection unit 34a of the control unit 34 causes the correction unit 322 to perform a second correction process on the image data of at least one area within the search range 190. Note that, when the second correction process is performed, it is assumed that the signals output from the pixels of the boundary blocks have been corrected by the first correction process. That is, when the second correction process is performed, the first image data is treated as having been obtained by imaging under the first imaging conditions, and the second image data is treated as having been obtained by imaging under the second imaging conditions.

[0283] 3-1. When image data to which the first imaging condition is applied and image data to which the second imaging condition is applied are not mixed in the image data of the search range 190 in FIG. 16 In this case, the correction unit 322 does not perform the second correction process, and the object detection unit 34a of the control unit 34 performs the subject detection process using the image data that constitutes the search range 190 as is.

[0284] 3-2. When image data in the search range 190 in FIG. 16 includes both image data to which the first imaging condition is applied and image data to which the second imaging condition is applied In this case, the object detection unit 34a of the control unit 34 causes the correction unit 322 corresponding to the block 111a to which the pixel to which the second imaging condition is applied belongs, in the image of the search range 190, to perform the second correction processing as in (Example 1) to (Example 3) described above as the case of performing focus detection processing. Then, the object detection unit 34a of the control unit 34 performs the subject detection processing using the image data of the pixel to which the first condition is applied and the image data after the second correction processing.

[0285] FIG. 27 is a diagram schematically showing the processing of the first image data and the second image data in the subject detection processing.

[0286] In the second correction processing, the second correction processing performed by the first processing unit 151 and / or the second processing unit 152 is the same as the second correction processing in FIG. 26 described above in connection with the case where focus detection processing is performed. The object detection unit 34a performs processing to detect subject elements based on the image data from the first processing unit 151 and the second processing unit 152, and outputs the detection results.

[0287] 4. Setting imaging conditions This section explains a case where the area of ​​the image capture screen is divided, different imaging conditions are set for each divided area, and then new photometry is performed to determine the exposure conditions. Note that when determining the exposure conditions, it is assumed that the signals output from each pixel in the boundary block have been corrected by the first correction process.

[0288] 4-1. When image data in the photometric range includes a mixture of image data to which the first imaging condition is applied and image data to which the second imaging condition is applied In this case, the correction unit 322 does not perform the second correction process, and the setting unit 34b of the control unit 34 performs exposure calculation processing using the image data that constitutes the photometric range as is.

[0289] 4-2. When image data in the photometric range includes both image data to which the first imaging condition was applied and image data to which the second imaging condition was applied In this case, the setting unit 34b of the control unit 34 causes the correction unit 322 corresponding to the block 111a to which the pixel to which the second imaging condition is applied belongs, among the image data of the photometric range, to perform the second correction process as in (Example 1) to (Example 3) described above for performing focus detection process. Then, the setting unit 34b of the control unit 34 performs exposure calculation process using the image data after the second correction process.

[0290] FIG. 28 is a diagram schematically showing the processing of the first image data and the second image data in relation to the setting of imaging conditions such as exposure calculation processing.

[0291] In the second correction processing, the second correction processing performed by the first processing unit 151 and / or the second processing unit 152 is the same as the second correction processing in FIG. 26 described above in connection with the case where focus detection processing is performed. The setting unit 34b performs calculation processing of imaging conditions such as exposure calculation processing based on image data from the first processing unit 151 and the second processing unit 152, and based on the calculation results, divides the image captured by the imaging unit 32 into multiple areas including the detected subject elements, and resets the imaging conditions for the multiple areas.

[0292] According to the third embodiment described above, the following advantageous effects can be obtained. (1) The image processing unit 32c has a first processing unit 151 that generates image data of the subject imaged in the first imaging region 141, and a second processing unit 152 that generates image data of the subject imaged in the second imaging region 142. The first processing unit 151 generates image data of the subject imaged in the first imaging region 141 from the image data of the subject imaged in the second imaging region 142. This allows pre-processing (first correction processing, second correction processing) of the image data to be performed in parallel by multiple correction units 322, thereby reducing the processing load on the correction units 322.

[0293] (2) The image processing unit 32c has a first processing unit 151 that generates a signal based on the object that entered the first imaging region 141, and a second processing unit 152 that generates a signal based on the object that entered the second imaging region 142. The first processing unit 151 generates a signal based on the object that was imaged in the first imaging region 141 from a signal based on the object that entered the second imaging region 142. This allows pre-processing of image data to be performed in parallel by multiple correction units 322, thereby reducing the processing load on the correction units 322 and, because the pre-processing by the multiple correction units 322 is performed in a short time by parallel processing, the time until the focus detection process in the lens movement control unit 34d starts can be shortened, contributing to faster focus detection process.

[0294] ---Modification of the third embodiment--- The following modifications are also within the scope of the present invention, and one or more of the modifications may be combined with the above-described embodiment. (Variation 13) In the third embodiment described above, one correction unit 322 corresponds to one of the blocks 111a (unit segments). However, one correction unit 322 may correspond to one of the composite blocks (composite segments) having a plurality of blocks 111a (unit segments). In this case, the correction unit 322 sequentially corrects image data from pixels belonging to the plurality of blocks 111a included in the composite block. Even if a plurality of correction units 322 are provided corresponding to each composite block having a plurality of blocks 111a, the second correction process of the image data can be processed in parallel by the plurality of correction units 322. This reduces the processing load on the correction units 322, and enables an appropriate image to be generated in a short time from image data generated in areas with different imaging conditions.

[0295] (Variation 14) In the third embodiment described above, the generation unit 323 is provided inside the imaging unit 32A. However, the generation unit 323 may be provided outside the imaging unit 32A. Even if the generation unit 323 is provided outside the imaging unit 32A, the same effects as those described above can be achieved.

[0296] (Variation 15) In the third embodiment described above, the stacked imaging element 100A further includes an image processing chip 114 that performs the pre-processing and image processing described above, in addition to the back-illuminated imaging chip 111, the signal processing chip 112, and the memory chip 113. However, the stacked imaging element 100A may not include the image processing chip 114, and the image processing unit 32c may be provided in the signal processing chip 112.

[0297] (Variation 16) In the third embodiment described above, the second processing unit 152 receives information about the first imaging conditions necessary to reduce the difference between the image data due to the difference in the imaging conditions from the first processing unit 151. Furthermore, the first processing unit 151 receives information about the second imaging conditions necessary to reduce the difference between the image data due to the difference in the imaging conditions from the second processing unit 152. However, the second processing unit 152 may receive information about the first imaging conditions necessary to reduce the difference between the image data due to the difference in the imaging conditions from the driving unit 32b or the control unit 34. Similarly, the first processing unit 151 may receive information about the second imaging conditions necessary to reduce the difference between the image data due to the difference in the imaging conditions from the driving unit 32b or the control unit 34. The above-described embodiments and modifications may be combined with each other.

[0298] The imaging optical system 31 described above may include a zoom lens or a tilt lens. The lens movement control unit 34d adjusts the angle of view of the imaging optical system 31 by moving the zoom lens in the optical axis direction. In other words, by moving the zoom lens, it is possible to adjust the image of the imaging optical system 31, such as obtaining an image of a wide range of subjects or obtaining a large image of a distant subject. Furthermore, the lens movement control unit 34d can adjust distortion of the image caused by the imaging optical system 31 by moving the tilt lens in a direction perpendicular to the optical axis. Then, in order to adjust the state of the image (for example, the state of the angle of view or the state of image distortion) obtained by the imaging optical system 31, it is preferable to perform the above-mentioned preprocessing based on the idea that it is preferable to use image data after the above-mentioned preprocessing.

[0299] Although various embodiments and modifications have been described above, the present invention is not limited to these. Other embodiments that are conceivable within the scope of the technical idea of ​​the present invention are also included within the scope of the present invention. [Explanation of symbols]

[0300] 1,1C...Camera 1B...imaging system 31...imaging optical system 32...imaging unit 32a, 100...imaging element 33...Image processing unit 33a, 321...input section 33b,322...Correction section 33c,323...Generation part 34...Control unit 34a...Object detection unit 34b…Setting section 34c...imaging control unit 34d...Lens movement control section 35...Display section 80...predetermined range 90...Area of ​​interest 1001...imaging device 1002...Display device P...pixel of interest

Claims

[Claim 1] an imaging element capable of setting imaging conditions for each block including a plurality of pixels; an imaging condition setting unit that sets a first imaging condition for the block in a first region into which a first light from a first object is incident, sets a second imaging condition for the block in a second region into which a second light from a second object is incident, and sets an imaging condition different from the first imaging condition and the second imaging condition for the block in a third region between the first region and the second region, The imaging condition setting unit sets the number of imaging conditions to be set in the third area based on a setting value set by the first imaging condition and a setting value set by the second imaging condition.

Images