Imaging apparatus

Abstract

To provide an imaging apparatus for performing proper processing, while considering an imaging condition, when processing image data respectively generated by areas having different imaging conditions.SOLUTION: An imaging apparatus 1 includes: an imaging element 32a which has a first imaging area for imaging a subject and outputting a first signal and a second imaging area for imaging the subject and outputting a second signal; a setting part 34b which sets an imaging condition of the first imaging area to an imaging condition different from an imaging condition of the second imaging area; a correction part 33b which performs correction used for interpolation of the first signal output from the first imaging area to the second signal output from the second imaging area; and a generation part 33c which generates an image of the subject imaged by the first imaging area by using a signal obtained by interpolating the first signal by the second signal corrected by the correction part and generates an image of the subject imaged by the second imaging area by using the second signal to which the correction used for the interpolation of the first signal is not performed.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an imaging device. [Background technology]

[0002] There is known an imaging device equipped with an imaging element that can set different imaging conditions for each area of ​​a screen (see Patent Document 1). However, the imaging conditions are not taken into consideration when processing image data generated in areas with different imaging conditions. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2006-197192 Summary of the Invention

[0004] According to a first aspect, an imaging device includes an imaging element having a first imaging area that images a subject and outputs a first signal, and a second imaging area that images a subject and outputs a second signal, a setting unit that sets imaging conditions of the first imaging area to imaging conditions different from imaging conditions of the second imaging area, a correction unit that corrects the second signal output from the second imaging area for use in interpolating the first signal output from the first imaging area, and a generation unit that generates an image of the subject imaged in the first imaging area using a signal obtained by interpolating the first signal with the second signal corrected by the correction unit, and generates an image of the subject imaged in the second imaging area using the second signal that has not been corrected for use in interpolating the first signal. [Brief description of the drawings]

[0005] [Figure 1] 1 is a block diagram illustrating a configuration of a camera according to a first embodiment. [Diagram 2] FIG. 2 is a cross-sectional view of a stacked type imaging element. [Diagram 3]2 is a diagram illustrating a pixel array and a unit area of ​​an imaging chip. FIG. [Figure 4] FIG. 2 is a diagram illustrating a circuit in a unit area. [Diagram 5] FIG. 2 is a diagram illustrating an example of an image of a subject formed on an image sensor of a camera. [Figure 6] FIG. 13 is a diagram illustrating an example of a setting screen for imaging conditions. [Figure 7] FIG. 7(a) is a diagram illustrating the vicinity of the boundary of the first region in a live view image, FIG. 7(b) is an enlarged view of the vicinity of the boundary, and FIG. 7(c) is an enlarged view of a pixel of interest and a reference pixel. [Figure 8] Figure 8(a) is a diagram illustrating an example of an arrangement of photoelectric conversion signals output from pixels, Figure 8(b) is a diagram explaining the interpolation of image data of the G color component, and Figure 8(c) is a diagram illustrating image data of the G color component after interpolation. [Figure 9] FIG. 9(a) is a diagram showing image data of the R color component extracted from FIG. 8(a), FIG. 9(b) is a diagram explaining the interpolation of the color difference component Cr, and FIG. 9(c) is a diagram explaining the interpolation of image data of the color difference component Cr. [Figure 10] FIG. 10(a) is a diagram showing image data of the B color component extracted from FIG. 8(a), FIG. 10(b) is a diagram explaining the interpolation of the color difference component Cb, and FIG. 10(c) is a diagram explaining the interpolation of image data of the color difference component Cb. [Figure 11] 4 is a diagram illustrating an example of the positions of focus detection pixels on an imaging surface; [Figure 12] FIG. 4 is an enlarged view of a partial area of ​​a focus detection pixel line. [Figure 13] FIG. 4 is an enlarged view of the focus detection positions. [Figure 14] FIG. 14(a) is a diagram illustrating a template image representing an object to be detected, and FIG. 14(b) is a diagram illustrating a live view image and a search range. [Figure 15] 11 is a flowchart illustrating a process of setting imaging conditions for each region and capturing an image. [Figure 16] 16(a) to 16(c) are diagrams illustrating examples of the arrangement of the first region and the second region on the imaging surface of an imaging element. [Figure 17] FIG. 23 is a block diagram illustrating the configuration of an imaging system according to an eleventh modification. [Figure 18] FIG. 2 is a diagram illustrating the supply of a program to a mobile device. [Figure 19] FIG. 11 is a block diagram illustrating the configuration of a camera according to a second embodiment. [Figure 20] 13 is a diagram illustrating a schematic diagram of a correspondence relationship between each block and a plurality of correction units according to the second embodiment. FIG. [Figure 21] FIG. 2 is a cross-sectional view of a stacked imaging element. [Figure 22] 3A to 3C are diagrams illustrating a schematic diagram of processing of first image data and second image data relating to image processing. [Figure 23] 5A to 5C are diagrams illustrating the processing of first image data and second image data in relation to focus detection processing. [Figure 24] 5A to 5C are diagrams illustrating the processing of first image data and second image data in relation to the subject detection processing. [Diagram 25] 5 is a diagram showing a schematic representation of processing of first image data and second image data relating to setting of imaging conditions such as exposure calculation processing. FIG. [Figure 26] 23 is a diagram showing a schematic diagram of processing of the first image data and the second image data according to a thirteenth modification example. FIG. [Figure 27] 23 is a diagram showing a schematic diagram of processing of the first image data and the second image data according to a thirteenth modification example. FIG. [Figure 28] 23 is a diagram showing a schematic diagram of processing of the first image data and the second image data according to a thirteenth modification example. FIG. [Figure 29] 23 is a diagram showing a schematic diagram of processing of the first image data and the second image data according to a thirteenth modification example. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] ---First embodiment --- A digital camera will be described as an example of an electronic device equipped with an image processing device according to the first embodiment. Camera 1 (FIG. 1) is configured to be able to capture images under different conditions for each region of the imaging surface of imaging element 32a. Image processing unit 33 performs appropriate processing for each region with different imaging conditions. Details of camera 1 will be described with reference to the drawings.

[0007] <Camera description> Fig. 1 is a block diagram illustrating the configuration of a camera 1 according to the first embodiment. In Fig. 1, camera 1 has an imaging optical system 31, an imaging unit 32, an image processing unit 33, a control unit 34, a display unit 35, an operation member 36, and a recording unit 37.

[0008] The imaging optical system 31 guides a light beam from the subject field to the imaging section 32. The imaging section 32 includes an imaging element 32a and a driving section 32b, and photoelectrically converts the image of the subject formed by the imaging optical system 31. The imaging section 32 can capture images under the same conditions over the entire imaging surface of the imaging element 32a, or capture images under different conditions for each region of the imaging surface of the imaging element 32a. The imaging section 32 will be described in detail later. The driving section 32b generates a driving signal required for the imaging element 32a to perform accumulation control. Imaging instructions such as charge accumulation time for the imaging section 32 are transmitted from the control section 34 to the driving section 32b.

[0009] The image processing unit 33 includes an input unit 33a, a correction unit 33b, and a generation unit 33c. The input unit 33a receives image data acquired by the imaging unit 32. The correction unit 33b performs pre-processing to correct the input image data. The details of the pre-processing will be described later. The generation unit 33c generates an image based on the input image data and the image data after the pre-processing. The generation unit 33c also performs image processing on the image data. The image processing includes, for example, color interpolation processing, pixel defect correction processing, contour enhancement processing, noise reduction processing, white balance adjustment processing, gamma correction processing, display brightness adjustment processing, saturation adjustment processing, and the like. Furthermore, the generation unit 33c generates an image to be displayed by the display unit 35.

[0010] The control unit 34 is constituted by, for example, a CPU, and controls the overall operation of the camera 1. For example, the control unit 34 performs a predetermined exposure calculation based on the photoelectric conversion signal acquired by the imaging unit 32, and determines exposure conditions such as the charge accumulation time (exposure time) of the image sensor 32a required for proper exposure, the aperture value of the imaging optical system 31, and ISO sensitivity, and instructs the driving unit 32b. In addition, the control unit 34 determines image processing conditions for adjusting saturation, contrast, sharpness, etc. according to the imaging scene mode set in the camera 1 and the type of the detected subject element, and instructs the image processing unit 33. Detection of the subject element will be described later.

[0011] The control unit 34 includes an object detection unit 34a, a setting unit 34b, an imaging control unit 34c, and an AF calculation unit 34d. These are realized in software by the control unit 34 executing a program stored in a non-volatile memory (not shown), but they may also be configured by ASIC or the like.

[0012] The object detection unit 34a performs a known object recognition process to detect subject elements such as people (human faces), animals (animal faces) such as dogs and cats, plants, vehicles such as bicycles, automobiles, and trains, buildings, stationary objects, landscapes such as mountains and clouds, and specific objects determined in advance from the image acquired by the imaging unit 32. The setting unit 34b divides the image captured by the imaging unit 32 into a plurality of regions including the subject elements detected as described above.

[0013] The setting unit 34b further sets imaging conditions for the multiple regions. The imaging conditions include the above-mentioned exposure conditions (charge accumulation time, gain, ISO sensitivity, frame rate, etc.) and the above-mentioned image processing conditions (for example, white balance adjustment parameters, gamma correction curve, display brightness adjustment parameters, saturation adjustment parameters, etc.). It is possible to set the same imaging conditions for all of the multiple regions, or to set different imaging conditions between the multiple regions.

[0014] The imaging control unit 34c applies the imaging conditions set for each region by the setting unit 34b to control the imaging unit 32 (imaging element 32a) and image processing unit 33. This makes it possible for the imaging unit 32 to perform imaging under different exposure conditions for each of the multiple regions, and for the image processing unit 33 to perform image processing under different image processing conditions for each of the multiple regions. The number of pixels constituting each region may be any number, and may be, for example, 1000 pixels or 1 pixel. Furthermore, the number of pixels may differ between regions.

[0015] The AF calculation unit 34d controls an automatic focus adjustment (autofocus: AF) operation for focusing on a corresponding subject at a predetermined position (called a focus detection position) on the imaging screen. Based on the calculation result, the AF calculation unit 34d sends a drive signal to the drive unit 32b for moving the focus lens of the imaging optical system 31 to a focus position. The process performed by the AF calculation unit 34d for automatic focus adjustment is also called a focus detection process. The focus detection process will be described in detail later.

[0016] The display unit 35 reproduces and displays images generated by the image processing unit 33, images that have been subjected to image processing, images that have been read out by the recording unit 37, etc. The display unit 35 also displays an operation menu screen, a setting screen for setting imaging conditions, etc.

[0017] The operation members 36 are composed of various operation members such as a release button, a menu button, etc. The operation members 36 send operation signals corresponding to each operation to the control unit 34. The operation members 36 also include a touch operation member provided on the display surface of the display unit 35.

[0018] The recording unit 37 records image data and the like in a recording medium such as a memory card (not shown) in response to an instruction from the control unit 34. The recording unit 37 also reads out image data recorded in the recording medium in response to an instruction from the control unit 34.

[0019] <Explanation of stacked image sensor> As an example of the above-mentioned imaging element 32a, a stacked imaging element 100 will be described. FIG. 2 is a cross-sectional view of the imaging element 100. The imaging element 100 includes an imaging chip 111, a signal processing chip 112, and a memory chip 113. The imaging chip 111 is stacked on the signal processing chip 112. The signal processing chip 112 is stacked on the memory chip 113. The imaging chip 111 and the signal processing chip 112, and the signal processing chip 112 and the memory chip 113 are electrically connected by a connection portion 109. The connection portion 109 is, for example, a bump or an electrode. The imaging chip 111 captures a light image from a subject and generates image data. The imaging chip 111 outputs the image data from the imaging chip 111 to the signal processing chip 112. The signal processing chip 112 performs signal processing on the image data output from the imaging chip 111. The memory chip 113 has a plurality of memories and stores the image data. The imaging element 100 may be composed of an imaging chip and a signal processing chip. When the imaging element 100 is composed of an imaging chip and a signal processing chip, a storage unit for storing image data may be provided in the signal processing chip or may be provided separately from the imaging element 100.

[0020] As shown in Figure 2, incident light mainly travels in the positive Z-axis direction, as indicated by the white arrow. Also, as shown by the coordinate axes, the left direction on the paper, perpendicular to the Z-axis, is the positive X-axis direction, and the front direction on the paper, perpendicular to the Z-axis and X-axis, is the positive Y-axis direction. In the following figures, the coordinate axes are displayed so that the orientation of each figure can be understood based on the coordinate axes in Figure 2.

[0021] The imaging chip 111 is, for example, a CMOS image sensor. Specifically, the imaging chip 111 is a back-illuminated CMOS image sensor. The imaging chip 111 has a microlens layer 101, a color filter layer 102, a passivation layer 103, a semiconductor layer 106, and a wiring layer 108. In the imaging chip 111, the microlens layer 101, the color filter layer 102, the passivation layer 103, the semiconductor layer 106, and the wiring layer 108 are arranged in this order in the positive direction of the Z axis.

[0022] The microlens layer 101 has a plurality of microlenses L. The microlenses L focus incident light on a photoelectric conversion unit 104, which will be described later. The color filter layer 102 has a plurality of color filters F. The color filter layer 102 has a plurality of types of color filters F with different spectral characteristics. Specifically, the color filter layer 102 has a first filter (R) with a spectral characteristic that mainly transmits red component light, a second filter (Gb, Gr) with a spectral characteristic that mainly transmits green component light, and a third filter (B) with a spectral characteristic that mainly transmits blue component light. In the color filter layer 102, the first filter, the second filter, and the third filter are arranged, for example, in a Bayer array. The passivation layer 103 is made of a nitride film or an oxide film, and protects the semiconductor layer 106.

[0023] The semiconductor layer 106 has a photoelectric conversion unit 104 and a readout circuit 105. The semiconductor layer 106 has a plurality of photoelectric conversion units 104 between a first surface 106a, which is a light incidence surface, and a second surface 106b opposite to the first surface 106a. The semiconductor layer 106 has a plurality of photoelectric conversion units 104 arranged in the X-axis direction and the Y-axis direction. The photoelectric conversion unit 104 has a photoelectric conversion function of converting light into electric charge. The photoelectric conversion unit 104 accumulates electric charge due to a photoelectric conversion signal. The photoelectric conversion unit 104 is, for example, a photodiode. The semiconductor layer 106 has a readout circuit 105 on the second surface 106b side of the photoelectric conversion unit 104. The semiconductor layer 106 has a plurality of readout circuits 105 arranged in the X-axis direction and the Y-axis direction. The readout circuit 105 is configured with a plurality of transistors, and reads out image data generated from charges photoelectrically converted by the photoelectric conversion unit 104, and outputs the image data to the wiring layer .

[0024] The wiring layer 108 has a plurality of metal layers. The metal layers are, for example, Al wiring, Cu wiring, etc. The wiring layer 108 outputs image data read by the readout circuit 105. The image data is output from the wiring layer 108 to the signal processing chip 112 via the connection unit 109.

[0025] The connection portion 109 may be provided for each photoelectric conversion portion 104. Furthermore, the connection portion 109 may be provided for each of a plurality of photoelectric conversion portions 104. When the connection portion 109 is provided for each of a plurality of photoelectric conversion portions 104, the pitch of the connection portions 109 may be larger than the pitch of the photoelectric conversion portions 104. Furthermore, the connection portions 109 may be provided in a peripheral region of the region in which the photoelectric conversion portions 104 are arranged.

[0026] The signal processing chip 112 has a plurality of signal processing circuits. The signal processing circuits perform signal processing on the image data output from the imaging chip 111. The signal processing circuits are, for example, an amplifier circuit that amplifies the signal value of the image data, a correlated double sampling circuit that performs noise reduction processing on the image data, and an analog / digital (A / D) conversion circuit that converts an analog signal into a digital signal. A signal processing circuit may be provided for each photoelectric conversion unit 104.

[0027] Furthermore, a signal processing circuit may be provided for each of the multiple photoelectric conversion units 104. The signal processing chip 112 has multiple through electrodes 110. The through electrodes 110 are, for example, silicon through electrodes. The through electrodes 110 connect the circuits provided in the signal processing chip 112 to each other. The through electrodes 110 may also be provided in the peripheral region of the imaging chip 111 and in the memory chip 113. Note that some of the elements constituting the signal processing circuit may be provided in the imaging chip 111. For example, in the case of an analog / digital conversion circuit, a comparator that compares an input voltage with a reference voltage may be provided in the imaging chip 111, and circuits such as a counter circuit and a latch circuit may be provided in the signal processing chip 112.

[0028] The memory chip 113 has a plurality of storage units. The storage units store image data that has been subjected to signal processing by the signal processing chip 112. The storage units are, for example, volatile memories such as DRAMs. A storage unit may be provided for each photoelectric conversion unit 104. Also, a storage unit may be provided for each of the plurality of photoelectric conversion units 104. The image data stored in the storage units is output to a downstream image processing unit.

[0029] FIG. 3 is a diagram for explaining the pixel array of the imaging chip 111 and the unit area 131. In particular, the imaging chip 111 is shown as observed from the back surface (imaging surface) side. For example, 20 million or more pixels are arranged in a matrix in the pixel area. In the example of FIG. 3, four pixels, 2 pixels by 2 pixels adjacent to each other, form one unit area 131. The grid lines in the figure show the concept of forming the unit area 131 by grouping adjacent pixels. The number of pixels forming the unit area 131 is not limited to this, and may be about 1000, for example, 32 pixels by 32 pixels, or may be more or less than that, or may be 1 pixel.

[0030] As shown in the partially enlarged view of the pixel region, the unit region 131 in Fig. 3 includes a so-called Bayer array consisting of four pixels: green pixels Gb, Gr, blue pixel B, and red pixel R. The green pixels Gb and Gr are pixels having a green filter as a color filter F, and receive light in the green wavelength band of incident light. Similarly, the blue pixel B is a pixel having a blue filter as a color filter F, and receives light in the blue wavelength band, and the red pixel R is a pixel having a red filter as a color filter F, and receives light in the red wavelength band.

[0031] In this embodiment, a plurality of blocks are defined so that each block includes at least one unit area 131. That is, the minimum unit of one block is one unit area 131. As described above, the smallest possible value of the number of pixels forming one unit area 131 is one pixel. Therefore, when one block is defined in pixel units, the smallest possible number of pixels that can define one block is one pixel. Each block can control the pixels included in each block with different control parameters. In each block, all unit areas 131 in that block, that is, all pixels in that block are controlled under the same imaging conditions. That is, photoelectric conversion signals with different imaging conditions can be obtained between a pixel group included in one block and a pixel group included in another block. Examples of the control parameters are a frame rate, a gain, a thinning rate, the number of summation rows or columns for summing photoelectric conversion signals, the charge accumulation time or number of accumulations, the number of digitization bits (word length), and the like. The imaging element 100 can freely perform thinning not only in the row direction (X-axis direction of the imaging chip 111) but also in the column direction (Y-axis direction of the imaging chip 111). Furthermore, the control parameters may be parameters for image processing.

[0032] Fig. 4 is a diagram for explaining a circuit in a unit area 131. In the example of Fig. 4, one unit area 131 is formed by four pixels, 2 pixels x 2 pixels adjacent to each other. As described above, the number of pixels included in the unit area 131 is not limited to this, and may be 1000 pixels or more, or may be as small as 1 pixel. The two-dimensional position of the unit area 131 is indicated by symbols A to D.

[0033] The reset transistors (RST) of the pixels included in the unit region 131 are configured to be able to be turned on and off individually for each pixel. In Fig. 4, a reset wiring 300 is provided to turn on and off the reset transistor of pixel A, and a reset wiring 310 to turn on and off the reset transistor of pixel B is provided separately from the reset wiring 300. Similarly, a reset wiring 320 to turn on and off the reset transistor of pixel C is provided separately from the reset wirings 300 and 310. A dedicated reset wiring 330 is also provided for another pixel D to turn on and off the reset transistor.

[0034] The transfer transistors (TX) of the pixels included in the unit region 131 are also configured to be able to be turned on and off individually for each pixel. In Fig. 4, a transfer wiring 302 that turns on and off the transfer transistor of pixel A, a transfer wiring 312 that turns on and off the transfer transistor of pixel B, and a transfer wiring 322 that turns on and off the transfer transistor of pixel C are provided separately. For another pixel D, a dedicated transfer wiring 332 is also provided to turn on and off the transfer transistor.

[0035] Furthermore, the selection transistors (SEL) of the pixels included in the unit region 131 are also configured to be able to be turned on and off individually for each pixel. In Fig. 4, a selection wiring 306 that turns on and off the selection transistor of pixel A, a selection wiring 316 that turns on and off the selection transistor of pixel B, and a selection wiring 326 that turns on and off the selection transistor of pixel C are provided separately. For another pixel D, a dedicated selection wiring 336 is also provided to turn on and off the selection transistor.

[0036] The power supply wiring 304 is commonly connected to pixels A to D included in the unit area 131. Similarly, the output wiring 308 is commonly connected to pixels A to D included in the unit area 131. The power supply wiring 304 is commonly connected between a plurality of unit areas, but the output wiring 308 is provided individually for each unit area 131. The load current source 309 supplies a current to the output wiring 308. The load current source 309 may be provided on the imaging chip 111 side or on the signal processing chip 112 side.

[0037] By individually turning on and off the reset transistors and transfer transistors of the unit area 131, it is possible to control charge accumulation including the accumulation start time, accumulation end time, and transfer timing for pixels A to D included in the unit area 131. In addition, by individually turning on and off the selection transistors of the unit area 131, it is possible to output the photoelectric conversion signals of each of pixels A to D via a common output wiring 308.

[0038] Here, a so-called rolling shutter method is known in which charge accumulation is controlled in a regular order for rows and columns for pixels A to D included in the unit area 131. When pixels are selected for each row using the rolling shutter method and then columns are specified, photoelectric conversion signals are output in the order of "ABCD" in the example of FIG.

[0039] In this way, by configuring the circuit based on the unit area 131, it is possible to control the charge accumulation time for each unit area 131. In other words, it is possible to output photoelectric conversion signals at different frame rates between the unit areas 131. Also, by resting the unit areas 131 included in some blocks of the imaging chip 111 while causing the unit areas 131 included in other blocks to accumulate charge (imaging), it is possible to cause imaging to be performed only in a predetermined block of the imaging chip 111 and output the photoelectric conversion signal. Furthermore, it is also possible to switch the block (block subject to accumulation control) that accumulates charge (imaging) between frames, and cause different blocks of the imaging chip 111 to sequentially capture images and output photoelectric conversion signals.

[0040] As described above, output wiring 308 is provided corresponding to each unit area 131. Since image pickup element 100 has imaging chip 111, signal processing chip 112, and memory chip 113 stacked on top of each other, by using electrical connection between the chips using connection parts 109 for output wiring 308, it is possible to route wiring without increasing the size of each chip in the planar direction.

[0041] <Image sensor block control> In this embodiment, imaging conditions can be set for each of a plurality of blocks in the image sensor 32a. The control unit 34 (imaging control unit 34c) associates the plurality of regions with the blocks and performs imaging under the imaging conditions set for each region.

[0042] 5 is a diagram showing a schematic diagram of an image of a subject formed on the imaging element 32a of the camera 1. Before an image capturing instruction is issued, the camera 1 photoelectrically converts the subject image to obtain a live view image. A live view image is an image for monitoring that is repeatedly captured at a predetermined frame rate (e.g., 60 fps).

[0043] Before the setting unit 34b divides the region, the control unit 34 sets the same imaging conditions for the entire imaging chip 111 (i.e., the entire imaging screen). The same imaging conditions refer to setting common imaging conditions for the entire imaging screen, and are considered to be the same even if there is a variation of less than about 0.3 stops in the Apex value, for example. The imaging conditions set to be the same for the entire imaging chip 111 are determined based on the exposure conditions according to the photometric value of the subject brightness, or the exposure conditions manually set by the user.

[0044] 5, an image including a person 61a, a car 62a, a bag 63a, a mountain 64a, and clouds 65a and 66a is formed on the imaging surface of the imaging chip 111. The person 61a is holding the bag 63a with both hands. A car 62a is parked to the right rear of the person 61a.

[0045] <Dividing the area> Based on the live view image, the control unit 34 divides the screen of the live view image into a plurality of regions as follows. First, the object detection unit 34a detects subject elements from the live view image. The detection of the subject elements uses a known subject recognition technique. In the example of Fig. 5, the object detection unit 34a detects a person 61a, a car 62a, a bag 63a, a mountain 64a, a cloud 65a, and a cloud 66a as subject elements.

[0046] Next, the setting unit 34b divides the screen of the live view image into regions including the above-mentioned subject elements. In this embodiment, the region including the person 61a is region 61, the region including the car 62a is region 62, the region including the bag 63a is region 63, the region including the mountain 64a is region 64, the region including the cloud 65a is region 65, and the region including the cloud 66a is region 66.

[0047] <Setting imaging conditions for each block> When the control unit 34 divides the screen into a plurality of regions by the setting unit 34b, the control unit 34 causes the display unit 35 to display a setting screen as exemplified in Fig. 6. In Fig. 6, a live view image 60a is displayed, and a setting screen 70 for setting imaging conditions is displayed to the right of the live view image 60a.

[0048] The setting screen 70 lists, from the top, frame rate, shutter speed (TV), and gain (ISO) as examples of setting items for imaging conditions. The frame rate is the number of frames of live view images captured per second or moving images recorded by the camera 1. The gain is the ISO sensitivity. Setting items for imaging conditions may be added as appropriate in addition to those exemplified in FIG. 6. If all setting items do not fit within the setting screen 70, other setting items may be displayed by scrolling the setting items up and down.

[0049] In this embodiment, the control unit 34 sets an area selected by the user among the areas divided by the setting unit 34b as a target for setting (changing) the imaging conditions. For example, in the camera 1 capable of touch operation, the user taps the display position of the main subject for which the imaging conditions are to be set (changed) on the display surface of the display unit 35 on which the live view image 60a is displayed. For example, when the display position of a person 61a is tapped, the control unit 34 sets the area 61 including the person 61a in the live view image 60a as a target area for setting (changing) the imaging conditions, and displays the outline of the area 61 with emphasis.

[0050] In FIG. 6, an area 61 that is displayed with an emphasized outline (bold, bright, changed color, dashed line, blinking, etc.) indicates an area for which imaging conditions are to be set (changed). In the example of FIG. 6, a live view image 60a is displayed with an emphasized outline of the area 61. In this case, the area 61 is an area for which imaging conditions are to be set (changed). For example, in a camera 1 that is capable of touch operation, when the user taps on the shutter speed (TV) display 71, the control unit 34 causes the current setting value of the shutter speed for the emphasized area (area 61) to be displayed on the screen (reference numeral 68). In the following explanation, the camera 1 will be described on the assumption that touch operations are performed. However, the imaging conditions may be set (changed) by operating the buttons and the like that constitute the operation member .

[0051] When the user taps the upper icon 71a or the lower icon 71b of the shutter speed (TV), the setting unit 34b increases or decreases the shutter speed display 68 from the current setting value in response to the tap operation, and sends an instruction to the imaging unit 32 (FIG. 1) to change the imaging conditions of the unit area 131 (FIG. 3) of the imaging element 32a corresponding to the highlighted area (area 61) in response to the tap operation. The decision icon 72 is an operation icon for finalizing the set imaging conditions. The setting unit 34b also sets (changes) the frame rate and gain (ISO) in the same way as the setting (change) of the shutter speed (TV).

[0052] Although the setting unit 34b has been described as setting the imaging conditions based on the user's operation, the present invention is not limited to this. The setting unit 34b may set the imaging conditions based on the judgment of the control unit 34, not based on the user's operation. For example, when whiteout or blackout occurs in an area including a subject element that has the maximum or minimum brightness in an image, the setting unit 34b may set the imaging conditions based on the judgment of the control unit 34 so as to eliminate the whiteout or blackout. For areas that are not highlighted (areas other than the area 61), the imaging conditions that have been set are maintained.

[0053] Instead of highlighting the outline of the region for which the imaging conditions are to be set (changed), the control unit 34 may display the entire target region brightly, display the entire target region with increased contrast, or display the entire target region with blinking. The target region may also be surrounded by a frame. The frame surrounding the target region may be displayed as a double frame or a single frame, and the display mode of the surrounding frame, such as the line type, color, brightness, etc., may be changed as appropriate. The control unit 34 may also display an arrow or the like near the target region to indicate the region for which the imaging conditions are to be set. The control unit 34 may display the region other than the target region for which the imaging conditions are to be set (changed) darkly, or display the region other than the target region with reduced contrast.

[0054] As described above, after the imaging conditions for each region are set, when a release button (not shown) constituting the operation member 36 or a display (release icon) instructing the start of imaging is operated, the control unit 34 controls the imaging unit 32 to capture images under the imaging conditions set for each of the divided regions. Then, the image processing unit 33 performs image processing on the image data acquired by the imaging unit 32. As described above, the image processing can be performed under different image processing conditions for each region.

[0055] After the image processing by the image processing unit 33, the recording unit 37 receives an instruction from the control unit 34 and records the image data after the image processing in a recording medium such as a memory card (not shown). This completes the series of imaging processes.

[0056] <Correction process> As described above, in this embodiment, after the area of ​​the imaging screen is divided by the setting unit 34b, it is possible to set (change) imaging conditions for an area selected by the user or an area determined by the control unit 34. When different imaging conditions are set for the divided areas, the control unit 34 performs the following correction processing as necessary.

[0057] 1. When performing image processing When the image processing for the image data acquired by applying different imaging conditions between the divided regions is a predetermined image processing, the image processing unit 33 (correction unit 33b) performs correction processing on the image data located near the boundary of the regions as pre-processing of the image processing. The predetermined image processing is a process for calculating image data of a target position to be processed in the image by referring to image data of multiple reference positions around the target position, and corresponds to, for example, pixel defect correction processing, color interpolation processing, edge enhancement processing, noise reduction processing, etc.

[0058] The correction process is performed to reduce discontinuity that occurs in the image after image processing due to different imaging conditions between the divided regions. In general, when the focus position is located near the boundary between the divided regions, image data to which the same imaging conditions as the image data of the focus position are applied and image data to which imaging conditions different from the image data of the focus position are applied may be mixed at multiple reference positions around the focus position. In this embodiment, the correction process is performed as follows based on the idea that it is preferable to calculate the image data of the focus position by referring to the image data of the reference position that has been subjected to a correction process so as to suppress the difference between the image data due to the difference in imaging conditions, rather than calculating the image data of the focus position by directly referring to the image data of the reference position to which different imaging conditions are applied.

[0059] FIG. 7(a) is a diagram illustrating an example of an area 80 near the boundary between an area 61 and an area 64 in a live view image 60a. In this example, it is assumed that a first imaging condition is set in the area 61 including at least a person, and a second imaging condition is set in the area 64 including a mountain. FIG. 7(b) is an enlarged view of the area 80 near the boundary in FIG. 7(a). Image data from pixels on the image sensor 32a corresponding to the area 61 to which the first imaging condition is set is shown in white, and image data from pixels on the image sensor 32a corresponding to the area 64 to which the second imaging condition is set is shown in shaded areas. In FIG. 7(b), image data from a pixel of interest P is located in the area 61 near a boundary 81 between the area 61 and the area 64, that is, in the boundary portion. Pixels (eight pixels in this example) surrounding the pixel of interest P included in a predetermined range 90 (e.g., 3×3 pixels) centered on the pixel of interest P are used as reference pixels. FIG. 7(c) is an enlarged view of the pixel of interest P and the reference pixels. The position of the pixel of interest P is the target position, and the positions of the reference pixels surrounding the pixel of interest P are the reference positions.

[0060] The image processing unit 33 (generation unit 33c) usually performs image processing by directly referring to the image data of the reference pixels without performing correction processing. However, when the imaging condition (first imaging condition) applied to the pixel of interest P is different from the imaging condition (second imaging condition) applied to the reference pixels around the pixel of interest P, the correction unit 33b performs correction processing on the image data of the second imaging condition among the image data of the reference pixels as shown in the following (Example 1) to (Example 3). Then, the generation unit 33c performs image processing to calculate the image data of the pixel of interest P by referring to the image data of the reference pixels after the correction processing. In FIG. 7(c), the image data output from the pixels shown in white is the image data of the first imaging condition, and the image data output from the pixels shown in diagonal lines is the image data of the second imaging condition.

[0061] (Example 1) For example, when only the ISO sensitivity differs between the first and second imaging conditions, and the ISO sensitivity of the first imaging condition is 100 and the ISO sensitivity of the second imaging condition is 800, the image processing unit 33 (correction unit 33b) applies 100 / 800 as a correction process to the image data of the reference pixels under the second imaging condition, thereby reducing the difference between the image data due to the difference in imaging conditions. Note that, when the amount of light incident on the pixel of interest P is the same as the amount of light incident on the reference pixel, the difference in image data becomes small, but when the amount of light incident on the pixel of interest P is originally different from the amount of light incident on the reference pixel, the difference in image data may not become small. The same applies to the examples described later.

[0062] (Example 2) For example, when only the shutter speed differs between the first and second imaging conditions, and the shutter speed of the first imaging condition is 1 / 1000 seconds and the shutter speed of the second imaging condition is 1 / 100 seconds, the image processing unit 33 (correction unit 33b) applies 1 / 1000 / 1 / 100=1 / 10 to the image data of the reference pixels under the second imaging condition as a correction process, thereby reducing the difference between the image data due to the difference in imaging conditions.

[0063] (Example 3) For example, when only the frame rate differs between the first and second imaging conditions (the charge accumulation time is the same), the frame rate of the first imaging condition is 30 fps, and the frame rate of the second imaging condition is 60 fps, the image processing unit 33 (correction unit 33b) performs correction processing to adopt image data of a frame image acquired under the first imaging condition (30 fps) and having an acquisition start timing close to that of the frame image acquired under the first imaging condition (30 fps) for the image data of the reference pixels under the second imaging condition (60 fps). This reduces the difference between the image data due to the difference in imaging conditions. In addition, the correction process may involve interpolating and calculating image data for a frame image whose acquisition start timing is close to that of a frame image acquired under the first imaging condition (30 fps) based on multiple adjacent frame images acquired under the second imaging condition (60 fps).

[0064] On the other hand, the image processing unit 33 (correction unit 33b) does not perform correction processing on the image data of the reference pixels when the imaging conditions (first imaging conditions) applied to the pixel of interest P are the same as the imaging conditions (second imaging conditions) applied to all reference pixels surrounding the pixel of interest P. In other words, the generation unit 33c performs image processing to calculate the image data of the pixel of interest P by directly referring to the image data of the reference pixels. As described above, even if there are some differences in the imaging conditions, they are considered to be the same imaging conditions.

[0065] <Image processing example> Image processing involving correction processing will be illustrated. (1) Pixel defect correction process In this embodiment, pixel defect correction processing is one of the image processing performed during imaging. In general, the imaging element 32a, which is a solid-state imaging element, may have pixel defects during or after the manufacturing process, and may output image data of an abnormal level. Therefore, the image processing unit 33 (generation unit 33c) corrects the image data output from the pixel where the pixel defect occurred, so that the image data at the pixel position where the pixel defect occurred is not noticeable.

[0066] An example of pixel defect correction processing will be described. The image processing unit 33 (generation unit 33c) sets a pixel at the position of a pixel defect, which is recorded in advance in a non-volatile memory (not shown) in an image of one frame, as a pixel of interest P (a pixel to be processed), and sets pixels (eight pixels in this example) around the pixel of interest P included in a predetermined range 90 (e.g., 3×3 pixels) centered on the pixel of interest P as reference pixels.

[0067] The image processing unit 33 (generation unit 33c) performs Max, Min filter processing to calculate the maximum and minimum values ​​of the image data of the reference pixels, and replaces the image data output from the target pixel P with the maximum or minimum value when the image data output from the target pixel P exceeds the maximum or minimum value. This processing is performed for all pixel defects whose position information is recorded in a non-volatile memory (not shown).

[0068] In the present embodiment, when the pixel to which the second imaging condition different from the first imaging condition applied to the target pixel P is included in the reference pixel, the image processing unit 33 (correction unit 33b) performs correction processing on the image data to which the second imaging condition is applied. Then, the image processing unit 33 (generation unit 33c) performs the above-described Max, Min filter processing.

[0069] (2) Color interpolation processing In the present embodiment, the color interpolation processing is one of the image processes performed during imaging. As illustrated in FIG. 3, in the imaging chip 111 of the image sensor 100, green pixels Gb, Gr, blue pixels B, and red pixels R are arranged in a Bayer array. Since the image data of the color components different from the color components of the color filter F arranged at each pixel position is insufficient, the image processing unit 33 (generation unit 33c) performs color interpolation processing to generate the image data of the insufficient color components by referring to the image data of the peripheral pixel positions.

[0070] An example of the color interpolation processing will be described. FIG. 8(a) is a diagram illustrating the arrangement of the image data output from the image sensor 32a. Corresponding to each pixel position, it has a color component of either R, G, or B according to the rule of the Bayer array. <G color interpolation> First, general G color interpolation will be described. The image processing unit 33 (generation unit 33c) that performs G color interpolation sequentially sets the positions of the R color component and the B color component as the target positions, and generates the image data of the G color component at the target position by referring to the image data of the four G color components at the reference positions around the target position. For example, when generating the image data of the G color component at the target position indicated by the thick frame in FIG. 8(b) (the second row and the second column counted from the upper left position; hereinafter, the target position counted from the upper left position is represented in the same way), the image data G1 to G4 of the four G color components located in the vicinity of the target position (the second row and the second column) are referred to. The image processing unit 33 (generation unit 33c) sets, for example, (aG1 + bG2 + cG3 + dG4) / 4 as the image data of the G color component at the target position (the second row and the second column). Note that a to d are weighting coefficients provided according to the distance between the reference position and the target position and the image structure.

[0071] Next, the G color interpolation of this embodiment will be described. In FIGS. 8(a) to 8(c), it is assumed that the first imaging condition is applied to the regions to the left and above the thick line, and the second imaging condition is applied to the regions to the right and below the thick line. Note that in FIGS. 8(a) to 8(c), the first imaging condition and the second imaging condition are different. Also, the image data G1 to G4 of the G color component in FIG. 8(b) are reference positions for image processing of the pixel at the attention position (the second row and the second column). In FIG. 8(b), the first imaging condition is applied to the attention position (the second row and the second column). Among the reference positions, the first imaging condition is applied to the image data G1 to G3. Also, among the reference positions, the second imaging condition is applied to the image data G4. Therefore, the image processing unit 33 (correction unit 33b) performs correction processing on the image data G4. After that, the image processing unit 33 (generation unit 33c) calculates the image data of the G color component at the attention position (the second row and the second column).

[0072] The image processing unit 33 (generation unit 33c) can obtain the image data of the G color component at each pixel position as shown in FIG. 8(c) by generating the image data of the G color component at the positions of the B color component and the R color component in FIG. 8(a).

[0073] <R color interpolation> FIG. 9(a) is a diagram in which the image data of the R color component is extracted from FIG. 8(a). The image processing unit 33 (generation unit 33c) calculates the image data of the color difference component Cr shown in FIG. 9(b) based on the image data of the G color component shown in FIG. 8(c) and the image data of the R color component shown in FIG. 9(a).

[0074] First, a general interpolation of the color difference component Cr will be described. When generating image data of the color difference component Cr at the target position shown in the thick frame (second row, second column) in FIG. 9(b), for example, the image processing unit 33 (generating unit 33c) refers to image data Cr1 to Cr4 of four color difference components located near the target position (second row, second column). The image processing unit 33 (generating unit 33c) sets, for example, (eCr1+fCr2+gCr3+hCr4) / 4 as the image data of the color difference component Cr at the target position (second row, second column). Note that e to h are weighting coefficients set according to the distance between the reference position and the target position and the image structure.

[0075] Similarly, when generating image data of the color difference component Cr at the position of interest shown in the thick frame (second row, third column) in FIG. 9(c), the image processing unit 33 (generation unit 33c) refers to image data Cr2, Cr4 to Cr6 of the four color difference components located near the position of interest (second row, third column). The image processing unit 33 (generation unit 33c) determines, for example, (qCr2+rCr4+sCr5+tCr6) / 4 as the image data of the color difference component Cr at the position of interest (second row, third column). Here, q to t are weighting coefficients set according to the distance between the reference position and the position of interest and the image structure. In this way, image data of the color difference component Cr is generated for each pixel position.

[0076] Next, the interpolation of the color difference component Cr in this embodiment will be described. In Fig. 9(a) to Fig. 9(c), for example, the first imaging condition is applied to the left and upper regions of the thick line, and the second imaging condition is applied to the right and lower regions of the thick line. Note that the first imaging condition and the second imaging condition are different in Fig. 9(a) to Fig. 9(c). In Fig. 9(b), the position indicated by the thick frame (second row, second column) is the target position of the color difference component Cr. Also, the image data Cr1 to Cr4 of the color difference components in Fig. 9(b) are reference positions for image processing the pixel at the target position (second row, second column). In Fig. 9(b), the first imaging condition is applied to the target position (second row, second column). Of the reference positions, the first imaging condition is applied to the image data Cr1, Cr3, and Cr4. Also, of the reference positions, the second imaging condition is applied to the image data Cr2. Therefore, the image processing unit 33 (correction unit 33b) performs correction processing on the image data Cr2. After that, the image processing unit 33 (the generation unit 33c) calculates image data of the color difference component Cr at the target position (second row, second column). In FIG. 9(c), the position indicated by the thick frame (second row, third column) is the target position of the color difference component Cr. In FIG. 9(c), the image data Cr2, Cr4, Cr5, and Cr6 of the color difference components are reference positions for image processing the pixel at the target position (second row, third column). In FIG. 9(c), the second imaging condition is applied to the target position (second row, third column). Among the reference positions, the image data Cr4 and Cr5 are applied with the first imaging condition. Among the reference positions, the image data Cr2 and Cr6 are applied with the second imaging condition. Therefore, the image processing unit 33 (correction unit 33b) performs correction processing on the image data Cr4 and Cr5, respectively. After that, the image processing unit 33 (generation unit 33c) calculates the image data of the color difference component Cr at the target position (second row, third column).

[0077] The image processing unit 33 (generation unit 33c) obtains image data of the color difference component Cr at each pixel position, and then adds the image data of the G color component shown in Figure 8 (c) corresponding to each pixel position, thereby obtaining image data of the R color component at each pixel position.

[0078] FIG. 10(a) is a diagram obtained by extracting the image data of the B color component from FIG. 8(a). The image processing unit 33 (generation unit 33c) calculates the image data of the color difference component Cb shown in FIG. 10(b) based on the image data of the G color component shown in FIG. 8(c) and the image data of the B color component shown in FIG. 10(a).

[0079] First, the interpolation of the general color difference component Cb will be described. When the image processing unit 33 (generation unit 33c) generates the image data of the color difference component Cb at the attention position indicated by the thick frame (the third row and the third column) in FIG. 10(b), for example, it refers to the image data Cb1 to Cb4 of the four color difference components located in the vicinity of the attention position (the third row and the third column). The image processing unit 33 (generation unit 33c) uses, for example, (uCb1 + vCb2 + wCb3 + xCb4) / 4 as the image data of the color difference component Cb at the attention position (the third row and the third column). Here, u to x are weight coefficients provided according to the distance between the reference position and the attention position and the image structure.

[0080] Similarly, when the image processing unit 33 (generation unit 33c) generates the image data of the color difference component Cb at the attention position indicated by the thick frame (the third row and the fourth column) in FIG. 10(c), for example, it refers to the image data Cb2, Cb4 to Cb6 of the four color difference components located in the vicinity of the attention position (the third row and the fourth column). The image processing unit 33 (generation unit 33c) uses, for example, (yCb2 + zCb4 + αCb5 + βCb6) / 4 as the image data of the color difference component Cb at the attention position (the third row and the fourth column). Here, y, z, α, and β are weight coefficients provided according to the distance between the reference position and the attention position and the image structure. In this way, the image data of the color difference component Cb is generated for each pixel position.

[0081] Next, the interpolation of the color difference component Cb in this embodiment will be described. In Fig. 10(a) to Fig. 10(c), for example, the first imaging condition is applied to the left and upper regions of the thick line, and the second imaging condition is applied to the right and lower regions of the thick line. Note that the first imaging condition and the second imaging condition are different in Fig. 10(a) to Fig. 10(c). In Fig. 10(b), the position indicated by the thick frame (third row, third column) is the target position of the color difference component Cb. Also, the image data Cb1 to Cb4 of the color difference components in Fig. 10(b) are reference positions for image processing the pixel at the target position (third row, third column). In Fig. 10(b), the second imaging condition is applied to the target position (third row, third column). Of the reference positions, the first imaging condition is applied to the image data Cb1 and Cb3. Also, of the reference positions, the second imaging condition is applied to the image data Cb2 and Cb4. Therefore, the image processing unit 33 (correction unit 33b) performs correction processing on each of the data Cb1 and Cb3. After that, the image processing unit 33 (generation unit 33c) calculates image data of the color difference component Cb at the target position (third row, third column). In addition, in FIG. 10(c), the position indicated by the thick frame (third row, fourth column) is the target position of the chrominance component Cb. In addition, the image data Cb2, Cb4 to Cb6 of the chrominance components in FIG. 10(c) are reference positions for performing image processing on the pixel at the target position (third row, fourth column). In FIG. 10(c), the second imaging condition is applied to the target position (third row, fourth column). In addition, the second imaging condition is applied to the image data Cb2, Cb4 to Cb6 at all reference positions. Therefore, the image processing unit 33 (generation unit 33c) calculates the image data of the chrominance component Cb at the target position (third row, fourth column) by referring to the image data Cb2, Cb4 to Cb6 at the reference positions that have not been corrected by the image processing unit 33 (correction unit 33b).

[0082] The image processing unit 33 (generation unit 33c) obtains image data of the color difference component Cb at each pixel position, and then adds the image data of the G color component shown in Figure 8 (c) corresponding to each pixel position, thereby obtaining image data of the B color component at each pixel position. In the above-mentioned "G color interpolation", for example, when generating image data of G color components at the attention position shown in the thick frame (second row, second column) in FIG. 8(b), image data G1 to G4 of four G color components located near the attention position are referred to, but the number of image data of G color components to be referred to may be changed depending on the image structure. For example, when the image near the attention position has similarity in the vertical direction (for example, a vertical stripe pattern), the interpolation process is performed using only the image data above and below the attention position (G1 and G2 in FIG. 8(b)). Also, when the image near the attention position has similarity in the horizontal direction (for example, a horizontal stripe pattern), the interpolation process is performed using only the image data to the left and right of the attention position (G3 and G4 in FIG. 8(b)). In these cases, the image data G4 to be corrected by the correction unit 33b may or may not be used.

[0083] (3) Edge enhancement processing An example of the contour enhancement process will be described. The image processing unit 33 (generation unit 33c) performs a known linear filter operation using a kernel of a predetermined size centered on a pixel of interest P (a pixel to be processed) in an image of one frame, for example. When the kernel size of a sharpening filter, which is an example of a linear filter, is N×N pixels, the position of the pixel of interest P is the position of interest, and the positions of (N2-1) reference pixels surrounding the pixel of interest P are the reference positions. The kernel size may be N×M pixels.

[0084] The image processing unit 33 (generation unit 33c) performs a filter process to replace the image data at the target pixel P with the result of a linear filter calculation, for example, by shifting the target pixel from left to right on each horizontal line from the top horizontal line to the bottom horizontal line of the frame image.

[0085] In this embodiment, the image processing unit 33 (correction unit 33b) performs a correction process on the image data to which the second imaging condition is applied when the reference pixels include a pixel to which a second imaging condition different from the first imaging condition applied to the pixel of interest P is applied. After that, the image processing unit 33 (generation unit 33c) performs the above-mentioned linear filter process.

[0086] (4) Noise reduction processing An example of noise reduction processing will be described. The image processing unit 33 (generation unit 33c) performs a known linear filter calculation using a kernel of a predetermined size centered on a pixel of interest P (a pixel to be processed) in an image of one frame, for example. When the kernel size of a smoothing filter, which is an example of a linear filter, is N×N pixels, the position of the pixel of interest P is the position of interest, and the positions of (N2-1) reference pixels surrounding the pixel of interest P are the reference positions. The kernel size may be N×M pixels.

[0087] The image processing unit 33 (generation unit 33c) performs a filter process to replace the image data at the target pixel P with the result of a linear filter calculation, for example, by shifting the target pixel from left to right on each horizontal line from the top horizontal line to the bottom horizontal line of the frame image.

[0088] In this embodiment, the image processing unit 33 (correction unit 33b) performs a correction process on the image data to which the second imaging condition is applied when the reference pixels include a pixel to which a second imaging condition different from the first imaging condition applied to the pixel of interest P is applied. After that, the image processing unit 33 (generation unit 33c) performs the above-mentioned linear filter process.

[0089] 2. When performing focus detection processing The control unit 34 (AF calculation unit 34d) performs focus detection processing using signal data (image data) corresponding to a predetermined position (focus detection position) on the imaging screen. When different imaging conditions are set between the divided areas and the focus detection position for the AF operation is located at the boundary between the divided areas, the control unit 34 (AF calculation unit 34d) performs correction processing on the signal data for focus detection of at least one area as pre-processing for the focus detection processing.

[0090] The correction process is performed to suppress a decrease in the accuracy of the focus detection process due to differences in imaging conditions between the regions of the imaging screen divided by the setting unit 34b. For example, when signal data for focus detection at a focus detection position where an image shift amount (phase difference) is detected in an image is located near the boundary of the divided regions, signal data for focus detection may contain signal data to which different imaging conditions are applied. In this embodiment, the correction process is performed as follows based on the idea that it is preferable to detect the image shift amount (phase difference) using signal data that has been subjected to a correction process so as to suppress differences between signal data due to differences in imaging conditions, rather than detecting the image shift amount (phase difference) using signal data to which different imaging conditions are applied as is.

[0091] <Example of focus detection processing> An example of focus detection processing accompanied by correction processing will be described. In the AF operation of this embodiment, for example, the focus is adjusted to a subject corresponding to a focus detection position selected by a user from among multiple focus detection positions on an imaging screen. The control unit 34 (AF calculation unit 34d (generation unit)) calculates the defocus amount of the imaging optical system 31 by detecting the image shift amount (phase difference) of multiple subject images generated by light beams passing through different pupil regions of the imaging optical system 31. The control unit 34 (AF calculation unit 34d) moves the focus lens of the imaging optical system 31 to a position where the defocus amount is zero (below a permissible value), that is, to a focused position.

[0092] Fig. 11 is a diagram illustrating the positions of focus detection pixels on the imaging surface of the image sensor 32a. In this embodiment, focus detection pixels are arranged discretely along the X-axis direction (horizontal direction) of the imaging chip 111. In the example of Fig. 11, 15 focus detection pixel lines 160 are provided at predetermined intervals. The focus detection pixels constituting the focus detection pixel lines 160 output photoelectric conversion signals for focus detection. In the imaging chip 111, normal imaging pixels are provided at pixel positions other than the focus detection pixel lines 160. The imaging pixels output photoelectric conversion signals for live view images and recording.

[0093] Fig. 12 is an enlarged view of a partial area of ​​the focus detection pixel line 160 corresponding to the focus detection position 80A shown in Fig. 11. In Fig. 12, red pixels R, green pixels G (Gb, Gr), and blue pixels B, focus detection pixels S1, and focus detection pixels S2 are illustrated. The red pixels R, green pixels G (Gb, Gr), and blue pixels B are arranged according to the Bayer array rules described above.

[0094] The square regions illustrated for the red pixel R, green pixel G (Gb, Gr), and blue pixel B indicate the light receiving regions of the imaging pixels. Each imaging pixel receives a light beam that passes through the exit pupil of the imaging optical system 31 (FIG. 1). That is, the red pixel R, green pixel G (Gb, Gr), and blue pixel B each have a square mask opening, and light that passes through these mask openings reaches the light receiving section of the imaging pixel.

[0095] The shape of the light receiving regions (mask openings) of the red pixel R, the green pixel G (Gb, Gr), and the blue pixel B is not limited to a rectangle, and may be, for example, a circle.

[0096] The semicircular regions illustrated for the focus detection pixels S1 and S2 indicate the light receiving regions of the focus detection pixels. That is, the focus detection pixel S1 has a semicircular mask opening on the left side of the pixel position in FIG. 12, and light passing through this mask opening reaches the light receiving portion of the focus detection pixel S1. On the other hand, the focus detection pixel S2 has a semicircular mask opening on the right side of the pixel position in FIG. 12, and light passing through this mask opening reaches the light receiving portion of the focus detection pixel S2. In this way, the focus detection pixels S1 and S2 each receive a pair of light beams that pass through different regions of the exit pupil of the imaging optical system 31 (FIG. 1).

[0097] The positions of the focus detection pixel lines 160 in the imaging chip 111 are not limited to the positions exemplified in Fig. 11. Furthermore, the number of focus detection pixel lines 160 is not limited to the example in Fig. 11. Furthermore, the shape of the mask openings in the focus detection pixels S1 and S2 is not limited to a semicircular shape, and may be, for example, a rectangular shape obtained by horizontally dividing the square-shaped light receiving regions (mask openings) in the imaging pixels R, G, and B.

[0098] Furthermore, the focus detection pixel line 160 in the imaging chip 111 may be formed by arranging focus detection pixels along the Y-axis direction (vertical direction) of the imaging chip 111. An imaging element in which imaging pixels and focus detection pixels are two-dimensionally arranged as in Fig. 12 is well known, and detailed illustration and description of these pixels will be omitted.

[0099] 12, a so-called 1PD structure has been described in which each of the focus detection pixels S1 and S2 receives one of a pair of light beams for focus detection. Alternatively, a so-called 2PD structure may be used in which each of the focus detection pixels receives both of a pair of light beams for focus detection. The 2PD structure makes it possible to use the photoelectric conversion signal obtained by the focus detection pixel as a photoelectric conversion signal for recording.

[0100] The control unit 34 (AF calculation unit 34d) detects the image shift amount (phase difference) of a pair of images formed by a pair of light beams passing through different regions of the imaging optical system 31 (FIG. 1) based on the photoelectric conversion signals for focus detection output from the focus detection pixel S1 and the focus detection pixel S2. Then, the control unit 34 calculates the defocus amount based on the image shift amount (phase difference). Such defocus amount calculation using the pupil division phase difference method is well known in the field of cameras, so a detailed description will be omitted.

[0101] It is assumed that focus detection position 80A (FIG. 11) is selected by the user at a position corresponding to area 80 near the boundary of area 61 in the live view image 60a illustrated in FIG. 7(a). FIG. 13 is an enlarged view of focus detection position 80A. White pixels indicate that the first imaging condition is set, and shaded pixels indicate that the second imaging condition is set. The position surrounded by a frame 170 in FIG. 13 corresponds to focus detection pixel line 160 (FIG. 11).

[0102] The control unit 34 (AF calculation unit 34d) normally performs focus detection processing using signal data from the focus detection pixels indicated by the frame 170 as is, without performing correction processing. However, when the signal data surrounded by the frame 170 includes a mixture of signal data to which the first imaging condition is applied and signal data to which the second imaging condition is applied, the control unit 34 (AF calculation unit 34d) performs correction processing as shown in the following (Example 1) to (Example 3) on the signal data under the second imaging condition among the signal data surrounded by the frame 170. Then, the control unit 34 (AF calculation unit 34d) performs focus detection processing using the signal data after correction processing.

[0103] (Example 1) For example, when only the ISO sensitivity differs between the first and second imaging conditions, and the ISO sensitivity of the first imaging condition is 100 and the ISO sensitivity of the second imaging condition is 800, the control unit 34 (AF calculation unit 34d) applies a correction process of 100 / 800 to the signal data of the second imaging condition, thereby reducing the difference between the signal data due to the difference in imaging conditions. In addition, when the amount of light incident on a pixel to which the first imaging condition is applied is the same as the amount of light incident on a pixel to which the second imaging condition is applied, the difference in signal data becomes small, but in cases where the amount of light incident on a pixel to which the first imaging condition is applied is different from the amount of light incident on a pixel to which the second imaging condition is applied, the difference in signal data may not become small.

[0104] (Example 2) For example, when only the shutter speed differs between the first and second imaging conditions, and the shutter speed of the first imaging condition is 1 / 1000 seconds and the shutter speed of the second imaging condition is 1 / 100 seconds, the control unit 34 (AF calculation unit 34d) applies 1 / 1000 / 1 / 100=1 / 10 as a correction process to the signal data of the second imaging condition, thereby reducing the difference between the signal data due to the difference in imaging conditions.

[0105] (Example 3) For example, when only the frame rate differs between the first and second imaging conditions (the charge accumulation time is the same), the frame rate of the first imaging condition is 30 fps, and the frame rate of the second imaging condition is 60 fps, the control unit 34 (AF calculation unit 34d) performs correction processing to adopt, for the signal data of the second imaging condition (60 fps), the signal data of the frame image acquired under the first imaging condition (30 fps) and the acquisition start timing of which are close to each other, thereby reducing the difference between the signal data due to the difference in imaging conditions. In addition, the correction process may involve interpolating and calculating signal data for a frame image whose acquisition start timing is close to that of a frame image acquired under the first imaging condition (30 fps) based on multiple adjacent frame images acquired under the second imaging condition (60 fps).

[0106] On the other hand, the control unit 34 (AF calculation unit 34d) does not perform the above correction process when the imaging conditions applied to the signal data surrounded by the frame 170 are the same. In other words, the control unit 34 (AF calculation unit 34d) performs focus detection processing using the signal data from the focus detection pixels shown in the frame 170 as is.

[0107] As described above, even if there are some differences in the imaging conditions, they are considered to be the same imaging conditions. In the above example, an example was described in which correction processing was performed on signal data of the second imaging condition among the signal data using the first imaging condition, but correction processing may also be performed on signal data of the first imaging condition among the signal data using the second imaging condition. The control unit 34 (AF calculation unit 34d) may determine whether to perform correction processing on the signal data under the first imaging condition or the signal data under the second imaging condition based on, for example, the ISO sensitivity. When the ISO sensitivity is different between the first imaging condition and the second imaging condition, it is desirable to perform correction processing on the signal data obtained under the imaging condition with the lower ISO sensitivity if the signal data obtained under the imaging condition with the higher ISO sensitivity is not saturated. In other words, when the ISO sensitivity is different between the first imaging condition and the second imaging condition, it is desirable to perform correction processing on the darker signal data so as to reduce the difference with the brighter signal data.

[0108] Furthermore, correction processing may be performed on the signal data under the first imaging condition and the signal data under the second imaging condition, respectively, to reduce the difference between the two sets of signal data after the correction processing.

[0109] In the above explanation, an example of focus detection processing using the pupil division phase difference method has been given, but the same can also be performed using a contrast detection method in which the focus lens of the imaging optical system 31 is moved to the in-focus position based on the level of contrast of the subject image.

[0110] When the contrast detection method is used, the control unit 34 performs a known focus evaluation value calculation based on signal data output from the imaging pixels of the image sensor 32a corresponding to the focus detection positions at each position of the focus lens while moving the focus lens of the imaging optical system 31. Then, the position of the focus lens that maximizes the focus evaluation value is obtained as the in-focus position.

[0111] The control unit 34 normally performs focus evaluation value calculation using the signal data output from the imaging pixel corresponding to the focus detection position as is without performing correction processing. However, when the signal data corresponding to the focus detection position includes a mixture of signal data to which the first imaging condition is applied and signal data to which the second imaging condition is applied, the control unit 34 performs the above-mentioned correction processing on the signal data of the second imaging condition among the signal data corresponding to the focus detection position. Then, the control unit 34 performs focus evaluation value calculation using the signal data after the correction processing.

[0112] 3. When performing subject detection processing Fig. 14(a) is a diagram illustrating a template image representing an object to be detected, and Fig. 14(b) is a diagram illustrating a live view image 60(a) and a search range 190. The control unit 34 (object detection unit 34a) detects an object (for example, a bag 63a, which is one of the subject elements in Fig. 5) from the live view image. The control unit 34 (object detection unit 34a) may set the range for detecting the object to the entire range of the live view image 60a, but may also set a part of the live view image 60a as the search range 190 to reduce the detection process.

[0113] When different imaging conditions are set between the divided areas and the search range 190 includes the boundary between the divided areas, the control unit 34 (object detection unit 34a) performs a correction process on the image data of at least one area within the search range 190 as pre-processing of the subject detection process.

[0114] The correction process is performed to suppress a decrease in accuracy of the detection process of the subject element due to differences in imaging conditions between the regions of the imaging screen divided by the setting unit 34b. In general, when the search range 190 used to detect the subject element includes the boundaries of the divided regions, image data to which different imaging conditions are applied may be mixed in the image data of the search range 190. In this embodiment, the correction process is performed as follows based on the idea that it is preferable to detect the subject element using image data that has been subjected to a correction process so as to suppress differences between image data due to differences in imaging conditions, rather than detecting the subject element using image data to which different imaging conditions are applied as is.

[0115] A case will be described where a bag 63a belonging to a person 61a is detected in a live view image 60a illustrated in Fig. 5. The control unit 34 (object detection unit 34a) sets a search range 190 in the vicinity of an area including the person 61a. Note that the area 61 including the person 61a may also be set as the search range.

[0116] When the search range 190 is not divided by two regions with different imaging conditions, the control unit 34 (object detection unit 34a) performs the subject detection process using the image data constituting the search range 190 as is without performing the correction process. However, if the image data of the search range 190 includes a mixture of image data to which the first imaging condition is applied and image data to which the second imaging condition is applied, the control unit 34 (object detection unit 34a) performs the correction process as in (Example 1) to (Example 3) described above as the case of performing the focus detection process on the image data of the second imaging condition among the image data of the search range 190. Then, the control unit 34 (object detection unit 34a) performs the subject detection process using the image data after the correction process. As described above, even if there are some differences in the imaging conditions, they are considered to be the same imaging conditions. In addition, in the above example, an example was described in which correction processing was performed on image data of the second imaging condition among the image data using the first imaging condition, but correction processing may also be performed on image data of the first imaging condition among the image data using the second imaging condition.

[0117] The correction process for image data in the search range 190 described above may be applied to a search range used to detect a specific subject such as a human face, or to an area used to determine an imaging scene.

[0118] Furthermore, the correction process for image data in the search range 190 described above may be applied not only to a search range used in a pattern matching method using a template image, but also to a search range when detecting features based on the color, edges, etc. of an image.

[0119] Also, a known template matching process may be performed using image data of a plurality of frames acquired at different times, to apply to a moving object tracking process in which an area similar to a tracking target in a previously acquired frame image is searched for in a later acquired frame image. In this case, when image data to which the first imaging condition is applied and image data to which the second imaging condition is applied are mixed in a search range set in a later acquired frame image, the control unit 34 performs a correction process as described above in (Example 1) to (Example 3) on the image data of the second imaging condition among the image data in the search range. Then, the control unit 34 performs a tracking process using the image data after the correction process.

[0120] The same is true when detecting a known motion vector using image data of a plurality of frames acquired at different times. When image data to which the first imaging condition is applied and image data to which the second imaging condition is applied are mixed in the detection area used for detecting the motion vector, the control unit 34 performs a correction process as described above in (Example 1) to (Example 3) on the image data of the detection area used for detecting the motion vector under the second imaging condition. Then, the control unit 34 detects the motion vector using the image data after the correction process.

[0121] 4. Setting imaging conditions When the control unit 34 (setting unit 34b) divides an area of ​​the imaging screen, sets different imaging conditions between the divided areas, and then performs new photometry to determine exposure conditions, the control unit 34 (setting unit 34b) performs a correction process on the image data of at least one area as a preprocessing step for setting the exposure conditions.

[0122] The correction process is performed to suppress a decrease in accuracy of the process for determining the exposure conditions due to differences in the imaging conditions between the regions of the imaging screen divided by the setting unit 34b. For example, when the photometry range set in the center of the imaging screen includes the boundaries of the divided regions, image data to which different imaging conditions are applied may be mixed in the image data of the photometry range. In this embodiment, the correction process is performed as follows based on the idea that it is preferable to perform exposure calculation processing using image data that has been subjected to a correction process so as to suppress differences between image data due to differences in imaging conditions, rather than performing exposure calculation processing using image data to which different imaging conditions are applied as is.

[0123] When the photometric range is not divided by a plurality of regions with different imaging conditions, the control unit 34 (setting unit 34b) performs exposure calculation processing using the image data constituting the photometric range as is without performing correction processing. However, if the image data of the photometric range includes a mixture of image data to which the first imaging condition is applied and image data to which the second imaging condition is applied, the control unit 34 (setting unit 34b) performs correction processing as in (Example 1) to (Example 3) described above as cases in which focus detection processing and subject detection processing are performed on the image data of the second imaging condition among the image data of the photometric range. Then, the control unit 34 (setting unit 34b) performs exposure calculation processing using the image data after correction processing. As described above, even if there are some differences in the imaging conditions, they are considered to be the same imaging conditions. In addition, in the above example, an example was described in which correction processing was performed on image data of the second imaging condition among the image data using the first imaging condition, but correction processing may also be performed on image data of the first imaging condition among the image data using the second imaging condition.

[0124] This is not limited to the metering range used when performing the above-mentioned exposure calculation processing, but also applies to the metering (colorimetry) range used when determining a white balance adjustment value, the metering range used when determining whether or not to emit fill light from a light source that emits fill light, and further to the metering range used when determining the amount of fill light emitted by the light source.

[0125] Furthermore, in the case where the readout resolution of the photoelectric conversion signal is made different between the regions divided from the imaging screen, the same can be applied to the regions used for determining the imaging scene when determining the readout resolution for each region.

[0126] <Explanation of the flow chart> Fig. 15 is a flowchart for explaining the flow of processing for setting imaging conditions for each area and capturing an image. When the main switch of the camera 1 is turned on, the control unit 34 starts a program for executing the processing shown in Fig. 15. In step S10, the control unit 34 causes the display unit 35 to start live view display, and the process proceeds to step S20.

[0127] Specifically, the control unit 34 instructs the imaging unit 32 to start acquiring live view images, and causes the acquired live view images to be sequentially displayed on the display unit 35. As described above, at this point in time, the same imaging conditions are set for the entire area of ​​the imaging chip 111, i.e., the entire screen. When the AF operation is set to be performed during live view display, the control unit 34 (AF calculation unit 34d) performs focus detection processing to control the AF operation for focusing on a subject element corresponding to a predetermined focus detection position. The AF calculation unit 34d performs the above correction processing as necessary before performing the focus detection processing. Furthermore, if a setting to perform AF operation during live view display is not made, the control unit 34 (AF calculation unit 34d) will perform AF operation when an AF operation is later instructed.

[0128] In step S20, the control unit 34 (object detection unit 34a) detects subject elements from the live view image and proceeds to step S30. The object detection unit 34a performs the above correction process as necessary and then performs the subject detection process. In step S30, the control unit 34 (setting unit 34b) divides the screen of the live view image into areas including subject elements and proceeds to step S40.

[0129] In step S40, the control unit 34 displays the regions on the display unit 35. As illustrated in Fig. 6, the control unit 34 highlights the region among the divided regions that is the target of setting (changing) the imaging conditions. The control unit 34 also displays an imaging condition setting screen 70 on the display unit 35, and the process proceeds to step S50. When the user taps the display position of another main subject on the display screen with his / her finger, the control unit 34 changes the region including the main subject to the region that is the target of setting (changing) the imaging conditions, and highlights the region.

[0130] In step S50, the control unit 34 determines whether or not an AF operation is necessary. For example, when the focus adjustment state has changed due to the movement of the subject, when the focus detection position has been changed by a user operation, or when an AF operation is instructed by a user operation, the control unit 34 makes a positive determination in step S50 and proceeds to step S70. When the focus adjustment state has not changed, when the focus detection position has not been changed by a user operation, and when an AF operation is not instructed by a user operation, the control unit 34 makes a negative determination in step S50 and proceeds to step 60.

[0131] In step S70, the control unit 34 performs an AF operation and returns to step S40. The AF calculation unit 34d performs the above-mentioned correction process as necessary, and then performs focus detection processing, which is an AF operation. After returning to step S40, the control unit 34 repeats the same process as the above-mentioned process based on the live view image acquired after the AF operation.

[0132] In step S60, the control unit 34 (setting unit 34b) sets the imaging conditions for the highlighted area in response to a user operation, and the process proceeds to step S80. Note that the display transition of the display unit 35 and the setting of the imaging conditions in response to the user operation in step S60 are as described above. The control unit 34 (setting unit 34b) performs the above correction process as necessary, and then performs the exposure calculation process.

[0133] In step S80, the control unit 34 determines whether or not an image capture instruction has been issued. If a release button (not shown) constituting the operation member 36 or a display icon instructing image capture has been operated, the control unit 34 makes an affirmative decision in step S80 and proceeds to step S90. If an image capture instruction has not been issued, the control unit 34 makes a negative decision in step S80 and returns to step S60.

[0134] In step S90, the control unit 34 performs a predetermined imaging process. That is, the imaging control unit 34c controls the imaging element 32a to perform imaging under the imaging conditions set for each of the above-mentioned regions, and the process proceeds to step S100.

[0135] In step S100, the control unit 34 (imaging control unit 34c) sends an instruction to the image processing unit 33 to perform a predetermined image processing on the image data obtained by the above imaging, and the process proceeds to step S110. The image processing includes the above pixel defect correction processing, color interpolation processing, edge enhancement processing, and noise reduction processing. If necessary, the image processing unit 33 (correction unit 33b) performs correction processing on image data located near the boundary between regions before performing image processing.

[0136] In step S110, the control unit 34 sends an instruction to the recording unit 37 to record the image data after image processing on a recording medium (not shown), and the process proceeds to step S120.

[0137] In step S120, the control unit 34 judges whether or not an end operation has been performed. If an end operation has been performed, the control unit 34 makes an affirmative decision in step S120 and ends the processing in Fig. 15. If an end operation has not been performed, the control unit 34 makes a negative decision in step S120 and returns to step S20. If the processing returns to step S20, the control unit 34 repeats the above-mentioned processing.

[0138] In the above description, the stacked imaging element 100 has been exemplified as the imaging element 32a. However, as long as imaging conditions can be set for each of a plurality of blocks in the imaging element (imaging chip 111), it is not necessarily required to configure the imaging element as a stacked imaging element.

[0139] According to the first embodiment described above, the following advantageous effects can be obtained. (1) The camera 1 equipped with the image processing device includes a control unit 34 (setting unit 34b) that sets the imaging conditions of the first region of the imaging unit 32 to be different from the imaging conditions of the second region of the imaging unit 32, an image processing unit 33 (correction unit 33b) that corrects first image data generated by capturing light incident on the first region under the imaging conditions set by the setting unit 34b based on the imaging conditions of the second region set by the setting unit 34b, and an image processing unit 33 (generation unit 33c) that generates an image using the first image data corrected by the correction unit 33b and the second image data generated by capturing light incident on the second region. This allows appropriate processing to be performed in the regions with different imaging conditions. That is, an image can be appropriately generated using the image data generated in each region. For example, discontinuity and discomfort that appear in the generated image due to differences in imaging conditions for each region can be suppressed.

[0140] (2) The correction unit 33b of the camera 1 corrects the first image data so that the difference between the values ​​of the first image data and the second image data is small, so that appropriate processing can be performed in areas with different imaging conditions.

[0141] (3) The correction unit 33b of the camera 1 corrects the first image data based on the difference between the imaging conditions of the first area and the imaging conditions of the second area, so that appropriate processing can be performed for each of the areas with different imaging conditions.

[0142] (4) The correction unit 33b of the camera 1 may correct the second image data based on the imaging conditions of the first region set by the setting unit 34b, and the generation unit 33c may generate an image using the first image data corrected by the correction unit 33b and the second image data corrected by the correction unit 33b. Even in this way, appropriate processing can be performed in each of the regions with different imaging conditions.

[0143] (5) The correction unit 33b of the camera 1 may correct the second image data using the first image data, and the generation unit 33c may generate an image using the first image data corrected by the correction unit 33b and the second image data corrected by the correction unit 33b. Even in this way, appropriate processing can be performed in areas with different imaging conditions.

[0144] (6) The generation unit 33c of the camera 1 performs image processing on the second image data using the first image data corrected by the correction unit 33b, so that appropriate processing can be performed in each area with different imaging conditions.

[0145] (7) The generation unit 33c of the camera 1 changes the values ​​of the second image data using the values ​​of the first image data corrected by the correction unit 33b, so that appropriate processing can be performed in each area with different imaging conditions.

[0146] (8) The generation unit 33c of the camera 1 replaces the values ​​of the second image data with the values ​​of the first image data corrected by the correction unit 33b, so that appropriate processing can be performed in each area having different imaging conditions.

[0147] (9) The generation unit of camera 1 generates third image data from the second image data and the first image data corrected by correction unit 33b, so that appropriate processing can be performed for each area with different imaging conditions.

[0148] (10) The setting unit 34b of the camera 1 sets the imaging sensitivity as the imaging condition for the first region of the imaging unit 32 to a first imaging sensitivity, and sets the imaging sensitivity as the imaging condition for the second region of the imaging unit 32 to a second imaging sensitivity different from the first imaging sensitivity, and the correction unit 33b corrects the first image data generated by capturing an image under the imaging condition set by the setting unit 34b based on the difference between the first imaging sensitivity and the second imaging sensitivity. This makes it possible to appropriately process the regions with different imaging sensitivities.

[0149] The system may be configured to be switchable between mode 1, in which the above-mentioned correction process is performed as a pre-processing, and mode 2, in which the correction process is not performed as a pre-processing. When mode 1 is selected, the control unit 34 performs the above-mentioned pre-processing and then performs image processing. On the other hand, when mode 2 is selected, the control unit 34 performs image processing and other processes without performing the above-mentioned pre-processing. For example, when a part of a face detected as a subject element has a shadow, if an image is captured under different settings for the imaging conditions of the area including the shadow part of the face and the area including the non-shadow part of the face so that the brightness of the shadow part of the face is about the same as the brightness of the non-shadow part of the face, and then a correction process is performed on the image generated by the imaging conditions, unintended color interpolation may be performed on the shadow part due to the difference in the imaging conditions set. By configuring the system to be switchable between mode 1 and mode 2 so that the color interpolation process can be performed using the image data as it is without performing the correction process, it is possible to avoid unintended color interpolation.

[0150] ---Modification of the first embodiment--- The following modifications are also within the scope of the present invention, and one or more of the modifications may be combined with the above-described embodiment. (Variation 1) 16(a) to 16(c) are diagrams illustrating the arrangement of the first and second regions on the imaging surface of the image sensor 32a. In the example of Fig. 16(a), the first region is made up of even-numbered columns, and the second region is made up of odd-numbered columns. That is, the imaging surface is divided into even-numbered columns and odd-numbered columns.

[0151] 16(b), the first region is made up of odd-numbered rows, and the second region is made up of even-numbered rows, i.e., the imaging surface is divided into odd-numbered rows and even-numbered rows.

[0152] In the example of Fig. 16(c), the first region is made up of blocks in even rows and odd columns, and blocks in odd rows and even columns. The second region is made up of blocks in even rows and even columns, and blocks in odd rows and odd columns. In other words, the imaging surface is divided into a checkerboard pattern.

[0153] 16(a) to 16(c), a first image based on the photoelectric conversion signal read out from the first region and a second image based on the photoelectric conversion signal read out from the second region are generated by the photoelectric conversion signal read out from the image sensor 32a that has captured one frame. According to the first modification, the first image and the second image are captured at the same angle of view and include a common subject image.

[0154] In the first modification, the control unit 34 uses the first image for display and the second image for detection. Specifically, the control unit 34 causes the display unit 35 to display the first image as a live view image. The control unit 34 also causes the object detection unit 34a to perform subject detection processing using the second image, the AF calculation unit 34 to perform focus detection processing using the second image, and the setting unit 34b to perform exposure calculation processing using the second image.

[0155] In the first modification, the imaging conditions set for the first region for capturing the first image are referred to as the first imaging conditions, and the imaging conditions set for the second region for capturing the second image are referred to as the second imaging conditions. The control unit 34 may make the first imaging conditions and the second imaging conditions different from each other.

[0156] 1. As an example, the control unit 34 sets the first imaging condition to a condition suitable for display by the display unit 35. For example, the first imaging condition set in the first area is made the same throughout the first area of ​​the imaging screen. Meanwhile, the control unit 34 sets the second imaging condition set in the second area to a condition suitable for focus detection processing, subject detection processing, and exposure calculation processing. The second imaging condition is made the same throughout the second area of ​​the imaging screen. In addition, when the conditions suitable for the focus detection process, the subject detection process, and the exposure calculation process are different, the control unit 34 may set the second imaging condition in the second area different for each frame. For example, the second imaging condition in the first frame is set to the condition suitable for the focus detection process, the second imaging condition in the second frame is set to the condition suitable for the subject detection process, and the second imaging condition in the third frame is set to the condition suitable for the exposure calculation process. In these cases, the second imaging condition in each frame is set to be the same in the entire second area of ​​the imaging screen.

[0157] 2. As another example, the control unit 34 may set the first imaging condition for the first region differently depending on the region. The control unit 34 (setting unit 34b) sets different first imaging conditions for each region including the subject elements divided by the setting unit 34b. On the other hand, the control unit 34 sets the second imaging condition for the second region to be the same for the entire second region of the imaging screen. The control unit 34 sets the second imaging condition to a condition suitable for focus detection processing, subject detection processing, and exposure calculation processing, but when the conditions suitable for focus detection processing, subject detection processing, and exposure calculation processing are different, the imaging condition set for the second region may be different for each frame.

[0158] 3. As another example, the control unit 34 may set the first imaging condition for the first region to be the same throughout the entire first region of the imaging screen, while setting the second imaging condition for the second region to be different in the imaging screen. For example, the setting unit 34b sets different second imaging conditions for each region including the divided subject elements. Even in this case, if the conditions suitable for the focus detection process, the subject detection process, and the exposure calculation process are different, the imaging condition set for the second region may be different for each frame.

[0159] 4. Furthermore, as another example, the control unit 34 makes the first imaging condition set for the first region different on the imaging screen, and makes the second imaging condition set for the second region different on the imaging screen. For example, the setting unit 34b sets different first imaging conditions for each region including the divided subject elements, and sets different second imaging conditions for each region including the divided subject elements.

[0160] In Fig. 16(a) to Fig. 16(c), the area ratio of the first region and the second region may be made different. For example, the control unit 34 sets the ratio of the first region higher than that of the second region, sets the ratio of the first region and the second region equal as illustrated in Fig. 16(a) to Fig. 16(c), or sets the ratio of the first region lower than that of the second region based on the operation by the user or the determination of the control unit 34. By making the area ratio of the first region and the second region different, it is possible to make the first image higher in resolution than the second image, make the resolution of the first image and the second image equal, or make the second image higher in resolution than the first image.

[0161] (Variation 2) In the correction process when performing image processing, when the imaging condition (first imaging condition) applied at the position of interest is different from the imaging condition (second imaging condition) applied at the reference position around the position of interest, the image processing unit 33 (correction unit 33b) corrects the image data of the second imaging condition (image data of the second imaging condition among the image data of the reference position) based on the first imaging condition. That is, by correcting the image data of the second imaging condition at the reference position, discontinuity of the image based on the difference between the first imaging condition and the second imaging condition is mitigated.

[0162] Instead, in Modification 2, the image processing unit 33 (correction unit 33b) may correct the image data of the first imaging condition (the image data of the interest position and the image data of the reference position under the first imaging condition) based on the second imaging condition. In this case, too, the discontinuity of the image based on the difference between the first imaging condition and the second imaging condition can be mitigated.

[0163] Alternatively, the image processing unit 33 (correction unit 33b) may correct both the image data under the first imaging condition and the image data under the second imaging condition. That is, the image data of the position of interest under the first imaging condition, the image data under the first imaging condition among the image data of the reference position, and the image data under the second imaging condition among the image data of the reference position may be subjected to correction processing, respectively, to mitigate discontinuity of images based on the difference between the first imaging condition and the second imaging condition. For example, in the above (Example 1), the image data of the reference pixel under the first imaging condition (ISO sensitivity is 100) is multiplied by 400 / 100 as a correction process, and the image data of the reference pixel under the second imaging condition (ISO sensitivity is 800) is multiplied by 400 / 800 as a correction process. This reduces the difference between the image data due to the difference in imaging conditions. The pixel data of the target pixel is corrected by multiplying it by 100 / 400 after color interpolation. This correction process can change the pixel data of the target pixel after color interpolation to a value similar to that when the pixel data is captured under the first imaging condition. Furthermore, in the above (Example 1), the degree of correction process may be changed depending on the distance from the boundary between the first area and the second area. Then, the rate at which the image data increases or decreases due to the correction process can be reduced compared to the case of the above (Example 1), and the noise caused by the correction process can be reduced. Although the above (Example 1) has been described, the same can be applied to the above (Example 2).

[0164] According to the second modification, similarly to the above-described embodiment, it is possible to perform appropriate image processing on image data generated for areas with different imaging conditions.

[0165] (Variation 3) In the above description, when performing correction processing on image data, the corrected image data is obtained by performing a calculation based on the difference between the first and second imaging conditions. Instead of the calculation, the corrected image data may be obtained by referring to a correction table. For example, the corrected image data is read by inputting the first and second imaging conditions as arguments. Alternatively, the correction coefficient may be read by inputting the first and second imaging conditions as arguments.

[0166] (Variation 4) In the above-mentioned correction process, upper and lower limits of the corrected image data may be set. By setting the upper and lower limits, it is possible to restrict the correction so as not to make more correction than necessary. The upper and lower limits may be determined in advance, or, in the case where a photometry sensor is provided in addition to the image sensor 32a, they may be determined based on the output signal from the photometry sensor.

[0167] (Variation 5) In the above embodiment, the control unit 34 (setting unit 34b) detects subject elements based on a live view image and divides the screen of the live view image into areas including the subject elements. In Modification 5, if the control unit 34 includes a photometry sensor in addition to the image sensor 32a, the control unit 34 may divide the areas based on an output signal from the photometry sensor.

[0168] The control unit 34 divides the image into a foreground and a background based on the output signal from the photometry sensor. Specifically, the control unit 34 divides the live view image acquired by the image sensor 32b into a foreground region corresponding to the region determined to be the foreground based on the output signal from the photometry sensor, and a background region corresponding to the region determined to be the background based on the output signal from the photometry sensor.

[0169] The control unit 34 further arranges the first region and the second region at a position corresponding to the foreground region on the imaging surface of the imaging element 32a, as illustrated in Figures 16(a) to 16(c). On the other hand, the control unit 34 arranges only the first region on the imaging surface of the imaging element 32a at a position corresponding to the background region on the imaging surface of the imaging element 32a. The control unit 34 uses the first image for display and the second image for detection.

[0170] According to the fifth modification, the output signal from the photometry sensor can be used to perform area division of the live view image captured by the image sensor 32b. Also, for the foreground area, a first image for display and a second image for detection can be obtained, and for the background area, only the first image for display can be obtained.

[0171] (Variation 6) In the sixth modification, the image processing unit 33 (the generating unit 33c) performs a contrast adjustment process as an example of a correction process. That is, the generating unit 33c reduces discontinuity of an image based on a difference between the first and second imaging conditions by varying a gradation curve (gamma curve).

[0172] For example, assume that only the ISO sensitivity differs between the first and second imaging conditions, and the ISO sensitivity of the first imaging condition is 100 and the ISO sensitivity of the second imaging condition is 800. The generating unit 33c compresses the value of the image data of the second imaging condition among the image data of the reference position to 1 / 8 by flattening the gradation curve.

[0173] Alternatively, the generating unit 33c may increase the value of the image data of the interest position and the image data of the reference position under the first imaging condition by eight times by increasing the gradation curve.

[0174] According to the sixth modification, similarly to the above-described embodiment, it is possible to appropriately perform image processing on image data generated in regions with different imaging conditions. For example, it is possible to suppress discontinuity or discomfort that appears in the image after image processing due to differences in imaging conditions at the boundary between regions.

[0175] (Variation 7) In the seventh modification, the image processing unit 33 performs the above-mentioned image processing (for example, noise reduction processing) so as not to impair the contours of the subject elements. Generally, smoothing filter processing is adopted when performing noise reduction. When a smoothing filter is used, while the noise reduction effect is achieved, the boundaries of the subject elements may become blurred.

[0176] Therefore, the image processing unit 33 (the generation unit 33c) compensates for the blurring of the boundaries of the above-mentioned subject elements by performing a contrast adjustment process, for example, in addition to or together with the noise reduction process. In the seventh modification, the image processing unit 33 (the generation unit 33c) sets an S-shaped curve as the density conversion (gradation conversion) curve (so-called S-shaped conversion). By performing contrast adjustment using the S-shaped conversion, the image processing unit 33 (the generation unit 33c) stretches the gradation parts of the bright data and the dark data, respectively, to increase the number of gradations of the bright data (and the dark data), and compresses the image data of intermediate gradations to reduce the number of gradations. This reduces the number of image data with medium brightness in the image, and increases the data classified as either bright or dark, thereby making it possible to compensate for the blurring of the boundaries of the subject elements.

[0177] According to the seventh modification, the blurring of the boundaries between object elements can be compensated for by making the contrast between light and dark in the image clearer.

[0178] (Variation 8) In the eighth modification, the image processing unit 33 (the generation unit 33c) changes the white balance adjustment gain so as to reduce discontinuity in the image based on the difference between the first and second imaging conditions.

[0179] For example, when the imaging conditions (referred to as the first imaging conditions) applied when capturing an image at the focus position are different from the imaging conditions (referred to as the second imaging conditions) applied when capturing an image at a reference position surrounding the focus position, the image processing unit 33 (generation unit 33c) changes the white balance adjustment gain so as to bring the white balance of the image data of the second imaging condition among the image data of the reference position closer to the white balance of the image data acquired under the first imaging conditions.

[0180] In addition, the image processing unit 33 (generation unit 33c) may change the white balance adjustment gain so as to bring the white balance of the image data of the first imaging condition among the image data of the reference position and the image data of the interest position closer to the white balance of the image data acquired under the second imaging condition.

[0181] According to variant example 8, by aligning the white balance adjustment gain for image data generated in areas with different imaging conditions to the adjustment gain for one of the areas with different imaging conditions, discontinuity in the image due to the difference between the first and second imaging conditions can be mitigated.

[0182] (Variation 9) A plurality of image processing units 33 may be provided to perform image processing in parallel. For example, image processing may be performed on image data captured in region A of the imaging unit 32 while image processing is performed on image data captured in region B of the imaging unit 32. The plurality of image processing units 33 may perform the same image processing, or may perform different image processing. In other words, the same parameters or the like may be applied to the image data of region A and region B to perform similar image processing, or different parameters or the like may be applied to the image data of region A and region B to perform different image processing.

[0183] In the case where a plurality of image processing units 33 are provided, image processing may be performed by one image processing unit on image data to which a first imaging condition is applied, and image processing may be performed by another image processing unit on image data to which a second imaging condition is applied. The number of image processing units is not limited to the above two, and may be the same number as the number of imaging conditions that can be set, for example. That is, each image processing unit is responsible for image processing for each region to which a different imaging condition is applied. According to the ninth modification, imaging under different imaging conditions for each region and image processing for image data of images obtained for each region can be performed in parallel.

[0184] (Variation 10) In the above description, the camera 1 has been taken as an example, but the camera 1 may be configured as a highly functional mobile phone 250 (FIG. 18) equipped with a camera function such as a smartphone, or as a mobile device such as a tablet terminal.

[0185] (Variation 11) In the above-described embodiment, the camera 1 is described as an example in which the imaging unit 32 and the control unit 34 are configured as a single electronic device. Instead of this, for example, the imaging unit 32 and the control unit 34 may be provided separately, and the imaging system 1B may be configured in which the imaging unit 32 is controlled by the control unit 34 via communication. Hereinafter, an example in which an imaging device 1001 including an imaging section 32 is controlled from a display device 1002 including a control section 34 will be described with reference to FIG.

[0186] Fig. 17 is a block diagram illustrating a configuration of an imaging system 1B according to Modification 11. In Fig. 17, the imaging system 1B is configured with an imaging device 1001 and a display device 1002. The imaging device 1001 includes a first communication unit 1003 in addition to the imaging optical system 31 and imaging unit 32 described in the above embodiment. The display device 1002 includes a second communication unit 1004 in addition to the image processing unit 33, control unit 34, display unit 35, operation member 36, and recording unit 37 described in the above embodiment.

[0187] The first communication unit 1003 and the second communication unit 1004 can perform two-way image data communication using, for example, well-known wireless communication technology or optical communication technology. The imaging device 1001 and the display device 1002 may be connected by a wired cable, and the first communication unit 1003 and the second communication unit 1004 may perform bidirectional image data communication.

[0188] In the imaging system 1B, the control unit 34 controls the imaging unit 32 by performing data communication via the second communication unit 1004 and the first communication unit 1003. For example, by transmitting and receiving predetermined control data between the imaging device 1001 and the display device 1002, the display device 1002 divides the screen into a plurality of regions based on the image as described above, sets different imaging conditions for each divided region, and reads out photoelectric conversion signals photoelectrically converted in each region.

[0189] According to variant example 11, a live view image acquired by the imaging device 1001 and transmitted to the display device 1002 is displayed on the display unit 35 of the display device 1002, allowing the user to remotely control the imaging device 1001 from the display device 1002 located away from the imaging device 1001. The display device 1002 can be configured, for example, by a highly functional mobile phone 250 such as a smartphone. The imaging device 1001 can be configured by an electronic device including the stacked imaging element 100 described above. Although an example has been described in which the control unit 34 of the display device 1002 is provided with the object detection unit 34a, the setting unit 34b, the imaging control unit 34c, and the AF calculation unit 34d, part of the object detection unit 34a, the setting unit 34b, the imaging control unit 34c, and the AF calculation unit 34d may be provided in the imaging device 1001.

[0190] (Variation 12) The program can be supplied to a mobile device such as the above-mentioned camera 1, high-function mobile phone 250, or tablet terminal by transmitting the program from a personal computer 205 storing the program to the mobile device via infrared communication or short-range wireless communication, as illustrated in FIG. 18, for example.

[0191] The program may be supplied to the personal computer 205 by inserting a recording medium 204 such as a CD-ROM storing the program into the personal computer 205, or by loading the program into the personal computer 205 via a communication line 201 such as a network. When the program is supplied via the communication line 201, the program is stored in a storage device 203 of a server 202 connected to the communication line.

[0192] The program can also be directly transmitted to the mobile device via a wireless LAN access point (not shown) connected to the communication line 201. Furthermore, a recording medium 204B such as a memory card storing the program may be set in the mobile device. In this way, the program can be supplied as a computer program product in various forms, such as a recording medium or via a communication line.

[0193] ---Second embodiment--- With reference to Figs. 19 to 25, a digital camera will be described as an example of an electronic device equipped with an image processing device according to the second embodiment. In the following description, the same components as those in the first embodiment are given the same reference numerals, and differences will be mainly described. Points that are not particularly described are the same as those in the first embodiment. This embodiment differs from the first embodiment mainly in that, instead of providing the image processing unit 33 of the first embodiment, the imaging unit 32A further includes an image processing unit 32c having the same function as the image processing unit 33 of the first embodiment.

[0194] Fig. 19 is a block diagram illustrating the configuration of a camera 1C according to the second embodiment. In Fig. 19, the camera 1C has an imaging optical system 31, an imaging unit 32A, a control unit 34, a display unit 35, an operation member 36, and a recording unit 37. The imaging unit 32A further includes an image processing unit 32c having the same function as the image processing unit 33 of the first embodiment.

[0195] The image processing unit 32c includes an input unit 321, a correction unit 322, and a generation unit 323. The input unit 321 receives image data from the imaging element 32a. The correction unit 322 performs pre-processing to correct the input image data. The pre-processing performed by the correction unit 322 is the same as the pre-processing performed by the correction unit 33b in the first embodiment. The generation unit 323 performs image processing on the input image data and the image data after the pre-processing to generate an image. The image processing performed by the generation unit 323 is the same as the image processing performed by the generation unit 33c in the first embodiment.

[0196] Fig. 20 is a diagram showing a schematic diagram of the correspondence between each block and a plurality of correction units 322 in this embodiment. In Fig. 20, one square of the imaging chip 111 represented by a rectangle represents one block 111a. Similarly, one square of the image processing chip 114, which will be described later, represented by a rectangle represents one correction unit 322.

[0197] In this embodiment, the correction unit 322 is provided for each block 111a. In other words, the correction unit 322 is provided for each block, which is the smallest unit of an area on the imaging surface where the imaging conditions can be changed. For example, the hatched block 111a in FIG. 20 corresponds to the hatched correction unit 322. The hatched correction unit 322 in FIG. 20 performs preprocessing on image data from pixels included in the hatched block 111a. Each correction unit 322 performs preprocessing on image data from pixels included in the corresponding block 111a. This allows pre-processing of image data to be performed in parallel by multiple correction units 322, thereby reducing the processing burden on the correction units 322 and enabling an appropriate image to be generated in a short period of time from image data generated in areas with different imaging conditions. In the following description, when explaining the relationship between a certain block 111a and the pixels contained in the block 111a, the block 111a may be referred to as the block 111a to which the pixel belongs. The block 111a may be referred to as a unit segment, and a collection of multiple blocks 111a, i.e., a collection of multiple unit segments, may be referred to as a composite segment.

[0198] 21 is a cross-sectional view of the stacked imaging element 100A. The stacked imaging element 100A further includes an image processing chip 114 that performs the above-mentioned pre-processing and image processing, in addition to a back-illuminated imaging chip 111, a signal processing chip 112, and a memory chip 113. That is, the above-mentioned image processing unit 32c is provided in the image processing chip 114. The imaging chip 111, the signal processing chip 112, the memory chip 113 and the image processing chip 114 are stacked and electrically connected to one another by conductive bumps 109 such as Cu.

[0199] A plurality of bumps 109 are arranged on the mutually opposing surfaces of the memory chip 113 and the image processing chip 114. These bumps 109 are aligned with each other, and the memory chip 113 and the image processing chip 114 are pressed together, whereby the aligned bumps 109 are bonded together and electrically connected.

[0200] <Correction process> As in the first embodiment, in the second embodiment, after the area of ​​the imaging screen is divided by the setting unit 34b, it is possible to set (change) imaging conditions for an area selected by a user or an area determined by the control unit 34. When different imaging conditions are set for the divided areas, the control unit 34 causes the correction unit 322 to perform the following correction processing as necessary.

[0201] 1. When performing image processing 1-1. When the imaging conditions of the pixel of interest P and the imaging conditions of the multiple reference pixels surrounding the pixel of interest P are the same In this case, in the image processing unit 32c, the correction unit 322 does not perform the correction process, and the generation unit 323 performs the image processing using the uncorrected image data of the multiple reference pixels.

[0202] 1-2. When the imaging conditions of the pixel of interest P differ from the imaging conditions of at least one of the reference pixels surrounding the pixel of interest P The imaging condition applied to the pixel of interest P is defined as the first imaging condition, the imaging condition applied to some of the reference pixels is defined as the first imaging condition, and the imaging condition applied to the remaining reference pixels is defined as the second imaging condition. In this case, the correction unit 322 corresponding to the block 111a to which the reference pixels to which the second imaging condition is applied belong performs correction processing on the image data of the reference pixels to which the second imaging condition is applied as in the following (Example 1) to (Example 3). Then, the generation unit 323 performs image processing to calculate image data of the pixel of interest P by referring to the image data of the reference pixels to which the first imaging condition is applied and the image data of the reference pixels after the correction processing.

[0203] (Example 1) The correction unit 322 corresponding to the block 111a to which the reference pixel to which the second imaging condition is applied belongs applies a correction process of 100 / 800 to the image data of the reference pixel when, for example, only the ISO sensitivity differs between the first and second imaging conditions and the ISO sensitivity of the first imaging condition is 100 and that of the second imaging condition is 800. This reduces the difference between the image data due to the difference in imaging conditions.

[0204] (Example 2) For example, when only the shutter speed differs between the first and second imaging conditions, and the shutter speed of the first imaging condition is 1 / 1000 second and the shutter speed of the second imaging condition is 1 / 100 second, the correction unit 322 corresponding to the block 111a to which the reference pixel to which the second imaging condition is applied applies 1 / 1000 / 1 / 100=1 / 10 as a correction process to the image data of the reference pixel, thereby reducing the difference between the image data due to the difference in imaging conditions.

[0205] (Example 3) The correction unit 322 corresponding to the block 111a to which the reference pixel to which the second imaging condition is applied belongs performs a correction process in which, for example, when only the frame rate differs between the first imaging condition and the second imaging condition (the charge accumulation time is the same) and the frame rate of the first imaging condition is 30 fps and the frame rate of the second imaging condition is 60 fps, the image data of the reference pixel, i.e., the image data of the second imaging condition (60 fps), is adopted as the image data of the frame image acquired under the first imaging condition (30 fps) and the acquisition start timing of the frame image are close to each other. This reduces the difference between the image data due to the difference in the imaging conditions. In addition, the correction process may involve interpolating and calculating image data for a frame image whose acquisition start timing is close to that of a frame image acquired under the first imaging condition (30 fps) based on multiple adjacent frame images acquired under the second imaging condition (60 fps).

[0206] The same applies when the imaging condition applied to the pixel of interest P is the second imaging condition and the imaging condition applied to the reference pixels surrounding the pixel of interest P is the first imaging condition. That is, in this case, the correction unit 322 corresponding to the block 111a to which the reference pixel to which the first imaging condition is applied belongs performs the correction process on the image data of the reference pixel as in the above-mentioned (Example 1) to (Example 3).

[0207] As described above, even if there are some differences in the imaging conditions, they are considered to be the same imaging conditions.

[0208] The generation unit 323 performs image processing such as pixel defect correction processing, color interpolation processing, contour enhancement processing, and noise reduction processing, similar to the image processing unit 33 (generation unit 33c) in the first embodiment, based on the image data of the reference pixel to which the same imaging conditions as those of the target pixel P have been applied and the image data of the reference pixel corrected by the correction unit 322.

[0209] Figure 22 is a schematic diagram showing the processing of image data (hereinafter referred to as first image data) from each pixel included in a partial area of ​​the imaging surface to which a first imaging condition is applied (hereinafter referred to as first area 141), and image data (hereinafter referred to as second image data) from each pixel included in a partial area of ​​the imaging surface to which a second imaging condition is applied (hereinafter referred to as second area 142).

[0210] Each pixel included in the first region 141 outputs first image data captured under a first imaging condition, and each pixel included in the second region 142 outputs second image data captured under a second imaging condition. The first image data is output to a correction unit 322 provided in the processing chip 114, the correction unit 322 corresponding to the block 111a to which the pixel that generated the first image data belongs. In the following description, the multiple correction units 322 corresponding to the multiple blocks 111a to which the pixel that generated the first image data belongs are referred to as a first processing unit 151. Similarly, the second image data is output to a correction unit 322 corresponding to the block 111a to which the pixel that generated the second image data belongs, among the correction units 322 provided in the processing chip 114. In the following description, the multiple correction units 322 corresponding to the multiple blocks 111a to which the pixels that generated the second image data belong will be referred to as second processing units 152.

[0211] For example, when the pixel of interest P is included in the first region 141, the second image data from the reference pixels included in the second region 142 is subjected to the above-mentioned correction process in the second processing unit 152 as shown in Fig. 22. Note that the second processing unit 152 receives information 181 on the first imaging condition required to reduce the difference between the image data due to the difference in the imaging condition from, for example, the first processing unit 151. Similarly, for example, when the pixel of interest P is included in the second region 142, the first image data from the reference pixels included in the first region 141 is subjected to the above-mentioned correction process in the first processing unit 151. Note that the first processing unit 151 receives information on the second imaging condition required to reduce the difference between the image data due to the difference in the imaging condition from the second processing unit 152.

[0212] When the pixel of interest P and the reference pixel are included in the first region 141, the first processing unit 151 does not correct the first image data from the reference pixel. Similarly, when the pixel of interest P and the reference pixel are included in the second region 142, the second processing unit 152 does not correct the second image data from the reference pixel. Alternatively, both the image data under the first imaging condition and the image data under the second imaging condition may be corrected by the first processing unit 151 and the second processing unit 152. That is, by performing correction processing on the image data of the position of interest under the first imaging condition, the image data under the first imaging condition among the image data of the reference position, and the image data under the second imaging condition among the image data of the reference position, respectively, discontinuity of images based on the difference between the first imaging condition and the second imaging condition may be mitigated. For example, in the above (Example 1), the image data of the reference pixel under the first imaging condition (ISO sensitivity is 100) is multiplied by 400 / 100 as a correction process, and the image data of the reference pixel under the second imaging condition (ISO sensitivity is 800) is multiplied by 400 / 800 as a correction process. This reduces the difference between the image data due to the difference in imaging conditions. The pixel data of the target pixel is corrected by multiplying it by 100 / 400 after color interpolation. This correction process can change the pixel data of the target pixel after color interpolation to a value similar to that when the pixel data is captured under the first imaging condition. Furthermore, in the above (Example 1), the degree of correction process may be changed depending on the distance from the boundary between the first area and the second area. Then, the rate at which the image data increases or decreases due to the correction process can be reduced compared to the case of the above (Example 1), and the noise caused by the correction process can be reduced. Although the above (Example 1) has been described, the same can be applied to the above (Example 2).

[0213] The generation unit 323 performs image processing such as pixel defect correction processing, color interpolation processing, contour enhancement processing, and noise reduction processing based on the image data from the first processing unit 151 and the second processing unit 152, and outputs the image data after image processing.

[0214] When the pixel of interest P is located in the second region 142, the first processing unit 151 may perform correction processing on the first image data from all pixels included in the first region 141, or may perform correction processing on only the first image data from pixels that may be used to interpolate the pixel of interest P in the second region 142 among the pixels included in the first region 141. Similarly, when the pixel of interest P is located in the first region 141, the second processing unit 152 may perform correction processing on the second image data from all pixels included in the second region 142, or may perform correction processing on only the second image data from pixels that may be used to interpolate the pixel of interest P in the first region 141 among the pixels included in the second region 142.

[0215] 2. When performing focus detection processing As in the first embodiment, the control unit 34 (AF calculation unit 34d) performs focus detection processing using signal data (image data) corresponding to a predetermined position (focus detection position) on the imaging screen. When different imaging conditions are set between the divided areas and the focus detection position for AF operation is located at the boundary between the divided areas, that is, when the focus detection position is divided into two areas, the first area and the second area, in this embodiment, as described in 2-2 below, the control unit 34 (AF calculation unit 34d) causes the correction unit 322 to perform correction processing on the signal data for focus detection of at least one area.

[0216] 2-1. When signal data from pixels within frame 170 in FIG. 13 does not include a mixture of signal data to which the first imaging condition is applied and signal data to which the second imaging condition is applied In this case, the correction unit 322 does not perform correction processing, and the control unit 34 (AF calculation unit 34d) performs focus detection processing using signal data from the focus detection pixels indicated by the frame 170 as is.

[0217] 2-2. When signal data from pixels within frame 170 in FIG. 13 includes a mixture of signal data to which the first imaging condition is applied and signal data to which the second imaging condition is applied In this case, the control unit 34 (AF calculation unit 34d) causes the correction unit 322 corresponding to the block 111a to which the pixel to which the second imaging condition is applied belongs among the pixels in the frame 170 to perform correction processing as in the following (Example 1) to (Example 3). Then, the control unit 34 (AF calculation unit 34d) performs focus detection processing using the signal data of the pixel to which the first imaging condition is applied and the signal data after the correction processing.

[0218] (Example 1) For example, when only the ISO sensitivity differs between the first and second imaging conditions and the ISO sensitivity of the first imaging condition is 100 and that of the second imaging condition is 800, the correction unit 322 corresponding to the block 111a to which the pixel to which the second imaging condition is applied applies 100 / 800 as a correction process to the signal data of the second imaging condition, thereby reducing the difference between the signal data due to the difference in imaging conditions.

[0219] (Example 2) For example, when only the shutter speed differs between the first and second imaging conditions, and the shutter speed of the first imaging condition is 1 / 1000 seconds and the shutter speed of the second imaging condition is 1 / 100 seconds, the correction unit 322 corresponding to the block 111a to which the pixel to which the second imaging condition is applied belongs applies 1 / 1000 / 1 / 100=1 / 10 to the signal data of the second imaging condition as a correction process, thereby reducing the difference between the signal data due to the difference in imaging conditions.

[0220] (Example 3) The correction unit 322 corresponding to the block 111a to which the pixel to which the second imaging condition is applied belongs performs correction processing such that, for example, when only the frame rate differs between the first imaging condition and the second imaging condition (the charge accumulation time is the same), the frame rate of the first imaging condition is 30 fps, and the frame rate of the second imaging condition is 60 fps, the signal data of the frame image acquired under the first imaging condition (30 fps) and the acquisition start timing of which are close to each other, are adopted for the signal data under the second imaging condition (60 fps). This reduces the difference between the signal data due to the difference in imaging conditions. In addition, the correction process may involve interpolating and calculating signal data for a frame image whose acquisition start timing is close to that of a frame image acquired under the first imaging condition (30 fps) based on multiple adjacent frame images acquired under the second imaging condition (60 fps).

[0221] As described above, even if there are some differences in the imaging conditions, they are considered to be the same imaging conditions. In the above example, the correction process is performed on the signal data under the second imaging condition among the signal data. However, the correction process may be performed on the signal data under the first imaging condition among the signal data.

[0222] Furthermore, correction processing may be performed on the signal data under the first imaging condition and the signal data under the second imaging condition, respectively, to reduce the difference between the two sets of signal data after the correction processing.

[0223] FIG. 23 is a diagram showing a schematic diagram of processing of the first signal data and the second signal data in relation to the focus detection processing.

[0224] First signal data captured under a first imaging condition is output from each pixel included in the first region 141, and second signal data captured under a second imaging condition is output from each pixel included in the second region 142. The first signal data from the first region 141 is output to a first processing unit 151. Similarly, the second signal data from the second region 142 is output to a second processing unit 152.

[0225] When performing a correction process on the signal data under the second imaging condition among the signal data to reduce the difference between the signal data after the correction process and the signal data under the first imaging condition, the second processing unit 152 performs the process. The second processing unit 152 performs the above-mentioned correction process on the second signal data from the pixels included in the second region 142. The second processing unit 152 receives information 181 on the first imaging condition required to reduce the difference between the signal data due to the difference in the imaging conditions from, for example, the first processing unit 151. In addition, when performing correction processing on the signal data of the second imaging condition among the signal data to reduce the difference between the signal data after the correction processing and the signal data of the first imaging condition, the first processing unit 151 does not correct the first signal data.

[0226] Furthermore, when performing a correction process on the signal data under the first imaging condition among the signal data to reduce the difference between the signal data after the correction process and the signal data under the first imaging condition, the first processing unit 151 performs the process. The first processing unit 151 performs the above-mentioned correction process on the first signal data from the pixels included in the first region 141. The first processing unit 151 receives information on the second imaging condition required to reduce the difference between the signal data due to the difference in the imaging condition from the second processing unit 152. In addition, when performing correction processing on the signal data of the first imaging condition among the signal data to reduce the difference between the signal data after the correction processing and the signal data of the first imaging condition, the second processing unit 152 does not correct the second signal data.

[0227] Furthermore, when performing correction processing on the signal data under the first imaging condition and the data under the second imaging condition among the signal data to reduce the difference between both sets of signal data after the correction processing, the first processing unit 151 and the second processing unit 152 perform processing. The first processing unit 151 performs the above-mentioned correction processing on the first signal data from the pixels included in the first region 141, and the second processing unit 152 performs the above-mentioned correction processing on the second signal data from the pixels included in the second region 142.

[0228] The AF calculation unit 34d performs focus detection processing based on signal data from the first processing unit 151 and the second processing unit 152, and outputs a drive signal for moving the focus lens of the imaging optical system 31 to a focus position based on the calculation result.

[0229] 3. When performing subject detection processing When different imaging conditions are set between the divided areas and the search range 190 includes the boundaries of the divided areas, in this embodiment, as described in 3-2 below, the control unit 34 (object detection unit 34a) causes the correction unit 322 to perform correction processing on the image data of at least one area within the search range 190.

[0230] 3-1. When image data in the search range 190 in FIG. 14 does not include a mixture of image data to which the first imaging condition is applied and image data to which the second imaging condition is applied In this case, the correction unit 322 does not perform the correction process, and the control unit 34 (object detection unit 34a) performs the subject detection process using the image data constituting the search range 190 as is.

[0231] 3-2. When image data in the search range 190 in FIG. 14 includes both image data to which the first imaging condition is applied and image data to which the second imaging condition is applied In this case, the control unit 34 (object detection unit 34a) causes the correction unit 322 corresponding to the block 111a to which the pixel to which the second imaging condition is applied belongs, in the image of the search range 190, to perform the correction process as described above in (Example 1) to (Example 3) for performing the focus detection process. Then, the control unit 34 (object detection unit 34a) performs the subject detection process using the image data of the pixel to which the first condition is applied and the image data after the correction process.

[0232] Fig. 24 is a diagram showing a schematic diagram of the processing of the first image data and the second image data in the subject detection processing. The correction processing performed by the first processing unit 151 and / or the second processing unit 152 is the same as the correction processing in Fig. 23 described above in the case of performing the focus detection processing.

[0233] The object detection unit 34a performs processing to detect subject elements based on the image data from the first processing unit 151 and the second processing unit 152, and outputs the detection result.

[0234] 4. Setting imaging conditions A case will be described in which the area of ​​the image capture screen is divided, different image capture conditions are set between the divided areas, and exposure conditions are determined by newly measuring the light.

[0235] 4-1. When image data in the photometric range does not include a mixture of image data to which the first imaging condition is applied and image data to which the second imaging condition is applied In this case, the correction unit 322 does not perform the correction process, and the control unit 34 (setting unit 34b) performs the exposure calculation process using the image data that constitutes the photometric range as is.

[0236] 4-2. When image data in the photometric range includes a mixture of image data to which the first imaging condition is applied and image data to which the second imaging condition is applied In this case, the control unit 34 (setting unit 34b) causes the correction unit 322 corresponding to the block 111a to which the pixel to which the second imaging condition is applied belongs, among the image data of the photometric range, to perform the correction process as described above in (Example 1) to (Example 3) for the case of performing the focus detection process. Then, the control unit 34 (setting unit 34b) performs the exposure calculation process using the image data after the correction process.

[0237] Fig. 25 is a diagram showing a schematic diagram of processing of the first image data and the second image data in relation to the setting of the imaging conditions such as the exposure calculation processing, etc. The correction processing performed by the first processing unit 151 and / or the second processing unit 152 is the same as the correction processing in Fig. 23 described above in relation to the case where the focus detection processing is performed.

[0238] The setting unit 34b performs calculation processing of the imaging conditions such as exposure calculation processing based on the image data from the first processing unit 151 and the second processing unit 152, and based on the calculation results, divides the image captured by the imaging unit 32 into multiple areas including the detected subject elements, and resets the imaging conditions for the multiple areas.

[0239] According to the second embodiment described above, the following advantageous effects can be obtained. (1) The camera 1C is capable of imaging by changing the imaging conditions for each unit section of the imaging surface, and is provided with an imaging element 32a that generates first image data from a first region consisting of at least one unit section imaged under a first imaging condition, and second image data from a second region consisting of at least one unit section imaged under a second imaging condition different from the first imaging condition. The camera 1C is provided with a plurality of correction units 322 that are provided corresponding to each unit section or each composite section having a plurality of unit sections, and that can correct image data from the corresponding unit section or the unit section in the corresponding composite section. The correction units 322 corresponding to the unit sections or composite sections in the second region correct the first image data using the second imaging condition. The imaging element 32a is provided in the back-illuminated imaging chip 111. The plurality of correction units 322 are provided in the image processing chip 114. This allows the correction processing of image data to be performed in parallel by a plurality of correction units 322, thereby reducing the processing load on the correction units 322.

[0240] (2) The back-illuminated imaging chip 111 and the image processing chip 114 are stacked together, which makes it easy to connect the imaging element 32a and the image processing unit 32c.

[0241] (3) Camera 1C includes generation unit 323 that generates an image based on the first image data corrected by correction unit 322 and the second image data. This allows pre-processing by a plurality of correction units 322 to be performed in parallel in a short time, thereby shortening the time until an image is generated.

[0242] (4) The camera 1C is capable of imaging by changing the imaging conditions for each unit section of the imaging surface, and includes an imaging element 32a that generates first image data from a first region consisting of at least one unit section in which an incident light image is imaged through the imaging optical system under a first imaging condition, and second image data from a second region consisting of at least one unit section in which the incident light image is imaged under a second imaging condition different from the first imaging condition. The camera 1C includes a plurality of correction units 322 that are provided corresponding to each unit section or each composite section having a plurality of unit sections and that can correct image data from the corresponding unit section or the unit section in the corresponding composite section. The camera 1C includes an AF calculation unit 34d that detects information for moving the imaging optical system. The correction unit 322 corresponding to the unit section or composite section in the first region corrects the first image data so that the difference between the first image data and the second image data is small. The AF calculation unit 34d detects information for moving the imaging optical system based on the first image data corrected by the correction unit 322 and the second image data. The imaging element 32a is provided in a back-illuminated imaging chip 111. The multiple correction units 322 are provided in the image processing chip 114 . As a result, the correction processing of image data can be processed in parallel by multiple correction units 322, thereby reducing the processing burden on the correction units 322. In addition, since pre-processing by the multiple correction units 322 can be performed in a short time through parallel processing, the time until focus detection processing begins in the AF calculation unit 34d can be shortened, contributing to faster focus detection processing.

[0243] (5) The camera 1C is capable of imaging by changing the imaging conditions for each unit section of the imaging surface, and includes an imaging element 32a that generates first image data from a first region consisting of at least one unit section in which an incident subject image is imaged under a first imaging condition, and second image data from a second region consisting of at least one unit section in which an incident subject image is imaged under a second imaging condition different from the first imaging condition. The camera 1C includes a plurality of correction units 322 that are provided corresponding to each unit section or each composite section having a plurality of unit sections and that can correct image data from the corresponding unit section or the unit section in the corresponding composite section. The camera 1C includes an object detection unit 34a that detects an object from the subject image. The correction unit 322 corresponding to the unit section or composite section in the first region corrects the first image data so that the difference between the first image data and the value of the second image data is small. The object detection unit 34a detects the object from the subject image based on the first image data corrected by the correction unit 322 and the second image data. The imaging element 32a is provided in a back-illuminated imaging chip 111. The multiple correction units 322 are provided in the image processing chip 114 . As a result, the correction processing of image data can be processed in parallel by multiple correction units 322, thereby reducing the processing burden on the correction units 322. In addition, pre-processing by the multiple correction units 322 can be performed in a short time through parallel processing, thereby shortening the time until the subject detection processing begins in the object detection unit 34a, thereby contributing to speeding up the subject detection processing.

[0244] (6) The camera 1C is capable of imaging by changing the imaging conditions for each unit section of the imaging surface, and includes an imaging element 32a that generates first image data from a first region consisting of at least one unit section in which an incident light image is imaged under a first imaging condition, and second image data from a second region consisting of at least one unit section in which an incident light image is imaged under a second imaging condition different from the first imaging condition. The camera 1C includes a plurality of correction units 322 that are provided corresponding to each unit section or each composite section having a plurality of unit sections and that can correct image data from the corresponding unit section or the unit section in the corresponding composite section. The camera 1C includes a setting unit 34b that sets the imaging conditions. The correction unit 322 corresponding to the unit section or composite section in the first region corrects the first image data so that the difference between the first image data and the second image data is small. The setting unit 34b sets the imaging conditions based on the first image data corrected by the correction unit 322 and the second image data. The imaging element 32a is provided in the back-illuminated imaging chip 111. The multiple correction units 322 are provided in the image processing chip 114 . As a result, the correction processing of image data can be processed in parallel by multiple correction units 322, thereby reducing the processing burden on the correction units 322. In addition, since pre-processing by the multiple correction units 322 can be performed in a short time through parallel processing, the time until the start of the setting processing of the imaging conditions in the setting unit 34b can be shortened, which contributes to speeding up the setting processing of the imaging conditions.

[0245] ---Modification of the second embodiment--- The following modifications are also within the scope of the present invention, and one or more of the modifications may be combined with the above-described embodiment. (Variation 13) We will now explain the processing of the first image data and the second image data when the first region and the second region are arranged on the imaging surface of the imaging element 32a as shown in Figures 16(a) to 16(c) in Variation 1 of the first embodiment. 16(a) to 16(c), a first image based on an image signal read from a first region and a second image based on an image signal read from a second region are generated by pixel signals read from the imaging element 32a that has captured one frame, similarly to the first modification. In this modification, similarly to the first modification, the control unit 34 uses the first image for display and the second image for detection. The imaging conditions set for the first region for capturing the first image are referred to as the first imaging conditions, and the imaging conditions set for the second region for capturing the second image are referred to as the second imaging conditions. The control unit 34 may make the first imaging conditions and the second imaging conditions different from each other.

[0246] 1. As an example, a case where a first imaging condition set in a first region is the same in the entire first region of an imaging screen, and a second imaging condition set in a second region is the same in the entire second region of an imaging screen will be described with reference to Fig. 26. Fig. 26 is a diagram that illustrates the processing of the first image data and the second image data.

[0247] First image data captured under a first imaging condition is output from each pixel included in the first region 141, and second image data captured under a second imaging condition is output from each pixel included in the second region 142. The first image data from the first region 141 is output to a first processing unit 151. Similarly, the second image data from the second region 142 is output to a second processing unit 152.

[0248] In this example, since the first imaging condition is the same throughout the first region of the imaging screen, the first processing unit 151 does not correct the first image data from the reference pixels included in the first region. Also, since the second imaging condition is the same throughout the second region of the imaging screen, the second processing unit 152 does not correct the second image data used in the focus detection process, the subject detection process, and the exposure calculation process. However, the second processing unit 152 performs a correction process on the second image data used to interpolate the first image data, to reduce the difference between the image data due to the difference between the first imaging condition and the second imaging condition. The second processing unit 152 outputs the second image data after the correction process to the first processing unit 151 as indicated by the arrow 182. Note that the second processing unit 152 may output the second image data after the correction process to the generation unit 323 as indicated by the dashed arrow 183. The second processing unit 152 receives, for example, from the first processing unit 151, information 181 about the first imaging condition required to reduce the difference between image data due to the difference in imaging condition.

[0249] The generation unit 323 performs image processing such as pixel defect correction processing, color interpolation processing, contour enhancement processing, and noise reduction processing based on the first image data from the first processing unit 151 and the second image data corrected by the second processing unit 152, and outputs the image data after image processing. The object detection unit 34a performs processing to detect subject elements based on the second image data from the second processing unit 152, and outputs the detection result. The setting unit 34b performs calculation processing of the imaging conditions such as exposure calculation processing based on the second image data from the second processing unit 152, and based on the calculation results, divides the image captured by the imaging unit 32 into multiple areas including the detected subject elements, and resets the imaging conditions for the multiple areas. The AF calculation section 34d performs focus detection processing based on the second signal data from the second processing section 152, and outputs a drive signal for moving the focus lens of the imaging optical system 31 to a focus position based on the calculation result.

[0250] 2. As another example, a case where the first imaging condition set for the first region differs depending on the region of the imaging screen, and the second imaging condition set for the second region is the same for the entire second region of the imaging screen will be described with reference to Fig. 27. Fig. 27 is a diagram that illustrates the processing of the first image data and the second image data.

[0251] Each pixel included in the first region 141 outputs first image data captured under first imaging conditions that differ depending on the region of the imaging screen, and each pixel included in the second region 142 outputs second image data captured under the same second imaging conditions throughout the entire second region of the imaging screen. The first image data from the first region 141 is output to a first processing unit 151. Similarly, the second image data from the second region 142 is output to a second processing unit 152.

[0252] As described above, in this example, the first imaging condition set in the first region differs depending on the region of the imaging screen. That is, the first imaging condition differs depending on the partial region in the first region. When different first imaging conditions are set for the pixel of interest P and the reference pixel, both of which are located in the first region, the first processing unit 151 performs a correction process similar to the correction process described in 1-2 above on the first image data from the reference pixel. Note that when the same first imaging condition is set for the pixel of interest P and the reference pixel, the first processing unit 151 does not perform a correction process on the first image data from the reference pixel.

[0253] In this example, since the second imaging conditions set for the second region are the same throughout the second region of the imaging screen, the second processing unit 152 does not correct the second image data used for focus detection processing, subject detection processing, and exposure calculation processing. For the second image data used for interpolation of the first image data, the second processing unit 152 performs a correction process to reduce the difference between the image data due to the difference between the imaging conditions for the target pixel P included in the first region and the second imaging conditions. The second processing unit 152 outputs the second image data after the correction process to the first processing unit 151. Note that the second processing unit 152 may output the second image data after the correction process to the generation unit 323. The second processing unit 152 receives, for example, from the first processing unit 151, information 181 on the imaging conditions for the pixel of interest P included in the first region, which is necessary to reduce the difference between image data due to differences in imaging conditions.

[0254] The generation unit 323 performs image processing such as pixel defect correction processing, color interpolation processing, contour enhancement processing, and noise reduction processing based on the first image data from the first processing unit 151 and the second image data corrected by the second processing unit 152, and outputs the image data after image processing. The object detection unit 34a performs processing to detect subject elements based on the second image data from the second processing unit 152, and outputs the detection result. The setting unit 34b performs calculation processing of the imaging conditions such as exposure calculation processing based on the second image data from the second processing unit 152, and based on the calculation results, divides the image captured by the imaging unit 32 into multiple areas including the detected subject elements, and resets the imaging conditions for the multiple areas. The AF calculation section 34d performs focus detection processing based on the second signal data from the second processing section 152, and outputs a drive signal for moving the focus lens of the imaging optical system 31 to a focus position based on the calculation result.

[0255] 3. As another example, a case where the first imaging condition set for the first region is the same for the entire first region of the imaging screen, and the second imaging condition set for the second region differs depending on the region of the imaging screen will be described with reference to Fig. 28. Fig. 28 is a diagram that illustrates the processing of the first image data and the second image data.

[0256] Each pixel included in the first region 141 outputs first image data captured under the same first imaging condition throughout the first region of the imaging screen, and each pixel included in the second region 142 outputs second image data captured under second imaging conditions that differ depending on the region of the imaging screen. The first image data from the first region 141 is output to a first processing unit 151. Similarly, the second image data from the second region 142 is output to a second processing unit 152.

[0257] In this example, since the first imaging conditions set for the first region are the same throughout the first region of the imaging screen, the first processing unit 151 does not correct the first image data from the reference pixels included in the first region.

[0258] In this example, since the second imaging conditions set for the second region differ depending on the region of the imaging screen, the second processing unit 152 performs a correction process on the second image data as follows: For example, the second processing unit 152 performs a correction process on the second image data captured under a certain imaging condition among the second image data, thereby reducing the difference between the second image data after the correction process and the second image data captured under another imaging condition different from the above-mentioned certain imaging condition.

[0259] In this example, the second processing unit 152 performs a correction process on the second image data used for the interpolation of the first image data to reduce a difference between the image data due to a difference between the imaging condition for the pixel of interest P included in the first region and the second imaging condition. The second processing unit 152 outputs the second image data after the correction process to the first processing unit 151. The second processing unit 152 may output the second image data after the correction process to the generation unit 323. In FIG. 26, the second image data after the correction process output to the first processing unit 151 is indicated by the reference numeral 182, and the second image data after the correction process output to the generation unit 323 is indicated by the reference numeral 183. The second processing unit 152 receives, for example, from the first processing unit 151, information 181 on the imaging conditions for the pixel of interest P included in the first region, which is necessary to reduce the difference between image data due to differences in imaging conditions.

[0260] The generation unit 323 performs image processing such as pixel defect correction processing, color interpolation processing, contour enhancement processing, and noise reduction processing based on the first image data from the first processing unit 151 and the second image data corrected by the second processing unit 152, and outputs the image data after image processing. The object detection unit 34a performs a process of detecting subject elements based on second image data captured under certain imaging conditions and second image data captured under other imaging conditions, which have been corrected by the second processing unit 152, and outputs the detection result. The setting unit 34b performs calculation processing of the imaging conditions such as exposure calculation processing based on the second image data captured under certain imaging conditions and the second image data captured under other imaging conditions, which have been corrected by the second processing unit 152. Based on the calculation results, the setting unit 34b divides the image captured by the imaging unit 32 into multiple areas including the detected subject elements, and resets the imaging conditions for the multiple areas. The AF calculation unit 34d performs focus detection processing based on the second signal data captured under certain imaging conditions and the second signal data captured under other imaging conditions, which have been corrected by the second processing unit 152. The AF calculation unit 34d outputs a drive signal for moving the focus lens of the imaging optical system 31 to a focus position based on the calculation result.

[0261] 4. Furthermore, as another example, a case where the first imaging condition set for the first area differs depending on the area of ​​the imaging screen, and the second imaging condition set for the second area differs depending on the area of ​​the imaging screen will be described with reference to Fig. 29. Fig. 29 is a diagram that illustrates the processing of the first image data and the second image data.

[0262] Each pixel included in the first region 141 outputs first image data captured under a first imaging condition that differs depending on the region of the imaging screen, and each pixel included in the second region 142 outputs second image data captured under a second imaging condition that differs depending on the region of the imaging screen. The first image data from the first region 141 is output to a first processing unit 151. Similarly, the second image data from the second region 142 is output to a second processing unit 152.

[0263] As described above, in this example, the first imaging condition set in the first region differs depending on the region of the imaging screen. That is, the first imaging condition differs depending on the partial region in the first region. When different first imaging conditions are set for the pixel of interest P and the reference pixel, both of which are located in the first region, the first processing unit 151 performs a correction process similar to the correction process described in 1-2 above on the first image data from the reference pixel. Note that when the same first imaging condition is set for the pixel of interest P and the reference pixel, the first processing unit 151 does not perform a correction process on the first image data from the reference pixel.

[0264] Furthermore, in this example, since the second imaging condition set for the second region differs depending on the region of the imaging screen, the second processing unit 152 performs correction processing on the second image data as an example of the above-mentioned 3.

[0265] The generation unit 323 performs image processing such as pixel defect correction processing, color interpolation processing, contour enhancement processing, and noise reduction processing based on the first image data from the first processing unit 151 and the second image data corrected by the second processing unit 152, and outputs the image data after image processing. The object detection unit 34a performs a process of detecting subject elements based on second image data captured under certain imaging conditions and second image data captured under other imaging conditions, which have been corrected by the second processing unit 152, and outputs the detection result. The setting unit 34b performs calculation processing of the imaging conditions such as exposure calculation processing based on the second image data captured under certain imaging conditions and the second image data captured under other imaging conditions, which have been corrected by the second processing unit 152. Based on the calculation results, the setting unit 34b divides the image captured by the imaging unit 32 into multiple areas including the detected subject elements, and resets the imaging conditions for the multiple areas. The AF calculation unit 34d performs focus detection processing based on the second signal data captured under certain imaging conditions and the second signal data captured under other imaging conditions, which have been corrected by the second processing unit 152. The AF calculation unit 34d outputs a drive signal for moving the focus lens of the imaging optical system 31 to a focus position based on the calculation result.

[0266] (Variation 14) In the second embodiment described above, one of the correction units 322 corresponds to one of the blocks 111a (unit divisions). However, one of the correction units 322 may correspond to one of the composite blocks (composite divisions) having a plurality of blocks 111a (unit divisions). In this case, the correction unit 322 sequentially corrects image data from pixels belonging to a plurality of blocks 111a included in the composite block. Even if a plurality of correction units 322 are provided corresponding to each composite block having a plurality of blocks 111a, the correction process of the image data can be processed in parallel by the plurality of correction units 322, so that the processing load on the correction unit 322 can be reduced, and an appropriate image can be generated in a short time from image data generated in areas with different imaging conditions.

[0267] (Variation 15) In the second embodiment described above, the generation unit 323 is provided inside the imaging unit 32A. However, the generation unit 323 may be provided outside the imaging unit 32A. Even if the generation unit 323 is provided outside the imaging unit 32A, the same effects as those described above can be achieved.

[0268] (Variation 16) In the above-described second embodiment, the stacked imaging element 100A further includes an image processing chip 114 that performs the above-described pre-processing and image processing, in addition to the back-illuminated imaging chip 111, the signal processing chip 112, and the memory chip 113. However, the stacked imaging element 100A may not include the image processing chip 114, and the image processing unit 32c may be provided in the signal processing chip 112.

[0269] (Variation 17) In the above-described second embodiment, the second processing unit 152 receives information on the first imaging condition required to reduce the difference between the image data due to the difference between the imaging conditions from the first processing unit 151. Also, the first processing unit 151 receives information on the second imaging condition required to reduce the difference between the image data due to the difference between the imaging conditions from the second processing unit 152. However, the second processing unit 152 may receive information on the first imaging condition required to reduce the difference between the image data due to the difference between the imaging conditions from the driving unit 32b or the control unit 34. Similarly, the first processing unit 151 may receive information on the second imaging condition required to reduce the difference between the image data due to the difference between the imaging conditions from the driving unit 32b or the control unit 34. The above-described embodiments and modifications may be combined with each other.

[0270] Although various embodiments and modifications have been described above, the present invention is not limited to these. Other embodiments that are conceivable within the scope of the technical idea of ​​the present invention are also included within the scope of the present invention. The above-mentioned embodiments and modifications also include an imaging device, an image processing device, and a display device as follows: (1) An imaging device including an imaging element having a first imaging area that images a subject and outputs a first signal, and a second imaging area that images the subject and outputs a second signal, a setting unit that sets an imaging condition of the first imaging area to an imaging condition different from an imaging condition of the second imaging area, a correction unit that corrects the second signal output from the second imaging area for use in interpolating the first signal output from the first imaging area, and a generation unit that generates an image of the subject imaged in the first imaging area by using a signal obtained by interpolating the first signal with the second signal corrected by the correction unit. (2) In an imaging device such as (1), the correction unit performs correction on the second signal output from the second imaging area based on imaging conditions of the first imaging area and imaging conditions of the second imaging area, as a correction for use in interpolation of the first signal output from the first imaging area. (3) In an imaging device such as (2), the correction unit performs correction on the second signal output from the second imaging area based on a difference between imaging conditions of the first imaging area and imaging conditions of the second imaging area, as a correction for use in interpolation of the first signal output from the first imaging area. (4) In an imaging device such as (3), the imaging conditions of the first imaging area and the imaging conditions of the second imaging area are exposure conditions, and the correction unit corrects the second signal output from the second imaging area based on the difference between the exposure conditions of the first imaging area and the exposure conditions of the second imaging area. (5) In an imaging device such as (4), the exposure condition of the first imaging area and the exposure condition of the second imaging area are charge accumulation times of the light receiving sections of the imaging areas, and the correction section corrects the second signal output from the second imaging area based on the ratio between the charge accumulation time of the first imaging area and the charge accumulation time of the second imaging area. (6) In an imaging device such as (4), the exposure condition of the first imaging area and the exposure condition of the second imaging area are the imaging sensitivities of the imaging areas, and the correction unit corrects the second signal output from the second imaging area based on the logarithmic ratio between the imaging sensitivity of the first imaging area and the imaging sensitivity of the second imaging area. (7) In an imaging device such as (1), the correction unit performs correction on the second signal output from the second imaging area based on the imaging conditions of the first imaging area or the imaging conditions of the second imaging area as a correction for use in interpolation of the first signal output from the first imaging area. (8) In an imaging device such as (1), the correction unit corrects the first signal output from the first imaging area and performs correction on the second signal output from the second imaging area for use in interpolating the corrected first signal, and the generation unit generates an image of the subject imaged in the first imaging area using a signal obtained by interpolating the corrected first signal with the second signal corrected by the correction unit. (9) In an imaging device such as those described in (1) to (8), the first imaging region photoelectrically converts light of a first color from among light from a subject to output the first signal, and the second imaging region photoelectrically converts light of a second color different from the first color from among light from the subject to output the second signal. (10) An imaging device comprising: an imaging element having a first imaging area that images a subject and outputs a first signal; and a second imaging area that images the subject and outputs a second signal; a setting unit that sets imaging conditions of the first imaging area to imaging conditions different from imaging conditions of the second imaging area; a correction unit that corrects the second signal output from the second imaging area for use in interpolating pixels that output the first signal of the first imaging area; and a generation unit that generates an image of the subject imaged in the first imaging area using a signal obtained by interpolating pixels that output the first signal using the second signal corrected by the correction unit. (11) In the imaging device as described in (10), a pixel in the first imaging region that outputs the first signal photoelectrically converts light of a first color from light from a subject to output the first signal, and the second imaging region photoelectrically converts light of a second color different from the first color from light from the subject to output the second signal. (12) An imaging device comprising: an imaging element having a first imaging region that captures an image of a subject and outputs a first signal, and a second imaging region that captures an image of the subject and outputs a second signal, a setting unit that sets an imaging condition of the first imaging region to an imaging condition different from that of the second imaging region, a correction unit that corrects the second signal output from the second imaging region to reduce noise contained in the first signal output from the first imaging region, and a generation unit that generates an image of the subject imaged in the first imaging region by reducing noise contained in the first signal using the second signal corrected by the correction unit. (13) In an imaging device such as (12), the correction unit performs correction on the second signal output from the second imaging area based on imaging conditions of the first imaging area or imaging conditions of the second imaging area as a correction to reduce noise contained in the first signal output from the first imaging area. (14) In an imaging device such as (12), the correction unit performs correction on the second signal output from the second imaging area based on imaging conditions of the first imaging area and imaging conditions of the second imaging area to reduce noise contained in the first signal output from the first imaging area. (15) In an imaging device such as (14), the correction unit performs correction on the second signal output from the second imaging area based on a difference between imaging conditions of the first imaging area and imaging conditions of the second imaging area, as a correction for use in interpolation of the first signal output from the first imaging area. (16) In an imaging device such as (15), the imaging conditions of the first imaging area and the imaging conditions of the second imaging area are exposure conditions, and the correction unit corrects the second signal output from the second imaging area based on a difference between the exposure conditions of the first imaging area and the exposure conditions of the second imaging area. (17) In an imaging device such as (16), the exposure condition of the first imaging area and the exposure condition of the second imaging area are charge accumulation times of the light receiving sections of the imaging areas, and the correction section corrects the second signal output from the second imaging area based on the ratio between the charge accumulation time of the first imaging area and the charge accumulation time of the second imaging area. (18) In an imaging device such as (16), the exposure condition of the first imaging region and the exposure condition of the second imaging region are the imaging sensitivities of the imaging regions, and the correction unit corrects the second signal output from the second imaging region based on a logarithmic ratio between the imaging sensitivity of the first imaging region and the imaging sensitivity of the second imaging region. (19) In an imaging device such as (15), the correction unit corrects the first signal output from the first imaging area and performs correction on the second signal output from the second imaging area to reduce noise contained in the corrected first signal, and the generation unit reduces the noise contained in the corrected first signal using the second signal corrected by the correction unit to generate an image of the subject imaged in the first imaging area. (20) An imaging device comprising: an imaging element having a first imaging area that images a subject and outputs a first signal; and a second imaging area that images the subject and outputs a second signal; a setting unit that sets imaging conditions of the first imaging area to imaging conditions different from imaging conditions of the second imaging area; a correction unit that corrects the second signal output from the second imaging area for use in image processing of the first signal output from the first imaging area; and a generation unit that processes the first signal using the second signal corrected by the correction unit to generate an image of the subject imaged in the first imaging area. (21) An imaging device comprising: an imaging element having a first imaging area that images a subject and outputs a first signal; and a second imaging area that images the subject and outputs a second signal; a setting unit that sets imaging conditions for the first imaging area and imaging conditions for the second imaging area; a processing unit that performs processing that differs depending on the imaging conditions set for the second imaging area by the setting unit as a process of interpolating the first signal using the second signal output from the second imaging area; and a generation unit that generates an image of the subject imaged in the first imaging area using a signal obtained by interpolating the first signal by the processing unit. (22) In an imaging device such as (21), the processing unit performs different processing, as a process of interpolating the first signal using the second signal output from the second imaging area, depending on whether a first imaging condition is set for the first imaging area and the second imaging area by the setting unit or whether the first imaging condition is set for the first imaging area and a second imaging condition different from the first imaging condition is set for the second imaging area. (23) In an imaging device such as (22), the processing unit has a correction unit that, when the setting unit sets the first imaging condition in the first imaging area and the second imaging condition different from the first imaging condition in the second imaging area, corrects the second signal output from the second imaging area for use in interpolating the first signal output from the first imaging area, and the generation unit generates an image of the subject imaged in the first imaging area using a signal obtained by interpolating the first signal using the second signal corrected by the correction unit, as a signal obtained by interpolating the first signal by the processing unit. In an imaging device such as (24)(22), when the first imaging condition is set for the first imaging area and the second imaging area by the setting unit, the generation unit generates an image of the subject imaged in the first imaging area using a signal obtained by interpolating the first signal by the processing unit and a signal obtained by interpolating the first signal by the second signal. (25) An imaging device comprising: an imaging element having a first imaging area that images a subject and outputs a first signal; and a second imaging area that images the subject and outputs a second signal; a setting unit that sets imaging conditions of the first imaging area to imaging conditions different from imaging conditions of the second imaging area; a correction unit that corrects the first signal output from the first imaging area and performs correction on the second signal output from the second imaging area for use in interpolation of the corrected first signal; and a generation unit that generates an image of the subject imaged in the first imaging area using a signal obtained by interpolating the first signal corrected by the correction unit using the second signal corrected by the correction unit. (26) In an imaging device such as (25), the correction unit corrects the first signal based on a setting value different from the setting value of the imaging conditions of the first imaging area and the setting value of the imaging conditions of the second imaging area, and corrects the second signal based on the setting value of the imaging conditions of the second imaging area and the different setting value for use in interpolating the corrected first signal. In an imaging device such as (27)(25), the correction unit corrects the first signal based on the difference between a setting value of an imaging condition of the first imaging area and the different setting value, and corrects the second signal based on the difference between a setting value of an imaging condition of the second imaging area and the different setting value. In an imaging device such as (28)(27), the correction unit corrects the first signal based on a ratio of the different setting value to a setting value of an imaging condition of the first imaging area, and corrects the second signal based on a ratio of the different setting value to a setting value of an imaging condition of the second imaging area. (29) An imaging device comprising: an imaging element having a first imaging area that images a subject and outputs a first signal; and a second imaging area that images the subject and outputs a second signal; a setting unit that sets an imaging condition of the first imaging area to an imaging condition different from an imaging condition of the second imaging area; a correction unit that corrects the first signal output from the first imaging area and performs correction on the second signal output from the second imaging area for use in interpolation of pixels that output the corrected first signal; and a generation unit that generates an image of the subject imaged in the first imaging area using a signal obtained by interpolating pixels that output the first signal corrected by the second signal corrected by the correction unit. In an imaging device such as (30)(29), a pixel in the first imaging region that outputs the first signal performs photoelectric conversion on light of a first color from among light from a subject, and outputs the first signal, and the second imaging region performs photoelectric conversion on light of a second color different from the first color from among light from a subject, and outputs the second signal. (31) An image processing device comprising: a correction unit that performs correction on a second signal output from a second imaging area of ​​an image sensor that images a subject, the second signal having imaging conditions different from those of the first imaging area, for use in interpolation of a first signal output from the first imaging area of ​​the image sensor that images a subject; and a generation unit that generates an image of the subject imaged in the first imaging area by using a signal obtained by interpolating the first signal using the second signal corrected by the correction unit. (32) An image processing device comprising: a correction unit that performs correction on a second signal output from a second imaging area of ​​an image sensor that images a subject, the second signal having imaging conditions different from those of the first imaging area, in order to reduce noise contained in the first signal output from the first imaging area of ​​the image sensor; and a generation unit that uses the second signal corrected by the correction unit to reduce noise contained in the first signal and generate an image of the subject imaged in the first imaging area. (33) A display device comprising: a correction unit that performs correction on a second signal output from a second imaging area of ​​an imaging element that images a subject, the second signal having imaging conditions different from those of the first imaging area, for use in interpolation of a first signal output from the first imaging area of ​​the imaging element that images the subject; and a display unit that displays an image of the subject imaged in the first imaging area using a signal obtained by interpolating the first signal using the second signal corrected by the correction unit. (34) A display device comprising: a correction unit that performs correction on a second signal output from a second imaging region of an imaging element that images a subject, the second signal having imaging conditions different from those of the first imaging region, in order to reduce noise contained in the first signal output from the first imaging region of the imaging element; and a display unit that displays an image of the subject imaged in the first imaging region, the image being generated by reducing the noise contained in the first signal using the second signal corrected by the correction unit. (35) An imaging device comprising: an imaging unit having a first region that images incident light and a second region different from the first region that images the incident light; a setting unit that sets imaging conditions for the first region to be different from imaging conditions for the second region; a correction unit that corrects first image data generated by imaging light incident on the first region under the imaging conditions set by the setting unit based on the imaging conditions for the second region set by the setting unit; and a generation unit that generates an image using the first image data corrected by the correction unit and second image data generated by imaging light incident on the second region. (36) An imaging device comprising: an imaging unit having a first region that captures incident light and a second region different from the first region that captures the incident light; a setting unit that sets imaging conditions for the first region to be different from imaging conditions for the second region; a correction unit that corrects first image data generated by capturing light incident on the first region under the imaging conditions set by the setting unit based on the imaging conditions for the second region set by the setting unit and corrects second image data generated by capturing light incident on the second region under the imaging conditions set by the setting unit based on the imaging conditions for the first region set by the setting unit; and a generation unit that generates an image using the first image data corrected by the correction unit and the second image data corrected by the correction unit. (37) An image processing device comprising: a setting unit that sets imaging conditions of a first region of an imaging unit to be different from imaging conditions of a second region of the imaging unit; a correction unit that corrects first image data generated by capturing light incident on the first region under the imaging conditions set by the setting unit based on the imaging conditions of the second region set by the setting unit; and a generation unit that generates an image using the first image data corrected by the correction unit and second image data generated by capturing light incident on the second region. (38) An image processing device comprising: a setting unit that sets imaging conditions of a first region of an imaging unit to be different from imaging conditions of a second region of the imaging unit; a correction unit that corrects first image data generated by imaging light incident on the first region under the imaging conditions set by the setting unit based on the imaging conditions of the second region set by the setting unit, and corrects second image data generated by imaging light incident on the second region under the imaging conditions set by the setting unit based on the imaging conditions of the first region set by the setting unit; and a generation unit that generates an image using the first image data corrected by the correction unit and the second image data generated by imaging light incident on the second region. (39) A display device comprising: a setting unit that sets imaging conditions of a first region of an imaging unit to be different from imaging conditions of a second region of the imaging unit; a correction unit that corrects first image data generated by imaging light incident on the first region under the imaging conditions set by the setting unit based on the imaging conditions of the second region set by the setting unit; and a display unit that displays an image generated from the first image data corrected by the correction unit and second image data generated by imaging light incident on the second region. (40) A display device comprising: a setting unit that sets imaging conditions of a first region of an imaging unit to be different from imaging conditions of a second region of the imaging unit; a correction unit that corrects first image data generated by imaging light incident on the first region under the imaging conditions set by the setting unit based on the imaging conditions of the second region set by the setting unit, and corrects second image data generated by imaging light incident on the second region under the imaging conditions set by the setting unit based on the imaging conditions of the first region set by the setting unit; and a display unit that displays an image generated from the first image data corrected by the correction unit and the second image data generated by imaging light incident on the second region.

[0271] The above-described embodiment and modifications also include an imaging device, an image processing device, and a display device as described below. (1) An imaging device comprising: an imaging unit having a first region that images incident light and a second region different from the first region that images the incident light; a setting unit that sets imaging conditions for the first region to be different from imaging conditions for the second region; a correction unit that corrects first image data generated by imaging light incident on the first region under the imaging conditions set by the setting unit based on the imaging conditions for the second region set by the setting unit; and a generation unit that generates an image using the first image data corrected by the correction unit and second image data generated by imaging light incident on the second region. (2) In the imaging device as described in (1), the correction unit corrects the first image data so that a difference between a value of the first image data and a value of the second image data becomes small. (3) In the imaging device as described in (1) or (2), the correction unit corrects the first image data based on a difference between an imaging condition of the first region and an imaging condition of the second region. (4) In the imaging device as described in (1) to (3), the generation unit performs image processing on the second image data using the first image data corrected by the correction unit. (5) In the imaging device as described in (1) to (4), the generation unit changes values ​​of the second image data using values ​​of the first image data corrected by the correction unit. (6) In the imaging device as described in (1) to (5), the generation unit replaces values ​​of the second image data with values ​​of the first image data corrected by the correction unit. (7) In the imaging device as described in (1) to (6), the generating unit generates third image data from the second image data and the first image data corrected by the correcting unit. (8) In an imaging device such as those described in (1) to (7), the setting unit sets the imaging sensitivity as an imaging condition for a first region of the imaging unit to a first imaging sensitivity and sets the imaging sensitivity as an imaging condition for a second region of the imaging unit to a second imaging sensitivity different from the first imaging sensitivity, and the correction unit corrects the first image data generated by imaging under the imaging conditions set by the setting unit based on a difference between the first imaging sensitivity and the second imaging sensitivity. (9) An imaging device comprising: an imaging unit having a first region that images incident light; and a second region different from the first region that images the incident light; a setting unit that sets imaging conditions for the first region to be different from imaging conditions for the second region; a correction unit that corrects first image data generated by imaging the light incident on the first region under the imaging conditions set by the setting unit based on the imaging conditions for the second region set by the setting unit, and corrects second image data generated by imaging the light incident on the second region under the imaging conditions set by the setting unit based on the imaging conditions for the first region set by the setting unit; and a generation unit that generates an image using the first image data corrected by the correction unit and the second image data corrected by the correction unit. In the imaging device as described in (10) and (9), the correction unit corrects the first image data and the second image data so that a difference between a value of the first image data and a value of the second image data becomes small. (11) In the imaging device as described in (10), the correction unit corrects the first image data so as to increase the value thereof and to decrease the value thereof. (12) In an imaging device such as (9) to (11), the correction unit corrects the first image data based on the imaging conditions of the first area and the imaging conditions of the second area set by the setting unit, and corrects the second image data based on the imaging conditions of the first area and the imaging conditions of the second area. (13) In the imaging device as described in (12), the correction unit corrects the first image data and the second image data based on a difference between an imaging condition of the first region and an imaging condition of the second region. (14) In an imaging device such as those described in (9) to (13), the generation unit performs image processing on the second image data corrected by the correction unit using the first image data corrected by the correction unit. (15) In the imaging device as described in (9) to (14), the generation unit corrects the generated image. (16) An image processing device comprising: a setting unit that sets imaging conditions of a first region of an imaging unit to be different from imaging conditions of a second region of the imaging unit; a correction unit that corrects first image data generated by capturing light incident on the first region under the imaging conditions set by the setting unit based on the imaging conditions of the second region set by the setting unit; and a generation unit that generates an image using the first image data corrected by the correction unit and second image data generated by capturing light incident on the second region. In the image processing device as described in (17) and (16), the correction unit corrects the first image data so that a difference between a value of the first image data and a value of the second image data becomes small. (18) In the image processing device as described in (16) or (17), the correction unit corrects the first image data based on a difference between an imaging condition of the first region and an imaging condition of the second region. (19) An image processing device comprising: a setting unit that sets imaging conditions of a first region of an imaging unit to be different from imaging conditions of a second region of the imaging unit; a correction unit that corrects first image data generated by imaging light incident on the first region under the imaging conditions set by the setting unit based on the imaging conditions of the second region set by the setting unit, and corrects second image data generated by imaging light incident on the second region under the imaging conditions set by the setting unit based on the imaging conditions of the first region set by the setting unit; and a generation unit that generates an image using the first image data corrected by the correction unit and the second image data generated by imaging light incident on the second region. In an image processing device such as (20)(19), the correction unit corrects the first image data and the second image data so that a difference between values ​​of the first image data and values ​​of the second image data becomes smaller. In an image processing device such as (21), (19) or (20), the correction unit corrects the first image data and the second image data based on a difference between imaging conditions of the first area and imaging conditions of the second area. (22) A display device comprising: a setting unit that sets imaging conditions of a first region of an imaging unit to be different from imaging conditions of a second region of the imaging unit; a correction unit that corrects first image data generated by imaging light incident on the first region under the imaging conditions set by the setting unit based on the imaging conditions of the second region set by the setting unit; and a display unit that displays an image generated from the first image data corrected by the correction unit and second image data generated by imaging light incident on the second region. (23) In the display device as described in (22), the correction unit corrects the first image data so that a difference between a value of the first image data and a value of the second image data becomes small. (24) In the display device as described in (22) or (23), the correction unit corrects the first image data based on a difference between an imaging condition of the first region and an imaging condition of the second region. (25) A display device comprising: a setting unit that sets imaging conditions of a first region of an imaging unit to be different from imaging conditions of a second region of the imaging unit; a correction unit that corrects first image data generated by imaging light incident on the first region under the imaging conditions set by the setting unit based on the imaging conditions of the second region set by the setting unit, and corrects second image data generated by imaging light incident on the second region under the imaging conditions set by the setting unit based on the imaging conditions of the first region set by the setting unit; and a display unit that displays an image generated from the first image data corrected by the correction unit and the second image data generated by imaging light incident on the second region. In a display device such as (26) and (25), the correction unit corrects the first image data and the second image data so that a difference between a value of the first image data and a value of the second image data becomes smaller. In a display device such as (27) (25) or (26), the correction unit corrects the first image data and the second image data based on a difference between imaging conditions of the first region and imaging conditions of the second region. (28) In an imaging device such as those described in (1) to (15), the correction unit has a first correction unit that corrects the first image data generated by capturing light incident on the first region under the imaging conditions set by the setting unit, and a second correction unit that corrects the second image data generated by capturing light incident on the second region under the imaging conditions set by the setting unit, the first correction unit corrects the first image data based on the imaging conditions of the second region, and the generation unit generates an image using the first image data corrected by the first correction unit and the second image data. In the imaging device as described in (29) and (28), the first correction unit corrects the first image data so that a difference between a value of the first image data and a value of the second image data becomes small. In the imaging device as described in (30), (28) or (29), the first correction unit corrects the first image data based on a difference between an imaging condition of the first region and an imaging condition of the second region. (31) In an imaging device such as (28) to (30), the second correction unit corrects the second image data based on imaging conditions of the first area, and the generation unit generates an image using the first image data corrected by the first correction unit and the second image data corrected by the second correction unit. (32) In an imaging device such as (28) to (31), the second correction unit corrects the second image data using the first image data, and the generation unit generates an image using the first image data corrected by the first correction unit and the second image data corrected by the second correction unit. (33) In the imaging device as described in (28) to (32), the imaging section is provided on a first semiconductor substrate, and the first correction section and the second correction section are provided on a second semiconductor substrate. In an imaging device such as (34) or (33), the first semiconductor substrate and the second semiconductor substrate are stacked.

[0272] The disclosures of the following priority applications are incorporated herein by reference: Japanese Patent Application No. 2015-194612 (filed September 30, 2015) [Explanation of symbols]

[0273] 1,1C…Camera 1B…Imaging system 32…Imaging unit 32a, 100...imaging element 33...Image processing unit 33a, 321...input section 33b,322...Correction section 33c,323...Generation part 34...Control section 34a...Object detection unit 34b…Setting section 34c...imaging control unit 34d…AF calculation section 35...Display section 90...Predetermined range 1001...imaging device 1002...Display device P…pixel of interest

Claims

[Claim 1] an image sensor having a first imaging region that captures an image of a subject and outputs a first signal, and a second imaging region that captures an image of the subject and outputs a second signal; A setting unit that sets an imaging condition of the first imaging region to an imaging condition different from an imaging condition of the second imaging region; a correction unit that performs correction on the second signal output from the second imaging region for use in interpolation of the first signal output from the first imaging region; a generation unit that generates an image of a subject imaged in the first imaging area using a signal obtained by interpolating the first signal with the second signal corrected by the correction unit, and generates an image of a subject imaged in the second imaging area using the second signal that has not been corrected for use in interpolating the first signal.

Images